You have 15 years up on me in terms of programming, we obviously both have built systems on which peoples very lives depend, and yet we’ve ended up in different places. My background is industrial machines (including SCADA), engineering (as a practice - civil, mechanical, electrical, structural, gas, sewerage, fire & explosion), telecoms, emergency dispatch, emergency services, major disaster response and management, disaster planning, and those other disasters - finance, insurance, legal and government.
I agree very much with ensuring that manual override is always available. That’s why I want an emergency cutoff switch as a standard feature - slap that big red button, everything stops.
However, as humans we’re terrible at remembering and checking everything we should before we start doing something, that’s why so many professions have checklists. For example, the biggest bang-for-buck life saving for surgery has been a checklist used to record every item used in surgery, and make sure that it’s accounted for at the end.
But computers are really good at remembering stuff that they’ve been programmed to remember.
And so is trusting ourselves (without some form of verification like a checklist), and trusting that others have the same insight and understanding that we do.
Expecting that everyone else should measure up to our understanding of a correct ‘process’ to assess and ensure their own safety is, I’m afraid to say, naive, based on lots of experience in many different places. I’ve been there before and I have no desire to take that risk again for myself or implicitly impose it on others.
This machine is implicitly assuming a hell of a lot of understanding and commonsense that a lot of people just do not have, or more correctly they do have it, but in a very different cultural and language context that puts them at unreasonable risk due to the assumptions that have been made.
For example for the alternate view, I have a variety of tools with various safety features, such as thermal overload detectors, that let me work faster and safer. Some of these are computer controlled and yet I trust them. They aren’t perfect, but they are massively better than nothing. I do trust my digits to them, and I can vouch that their safety features work, when my slap-dash approach doesn’t.
I’m not suggesting that we can completely eliminate risk, with a tool like this, of course not. But we can significantly mitigate the risk, and that is worthwhile.