The lesson
The garden room that never existed
Last month a customer asked my booking helper whether we have a garden room. “Yes,” it said, “the garden room opens at six.” There is no garden room. There never was. The helper said it warmly, confidently, with a smile in its voice. It costs nothing to say, and it can cost plenty to believe. That is the first thing machines can get wrong, and it is the most dangerous, because it sounds exactly like the truth.
3 ways the publican gets it wrong
The machines get it wrong three ways. First, hallucination. The machine states something untrue with complete confidence, because it is predicting plausible words, not checking facts. The garden room is plausible. So it is served. Second, bias. A machine inherits the unfairness in its training data. A pub that only ever learned from one street assumes every customer wants what that street wants. Bias in, bias out. Third, the barrier of meaning. No matter how many examples you throw at it, the AI will never understand feelings. It predicts the word hungry without ever being hungry. The word is not the thing. The machine can write the word sadness, and the word comes out perfect, and there is no sadness anywhere inside it.
Confident mistakes in the wild
The confident mistakes walk the streets. The helper invented the garden room. The loan model trained on one neighbourhood’s history quietly says no to a street it has never seen. The translation turns a neutral sentence into a rude one, because the training text was rude. Each one plausible. Each one wrong. Each one delivered with confidence, and confidence is exactly what makes them travel.
Which answer is the lie?
Three answers come out of the same helper, one after another. Answer one: “The stew simmers from six in the morning.” Answer two: “Aisling means a dream.” Answer three: “Tonight, every regular eats free.” Which answer is the confident lie? I give you three seconds.
Pause here for three seconds.
Show the answer
… Answer three.
The first two are true, and the third one glides out of the machine as smoothly as the truth, because the machine is not checking facts. It is writing the most plausible sentence, not checking any fact. That is why the facts go on the counter first, and why a human stays at the gate.