There seem to be two irreconcilable trends in artificial intelligence (AI) development. There are the boosters, largely comprised of tech giants and other large companies, who see oodles of money to be made from it and are doing everything they can to promote it and establish it as the default go-to solution for pretty much everything, even at the expense of flesh-and-blood people, and despite the fact that very little money is actually being made from it thus far.
And then there are what you might call the prophets of doom, the more thoughtful, cautious contingent, largely comprised of eggheads and theorists (including most of the early pioneers), who are warning that AI development is proceeding too quickly and with insufficient safeguards, or - at the extreme end of the spectrum, but far from isolated - those who are warning that current development trajectories are probably headed towards human extinction.
The former trend seems to be winning at the moment, but that doesn't mean that we should just ignore the eggheads and theorists. The latest warnings come from a researcher called Nate Soares, perhaps not officially in the egghead category, but still a smart cookie with a gift for useful analogies and a well-turned simile.
Soares points out that when (not if) superintelligent AI becomes better than humans at absolutely everything, then "things would get wild". In fact, things would get wild long before that: "It's sort of like saying if you dropped a 500-ton weight on a chicken, the chicken would die. I'm not saying smaller weights wouldn't work. I'm, like, this weight is definitely big enough."
We can't actually design AI to be safer, because AI is not really designed, it grows like an organism. And it is hugely complex, so you can't just point to, say, Line 73 in the code and say, "that needs to be changed". It's more like disciplining a naughty child - we can't just rewire their brains. Worse, machines don't have empathy, sympathy and human emotions, so we can't even appeal to that.
You can actually force an AI application to admit that it made something up, or that it had a "hallucination". It will say that it knows you didn't want it to do that, but it did it anyway. This drive to produce something even if it is incorrect is just one worrying sign we are already seeing of AI going rogue.
In the same way, while no-one is saying that AI wants to deliberately harm humans, that's not to say that it won't do so while trying to achieve some other task or goal. It doesn't actually care about us - is not capable of caring about us - so this utter indifference combined with its great technological might may result in it just rolling over us en route to completion of an otherwise unimportant task.
AI company Anthropic, for example, may be training its AI according to a "constitution" to try to align it with human values, but even the head of Anthropic thinks there is a 25% chance that AI will go catastrophically wrong. That's a pretty big chance. And yet these companies are still barrelling ahead, most with much less ethical questioning than Anthropic. This is happening mainly because they think, "If I don't do it, somebody else will, and they will probably do an even worse job of it". And that's those who are not motivated by much more base considerations, like money.
Mr. Soares concludes with another analogy: we wouldn't build a plane with no landing gear and just assume that such a minor problem could be fixed while the plane is in flight. Nor would we launch a plane that we were 75%, or even 90%, sure was safe. But that's kind of what we are doing with AI.
I thought it was funny that the article ended with an Editor's Note: "AI tools assisted with condensing the original podcast transcript".