On the heels of warnings from OpenAI’s Sam Altman and Anthropic’s Dario Amodei comes reassurance from Nvidia’s Jensen Huang. The industry leaders building the future of AI appear to have very different views of where it is taking us.
Huang puts the chance of the world ending by 2030 at “0%.” (The Guardian)
That is quite a degree of certainty. But certainty about what?
Even if we accept that AI will not destroy the world, I can think of an enormous range of really bad outcomes between here and the end of humanity. A business can make damaging decisions. A patient can receive an inappropriate recommendation. These are some moderately not-good outcomes that we should strive to avoid. The world is not destroyed, but the consequences for the people affected are very real.
How do we address those outcomes? That is why we cannot sit back and relax.
For enterprises, I believe a central question is what happens to the people who understand the business as AI takes on more of its reasoning.
We ask AI a question. It responds with an answer and a detailed explanation. We read it, perhaps ask a follow-up question, and accept it. There is a human in the loop. But what is that human actually doing? Just saying, “Yes, approved”?
There is a significant difference between AI reasoning while humans nod yes, and humans actively participating in the reasoning. A person can approve an answer without examining its assumptions. A team can accept an analysis without contributing the experience that would have changed it. The presence of a human does not, by itself, establish the presence of human judgment.
Of course we need controls. We need permissions, checks, approvals, and limits on what agents can do. But accepting reasoning from AI and then putting controls around it is necessary, not sufficient. An agent can stay within its permissions and still apply the wrong business logic. An approved analysis can still overlook something an experienced person would have recognized.
I hope Jensen Huang is right. His confidence is reassuring, but it might also lull us into a false sense of security.
If the people who understand the business must rely on AI to understand and change the reasoning behind decisions, how independent is their judgment? We should be building AI systems in which people can work directly on that reasoning, without having to ask AI to make every change. And their contributions should last, becoming part of the reasoning used for subsequent decisions.
Perhaps there is work to do before we can relax.