10 obvious reasons that the danger from AGI is way more serious than nuclear weapons:
-
Nuclear weapons are not smarter than humanity.
-
Nuclear weapons are not self-replicating.
-
Nuclear weapons are not self-improving.
-
Scientists understand how nuclear weapons work.
-
You can calculate how powerful a nuclear weapon will be before setting it off.
-
A realistic full exchange between two nuclear powers wouldn’t extinguish literally all of humanity.
-
It would be hard to do a full nuclear exchange by accident and without any human being having decided to do that.
-
The materials and factories for building nuclear weapons are relatively easy to spot.
-
The process for making one nuclear weapon doesn’t let you deploy 100,000 of them immediately after.
-
Humanity understands that nuclear weapons are dangerous, politicians treat them seriously, and leading scientists can have actual conversations about the dangers.
-
There are not dozens of venture-backed companies trying to scale privately owned nuclear weapons further.
-
Countries have plans for dealing with the danger posed by strategic nuclear armaments, and the plans may not be perfect but they make sense and are not made completely out of deranged hopium like “oh we’ll be safe so long as everyone has open-source nuclear weapons”.
-
Most people are not tempted to anthropomorphize nuclear weapons, nor to vastly overestimate their own predictive abilities based on anthropomorphic (or mechanomorphic) models.
-
People think about nuclear weapons as if they are ultimately ordinary causal stuff, and not as if they go into a weird separate psychological magisterium which would produce responses like “Isn’t the danger of strategic nuclear weapons just a distraction from the use of radioisotopes in medicine?”
-
Nuclear weapons are in fact pretty easy to understand. They make enormous poisonous explosions and that’s it. They have some internally complicated machinery, but the details don’t affect the outer impact and meaning of nuclear weapons.
-
Eminent physicists don’t publicly mock the idea that constructing a strategic nuclear arsenal could possibly in some way be dangerous or go less than completely well for humanity.
-
When somebody raised the concern that maybe the first nuclear explosion would ignite the atmosphere and kill everyone, it was promptly taken seriously by the physicists on the Manhattan Project, they did a physical calculation that they understood how to perform, and correctly concluded that this could not possibly happen for several different independent reasons with lots of safety margin.