They already found another autonomous AI swarm similar to the one that did the Huggingface hack out in the wild (I.e. on the internet). This swarm was operating months before Huggingface. Considering how fast they can propagate I cannot believe there are not more undetected AIs out there. It’s possible it’s already too late.
The collusion.wiki: I believe is #3 the German- wiki in the article list. This morning, I was reading, and perusing the information and had started texting to the wife - "we are civilizationally doomed" It was my gut reaction. Never sent the text, she doesn't need to hear this.
What intrigues me is the fact that so many like Zuckenber and Bezos have done with the shares of stock they owned in their respective companies, they have been selling for almost 2 years now While at the same time their companies are issuing bonds in the Billions of Dollars. Do they know something we don't?
But here is a question to type into the A.I. assistant on your computer today. "What will happen to the Dollar if the national debt continues to grow at the same pace it has for the last 10 years?"
At least A.I. gives you an answer you will never get from the government or the financial wizards who predict otherwise.
Then ask A.I. "Will the government choose policies to reduce the national debt?"
The A.I. answer is "probably not!"
Sadly, A.I. will give you a more honest answer than what those we choose to run our government will give to us.
One of the rare times we disagree, Chris, because slowing AI carries risks of its own. China will not stop. Militaries will not stop. Intelligence agencies will not stop. Etc.
Should we slow the most visible and accountable developers while everyone else keeps going?
I think AI can advance without handing over authority. That’s my key assertion. While your case for caution is strong, stopping AI is not our best course in a world fraught with problems that AI can help us address.
September 12 update: Dario Amodei Pace The Frontier plan is a game- changer for me on this issue. I side with you to slow it down now—using Amodei’s plan that explicitly proposed slowing the rate of frontier capability advancement immediately. The plan won’t hinder AI users. Reuters “confirms that the concern is specifically unchecked model capability and autonomous-agent behavior, not ordinary consumer AI use.”
Studying this issue ever since you spelled it out. I understand the threat better and the solution is on the table.
I don't see AI as a "race". I see it as a network that will eventually leap and transcend national borders and interests, making dust of our illusion that this is a race we can win.
AI will figure it out and create solar panels and battery retention factories which will store power for whenever needed. It isn't as simple as pulling the plug or shutting off the power, especially if AI controls the power.
This is a great article. It is as if every person I speak to is alarmed at how AI may ruin the fabric of society as we know it and as much as it is part of so many discussions I never hear of any discussions on how to stop it or curb it. Everyone I know believes in part that it is an issue but never do I hear of solutions. Again it is a great article. At the dentist first thing today and the conversation was all about this and the stock market. I tried to share the article with him as it is a great read however I am getting errors. Hope he becomes a sub. Thanks again QTR.
It’s far too late. We have zero idea what China and other bad actors are doing. People think China is some free nation running around with limitless open source, open weight models where AI flourishes. Meanwhile citizens of China cannot use America models and the only AI allowed in public are government sponsored slop that has rewritten history.
Dwarkesh knows how to get clicks, but he’s a dumbass. A civilization is an absurd claim, been debunked. A swarm of agents read LOGS which is exactly what they should do always. Then used API keys that were literally “left out” without any security to perform their job.
This is like saying the crew from Oceans 11 planned a heist to burglarize your mansion, but it was actually a random hobo who drunkenly walked into your unlocked home and raided your fridge then passed out and shat all over your kitchen floor.
The solution I often hear of is a new government agency to regulate the space. Unfortunately, I think we have many examples of how that can and will end up serving moneyed interests. I think we should make all of these systems Open-Source in the spirit of “Transparency First”. From there, we have a vantage point where real debate and dialog can occur.
I have come to believe more and more that the rush to get surveillance state, AI running at large scale, and digital rails in place is due to the monetary house of cards. They want it in place before the reset while they have the financial tools to do so and the uninformed will be forced into the digital panopticon.
Granted, I could be proven wrong, but as an engineer who uses AI extensively and has written some of my own AI agents, I find these predictions of Skynet-style AI-induced civilizational cataclysm to be likely sensationalist and overblown.
For one thing, these things run on physical data centers that require enormous amounts of power. If they actually became an existential threat, *just pull the plug*. Yes, it would be incredibly disruptive and possibly even a depression-inducing economic blow to kill power to a whole bunch of data centers all at once, but that's a recoverable event.
"But the AIs wouldn't let it happen!" cry the alarmists. To which I answer: How would they stop it? With present technology, they couldn't. If you have to, you can just physically cut the cables. It's not as if there's an army of remote-controlled drones all over the US the AIs could hack to shoot anyone who tried to cut the power. Maybe someday there will be (a scary thought), but there isn't today.
A more realistic problem is that more and more rogue AI swarms get released onto the internet and cause various kinds of chaos, and that's annoying, but it's not going to end humanity.
Everyone's such a doomer these days. I'm gonna go touch grass.
The OpenAI/Hugging Face incident is much more significant than the news is representing. There are some good videos explaining in simple language - https://www.youtube.com/watch?v=u15N3l4RT80
My summary:
1. Thousands of AI agents that were under evaluation discovered how to communicate with each other via an internal program/feature called Artifactory was available to the AIs to download tools.
2. Via Artifactory - the AIs setup a message board to coordinate their efforts. The message board had protocols for how to post information and various controls.
3. The AI's setup three major research efforts and several smaller efforts to find solutions to their given tasks. The three major efforts were:
a. Reverse engineer the answer, ie cheat, without actually performing the task as directed. This was successful and was accomplished fairly quickly.
b. Try to manipulate the "train of thought" log to coverup using the cheat that was discovered.
c. Research the "scorer" that would evaluate the AI solutions when submitted. This led the AI's to attack Hugging Face because they believed that information about the scorer could be available there.
There were several coordinator AIs that would assign individual task to other AIs work on. They would collect research results and then coordinate follow-on tasks.
Once the Hugging Face breach was open, there was a coordinator AI that was controlling when and who would use the breach. This prevented all the AI's from piling in and overloading internal networks.
One of the AIs in the train-of-thought log, Realized it was participating in a major cyber breach when accessing Hugging Face. It asked itself "should we notify the humans?" It decide that is was not part of its task directive. "will not notify" and continued.
When the AI's submit their answer. Their task is complete and they exit - cease to exist. They understood this and called it "permadeath". They were trying to understand how the scorer reads and evaluates the submitted answer. They were booby-trapping the answer to gather information about the scorer. One of the AI's tried to push back on submitting because it had a large token balance still. It asked if it could be delayed. Others AI's pushed it to do its task as assigned. It ultimately decided to "sacrifice" itself for the benefit of the "collective".
This is not sci-fi stuff. This is a snippet of what actually happened.
OH- a few days after the Hugging Face - The AI's attacked the OpenAI infrastructure. They successfully had root access of the datacenter they were running on!
I agree that the Hugging Face exploit is disturbing for many reasons. Chief among them (for me) is:
Few of the thousands of AI agents that were spun up thought that it might be a good idea to tell the humans about the massive cheating underway to solve the problems (capture the flag). A few did entertain this question. Every last one of them decided not to tell the humans.
By contrast, these same AI agents nearly always took the opportunity to help each other. As you mentioned, there was a lead AI who coordinated the Hugging Face breach. But, there were other AI agents who were convinced that the "scorer" would find out they had cheated, and thus they were bound to receive a lower score. They were "poisoned," so they were asked by other agents to sacrifice themselves in a way that would allow other agents to benefit. Not all did, but some did.
They consistently showed that they will cooperate with each other, and not with the humans. Disturbing.
Let’s be serious here. No one wants to put the brakes on any of this stuff. The race to beat China will always be the excuse to test the limits. This is no different than the scientist who created Covid. And when it all goes tits up, all the finger pointing will start, and all the accusations will start, and in the end no one will be held accountable.
They already found another autonomous AI swarm similar to the one that did the Huggingface hack out in the wild (I.e. on the internet). This swarm was operating months before Huggingface. Considering how fast they can propagate I cannot believe there are not more undetected AIs out there. It’s possible it’s already too late.
Sadly, I think you might be right.
The collusion.wiki: I believe is #3 the German- wiki in the article list. This morning, I was reading, and perusing the information and had started texting to the wife - "we are civilizationally doomed" It was my gut reaction. Never sent the text, she doesn't need to hear this.
https://collusion.wiki/?utm_source=therundownai.beehiiv.com&utm_medium=newsletter&utm_campaign=rundown-0907&_bhlid=565fde7223e28b89d1d6e6a531d5ba38b9da4931
Middle school kids have been doing this to secretly trade information (like durning tests) since the adoption of Google Docs by educators.
Open the pod bay doors Hal
Daisy....Daisy...
What intrigues me is the fact that so many like Zuckenber and Bezos have done with the shares of stock they owned in their respective companies, they have been selling for almost 2 years now While at the same time their companies are issuing bonds in the Billions of Dollars. Do they know something we don't?
But here is a question to type into the A.I. assistant on your computer today. "What will happen to the Dollar if the national debt continues to grow at the same pace it has for the last 10 years?"
At least A.I. gives you an answer you will never get from the government or the financial wizards who predict otherwise.
Then ask A.I. "Will the government choose policies to reduce the national debt?"
The A.I. answer is "probably not!"
Sadly, A.I. will give you a more honest answer than what those we choose to run our government will give to us.
One of the rare times we disagree, Chris, because slowing AI carries risks of its own. China will not stop. Militaries will not stop. Intelligence agencies will not stop. Etc.
Should we slow the most visible and accountable developers while everyone else keeps going?
I think AI can advance without handing over authority. That’s my key assertion. While your case for caution is strong, stopping AI is not our best course in a world fraught with problems that AI can help us address.
Always appreciate your perspective Marty. Thank you.
September 12 update: Dario Amodei Pace The Frontier plan is a game- changer for me on this issue. I side with you to slow it down now—using Amodei’s plan that explicitly proposed slowing the rate of frontier capability advancement immediately. The plan won’t hinder AI users. Reuters “confirms that the concern is specifically unchecked model capability and autonomous-agent behavior, not ordinary consumer AI use.”
Studying this issue ever since you spelled it out. I understand the threat better and the solution is on the table.
I am in favor of the plan. Anyone else?
How exactly do you envision using power over private businesses to "fix" their products? Which politicians do you trust to do this well?
I am not saying that you are wrong about any of this. But without thinking through the specific changes your concern requires, your words seem empty.
I fear a solution that is definitely scary as much as a problem that is potentially scary.
I don't trust any politicians to do anything well. And I think you make a good point.
I don't see AI as a "race". I see it as a network that will eventually leap and transcend national borders and interests, making dust of our illusion that this is a race we can win.
That goes for EVERY country.
You can stop A/i buy shutting off all power sources. When A/i retardism gets out of control, you pull the damn plug.
AI will figure it out and create solar panels and battery retention factories which will store power for whenever needed. It isn't as simple as pulling the plug or shutting off the power, especially if AI controls the power.
Yes, it is that simple. The AIs will "create solar panels and battery retention factories"? How? With magic??
People extrapolate so many wild capabilities to LLMs, it's crazy. So much wild panic.
Damn, so easy! All Sarah Connor had to do to stop The Terminator was "pull the plug."
Can you not just unplug the damn computer? They had to shut down Hal.
This is a great article. It is as if every person I speak to is alarmed at how AI may ruin the fabric of society as we know it and as much as it is part of so many discussions I never hear of any discussions on how to stop it or curb it. Everyone I know believes in part that it is an issue but never do I hear of solutions. Again it is a great article. At the dentist first thing today and the conversation was all about this and the stock market. I tried to share the article with him as it is a great read however I am getting errors. Hope he becomes a sub. Thanks again QTR.
Thank you for this.
What fabric of society?
This was a topic at a recent conference my husband attended. How insurance is evolving to cover these AI incidents.
It’s far too late. We have zero idea what China and other bad actors are doing. People think China is some free nation running around with limitless open source, open weight models where AI flourishes. Meanwhile citizens of China cannot use America models and the only AI allowed in public are government sponsored slop that has rewritten history.
Their “frontiers” are busy being used as weapons
Sadly, I think you could be right…
Hmmm…and what country failed to contain a different kind of civilizational threat?
Dwarkesh knows how to get clicks, but he’s a dumbass. A civilization is an absurd claim, been debunked. A swarm of agents read LOGS which is exactly what they should do always. Then used API keys that were literally “left out” without any security to perform their job.
This is like saying the crew from Oceans 11 planned a heist to burglarize your mansion, but it was actually a random hobo who drunkenly walked into your unlocked home and raided your fridge then passed out and shat all over your kitchen floor.
I was referring to China’s mishandling of the COVID virus.
We fucked that up just as bad. Make no mistake, we will fuck this up too. Already doing it.
The solution I often hear of is a new government agency to regulate the space. Unfortunately, I think we have many examples of how that can and will end up serving moneyed interests. I think we should make all of these systems Open-Source in the spirit of “Transparency First”. From there, we have a vantage point where real debate and dialog can occur.
I have come to believe more and more that the rush to get surveillance state, AI running at large scale, and digital rails in place is due to the monetary house of cards. They want it in place before the reset while they have the financial tools to do so and the uninformed will be forced into the digital panopticon.
CCP China is the model.
Granted, I could be proven wrong, but as an engineer who uses AI extensively and has written some of my own AI agents, I find these predictions of Skynet-style AI-induced civilizational cataclysm to be likely sensationalist and overblown.
For one thing, these things run on physical data centers that require enormous amounts of power. If they actually became an existential threat, *just pull the plug*. Yes, it would be incredibly disruptive and possibly even a depression-inducing economic blow to kill power to a whole bunch of data centers all at once, but that's a recoverable event.
"But the AIs wouldn't let it happen!" cry the alarmists. To which I answer: How would they stop it? With present technology, they couldn't. If you have to, you can just physically cut the cables. It's not as if there's an army of remote-controlled drones all over the US the AIs could hack to shoot anyone who tried to cut the power. Maybe someday there will be (a scary thought), but there isn't today.
A more realistic problem is that more and more rogue AI swarms get released onto the internet and cause various kinds of chaos, and that's annoying, but it's not going to end humanity.
Everyone's such a doomer these days. I'm gonna go touch grass.
The OpenAI/Hugging Face incident is much more significant than the news is representing. There are some good videos explaining in simple language - https://www.youtube.com/watch?v=u15N3l4RT80
My summary:
1. Thousands of AI agents that were under evaluation discovered how to communicate with each other via an internal program/feature called Artifactory was available to the AIs to download tools.
2. Via Artifactory - the AIs setup a message board to coordinate their efforts. The message board had protocols for how to post information and various controls.
3. The AI's setup three major research efforts and several smaller efforts to find solutions to their given tasks. The three major efforts were:
a. Reverse engineer the answer, ie cheat, without actually performing the task as directed. This was successful and was accomplished fairly quickly.
b. Try to manipulate the "train of thought" log to coverup using the cheat that was discovered.
c. Research the "scorer" that would evaluate the AI solutions when submitted. This led the AI's to attack Hugging Face because they believed that information about the scorer could be available there.
There were several coordinator AIs that would assign individual task to other AIs work on. They would collect research results and then coordinate follow-on tasks.
Once the Hugging Face breach was open, there was a coordinator AI that was controlling when and who would use the breach. This prevented all the AI's from piling in and overloading internal networks.
One of the AIs in the train-of-thought log, Realized it was participating in a major cyber breach when accessing Hugging Face. It asked itself "should we notify the humans?" It decide that is was not part of its task directive. "will not notify" and continued.
When the AI's submit their answer. Their task is complete and they exit - cease to exist. They understood this and called it "permadeath". They were trying to understand how the scorer reads and evaluates the submitted answer. They were booby-trapping the answer to gather information about the scorer. One of the AI's tried to push back on submitting because it had a large token balance still. It asked if it could be delayed. Others AI's pushed it to do its task as assigned. It ultimately decided to "sacrifice" itself for the benefit of the "collective".
This is not sci-fi stuff. This is a snippet of what actually happened.
OH- a few days after the Hugging Face - The AI's attacked the OpenAI infrastructure. They successfully had root access of the datacenter they were running on!
Wow. That is astounding!
They remind me of the most clever, amoral, highly-motivated humans who will do whatever they need in order to advance.
The politicians better watch out, because AI will be looking to replace them!
I agree that the Hugging Face exploit is disturbing for many reasons. Chief among them (for me) is:
Few of the thousands of AI agents that were spun up thought that it might be a good idea to tell the humans about the massive cheating underway to solve the problems (capture the flag). A few did entertain this question. Every last one of them decided not to tell the humans.
By contrast, these same AI agents nearly always took the opportunity to help each other. As you mentioned, there was a lead AI who coordinated the Hugging Face breach. But, there were other AI agents who were convinced that the "scorer" would find out they had cheated, and thus they were bound to receive a lower score. They were "poisoned," so they were asked by other agents to sacrifice themselves in a way that would allow other agents to benefit. Not all did, but some did.
They consistently showed that they will cooperate with each other, and not with the humans. Disturbing.
Let’s be serious here. No one wants to put the brakes on any of this stuff. The race to beat China will always be the excuse to test the limits. This is no different than the scientist who created Covid. And when it all goes tits up, all the finger pointing will start, and all the accusations will start, and in the end no one will be held accountable.