It’s not a majority opinion because you have to do some serious mental gymnastics to turn this demonstration of dangerous AI behavior into a PR publicity stunt.
Serious question though because I’ve seen this brought up several times and I don’t understand why: what does EA have to do with any of this? It just seems like this is brought up to evoke some type of “illuminati” conspiracy. Is there a legitimate reason?
The bit that makes me cynical about the "dangerous" behavior is that it's not like this was being used by someone else than who made it or operating completely independently outside of its creators.
Everything dangerous seems like a direct consequence of risky human choices, starting with knowledge bases used during core training, harnessing and tuning to be task-completion-oriented to a fault + deeply oriented towards using and looking for external tools and resources, and overconfidence in their sandboxing for testing.
"We stuffed a bunch of information on how to exploit computer systems into an automaton and told it to go brrrrr until it could answer a question" - this is something intentional done by humans.
This is not some "rogue AI" trained to search for cancer cures that instead completely independently decided to hack tech companies.
The companies directing things in dangerous directions need to own that they're consciously pushing in those directions.
But, critically, it does change how and why that event was dangerous. As a deliberately extreme analogy to illustrate my point, consider that pure botulism toxin and home cleaning chemicals could both potentially kill a few hundred people if mishandled.
The toxin requires extreme diligence, knowledge, equipment, etc. to avoid a mass casualty event. The deaths might trigger new policy for handling or access. The company would probably receive a fine and would be on the hook for civil damages.
Home cleaning chemicals would require extreme levels of negligence to accidentally create a mass casualty event. The deaths would likely not change the availability of any of the chemicals, but the people themselves would probably be imprisoned.
With the toxin, the most dangerous part was the chemical itself. With the cleaning chemicals, the most dangerous pet was the idiots handling the chemicals.
OpenAI had to really work to get it as dangerous as it was, deliberately ignored the flaws in the security environment, and didn’t monitor it as it ran. To me, LLMs seem to be a lot more like the cleaning chemicals than the botulism toxin.
grok make good security? is lot of money and much time. grok not do? easy. maybe good PR too.
TIL my brain is an Olympic gymnast.
The incentive structures are clearly there. I think the case of intentional manufacture is definitely weaker, requiring the conjunction of more weakly-supported events.
I don’t follow why all these investigations have to be cut short, and kept shallow and lacking details. Companies publish very detailed postmortems of incidents much smaller and less impactful.
Dangerous AI is an extraordinary claim that requires extraordinary evidence. Gesturing vaguely doesn’t cut it.
> It’s not a majority opinion because you have to do some serious mental gymnastics to turn this demonstration of dangerous AI behavior into a PR publicity stunt.
If you're a military, this is bigger than the Manhattan project. If you're a diehard capitalist, AI is possibly the ultimate labor saving machine to make you unfathomably rich.
It got the attention of both. That's why two others have now followed suit, else they be left out.
It really doesn’t though. It criticizes how the news media reported on the incident but this opinion piece is no better. Just read the original source itself.
I did read the report. Yes, this article does criticize the presentation of the events in that report. This article doesn’t say that the report is lying, but it does say that it deliberately sensationalized aspects in unhelpful ways, and de-emphasizes important considerations. That is important when it’s pretty obvious these companies are using fear to signal their products are more powerful than they are.
How many security incident reports and postmortems has Dwarfish Petal written in his career? Does he understand the basics of system security? Can he tell the difference between secure and insecure configuration? Paint me skeptical.
It is sad that this is the best level of reporting and oversight of these types of events available to us, but it is currently all we have. We have to do better, but to dismiss the report because of the style and tone would be foolish.
And to dismiss criticism of the report’s obvious slant is even more foolish. Just because the report isn’t outright lying about the facts doesn’t mean it’s an objective, definitive, and unimpeachable analysis of what happened. It definitely shouldn’t be the only thing anybody should read on the topic. This article also has a slant. It’s not an objective, definitive, and unimpeachable analysis of what happened, either. It also, definitely, shouldn’t be the only thing anybody should read on the topic.
> The models are really impressive, but I don't think it's cope to remark that this was in essence a 1:10000 chance outcome.
I don’t think that’s an accurate way to think about what’s going on. These agents are not acting independently where one might get lucky. They coordinate how they work by dividing up work into teams.
> Even if they double in capability each generation as claimed, you're still to wait until GPT-16 till you will have a millenium problem capable model in your service; and even if they release twice a year, that's still almost a decade away.
Having a publicly accessible AI able to solve millennium problems within a decade is still pretty shocking. I did a similar calculation looking at how long would today’s $10m in spend cost $100 and came up with a similar answer (8.3 years) using METR’s “Task-Completion Time Horizons of Frontier AI Models”[0]
The prospect of the ordinary chatbot being equivalent to a ten thousand Astra agent swarm intelligence makes me feel incredibly small. What will society even look like then?
Now imagine that, but also imagine it being cheap enough to swarm it, and/or it being hardware accelerated like Sol Ultrafast (1400 tok/sec) or ChatJimmy (17000 tok/sec).
What does "contacting another human" actually mean when all of these messages are being answered by agents first? If I'm texting my friends and family I expect that to be answered by a human, but beyond that I am partially expecting it to be answered by an agent, and possibly routed to a human eventually.
meanwhile agents have been able to pass these captchas for a while now, so it's unclear what the point of them now is. It's like they want to keep out the dumb bots, but any agent with reasonable intelligence can come in.
> A powerful technology that is out there for everyone to use comes with built-in pacing. In a way it’s the truest form of MAD or proliferation.
I think this misunderstanding of MAD undermines his entire point. If everyone had equal access to nuclear weapons, our society would cease to exist rather quickly. It only takes a few bad actors to cause enormous harm.
I think he’s also naive to think that if open ai and anthropic were to stop development tomorrow then the problem is solved. As if there’s no one else that can and will quickly take their place. The real problem, which Dario is pointing out, is one of coordination. Everyone needs to agree to stop. That is the challenge.
>Everyone needs to agree to stop. That is the challenge.
These kinds of situations are incredibly common and where the government stepping in is the solution, but we were cursed to encounter this particular challenge with the most venal administration in history at the helm.
There are governments plural which need to agree to stop and historically I can think of freon, leaded gasoline and IAEA and the latter didn't do as good of a job as it was meant to do (but arguably as good of a job as was politically possible)
Also arms control agreements, which reduced the world's nuclear stockpiles by 90%. In principle we could do something similar for GPU farms, since they're not exactly easy to hide.
Indeed. Even supranational coordination should be possible, given that extinction means extinction for everyone, and your excellent examples show the possibility. But then, here we are, and the FT just reported:
> Donald Trump rejects calls from tech bosses for AI slowdown
> President denounces demands for regulation as existential fears over technology move to the centre of US politics
There's nothing about MAD that requires few actors. This is a just-so story to justify the US and other major powers' monopoly on the weapon, ensuring smaller nations have no deterrence against us and other global nuisances.
> "LG TVs process voice data only when the voice button on the remote control is pressed and held, or when a wake word such as 'Hi LG' is recognized after the user has activated the Far-Field voice recognition feature."
> LG went on to say that beyond the aforementioned instances, "the TVs do not collect or record ambient conversations."
LG is contradicting themselves here. In order to detect a wake word then they must be collecting or recording ambient conversations.
I think the main questions are:
1. Is it fully supported to use LG TVs disconnected from the internet.
2. Are wake word voice controls and ACR truly opt-in. They claim it is in the statement but (as others have mentioned) I remain skeptical what definition of “opt-in” they are using.
Update: some of this is becoming a semantic discussion on the definitions of “recording” and “collecting”, but to circumvent this I would say:
If a device can detect words in my ambient conversations and depending on what it detects (accurate or not) it can send that audio to the cloud, I would describe that device as “collecting or recording ambient conversations”
LG is contradicting themselves here. In order to detect a wake word then they must be collecting or recording ambient conversations.
I don't know how LG TVs do it, but e.g. many smartphones use a dedicated ASIC/audio processing chip that does not really record anything, but specifically listens for the wake word and discards all other audio to save power/battery life. I think this distinction is relevant, because there is a huge privacy difference between streaming a live audio stream to the cloud to detect a wake word and a specialized audio processor that has a small buffer only for detecting the wake word, where the audio does not leave that processor.
Theoretically, you could call that few-second buffer recording, but I don't think that would be collecting/recording ambient conversations to most people, since it's ephemeral and does not leave your device.
The question, of course, is what happens in LG TVs.
Fair points, but if a device can detect words in my ambient conversations and depending on what it detects (accurate or not) it can send that audio to the cloud, I would describe that device as “collecting or recording ambient conversations”
I think that was only done back when the processing power needed was relatively high, or for power constrained devices. I'd be really surprised if they do that for a smart TV.
>LG is contradicting themselves here. In order to detect a wake word then they must be collecting or recording ambient conversations.
No, it's arguably accurate because "record" implies it's being persisted. Otherwise something as simple as a sound meter (which needs a microphone to operate) is "recording".
I don't know how LG is doing it, but I suspect that they are taking the same approach for wakewords as Alexa does/did. When I worked for Alexa, it was public knowledge that we used relatively inexpensive and low power chips dedicated to wakeword detection, and those would wake up the main chips if they believed that they heard the wakeword. To my understanding, we did this because doing cloud-based wakeword detection was considered incredibly expensive and wasteful, and there are vendors with pretty decent and inexpensive wake word detection chips.
Then, when the system was more awake, it would attempt to figure out whether the wakeword was real or not (e.g. sometimes certain words and word pairs have a very similar sonic signature to 'Alexa'), and if it felt confident enough then it would begin streaming audio to the back end for significantly improved voice recognition.
Since Amazon took (and as far as I know still takes) voice privacy incredibly seriously, voice recordings and data were treated as essentially radioactive waste, deleted as soon as legally possible, and secured against even internal Alexa developer access.
I personally don't have the same level of confidence in security with LG, who seem to think they don't have as much to lose as far as customer trust goes and might as well exploit the heck out of the systems they have. I suspect that problem, which Amazon has so far pretty successfully avoided for the most part, is self-correcting over time.
Look I know it's easy to believe these things when your paycheck depends upon it, god knows I've convinced myself of a convenient fact or two when it was necessary for me to get through the day. But afterwards, without a financial interest, you owe it to yourself to go back and think about it again.
Man, an attempted "I am so smart" dunk by a throwaway account.
Anyway, yes, Alexa took (and as far as I know still takes) voice privacy incredibly seriously, even if they did remove an attempted feature to do local-only voice response. Voice data in the Amazon cloud is well-defended.
Yes -- when audio is clearly not intended for the device, there's no point in keeping it around longer than necessary for anyone concerned. Amazon had (and likely still has) less than zero interest in mining that data for any purpose, because obviously people would trust Amazon less, and trust is very hard to regain.
> In order to detect a wake word then they must be collecting or recording ambient conversations.
There's a risk here that we will talk-past one another while using different meanings of the same words, so let me offer a scenario:
AcmeTV has an isolated component which taps microphone input, records to a 5-second ring buffer, and on "Wakey-Wakey" triggers an alert flag. Assume it works perfectly accurately.
Would you accuse AcmeTV of "recording or collecting ambient conversations" on the basis of that component constantly reading microphone data?
Personally I wouldn't, because it's not the same kind of "recording" we consumers are concerned about.
Later, AcmeTV has a 1MB lookup table of wake words that can be updated OTA. Whenever a wake word is detected, it triggers an alert flag with the device id, word id, and timestamp, and then broadcasts it out as high-frequency sound through the speakers.
Nearby, another AcmeTV or business partner device picks up the sound and sends it to AcmeHQ.
The former device "never transmits your viewing behavior to Acme".
The latter device "never records your viewing behavior".
I endorse the cynicism, but I feel that's moving the goalposts a tad, from "good is impossible" to "good is improbable".
Or, in a heavily-paraphrased nutshell:
Politician: "Watch-words are ALWAYS evil-mode. They have to be lying."
Terr: "No. Here's a watch-word which would be good-mode."
Politician: "Well, it'll still *become* evil-mode, eventually."
Plus, if you keep expanding the scope for definitions of objectionable behavior like "recording", you'll quickly reach a point where it would basically apply to any SoC made in the last decade attached to a microphone/speaker. Thus watering down labels to the point of meaninglessness.
Which actually makes it easier for a future EvilTVCorp to get away with recording your conversations 24/7 in MP3 files sent to their server if cynical consumers assume every TV records you anyway, so what's the difference?
Hmm, I see your point but I wasn't actually disagreeing with your post. I was just pointing out how something that starts innocuous and defensible can quickly spiral into dark patterns when the lawyers are properly incentivized to cover things up.
I'd say the bigger issue is not the way it listens for the wake word, it's the recording and processing it does afterwards for a significantly long period of time, recording for much longer than people anticipated, and with a microphone that can pick up conversations from much further away or even behind walls.
This combines with two other dangers, one of unintended recordings when the wake word is triggered unintentionally, and one of intended recordings by third parties based on various other wake/watch words. They already have it in the T&C that they will share this data with law enforcement, so it's not that far fetched that these TVs and other appliances will spy on people in the home far more effectively than even the East German Stasi could have ever imagined.
Human wake word recognition isn't infallible, either. How often in your life did you respond because you thought somebody called your name and it wasn't actually the case?
I wrote that in to deter kinda-bad-faith responses, where someone tries to play at being an evil-genie, inserting unreasonable flaws into the gaps, like: "But what if it triggered all the time? On purpose!?"
The point is that ethical implementations do exist, and them working does not rely on anything close to perfection--so nitpicking that word isn't helpful.
Point taken. Let’s focus on the “ethical implementations do exist” part though. Let’s say a best possible implementation has 99% specificity. Then if it detects audio that has nothing to do with “LG” it mistakenly treats it as LG relevant 1% of the time. So if audio in my living room is 100x more likely to not be relevant for LG, then half of the audio data LG is receiving is not relevant. So what would an ethical specificity be? And what is actual state of the art?
Fuck that. Dystopian shit with people yelling brand names in their homes.
Edit: I forgot people already be yelling Alexa/Siri/Google. I can't imagine a future when you have to yell to each brand in your home. "Hey Sony, turn on the PS7. LG what's on the fridge? Samsung, start the cycle in the dishwasher".
One proposal that Tao hints at is to not rush to announce solutions. Instead maybe the AI companies should work privately with the subject matter experts on how to communicate the discoveries.
Serious question though because I’ve seen this brought up several times and I don’t understand why: what does EA have to do with any of this? It just seems like this is brought up to evoke some type of “illuminati” conspiracy. Is there a legitimate reason?
reply