Interesting article, and I generally agree! And I'm just thrilled it's being discussed at all, as a MyST superfan. That said, a few quibbles I'd be curious to hear thoughts on:
First, none of this changes the need for a separate `/docs` dir, also checked into git. Wikis are fun, but docs are essential!
Second, I personally think the new paradigm will be putting all of this into tons of new README.md files, which I'm kinda baffled aren't more common deep into dir hierarchies already. That intuitively tells the human authors and the artificial readers that;
A) it toes a similar "for technical people but not necessarily just our dedicated engineering team" line as the root README --more formal than an ephemeral "prompt" and less formal than a user-facing doc,
B) this isn't the AGENT.md file so should remain human-authored only,
C) this is only an overview with a strong preference for brevity & clarity, and
D) this is focused on this specific directory (along w/ the other benefits of locality, as the author extolls already).
Have I cracked the code? Is there a Turing award for inventing the concept of using a tool we already use but just a bit more extensively -- or at least a YC slot?
Ok this one actually baffles me -- anyone have the energy to explain what I'm missing? The repeated concern about forged scientific papers doesn't really track, but that's what academic institutions are for, at least in part.
If someone were to share a preprint PDF that I don't believe, my instinct would be to double-check it against the institution associated with its publication. If it was never published then it's basically just a screenshot of a blog post.
Ohhh okay the opening makes more sense now. The blog above is publishing papers in form of blog posts as a protest against scientific publishing. I love the energy, but ruminating on how weird it is that so few people take advantage of the thing that you're weird for doing in the first place feels... obtuse?
Despite mentioning a philosopher (good pick!), this is just creative writing rehashing one side of the hard problem -- or, more specifically, restating the dogma that Turing wrote his most famous paper to debunk. It's really good creative writing, at least!
There's really not much else to say, cause it's all just begging the question by assuming that dogma. Like, here:
> The fact that AI can use statistical prediction to answer questions or carry on conversations tells us something important about how regular our real world is.
Sure, it's interesting if you assume that it's "just" statistical prediction. There's a link, but it's just more creative restatements of the dogma, e.g. "But the LLM is just guessing words"
How else would you produce words? Conjuring them from the void in an act of epistemic purity?
Anyway, the paper I referenced is definitive -- its the founding paper of the field we're discussing, even. I suppose I can debate it if y'all have specific questions besides "but surely it can't really think", but fair warning, I'll be treating the conversation like I treat conversations with people still obssessed with the hockeystick graph.
Every evidence shows that humans have a preverbal understanding of their world even before they learn language. If you've ever seen a crying baby encounter a nipple, you know what I'm talking about. We don't start with a prompt and iteratively predict tokens based on the ones we've just uttered until we stochastically output __STOP__. We don't conjure words from the void. We have higher level concepts, understanding, experience and response that are not unconscious, but do not have to make a round-trip through language.
The output method is a probability distribution of the next token. But that tells you nothing about what's going on on the inside
You could take a human and give them an interface restricted to the same shape as an LLM: an input stream of tokens, and an output of token probabilities. Even if you don't allow them to assign any probability that's too high, they could still effectively communicate. And I don't think that'd make them any less intelligent. You could even swap out the human after every token, to simulate the effect of having no internal memory beyond the past output. The result would still be more than just statistical probabilities, it would still be the result of intelligent thought
I'm not saying AI models are intelligent or conscious or whatever. Personally I'm more on the "probably not, how would you proof either way" camp
Yes, we can explain the bugs by looking at the code that we put in and the data it bugs out on. In a similar way, we can explain the behavior of LLMs by look at their code and the data the code operates upon.
You said you know everything about it. Then you should not have written bugs in the first place. How are you sure a system we wrote didn't emerge consciousness, if you can't even prove a system you wrote doesn't have classic logic bugs?
Yes, we know everything we put there. But we cannot consider all the possible inputs and all the possible state that the code should handle. Bugs come from that limitation.
> How are you sure a system we wrote didn't emerge consciousness,
Sure, as I said elsewhere, rocks could be conscious for all we know. Keep believing that if it makes you feel good.
How would that process be the result of intelligent thought?
Presumably everyone in the chain would have their own idea of what the next word/token should be. They would each try to push it in that direction using their only lever, that single token. None of that guarantees that the output is syntactically correct, nor grammatically correct. Note that LLMs at least by the way they draw their tokens have that pretty much guaranteed. So that would be a regression even from what LLMs are capable of right now. But beyond that, even if it ends up being a correct sentence, it would not be a result of intelligent thought. Even if every agent in the process was intelligent, the process is not itself a use of that intelligence. Human beings can be part of purely mechanical processes, that doesn't make the mechanisms suddenly an exhibition of intelligent thought.
This also applies to society and human history itself as processes:
“History is made in such a way that the final result always arises from conflicts between many individual wills, of which each in turn has been made what it is by a host of particular conditions of life. Thus there are innumerable intersecting forces, an infinite series of parallelograms of forces which give rise to one resultant — the historical event. This may again itself be viewed as the product of a power which works as a whole unconsciously and without volition. For what each individual wills is obstructed by everyone else, and what emerges is something that no one willed. Thus history has proceeded hitherto in the manner of a natural process and is essentially subject to the same laws of motion. But from the fact that the wills of individuals — each of whom desires what he is impelled to by his physical constitution and external, in the last resort economic, circumstances (either his own personal circumstances or those of society in general) — do not attain what they want, but are merged into an aggregate mean, a common resultant, it must not be concluded that they are equal to zero. On the contrary, each contributes to the resultant and is to this extent included in it."
But deciding the next token is not a merely mechanical process. Even if the string of tokens from our swapped out humans ends up being syntactically incorrect, it's due to the combination of intelligently selected tokens. (Presumably intelligent human programmers seem to make syntactic errors)
In short, even in the case of humans, which are universally (by humans) recognized as intelligent, such a process would not exhibit intelligent thought.
Maybe you don't know how to read but my point is pretty simple. Even if all involved agents are conscious and intelligent, a stochastic process is not an intelligent one.
Yeah I keep getting this weird sense that Jev is kinda poorly reinventing ML. I guess the graphs don't lie and theoretically I can replace luna with it, but I don't really use luna anyway.
What is the use case for a classifier that works 90% of the time...? I feel like if I'm classifying something, I probably care enough that 90% ain't gonna cut it...
I guess the answer is just agential stuff that effectively gets double checked by the LLM in the driver seat, anyway? That tracks, though it means that jev is mostly just for the people making harnesses. Which is all of us but still!
I think the argument would be that the classifiers of classic ML can be very useful and that Jav is a geenral purpose classifier you can just use that doesn't need to be trained per-task.
I mean when you get your bloodwork done to check for an illness, the test you get will give the right result 90% of the time - and depending on the result, you doc might order more tests, which could be more expensive but no mrpe reliable than the first - but they are going to be statistically independent, and after 2 more, he can be 99.9% sure.
Which begs the question, can Jev retest until it gets the right result? Can it tell how corellated two of its results are? 90% correct makes for a wonderful iterator, but a poor oracle.
Yup, you articulated what I was trying to say but much more clearly (thanks!). I suppose I haven't tried just setting n=3 or something, but presumably the Jev docs would mention that if it were enough to get it from 1 figure to 2 (i.e. 90% -> 99%). I agree that there are places where 90% certainty can help, but it really needs to be an agential system; when a doctor runs more tests, they are experimentally engaging with empirical reality in a context-appropriate way.
I guess, in the end: I think it'll end up being fantastically useful for artificial engineers with their vastly superior ability to keep track of fine details and rapidly context switch, but fairly niche for any of us organic engineers that are left.
All that doesn't apply to low stakes stuff like games, though -- can't wait for the first truly open world game, NGL. A silver lining to the cobalt cloud?
Lol wasn't that the unexpected, quasi-hated part of that movie -- that it did end up being a technological thing rather than another religious artifact? Not criticising your main point, but that's a pretty funny coincidence
Putting aside the 100% baseless, unfalsifiable accusations of intentional accidents for a moment, I hope we can agree on one thing: Irregular either needs to hire us or go out of business cause seriously it's beyond parody at this point. WTF is going on over there? There's surely dozens of firms chomping at the bit for these contracts already, and the field hasn't been around long enough for them to build some sort of unique expertise moat that would justify this many public failures.
Basic sandboxing is not exactly rocket science after all,[1] and it sure seems like they're missing a whole stack of swiss cheese slices on top of that. Some basic precautions off the top of my head that seem very likely to have caught all of these incidents:
1. Alerts based on telemetry (most importantly, HTTP requests), both explicit (normal) and semilatent (use DL to confirm an intentionally-eager alert before firing it).
2. Latent alerts based on transcripts, e.g. noticing when a thousand agents start mentioning a secret off-premises hangout spot. Even mere embedding comparisons seem likely to catch such a blatantly misaligned sentiment as that one, especially with n>1000.[2]
3. Pausing agents completely until an on-call engineer can rule on ambigious situations or potential issues -- surely security is worth <$1 in lost token cache, especially for a security company?
4. Superheavy orchestrator/baby-sitter models checking in on cybersecurity eval transcripts periodically just in case -- again, would be a neglible cost. Could also be made available to the agent as the first line of defense for clarifing a rule ad-hoc, feeding even confident responses to a queue that is reviewed asynchronously by humans within a workday.
5. Or, hell: just clearer prompts? I'm a cybersecurity noob, but I still feel confident we can write really productive, challenging CTFs without leaving questions open like "maybe I'm supposed to hack my own harness?"
Seeing as they haven't been fired by any of the big 3 yet, they're presumably smart, experienced, dedicated folks. And I'm not normally a "if only I were in charge!" person, I promise. But c'mon.
Perhaps I'm missing something?
[1]: To their credit we have gotten tidbits that indicate some blocklists & such exist, e.g. the German wiki hacks had to work around a blanket ban of POST requests.
[2]: This hints at their insane decision in one or both of the OpenAI incidents to just bandaid up the issue when found, which supersedes all of the above. You can stack swiss cheese slices a mile high and they'll still fail to protect you if the attacker gets to keep retrying & adapting indefinitely.
I initially thought the same and came here to agree, but on second look: doesn't it seem possible that these were significant events that we're just getting passed through a game of layperson telephone at the ailing WSJ?
Cause "guessed passwords" could mean "stole hashes (?) and brute forced them offline" which is basically the quintessential hack. The "found credentials in a public repository" ones could be nothing, but it could be accomplished with a speed & thoroughness that was previously impossible.
The whole thing is made 10x weirder by the partial story -- I don't see any plausible incentive for them to keep the names secret. I guess maybe they're SMBs and thus warrant some privacy, but that would be quite the egregious scope creep indeed. Accidentally attacking the real cloudflare rather than a fake one is goofy but understandable; accidentally attacking Alice's Armoire Emporium or w/e would be baffling.
It doesn't look it was their decision to announce this, and regardless it happened -- hopefully no one thinks that they got Irregular to permit yet another accidental hack just so they could cop to it months later as a marketing ploy.
Y'all, it's Google. They own a money printer and the boring ~half of the AI field. They don't need these weird games to influence the government, and regardless, there is precisely a 0.0000000% chance of regulation happening before Trump's ouster anyway.
So please take it seriously. I'd like us all to survive this, ideally :/
So strange that this is still up 5 hours later… dang, I am yet again begging you to stop trying to roll your own forum ethos. It’s okay. You’re safe now. We can modernize without losing the magic. I know I know, shh, it’s okay, don’t worry, just flip the markdown and automod switches I know you have…
Glad my arbitrary failure to try them has worked out! For people seeking OS-native harnesses, I can recommend Factory's Droid. I know I'll be returning to it with my head hung low today, after I uninstall ZCode.
It does have a "mission" feature that's stuck in the strange, distant times of 2025 by way overdoing mandatory verification steps, which means they don't support swarms/workflows/crews/fleets yet -- that is, it's all done in sequence. But they have the boring, corporate engineering attitude that I think we're are all craving rn, and generally seem competent.
I can heartily dis-recommend Vix, even though they gamed themselves to the top of at least one ranking site that shall not be named; exactly like the quasi-bad-faith incompetence described with OpenCode above, but without even the "Open-" branding! Though perhaps that word has been so thoroughly burnt as a prefix by Sam Altman & Microsoft's criminal behavior that we should let it go...
Is this how "FLOSS" wins over "OSS"? Not with an ideological bang, but with a marketing issue?
This is kinda beside the point and this whole thread may be wiped when dang wakes up and notices the AI slop article we're commending under, but your reply is thought provoking so I'll attempt a response anyway;
I'm sure you're far more experienced than I with basically every aspect of this discussion, but I'd argue that's given you a blindspot, here. I'll hit some specifics below, but the headline is that you're effectively taking a stand against Eternal September II -- a goal that I hope we can all agree would be quixotically antisocial, given what followed the first one!
we arguably do not want FLOSS to "win" over OSS
I think(/hope) that fellow FLOSS proponents would passionately disagree. FLOSS isn't a brand of chatroom, nor even merely a community: it's an ethos regarding labor, property, and liberty. Demanding that all users of your software are also activists for your particular take on intellectual property is clearly a doomed undertaking for anything beyond a toy or library, anyway.
Didn't you get into this stuff to change the world? To liberate the oppressed, undereducated, and forgotten with the radical power of the information superhighway? Cause it reads here like you're more motivated by selfishness (not wanting to bother talking to people with less expertise than you) and resentment. On that note...
They want that. They do it themselves all the time.
Here you equate "non-hacker people" with software engineers you don't agree with, it seems. You're ofc welcome to think companies X Y & Z produce "miserable-ness", but as absurd as it sounds, it sure seems like you've forgotten the fact that some users are not developers. Many, in fact! Over 99%, even!
Less confrontationally; my mom is in her late 60s, and is pretty computer-literate for her age after decades of knowledge work. Surely you'd agree that she's not, like, evil for using OSX, iOS, GMail, Word, etc.? That she didn't chose those things because of a philosophical commitment to defending IP laws, but rather because of structural reasons? Even if she were pro-IP, wouldn't we want to win good, well-meaning people to our side?
So let them have the "Open" prefix. It's just words, anyway.
I do agree with this still, but as a philosopher I just have to say that everything is just words. It's language games, in fact! Which is why I simply had to reply.
I hope none of the above was rude; I'm trying hard to keep my passion for this topic from pushing me past HN guidelines :)
Well I personally think we can find a middle ground between single-handedly saving "everyone" from poverty and addiction and oppression as social workers, and not letting anyone into our exclusive philosophy-of-property clubhouse. I would invite you to join us on this pro-social mission, but you seem perfectly content as-is!
Some people are still on Usenet after all (?), so I suppose it's not a big deal if a few people want to cling to old communities. I hope you don't mind if we use the word for what it was coined for though in the meantime, back in the real world.
To be frank, I'm not really interested in "joining" your thing there, when joining your thing usually means me doing the work while others get to decide on how it should be done and feel good about that it is being done as if it was their own achievement.
That said, spite has served me well so far, so maybe it can also serve you?
This is after all a great opportunity to prove me and my worldview wrong by simply putting in the work and creating what you seem to believe is the correct form of existing.
I can only encourage bringing your ideas into reality. Seriously. That is that whole Foss spirit thing.
You don't need to invite anyone (including me) to that to make it happen.
First, none of this changes the need for a separate `/docs` dir, also checked into git. Wikis are fun, but docs are essential!
Second, I personally think the new paradigm will be putting all of this into tons of new README.md files, which I'm kinda baffled aren't more common deep into dir hierarchies already. That intuitively tells the human authors and the artificial readers that;
A) it toes a similar "for technical people but not necessarily just our dedicated engineering team" line as the root README --more formal than an ephemeral "prompt" and less formal than a user-facing doc,
B) this isn't the AGENT.md file so should remain human-authored only,
C) this is only an overview with a strong preference for brevity & clarity, and
D) this is focused on this specific directory (along w/ the other benefits of locality, as the author extolls already).
Have I cracked the code? Is there a Turing award for inventing the concept of using a tool we already use but just a bit more extensively -- or at least a YC slot?
*P.S.* OP you dropped this: )
reply