While sub-agents could theoretically stack turtles indefinitely, there’s limited room when going up the stack because someone will have to take responsibility beyond “git blame”, and that lands squarely with a person or legal entity.
A big part of that responsibility can be put in code tests.
Ensuring good test coverage and quality is how you purchase trust in the work agents do. This also reduces the context problem - a test collection has no recall issues, it just runs every time you call it, the whole battery, checks all the things we could check by code in one fast tool call. For the rest, the things we can't test by code, I use manual testing.
A large class of problems are intent divergence, when the model passes tests but it didn't do what I asked. For that I keep a log of all user messages in the project history and review it with agents. This intent alignment is repeated from time to time to catch drift.
So I see the "why should I trust the work agent did?" problem as a combination of 1. ensure good testing 2. review intent alignment.
Having a coding harness is critical but the differences between them are overstated.
Personally, I’ve replaced OpenCode with a thin wrapper around Pydantic-AI as the pythonic analogue to Pi-Agent for headless use via Hermes
They’d all do the job - I just prefer to compartmentalize for access control.
Keeping the harness’ surface area tiny had the added benefit of preserving my understanding and being able to adapt it to my preferred workflow effortlessly
One thing I am missing to be able to move out of Claude Code, is the auto mode (and the soft_deny and hard_deny settings that can be tuned), with it's classifier checking the output.
It's the killer feature from me personally, often when wanting to troubleshoot for example things like Kubernetes workloads. LLMs are now really good at it, but we doesn't want them to like delete a pod.
Other harnesses like Codex have often on static rules, like the allow/deny of claude code, that can filter out based on regex. It's quite good already, but sometimes the model can find a way to write something that wasn't anticipated, or in a convoluted way.
After, I guess it's something that can be added in an open-source harness like Pi, and add like this new Jev model or something else equivalent
> Personally, I’ve replaced OpenCode with a thin wrapper around Pydantic-AI as the pythonic analogue to Pi-Agent for headless use via Hermes
That's really interesting. I like Pydantic AI a lot and wondered why all of the harnesses seem to be written in Javascript instead of it. What do you use it for headless, though? I haven't tried Hermes or similar yet, so don't have a handle on what you do with them.
I like your constructive outlook and I want to believe that we can now focus on the bigger picture, but I'm not really seeing it around me. Most everyone I know that's "AI-pilled" has spent the newfound capacity on throughput instead of altitude.
Is anyone actually seeing a shift towards improved structure rather than more code, faster? We seem to be living in the Gatling-gun version of the picture John Carmack drew when modernity also ushered into precision marksmanship...
The people who are using AI to improve what they were already doing anyway, aren't going around expounding "I made this with AI!"
Like one of my personal projects is a sort of "middleware" for Godot, and it needs to be lean, precise and match certain rules, because it's meant to be used by many games, so I can't blindly trust AI to generate code for it, it'd be too sloppy,
BUT Codex reviews have been a great help in finding bugs that would have taken me ages to even notice on my own. Some of those AI findings helped me overhaul entire subsystems, still manually coded.
I feel like using LLMs as a sort of deep search to find bugs and security issues is going to end up relatively uncontroversial all around, if the externalities of LLM use are ever taken care of.
I use vibecoding to capture UX. It's the best way of capturing requirements I've seen so far. Then once it's all captured and dies down towards requirements that are all in the "could have" category of MoSCoW, then I use AI-assisted coding (and the fun task of reading tens of thousands of source code lines) to refactor or create an entirely new application that is more technically sound.
Are those people working on products or platforms? Platforms still have to be rock solid and deliver performance guarantees to the products built on top of them.
>> Is anyone actually seeing a shift towards improved structure rather than more code, faster?
Yes. At work we recently finished a complete rewrite of the platform. The old codebase got abandoned and two new codebases got stood up. Previous stack was Phoenix LiveView and the new one is Phoenix API + Vue /w TypeScript. Zero code shared between the two. We took the opportunity to re-architect a lot of the core functionality and get rid of the tech debt that had been hounding us and killing our velocity for the past six years.
We finished the rewrite in two months. A little over 700k LoC total. 95% of it was planned, orchestrated and written by AI. We could have gotten it done faster, but we were well aware of the potential for slop and thus paid an absurd amount of attention to (and tokens for) code quality. For example, there is a 100% test coverage requirement in both server and client, combined with AI-driven review rules that say all tests must be non-vacuous, plus a ton of invariants enforced via deterministic checks. Everything from standard linting and formatting rules to AI-necessitated stuff like "comments cannot be more than five lines each" is in there. At the end of each week, we get together for ~30 minutes to review each bug that was fixed that week (there haven't been that many) and try to figure out if more deterministic checks can be added (or new ones extended) to prevent that class of bug from re-appearing. We actually have an Architecture epic with a dedicated Fable agent orchestrating all the initiatives and tickets in it and keeping track of improvements (and the occasional regression). So things like accessibility, usage of shared helpers, enforcement of common patterns, etc. are all kept track of.
The difference between the old codebase and the two new ones has been night and day. The new codebases, combined, are ~35% larger than what we had before, and yet there are way fewer bugs (we actually got suspicious at first and had to check to make sure our logging instrumentation was configured correctly). Performance is also stellar, features have been a lot easier to add and extend, and everything is a lot easier to find. When someone asks a question or has an issue we have a pretty good idea of what part of the codebase to look in and we're usually right.
It wasn't all unicorns and rainbows. Not everyone on the team was proficient with AI-driven development, so they had to be brought up to speed quickly and taught the tools and also what works and doesn't. Code reviews were also difficult and time-consuming. We had to deal with our anxieties about working in two new codebases none of us had authored much code in ourselves. What made all of it worse was that we were racing towards a deadline imposed by external factors so we all ended up working a lot of hours. It was essentially between getting it done this summer or waiting until next summer and we bit the bullet and did it. It absolutely would not have been possible without AI.
That is interesting, but... is it actually improving structure?
We're mid-way through a similar process at work. Rewriting a legacy app in a new language, with new architecture and new features.
And it's a mess.
We're at 10x loc (admittedly, the new programming language is more verbose than the old one), comments make no sense. Yes, we have ~100% coverage, but most of the tests are meaningless. The agent keeps removing our tests to replace them with tests that are easier to pass, breaking code invariants, removing all the engineered data structures and replacing them with stringly-typed code, etc.
And of course, given the number of LoC (and the fact that the agent rewrites so much code all the time), it's physically impossible that all of them were reviewed by a human being.
AI made it possible, insofar as upper management would never have greenlit the project without AI, but I can't escape the feeling that we're building on quicksands.
In my experience, when you attempt something like this it grows like a crystal. If you start badly, with poor tests and poor architecture, that is the code that has the attention of the model going forward. More than any rules you give it, what matters to the model is all the code that’s already there. I’ve found myself in such a situation from going to fast and expecting new hyped models to “just take care of it”. The solution was to stop and take stock. I ended up deleting most of the dead-end and using the models to refactor the rest. From that point on, new code from the models was at the standard I needed, following rules already expressed in the code base.
If your code base is script-kiddy garbage, your models are only going to keep writing more garbage. You should throw it all away. I expect that isn’t an option. Try instead instructing the model that the code you have is legacy garbage written in a hurry that needs siginificant refactoring, but that right now I need one new feature written to these new standards instead. Take it one small piece at a time. Don’t let context go beyond 100k. Good luck!
10x loc? wow. That makes me wonder how it will perform. And how you will review that, since if it is physically impossible for a human then who will take responsibility for what you release?
>> That is interesting, but... is it actually improving structure?
Can you be more specific? "Improve structure" can mean different things to different people.
We've ensured that agents strictly adhere to code architecture rules, using both agentic review rules and deterministic CI gates. Everything from file naming, location and namespacing to the "shape" of each file that shares the same role are consistent. We can trace functions of the same role across the same set of modules, with abstractions and code reuse where they make sense. Once someone learns the code in one part of the codebase, their understanding of that "spine" translates to other areas - the only differences relate to the specific business rules governing those areas.
>> And of course, given the number of LoC (and the fact that the agent rewrites so much code all the time), it's physically impossible that all of them were reviewed by a human being.
Code review was difficult at the start due to the volume of code. One insight we gained midway through though is that the value humans bring to code review is judgment and business context. So we created a code-atlas skill that creates an artifact for PR reviewers. That artifact highlights the most important parts of the code. For example, if a PR author has made the choice to use soft-deletes when the product uses hard-deletes for everything else, that is flagged by the code atlas, and a human reviewer can use that to ask questions about it. Meanwhile, things like boilerplate are de-emphasized; any problems associated with them are reliably found and flagged by AI reviewers anyway. The refreshing part about this is that this has almost completely eliminated bike-shedding: people no longer argue about module naming or whether a comment is worded properly.
> Code review was difficult at the start due to the volume of code. One insight we gained midway through though is that the value humans bring to code review is judgment and business context. So we created a code-atlas skill that creates an artifact for PR reviewers. That artifact highlights the most important parts of the code.
Yeah, we did that, too.
But every time I end up, for some reason, digging up deep in the code, I realize that it's not nearly sufficient in our case.
Assuming the LLM never got anything wrong or otherwise had to be re-prompted, that means your devs were reviewing 130 SLOC per hour, on what was described as moderately greenfield (examining new implementations rather than comparing to old historical accidents).
How?
I don't want to sound flippant, but if the point is to add human thought to the mix, that's a high review rate even when examining small tweaks to an existing, working product, even with substantial AI help to pre-filter major gotchas before you bother spending a lot of human effort on the review. That's only 20-30wpm, but a review isn't just scanning or reading code, especially if you're trying to figure out how a new system which doesn't run yet will fit together.
The fatigue aspect matters too, but another important point I was trying to get across is that not all reviews are created equally. In a working, well tested, properly factored system, with a high quality PR, I can do a cursory, local analysis of the new code and be very confident that it works. AI code I've seen is ... not that. I wouldn't be comfortable in that review without manually checking preconditions and postconditions, defining invariants, examining lifetimes, and a host of other activities. Moreover, for a greenfield replacement of lots of SLOC, I'd expect that most of the code for the early PRs can't even meaningfully run or do anything important, making any sort of broader architectural analysis impossible.
I could see a world potentially where they came up with a magic prompt allowing each proposed PR to be cohesive, shippable, well factored, and everything else you need to be able to actually review it at a higher level and be comfortable with the results, but I'm skeptical. That's a major innovation if they managed to do so even as a one-off, and that wasn't the thing they highlighted when talking about the project.
Is that actually a high review rate? Especially if you know the language and domain. Sure, initially there's a learning curve for a new codebase structure, but lots of lines will also be trivial and many changes might also be similar to each other.
For small separate changes in isolation then maybe it's ok? But not for whole days 8 hours each.
But then you need to watch for bugs coming from interaction with previous changes and in 700k loc that might be nontrivial. How do you know which states are reachable and which are not? That takes time.
It only takes a botched condition here (forgot a "not"? swapped "and"/"or"?), a swapped variable name there, code that looks ok, but isn't.
Could you share more about how you guys set up the rewrite? I am interested in the shape of the software as well as how you achieved (or aimed to achieve) high quality.
> We could have gotten it done faster, but we were well aware of the potential for slop and thus paid an absurd amount of attention to (and tokens for) code quality. For example, there is a 100% test coverage requirement in both server and client, combined with AI-driven review rules that say all tests must be non-vacuous, plus a ton of invariants enforced via deterministic checks.
This is why my current view in terms of LLM productivity gains are:
- debugging and triaging 5x
- refactoring when you have tests in place 3x
- greenfield where you have zero tests is about 1.5x
> For example, there is a 100% test coverage requirement in both server and client, combined with AI-driven review rules that say all tests must be non-vacuous,
look, your application works, right? so it doesn't really matter what you or i think, and this is why AI matters. but this, your "100% test coverage" - that is pure slop. just 20 years ago, all the most popular software shipped with NO tests. are you getting it?
> just 20 years ago, all the most popular software shipped with NO tests
not sure what you're point is here. It sounds similar to "we use to use blood letting and leeches and doctors didn't clean their hands and everything was fine so what are you getting at?"
Good tests have real benefits. The fact that people shipped without them in the past in no way suggests they aren't needed or have no point.
Also, 20 years ago SW was tested by QA department and approved before shipping. Don't want to go back to that, but there were testing, just differently.
IMHO getting rid of proper QA done by teams of QA specialists is the main reason for the current software quality crisis (and that already started 15 years ago or so). We should go back to QA teams and proper QA procedures! Automated tests are no replacement, especially when they are set up by the same people designing and building the product.
> just 20 years ago, all the most popular software shipped with NO tests
Are you getting older? A lot of people anchor their intuition of time and history to a certain year. There are probably still lots of people who think the 1990s is not that long ago even though it’s now over a quarter century since it ended. Maybe you mentally default to 2012 or so, when it might be true that most popular software shipped without automated tests (although manual QA was a lot more extensive in 1992).
But 20 years ago is now 2006, and unit tests were well established as a best practice. Perl had extensive automated tests in the late 1990s that everyone who ever compiled Perl would have noticed, since they were run by default and produced obvious output. Kent Beck’s “Test Driven Development: By Example” was released in 2002, and popularized both the name and practice.
But 20 years ago is now 2006, and unit tests were well established as a best practice.
I agree with the spirit of what you wrote, but my recollection of the timeline is different. The first decade of the 2000s was peak Crazy Agile Advocacy, but IIRC it wasn’t until the 2010s that unit testing really became almost universal practice. Much before that and it was still tangled up with XP, TDD and lots of other things that certainly weren’t universally accepted as good practices (notwithstanding the strident advocacy of a certain group of consultants/authors/speakers/bloggers and their fans).
I remember, back in the mid-2000s, when we had some consultants brought in to talk about different aspects of quality and testing. There were several working groups, each led by one of those external consultants, and one of them was about unit testing. This was in a relatively large software development organisation for the time, a few thousand people, and while some parts of the organisation had some form of automated testing operating by then, it definitely was not the case that the well-known products produced by the organisation all had a unit test suite. Other practices we’d consider routine today, such as peer code reviews, were also in their infancy during that period: some were doing them, many were not, and generally we had much less experience of how to do them effectively than we have today.
As an industry, I don’t think we really matured in how even the most ardent fans of unit testing were writing test suites until the 2010s either. In the 2000s, we still had lots of people mocking the entire universe and then writing unit tests that were 99% testing those mocks because of 100% test coverage requirements, and similar dogmatic nonsense.
By the 2020s, I think there was much more awareness of that automated testing is generally a good idea, but there are different kinds/levels of automated testing and finding a mix that suits each project’s specific needs is important. One of the great benefits from the more recent AI tools, particularly the agentic ones over the past year or so, has been that it has clearly demonstrated both the value of a good automated test strategy and how much of a waste of time vacuous tests are.
>> but this, your "100% test coverage" - that is pure slop.
Not really, but I can see why some people think that.
We treat 100% test coverage as "required, but by itself not sufficient". It doesn't give us false confidence that everything will be perfect or anything like that. But it provides us with the discipline to make sure no corners are cut, and the bugs that are fixed don't come back.
One refreshing aspect was that during PR reviews we stopped debating whether something needed test coverage. Instead we focused on what was being tested and how.
> One refreshing aspect was that during PR reviews we stopped debating whether something needed test coverage.
I’d be curious to know what percentage of the time spent implementing tests would have otherwise gone to discussions about whether to implement them or not. ;)
> just 20 years ago, all the most popular software shipped with NO tests. are you getting it?
Not really?
About 20 years ago, I was working on Firefox and we had millions of tests on CI. I was working on a host of other open source apps and they all had tests (most of them had no CI, of course).
If only the careless drivers were those crippled or killed by their poor driving, it'd be a self-resolving issue as you imply, and nobody should care.
However, what actually happens is that careless drivers often cripple or kill innocent bystanders in other vehicles as a result of their poor driving. That's why seatbelt laws and improved vehicle safety features are a good thing.
> You won't need to wear sunglasses because apparently there's a >10% chance there is no future.
That's a reasonable risk to take if there's a possibility of Dario Amodei becoming a trillionaire with his very own Stasi. I think AI could be very useful to implement Zersetzung (https://en.wikipedia.org/wiki/Zersetzung), and I am excited about the future technological advances in that area. It'll be a really interesting white paper!
Well, I'm a cynical nobody so my predictions aren't worth much, but I suspect we'll get the trillionaires and really dark dystopia (compared to the one we have now) for at least a short time before The End comes.
I may have explored a related approach but as an append-only log riding on source-control to sync state between checkouts (git trailer metadata specifically)
Interesting but at the commit level seems too coarse. I have a model which models decisions similar to QOC (questions options criteria) and requires line level coverage from accepted answers.
gbrain is an awesome project, but yeah, that was exactly the itch I was trying to scratch! I wanted something that felt as fast, portable, and bulletproof as a single .db file. Zero background daemons or cloud dependencies—just fast, hybrid queries.
However we've now got local AI to do it for us:
https://github.com/corv89/rookmark
reply