This always feels odd to me. It would seem a fairly straight forward optimization of the interpreter to special case the different types that it can reduce.
That is, why couldn't they have done the essentially same trick that you reference for += with reduce?
There is no such trick. Python is only now getting those sort of JIT style optimizations, and that one in particular still hasn't hit. Do not use += on strings in a loop unless you are certain the iteration count will be small.
There is an optimization for lists, and maybe that's what GP is remembering. l += is functionally different from l = l +. The former mutates l, whereas the latter creates a new l. The difference matters when the line above is m = l. The mutation version will mutate m as well (they're the same reference), the creates new version will not. This optimization can just as easily turn into a footgun if the programmer is unaware of it, and in that sense is unpythonic.
I’m not a fan of this kind of “fancy” optimization anyway.
It’s too fragile. I may make some innocuous change, now the compiler cannot recognize the pattern and performance falls off the cliff.
I’d rather have the reliabile performance than the absolute fastest possible result. Then if there’s an issue I can catch and fix it reliably with profiling, not deal with a heisenbug based on whether the compiler can match the pattern.
That sounds nice but in practice it’s probably not super helpful. Yes you could make a special case for reducing “+” over integers. But in python you can generally not promise that all the inputs are strictly integers, and you can’t even promise that your “+” function has no side effects.
I used to think of it more as O(1) being the expected average of the cases. My guess is I'm probably thinking of it more as an amortized cost, in that framing? (That is, not that it is the average case. Is the average of all cases.)
To your point on the worst case being something you may worry about in denial of service, I think it is often the case that people should set bounds on what size N they will deal with in a program. And then decide from there on whether you are worried about some of the more esoteric growth patterns.
It is both amortized and average, because the map may need to grow. But the complexity without growing is average, not amortized (it's possible to build hash functions for which the probability will mean O(1) for all accesses, and hash functions which will be O(N) for all accesses).
Amortized is a bit different, though? It largely implies that there is a heavy cost periodically. Average just implies that it varies. Which, fair that "massive cost periodically" is compatible with that. Just seems to have a very different context, to me.
With hashtables, it was more that it was not guaranteed to be the minimal cost. Just, depending on statistics of the data that you feed to it, it shouldn't be the worst case.
But is it really what we want, machines with the same defects as humans? I don't want a pocket calculator that make mistakes "sometimes" so I have to double-check the results, I want a pocket calculator that works (to those who want to argue that pocket calculators don't give the correct result for (1/3)*3: STFU).
>But is it really what we want, machines with the same defects as humans?
Sort of, actually. I think we humans actually have some intuition that we'd be more effective if our cognition were augmented more directly by machine strengths: the ability to run precise calculations, more memory, ability to look facts in some sort of knowledge graph.
I think we're on the right track, but instead of augmenting humans with machine strengths, we're building intelligence in hardware in a way where it can access that augmentation. Plus, then we can quickly distribute updates, run parallel instances, etc.
If intelligence is compression, and hallucinations are essentially loss, then as the models grow in size performance (at least as far as hallucinations) should reduce. Or we'll get things fast enough that we can afford to stop relying on model weights for memory and check an increasingly larger set of discrete facts as part of reasoning.
Right now, the models are making trade-offs. As compute grows, and inference gets faster, we can make fewer of those trade-offs and start to use the unique strengths of machines to fill the gaps we're seeing, I suspect.
One of the biggest strengths of a computer is reproducibility. The worst software bugs are inconsistent or non reproducible. The least useful calculators apply rules inconsistently, to your example.
the inconsistency of LLMs is by far one of the biggest gripes I have with them. Closely related to their apparently deep desire to avoid following instructions.
I know these are both a byproduct of noise (which is somewhat tunable) and noise is inherent to these systems in a lode bearing way.
I still hate it. it’s holding the technology back. I don’t honestly see how we can safely or even successfully approach the idealized realm of AI without bypassing this problem, which to my understanding, probably means not using language models at all and trying a totally different approach. But I really don’t know much about machine learning, I’m a super novice compared to a lot on this website.
When we are really thinking about something we do it forwards, backwards and middle out, and regenerate and distill many times.
When we do meta thinking about that process after the fact, two things happen. 1, we change our total “thought” by adding that meta thinking to it. And 2: it’s a very lossy process, because we don’t have very good data about what our brain or mind was actually doing during that first think and emotional factors are nearly always at play and even more complex.
Now for the more complex AI, the fragmented process of multiple agents and loops and reruns are pretty similar to that first think we do. At least structurally. But the meta think is where they differ. They have no emotion, but they also have even worse data about its own function. They constantly degenerate so I would argue their “changing the thought by thinking about it” factor is also generally way higher than ours.
Getting better at consistent/reproducible thinking, with many ‘steps’, that leaves good documentation of that thinking behind for future analysis, has to be one of the more important areas for the big flagships going forward. I’m certain that “what is this fucker doing and why” is the biggest pain point for AI researchers. Or the math, it’s usually the math.
But you’re correct in the general structure; they generally do the same post hoc analysis we do, just noticeably worse because of their opaque nature(even to themselves) and general degenerative instability.
No. People have an inner monologue, partial results and ideas and they remember that.
If they've worked some minutes/hours/weeks on something and you ask them why did they do that, they will either answer honestly and truthfully, lie, or say "I missed that/didn't seem important so I just chose something at random".
Perhaps up to 50% of people actually don't have an inner monologue, much like many have aphantasia where they can't actually see anything in their mind either.
That’s not how that works. I don’t usually have an inner monologue either, but I do have an abstract stream of thought. It’s not as if I am always acting on instinct.
If you have examples of studies that verify that people are always aware of the gaps in their memories of why they did things rather than their memories sometimes "filling in the blanks", so to speak, I'd be interested. My impression is that the opposite has a lot more evidence in studies (e.g. around the reliability of eyewitness testimony).
It's not clear to me whether you're aware of a rigorous basis for your claim or you're just inferring based on what you think makes sense, but I can't help but wonder if it's the later, in which case regardless of the mechanism, the outcome certainly seems to resemble what happens with LLMs.
No, I was asking why you were confident that 3 didn't exist because the comment you said before was that people will either recall correct, lie, or not remember. Lying is not the same as remembering incorrectly but not realizing it, so I agree with your relaxed list. I still don't understand how you think this is any different from an LLM though, which will also always give one of the three options you listed just now.
I think this is true with some people, but I don't think this holds true for some (or even most) people across the US (at least not all the folks I've worked with)
Models with reasoning > Instant have an inner monologue and when asked why they did something they can deduce based on that monologue. I have asked things like "what steps did you take" and "what was your reasoning" and the answer matched the thinking output of the model.
>No. People have an inner monologue, partial results and ideas and they remember that.
Yes, but the vast, vast majority of decisions you make either don't take place via an inner monologue, or include details that were not actively/consciously "thought" and reasoned with in your inner monologue.
And yet, when asked why you did something, you're not likely to respond "sorry, that decision was made subconsciously". Instead, you use your inner monologue to try to backfill in a reason why. That reason may be correct, or it may not be. You don't actually know, since you have new data that may be updating your own internal state as you try to rationalize it after the fact.
Actually split brain experiments tells a different story. The left hemisphere actively confabulates, inventing plausible explanations for actions it didn’t initiate, suggesting that much of human self-narrative may be post-hoc storytelling.
> The left hemisphere actively confabulates, inventing plausible explanations for actions it didn’t initiate, suggesting that much of human self-narrative may be post-hoc storytelling.
Sounds like the left hemisphere usually uses something from the right hemisphere to answer those questions and it can't do that if it's been cut off?
We know that our brains are capable of hallucinating due to substances (drugs), being asleep, brain damage (including split brain), hypnosis, etc. Just as RAM damage make your computer do weird shit. That doesn't mean it operates that way normally.
You can't just remove a huge part of a system and then assume that the whole system behaves the same.
Perhaps, but it's also a rather questionable achievement: We've already had several decades of program-output that can match humans with literal brain damage.
If anything, biological comparisons should make us cautious. Consider the vast gulf that still exists between the finest artificial organ/limb versus the OEM parts of natural nanobots.
That was my experience with Claude when my vibe-coded project was small.
But now that I've been working on it a month and there's a lot of documentation, it's pretty clearly ignoring parts of the documentation and parts of the code. It will come up with some ridiculous statement about how something works, and I'll challenge it, and it'll admit I'm right.
It definitely reads more documentation than any programmer I've ever worked with (myself included) but because it doesn't have a memory other than the documentation, it still makes mistakes like that.
I haven't turned on "memory" or tried it with Codex, so I don't know how that'll change soon, though.
Yeah the biggest task these days that I do manually is curating the documentation. AGENTS.md in every major directory, and a variety of reference docs that are explicitly referenced in those files.
# See DOC-ITEM-NAME
DOC-ITEM-NAME.md
When referencing documents, always use the exact syntax See <TAG> - this is enforced by a lint on precommit
And those doc items are basically all of the values, architectural, strategic, and tactical items. It's a poor man's in-repo RAG but it's shockingly effective, especially if you keep them small. I may migrate some/all of them to skills over time, but I usually update them biweekly, and I only allow agents to make small edits or propose new notes. And typically I go through and delete or curate any agent edits before merge.
Depending on language I've seen this scale past multiple millions of lines of code, as long as you pair it with all of the linting and tooling that you can possibly build.
I see this observation frequently, and I dislike how it often has the unsound subtext of: "Therefore something is going well or at least not too badly."
If you build a robot where a pressurized hose leaks causing fluid to destroy part of the circuitry, we don't praise it as progress towards the human ideal of having brain aneurysms. A similar failure-path is not a reliable indicator of a similar success-path.
AFAIK this really is true. I've seen some videos about patients who had the connecting part between the left and right halves of the brain cut as a (archaic) treatment for epilepsy.
While it did help the epilepsy, their brain was essentially two brains controlling two halves of the body. With one controlling speech. There were experiments where one eye was shown some instruction text, the corresponding hand performed that instruction, and when asked why they dix that action, the speaking half just made up some plausible, yet completely wrong reason, just like an LLM.
People do both. Some choices aren't worth the time and effort of detailed analysis and contemplation and some are basically instinctual, but there are plenty of times that choices are carefully considered and well reasoned before being made and acted on.
Every time I hear someone complain about hallucinations, I laugh at the total lack of self awareness about our species. Humans are just as bad (now, probably worse) at telling the truth, whether due to intention or poor memory.
This is true of a ton of online discourse. Worse, when the headline of a claim doesn't even match the article it is fronting. I've seen more than a few articles that basically contradict the headline, but end in a "despite all evidence, we think it is correct to say X."
Huge shout out that if you are going to be in Japan, you need one of these cards. I had thought I could just buy a two day pass in Tokyo. Which, you can, but you are almost certainly going to be limited to what routes and lines you can use, then. Which is brutal when you can't tell google maps which card you have.
The Tokyo Metro Ticket (24h/48h/72h) is very convenient, though. I just looked at the prices and the longest one went from ¥1500 to ¥2000 since November 2025 but I think it's still very valuable, especially if you've never been to Tokyo and want to travel between a lot of places.
JR is better if you go to the countryside of Tokyo but for the city itself Metro is much more convenient, IMO. Perhaps it depends on the station you commute to, though.
When I travel to Tokyo, I usually do at the very least 4 Metro trips per day, and let's say the minimum fare is ¥178,it would come down to ¥712 daily.
Allows you to specify up front what passes you have, worked brilliantly on our honeymoon, especially when figuring out which Shinkansen reservations we needed to make.
As a rule, unless you have a very specific itinerary that involves a ton of travel within a single train company's network, passes in Japan are not worth it. This applies to both local subway passes and the sadly nerfed Japan Rail Pass for bullet trains. Even the Seishun 18 (unlimited travel for ¥2000/day, but local JR trains only) ain't what it used to be since they switched to requiring continuous days.
And there are several calculators online where you can punch in your itinerary and it will tell you the best pass to buy, if any.
On my last trip it only made sense to buy a pass for a single ~4 day segment that included two long Shinkansen rides plus an expensive sight-seeing bus.
yeah they still have the ticket stations next to the one to refill/buy IC cards. and thanks to the 20/21 Olympics in english now too(not 2 levels of menus deep like many of them used to be) so pretty easy to do
tbh if you only stay in Tokyo then the combined Tokyo subway pass (Tokyo Metro and Toei Subway) is the best all around deal, you don't need anything else. ¥2000 for 72 hours
If you do 5 Metro journeys per day—which is very reasonable if you're a tourist—at the bare minimum of ¥160 per trip, you would save money by buying the Tokyo Metro Ticket for 72h.
I think, in general, one should have the view that anything that can miss, will miss. Agreed that it is better to have something with a lower miss rate.
I always assumed it was a bit of a common practice from image based development practices. I'm now assuming I was incorrect on that being the case?
I'd love to read a good history of the different development styles. I know we often have a good exploration of the parts of an executable file. I don't recall seeing a good exploration of different compilation/build styles through the years.
Yeah, me too. I hear Ada has something really weird, but I don't know anything about it. Smalltalk and especially Gemstone also have some really different approaches, and OS/400 also does some pretty strange stuff.
Mostly I've noticed unusual compilation/build styles when they come into conflict with version control systems. One weird one was when I was contracting 01996 to 01997 at TCSI, Teknekron Communication Systems, Incorporated, which I guess was a spinoff of Teknekron dedicated to building telco software. I was working on a project called STNMS, the SONET Tunnel Network Management System. There were some pretty funny things about the project, including some amazing recursive acronym soup. One of the major components of STNMS was the NGEMS TOM, which stood or NGDLC Element Management System TCSI Object Manager, or, fully expanded, New Generation Digital Loop Carrier Element Management System Teknekron Communication Systems, Incorporated, Object Manager.
Anyway, one of the interesting things about the system was that it was built on a distributed object system like CORBA or PARC's ILU or DCOM or eventually Java RMI, where you would invoke methods on objects, and the methods could execute in another process or even on another machine. So the NGEMS GUI would invoke methods on NGEMS-related objects which were actually implemented by code running in the NGEMS TOM, which might be on another machine, so that you could be running the NGEMS GUI on lots of machines, all calling methods on the same NGEMS-related objects, which could therefore avoid any concurrency-related bugs, I guess.
I'm not totally sold on this actually being a good idea, among other things, because the various TOMs stored their data in an INFORMIX-ONLINE SQL database server, which had pretty decent concurrency control already, and was pretty fast.
I was responsible for maintaining the bug-tracking system, the INFORMIX servers in the test environment, the build system, and the software version-tracking system, which was a pile of csh scripts built on top of RCS, running on NFS. And I was also responsible for doing a full system build, I think about once a month, which usually took me about a week of tweaking things. I was 19, and it was 01996, so I didn't know any better. I'd learned SQL and RCS by reading the O'Reilly books on the plane out to California.
Anyway, TCSI wasn't using a CORBA ORB or an ILU ORB or a DCOM ORB for their RPC layer. They'd written their own ORB, whose name I forget, but its IDL was generally called "MOSU", and lived in files with the .mosu extension. This stood for Methods, Objects, and something or other I can't remember; just as with any ORB IDL, you needed to define the class interfaces and methods and parameters that you could invoke through the ORB, so that the clients and servers would agree on what they were. You'll have to forgive me for the vagueness — it's been almost 30 years, and even at the time I wasn't writing code that went into the system, just tweaking it until it compiled so the 30 or so programmers and testers on the project could get on with their work.
The thing that caused trouble with version control was the .moslog file that accompanied the .mosu file, assigning numbers to methods and objects.
For efficiency, there was a small positive integer assigned to each of the methods, objects, etc., that the ORB's protocol could refer to. The client and server also had to agree on these numbers, just as in Protocol Buffers you need a unique tag number for each field that all users of the .proto agree on. You could solve this by burdening programmers with the need to declare these numbers in their IDL (the .mosu), as protobuf does; or you could solve it by reaching agreement on them when the connection was opened and then doing the mapping dynamically during the session, as X-Windows's XInternAtoms() does; but the MOSU IDL compiler solved it by assigning these MOS numbers automatically, starting with the smallest natural numbers.
But, to ensure that it assigned them consistently, it would log the assignments to the .moslog file.
Therefore, the .moslog file had to be checked in to source control, but the build process might modify it. And, if different programmers added things to the same .mosu, their .moslog changes were almost certain to conflict, because the compiler would assign the same next number to both things — conflicts I seem to recall I sometimes had to clean up by hand. Of course this required sneaking around behind the back of the version-tracking system, since RCS requires exclusive locks to modify a file, but often the programmers did have to sneak around thus in order to get their work done, for example because someone else had the .mosu and the .moslog locked that day.
TCSI is now dead and gone; https://en.wikipedia.org/wiki/Teknekron_Corporation tells me that it was bought by Rocket Software in 02003. I think the STNMS project failed, and BellSouth, the company that was paying TCSI to build it, also no longer exists. So, I doubt anybody has used these primitive tools in 20+ years, and I haven't seen them described anywhere publicly before, perhaps because TCSI foolishly thought they were a competitive advantage at the time. (I hope I haven't made too many errors in my recollections.) But it's interesting what we can learn from them for the future.
Today, there's a somewhat similar thing that happens with generative testing systems like Hypothesis: when they find a failing "instance" of a property-based test, they record it in a database of failing tests, so that you can always use it in the future. So the "build process", or rather the test process, sometimes produces data that must be checked in to the version tracking system. This doesn't mesh well with conventional version-control workflows, and especially established continuous-integration systems which blow away the universe when the test suite finishes running.
Of course, training AI neural networks doesn't fit cleanly into the conventional version-control model either.
This is silly. Fun. But silly. Is like claiming that math on the numbers that everyone knows is actually typed. Ignoring that that is only true if you do the effort to also do your operations on the types.
There are CDNs everywhere, true. It is not true that everyone deploys to them all, though. Also it dodges that the networks available to everyone are still not equal.
That is, why couldn't they have done the essentially same trick that you reference for += with reduce?
reply