>If that were true, we should have seen LLMs play good chess by now.
Not at all. LLMs learn by imbibing a mass of relationships as isolated fragments of information. There is a certain amount of sorting and indexing that happens during the training phase. There is also a certain amount of compute executed on these relationships during inference. LLMs can model processes that fit within the compute budget. Language translation works well because language is lookup-heavy while being light on compute.
Chess is a compute heavy game of finding the best move out of many possibilities with wide variation in the quality of each move. Humans cut through the compute requirements by reinforcement and learning intuition. LLMs don't get reinforcement on chess so they must compute during inference a unified model of chess. Developing a strong model of chess from raw fragments of information is simply not in their compute budget.
>> Not at all. LLMs learn by imbibing a mass of relationships as isolated fragments of information.
You gotta be careful how you use the word "relation" here because there's an informal meaning (I'm related to my cousin) and a more strict, formal meaning, that is used in computer science e.g. in the "Relational Calculus" etc. In the formal sense, the one relation that LLMs learn during training is the co-occurrence of tokens in a corpus of text, what's called more technically a "collocation" relation. Nothing says that this is enough to play chess, so I'm indeed doubtful that they can.
So you weren't giving it an updated board state after every move? If you want to compare apples to apples, it should give an updated board state for each move, or you should play blindfolded.
Blindfolded flex by OP aside (I can barely play when seeing the board), considering reasoning traces and their nature, if we want to be fair, a person would have to get the moves, but be allowed to write them down or draw up a board in their notepad. My working memory can barely handle five chunks, a models reasoning tokens are masses of written text in comparison.
An LLM has been trained to do everything it does blindfolded, "only" using perfect recall of everything in it's hundreds of thousands of steps of context, and hundreds of layers of KV cache. It's a computer - it has a massive advantage over a human.
The fairest apples-to-apples comparison of an LLM whose training data included chess games would be a trained human such as Magnus Carlson, who can quite happily play a dozen or more simultaneous blindfold chess games.
>Where is all the training and practice LLMs did to become so good at coding?
Coding is a matter of translating the natural language description of a problem to the code specification while keeping the semantics fixed (and imputing the unspecified semantics as necessary). It is not considerably more difficult than translating between two dissimilar natural languages. Chess isn't a matter of language translation, but a compute heavy game of finding the best move out of many possibilities with wide variation in the quality of each move. Chess takes directed practice and reinforcement whereas language translation does not.
And how many years of direct play and study does it take for a human to get good at chess or any other game? Absolutely no human ever could be good at chess just by reading a few books, or even every book on chess. That's just not how the brain works. If LLMs could do that they would truly be superintelligence.
> Absolutely no human ever could be good at chess just by reading a few books, or even every book on chess.
Maybe not, but you'd be surprised how little it takes.
A six year old child can learn the rules of chess well enough to be able to play legal moves only in a single day. And they can improve their game at a pace which is almost frightening to behold. I have taught children, and I've witnessed significant improvement materialise in a single game. LLMs have probably thousands of chess books, games, videos, etc in their training data, yet they are unable to even follow the rules.
This is, at the very least, interesting. It illustrates many of the things brains can do, which current ML systems in general, and LLMs in particular, can't.
It is interesting, but people are drawing the wrong conclusion from it. For one, LLMs don't go through a "chess learning phase". They're not analyzing a board as they're learning the rules or studying games to create a coherent model of chess. They're just imbibing raw relationships as isolated fragments of information. The fact that they can't unify this into a coherent model of chess playing in one shot and execute a competent game says nothing interesting about the limits of their intelligence. If you give frontier models the rules of chess in their context window, could they perform only legal moves? I bet they could, excepting trickier scenarios like pins and failing to respond to a check. But those kinds of scenarios have to be reinforced in any human player as well. Even Super GMs fall for mate-in-1's occasionally which is functionally equivalent to those kinds of failures.
We're obviously using different meanings for "good" here. But aside from that, it took 100's to 1000's of reinforcement iterations for your brother to play competently. While certainly impressive, that is still an entirely different category from piecing together disparate facts learned during training (LLMs aren't analyzing a board as they're learning the rules or ingesting PGN files), to executing a competent performance in one shot.
No, learning is definitely not a sign of super intelligence. I know words don’t mean anything anymore, but that is simply general intelligence, despite the claims we have reached this milestone.
If something has general intelligence it should be able to read the rules of a game and follow them. Therefore an artificial general intelligence (AGI) should be able to do this.
So we have a situation where very powerful and influential people are saying we will have AGI in 6 months (if we don’t already), yet the facts on the ground are so clearly pointing in the opposite direction.
I would bet a lot of money that Astra can follow the rules of chess (perhaps if repeated within the context window). Also, this is a different argument than what I responded to.
I can write you a benchmark to prove it even with a heavy handed system prompt Astra will make an illegal move during the course of the games first few moves are generally ok since it's just throwing out learned moves.
I wonder if I would do better as a human, maybe? Or would I happen to have one move in 1500+ that's not valid?
I could see myself messing up something at some point if the board is complicated enough and trying an illegal move, perhaps if a piece somewhere would attack my king if I moved another piece. Even through I do know the rules of chess, and I have played a few games once every so often.
> generally read the rules of a game and then follow them
How many times do you think chess.com prevents illegal moves from being executed? Even Super GM's fall for mate-in-1's occasionally, which is functionally equivalent to missing a pin or a check. This idea that LLMs failing to only ever make legal moves undermines their intelligence doesn't pass the smell test.
Chess.com has to accommodate people who haven't learned the rules yet on the low end. On the high end, people are commonly playing fast enough that they're often outlining sequences of multiple "pre-moves" during the opponent's turn in order to avoid losing on time. And no, I would not agree with that functional equivalence.
It only matters if you are claiming it to be general purpose.
If you admit that it's just a collection of narrow capabilities - whose strength is mostly confined to the 1000 or so RL environments it was post-trained in, then there is of course no expectation of it being general purpose.
The AI companies seem to heavily want you to believe it is some some near human level general intelligence, so therefore pointing out all the things it can't do is very relevant.
Because we want to use this as a replacement for humans, and the average human can learn the rules of chess without needing to see the rules explained hundreds of thousands of times in millions of games.
So, yeah, it matters if a model has millions of examples of something in its training set and still cannot follow the rules.
We're not talking about learning the rules of chess here, but playing a competent game from just being shown the rules. Why is it so hard for people to keep track of the thread of discussion?
> We're not talking about learning the rules of chess here, but playing a competent game from just being shown the rules.
Okay, lets go with that: it's the "shown the rules" bit that we are arguing about.
The argument is that a human may play maybe a dozen games after learning the rules, after which they won't be inadvertently attempting illegal moves. What we are observing with SOTA models is that, even after seeing millions of chess rules, rulebooks, actual games, etc, they still attempt illegal moves.
This does not point to generalisable and adaptable intelligence, such as we see in the average human.
This is not good reasoning. Humans need at least dozens if not hundreds of reinforcement sessions to only make legal moves, and still occasionally fail (consider pins, discovered check, failing to respond to check). LLMs must one-shot a competent game after imbibing a mass of disconnected units of information about chess. Nothing about the two are similar.
It's also immoral to expose isolated populations to all manner of the modern world's vices, optimized to be maximally addictive and maximally extractive, for which they have zero defense or even knowledge of their dangers.
But they're not equipped to decide whether they want it or not. It's like offering drugs to children saying they can choose if they want it. Children simply aren't equipped to make that decision, neither are people in these populations. I don't think we should keep them isolated, but contact should be limited initially. The technology we introduce them should be limited, enough to increase their quality of life and eliminate disease and so on without being utterly disruptive. Over time more technology and information can be introduced.
No population should be kept living a life we would consider extreme poverty out of some noble ideal. But we also shouldn't completely disrupt their way of life out of a different noble ideal. Wealth as we understand it isn't the only thing of value. Human connection, meaning anchored in community and deep roots, a sense of belonging and so on, are what really matter. Introducing modernity to such populations will sweep all of that away in one motion. These people would be left adrift in an alien world devoid of personal meaning, their only solace whatever addiction appeals to them most.
> But they're not equipped to decide whether they want it or not.
Why can't you say that about, idk, folks on welfare in the United States? How are they in a position to decide whether buying pop on food stamps is what they want or not? Maybe a woman doesn't really have the tools she needs to decide whether she wants to keep a pregnancy until she's had some counseling and time to cool off?
Obviously I'm not comparing these situations directly, and frankly I'm not sure you're wrong, but the point of view you are expressing here leads us to all of those scenarios and I think we should reconcile that.
I agree the application of the logic doesn't end with isolated populations. The relevant difference is that those brought up in the modern world have/had the potential to engage with knowledge of addiction and other vices, and decide for themselves how to steer their lives in an informed way. We draw a somewhat arbitrary line for being responsible for oneself as the best way to structure society given the tradeoffs. There's just no sense in which these isolated populations as a whole can be sufficiently informed about the modern world to suitably navigate its complexities to lead to their prospering in it.
But on the other hand you are assuming the modern world is bad and so therefore exposure to it is a dangerous and negative thing. I'm here to tell you that having access to medicine and playing sports and whatever is pretty great too. It's not all bad. I think you're taking a little bit of this Star Trek version of "leave them alone to develop on their own" but if the shoe were reversed and there was an alien species out there that could come down and talk to us and share knowledge and potentially harmful things as well, I'd like to make that choice for myself.
It's certainly a complicated subject, but I'd just challenge you to think of the positive version as well, not just the negative (addiction/vices/&c). It seems you have a default negative view of the world, some of us have a default positive view of the world. Modern life is pretty awesome!
My concern isn't mainly about addiction and vices, its more about a loss of meaning and being adrift in an alien world. The addiction/vices are just what one falls into when your life becomes devoid of meaning. I think the mental health crisis that most of the western world seems to be engulfed by, despite living in an age of endless wonders, goes to show that there is something deeply wrong with our modern way of living. My caution is against introducing this "sickness" (for lack of a better term) into populations even less able to defend and preserve personal meaning against an onslaught of technology and ideas that would evaporate their personal sources of meaning in an instant. That is no gift to anyone.
I just think you are super focused on the negatives. My life is great, I don't have a mental health crisis, most of my friends are doing well and having kids and enjoying their lives, and more.
There might be something deeply wrong with the modern world (which by the way includes other countries ranging from, idk, Iran, to Vietnam, to Korea) but there are a lot of things deeply right about it: opportunity, human rights, women's rights, independence, opportunities to build rocket ships or sail the world. That seems like one hell of a gift that I wouldn't want to deny anyone. If anything, providing opportunities and giving human minds the chance to flourish and be creative seems less of a sickness and more of a cure.
The danger when making decisions on behalf of other people is in imagining yourself in their position, rather than imagining them in their position. So you have a happy fulfilled life, you have the kinds of human relationships that provide deep meaning to you, you have the right kind of curiosity and wonder to be fulfilled by witnessing burgeoning technological progress, etc etc. Yes, being deprived of all that would be very bad for you.
Now imagine someone psychologically very average whose source of meaning is in the familiar environment where they have an acceptable amount of status, broad community support and much social engagement. They have a peculiar culture and social practices which gives them deep personal meaning. Now imagine all of that is swept away; no social status, no familiar environment, cultural practices gone, extended family scattered to the wind. What value does the wonders of the modern world have to someone who has lost everything they value?
Consumerism, technological marvels, and endless entertainment are the modern world's answers to the waning sources of personal meaning in modern life. Turns out that it's all a very poor replacement.
It's a real phenomena. Mental imagery corresponds to voluntary activation of the visual cortex[1]. The quality of the self-reported imagery corresponds to the degree of activity in the visual cortex[2] while imagining some visual scene. People with aphantasia have little to no visual cortex activity.
I had thought this was all people misunderstanding each other's descriptions of their inner subjective experiences; but if this is accurate I switch sides.
Electricity in the brain is all we are.
Also its more aligned with the general theory of mind, that destinct variety of peoples would naturally observe destinct variations of sensory qualia and exhibit different internal subjective experiences.
Naturally conscious language fails to articulate these gaps because they are themselves the lack of words.
And the claim implied by TFA here is not that the VC is summoning or rendering images, but actually receiving and using a signal to suppress the visual attention from the eyes during imagination. Aphantasics suppress real vision less, so in theory they can’t see their imagination model of the world as cleanly.
I'm familiar with these studies but I don't think you can assume that activity in the visual cortex is phenomenal experience. I'm sure there are differences in self-reported aphantasics and people who don't claim to be aphantasics, but I don't think the existence of differences proves there is a difference in phenomenal experience.
Prove is a strong word. But it supports the existence of the condition as it coheres with our scientific models of visual experience. You can always reach for obscure philosophical puzzles about phenomenal experience to try to invalidate the science, but at that point you're just having a different debate than the rest of us.
>pacing between the us labs? what does that do for china?
I've seen no indications that China is in any kind of race with the US. They seem to be content to be 6 months behind and just copy what we do. They would probably be content with a bilateral agreement to pause progress.
The China bogeyman serves only one purpose, and that's to clear the way against anything that may cause friction with forward progress.
>endless abundance of solutions and ideas cannot coexist with a healthy society.
This really cuts to the heart of the problem with AI. Not only does AI undermine the monetary economy, it undermines the intellectual economy. What is humanity without the need for collaboration for survival or for intellectual progress, ultimately providing the impetus to build something greater as a result? I don't know, and I'm not looking forward to finding out.
It is not reasonable to not believe anything unless there is "evidence" (narrowly construed as an observation incompatible with the negation of some state of affairs). Beliefs have a wide spectrum of characterizations, and not all belief must wait until publicly corroborated evidence is available. Some events defy evidence and we can and should use experience and reasoning to infer unobservable states of affairs.
This is olympic level mental gymnastics to justify believing things without evidence. The double negative with the word evidence in scare quotes is chef's kiss.
I believe the sun will rise tomorrow without "evidence" (again, narrowly construed). We all do. It's only those who abuse the idea of epistemic hygiene who claim otherwise, usually with ulterior motives.
The evidence is the history of the sun rising since time immemorial, as well as the science of physics and cosmology that models the sun's motion with respect to the earth.
Yes, reasoning with models and making inferences are perfectly acceptable forms of evidence. But you can model the world based on ones knowledge and experience and infer when some event doesn't fit the typical pattern, then form beliefs about what that means. Also perfectly fine from an epistemic perspective. The rejoinder "there's no evidence" to a belief based on such an inference does no work.
Not at all. LLMs learn by imbibing a mass of relationships as isolated fragments of information. There is a certain amount of sorting and indexing that happens during the training phase. There is also a certain amount of compute executed on these relationships during inference. LLMs can model processes that fit within the compute budget. Language translation works well because language is lookup-heavy while being light on compute.
Chess is a compute heavy game of finding the best move out of many possibilities with wide variation in the quality of each move. Humans cut through the compute requirements by reinforcement and learning intuition. LLMs don't get reinforcement on chess so they must compute during inference a unified model of chess. Developing a strong model of chess from raw fragments of information is simply not in their compute budget.
reply