I can compile stockfish from source and use it to beat other kids in my class in chess. Behold, I am a chess genius.

I suspect (in a probably ignorant fashion) that this is because learning process has been reading a lot of algebraic chess notation (such as "1. e4 e5 2. Nf3 f6 3. Nxf6 gxf6 4. Qh5! +-") then, to play, generating more of it without considering the rules of the game. This is exactly how it's always felt to me when playing chess against LLMs. Sure, "1. e4 e5 2. Nf3 Nc3" looks innocent to somebody simply learning the syntax of algebraic notation, but that Nc3 by black is an illegal move.

An LLM is the wrong approach for playing chess.

Are you saying that modern LLMs cannot play chess now, or that LLMs (GPT architecture) cannot be trained to play chess well?

Or are you saying that neural networks in general cannot (practically) be trained to be an above-average chess player?

Or are you saying that it depends on the input? Would it be better if they were given a picture/drawing/ascii art of the board? If so, surely they can produce it at will?

Neural Nets can be trained to play chess very well and have been doing so for a long time (see Stockfish and Leela as some of the most popular/strongest ones - top GMs have no chance against them), but these are dedicated models, where the game rules are encoded in the learning process, as opposed to large language models which are natural language processing models. Technically you can give an LLM a lot of chess books and games and it will be able to spit out chess notation. Put a webapp on top that renders text moves to the board and it looks like it's playing chess. But it isn't really.

https://en.wikipedia.org/wiki/Chinese_room I think about this once in a while. At some point if it does the thing almost perfectly is it still not doing the thing?

I suppose if you all you need is a good enough opponent for the average person out there, sure this is good enough.

I was more talking in reference to why the LLMs in the above linked paper were producing so many illegal moves, and it is because they are not hard constrained by the rules of the game. Of course, a loop can prompt until a valid move is produced and then rendered on a screen. But why do this? I suppose, who am I to say what should be done or not, but a specialized tool being better than a general one at its specific job isn't particularly surprising.