This is really interesting because i ran into a pokemon bot the other day were its training led to calibration of 50% winrste at all levels of play on Pokémon showdown. It was a complete accident.
It's not hard to make a chess bot that plays at a 1300 strength, i.e. its rating would converge to 1300 if it were allowed to compete. But it will not play like a 1300-rated human. It would play like a superhuman genius on most moves and then make beginner-level blunders at random moments.
Making one that realistically plays like a human is an unsolved problem.
Of course, you are right. But (the linked site) at least has a bot that plays the opening like a human of chosen rating perfectly. It stops working after the opening-stage (since it just copies moves from humans in the lichess game database), but it is still very impressive. For later game stages, some other method would have to be used (unless we play multiple orders of magnintude more games on lichess).
Now that i think about it, i remember the people in the alphago documentary talking about the bot giving its moves percentage scores in both how high winning % the move had and how high % chance that a human would have made the same move that it just played. I wonder why they never showed what a full game of the most human-like moves from alphago would look like. Maybe it actually worked, by feeding it all the pro games in existence, and training it to play the high human % instead of the higest win probability moves like they did in the end.
I think this can be achieved with some ease with a machine learning model. You will have to train it on games between 1300-rated players and below. A transformer model might work even better in terms of the evenness of play (behaving like a 1300 rated player throughout the game).
Take the computer which beats Magnus and restrain it to never make the best move in a position. Expand this to N best moves as needed to reach 1300 rating.
Even 1300s sometimes make the best move. Sometimes the best move is really easy to see or even mandatory, like if you are in check and MUST take that checking piece. Sometimes the best move is only obvious if you can look 20 moves ahead. Sometimes the best move is only obvious if you can look 5 moves ahead, but the line is so forcing that even 1300s can look that far ahead.
Despite decades of research, nobody has found a good way to make computers play like humans.
Then I can't refrain from asking: and what's the style of LLMs? For example the ChatGPT which is apparently rated around 1800? That should be completely different from that of a classic chess engine.
LLMs can be trained on chess games, but the tree of possible board states branches so fast that for any given position there is simply very little training data available. Even the billions of games played on chess.com and lichess are only a drop in the bucket compared to how many possible board states there are. This would have to be split further by rating range, so the amount of games for any given rating range would be even lower.
This means that the LLM does not actually have a lot of training data available to learn how a 1300 would play, and subsequently does a poor job at imitating it. There is a bunch of papers available online if you want more info.
LLMs already do play at elo ~1400-1800. The question was how does their style feels like to someone who can appreciate the difference between a human player and a chess engine (and the different styles of different human players).
I can’t speak for ChatGPT, but your intuition is correct that LLMs tend to play more like “humans” than Stockfish or other semi-brute force approaches.
You've identified a potential strategy by which a computer can play like a 1300-rated player, but not one where it will "play like a 1300-rated human". Patzers can still find and make moves in your set of N (if only by blind chance).
Yeah, you would have to weigh the moves based on how "obvious" it is, such as how active the piece has been, how many turns until it leads to winning material, or other such 'bad habits' humans fall for.
I think there's a real difference between "a computer"— in this context meaning an algorithm written by a human, possibly calibrated with a small number of parameters but not trained in any meaningful sense, and a "chess model" which works as you describe.
I think the chess model would be successful at producing the desired outcome but it's not as interesting. There's something to be said for being able to write down in precise terms how to play imperfectly in a manner that feels like a single cohesive intelligence strategizing against you.
