Blog
Have you ever wondered if ChatGPT can play chess?
By: Gustav Rånby Senior Data Scientist
I have been playing around with the new custom GPT functionality released to plus users recently, and I decided to create a chess playing GPT - The Chess Challenger. It actually turned out to be a real chess challenge.
If you have ever tried playing chess with ChatGPT, you will probably have noticed that chess pieces mysteriously can disappear from the board or that ChatGPT “cheats” with illegal moves. In generative AI terminology, this is called hallucination, and in order to mitigate this, we need to give ChatGPT some tools for managing the game state and to validate moves.
How does it work?
The main idea is to keep the game state in the chat, and instruct ChatGPT to only update the state using the provided tools. ChatGPT suggested the FEN representation for the game state, and the SAN representation for the chess moves. FEN is the abbreviation of Forsyth-Edwards Notation, and it is the standard notation to describe positions of a chess game, and SAN (Standard Algebraic Notation) is the annotation that you might have seen in a chess book or chess article, for example, Bxc6 (bishop captures on c6). I found these representations reasonable considering that they are commonly used (ChatGPT is trained on them), they are both language-like, and can be said to define a chess language - a perfect match for our language model.
Another aspect is that as a human player, maybe the FEN format could be a bit unintuitive to work with, we tend to prefer a visual representation of the chess board. I figured out a way to generate a chess board using the python plotting library matplotlib, and I created a tool for ChatGPT for generating a visual representation from the game state (the FEN).
A challenging part turned out to be the parsing of the moves. As you may have noticed from the example Bxc6, the SAN annotation often omits the important detail about what exact piece it is referring to, and even if we know that a bishop should capture on c6, we have to figure out based on the current chess board, what bishop we are referring to. A starting sequence in chess could be e4, c5, Nf3, Nc6, d4, and needless to say, there are a few (many) corner cases to address when translating sequences like this into a current game state.
In summary, I provided ChatGPT with tools to validate moves and to update the game state given a move and a game state, together with a system prompt describing the role of the Chess Challenger, and how I want it to use the tools. The result is a chess-playing GPT.
So, what is the answer then?
ChatGPT can play chess to some extent, it absolutely makes some blunders, and thanks to the validation tools, it tends to recover from them. In this case, we are only relying on its inherent ability to play chess, and I have not provided it with any tools for detecting threats, finding valid moves, or for scoring game states. I have been able to complete a couple of games, it tends to have a challenge figuring out that it is checkmated, and I am considering giving it a tool for checking (no pun intended) that.
In conclusion this highlights the capabilities of Open AI's new feature custom GPTs, and also how you can address hallucinations when working with generative AI, and in this case this was managed by giving the language model access to some python functions together with an instruction on how to use them. It would be interesting to repeat the experiment with another representation for the game state, to see how that affects the game play (like an ASCII-art representation of the board). Some work remains regarding the python utilities, for example updating of the castling availability is not implemented yet.
How do we at Scania think about AI?
AI is a great enabler in many areas in order to Increase our precision and increase productivity across Scania.
This also includes areas where Scania already has a strong AI presence; like autonomous driving, automated production units (like Scania´s ultra-modern robotised foundry), computer vision for securing quality, connected intelligence where the data from our connected vehicles are used for optimising transport or for dimensioning charging infrastructure for Scania´s fleet of electric vehicles and so on. At Scania Ai is improving processes in smaller or larger ways, as part of Scania’s relentless work with continuous improvement of our methods and processes.
With the cloud first policy at Scania, it gives the developers access to AI services at providers such as; AWS and/or Microsoft Azure. AI will for sure shape the digital landscape in a rapid fashion, creating new opportunities in ways that are hard to foresee. What I am sure of is that working at Scania will allow me to be part of that journey.
If you want to experience more within this field of AI
If you have a plus subscription, you can play around with this GPT here: https://chat.openai.com/g/g-NulWr5P6g-chess-challenger.
