This article was contributed to the Roboflow blog by the team at A1H1.

Introduction

The project of making an automated chess game recorder started after finding out that a chess set with sensors costs ~$500 USD! This price tag is not very accessible for everyday chess players or local chess clubs, and this is what motivated us to start working on Clio 𝛼 – a smart chess recorder that can record and analyze your chess games.

In this post, we will share snapshots of the journey in making this chess game recording system, a cheaper and more portable alternative to chess sets with sensors for recording and analyzing chess games. You’ll learn how we address some of the challenges of digitizing real-life chess games.

Challenges with digitizing chess games with computer vision

When it comes to digitizing chess games with computer vision, the problem can be divided into:

1. chessboard detection - detecting the boundaries of the chessboard and transforming the perspective view of the chessboard from the image to a 2D orthogonal view
2. chess piece recognition - recognizing a chess piece and its position on the board

The variation in chessboard and chess piece appearances with unique lighting conditions in real-life settings are some of the core challenges for computer vision techniques.

If we want to make a portable and small system, then we run into issues from obscured corners of the chessboard and pieces due to the placement of the camera at a lower height. The lower the camera system, the more serious these issues become. The following picture showcases some of these problems.

Many commercial solutions have moved away from computer vision for these reasons and lean toward hardware-based solutions like sensors. Unfortunately, using sensors is both costly and limited to a specific chessboard.

Our objective is to overcome these limitations by employing computer vision and machine learning to create a more universal and accessible chess game recording experience that can be applied to most standard chess sets available in the market.

Portability, cost, and aesthetics are the key requirements we wanted for our system and decided to make a device that meets these requirements. Our approach may not be the simplest, but we have found it to be successful in real-life situations.

A Custom Chessboard Recording Device

After numerous brainstorming and design cycles, we ended up with Clio lite, which is a portable camera designed specifically for recording board games. Clio lite has a fixed camera angle and field of view (FOV) which simplifies computer vision problems and allows us to make certain assumptions when developing the algorithms.

The fully extended Clio lite is only 55(W) x 50(L) x 210(H)mm so smaller than most tripods. In terms of electronics, it has a microcontroller and a wide-angle camera which helps keep costs low.

Chessboard Detection Using Computer Vision

There are many approaches for chessboard detection in literature and open source projects, and we have decided to utilize the idea of x-corners (the corner between the black and white squares) in this repo for our detection algorithm. The steps of our chessboard detection algorithm are summarized in the diagram below.

Pre-wrap input image based on calibration

Since our camera angle is fixed, we can simplify the subsequent steps of the algorithm by pre-determining the homography matrix and using it to rectify input images such that the chessboard is square and appears to be parallel to the image plane.

For this calibration, we wrote a tool with OpenCV to select 4 points (the red dots in the picture below) on the raw input image, which is then passed to the OpenCV findHomography() function to wrap those 4 selected points to the vertices of an ideal square. The output homography matrix is then saved to a calibration file to be used later.

Estimating the rotation and translation of the image

After rectifying the input images with the predetermined homography matrix from the previous step, the rotation and translation of the image are still unknown. We estimate the transformation of the image by first learning the locations of the x-corners and squares on the image. Using Roboflow Annotate, we created a dataset and trained it with Ultralytics YOLOv5.

When we perform inference on the trained model, we will get point clouds for the x-corners and the squares on the chessboard. The idea is to estimate the transformation between these inferred point clouds and the ideal 7x7 x-corner and 8x8 square grids.

Finding the rotation: We first assume the camera is placed approximately near the centerline of the chessboard such that the image is rotated no more than 45 degrees. Then, we compute the following:

1. Randomly pair two points and compute the angle, ɑ between the line and the origin
2. Using angle_array = [-90, 90, 180, -180], find argmin(abs(angle_array - ɑ))
3. If the two points form a near-vertical or near-horizontal line, the angle difference should be less than 45 degrees. If they form a diagonal line, we skip this iteration
4. Repeat this N time and the average of the selected angle difference is the estimated rotation angle

Finding the translation: After estimating the rotation and applying it to the point clouds, estimating translation is relatively straightforward since the point clouds are very much grid-like. We can compute the centroids of point clouds and find the difference between them, which is equivalent to their approximate translation.

Pair the Transformed x-corners with an Ideal x-corner Grid

We do not expect the x-corner detection rate from YOLO to be 100%, which is why we need to determine which x-corners have been detected. Once the x-corner point cloud has been transformed close to the pre-defined ideal x-corner grid (with known pixel locations), we can start pairing them and create an occupancy grid for the input x corners.

Determine the Final Homograph Matrix

The final step is to “fine-tune” the transformation of the point clouds by mapping the x-corners to the ideal grid and computing the homograph matrix again.

Here, we want to find the 4 points that make up the largest possible box to reduce sensitivity to pixel error. As shown in the diagram below, we can create a box and loop through the point cloud until all 4 vertices of the boxes have a corresponding x-corner point. Once we have identified the x-corner/ideal grid pairs, we can compute the final homograph matrix.

Finally, to process a new raw input image, we first obtain the x-corners and square point clouds from the YOLO inference. Then, we apply the pre-defined homography matrix -> apply transformation -> apply the final homograph matrix to the point clouds so that we have a chessboard grid with known pixel locations.

Chess Board Position Recognition

In order to localize the chess pieces, a typical approach is to use machine learning (like YOLO models) to recognise and classify the individual pieces. This works great with a bird-eye camera view angle where the pieces’ obscurity is less severe. Our camera system views the chessboard at a lower height, and can drastically decrease the detection accuracy with this approach. We decided to use a square occupancy-based method to solve this problem.

Once again, we used Roboflow to annotate the chessboard squares and train the occupancy model using YOLOv5. Then, we are able to infer the positions of the chess pieces using our square occupancy-based method.

For better useability, we have developed a mobile app to use with our Clio lite device. It allows users to stream the digitized chess games live or replay recorded games. See the clip below for a simple live stream and game replay demo.

Where do we go from here?

We are super excited about bringing Clio lite to life and can’t wait to see it being used in the community! Improving the robustness of our software with future versions of our algorithm is one of our top priorities. We have plans to extend our piece and board detection capabilities to encompass other popular chess-like games, including Chinese Chess and Go, employing similar methodologies.

If you're interested in learning more about our project or even joining us, we invite you to visit our website at https://a1h1.com.au/, contact us via email info@a1h1.com.au  or check out the Clio lite demos on our YouTube channel.