How to Make Multiplayer Rock, Paper, Scissors with AI

This is a template for making multiplayer games that involve your hands and body using AI or computer vision. You can even submit new games to the repo and I will host them at https://handland.lol

Included Games

The repo currently comes with three complete two-player games:

1. Rock, Paper, Scissors
2. Staring Contest
3. 007 (Standoff or Block, Reload, Shoot, and Shotgun) - How to play

About the Code

The files are very short so you can mess around with them and make new games or just learn how websockets work. It has the most basic matchmaking possible as well as a shareable links.

Whoever has the same link as you will be in your "room".

The tech is very simple, it's a very minimal Express JS server with 3 folders of 3 multiplayer games. The games use WebRTC and native Websockets to connect in real time. AI Vision models were made and provided by Roboflow.

The blinking and Rock, Paper, Scissors models were already made on Roboflow Universe 

How to Make a Game from Scratch

I started off by training one of the models myself by taking a video of myself and annotating ~200 hundred frames for object detection.

This process sounds harder than it is. You upload a video and just drag boxes around you hands and write the game move you want to recognize. It may take trial and error with similar hand motions.

As I realized how similar some of the hand motions were, I took a few versions to correct and add more images that make it more clear when a player is doing one motion over another.

For instance, in "007" you typically point your hands straight up to reload which also looks very similar to the shooting motion which is also pointing your hands. So I tried to include more of the arm in the reload motion so the model doesn't confuse the two easily.

I also included a variety of poses as well as different clothes, rooms, and friends to make it as flexible as possible.

There is an option to just draw boxes roughly around your hands to annotate versus taking more time segment just your hands and fingers in each picture.

I found it is better to segment the hands fingers even if it takes a bit more time at the start. The smart annotating tool gets really good and doing this for you after awhile.

Streaming Video for Online Games

The webcams are streamed peer to peer with WebRTC through stunprotocol and google servers. This is a method to make sure you connect peer to peer even though people may be getting their internet inside a series of complicated networks.

You don't have to understand how it works though, all the functionality in common/common.js:

// Calling start(true) will start streaming your webcam


function start(isCaller) {
    peerConnection = new RTCPeerConnection(peerConnectionConfig);
    peerConnection.onicecandidate = gotIceCandidate;
    peerConnection.ontrack = gotRemoteStream;

    for(const track of localStream.getTracks()) {
        peerConnection.addTrack(track, localStream);
    }

    if(isCaller) {
            peerConnection.createOffer().then(createdDescription).catch(errorHandler);
    }
}

// Your device uuid is sent to the server so it can put you in a room

function gotIceCandidate(event) {
    if(event.candidate != null) {
        serverConnection.send(JSON.stringify({'ice': event.candidate, 'uuid': uuid}));
    }
}

Server Matchmaking

In server/main.js, the server has two "global" dictionaries that keep track of the state of all the games being played.

let clients = {};
let gameStates = {};

Both receive the uuid and "room" from the websocket as ws.uuid and ws.room.

• uuid is the unique user device id as we talked about before
• room is the url the user is at

Every url is a random string like say https://handland.lol/random. If any other two users are at the same url, the will be in the same "room." This keeps things simple and shareable.

The room is also used as the key names for the clients and gameStates dictionaries.

if (clients[ws.room].length >= 2) {
  ws.send("Room is full");
  ws.close();
  return;
}
      
clients[ws.room].push(ws);

if (game === 'rps') {
  gameStates[ws.room] = { ...defaultGameState_rps };

  if (clients[ws.room].length === 2) {

    gameStates[ws.room].id_p1 = clients[ws.room][0].uuid;
    gameStates[ws.room].id_p2 = clients[ws.room][1].uuid;

    roundLoop_rps(gameStates, ws, clients);
  }
}

Think of the websocket (or ws) as a single player. So it will push them into a room and check if the a room has exactly 2 people before allowing a game to start.

Multiplayer Logic

Since the game is played on a beat every few seconds, all the game logic is done in the server.

In Rock, Paper, Scissors, we wouldn't want one player throwing their hand early and repeatedly losing.

It is all handled in games/rps/server.js.

In the function roundLoop_rps, we have a bunch of timers inside of each other that run after 2000 milliseconds pass. It updates information in gameStates[ws.room].

function roundLoop_rps(gameStates, ws, clients) {
    gameStates[ws.room].remoteVideoVisible = false;
    broadcast_game_rps(...);
    setTimeout(() => {
        gameStates[ws.room].gameStateText = "1";
        broadcast_game_rps(...);
        setTimeout(() => {
            gameStates[ws.room].gameStateText = "1, 2";
            broadcast_game_rps(...);
            setTimeout(() => {
                gameStates[ws.room].gameStateText = "1, 2, Go!";
                gameStates[ws.room].is_resolving = true;
                gameStates[ws.room].remoteVideoVisible = true;
                handleRound_rps(gameStates, ws, clients);
                broadcast_game_rps(...);
                if(!gameStates[ws.room].end_game) {
                    setTimeout(() => { 
                        roundLoop_rps(gameStates, ws, clients);
                        gameStates[ws.room].gameStateText = "";
                        gameStates[ws.room].is_resolving = false;
                        broadcast_game_rps(...);
                    }, 2000);
                }
                broadcast_game_rps(...);
            }, 2000);
        }, 2000);
    }, 2000);
}

This will run on an infinite loop until the server decides gameStates[ws.room].end_game is true.

Every time broadcast_game_rps is called, it is telling each player in the room the updated game state.

function broadcast_game_rps(data, room, clients) {
  clients[room].forEach(client => {
      if (client.readyState === WebSocket.OPEN) {
          let gameData = JSON.parse(data);
          // Set the "you" and "other" fields based on the player's UUID.
          if (client.uuid === gameData.gameState.id_p1) {
              ...
          } else {
              ...
          client.send(JSON.stringify(gameData), {binary: false});
      }
  });
}

Both players should not have exactly the same information from the server. There may even be games where you want to purposely hide that!

In Rock, Paper, Scissors, the server will want to tell one player they won and another they lost. It knows which player to send what information to by checking client.uuid and id_p1 or id_p2. You can check where those were set in the Server Matchmaking section above.

Who Wins and Keeping Score

Inside the deepest loop of the multiplayer logic, the function handleRound_rps is called to decide who wins. This is just a bunch of if statements to determine if it is a tie or who won the round.

The server will substract a health point with gameStates[ws.room].health_p1 or gameStates[ws.room].health_p2 depending on who lost.

function handleRound_rps(gameStates, ws, clients) {
  if (gameStates[ws.room].move_p1 == "ROCK") {
      if (gameStates[ws.room].move_p2 == "SCISSORS") {
          gameStates[ws.room].health_p2 = gameStates[ws.room].health_p2 - 1;
      } else if (gameStates[ws.room].move_p2 == "PAPER") {
          gameStates[ws.room].health_p1 = gameStates[ws.room].health_p1 - 1;
      } else if (gameStates[ws.room].move_p2 == "NOTHING") {
          gameStates[ws.room].health_p2 = gameStates[ws.room].health_p2 - 1;
      }
  } else if (gameStates[ws.room].move_p1 == "PAPER") {
      if (gameStates[ws.room].move_p2 == "ROCK") {
          gameStates[ws.room].health_p2 = gameStates[ws.room].health_p2 - 1;
      } else if (gameStates[ws.room].move_p2 == "SCISSORS") {
          gameStates[ws.room].health_p1 = gameStates[ws.room].health_p1 - 1;
      } else if (gameStates[ws.room].move_p2 == "NOTHING") {
          gameStates[ws.room].health_p2 = gameStates[ws.room].health_p2 - 1;
      }
  } else if (gameStates[ws.room].move_p1 == "SCISSORS") {
      if (gameStates[ws.room].move_p2 == "PAPER") {
          gameStates[ws.room].health_p2 = gameStates[ws.room].health_p2 - 1;
      } else if (gameStates[ws.room].move_p2 == "ROCK") {
          gameStates[ws.room].health_p1 = gameStates[ws.room].health_p1 - 1;
      } else if (gameStates[ws.room].move_p2 == "NOTHING") {
          gameStates[ws.room].health_p2 = gameStates[ws.room].health_p2 - 1;
      }
  } else {
      gameStates[ws.room].health_p1 = gameStates[ws.room].health_p1 - 1;
      if (gameStates[ws.room].move_p2 == "NOTHING") {
          gameStates[ws.room].health_p2 = gameStates[ws.room].health_p2 - 1;
      }
  }
  
  
  if (gameStates[ws.room].health_p1 <= 0 || gameStates[ws.room].health_p2 <= 0) {
      gameStates[ws.room].end_game = true;
      if (gameStates[ws.room].health_p1 > 0) {
          gameStates[ws.room].winnerText_p1 = "WINNER!"
          gameStates[ws.room].winnerText_p2 = "DEFEAT!"
      } else if (gameStates[ws.room].health_p2 > 0) {
          gameStates[ws.room].winnerText_p1 = "DEFEAT!"
          gameStates[ws.room].winnerText_p2 = "WINNER!"
      } else {
          gameStates[ws.room].winnerText_p1 = "DRAW!"
          gameStates[ws.room].winnerText_p2 = "DRAW!"
      }
      broadcast_game_rps(...);
  }
}

I set the default health at 3 for both players. When one gets below 0, gameStates[ws.room].end_game is set to True and the game is over.

Conclusion

I encourage you to play the game and start messing with the logic to make your own games.

HandLand can be uploaded as is to popular cloud platforms like Vercel or Heroku.