Teachable Snake is a classic snake game controlled by webcam image, powered by Tensorflow.js and Teachable Machine by Google, using pre-trained neural network models.
Google Creative Lab came to our class to introduce the latest version of their Teachable Machine, which provides ability to store trained neural network model in the cloud, and using it in other projects outside the Teachable Machine website. Impressed by the project, I decided to use it for my final project in the class.
Teachable Snake is an interactive web game powered by the beta version of Teachable Machine 2 and React.js, inspired by Webcam Pacman project.
The idea is to instead of using physical buttons to control the game, everyone can draw a black arrow on a piece of white paper as controller, and move the snake by turning the paper in different directions in front of webcam.
To recognize the paper controller pointing at different directions, I pre-trained the model with 500 photos each arrow direction (up, down, right and left), and store the model in cloud with the beta version of Teachable Machine 2. With the API endpoint published, I can then apply the model in my React.js app.
The framework of this project is Create React App. To manage the complicated state of the snake and the controllers gracefully, React Hooks has been an incredible helpful tool. And to pave a way for future possibilities, including multiple players, the code is structured as modulized as possible.
At first, the website was running extremely slow and consuming a lot of CPU power. I thought the issue was coming from memory management of
Tensorflow model. But turned out it was because I was calling the functions inside of React hook
useEffect API, instead of traditional
componentDidMount life cycles, the mobilenet
loadModel() function was accidentally being called repeatedly along with other looping functions. After I refactored the structure to set
loadModel() function to be called only once after component mounted, the website seems to be working more smoothly now.
Using the trained model provided by Teachable Machine is pretty straight forward. Just load the model and webcam in the first place, and make prediction in every
To make improvements on this project, other than enhancement of the user flow and user interface, the ambitious goal is to setup a
websocket for multiple players to experience the game at the same time!