Teachable Snake


    Teachable Snake is a classic snake game controlled by webcam image, powered by Tensorflow.js and Teachable Machine by Google, using pre-trained neural network models.

    role designer, programmer

    stack React.js, Tensorflow.js, Teachable Machine

    project link Teachable Snake

    Google Showcase Experiments with Google

    ITP class Machine Learning for the Web

    source code GitHub

    About the project

    Google Creative Lab came to our class to introduce the latest version of their Teachable Machine, which provides ability to store trained neural network model in the cloud, and using it in other projects outside the Teachable Machine website. Impressed by the project, I decided to use it for my final project in the class.

    Teachable Snake is an interactive web game powered by the beta version of Teachable Machine 2 and React.js, inspired by Webcam Pacman project.

    The idea is to instead of using physical buttons to control the game, everyone can draw a black arrow on a piece of white paper as controller, and move the snake by turning the paper in different directions in front of webcam.

    To recognize the paper controller pointing at different directions, I pre-trained the model with 500 photos each arrow direction (up, down, right and left), and store the model in cloud with the beta version of Teachable Machine 2. With the API endpoint published, I can then apply the model in my React.js app.

    Code structure

    The framework of this project is Create React App. To manage the complicated state of the snake and the controllers gracefully, React Hooks has been an incredible helpful tool. And to pave a way for future possibilities, including multiple players, the code is structured as modulized as possible.

    store context store context

    Performance issue

    At first, the website was running extremely slow and consuming a lot of CPU power. I thought the issue was coming from memory management of Tensorflow model. But turned out it was because I was calling the functions inside of React hook useEffect API, instead of traditional componentDidMount life cycles, the mobilenet loadModel() function was accidentally being called repeatedly along with other looping functions. After I refactored the structure to set loadModel() function to be called only once after component mounted, the website seems to be working more smoothly now.

    useEffect and loadModel useEffect and loadModel

    Loading model

    Using the trained model provided by Teachable Machine is pretty straight forward. Just load the model and webcam in the first place, and make prediction in every requestAnimationFrame.

    loading model loading model

    Future goals

    To make improvements on this project, other than enhancement of the user flow and user interface, the ambitious goal is to setup a websocket for multiple players to experience the game at the same time!

    demo gif demo snake gif