Teachable Snake

2019

    Teachable Snake is a classic snake game controlled by webcam image, powered by Tensorflow.js and Teachable Machine by Google, using pre-trained neural network models.

    role designer, programmer

    stack React.js, Tensorflow.js, Teachable Machine

    project link Teachable Snake

    ITP class Machine Learning for the Web

    source code GitHub

    About the project

    Google Creative Lab came to our class to introduce the latest version of their Teachable Machine, which provides ability to store trained neural network model in the cloud, and using it in other projects outside the Teachable Machine website. Impressed by the project, I decided to use it for my final project in the class.

    The project is inspired by the Webcam Pacman project. The idea here is to create a classic snake game powered by machine learning neural network controller instead of traditional physical ones.

    Code structure

    The framework of this project is Create React App. To manage the complicated state of the snake and the controllers gracefully, React Hooks has been an incredible helpful tool. And to pave a way for future possibilities, including multiple players, the code is structured as modulized as possible.

    store context store context

    Performance issue

    At first, the website was running extremely slow and consuming a lot of CPU power. I thought the issue was coming from memory management of Tensorflow model. But turned out it was because I was calling the functions inside of React hook useEffect API, instead of traditional componentDidMount life cycles, the mobilenet loadModel() function was accidentally being called repeatedly along with other looping functions. After I refactored the structure to set loadModel() function to be called only once after component mounted, the website seems to be working more smoothly now.

    useEffect and loadModel useEffect and loadModel

    Loading model

    Using the trained model provided by Teachable Machine is pretty straight forward. Just load the model and webcam in the first place, and make prediction in every requestAnimationFrame.

    loading model loading model

    Future goals

    To make improvements on this project, other than enhancement of the user flow and user interface, the ambitious goal is to setup a websocket for multiple players to experience the game at the same time!

    demo gif demo snake gif