LipSync experience was first built as a showcase of TensorFlow.js for Google in 2019, and later optimized for desktop browsers as LipSync with YouTube.
Since this campaign aimed to capture more users globally, we designed the experience to be played on mobile, via web browsers (no need to download any app). This made the experience easier for target users to access than ever, and thus, helped the campaign to capture a wider range of users.
Here are some major changes we made to achieve the best performance on mobile. First, we optimized the ML pipeline. Since our experience was designed for a single user, we removed the multi-face detection which is unnecessary. We also moved the calculation process into the ML model and upgraded the TensorFlow.js model to the up-to-date version to make the whole experience faster. Other changes are on the front end. We removed some of the effects that consume the device resources, such as the Face polygon effect, and updated the API used for generating GIF files for the users. We also split our code into small modules for even faster development.
The campaign was launched on June 10, 2021. Miley’s fans from around the globe joined our challenge with so much fun and got 4.5 out of 5 on their first try. Try out yours at https://lipsync.magnumicecream.com/ and see if you can get 5 out of 5!
