Work

It’s time to Flexpress Yourself! Create your own virtual wearable art and share it to the world!

In collaboration with Google and dentsu One Bangkok, we developed Flexpress Yourself experience for the promotion of Samsung Galaxy Z Flip4. Flexpress Yourself is an AI-powered web application that allows users to freely express themselves through creating Flexi – a virtual wearable art.

Three simple steps to Flexpress Yourself: Create, Pose, and Share.

1. Create – Users simply move their arms to create Flexi. As you move, the Flexi develops in real-time.
2. Pose – After finishing creating the Flexi, it will be put on your body. Now let’s move your own way.
3. Share – Then share your Flexi and movements to the world!

With the power of AI, we use BlazePose to detect key points of the human body. The web app creates Flexi following the movements of the user’s left and right arms. Whether you raise your hands up or down, forward or backward, every movement continuously generates Flexi. This means each Flexi is uniquely yours! Then the web app augments/maps Flexi to the key points of the user’s body in real-world 3D. So you can enjoy moving and dancing around with Flexi without limit. Share your creations with the world, and let’s embrace individuality & creativity.

Ready to Flexpress Yourself? Go to https://flexpressyourself.com/, create and show yours!

What is Pannana?

Pannana is a mobile app that enables Thai people with visual impairment to enjoy movies and any visual contents by listening to audio description along with the movie.

Audio description (AD) is an additional narration that describes what is happening on the screen when there is no dialogue. It depicts the key visual elements that are necessary to understanding the content on the screen.

You can learn more about Pannana at https://klongdinsor.com/pannana/.

Introducing VOHAN

In order to create audio descriptions, the staff have to do a few works including; finding the time code where they should place the audio descriptions (the part of audio that has no dialogue), interpreting the context of the video and writing one or a few sentences that best describe each scene, recording the narration of AD, post-processing the sound and exporting the audio file.

The whole process is quite complicated and requires a lot of time and human effort. Currently, only a few groups of people are able to create the ADs. As a consequence, the amount of content in the Pannana app is limited and it is hard to catch up with the current trend.

VOHAN aims to apply the concept of crowdsourcing to the AD generation process by letting AI handle the laborious and repetitive tasks. This means the staff can finish the process more quickly, adding more up-to-date content to the Pannana app. In addition, VOHAN makes writing AD scripts an effortless task that anyone can easily come and write the descriptions. As a result, more people can participate and help Pannana grow in order to increase equal access to media, which is the basic right for everyone.

How VOHAN works?

The first step is to identify the slots where we can put the ADs in. To assist the admin, we’ve created an audio segmentation ML model. This model detects and identifies the part of the video that has no dialogue. It will then return the timestamp as an editable card which can be used to fill in the AD script.

The volunteer writer enters the VOHAN web application and selects a video from the list. When the video starts, the writer interprets the scenes and adds the AD script simultaneously. They can come back and edit the text anytime they want.

After AD scripts are submitted, the admin reviews those ADs and chooses the one that is best-suited. Thanks to a text-to-speech model embedded, the admin is able to compile the selected AD scripts into an audio file immediately right within the web app.

Design concept and Technical description

As crowdsourcing is the core concept of VOHAN, we designed the web app to support users who have no or few experience in making the audio description. VOHAN is extremely easy to access via web browser – no app download needed. The user interface (UI) guides the user from the very beginning till the end of the video. So, people who have never known about audio description can try making the audio description at the same time.

The audio segmentation has played an important role in this project. The Audio Segmentation algorithm starts by turning an audio wave into Spectrogram by Short-time Fourier Transform (STFT). Then our Machine learning (ML) model segments the Spectrogram into slots where there is dialogue and no dialogue.

The challenge of this project is to collect the dataset used to train the ML model, which has to be a clean Thai-language speech audio. Another challenge is to design the ML model to be light enough to run on the web browser.

Conclusion

By using VOHAN as a tool, the audio description creation would be an activity that anyone can participate in. As a result, the Pannana application will be able to serve more content to our Thai visually impaired friends and help increase equal access to the media, which is the basic right for everyone.

VOHAN is going live soon! We welcome everyone to be a part of this project. Please stay tuned for more details.

When imagining a beehive, the first thing that pops into your mind might be a group of hexagons joined together. But what if the hive is built of other shapes? And what would the consequences of that be?

bit.studio, in partnership with Advent Children’s Museum, created ‘Beehive Interactive Experience’ to educate children about animal habitats. We have chosen a beehive’s hexagonal cells as an example due to its unique qualities. However, explaining the science behind beehives might be too complicated for children, so we applied a Learning-by-Doing method to simplify the concept.

The Beehive Interactive Experience is a fun experiment-based activity. Children write their name to start the experience. Then, draw a shape they want their bees to build the hive with. They can draw whatever they want – there are no limited choices. Our machine learning model will identify the drawing and morph it into the closest basic shape. The software will then evaluate the efficiency of the shape. The most efficient shape can store the maximum amount of honey, and use the least amount of wax. Children can try drawing other shapes to see different evaluation results and decide which shape would be best for the bees. Once the shape has been submitted, a simulation will be shown on the screen of the bees building their hive based on the chosen shape. At the end of the experience, the children’s beehive will be rated and be provided some feedback on how to improve their design.

In order for the drawing to be recognized and converted into the closest shape, we applied the machine learning algorithms, Convolutional Neural Network (CNN) which analyzes and classifies hand-drawn images.

The Beehive Interactive Experience was displayed as part of the “Surviving the Planet – A Life Science Exhibition” on October 1, 2020 to January 5, 2021 in Nanning, China. Children were overjoyed to see their bees building hives. And we truly hope that this fun experience can inspire them to explore nature.

Collaboration
Academic content support: Advent Children’s Museum
UX & UI design: bit.studio
Software development: bit.studio

Animating characters may not be a task for everyone. Scroobly aims to make it easier by being a fun and easy tool to create animations in real-time with your camera.

bit.studio in collaboration with Google, developed a web app that lets ANYONE turn their doodles to life – no coding or design skills required, just follow these 3 steps:
First, choose a shape and start scribbling. Second, make it move using your body and the doodles will mirror your moves in real-time. Last, record and save. Now your animation is ready to be shared!

Scroobly is designed to be enjoyed anywhere and anytime. Just a device with a camera and you are good to go. No additional app is required since our web app can be accessed via your default browser.

Scroobly is fully powered by TensorFlow.js. We implemented Facemesh and PoseNet Machine Learning models to detect the key points of your face and body. Then the system will translate those key points into motion and map that motion to the character. We optimized the model to make all these processes run simultaneously as you move. So, you can see how characters are animated in real-time.

Scroobly is an AI experiment featured on Experiments with Google, visit the blog to see more about the project: https://experiments.withgoogle.com/scroobly.

If you want to try it yourself, go to https://scroobly.com.
Scrooble a doodle and bring it to life!

Tech: Augmented Reality (AR), AI/ML – Pose Estimation, Web application, Online experience

Burrito? Nachos? Quesadilla? Taco?
Now ordering Mexican food is more special than ever!

In collaboration with Google, we created a fun food ordering experience for DiDi Food. ‘Dibuja tu Antojo’ is a web app that turns lovely food drawing into real food order.

To start the experience, draw a favorite dish and capture the drawing with our web app. It will guess what you have drawn and suggest the nearby restaurants that are able to serve that dish on DiDi Food. Finish your order and let’s enjoy the meal!

Try Dibuja tu Antojo on your mobile at https://dibujacondidifood.com/.