A Gym in the Metaverse - POC

Posted by Encora on Apr 4, 2023 2:41:12 PM
The Data Hub solution automated resource-intensive and complex tasks. This automation enabled the client to focus on exploring and analyzing their data for improved commercial strategy.

Topics: Digital Experience, Metaverse

Industry Context

Finding means to have a healthy way of life and exercising regularly is a constant issue for a lot of people. Many of us felt the difficulties that the COVID pandemic imposed on our lives, as the restrictions caused gyms to close and interrupted group sports activities, forcing us to look for alternatives. Some research also points out that group exercises and exercise communities can help boost people’s participation by providing regular schedules and motivation.



For the PoC we chose the case of workforce exercise referred to as “labor gymnastics” programs frequently offered at companies. The basic idea is to provide an app that both trainers and participants can use to participate in an exercise session. The app would allow the participants to move around and visualize the trainer's movements from different angles. The instructor's movements will be captured from the instructor's mobile phone camera and projected to the students’ cellphones as an avatar. Using cameras to track movements is not the best method, but for this proof of concept, the goal was to explore different technologies.

Gym in the Metaverse Idea


To achieve that, there were a series of challenges that needed to be addressed. The following list provides a description of each of those challenges, the solution we used and some alternatives.

  • 3D environment: We wanted to create a simple Gym environment where both instructor and participants would be located, during the exercise class. For this job we chose Unity.
  • Tracking and rendering instructor's movements: We needed to be able to render the trainer’s avatar inside the Gym and change the body position accordingly to the movements captured by the trainer’s cellphone camera. To help us with that we used MediaPipe an open-source cross-platform customizable Machine Learning solution. To integrate MediaPipe inside Unity we used an open-source plugin. Although the plugin facilitates the job, we encountered some issues regarding correctly tracking the instructor's movements.
  • Handling session between instructor and participants: We started evaluating the possibility to use WebRTC to transmit the 3D instructor and virtual room to the participants’ application. Unity has an open-source library called Unity Render Streaming that allows that, including the possibility for the clients to interact with the rendering scene. 
  • Spatial Voice Chat: Spatial Voice chat not only allows users to chat with each other within the application or game but also takes into consideration the user’s location, changing sound direction and volume accordingly. We wanted to incorporate this functionality in the application as it is an important factor for the user’s immersive experience inside our environment. 
Gym Challenge


In general, the technologies used proved to be ready to deliver a good and immersive experience. Without too much effort we were able to create a simple Gym application. Unity and its assets (Mirror and Agora) abstract a lot of the complexity behind the features, making the development process easier, allowing more configuration and less coding. For example, to include a video on a TV inside the Gym we just had to drag the video file inside the TV model.

MediaPipe, even with the difficulties reported above, proved to be fairly accurate when it came to identifying the instructor's movements, and it tends to get better as BlazePose GHUM Holistic, a new algorithm focused on fitness tracking, is being released. On our tests, MediaPipe was able to properly recognize a variety of positions as demonstrated on the video below. It is important to note that we did a basic mapping of the MediaPipe information to the avatar model, and we also did not implement finger tracking, so some inconsistencies in the movements were expected.

Gym in the Metaverse - Result