Pose estimation is a subset of AI problems that focuses on identifying and tracking characteristics of a person such as their gestures, movement, position, and other actions.
Hand Gesture Recognition
The focus of this project was to develop a complex hand gesture recognition system that can interface with a website, games and a robot all using the built-in webcam of a computer. I worked on the deep learning based hand gesture recognition component of the project alongside members from UofT's Machine Intelligence Team who worked on other aspects of the problem. The contributions and entire system are described in the presentation below.
Hand gesture recognition project completed in 2021.
Real Time Pose Estimation For Exercises At The Gym
The goal of this project was to identify the pose of gym-goers in real time using a Raspberry Pi 4, camera module, and AI. A program I wrote uses this pose information and checks if people are doing a series of exercises such as squats, curls, bench presses, etc. correctly and with a consistent speed. If they do the exercises incorrectly or too slowly, then the robot initiates a speaker with a pre-recorded set of "words of encouragement". The pose estimation is achieved using an open source library by Google called Mediapipe while the logic that identifies exercises and the tricks that I used to make this run in real time were my own. For instance, the model complexity of pose detection models can be adjusted and the confidence threshold for detections can be tuned. Additionally, I multi-threaded the program to have one thread contain the detections while the other thread contain the logic to identify exercises and initiate the hardware speakers to encourage the person doing the exercise. This greatly improved real-time performance.
The Mickey Mocker 3000, a robot that roasts people at the gym for not doing their exercises properly.
Pose Estimation For Dance Movements
My team and I won at Hack The North 2021 by creating SmartDance, an app that teaches you how to dance using AI. Smart Dance uses pose estimation to store the key points of dances from popular dance videos and use those poses to guide the user on how to move their body to do the dance. This was so well received that one of the judges, VC Danielle Strachman from 1517 Fund gave us a grant to continue developing the application after the hackathon was over. We took this idea to our university's startup accelerator program called the UofT Hatchery where not only were we selected to be in the accelerator's program but have made it to the very end of the program the entire summer of 2022 and are awaiting further funding decisions. Look out for details and an update on what this idea has evolved into! For now, here's what we made at the hackathon:
SmartDance, a winning idea my team and I made at Hack The North 2021.