Tetra AR Projector
Machine Learning (YOLO) | AR | Unity Development | Experience Design
As part of a team, we custom-designed an Augmented Reality (AR) projector as well as a framework for non-programmers that allow them to build AR experiences. As a proof of concept, we built an educational interactive magazine and projection mapping using our framework.
Tetra is a user-friendly tool for building AR experiences. It features intuitive interaction using gesture recognition and the AR Creator ARc. ARc is an interface that allows you to develop your own AR and projection mapping software with visual scripting and object identification.
Interactions with students in the design department revealed a strong resistance from my colleagues to adopt new technologies to complement their projects. One of the main barriers to incorporate technology in design is the need for coding skills. Through research, we found that projection mapping as well as creating custom AR applications is expensive, difficult and non-standardised. A tool with modular components that would allow an interactive and standardised way of developing applications is lacking, and thus would be a new addition to the local and global markets.
CAMERA: INTEL REALSENSE D415
PROJECTOR: WOWOTO H9
COMPUTER: ANY LINUUX COMPUTER
The object identification system is using ARToolkit SDK with Unity to extract the image features allowing for accurate object detection
The gesture system is responsible for the detection of the user’s hand gestures and interaction with the camera. The system relies on the state-of-the-art machine learning algorithm YOLO (You Only Look Once) and computer vision components for efficient recognition of multiple hand gestures.
AR PROJECTIONS CONTROL:
Projections can be controlled by simple hand gestures captured by the camera. Here is an example of an application: By swiping [left or right] the user can rotate the 3D projected model 360 degrees.
By connecting simple blocks, users can create a complete AR application. As an example, this one displays a 3D model
Allows users to create dynamic Augmented Reality objects using Unity such as images, 3D models, text, videos, etc... The system interacts with the gesture recognition module and allows the user to create or edit a certain AR object in a scene using specific gestures. The AR creator also interacts with the object identification system to augment some of recognized objects.
Users can create projection mapping masks and edit these masks via anchor points or nodes.