Motion and depth sensors for AR and Mixed Reality

We have been using several types of sensor and depth cameras over the years to enable advance 3d tracking of peoples positions or even full skeletons. More and more this is being integrated into live visual experiences, used to control the latest real time projected AR graphics. We are aiming to support a variety of sensors in the refleXion augmented reality projections toolkit and I thought I could list some of the most interesting systems I came across in my research.

When Microsoft first came out with the Kinect sensor it was a game changer for people wanting access to affordable and high quality motion tracking. I was excited to see this kind of tech rise and find ways to use it myself like we are doing with unity 3d and refleXion Visuals . Many in the field were sad to hear the Kinect was going off the market but I am happy to say now we have a new Kinect-DK coming and it is more designed for developers this time. It is cloud enabled for deep learning and I guess whatever you can dream upI am very excited to try it out hopefully in the next month or so. When I do I will write up a little report and post a example.

New hotness !

https://azure.microsoft.com/en-us/services/kinect-dk/

The ZED camera by Stereo Labs is a special device that I see a lot of potential in. It has two depth cameras for 2K stereoscopic vision and Zdepth detection.

Also built into the camera body is a set of motion and gyro sensors that can detect positional and orientation movements for the on board SLAM (Similtanious Localization And Mapping). ZED can act as the eyes and positional brain for and autonomous navigating robot.  With a range of up to 20 Meters I am very interested in using the ZED camera in our refleXion toolkit for full room depth and motion tracking. There is the regular and the ZED mini that is intended for a VR head mount display upgrade so you can do pass through AR and depth tracking

For developers and researchers looking for a small light weight low er cost solution for machine vision sensor  Tera Bee has some interesting products I could see being very useful in full room AR installations . The Active IR time of flight depth sensors and mini Lidar are meant for high speed depth detection and can be used in scenarios where privacy is protected and vision cameras are not acceptable.

The one system I know I want but can’t afford unless we get a big BIG project is Black Trax from Cast software . Black Trax is a ultra high end solution for detailed positional tracking. It uses a proprietary beacon system and TOF IR cameras to locate up to 80 different objects precise  movement and orientation. with this you can do things like have auto tracking spotlights on performers and projection mapping on moving objects.  Black Trax is a full system of hardware components and PC server to calibrate and stream the data. The positional data can be sent to any number of inputs like a media server or game engine.

Best regards to anyone who I met this week at FITC , Thank you Shawn and the organizers it was a lot of fun and there were really good presentations and demos throughout . Shout out to Beeple for 12 years of every days . I don’t know how you do it . Good guy, gave me a few moments to talk shop. More about that later.