Microsoft Corporation Shows How It Uses HoloLens To Record Streamable Live Action Video

Microsoft Corporation (NASDAQ:MSFT) is fairly close to creating holograms in much the same way that futuristic shows and movies portray the technology, the company revealed in a new research paper. Microsoft calls the holograms “high-quality free viewpoint video” and believes that it can be compressed to work in tandem with suitable-for-consumer applications. The consumer applications part reveals that the finalized holographic video output will not be limited to studios, and that Microsoft aims to make the technology available to the public at large.



Microsoft also released a YouTube video yesterday illustrating how the company captures holographic video for its virtual reality Hololens brand, illustrating an example of a traditional Maori Haka performance. Microsoft uses 106 RGB and infrared cameras, focused on a calibrated green screen stage, subtracted from the final video, which is how Microsoft computes silhouettes for the final hologram. Microsoft creates 3D point-cloud stereo depth maps by merging the points from RGB cameras and the Infrared cameras, that are refined locally.

The cloud is then refined globally using a multi-view stereo algorithm to define 2.7 million points per frame. Microsoft’s software then creates a tight mesh around the points for each frame; in the YouTube video, there are two individuals in the two meshes, so there are two watertight meshes representing each person. Microsoft cleans them up by applying topological de-noising and island removal, resulting in over one million triangles per frame.

Microsoft’s software even figures out which areas contain perceptually important details such as hands or faces and preserves the quality and geometry of these textures when reducing the mesh to around 20,000 triangles per frame (depending on the target density). All in all, this is not something that cannot be found in traditional motion capture methods, but Microsoft deviates from that by capturing the motions and simultaneously creating a model of the video, rather than creating the model later.


Microsoft’s HoloLens will be considerably boosted by this technology when Microsoft finally implements it. Making these videos streamable over the Internet, as the company plans to do, will market the videos to the entire world and with precise results such as those seen in the video, it will be interesting to see how the world takes it. The software obviously gives more attention to key areas such as faces, which are more precise than the body.

Microsoft can encourage a number of applications for its technology. One has to wonder if this will finally provide a way for people to communicate with lifelike holographic real-time representations of the recipients when talking to them through a smartwatch? Will the holographs eventually allow for interaction? The applications for the technology are endless, and it will be interesting to see if Microsoft’s HoloLens enables holographic technology to reach its potential.