Ideal Immersion

What will immersive technology look like in the future?

Before considering the options, let me define immersive in terms of technical presentation rather than subjective perception. This caveat bypasses the issue of user commitment to immersion by assuming that if the technical parameters are adequate, an immersive experience is assured.

With this in mind, the ideal, if unattainable, immersive technology would present an experience identical to physical presence.  We have two fictional versions of this, the Holodeck and the Matrix. 

What are the audio-visual technical parameters for these two imaginary immersive technologies that practical technologies can strive to approximate?

1. Images and sound are completely surrounding. There is no sense of looking through a window or at a screen.
2. The resolution limit is set by the user's own visual acuity.
3. The viewing position is user defined and continuous. A shift or tilt of the head presents a different point of view and an accompanying change in depth perception.
4. Audio visual directionality is continuously correlated with where the user is facing and where the user moves. A head turn or body repositioning reorients the sensory input accordingly.

The current technology that comes closest to these four ideal indices is Virtual Reality (VR) viewed through a Head Mounted Display (HMD).  Wearing an HMD device introduces tactile and kinesthetic sensory distractions, and the field of view of any optical design will restrict peripheral vision. The content must be provided as a volumetric database visualized by the device with no lag time. Nevertheless, VR HMD is a very active commercial research topic, driven by the large market for first person Point-of-View (POV) computer games.

The next closest technology exists only as a dream, but is theoretically attainable: a cyclorama consisting of an exceedingly fine micro lens array. To picture how this would work, imagine standing in a space surrounded by door screening or window screening material, the kind of mesh that keeps out bugs, configured into a dome or a cylindrical wall. Now imagine each one of the spaces in the mesh is actually a tiny lens that projects only a part of its particular image according to the position of the viewer. This would simulate the effect of light from objects beyond the mesh screen passing through the spaces in the mesh. Taken together, this micro lens array could create a light field, which would appear much like a hologram. 

The computational complexity for this approach would be daunting, in effect, requiring the production a tiny but high resolution image corresponding to a hemispheric view for each micro lens in the array. There is no question this would work when the component technologies catch up to the dream, but it will always be inefficient, producing at any moment far more imagery than the eyes are perceiving. 

It does, however, pose the potential for live action capture, with the "camera" being a micro lens array with the same configuration as the viewing array. Incident light would be captured at the recording side of the camera array surface, and its path regenerated on the projection side of the matching viewing array surface. This is synthetic holography, using sensors and digital encoding to simulate what coherent light does with wave phase interference and reconstruction. 

Apple has a patent for a clever solution that would reduce the computational requirement for a Massive Micro Lens Array (MMLA). In Apple's design, the lenses are convex micro mirrors that redirect light from a projection laser according to where the viewer's eyes are located, as tracked remotely through face recognition. This is somewhat like Digital Light Projection (DLP) except that the micro mirrors are stationary and the laser movements make all the positioning adjustments. Again, an anticipatory idea that current optics, mechanics and sensors can implement only crudely.

After VR HMD devices and the highly theoretical Massive Micro Lens Array technology, the contenders for the immersive technology of the future all utilize matched image pairs for Stereo 3D (S3D).  I call them "nearly ideal" because they lock the viewer's eyes into the POV choices made during content creation, affording the audience less of the exploratory interaction that contributes to an immersive experience. In other words, a viewer can't look around things, can't move relative to things, but rather, the viewer must passively accept camera and editing choices.

This constraint is not so much a loss as it is a trade off. In a later post I'll cover Nearly Ideal Immersion and it's advantages for storytelling.