Ideal Immersion

What will immersive technology look like in the future?

Before considering the options, let me define immersive in terms of technical presentation rather than subjective perception. This caveat bypasses the issue of user commitment to immersion by assuming that if the technical parameters are adequate, an immersive experience is assured.

With this in mind, the ideal, if unattainable, immersive technology would present an experience identical to physical presence.  We have two fictional versions of this, the Holodeck and the Matrix. 

What are the audio-visual technical parameters for these two imaginary immersive technologies that practical technologies can strive to approximate?

1. Images and sound are completely surrounding. There is no sense of looking through a window or at a screen.
2. The resolution limit is set by the user's own visual acuity.
3. The viewing position is user defined and continuous. A shift or tilt of the head presents a different point of view and an accompanying change in depth perception.
4. Audio visual directionality is continuously correlated with where the user is facing and where the user moves. A head turn or body repositioning reorients the sensory input accordingly.

The current technology that comes closest to these four ideal indices is Virtual Reality (VR) viewed through a Head Mounted Display (HMD).  Wearing an HMD device introduces tactile and kinesthetic sensory distractions, and the field of view of any optical design will restrict peripheral vision. The content must be provided as a volumetric database visualized by the device with no lag time. Nevertheless, VR HMD is a very active commercial research topic, driven by the large market for first person Point-of-View (POV) computer games.

The next closest technology exists only as a dream, but is theoretically attainable: a cyclorama consisting of an exceedingly fine micro lens array. To picture how this would work, imagine standing in a space surrounded by door screening or window screening material, the kind of mesh that keeps out bugs, configured into a dome or a cylindrical wall. Now imagine each one of the spaces in the mesh is actually a tiny lens that projects only a part of its particular image according to the position of the viewer. This would simulate the effect of light from objects beyond the mesh screen passing through the spaces in the mesh. Taken together, this micro lens array could create a light field, which would appear much like a hologram. 

The computational complexity for this approach would be daunting, in effect, requiring the production a tiny but high resolution image corresponding to a hemispheric view for each micro lens in the array. There is no question this would work when the component technologies catch up to the dream, but it will always be inefficient, producing at any moment far more imagery than the eyes are perceiving. 

It does, however, pose the potential for live action capture, with the "camera" being a micro lens array with the same configuration as the viewing array. Incident light would be captured at the recording side of the camera array surface, and its path regenerated on the projection side of the matching viewing array surface. This is synthetic holography, using sensors and digital encoding to simulate what coherent light does with wave phase interference and reconstruction. 

Apple has a patent for a clever solution that would reduce the computational requirement for a Massive Micro Lens Array (MMLA). In Apple's design, the lenses are convex micro mirrors that redirect light from a projection laser according to where the viewer's eyes are located, as tracked remotely through face recognition. This is somewhat like Digital Light Projection (DLP) except that the micro mirrors are stationary and the laser movements make all the positioning adjustments. Again, an anticipatory idea that current optics, mechanics and sensors can implement only crudely.

After VR HMD devices and the highly theoretical Massive Micro Lens Array technology, the contenders for the immersive technology of the future all utilize matched image pairs for Stereo 3D (S3D).  I call them "nearly ideal" because they lock the viewer's eyes into the POV choices made during content creation, affording the audience less of the exploratory interaction that contributes to an immersive experience. In other words, a viewer can't look around things, can't move relative to things, but rather, the viewer must passively accept camera and editing choices.

This constraint is not so much a loss as it is a trade off. In a later post I'll cover Nearly Ideal Immersion and it's advantages for storytelling.

Easter Egg Ironies

In some movies there's the ending everybody wants to believe, and there's the ending the movie makers insert like an inconspicuous Easter egg, in the tradition of computer games. I'll bet you missed the concealed ironic twists below.

The Prestige: As the Christian Bale character walks away from the scene after murdering the Hugh Jackman character, another Hugh Jackman character in the shadows is shown briefly, hiding in one of the water chambers. He had duplicated himself in expectation of being killed.

Terminator 2: The "good" terminator battles the liquid metal terminator, who immobilizes the good terminator by trapping his arm in machinery. When Sarah Connors is about to be killed by the liquid metal terminator, the good terminator appears, a stump where his arm used to be and a grenade launcher in his other arm. After he shoots the liquid metal terminator and it falls into the vat of molten metal, destroying it, the good terminator takes the cyborg hand the previous terminator in movie #1 had left behind, and lowers himself with the hand into the molten metal. However, his own arm, the one he had just torn off, is not with him. Thus the second movie ends just as the first one, with terminator technology left behind to reverse engineer and bring about the rise of the machines. Later movies in the series were not directed by James Cameron, and did not capitalize on this concealed irony.

12 Monkeys: At the conclusion, when the mad scientist finds his seat on the plane, one of the scientists from the future is already in the adjacent seat. The mad scientist asks her what her business is, and she says, "Insurance." It is made clear at that point that the Bruce Willis character, at that very moment shot dead by airport security, has been set up from the beginning to instigate the gun scene in the airport as a diversion, thus facilitating the mad scientist's dash to the plane when security was distracted. The scientists from the future were presumed to be averting the global pandemic that created their timeline, caused by the engineered virus sown by the mad scientist in the journey around the world upon which he was embarking. The ending made clear, however, that the scientists from the future were "insuring" that the mad scientist carried out his mission, thus setting into motion the events that resulted in their world.

I have never come across any references to these three covert endings. The movie critics certainly missed them entirely. A movie's structure and momentum carries it to the conclusion people prefer, and the audience forcibly fails to notice anything contradictory.

Rod Howard's commentary on the Beautiful Mind disc remarks repeatedly how test audiences resisted any reveal that the Russell Crowe character, based on John Nash, was suffering schizophrenic delusions. There were several reveals in succession, but many in the audience still held out for the possibility that he really wasn't nuts and that the government really had masterminded a coverup. The final reveal was beyond anyone's power of denial, when the character says to his wife that she can't see his friend because he is wearing an invisibility cloak. (The movie was released in 2001, the same year as the first Harry Potter book, which introduces his invisibility cloak.)

Many people with me in the audience for a showing of Shutter Island exited the theater looking dazed. In a very rare display of public post-movie discourse, they gathered in little groups trying to figure out what happened at the end. The ending isn't confusing at all as story logic, but the audience could not accept what they had witnessed and they had no alternative. Leonardo DiCaprio's character clearly decided that without his mental illness to protect him from the truth, he would rather have a lobotomy than suffer horrible memories. The audience when I saw the movie could not accept this, and thus chose to be confused rather than suffer a horrible truth. I thought Scorsese very clever to create for the audience a fate parallel to the fate of the main character.

Stage magic uses diversion to fool the eye, but movies can use clear logic and self evident visuals to inform the eye, and the audience will still insist on fooling itself. The ultimate irony is that audiences see what they want to see, not necessarily what the storyteller presents to them.

Movies as Lucid Dreams

The critical, business and celebrity buzz about movies obscures their deep psychological influence over the way we think after seeing them.

For all of my own analytical detachment, I can be extremely influenced for minutes, hours or even days afterward. This is not so much an emotional effect as a cognitive one. The experience of the movie creates a temporary lens through which I interpret the world, a lot like the minutes after waking up from a lucid dream.

The fuzzy pseudo-logic of lucid dreams can pervade one's waking moments all the way to morning coffee. The after effect of certain movies, not all of them, lingers similarly.

I just watched on Blu-ray "A Beautiful Mind" for the first time. When movies are viewed into the wee hours in a dark room when one is sleepy, they can become very much like dreams, evoking similar reactions. I found myself wondering at the end how much of my own life is purely imagined, and even reread my email and blog posts as a reality check. No, I hadn't created imaginary characters or events, but for some minutes the prospect of this worried me.

Movies gain their power, and suffer their limitations, from their similarity to the dream state. They can influence us greatly but the effect fades. If that fading happens rapidly, the most appropriate response is to laugh at oneself.

For some people and some movies, the effect lingers as a personal, recurrent reference point. Do they risk, like Inception's Cobb, becoming stuck in the dream, a moment that lasts forever, a sleep from which they never awake?

Climate Change: The Movie

A fictional movie about climate change greatly exaggerated the science, but the ploy didn't work, society wasn't scared by special effects into doing the right thing.

A documentary movie about climate change used reason and data charts, to much acclaim but for little result. The truth of climate change is still inconvenient.

What a movie about climate change has not tried is reverse psychology, which just might do the job, not on believers but on the deniers.

Here's the formula: 1) Choose which audience you want to scare; 2) Appeal to the fear they hold most dear.

The people who understand that climate change is real—that the disappearance of summer arctic ice, the acidification of the oceans and the release of methane from melting permafrost are all world class catastrophes—these rational, informed people are not the audience a fictional movie should address. They are plenty scared already, for good reason.

By contrast, the smug confidence of climate deniers will never be shaken by the evidence and logic of a careful documentary. Nor will a fictional movie ever make them feel afraid of an actual threat.

No, conservative climate deniers would much rather be afraid of things that don't exist, and they reveal these considerable fears in their attacks on climate science. That vulnerability, the conservative bubble of delusionary paranoia, should be the target of a climate change movie.

Deniers say they are afraid that the scientific community is joined in a conspiracy so vast that all the mounting evidence from field expeditions, satellite monitoring and direct measurements are completely confabulated, a lie breathtaking in scope being perpetrated by the smartest people in society. There's plot point number 1.

Deniers say they are afraid that liberals wish to use climate change as an excuse to expand government power into every aspect of our lives, controlling our behavior and restricting our marketplace freedoms. There's plot point number 2.

So, here's the story premise that exploits these fears:

Conservative opposition successfully forstalls effective action on climate change when a seemingly liberal senator flips his position on the issue and joins them, mouthing the rhetoric of plot points 1 and 2.

Soon, major disaster strikes. Frozen methane deposits in the deep ocean suddenly belch immense quantities of the gas into the atmosphere. The Ross Ice Shelf breaks up, thus making way for rapid glacier movement from the Antarctic highlands into the ocean, raising sea levels dramatically. Ocean acidification wipes out entire food chains, devastating global fisheries. Major ocean current circulation systems simply shut down, radically altering local climates.

The once liberal senator now running for president suddenly flips again. He says these catastrophes can all be blamed on the conservative climate deniers, and that the nation needs to punish them for betraying the national interest.

As conservatives see their most treasured fear coming true, they realize they've been tricked into serving as scapegoats. The liberal senator was setting them up to take the fall. Climate change was never in doubt and the denial enablers would most certainly be blamed for making it worse.

The conservatives, from religious right to neocon, realize too late that they were on the wrong side of an issue that should have been theirs. They could have used climate change as an excuse for asserting global power aggressively to suit American business interests. They could have justified in the name of saving God's Creation, the conquest of all other nations.

They witness a liberal president living out the conservative wet dream, preaching and implementing the gospel of green imperialism, and realize their own traditional strong hand has been taken away from them, even as they face trials for treason and calls for the death penalty.

Yes, a terribly paranoid scenario, in which climate change plays a secondary part. What conservatives fear most is their oppression by liberal policies, and their foolish opposition to dealing with climate change dooms them to that very fate.

If a movie about climate change wasn't really about climate change, it was about the conservative fear of liberal power reinforced by climate change consequences, then conservatives might start thinking about doing the right thing, albeit for the wrong reason.

Reverse psychology doesn't try to make people afraid, it leverages the fear that already exists by showing how people can bring their own worst nightmares upon themselves.

Claws, Jaws, & Maws: Cinema Monsters

Specific attributes seem obligatory in the design of modern cinema monsters, particularly, the accoutrements signifying a desire to devour human flesh.

In this modern era, when large predators like tigers, grizzly bears and even sharks are being driven to extinction, our apprehensions remain insistently primal. Whether the manifestation comes from outer space, the abyssal depths, or the lab of a mad scientist, the fate we prefer to fear is consistent: we are afraid of being eaten alive.

As with T. Rex, the fiercest will hunt us down in our nightmares long after they have vanished from Earth.

The danger of becoming a meal is no longer instilled in us by the fables we are told as children. The persistence of this visceral fear despite the absence of everyday cultural reinforcement suggests it is hard wired into our being, a genetic trait that once served us well because it kept us alert.

Other movie monsters exploit a considerable fear we each possess: the fear of our own kind. Vampires and zombies and faceless serial killers are but stand-ins for the mindless masses and malevolent loners living in our midst, who could turn on us without provocation. The primal fear in this case stems from a human predisposition toward paranoia. Instinctive xenophobia is particularly rampant in an anonymous society where anybody, even a family member, can become a malevolent stranger who was only pretending to be a friend.

Of course, we have tamed vampires and zombies with soap opera characterization, because we are not truly afraid of the mindless masses and malevolent loners, we just resent having to accommodate them during commute hour.

Despite the pacification of the planet, there are yet things roving the real world that wrap around our spines with an icy grip. The one true mortal threat that we fear most is so scary that it has never, to my knowledge, been portrayed as a movie monster. This is the horror of our own cells commandeered to kill us, the inscrutable threat of Cancer.

I thought about what Cancer might look like as a movie villain. The claw and fang cliches are boring. I am not professionally skilled at creature creation with 3D modeling programs, so I offer a sketch combining description with a conceptual image.

Imagine the Cancer Sisters, Maligna and Metasta. Maligna is a faceless, amorphous medusa head extending long tendrils outward in all directions. Metasta is a volumetric Mandelbrot beetle swarm hovering to Maligna's side. They are both glossy black, reflecting everything around them, showing no color of their own save a malevolent aura. When they move in for the kill, the victim is unaware, but soon the victim suffers the full range of cancer symptoms either compressed into the span of moments, or drawn out over months, at the Sisters' cruel discretion.

If you don't get a chill imagining this, if you don't fear superstitiously that even thinking about such things runs the risk of making them true, then you are made of stronger stuff than most of the human race.

Maligna and Metasta are not likely to appear at the multiplex. They are so scary no one would want to face them as phantoms on the screen. It is frightful enough that we face them when loved ones die, or the doctor brings us bad news.

We reserve our tame cinema fears for beasts with the predator characteristics we are vanquishing from the planet, as a kind of justification for our ignorant, thoughtless, relentless, selfish behavior: extermination as self defense.

The truly frightening threats in real life, all of our own creation now, are too scary for entertainment.

The Surface of the Thing is the Thing?

Nearly all that we call a thing, has a surface which bounds its existence. The surface is what we see, the surface is what we touch and hold. The surface contains the essence of the thing. Beyond the surface a thing can have a certain range of action impinging on other things with defining surfaces of their own.

If there is no surface, as with an evanescent cloud or transient musical sounds, we call these not things, but phenomena.

So what are we to make of this computer graphic rendering I created six years ago?

There are three things shown at different scales, the thing in the lower right being a part of the thing in the upper left, and the thing in the middle being a part of the thing in the lower right.

The thing in the upper left is a model I made of Bacteriophage T4, a virus of E. Coli bacteria made famous for its use in many laboratories for many purposes. 

The thing in the lower right is the baseplate of Bacteriophage T4, a mechanism that grabs a bacterium cell when the long fibers seen in the upper left are triggered by touching it.

The thing in the middle, looking evil and potent, is the needle structure that penetrates the cell wall of the bacterium, injecting into the bacterium the virus DNA, thus commandeering the bacterium cell to make hundreds of new virus copies.

A portrait of a weapon of mass destruction at the nanometer scale, disassembled into components like a field artillery piece. 

The Bacteriophage T4 is about 200 nanometers long. The shortest wavelength of visible light is 400 nanometers. Therefore, this picture could not be taken with a camera. In fact, it is not a picture at all, it is a rendering of data.

You could say that X-rays were used to gather the data, but not in any photographic sense. Thousands of frozen samples of Bacteriophage T4 were exposed to X-ray bombardment, being destroyed in the process. The X-rays were deflected according to the density of electrons in the atoms of the proteins making up the virus. Sensors surrounding the sub-microscopic virus registered these deflections.

Each of the individual T4 viruses was too small to arrange with any particular orientation. But the overall symmetry was known, and the top was distinguishable from the bottom. So the deflection data was converted by computation into a density volume fitted to be congruent with the expected symmetry.

In truth, the virus in its entirety was never examined this way. Instead, each major component, with five-fold or six-fold symmetry, was turned into data using this method, Cryo Electron Microscopy. These components were fitted together by scale and structure, to make a model of the whole virus. The volumetric density data was produced by the Rossmann Lab at Purdue University. The long tail fibers were imaged by transmission electron microscopy, also using X-rays.

I took that data, all publicly available, and used various software applications to turn the volumetric densities into geometric meshes. The meshes represent values where there is a sharp drop off in density, in other words, they represent surfaces.

But a surface of what? Think of what you see in this rendering as a shrink wrap of a complex compound of protein molecules. The atoms of the molecules are only hinted at in the surface shown because the resolution afforded by X-rays does not tease out distinct atoms. Instead, the structure of a given protein is synthesized in computer graphic programs and fitted into this shrink wrap to make sure it is accurate.

The rendering does not show these molecules themselves, but it does show the structural arrangement of the molecules in the needle. The ribbons are symbols of molecules with their atoms arranged in chains. The surface portrayal of the ribbons is purely arbitrary, they could as well be shown flat rather than plump.

Mostly absent from the data are the water molecules embedded in the structure. The water molecules might play essential roles in the adhesion of component parts and the mechanics of the components when the virus is triggered to penetrate the bacterium cell wall, but this is not known and not well studied.

What we don't see at all are are the surrounding water molecules. A water molecule is not spherical but it would just fit into a sphere two tenths of a nanometer in diameter. This means that a 200 nanometer high Bacteriophage T4 is to a 0.2 nanometer wide water molecule, as a 12 foot high model of T4 is to a BB, or small shotgun pellet. A twelve foot high model would be about five million times life size.

Water molecules vibrate and move violently. The effect of this pummeling on microscopic things in the water is called Brownian Motion. A T4 virus doesn't float around serenely in a placid fluid, rather, it is whacked around vigorously and constantly. Imagine yourself in a vat of violently moving shotgun pellets.

So look at the image again and think about what it shows. The surface of the thing is not the thing at all, it is a representation of data constructed not by optical imaging but by computation. The most significant aspects of the thing, like its structure, can only be shown symbolically. The environment of the thing isn't shown at all, and yet the whole apparatus that we call a virus must function within that environment.

And for all of the exquisite detail revealed by scientific visualization, we still don't know very much about how this thing, not so much a creature as a protein robot, makes its way in the wild. 

Bacteriophages. There are more of their kind, by individual count, than any other biological entity on earth.

Keeping Things In Perspective

Movies in 2D manipulate perspective, not depth per se.

The latest Star Trek movie 3D version was converted from the 2D master, so I watched it to see what marvelous or abhorrent transformation would be wrought by a high quality synthetic conversion.

Turns out, the retrofitting of 3D onto a 2D conceptualization doesn't do much of anything. J.J. Abrams is a 2D director, that is his cinematic eye. The 3D from the conversion is a presence without significance, in most cases depth without roundness, adding nothing at all and in some scenes, detracting.

This is a well crafted movie, with many examples of creative lens choices. A private tête-à-tête between Kirk and Spock, conducted on the command deck with crew all around, is made confidential by the use of a long lens that foreshortens the distance between the two heads in a tight shot. The camera had to have been far away to make the two actors seem so close together. Their faces are flattened, enhancing the guardedness of their expressions. But in 3D we are looking at the back of one head as a blur on a plane, and the front of the other head as a picture on a billboard.

Depth in the action montages is scarcely noticeable, although realistic. Quick cutting unavoidably slices and dices any perceivable volume into nothingness.

I am a devotee of stereo 3D used to tell a story, but this movie demonstrates that adding stereo 3D as a special effect afterthought is a wasted effort.

What is the heart of the problem? I don't think the issue is conversion technology, which can be quite good with adequate time and budget. After viewing many converted 2D movies, as well as numerous CG features originated as stereo 3D, I've concluded that the use of perspective in 2D movies, and the use of Z depth in stereo 3D movies, are antithetical to each other. They clash when employed together.

In 2D photography the combination of different lenses is a combination of different perspective croppings. The longer lenses employ foreshortening for certain dramatic effects. The wider lenses utilize vanishing point perspective for other dramatic effects. The projection of perspective onto a flat image is the artistic medium for 2D, geometrically.

Perspective does not show depth so much as it creates an impression of distances between objects as measured away from the camera. There are many other depth cues in a 2D image, such as the play of light across surfaces. Painting and cinema have established conventions that allow us to interpret the use of perspective and lighting as indicators of depth, and many movie sets since Casablanca have used these conventions in order to "cheat" a shot so that objects look more distant than they really were on set.

You could not fool the eye with that kind of cheat in stereo 3D. Why not? True depth perception is the product of parallax, not perspective. You can see depth in real life with only one eye open if you bob your head side to side, what some cats do in order to better gauge depth with a wider de facto eye separation. The side to side shift differentiates the depth of objects according to how much they move relative to each other, revealing the parallax differences between adjacent points of view.

Parallax shift is more apparent when seen from wide angle vanishing point perspectives. It is scarcely noticeable at practical lens separations for telephoto lenses. Look through binoculars at the tree trunks in a forest. You know that tree trunks are cylindrically round but through the binoculars they appear flat. The depth from nearest to farthest in view is apparent, but not the roundness of shapes.

Take your binoculars to the nearest railroad line and stand on the tracks, looking along them. This is a good demonstration of the difference between depth and perspective, for with 8x power binoculars, perspective lines scarcely converge to a vanishing point, and yet, you can see the flattened depth planes that are characteristic of binoculars. You could see more roundness if the binocular lenses were very widely separated, in the manner of old style gunnery binoculars.

However, unless one is simulating the view through binoculars, for good stereo 3D, long lenses should be avoided. It would be possible to make an excellent stereo 3D movie with only a matched pair of fixed focal length wide angle lenses, about 53 to 58 degrees horizontal angle of view (about 24 mm to 28 mm on a super-35 sized sensor). What a 2D director of photography does by varying focal length and camera placement in a motif set up for quick cuts, a 3D director of photography should do by varying camera separation (the stereo baseline) and camera rig placement in a motif set up for long takes on a mobile platform. This is the difference between manipulating perspective and manipulating depth.

Good 2D movie makers should just stay away from stereo 3D, and continue to do what they do, without dilution. Those who aspire to be good stereo 3D movie makers should abandon most of what is written in books about 2D photography. They should also study established stereo 3D techniques with detachment, feeling free to discover new variations on the old verities through brave experimentation.

The great stereo 3D productions are yet to come, and they will be created by young people who, because they were never 2D lensmen, have no set of best practices to forget in order to invent a new set of best practices for a fundamentally different medium.

Manipulate perspective or manipulate depth, make your choice. Used together, each detracts from the other.

Immersion is Absolute

In the English language the concept of immersion allows no qualification or quantification. It is an absolute. You are never more immersed or less immersed, well immersed or poorly immersed. You are either immersed or you are not.

A workaround for this absolutism is to carve out a part and call it a whole. Your toes are immersed in the cold lake water, but the rest of you remains dry.

For the term, immersive media, how does the dictionary definition square with usage?

While it would make no sense to say you were partially immersed in a media experience, it would be proper to say you felt immersed during certain scenes. The parts you have carved out for immersion are segments of time.

Here's a question: Would I claim, having recently plunged into an assortment of media offerings all grouped under the category, immersive media, that the intensive three day exposure felt immersive? Here's my answer: I felt immersed at times.

To be fair, critical detachment primes a person to be pulled out of the water at the slightest pretext.

The immersive media included stereo 3D watched on a 47 inch diagonal flat screen, stereo 3D watched on a multiplex sized screen, and stereo 3D watched on an IMAX screen, plus features shown on a full planetarium dome.

My observations in this post are solely about the media platforms mentioned above, with an additional platform, my own monitor, brought in for comparison. The discussion is content agnostic, but it is not technique agnostic. Specific titles are not relevant, specific production techniques must be mentioned.

What does it mean to say you felt immersed in a medium? In simplest terms, a media experience is immersive if it shuts out your awareness of all rival stimuli. You forget the viewing space, you no longer notice the people seated nearby, you put out of mind any distracting thoughts, you don't wonder how the movie was made. You are not looking at a production manufactured for viewing on a screen, rather, you are within a subsuming environment of sights and sounds that carries you along on its own currents. When immersed, you might feel surrounded, you might feel absorbed, but you never feel detached.

By these criteria, I thought IMAX 3D felt immersive most frequently, when the stereo 3D technique used the negative parallax volume in front of the screen.

The effectiveness of IMAX 3D owes to multiple factors designed to eliminate distraction. The immense size of the screen extends to peripheral vision. The seats are mounted on steep risers fairly close to the screen. The speakers behind the screen fuse auditory and visual attention. Without utilization of the spatial volume in front of the screen, however, IMAX 3D is just a very large picture window. When action in negative parallax puts depth within reach, you are immersed.

A 3D venue at the local multiplex, or a very large 3DTV. could approximate this effect in a darkened theater space when the content technique places image depth in front of the screen and the viewer sits at a distance from the screen equal to or somewhat less than the screen width.

For a couple of weeks after the event, this is where my assessment stood, until I viewed a number of HD GoPro videos recently.

The GoPro videos were not in 3D but they used the full frame fisheye default setting for the camera. They all showed an extremely wide scope of view, the cameras were all mounted on a moving platform, usually a person performing an extreme sport in spectacular scenery, and I watched them all at full screen 1080 HD on a 23 inch diagonal monitor from three feet away, in a room illuminated by indirect daylight.

Initially, I watched these without bothering to put on my headphones. Much of the time I felt melded with the camera, fully present in the action. I was pulling backflips on a dirtbike, I was leaping off a cliff in the Alps, I was free diving in the Caribbean, I was motorbike touring across the Himalayas.

I put on my headphones and watched the videos a second time. The immersive feeling vanished. With the headphones off again, the immersive feeling returned.

Revelation forces reassessment. Wide camera lens angles of view are just as important to immersion as wide personal angles of view in the theater. A moving point of view creates an immersive sense of spatial volume. An image without artificial intrusion from narration or music is more absorbing, although environmental sound might enhance the sense of presence.

Most importantly, these factors inherent in the technique of the content itself, can trump factors designed into the display technology. I felt immersed in a room illuminated by daylight, viewing 2D on a relatively small screen.

In the near future, when immersive imaging best practices are combined with an ideal immersive display to tell a compellingly immersive story, will technology present the ultimate immersive experience?

On my porch through the wee hours of a night six years ago, I began reading by headlamp the last novel in the Harry Potter series, purchased minutes earlier on the midnight of its release. For several hours of autumnal darkness the real world, including my own body, ceased to exist. I was immersed while reading printed words in a book.

The ultimate immersive experience is not dependent on any particular technology, it is dependent on the user's commitment.

The Ultimate Interactivity

In media, what are the standards by which interactivity should be judged?

Should we assess the degree to which interactive media exceed words printed on paper in their ability to allow us control over outcomes, or should we measure the degree to which they fall short of the free form complexities in real world experience?

Let us first examine those apparently polar opposites for what they really are.

Words on paper are not without interactivity. Readers can control many variables. They can access content at random, they can visit any page as many times as they wish at intervals of their own choosing, they can visualize from the words what the characters and the action look like, they can revise their responses with repeated readings, they can imagine life for the characters after the denouement, they can conjure endings they prefer over the one the author wrote. Reading a book can be as interactive as the reader wishes it to be.

Real life, by contrast only appears free before it is actually lived. In retrospect, it seems like events turn out as they must, scripted by the choices expressing who we were then and the role we played, with sudden interventions by fate over which we have no control. There is not much interactivity where the actions are automatic, resulting from training and best practices. Any two climbers will likely ascend to the top of a Class 5.11 mountain, a similar end for similar decisions. An avalanche triggered by one skier but not another the previous day under seemingly identical conditions, is just bad luck. One got away with the risk, one didn't. And a fool may defy all odds by succeeding where experts succumb. Are the combined workings of skill and fate truly interactive, or do they just play out the notated moves in a cosmic ballet?

If our two end anchors are not what they first seemed, what are we to make of the middle? Can we define interactivity without comparison with absolutes that are not so absolute after all?

Consider a thought experiment that someday soon might be achievable. Imagine a Virtual Reality experience furnished by a headset device that delivers sights and sounds according to our movements. The audio and visual imagery is convincingly realistic because it responds to our behavior in that environment. This is pure simulation. How do we assess its interactivity?

We have control over ourselves in this virtual world but cannot be completely sure of the responses. We can stumble, and then we can get up and keep on going. We have no skill set other than our wits, and there are no rules and objectives other than what we set for ourselves. This simulated world clearly has its own structure of cause and effect, but this is not a game, not a story, this is an exploration of an environment.

I would argue that this thought experiment represents interactivity with penultimate purity, as self directed learning.

What is the ultimate form? We look back upon an experience, enjoining a dialog with ourselves about meaning. Reflection provides the ultimate interactivity, and it happens entirely within our minds.

Birefringence in a Crystal Ball

When Stereo 3D practitioners are asked about the S3D camera of the future, they almost always expect more of what they have, for less money. I don't think that is what they are going to get.

My wild guess, as wild as a grizzly bear and every bit as ornery, is that two big changes are on the way, one for the low end budget and one for the Hollywood blockbuster budget.

On the low end, I suspect without any evidence to back me up that the GoPro 3D development crew is working on a specialized, unitary S3D rig. The company was conspicuously silent about the absence of a 3D housing for its Hero3, but they haven't announced any abandonment of S3D, either. I think they are going to surprise everybody with a rig that allows interaxial adjustment automatically linked to focal length adjustment, coupled with offset of image placement on the sensors, in a watertight housing, for a price point less than one thousand dollars. I think the user will be able to monitor the setup through an Oculus Rift and control the camera through an iPad. I think this will blow wide open the production of Indie S3D.

By contrast, I suspect that development on the high end, for blockbuster projects, will make shooting S3D more like shooting the 2D with which people in the industry are most familiar. I think such a camera will use a dual lens fixed separation system, with an integral depth recorder rig. The depth recording rig will use the light field technology of the Lytro cameras to register depth information which will be synthesized by software into a full depth field for the entire range of parallax between the two lenses set about 100 mm apart. The stereo movie will be created in post production, by joining the imagery with the depth information at any desired interaxial distance equal to or less than 100mm. The price point? At least as expensive as the best mirror rig on the market, but the system will include a full post production suite.

Ironically, the most stereographically adept movies will be low budget Indies. The S3D Indie filmmakers will define the new cinema tropes of the new medium, while traditional blockbuster movie makers will continue to make FX laden 2D movies with 3D as just another special effect. The two approaches will be on even footing regarding technical quality of both the imagery and the 3D, meaning that the contest between them will become one of style.

The market is trusted to differentiate but I don't think that is the way the contest will play out. Indies and blockbusters don't vie for screen exposure or box office receipts, they battle for dominance of style, and the new always wins. The young movie makers who use budget equipment will create the new styles that land them bigger budget projects financed by studios.

Eventually, stereo 3D will fulfill its promise as a new mainstream medium with new creative possibilities.

Stereo 3D Verities and Balderdash

Stereo 3D oldtimers sometimes engage in dustups that resemble John Wayne westerns, with their obligatory barroom brawls. The fiercest arguments seem to be about semantics and math, each party insisting on the correctness of a certain label, each party asserting its math mostly with words. Not a lot of communication happens during these dueling decrees, and when they take place online, they can erupt into tiresome flame wars that obliterate the usefulness of a channel for weeks.

I thought this was only a geezer phenomenon, until I looked at the hottest spot at the moment for discussions about S3D, the developers' forum for the headset viewing device, Oculus Rift. Same old, same old, but with a cast of young people who are somewhat more polite.

For technical subjects, perhaps a new generation is better off not learning from the old. Perhaps the young are entitled to learn from their own mistakes, just like their forebears did.

Cruise online sources about stereo 3D if you suffer from insomnia. I'm up to my eyeballs with pictures of eyeballs. The "explanations" of stereo 3D tend to over-technologize stereo vision, both natural and synthetic. The thing about natural vision is, it just works, we don't need a manual to make sure it works right. The thing about synthetic stereo vision is, when it isn't done right it hurts, a terrific learning reinforcement. For this reason, common sense based on actual experience is just as important as a calculator when designing S3D.

One professional from the computer graphic side of Hollywood S3D, with whom I've conversed, seems to take a seat-of-the-pants approach. He tests various settings on a big screen with his in-house cohorts and notes for extreme settings the number of yelps. (People who work with S3D are just as sensitive to eyeball pain as anyone else.) In this way, one CG stereo designer with an animation studio gets a good feeling for how far to push the envelope.

I like this bold but viewer centric approach to S3D design. It leads to experience-based conclusions that old timers and newbies often miss, or misstate.

First of all, you CAN go to infinity and beyond. The traditional wisdom is that the on-screen separation of right and left points should never exceed the separation of your two eyes, which together gaze into infinity along parallel lines of sight. Regard this constraint as you would regard the speed limit on Highway 50 across Nevada: advisory, use discretion. Discretion is well advised by one's own awareness of what is actually happening. We can accept divergence, seeing "beyond infinity," but within limits and only for a while.

Next time you watch a 3D IMAX production, remove your 3D glasses to see to what extent distant objects are separated on the screen. In many cases, the separation is noticeably further apart than the separation of our eyes, which means our eyeballs are diverging from parallel alignment when we try to fuse right and left. But this doesn't happen for long, and it doesn't happen where our eyes are attracted to the main action in the scene. So divergence isn't a problem when used with discretion.

Second, the volume in front of the screen is the spatial bonanza zone for depth perception. The conventional constraint that a scene should stay mostly behind the stereo window frame deprives the stereo designer of the most valuable real estate. Our natural stereo perception is optimized for discerning depth at close range, and the stereo designer can show more roundness for close objects that extend from the screen into the theater. 

The next time you view an animated stereo 3D movie in a theater and the scene shows a medium or close up shot of the characters, glance to the vertical edges of the frame to gauge your distance to the screen. You will notice that the characters are well in front of the screen, and that their shapes are nicely rounded. You will also notice that you wouldn't have noticed if you hadn't glanced away from the center of attention.

As a new generation teaches itself stereo 3D, my advice is this: You can trust what you see if you take care to notice what you see. As an audience member or as a practitioner, don't be intimidated by confusing explanations, but rather, be empirically guided by what actually works and what actually fails to work. There's truth behind the doctrines, but you have to find that out for yourself. In so doing, you may discover new truths.