High resolution display screens, motion and touch sensors, interaction with other people and virtual objects, channels that deliver sound, touch, taste and smell, stereo vision and sound, all wrapped into a 360 degree 3D sphere of experience — that’s the dream and expectation of VR (virtual reality). Is 360 degree video on the way to that ideal? Is what you experience with 360 degree video more real than video as recorded via a standard digital camera or via your smartphone camera and displayed on a flat rectangular screen?
In a blog post in March 2014, Mark Zuckerberg wrote about Facebook’s acquisition of Oculus Rift’s VR technology: “When you put it on, you enter a completely immersive computer-generated environment, like a game or a movie scene or a place far away. The incredible thing about the technology is that you feel like you’re actually present in another place with other people.”
Along with many others, I am sceptical about the claim that digital technologies can deliver such a convincing “feeling” of presence.
But I have to concur with Zuckerberg’s sentiment that follows: “People who try it say it’s different from anything they’ve ever experienced in their lives.”
That’s practically an admission that virtual reality is nothing like reality.
In fact there are many contributing technologies that are impressive individually (digital imagery, audio, sensors, actuators, etc), but it turns out to be extremely difficult simply to combine them all to deliver the completely immersive experience that’s promised.
Apart from the technical limitations, there are many who say human experience of the world is not like that anyway. Visual experience is not formed from geometrically constructed images projected before the retina, at whatever resolution, nor an amalgam of sense data projected onto or wired into the sense organs of the human body.
Nor do we even need this technology to have completely engaging immersive experiences. You can watch a cinema, television or tablet screen and be immersed. You can lose yourself, and lose awareness of the frame, the room, the background noise, and clock-time in many encounters with media, e.g. even while reading a book or playing a low-res video game.
Whatever the degree of media channel resolution, we human beings deploy our imaginations to make up for any media gaps, deficiencies and defects. In fact, I am prepared to say that our perceptual apparatus thrives on noisy media channels.
So it is worth developing a reasonable technical understanding of media such as 360 video as it exists now, with all its positives and drawbacks, rather than propose that 360 video is on the way to something even better, bringing us ever closer to the dream of true virtual reality as suggested in an article in Wired Magazine. The article alerts us that this technology is not real VR, especially as it makes people nauseous after prolonged use. Don’t be fooled, don’t believe the hype, and don’t be put off. The best is yet to come, says the article.
With it’s origins in flight simulators and immersive architectural renderings, VR developments were motivated by a quest to provide simulated realities, useful in learning and training environments. But seen through the lens of art, VR does not have to deliver what its name suggests, i.e. reality.
I am content for VR to offer something strange, different, and challenging. VR also brings to light certain aspects of what perception is and is not. VR is a technology that also brings to light defects in common understandings of perception and our sense of place in the world.
What is 360 degree video?
360 degree videos are video recordings, usually created by more than one camera to cover a near 360 degree field of view, i.e. the view in all directions, including pivoting right and left, looking in front and behind you and up and down, and from one viewing position. Sometimes several cameras are positioned on the same mount pointing in different directions to provide the full spherical coverage. If more than one camera is used then the output is stitched together to create a complete image. The video image is usually stored as a flat ultra-fisheye image on a flat surface.
The viewing experience makes more sense if the video image is projected onto a virtual sphere, with you, the viewer in the middle of it. A smartphone is a great way to view such an image as smartphones have motion and position sensors. With the right app, you can move your smartphone around and gaze at the screen as if looking through a window into the virtual sphere of the video.
VR viewers
Thanks to the high resolution of smartphone screens, developers have created apps that position two nearly identical video images side by side in the viewing screen while in landscape format. If you put a lens in front of each image and hold the phone screen close to your eyes then that gives you an image for each eye.
A wearable mount such as Google Cardboard holds everything in place before your eyes providing something like an immersive view of the video. You can move your head in any direction and see different parts of this spherical video. This is monoscopic 360 degree video. It is monoscopic as both eyes see the same image. You are seeing a flat image projected onto a virtual sphere.
Stereoscopic 360 video
Conventional stereoscopic video is recorded by two cameras next two each other, roughly the same distance apart as a person’s eyes. That works fine for a framed image as when viewing a 3D movie in a cinema. But stereoscopic 360 video is much harder to record. It is not enough simply to position two 360 degree cameras next to each other, as the person viewing the video could be looking in any direction.
Not only would the viewer see the other camera when they look to the right or the left, but the stereoscopic image will only be apparent when looking forward, i.e. in the same orientation as the two recording cameras. If you look behind you then the scene will look inside out — concave surfaces will look convex and vice versa, as when you swap the image in a 3D slide viewer. In any other position you will get double vision, or at least severe eye strain.
To work, stereoscopic camera arrangements require several binocular cameras looking in key directions up, down, right, left, etc. The software used for viewing this complex composite of images must select the stereo view according to the direction the viewer is looking, and stitch and blend images on the fly. This is all computationally expensive and the imagery is unlikely to be as clear as for monoscopic 360 video considering the current state of display technology.
3D models
360 degree stereoscopic imagery is more straightforward in the case of virtual 3D models as in immersive video game environments and CAD (computer aided design) architectural models. The virtual camera positions are constantly recalculated by the gaming software. As the viewer turns her head so the two virtual cameras move in the model to match your eye positions. With the current state of smartphone technology such models have to be much simpler than what you see in 3D cinema CGI, and there’s the overhead of having to create everything as a computer model.
The further advantage of 3D model based 360 imagery is that the viewer is not fixed in one position, but can move through the environment if the game space allows it. In the case of 360 degree video the viewer is fixed in the same position as the camera when the recording was made. Whether the camera is fixed in place or moving, the viewer is fixed to the recording camera.
So 360 degree video does not allow free movement through the space in which the video was recorded. But the payoff from 360 degree videos is that we can be watching a recording of a scuba dive in the Great Barrier Reef, a trek across a glacier, be amongst a pride of lions stalking a wilder beast and watch a small boy tell his story of life in South Sudan. Such videos are recorded, edited and produced with recourse to all the creative talents of regular video production and documentary film making.
Truthful representation
The term “virtual reality” suggests the viewer gains access to some unmediated truth via the medium of 360 degree video. Does 360 degree video usher in a new wave of journalism that puts the viewer into conflict-damaged Ukraine to see the devastation for themselves? After all you can look around in any direction rather than looking where the camera operator tells you to look.
The New York Times (NYT) has made much of its own foray into 360 monoscopic video journalism. The award winning short video The Displaced was a supplement to a text-based story in the NYT about refugee children in Ukraine, South Sudan and Lebanon.
In an interview reported online, NYT editor Jake Silverstein indicates some of the intriguing new practices that accompany the creation of 360 degree video, but there’s also much that draws on standard documentary technique.
There’s ambient music, the sequences only last for 15-20 seconds or so. The viewer is certainly not free to explore the space as they wish. There is a great deal of skilful editing.
With 360 degree video the producers face challengers different to those encountered in creating videos set within the standard frame. There are new considerations in positioning the 360 degree camera. The initial position of the view when the particular scene opens is clearly important, and how the producer directs attention within the 360 degree field of view.
Silverstein mentions the camera operator having to hide from view, rather than simply hide behind the camera. That’s an interesting challenge in the case of the scene of an open cucumber field being harvested by Syrian refugees in Lebanon.
One might also notice there are optimal camera sitings to give the right amount of foreground and background interest and detail. It is also interesting in the video what happens when the kids hold the camera, which seems to be on a monopod, as they run around a village.
There are many journalistic devices deployed in “The Displaced,” acceptable and expected in documentary making, such as choosing dramatic scenarios: the air drop, boys throwing ammunition shells from a roof, the evocative boat ride through the reeds, and the boy’s narrative about crocodiles.
There’s a lot happening in each scene, and a slight frustration by the viewer that they may be missing something if they look in the wrong direction. If the video has a rectangular frame then that’s less work for viewers as they know where to look.
So 360 degree video is not a means of putting you the viewer in the scene in a way never before encountered. It deploys all the devices of artifice and creative production, and I for one wouldn’t expect or want it to be otherwise.
References
- Fitzgerald, Michael. 2016. Jake Silverstein: ‘Immersing you in worlds not your own’ — New York Times Magazine editor-in-chief Jake Silverstein on telling stories with virtual reality. Nieman Storyboard online.
- Smith, Will. 2015. Stop calling Google Cardboard’s 360-degree videos ‘VR’. Wired Magazine, (November) online.
Notes
- I am grateful to MSc Research student Eva Zheng for alerting me to some of the journalistic challenges of immersive VR.
- See related posts: Is cyberspace real? Pictures devour reality, 3D passive unrealities, Computer images and realism, Wonders exposed, and Hygienic reality.
- The panoramic scenes above are of the harbour at Bastia, Corsica.
Reflections on Technology, Media & Culture 