The role of the Cinematographer in Virtual Reality
What is the role of the traditional Cinematographer in the new medium of Virtual Reality?
I’ve always been eager to try out new technology and figure out the advantages and limitations of each. But recently there’s a new medium in the market: Virtual Reality. With this new medium, there are new storytelling opportunities, but also potential storytelling problems. As we move forward with our Virtual Reality narrative film project “Universe 231” (http://universe231.com), these are some of the things that I hope to discover in the production process.
So the main question that I’ve been obsessing with is: in the emerging field of Virtual Reality film making, is there a role for the traditional Cinematographer, and if there is, what does that look like? The first step is to determine what the traditional role and available tool set of the Cinematographer is. The second step is to look at all prior crossover connections there are between Cinematographers and animations and video games. The third step is to examine all those elements and synthesize what tools and responsibilities could carry over to the emerging field of Virtual Reality for Cinematographers.
Because this industry is so young, there are very few standards established, so before we begin it’s important to clarify some definitions of terms. As of this writing, there are two flavors of Virtual Reality “Films”. The first are films that are shot with live actors on a video camera. This is style of VR is most closely related to traditional film making. The second style of VR films are fully animated films. They are basically animations generated by a game engine in real time.
It is this second style of VR film making that we will be examining. In order to understand the possible role of a cinematographer in VR we must first catalog the tools and responsibilities of the Cinematographer in traditional film production. After that we can look at the roll of the cinematographer in game and animation industries. From that we can deduce what the roll of the cinematographer in VR might be.
In the traditional film making, the role of the Cinematographer can be broken down into three phases: pre-production, production and post-production. Below are the responsibilities broken down for each of the phases.
Pre-production:
- Work with the Director to understand their vision.
- Work with the producer to select the camera, grip and electric crew.
- Participate on location scouting and provide feedback to Director, Producer and department leaders.
- Work with the Art Department to help plan production design.
- Work with the storyboard artist to determine framing and angle.
- Work with the Producer and 1st AD to schedule the shoot.
- Work with the Producer to select the best camera, grip and lighting equipment for the budget available.
- Work with the Producer to determine the optimal workflow from acquisition to distribution.
- Work with the camera department to shoot test footage necessary to ensure successful acquisition.
- Utilize the crew and equipment as necessary to accurately capture the Director’s vision.
- Utilize the crew and equipment in the most efficient manner possible to keep the production on time.
- Capture adequate coverage to ensure a problem free edit.
- Capture the footage with a sufficient quality to avoid post production problems.
- Adhere to the production workflow plan.
- Assist with film transfer/grading.
- Assist with producing distribution promotional materials.
- Color palate
- Framing
- Blocking
- Lighting
- Set Design
- Set Decoration (wardrobe, makeup)
- Film stock/sensor choice
- Lens choice
- Lens filtration
- Depth of Field
- Camera movement
- Coverage
- Color grading
For Animation, traditionally there is no official role of a Cinematographer. Usually the decisions that a cinematographer would make would be made entirely in preproduction mostly by the storyboard artist. In contrast, on a film production, the storyboard artist would work under the cinematographer. Is the cinematographer important then in animation? As far as blocking and rough look goes, those roles can be filled by the Storyboard Artist and the Concept Artist. This has been the tradtional workflow for animation production.
In 2008 however, there was an attempt at Pixar to bridge the animation/film gap by Andrew Stanton as he directed the animated feature film “Wall E”. He wanted to take his animation to the next level of realism. To his credit he brought in academy award winning cinematographer Roger Deakins (O Brother etc) to add his perspective to the visual process. Initially he was intending to a single day workshop on camera set up and general lighting. Instead he spent several weeks working with the animation production team to add cinematography realism to the production. Deakens was able to incorporate many of the traditional cinematography tools such as lens choice, focus, lighting, camera placement and movement. Since then Deakins has consulted with both Pixar and Dreamworks for How to train your Dragon, Rise of the Guardians, The Croods, Rango and more.
These are essentially production tools, however not pre or post production. Those are other areas for greater cinematographer integration.
For Video Game design, there are even fewer cross over opportunities for the traditional cinematographer. Usually those are in the game trailer for in “cut scenes” which are essentially animations embedded in the game at strategic points. Even though many games are described as “Filmic” or “Cinematic”, in the normal area of game play there usually is no collaboration with a cinematographer. Often those reviews are describing lighting, texture and pacing. The main difficulty with making a video game more of a “filmic” experience is that by it’s nature the game play is variable. With film the Director and the Cinematographer carefully control the viewer experience – where to look, how long to look, where to stand, where and when to move. With games, all of that control is ceded to the player. It’s all out of the Cinematographer’s hands. This translates into things like editing. It could be perceived that the closest analogy to a video game experience is the “Meis-en-scene” (one take, uninterrupted) shot. One of the most famous examples is the opening scene for Orson Wells’ “Touch of Evil” This one-take, ten minute shot follows a ticking time bomb from planting in a car, as it moves through the city until it finally explodes. While this is similar to a game experience in that the viewer follows the action uninterrupted, the difference is that in a game the player determines where to look (camera angle) where to stand (camera placement) where and when to move (camera movement). The player can choose to blow through a scene in record time, or linger all day, they can even decide to go backwards and revisit prior scenes. All of this is outside of the Director’s and Cinematographer’s control.
Now that we have examined the differences in Cinematography between Film, Game and Animation it’s possible to synthesize what the role of a Cinematographer might be in a Virtual Reality film. Virtual Reality is essentially an animation rendered in real time inside a video game engine (commonly Unity or Unreal) controlled by the player/viewer. So how can the traditional Cinematographer fit in to this mix? There are several areas that are common to all platforms:
Lens choice. Not just field of view, but also stylistic imperfections like lens flair, distortion, chromatic aberration, as well as any filters that may go over those lenses like Tiffen Black Promist, behind the lens or in front of the lens netting.
Film stock. One of the greatest weapons at the disposal of the cinematographer is in the choice of film stock. In the modern digital era, LUTs or Look Up Tables for film. In the game engine it’s referred to as Look Up Textures. Cinema Suite Inc. is a company that manufactures a plug in for Unity that can emulate specific film stocks or processing effects. In the game engine this is referred to as a “shader”
Lighting. Motion pictures have been around for over a hundred years and lighting has been a part of that ever since. It is not a surprise that both Animation and Game producers are turning to cinematographers to make their lighting more “filmic” In traditional film, lighting serves two main functions. The first is to establish a base level of illumination so that you can see what objects are where. The second function is to set the tone. You can build all the assets and texture them any way you want, but the lighting will determine how the player/viewer feels. There is a third function of lighting that is even more essential in the VR environment. Lighting can be used to direct the eye. In the most obvious example, imagine walking into a dark chamber but for a shaft of light falling onto an object of great importance in the back corner. Lighting can be used in a million other ways as well to advance the story in the direction the Director wants it to go.
Blocking. In simple terms, blocking a scene is the process of figuring out who stands where for optimal effect for the camera. On a film set it is forbidden for a Cinematographer to make acting suggestions, but they allowed to suggest changes in position or movement if it can help guide the viewer or motivate camera movement. In animation, blocking is determined by the storyboard artist. In the game environment, blocking is rarely a consideration and is a greatly under-utilized tool Blocking can be set up in the first image of a game, scene or level. The opening shot is one of the most important shots in a film, so too is should be in a video game. It is in this opening frame that a game cinematographer can control all aspects of the player/viewer experience. Once that player moves the controller, that opportunity is over. From then on there are a few other precious opportunities where blocking can be utilized. For instance whenever a player enters a room through door, the natural bottleneck can force the otherwise free roaming player/viewer to see from a specific angle. This is an opportunity to sculpt what the player sees, feels and learns. The same can also be done with blocking objects and walls.
A non-intuitive but important area of contribution by Cinematographers could be in the area level design. Level design in games is assembling the playing field. What room is connected to what room by which hallway, secret passage or warp transporter point. In one sense a level designer for a game is like an architect for a building. A good level designer understands that how a space reveals itself is part of how a story unfolds. The famous Architect Frank Lloyd Wright understood this. Good cinematographers are almost always good visual-spatial learners. They have to know how to set up a shot with blocking and camera placement in order to build up a space. Imagine the movie set of the Starship Enterprise. There is the Bridge, Engineering, the Warp Core, Ten Forward, Sickbay, all connected by hallways, turbo shafts and Jeffries Tubes. Now, of course all of those in real life are disconnected stage sets, but the Cinematographer is able piece them together in their head, photograph them and present them to an audience in such a way as to make them believe it is all one interconnected space ship. It is this ability for 3D visualization that can make the Cinematographer essential to the story when creating the 3d Virtual world for the player/viewer to experience without getting confused or frustrated.
