Stereo post-production techniques
Post-production steps that work on the stereo pair: generating new views from depth, the depth script, transferring mattes and subtitles to both eyes, and delivery formats. For depth grading and target-screen masters, see Post-production and mastering.
Depth-image-based rendering
Depth-image-based rendering (DIBR) generates the image of another viewpoint from one image and its depth map. Each pixel is shifted horizontally by the disparity computed from its depth:
where: d is the pixel shift (fraction of image width), Z the depth of the pixel, b' the desired interaxial, C' the desired convergence distance
Where it is used:
- 2D-to-3D conversion: the second eye from a single image and a depth map made by hand or by software.
- Depth compression: narrowing the depth range of a shot that has already been filmed, that is, making it look as if it had been shot with a smaller interaxial. HIT cannot do this; see Post-production and mastering.
- Multi-view displays: in-between views for displays that need more than two viewpoints. See Viewing devices.
Hole filling
Regions that are visible from the new viewpoint but hidden behind a foreground object in the original image have no data. These holes (disocclusion) open at object edges, in the direction of the shift, and are filled by extending the background or by image completion (inpainting).
- The larger the change in interaxial, the wider the holes. DIBR is more reliable for small changes.
- Hair, smoke, semi-transparent and reflective surfaces have no single depth value; these regions are often fixed by hand.
Depth script
The depth script is the shot-by-shot depth plan of a film. For each shot, the parallax range (nearest and farthest point) and the parallax of the main subject are shown on a chart. It is prepared in pre-production and used during shooting and depth grading.
- Depth is planned according to the dramatic structure of the scene; the widest range is not used throughout the film.
- The parallax of the main subject is kept close between consecutive shots. If a cut jumps, HIT is changed gradually before and after the cut; see Post-production and mastering, Editing continuity.
- Each shot is drawn within the comfort window of the target screen. With more than one target screen, each needs its own limit line.
Stereo roto and matte transfer
A roto, color-correction or visual-effects matte is first drawn in one eye. To transfer it to the other eye, the matte is shifted horizontally by the disparity of its depth. If a disparity map is available, each point is moved by its own disparity.
The transferred matte is checked in the other eye. If the matte edge does not have the same disparity in both eyes, the edge of the correction appears to the viewer as a separate line in front of or behind the object.
Depth-aware subtitling
In a stereo image a subtitle also has a parallax. Because the subtitle is drawn over the image, the occlusion cue puts it in front. If its parallax places it behind the object it covers, the two depth cues conflict and reading becomes harder. The subtitle is therefore placed slightly in front of the nearest object in the area it covers (depth-aware subtitling).
- Subtitle parallax is set per shot or per subtitle; consecutive subtitles should not jump far in depth.
- The subtitle also stays within the comfort window of the target screen. A subtitle far in front of the screen makes the eyes re-converge at every reading.
Delivery formats
A stereo pair is put into a video file in two ways:
- Frame-compatible formats: both eyes sit in the same frame, side by side (SbS) or top-and-bottom. In half formats each eye gets half the resolution: in a 1920-pixel-wide half SbS each eye is 960 pixels wide and is scaled back to double width for display. In full formats the frame is twice the size.
- Multiview coding: the two eyes are coded in separate layers, each at full resolution. See Spatial video and MV-HEVC.
In H.264 and HEVC the frame-compatible layout is signaled by frame packing information added to the video stream (frame packing arrangement SEI). If this information is missing, or the player does not read it, the two eyes appear side by side in a flat image and the layout is selected by hand.
The loss from half resolution is more noticeable on headsets with many pixels per eye and on large virtual screens. For these devices full SbS or multiview coding is preferred.
Multi-device mastering
The comfort window of the same content differs on a cinema screen, a TV, a phone and the virtual screen of a headset, because the window depends on screen width and viewing distance. On headsets and glasses, the apparent size and distance of the virtual screen are used instead of a physical screen. Target screens are grouped into the fewest masters: the method is on the Post-production and mastering page, the device classes on the Viewing devices page.