Using Motion Photogrammetry for Rendering Realistic 3D Movement in Movies

Movie makers are like magicians - when they show you a new trick, you want to know how it was done and what inspired it. This is certainly true of the creators of the movie Ghost in the Shell starring Scarlett Johansson. Live-action holograms of figures like kendo fighters or a geisha pouring tea appear throughout the movie's city landscapes. These live-action holograms, or "solograms", represent the 3D advertisements of the future and are shown suspended in the air, over and between buildings, and flowing between people on the street. 

To create the solograms, Digital Air Inc., a VFX and camera-array-technology company, created a special motion photogrammetry camera system. Photogrammetry is used extensively in motion pictures and video games to produce textured, volumetric 3D scans that can be rigged and animated to move. While this kind of still photogrammetry can produce a very realistic static 3D image, its dependence on the post-animation of a single texture map creates issues when the scans are animated. For example, the movement of complex surfaces like faces and fabric look less natural. Traditional photogrammetry is one moment in time, reanimated. In a film like Ghost in the Shell, set in a technologically advanced future, viewers would expect moving 3D images to be flawless.

Enter Digital Air and their new motion photogrammetry system. The system Digital Air used for the movie took the form of a dome-shaped rig made up of eighty synchronised FLIR Grasshopper (GS3-U3-50S5C-C) cameras. The cameras were recording at 2.5 k resolution and 24 FPS to create multi-frame textured object sequences with the same origin for each 3D model. The Grasshopper's auto synchronization feature, which ensured perfect timing of all camera shutters, was crucial to the proper functioning of the motion photogrammetry rig. All actor performances used to create the film's 3D animations were recorded inside it.

Digital Air's hardware system produced the consistent RGB data required for twenty-four full-body photogrammetric scans per second. Reality Capture photogrammetry software was used to create the sequential 3D models needed to make the photogrammetric scans into animated motion sequences consistent in scale and origin. Every frame is a completely new 3D model with different texture. The process created assets that could be rendered from any point of view and unlike conventional 3D scans captured the original motion of the performance photographically. The resulting scans could then be relit and rendered sequentially in post-production to reproduce the original performance in harmony with the CG constructed backplate and camera moves. The process also provided the ability to add nuances such as high-density, high-resolution sonograms for the more affluent parts of the city and pixelated sonograms with artifacts in other neighbourhoods like the red-light district.

Dayton Taylor, founder and president of Digital Air, notes that for him the solograms needed to resemble a "future retro tech visual"; like nothing ever seen before and also different from the actors in the film. He felt that this need made the film the perfect showcase for the possibilities of motion photogrammetry. Given the accolades the film has received for its stunning visuals, it would seem the world agrees.

About Digital Air

Digital Air develops and produces visual effects and camera-array-related technologies. They provide camera systems as a visual effects production service on film shoots around the world. They also license, build and install custom camera systems for events and permanent installations. 

www.digitalair.com