Issue link: http://digital.copcomm.com/i/371911
10 cgw j u ly . a u g u s t 2 0 1 4 "We used the height for scenes when they're standing around talking to each other. In the big action scenes, they are in 'turtle space,' so the stunt actors didn't need it." Stunt actors wore the iMocap suits but not the facial-capture rig, and all the actors playing turtles had big foam cushions to approximate a turtle shell on their backs. "If we didn't dress the stunt performers with the foam cushions, they'd move without regard to what they were meant to be," Weaver says. Filming the shots were two main stereo cameras, three witness cameras, and the two cameras on the head rigs on each of the fi ve actors playing the CG characters. "On a good day, we'd gather 1 tB of data per actor per day," Helman says. "That allowed us to fi nd all the subtleties." But it meant a total of between 4 tB to 6tB of data per day. Dealing with all that data was a he y job. A 14-person team from ILM managed the performance cap- ture. "Our footprint on set was pretty big," Helman says. "But we needed that many to capture the performances simultane- ously – manage the logistics, put the helmets on, recalibrate the cameras, check the marker makeup, and make sure the wireless data all transmitted. Doing facial and iMocap perfor- mance on set is challenging." R A W D A T A M A T T E R S The dual cameras mounted to the head rigs fi lmed the markers on the le and right sides of the actors' faces, taking RAW footage compressed on the fl y. Later, layout artists at ILM "solved" images from these cameras, from the witness camera footage of the iMocap suits, and the position of the stereo cameras used to fi lm the shots. They turned the dots in the images on the actors' faces and patterns on the iMocap suits into useful data. Michael Koperwas, creature CG supervisor at ILM, and R&D Lead Kiran Bhat led teams of artists and re- searchers who created and refi ned ILM's new facial- capture system. "There is a lot of action in this fi lm," Koperwas says. "The actors would even be jumping on a hydraulic rig moving up and down. Again, we partnered with Video- Hawks. They have a solid, 30 fps, 720p, dual- camera rig and, equally important, good helmets that are com- fortable for the actors." Before the on-set cap- ture, the team used Disney Research's Medusa system to do 4D scans of the actors (3D plus time). "We had the actors do expressions, smiles, frowns, and so forth, and then used Fez, our facial animation system, to break the expressions into animator- friendly compo- nents," Koperwas says. "To give animators more control, we didn't restrict ourselves to FACS expressions." The team would search through video of the scanned faces, pick the best example of a smile, a squint, and other expres- sions, select the Medusa- created mesh associated with each, and bring that mesh into Fez. "We wanted the best, most isolated example of an expression, like a smile," Ko- perwas says. "The actor may be squinting or have raised eyebrows, but we want only that smile. We extract it from the mesh, solve it (translate it to animation curves), and put it on sliders in Fez. The animators can make the smile bigger or smaller and open or close the lips. Fez automatically corrects it to keep it on model." The Muse team then transferred the actor's expressions (broken down into sliders) onto the corre- sponding turtle's facial rig. "If the actor smiles, the turtle smiles, and it's the same amount of smile," Koperwas says. That set the stage for applying facial expres- sions captured from the head-mounted cameras as the actors performed on location and in motion- capture studios. "We'd get footage from the plate cameras with the selected frames," Koperwas says. "This is a comedy, so the editor cut the dialog smoothly for maximum comedic eff ects. They might have captured the fi rst line of dialog in New York, the second at Digital Domain in LA, the third at House of Moves. We had to piece together the motion from those capture stages." And that's why the new system became important. "The fi rst line of dialog might have been delivered with a happy smile and the second with a neutral face," Koperwas says. "We'd get The Way Muse Works ILM'S NEW MUSE SYSTEM TRANSLATED FACIAL MOTION CAPTURE INTO DATA ANIMATORS COULD USE TO PERFORM TURTLES IN DIALOG-DRIVEN SCENES. P E R F O R M A N C E C A P T U R E "THE HEAD RIG ALSO HAD LIGHTS. SO WE COULD CAPTURE IN A VARIETY OF SITUATIONS – EVEN ON A ROOFTOP IN THE PITCH OF NIGHT."