Issue link: https://digital.copcomm.com/i/196542
Motion Capture makeup and with a smaller number of cameras. " A typical system utilizes three pairs of cameras: two monochrome sets for shape and tracking and one set for texture and color. "The fundamental difference between our system and others that capture 3D data at video rates is that our tracking works at the pixel level rather than tracking features on the face. We use the natural skin texture and the pores. " The process begins with a template mesh that has the topology the customer wants. "We fit that to one frame in the sequence to get the initial mapping onto the individual actor, and then we track every vertex on the mesh, Urquhart explains, " "every single one of the 2,000 vertices. Other systems track a sparse set of data that drives a rig that deforms a character. We deform the face mesh directly. Our approach is full-3D. " The data lands on a facial animation rig eventually so animators can refine the performance, and Dimensional Imaging helps its customers build appropriate systems. "When you have a 2,000-vertice mesh, you need a detailed, accurate, and well-defined rig, Urquhart says. "We have our customers " go through 100 or more expressions that we track using the detailed mesh to create a rig. " Recently, the company has been working with a hardware company on a head-mounted capture system to make it possible for customers to capture the body and face simultaneously. "Our technology is amenable to a two-camera system, " Urquhart says. "The problem has been that the video from existing cameras is highly compressed. The compression method looks for similarities between two frames and deletes that, but it's exactly that similarity that our tracking software requires. The Kinect system is interesting because it produces a videorate depth map, which is similar to our system, but the fidelity of machine-vision cameras is an order of magnitude higher. The holy grail is a head-mounted system with high fidelity that can track thousands of virtual [not painted] marker points on a face. " Currently, Dimensional Imaging sells software systems, licenses software, and does service work. "The take-home message from me is that traditional methods based on sparse sets of points don't provide enough detail, Urquhart says, " so people are looking for an alternative, and hopefully that's where we come in. " The facial animation created for Davy Jones and the Hulk brought a visual effects Oscar and two nominations to ILM for Pirates of the Caribbean: Dead Man's Chest and The Avengers, but this magic didn't happen overnight. "We started research into facial animation back in 1999," says Mike Sanders, who is a digital supervisor at ILM and works in the Advanced Development Group for Lucasfilm. "Facial capture is just getting to the point where it's costeffective, scalable, and stable." As do other studios, the crew typically starts by scanning an actor and using that information to translate shapes derived from a set of facial expressions onto a facial animation rig. "For us, each show is a little different," Sanders says. "We like to stay technology agnostic because of client requests, but we also have developed proprietary technology. And, after 15 years of developing in this space, we've converged on a good method for translating performances into creatures and for giving artists controls to embellish the performance." To start the process, ILM, which is now part of Disney, uses Zurich-based Disney Research's Medusa system to capture expressions – that is, to do a "facial survey." The expressions include a combination of phonemes, FACs-based facial calisthenics, and, as Sanders puts it, "anything an actor can throw in." "The idea with Medusa is to use machine-vision-based depth stereo reconstruction space to acquire a likeness of an actor with temporal coherence," Sanders says. "We keep the deformation of the skin as it moves into different expressions. Using proprietary techniques to make the movement from shape to shape coherent, we acquire up to 100 shapes. The vertices are the same as the skin stretches around to form each expression. There are no cheats. No mesh shrink-wrapping, which would lose the local deformation." A proprietary "decomposition" tool then translates those shapes and moves them into ILM's FEZ animation system and onto a rig, which animators can use to refine the performance. When the actor's face drives a creature rather than a digital double, ILM has introduced an intermediate, validation path. "We build a digital version of the actor and have the performance drive the shape space on the actor," Sanders says. "That's what we did for the Hulk in Avengers. We captured a shape library from Mark Ruffalo and built a FEZ rig with his likeness. Then we had his performance drive his face. That way we could say to the director, 'Yes, we acquired that performance and that is what the actor did. Now, what would you like the creature to do from here?'" For capture, although they can use the plate from a principal camera, they tend to put 100 to 200 dots on the actor's face and use them to drive the shape space. "We also use the con- 16 ■ CGW Se pt em ber / O c t ober 2013 ■ DYNAMIXYZ OFFERS markerless tracking via a 3D sensor.