Issue link: https://digital.copcomm.com/i/196542
dynamixyz Markerless tracking with 3D sensor Gaspard Breton, CEO of Dynamixyz, has been working on facial capture and performance since he was a PhD student 15 years ago. He now heads a team of former PhD students and a company that sells a markerless, model-based facial-capture system and provides facial motion-capture services. "Like most of these facial systems, we have a [CG] model with the actor's expressions, Breton says. "When the camera " sends data from the actor, the system combines expressions on the model and tries to match what the camera sees. That way we produce the same expressions on the model as the ones the actor performs. When that is successful, every pixel of the face is a marker. We also construct the color so you can see a blush, and if the actor has wrinkles, we rebuild the wrinkles. We really reconstruct the image. " The company's first systems worked with 2D cameras, but at SIGGRAPH, Dynamixyz released a "4D" system – 3D plus time using a head-mounted rig. "You can capture 3D from a multi-view camera system, like Vicon's Cara, or with a 3D sensor, like Kinect, SoftKinetic's cameras, or ones from Leap Motion, Breton says. "We use " ones from SoftKinetic. The quality is not quite as good as the Kinect, but when it is on a helmet, it's less than one meter tours of the face, lips, eyes, jawline, and potentially optical flow, as well," Sanders says. "We can capture from the taking camera, witness cameras, or head-mounted cameras, but in any case, we prefer high-definition and global shutters that integrate the image in one instant rather than line by line. The future will be when we can incorporate depth sensing. We've toyed with putting Kinect-like depth sensors on head rigs, which is a good idea in theory. In practice, there are limitations – the noise level in the sensor, the movie lights, and the weight of the sensor. But, this space will be interesting to watch." Also in the future, Sanders sees the pipeline becoming more cost effective and scalable. "We're becoming smarter about techniques that transfer across characters so we don't always have to start from scratch," he says. "And, the fidelity of the shape space has evolved. We have more control, and the controls are more intuitive. Before, we'd build a smile shape and use the smile dial. But when someone emotes, the entire face deforms. The skin stretching on the forehead and neck is subtle, and you have to build that into the shape space. You have to leverage the information about the shape space of the actor's face to affect the track. That's the critical phase. Anyone can track 2D dots. But translating 3D dots to a shape space while maintaining the localized deformation is critical. If you don't have that under the hood, the character will always look flat and lifeless." – Barbara Robertson C G W S e p t e mb e r / O c t o b e r 2 013 ■ 17