Issue link: https://digital.copcomm.com/i/18322
n n n n VR•Medical we don’t have physical backing for the x, y, z coordinates. We lay out the network to make it attractive and informative.” To produce the images, the team uses three programs. Avizo from the Visualization Sci- ences Group for the voxel-based images based on MRI scans, confocal microscopsy, and OCT. “We also use Avizo for vector data,” Banfelder says. “In collaboration with the Cooper Union, a local engineering school, we’ve visualized airflow in the nose and throat. It’s like airflow over a wing, but it’s inside the nose. We use it for 3D fields like that.” For molecular visualization, the team draws on the public-domain software Visual Mo- lecular Dynamics (VMD), supported by the Teoretical and Computational Biophysics Group, an NIH Resource for Macromolecular Modeling and Bioinformatics at the Beckman Institute, University of Illinois at Urbana- Champaign. “It’s software that many research- ers use,” Gracia says, “so it’s very convenient.” Lastly, the group writes proprietary soft- Surgeons can move intuitively inside the large voxel-based images using a wand or by moving their heads. Inside the Cave, they can quickly see elements difficult to visualize from slices produced by medical scanners, and sometimes discover relationships impossible to determine from the slices. says. “But it’s an intuitive interface. You can move around and see objects drawn from a new per- spective. Or, you can just turn your head.” Banfelder adds, “It takes people about 15 minutes to get oriented. After one or two ses- sions, they don’t need any help.” Gracia joins in: “Tey get confident after a couple of hours. Tey quickly learn that if they want to turn things around, they can move themselves around the object.” When the team first introduced the Cave to people at the medical college, they assumed that radiologists would be the ones most interested. “Tey were the least interested,” Borcherding points out. “Tey already had the 3D model in their heads.” Instead, the large 3D images inside the Cave were more interesting to sur- geons and others who hadn’t spent years men- tally visualizing the models. Te first project the team worked on was for a graduate student who was studying zebra- fish. “He had found a publicly available data set of an optical scan of a zebrafish that we ren- dered,” Borcherding says. “We also rendered images from his lab of zebrafish that had been injected with calcium imaging dyes to study patterns of neuronal activity.” Te team displayed the stereo 3D images 30 October 2010 on the walls of the Cave. “When he brought in the rest of the lab,” Borcherding continues, “they said things like, ‘I can see how the hind- brain connects now.’ Tey understood how things were connected; they could see the big picture and the fine details.” Adds Banfelder, “Tey said they wished they had the Cave when they were trying to learn anatomy in the first place. Tey wouldn’t have had to build a mental model; it was right in front of them.” Tools Now, the surgeons and researchers typically use the Cave to look at data in three broad catego- ries. Te first involves data from various types of body scans, MRIs, OCTs, and confocal mi- croscopsy, which, in the Cave, become voxel- based images. A second category includes x, y, and z coordinates of atoms and the bonds that connect them for displaying molecular simula- tions of proteins and the small molecules they interact with, particularly lipid membrane- bound proteins involved in cell signalings. Te third category includes abstract data sets, that is, networks showing gene interactions. “Te data- bases come from a variety of experiments,” Ban- felder says. “When we render these networks, ware. “For the general network visualization, we’ve been writing our own code based on Vrui VR Toolkit, a framework developed for immersive applications by Oliver Kreylos at UC Davis,” Banfelder says. “It’s not well known, but it works well. We write OpenGL code, and it renders across all the projectors. It doesn’t care whether the user has a mouse or a 3D wand; it just makes it all work.” Discoveries Many researchers have now joined the oph- thalmologist in making discoveries inside the Cave, sometimes to their surprise. Gracia and Banfelder tell one story: Gracia: “Te researcher was working on integrins, which are proteins that are so huge they’re difficult to see even with a big monitor and in 3D. Tere are so many atoms, it’s hard to see every detail. He had been looking at this protein in every direction for years.” Banfelder: “We were amused because he was a little skeptical. He had a 3D printout of the protein, a little 3D model he had made with stereolithography that he held in his hand. In the Cave, the model was so much bigger. He could change the colors, look at everything close to one part. He had flexibility.” Gracia: “I don’t remember his exact words, but he said something like, ‘I’ve never seen it from this angle.’ He could see something in there, some interaction that he had not seen before.” Alongside the researcher in the room were several students, all discussing the structure of the protein. Te researcher wore the tracker and moved the wand. “He had been work-