Order of Multitudes

Different Textures of Data: Talking with Holly Rushmeier

Holly Rushmeier is the John C. Malone Professor of Computer Science at Yale University. Her research and teaching interests include data visualization, object shape and texture capture with digital tools, and the application of human visual perception to computer graphics. These research interests have also resulted in a number of collaborative projects at Yale and elsewhere, using digital technologies for cultural heritage preservation. Before coming to Yale, Professor Rushmeier worked as a researcher at the IBM T. J. Watson Research Center. We discussed some of her recent work and how the global interest in big data has impacted Yale.

Sarah Pickman: Can you speak briefly about your background and professional trajectory before coming to Yale, and some of your broad research interests?

Holly Rushmeier: I’ve had a long path to Yale. My background is in mechanical engineering, which was the subject of my undergraduate studies and Ph.D. I’ve worked as a researcher in industry, for various companies, and taught before coming to Yale in 2004. I’ve been interested in graphics and data visualization for a long time, going back to the 1980s. More recently, I’ve also been interested in how to apply some of these same visualization technologies for cultural heritage preservation—things like capturing data on cultural objects’ shapes and textures to recreate them digitally. Recreating textures with digital tools is a huge field, and some of the biggest and most obvious applications are in movies and digital gaming, but there are also many applications for creating digital replicas of historic objects. One of the things I liked about coming to Yale was the opportunity to work on cultural heritage projects and with the collections and expertise here. These interests also relate to larger issues of how to translate human visual perception into digital tools, so most recently I’ve been spending a lot of time reviewing the scholarly literature on perception and thinking about how this research can be used to create more effective synthetic images.

SP: You’ve worked on a number of projects to replicate cultural heritage digitally, from ancient Egyptian archaeological sites to rare books in the collection of the Beinecke Library at Yale. As a computer scientist, what kinds of reactions to your work have you received from your colleagues in the humanities ?  

HR: The reaction to these kinds of projects has largely been very positive. I think initially there were some people in the humanities who were resistant to digitization projects because they were afraid that objects would be digitized so they could then be discarded! But today people are really interested in digital versions of objects and books being made available so that they can be more widely accessible. Even those people who don’t have the money or resources to study the originals can interact with these things. I also see colleagues who are now thinking about digital replicas as not just copies of objects, but as a form of evidence or “ground truth” for academic study. They are kinds of citations, in a way—just like a textual footnote, a sophisticated digitally-rendered version of an object can serve the function of supporting a scholarly argument.

In general, what we think of as “cultural heritage preservation” or “cultural heritage studies” has always had a strong scientific foundation. For example, the physical conservation of objects has been informed by art history, but the techniques to conserve or restore objects have come from chemistry and physics. In this field there’s always been a collaboration, in a way, between the sciences and the humanities.

SP: A large part of your work consists in researching how to replicate the texture and appearance of real-world materials in digital formats. Can you talk a bit about what’s involved in recreating textures? What does it look like to collect data for this kind of research?

HR: In general, working on textures involves collecting a huge amount of data, but all different kinds of data. Some of this work builds on the physics of how light interacts with different surfaces. But then there is the more subjective, descriptive data we collect from human viewers. We can do this using Mechanical Turk-style research, asking people to look at things and pick an option that best describes them visually—which of these images are of metals? Which of these are “not metals”? Are their surfaces hard or soft? Are the colors regular or are there variations? If we allow people to give textures different attributes, how do we as computer scientists then attach these descriptive attributes to mathematical models? Then you have to figure out how to improve the models by creating meaningful “dials” for people to adjust the digital image to more closely match what they see in the real world.

As another example, last autumn I published a paper with two Yale colleagues where we said, “OK, let’s just look at textures that are important for rendering buildings. Let’s collect a huge number of images of shingles, of bricks, of grass, of stucco, that might be used as models for digitally representing buildings.” And you might think—well, a brick’s a brick. But there’s a huge number of things that people call “bricks.” And even if you just walk around New Haven, you will see materials in all sorts of different colors and shapes that might conceivably be labeled as “bricks.” Even an attempt to narrow down the possible pool of data you could pull from opens up many doors! You have to think carefully about what your categories and sub-categories might be. You also want to make sure that whether you’re creating these data sets yourself or mining data sets that are already out there, you’re thinking about the ways in which this information might be biased towards or against particular groups of people.

SP: You’ve worked in the field of data visualization for a long time, but I think it’s fair to say that in the last ten years or so it’s really become a “hot” field in the larger world, not just in academia. As someone who has been researching data visualization for several decades, what changes have you seen as this field has attracted more attention?

HR: I came to Yale in 2004, and for the first couple of terms I was teaching an introductory course in visualization. But the interest just wasn’t there. People didn’t seem to get how important visualization is. So I mostly dropped it from my teaching for a few years. And then ten years ago it finally hit, and places like the New York Times and Washington Post started having visualization specialists as part of their staff, doing what I’d call responsible visualizations. They were really engaging with the questions of when to use hue variations, when to use color variations, when you use numbers of dots instead of radii of circles —basic principles of visualization. There was data visualization in the 1980s and 1990s of course, including in news media, but it’s only been more recently that we’ve had the convergence of people familiar with best principles and good tools—especially particular languages, interfaces, and products like Tableau.

The highest award in computer science is the Turing Award, and this year it was awarded to two people, Dr. Pat Hanrahan and Dr. Ed Catmull, for their work in computer animation—they were both early figures at Pixar. But interestingly, even though he is connected to this pioneering company in digital animation, I’d say Dr. Hanrahan is even more notable for his work as one of the co-founders of the company that developed Tableau, because Tableau has really enabled people to effectively and responsibly make very sophisticated visualizations of data. And data visualization has become such an important tool for giving the public access to critical information.

SP: I imagine that at Yale, this broader interest in, and demand for, good data visualization has translated into more interest in this area than in 2004!

HR: Yes, in the last few years I’ve taught a junior/senior level undergraduate class on visualization a few times and it’s become much more popular than when I first started teaching at Yale, especially since computer science has also become a much more popular major. One of the great things to see is that in general a lot of students are interested in using digital technologies to address societal problems: conveying accurate information to the public, engaging in green engineering, applying them to medicine and law.

For example, graduating Yale seniors submitted their senior projects in the spring, and we had a number of students in computer science researching topics related to law and privacy and security in the digital realm. These projects had an obvious technical component, but also asked humanities-type questions about human wants and behaviors. There are also students working on projects with significant visualization components, including one student who looked at capital punishment in the U.S. going back to the seventeenth century, and one looking at the impact of the digital divide; that is, unequal access to computing and the Internet. These students are using computing for projects that some people may not have thought about as being “computer” problems in the past.