New Dimensions in Testimony

New Dimensions Envisioned

Language: English

A simulation of New Dimensions technology as it may appear in a classroom environment.

New Dimensions in Testimony is a collection of testimonies designed as an interactive educational tool to permit students far into the future to “talk” with Holocaust survivors about their life experiences.

Though this emergent technology is still developing, the beta version already enables students to interact with the projected image of a real Holocaust survivor, which responds to questions in real time. The survivor selected for the demo project, Pinchas Gutter, is among the nearly 55,000 survivors and witnesses of the Holocaust and genocide whose testimonies live in the Visual History Archive. But the NDT project is a different initiative. For this, Gutter was interviewed again, this time before 50 cameras arranged in a rig to capture a three-dimensional recording of him telling his story in a new way, by answering questions that people are most likely to ask.

Work is underway to interview a handful of other survivors in similar fashion; all have also given testimonies that are stored in the Visual History Archive.

The value of the project is to provide an intimate experience with eyewitnesses to history who are uniquely qualified to reflect on life.

Extending the Conversation of Testimony

For more than 70 years, Holocaust survivors have recounted their stories thousands hundreds of times to tens of thousands of people all over the world, providing invaluable insights that shape and inform perspectives. As a part of those encounters, people have asked questions in their own words about issues that are important to them.

New Dimensions in TestimonySM is an initiative by USC Shoah Foundation to record and display testimony in a way that will preserve the dialogue between Holocaust survivors and learners far into the future. Collaborating within the project are Illinois Holocaust Museum and Education Center, with technology by USC Institute for Creative Technologies, and concept by Conscience Display. Funding for New Dimensions in TestimonySM was provided in part by Pears Foundation, Louis. F. Smith, Melinda Goldrich and Andrea Cayton/Goldrich Family Foundation in honor of Jona Goldrich, and Illinois Holocaust Museum and Education Center. Other partners include CANDLES Holocaust Museum and Education Center.

It will eventually be available at selected museums and learning institutions around the world to enhance USC Shoah Foundation’s mission of keeping voices of the Holocaust and other genocides alive for education and action.

New Dimensions in Testimony will incorporate interview content recorded with advanced filming techniques, next-generation natural language processing, and specialized display technologies to deliver a learning environment where a survivor will answer questions as if he or she were in the room.

USC Shoah Foundation and ICT compiled the questions posed for New Dimensions in Testimony from a variety of sources, including hundreds of students and members of the public. In all, more than 2,000 questions have been compiled, covering a vast range of subjects.

Students and the general public will “meet” moving images of survivors, and when they ask a question, “natural language” technology software will retrieve an appropriate response from a database of the survivors’ answers, creating an interactive dialogue. Audio-visual recordings of the survivors can be projected via two-dimensional screens, but in appropriate settings new visualization techniques pioneered by ICT will display the survivor in three dimensions – no 3-D glasses required – to provide an experience as close as possible to face-to-face interaction.

This emergent technology will allow viewers to see the survivor from any vantage point and will be adaptive. Whether in a classroom or a museum setting, it will look and feel realistic, as if the survivor is making “eye contact” while responding conversationally, with answers that are authentic and spontaneous.

Advancing Technology for Humanity

The key element of the project is the ability of the system to interpret questions being asked and find the most relevant response. ICT pioneered the natural language understanding technology that is used in New Dimensions in Testimony to provide viewers with a truly immersive experience. Whether people ask, “Where were you born?” “Do you believe in God?” “How did you survive?” Data is captured and processed into video segments that can be played back verbatim, precisely as the survivors delivered them. The playback technology enables the survivor to seamlessly answer the question posed and is able to recognize similarities between word patterns in questions and answers. Still in development, the playback system allows the testimony to be viewed in 3-D from the front, as if it were given in a classroom or museum setting, diffusely lit to blend with the environment. ICT has developed a number of virtual human characters with this capability for a variety of purposes, but this is the first time a real person has been integrated into the technology.

Words such as “hologram” and “avatar” fail to accurately describe New Dimensions in Testimony. Rather, New Dimensions in Testimony is a “virtual conversation.” What sets New Dimensions in Testimony apart is the ability to engage conversationally with the survivor by asking questions that trigger relevant, spoken responses.

Years from now, long after the last survivor has left us, New Dimensions in Testimony will be able to provide a valuable opportunity to engage with a survivor and ask them questions directly, encouraging them, each in their own way, to reflect on the deep and meaningful consequences of the Holocaust. Being able to ask a survivor questions will allow students to be active participants in their learning and develop important communication and critical-thinking skills. Being able to ask their own questions will teach students things about the Holocaust they never could learn from just reading a history book. By providing valuable points of view from someone who was there, students can better understand the human story behind the Holocaust and understand the impact it had on real people in a manner that is responsive and engaging, making history more relevant to their lives.

Conscience Display envisioned the idea and brought it to USC Shoah Foundation and ICT. USC Shoah Foundation supports the overall coordination and production of the project, advises and consults partners in collection development, coordinates implementation in a museum setting, and works with other partners to develop the questions. USC Shoah Foundation will also preserve the video testimony files and resulting metadata (questions and answers). ICT is developing all technology for the project; advanced filming, natural language processing, and display technologies.

A pilot installation was developed with the Illinois Holocaust Museum and Education Center. 

Pilot installations have also been shown at the United States Holocaust Memorial Museum, CANDLES Holocaust Museum and Education Center, the Sarah and Chaim Neuberger Holocaust Education Centre,  Holocaust Museum Houston, and the Museum of Jewish Heritage.

A permanent exhibit will open at the Illinois Holocaust Museum and Education Center in October.

New Dimensions in Testimony is expanding the scope of research in the field of Holocaust and genocide studies and represents a compelling addition to the nearly 55,000 eyewitness testimonies currently in USC Shoah Foundation’s Visual History Archive.

How Natural Language Technology Works

One critical element of the new project is developing the ability to understand questions put to a survivor and select appropriate responses. This will allow a person to engage in an interactive dialogue with the survivor, asking questions in their own words and hearing the survivor’s response. The technology must be able to recognize similarities between word patterns in questions and answers and choose answers to new questions that are similar, but not the same as, questions for which the answers are known.

The Institute for Creative Technologies (ICT) has developed a number of virtual human characters with this capability, for a variety of purposes; for example, they are used for training clinical psychologists and soldiers to conduct different kinds of interviews or recognize when someone might be lying. To achieve the level of authenticity necessary to provide viewers with a truly immersive experience, we will apply the technology to the answers Holocaust survivors give us, so that young people interacting with them will have their questions answered.

—David Traum