Syracuse University Magazine

Virtual Explorations

Virtual Explorations

Through immersive media technology, Syracuse faculty and students are finding new ways to enhance their disciplines and work, while also considering its impact on the future

By Wendy S. Loughlin

Newhouse communications professor Frank Biocca wrote the book—literally—on virtual reality with the publication of Communication in the Age of Virtual Reality (Routledge) in 1995. At that time, most people still viewed the technology as the stuff of science fiction. Today, Biocca is director of Newhouse’s Media, Interface, and Network Design Lab, known as the M.I.N.D. Lab, part of an international network of human computer interaction labs. At the M.I.N.D. Lab, what was once a thing of the future has become a main research focus, as the rapidly developing medium continues to grow. 

Virtual reality (VR) technology allows users to interact with computer-generated simulations of images or environments in a realistic, nearly true-to-life manner through the use of a specialized headset. Similarly, augmented reality (AR) introduces computer-generated effects, such as video, audio, or graphics, but presents them in a real-world setting (think Pokémon Go). They are often referred to collectively as immersive media. “This is an entirely new stage of media,” says Dan Pacheco, Peter A. Horvitz Endowed Chair in Journalism Innovation at the Newhouse School. “Much in the way that television was a new thing, this is a brand-new medium—some people call it the ‘fifth medium.’”



Dan PachecoPacheco (left) has been working with immersive media for several years. In 2014, he and then-student Irfan Uraizee ’15 spent the summer at the Des Moines Register in Iowa, participating in the groundbreaking virtual reality storytelling project “Harvest of Change,” produced through Gannett Digital. The project, which tells the story of a changing family farm in Page County, Iowa, through 3D game interaction and 360-degree video, went on to win a National Edward R. Murrow Award, which honors outstanding achievements in digital journalism. 

Pacheco had also experienced the innovative immersive journalism projects “Hunger in Los Angeles” and “Project Syria,” developed by VR pioneer Nonny de la Peña, and he could see VR’s powerful potential as a storytelling tool. In spring 2015, he launched Virtual Reality Storytelling, the first course of its kind at any journalism school. Though he was initially worried about a lack of student interest, the class filled in the first three days of registration. This spring, it filled in three hours. 

The interdisciplinary class is based in Newhouse’s Alan Gerry Center for Media Innovation, a creative hub where students from across disciplines work with next-generation technology. Pacheco’s students explore the differences between immersive media and traditional media—including possible health or ethical considerations that might accompany this new form of storytelling—and analyze existing immersive media packages created by Vive and Oculus developers. At the same time, they gain hands-on skills in 3D modeling, 360-degree video, and computer-generated imagery, creating immersive stories that they promote on Newhouse’s Interactive Media Wall and online at www.vrstorytelling.org.

Ultimately, students gain experience creating stories for various types of communications, including entertainment, journalism, advertising, and public relations. “We’re at a point now where every serious media company is doing something in this area,” Pacheco says. “They’re asking us, ‘Do you have qualified students who are interested in interning?’ So we’re at this really fun initial time.” Pacheco’s students have produced stories on topics ranging from the extinction of dinosaurs to color blindness to the Dakota Access pipeline. 

Carly Port ’17, a television, radio, and film major, created a story package focusing on present day Ausch­witz. Port had visited the concentration camp in 2012 and wanted to recreate the powerful experience for people who will never see it in person. Her story takes viewers on a virtual tour of Auschwitz II Birkenau as it looks today, based in part on photos she took during her visit. The tour is overlaid with audio from an interview with Holocaust survivor Israel Arbeiter, originally recorded by the World War II Foundation. Port had met and heard from a survivor on her own visit to the camp, a profound experience that she wanted to share with her audience.  

Her project touched on what many see as the most powerful potential for VR storytelling—the ability to create empathy. “I believe VR has the power to inspire empathy unlike any other storytelling platform, and I wanted those who will not be fortunate enough to physically visit this historic site and feel all the emotions I felt to have this virtual experience,” she says. “I want to tell the stories of survivors, who have so much insight into a world we never knew.” Port says she would like to continue to work with VR and social justice topics after she graduates this spring.



Visualizing Future Designs 

Ralf SchneiderNewhouse isn’t the only place where people are working with immersive media. Across campus, students and faculty are experimenting with the technology’s potential in areas like product design, music, gaming, and architecture. In the College of Visual and Performing Artsindustrial and interaction design (IID) students in the Product Design course explored the potential impact of VR and AR on the field of design. Under the direction of IID professor Ralf Schneider (above), students used Microsoft HoloLens to envision future applications of the technology to improve the design process. “Our program focuses on the user-centered design approach,” Schneider says. “Our work focuses on scenarios and visualizing ideas of improving the future.” 

HoloLes view of studioStudents researched the social, technological, and economic factors driving AR and VR and made projections into the future, looking at how the technology might play into the brainstorming, usability testing, and branding phases of design. For example, one student showed how user testing might be accomplished through AR in the near term, and through an artificial intelligence-AR interaction in the long term.



DSCF6556.jpg

The technology also has great potential in architectural design. School of Architecture professor Bess Krietemeyer (above) is head of the Interactive Design and Visualization Lab at the Syracuse Center of Excellence, where she is a faculty fellow. Krietemeyer’s research focuses on the ways emerging material technologies, human interaction, and computer simulations can influence the design of sustainable buildings and cities.

Together with immersive display artist Lorne Covington of NOIRFLUX, Krietemeyer developed an immersive projection environment of interactive architectural systems that users can also experience while wearing a VR headset. This allows her and collaborators—architecture faculty colleague Amber Bartosh and mechanical and aerospace engineering professor Jianshun Zhang of the College of Engineering and Computer Science—to virtually test and study architectural systems and technologies without having to develop actual working prototypes, which can be cost prohibitive. Their research also allows for modifications to proposed design elements before implementation. 

Their work facilitates one of the most challenging but important aspects of architectural design: human interaction. For example, when the team tested a solar responsive building envelope—a façade that adapts to environmental conditions—they measured not only its energy efficiency, but also its effects on user control and comfort. Ultimately, this promotes a kind of “co-design” process in which end users contribute to the development of responsive environments that better serve building occupants. 

“VR and immersive technologies enable designers to better visualize and engage with spatially and temporally dynamic conditions of the built environment—lighting, atmosphere, adaptive systems, and user behaviors—in ways we haven’t done before,” Kriete­meyer says. “We can test design ideas four-dimensionally and through first-person perspectives. For example, by providing a user-driven, walk-through experience of an interior space, while simultaneously mapping its ambient thermal energy flows, we can better understand how a space functions and feels at the same time. VR technologies not only reveal uncertainties or opportunities within the design process, but they explore new modes of architectural representation by translating between technical data and the imaginative potentials of projective space.”

Innovative Advances 

At the technology lab NEXIS (New Explorations in Information and Science) in the School of Information Studies (iSchool), students work independently on innovative projects that advance the fields of information science, engineering, and technology—including VR technology. Jorge Forero ’17, an information management and technology major in the iSchool, built a virtual reality music visualizer to “create the ultimate audio and visual experience,” he says. Conor Ried ’17, an economics and political science major in the College of Arts and Sciences, used VR software to create virtual buildings for his gaming project. 

As with any new technology, there are hiccups along the way. Kevin Spector ’17, also an information management and technology major, set his sights on making a more realistic virtual experience. His starting point was the “uncanny valley” concept, which refers to the discomfort people often feel when viewing a computer-generated figure that has almost—but not quite—human qualities. “I wanted to identify a process that would take real environments and virtualize them without reducing quality,” he says. 

Spector used 3D modeling and photogrammetry software, which extracts geometric information from two-dimensional images, to recreate a room based on photos. But the sheer number of high-quality photos needed to recreate the space, the required amount of computing power, and the need for a data center proved to be insurmountable hurdles. Spector ultimately put the project on hold, but remains interested in the “massive potential” of VR technology and plans to continue exploring its applications. 



Cognitive Impact 

We might not yet understand how to harness and responsibly manage the power of immersive media, but the work being done at the M.I.N.D. Lab is moving us toward that understanding. Newhouse communications professor Makana Chock is an expert in media psychology who conducts research on media processes and effects. Chock has begun to explore the impact of virtual and augmented reality stories on the human brain. “This is a relatively new, under-explored area,” she says. 

She suspects the effects of immersive media may be substantially different from those of traditional media, particularly in terms of memory and perceived realism—a measure of how realistic users judge media content to be. Studies have found that people will sometimes misremember events from traditional media as real-life experiences. “Is that stronger in a completely immersive environment? That’s what we’re trying to figure out,” she says. 

Chock brings people into the lab to experience immersive stories and scans their brains to measure which parts of the brain are responding. “We have old brains—our brains developed long before we had TV sets or movies—so they respond to stimuli to a great extent as if [the stimuli] were real, regardless of whether or not they are,” she says. This holds true even with traditional media; the fact that immersive media may have the same effect but magnified is central to its potential impact. “In VR your senses are telling you this is real—so how do you parse that out?” she says.

Chock is also using immersive media to take a deeper look at confirmation bias, which holds that people seek out or interpret information based on their preconceived notions, ultimately leading to errors in recollection and interpretation. In the lab, subjects are pre-screened for biases before experiencing a virtual reality scenario involving four men of different races at an airport. Chock measures subjects’ physiological responses both as they observe the scene and later, as they recall it. One theory is that immersive media might be used as a tool for changing attitudes. 

There is currently no code of ethics in place to help guide producers of this kind of content, Chock notes. Some organizations, like The New York Times, use existing journalistic standards when producing immersive stories, but without truly knowing the effects of this kind of content, it’s hard to know if those standards are enough, she says. 

The University is the ideal place to ask these kinds of questions, she says, hoping to see a center established at the Newhouse School that would advance education, training, and research in immersive media. “We have student storytellers creating content, we have faculty studying the effects of the content, we have the M.I.N.D. Lab, where we can measure these things in real time, and we have industry connections to get the information into the field,” Chock says.

Understanding the Ramifications

Of course, immersive media technology will continue to take off, at the academy as well as in the industry, even before its ramifications are fully understood. While it is no longer a thing of the future, its full potential still lies in the future. Learning how and where best to use it, as well as using it with care and according to an agreed-upon set of standards, will be key to the further development and success of VR and AR. 

“These are a whole new set of powerful capabilities that can be used for good, but also for scary things,” Pacheco says. “The fact that you can trick someone’s brain into thinking that they are somewhere else, that they are someone else…that can be really powerful.”«



MINDLab3.jpg

In the Newhouse M.I.N.D. Lab, Professor Makana Chock (right) looks on as media studies graduate student Yeonhee Cho (left) helps fellow graduate student Se Jung Kim prepare to use Vive, a virtual reality system. As Kim enters virtual reality (below), his brain activity is monitored on the screen at left. Chock is studying the impact of immersive media on the brain.



MIND-Lab-2.jpg

A Variety of Virtual Reality Experiences

Beyond the educational sphere, immersive media technology has already taken off in the field of entertainment. Alumna Mary Spio ’98 is founder and president of CEEK VR, which develops virtual reality experiences that run the gamut from entertainment to education. 

Last year, through a partnership with Universal Music, the company worked with thrash metal band Megadeth to create a VR experience for the album Dystopia. Fans who purchased the album’s deluxe version received a cardboard headset, compatible with Android and Apple, and an access code that allowed them to watch the band perform five songs from the album. The “mini concert,” as it was billed, was shot in 360-degree video and set in a “dystopian universe” that fans are able to enter and explore. The project won a silver Clio Award, which recognizes innovation and excellence in the creative business, and Dystopia won a 2017 Grammy Award, in part due to the VR experience and the album awareness it created. 

At the same time, CEEK is exploring the educational potential of virtual reality, particularly in the field of medicine. “The most promising aspect of VR technology is the tremendous potential for human learning,” Spio says. CEEK is looking at ways to increase patient safety and reduce costs by providing online VR training for health care professionals. The company is currently working on a first-of-its kind VR CPR training package for Miami Children’s Hospital. 

Spio, who earned a bachelor’s degree in electrical engineering from the College of Engineering and Computer Science, is a deep space scientist who started her career with the U.S. Air Force as a satellite communications technician. She was drawn toward the idea of VR even before it existed when, at age 10, she watched a recording of the moon landing and tried to mentally project herself into the scene. “Through VR, [people] can actually experience things that otherwise would be unattainable due to geography, finances, or physical ability,” she says. “I believe virtual reality represents humanity’s hope for a truly connected world—a world where every human on the planet can have access to life-transforming encounters and knowledge.”  



patient1.jpg

Mary Spio ’98 believes virtual reality has potential for numerous applications, from creating event experiences to assisting with medical training.

Images courtesy of CEEK VR



CEEK_Stage2.jpg

Schneider and IID class photos courtesy of College of Visual and Performing Arts

Pacheco and M.I.N.D. photos by Steve Sartori

Krietemeyer photo by Charles Wainwright



  • Lifting Lives
  • Virtual Explorations
  • Solar Solution
  • You’re Hired