Linking the Virtual and the Real: Gesture based Interaction with Digital Heritage

One of the aims of meSch is to realize tangible interaction with digital heritage. This could for instance be realized by interaction based on sensors in smart objects or – much more physically challenging for visitors – by using gesture based interaction. The April Meetup of the Virtual Museum Network Amsterdam was about ‘Linking the Virtual and the Real’ and focussed on a specific project that uses gestures to interact with the virtual reconstruction of a tomb. Opportunities and challenges that arise when museums use virtual reconstructions to re-contextualise objects were discussed.

Still of Etruscanning interactive

Still of Etruscanning interactive

Starting point of the discussion was an interactive in the Allard Pierson museum which combines virtual reconstruction and Kinect technology to allow visitors to virtually explore museum objects.  The Etruscanning application is  a virtual reconstruction of an Etruscan tomb containing virtually restored objects which were found in the tomb and which are now mostly on display at the Vatican museums. Visitors can navigate through the tomb by making hand and arm gestures and can select objects to find out more about them.

The fragile laws of interaction

First, the conversation focussed on the navigation functionality. Research by others has shown that visitors are sometimes inclined not to use an interactive if they feel like they have an audience watching them. Similarly, not knowing how an interactive should be used or what it does might deter potential users. In addition to this, research at the Allard Pierson Museum seemed to indicate that people using the interactive felt social pressure to use it for longer than they wanted to. They didn’t want to let down the people who were watching. What we learned from this is that the social and physical context of a digital interactive highly influence visitors’ response to the interactive. This aligns with the interactive experience model developed by John Falk and Lynn Dierking, described in the museum theory classic The Museum Experience.

Evaluating the interactive space

In the Allard Pierson Museum visitors can navigate the Virtual Reconstruction of the tomb and select objects by making various hand and arm gestures. Clear instructions are given at the start of the interactive, but understanding and remembering the various gestures might still prove to be an issue. The group discussed how visitors who entered the space halfway through the experience could be informed about the use of the interactive. Potential aides that were discussed during the Meetup were a permanent line of text on the screen, markings on the floor or text labels. The Allard Pierson Museum is hoping to experiment with floor markings and text panels and to evaluate visitors responses to these changes. A comment was also made about the small blue icon at the bottom of the screen, representing the user and mimicking his or her gestures. Although this icon was introduced as an aid to help users navigate, some thought it to be distracting rather than helpful and wondered if it could be replaced by something more neutral. Thirdly, there is the potential issue of onlookers entering the interactive zone and (perhaps unknowingly) interfering with the interaction. Various ways of preventing this were discussed, including the current use of lighting to create a barrier, markings on the floor or physical barriers.

What’s next?

After exploring the existing interactive, the Meetup focussed on a planned follow-up project which will consist of a Virtual Representation of a Roman domus, filled with virtual replicas of objects that are on display in an exhibition space next door. The interactive experience will be  a game and the main question posed by the Allard Pierson Museum was: How can we encourage people to also look at the real objects and how can we facilitate the transition from digital interactive to gallery space? One suggestion was to use actors, representing characters from the game who can help to extend the gaming experience to the real world. Although this is not a permanent solution for the Allard Pierson Museum, due to costs and staffing issues, it is certainly something that can be experimented with during events such as Museumnacht (museum night). Another solution worth exploring is the idea that some sort of ‘reward’ can be found/discovered in the proximity of the real objects. These rewards could be short films, handling objects, or additional information about the objects on display. This could also be a way to use the flow of the game to make the transition between the virtual world and real objects on display on gallery. The diverse backgrounds of participants meant that the challenges encountered by the Allard Pierson Museum could be approached from various angles. This provided the staff of the Allard Pierson Museum (and hopefully all participants) with plenty of food for thought and new ideas to test and evaluate.

More on the Meetups

The Virtual Museum Network Amsterdam Meetup is organized every eight weeks and hosted by the Heritage Lab of the University of Amsterdam and meSch partner Allard Pierson Museum. The aim of this event is to bring together heritage professionals, developers and researchers to discuss issues related to digital heritage innovation.

More on Etruscanning

More information about the Etruscanning project can be found at

  • BLOG