To kick off the start of this workpackage we held a joint meeting in July to collaboratively reflect on our evaluation strategy, brainstorm evaluation challenges and to make sure that all partners get actively involved.

Team members with different expertise taking part in the joint workshop of the Co-design (WP1), Case-studies (WP6), and Evaluation (WP7) workpackages in Stuttgart, July 2014. (Photo by Areti Damala, UoS.)

All partners were asked to reflect on the opportunities and challenges related with the existing meSch prototypes. Challenges and opportunities were examined by three groups in which at least one Cultural Heritage professional, one designer and one engineer participated. One group worked on the Loupe, the second one on the Interactive Showcases and the third one on the Companion Novel (initial Book version). The groups were asked to reflect, then report on key-research questions and issues they felt it is important to evaluate.

Group work on evaluation issues and questions for the Loupe prototype. (Photo by Areti Damala)

A discussion followed in which two categories of key research and evaluation questions were identified, the first one being common and thus “portable” for all of the three prototypes that were examined, the second one being specific to each prototype examined. Similarly, the research questions that were identified can further be divided in three categories, not surprisingly to be mapped onto the three main Evaluation (WP7) tasks.

Devising an evaluation strategy

During a first phase, we will be evaluating important yet individual system components, for example; the meSch authoring tool, its visual interface, the login data visualisation (as well as aspects related with the utility and usability of both the components) and the first meSch prototypes that are being designed and implemented.

Then, and once the first exhibitions integrating meSch technology will be open to the public, formative and summative evaluations will be conducted in order to gain insights on the museum visiting experience and the way it is shaped, transformed, enhanced and altered when the meSch prototypes are in use. Formative evaluation is going to be carried out prior to the opening of the main exhibitions in which the meSch technology is going to be integrated while summative evaluation is going to take place once the exhibitions are open to the public.

Finally, and since one of the most important goals is to enable Cultural Heritage professionals to easily create their own smart exhibits and the adaptive content to be stored on them, a third important task consists of evaluating the ease-of-use of the authoring development environment from A to Z.

In the picture below we see some of the main questions raised by the team that worked on the book version of the Companion Novel.

 Evaluation issues for the Companion Novel (Book version) as identified by the workshop participants.
(Slide ©the meSch consortium).

Discussing evaluation challenges with the Cultural Heritage community

The first important evaluation questions as identified in the first internally held brainstorming workshop were presented to a greater audience during a research seminar at HATII (Humanities Advanced Technology Institute) of the University of Glasgow, on September 30^th. The meSch project presentation as well as the presentation of the most important evaluation – research questions identified in Stuttgart, generated a very lively discussion with the seminar participants (academics, Cultural Heritage professionals, post-graduate and undergraduate students in Digital Cultural Heritage).

Is there really a gap between the digital and the physical?)

One of the questions that was asked concerned the gap between the digital and the physical. Is the gap also perceived as such by museum visitors? Or is this solely a concern of museums and museum visitor studies professionals? One of the participants also asked whether we plan to involve visitors in the design (or re-design process) of one or more prototypes as part of our evaluation strategy. Finally a large number of ideas, suggestions and questions concerned the evaluation plans for the meSch prototypes presented, namely the Loupe, the Companion Novel (book and belt version) and the Interactive Showcases. We were asked whether it might be possible to have adult and children’s versions as well as evaluate the prototypes (and compare the results) with different age groups. A lot of interest was generated as well by the possible ways through which logging in data can, and will be exploited to provide museum professionals with valuable insights as to how the prototypes are used and which contents and narratives are the most popular so that the meSch museum visiting experience is further enhanced.

Interested in finding out more on evaluation studies in meSch?

Three three totally different, yet complementary museums participate in the meSch project, these are;  Museon, the Allard Pierson Museum and the Museo Storico Italiano della Guerra, which also features an outdoor Cultural Heritage site and at least three forthcoming exhibitions integrating the meSch technology, several evaluation studies and activities – qualitative as well as quantitative. Each of which promise to shed light on the ways through which meSch can make a difference both for museums and museum visitors alike.

If you are a researcher, a museum visitor or a museum with an interest in meSch evaluation activities, you can make your interest know by contacting us on this address: meSch@shu.ac.uk

We will get in touch with you for disseminating our evaluation findings and/or inform you about on-line and on-site studies in case you would be interested in participating in one of our studies or surveys. Stay tuned and consider of subscribing to our newsletter to follow all the latest news and developments from the meSch project.