"A series of preliminary studies has tested the influence of virtual environment customisation, the influence of a soundscape, and photorealism on the feeling of presence."
Evaluating User eXperience (UX): In the lab or in situ?
UX refers to “perceptions and responses of a user resulting from the use and/or intended use of a system, product or service.“ (ISO, 2019). In this area, the following are considered and studied: emotions, beliefs, preferences, perceptions, physical and physiological responses, behaviours and actions that occur before, during and after use. This experience depends on the characteristics of the product and the personal characteristics of the user such as their physical condition, past experiences, attitude, personality etc. In addition, the context in which the product is used, and by extension, the context in which laboratory testing takes place, can also influence the quality of the experience.
This influence and the impact of so-called “parasitic” variables are driving researchers to conduct laboratory studies in neutral and similar environments for all testers. This increases the internal validity of the tests, i.e. the power of the conclusions that can be drawn from the sample under test. The impact of such precautions lies in the decrease in the ecological validity of the study, i.e. the decrease in the ability to extend the study’s findings beyond the laboratory. This is why tests are sometimes carried out in situ, in the real context of the use of the product or service. In the latter case, ecological validity is stronger but internal validity decreases. These tests cost more time and money, and are also more difficult to implement. In any case, the reduction of a type of validity may prevent a company from effectively predicting the reception of a product on the market. As such, the ideal way forward would be to find a way to combine the controlled aspect of the laboratory with the high ecological validity of in-situ tests.
Immersive environments in UX
Immersive environments reproduce specific laboratory environments (see Figure 1). A room is designed to make it as convincing as possible (see Bangcuyo et al., 2015; Hathaway & Simons, 2017; Sester et al., 2013, for example). In order to increase the immersion of testers, researchers sometimes add screens and speakers that play relevant content, and sometimes release different smells in the room used, such as a smell of coffee or cookies in an immersive kitchen. However, although these reproductions in the laboratory have clear advantages, they also have their fair share of disadvantages. They are very expensive: furniture corresponding to the environment in question must be bought, with new furniture every time you change the environment. Finally, this all takes up space: as needed, an entire room can be requisitioned, or even multiple rooms. An alternative solution, while remaining immersive, should therefore be compact, mobile, easily modifiable and customisable. The virtual reality option seems to be a no-brainer.
Figure 1
Virtual Reality (VR) track
Immersion is an increasingly common term, and somewhat goes everywhere. Researchers use it to describe the level of technology involved in a situation that aims to plunge an individual into an environment that is not reality (Slater & Wilbur, 1997). The technology involved is then described in four dimensions. The inclusive dimension refers to the degree to which reality is obscured. The extensive dimension indicates the range of sensory modalities covered by the device considered to be immersive. The third dimension, the surrounding dimension, describes the extent of the field of vision covered by the technology involved. Is this field of vision narrow or panoramic? The fourth and final dimension is the intensity dimension. It encompasses the concepts of display quality, and the richness of the content of the virtual environment.
Virtual reality is a highly inclusive technology. It is completely superimposed on real visual information. Virtual reality headsets are equipped with earphones and are accompanied by joysticks. They therefore cover sight, hearing, and touch and can be easily combined with the diffusion of smells, which places them very high on the extensive dimension. These helmets also offer a panoramic field of view with a trace of the movements of the individual’s head to allow them to explore the space 360° around them. They are therefore strongly surrounding. Finally, although the display quality is below what television screens, for example, can offer, it is still good. In addition, the intensity dimension also takes into account the richness of the environment in terms of content. While this may be affected by the available computing power, it is limited only by the creativity of the developers.
Nevertheless, while the relatively optimal degree of immersion offered by virtual reality is an immeasurable asset, it is not an end in itself. Immersion is linked to another key concept: presence. This is the feeling of existence in the virtual environment, of being there. The highest level of immersion can be achieved without the person feeling that they are present in the virtual environment (Heeter, 1992). Conversely, the degree of technology involved may be minimal, and yet the individual may feel completely captivated by (and feel present in) the environment. This dissociation between immersion and presence showcases the individual characteristics. For some authors, the feeling of presence must be at the heart of virtual reality (Cummings & Bailenson, 2016). What is the most advanced technology to plunge an individual into a virtual environment if, despite everything, that individual remains insensitive to, and thus disengaged from, what is presented to them?
The contribution of Orange
As part of the “Immersive Experience” anticipation programme, Orange is interested in virtual reality as a tool for “sketching” places to live (see the “Sketch Up Your Place“ tool used in this study) or as a collaboration tool with Orange VR Collaboration that allows you to organise collaborative meetings or training enriched in virtual reality, thanks in particular to the proposal of tools that do not exist in reality (https://www.youtube.com/watch?v=17qJ8-PNrN4&feature=youtube)(https://www.youtube.com/watch?v=17qJ8-PNrN4&feature=youtube). Work in augmented reality is also being carried out within the framework of the Maori research project.
But it is within the framework of its research programme Augmented Coms and Memories that Orange is interested in VR as a context simulation tool for evaluating the user experience combining both internal validity and ecological validity.
The first phase of this work aims to compare measurements of UX obtained in VR, the lab and in situ. To do this, it is necessary to determine the appropriate virtual environment for this comparison, in relation to the product being tested. For the latter, our choice quickly focused on Djingo, the voice assistant developed by Orange. The interaction is via voice commands and would therefore be similar between the different test contexts. A virtual version of Djingo can also be incorporated into the virtual environment to simulate its presence. From there, the lounge option as a test context was needed. This is the space where Djingo is installed by most users.
A series of preliminary studies tested the influence of virtual environment customisation, the influence of a soundscape and photorealism on the feeling of presence. These studies also aimed to assess the influence of individual factors such as personality or habits in terms of immersion on the presence experience in a virtual environment. To do this, a comparison was made between an unknown participant lounge and a personalised virtual lounge for each tester, recreated from the Orange Sketch Up Your Place tool (Figure 2).
Figure 2
The feeling of presence in the virtual lounge was assessed with and without sound. And finally, the feelings of presence evoked by a lounge with a low level of photorealism and a lounge with a higher level of photorealism were compared (Figure 3). Each participant also completed a Big Five Inventory personality questionnaire and an Immersion Tendencies Questionnaire, with the aim of linking the dimensions assessed by these questionnaires to the sense of presence experienced.
Figure 3
The results showed that the personalised lounge did not create a higher sense of presence than the unknown lounge of the testers. The soundscape and photorealism, on the other hand, improve the sense of presence. It therefore seemed justified to keep the photorealistic lounge accompanied by a soundscape for the rest of our work.
With its initial work focusing solely on the virtual environment, the heart of the project could be addressed: the comparison of UX evaluations carried out in the laboratory, in virtual reality and at home. To do this, three groups of testers, each in one of the three contexts of interest, conducted an audio quality test (hedonic judgement test) and interacted with Djingo (user test) before completing questionnaires that enabled them to express their feelings about this interaction.
The results showed, first, that the objective quality of interaction with Djingo (obtained from the success rate of requests made to Djingo) did not vary according to the context. UX questionnaires did not show any major difference between contexts. Only four questions in each questionnaire (including 28 and 34) revealed differences. This tends to show that VR group testers would be more positive in their feelings. These differences also tend to show that testers from the groups invited to come to our premises (laboratory and VR) found Djingo more innovative than those staying at home. For example, testers, coming to the premises of an innovative information and communication technology company like Orange, would tend to overestimate the innovative nature of the products and services to be tested, compared to their first use at home. These interpretations should however be viewed in perspective since they are based on only a few questions.
With regard to the audio quality test, the results show that VR group testers seem less able to discriminate between the musical extracts presented, noting more favourably the intermediate audio qualities (Figure 4).
Figure 4
Two interpretations are possible. The first is based on the “wow” effect of virtual reality, which tends to make the judgements of the testers in this group more positive, except for the extreme qualities that remain insensitive to this effect. The second interpretation is based on the sound that is played through the VR headset earphones: although the volume has been adjusted so that testers can hear the music on a speaker near them, it may be that the artificial sound atmosphere of VR has interfered in the assessment of intermediate qualities by disrupting the focus of hearing attention on the task of hedonic judgement.
Conclusion
The results of this first test session are very encouraging in that they show that VR does not introduce major bias in the user experience and its evaluation. Nevertheless, caution must be exercised, as the use of VR itself may have biased the answers of the testers in the context of the audio test in particular.
In order to validate the relevance of VR in the evaluation of the UX, it is also necessary to ensure that the measures collected are accurate. Typically, accuracy is stronger in the laboratory than at home, and the ideal situation, for VR to be a viable alternative to both methods, would be that VR responses are more accurate than at home, or as good as in the laboratory. To measure this accuracy, it is necessary to perform a second test session in which the testers will perform the same tasks (audio hedonic judgement test and user test with Djingo).
Furthermore, if VR accuracy is established, the next step would be to attempt to replicate in VR the context effects described in the literature and known to influence judgement and decision-making processes. This would allow the credibility of VR to be established as a context simulator during user testing.
