Ethical aspects and principles of usability tests

As part of the psychology degree programme, students have to complete a certain number of so-called subject hours for admission to the final thesis. This amounts to around 30 hours in total and serves to familiarise students with research from the perspective of the participants before they plan and carry out psychological studies themselves. Although this is time-consuming, it is really valuable as you become more familiar with some of the pitfalls in communication and the design of the study. You also learn how you feel as a "test subject".

Knowing about the feelings and concerns of the test subjects is not insignificant, because studies such as usability tests are also accompanied by a certain psychological pressure on the participants, which in turn can also have a negative impact on the test result.

Possible fears of test subjects

• "Stage fright" or performance anxiety. This pressure to perform can in turn affect self-confidence and self-efficacy and lead to a kind of self-fulfilling prophecy.
• Test subjects might feel "stupid" if they can't solve tasks or find things in the system. They blame themselves and not the system, it feels like an IQ test to them.
• They compare themselves inwardly with other users according to the motto: "Surely others don't behave so stupidly..."

As moderators of a test, we should therefore always try to put ourselves in the participants' shoes: As test subjects, they are sitting in the "spotlight", i.e. they are the centre of observation and are asked to perform tasks with a product that is unknown to them and possibly not optimally usable in front of strangers. It is quite natural to be a little nervous in this situation. The participants then have thoughts such as "Am I doing this right?" or: "Do these people think I'm stupid because I didn't understand?"

This pressure is real. Jared Spool, an American usability consultant, told a story about seeing a test subject cry during a usability test. This situation came about due to an accumulation of careless behaviour on the part of the moderators and test leaders: the original participant didn't turn up on the day of the test and then an employee who had just completed her first day of work was simply brought in as a quick replacement. The team thought it was a good idea to use her because, unlike other employees, she knew little about the product. There were also a number of observers in the observation room who were not informed about how to behave during a test, including the test person's boss. And last but not least, no pilot test was conducted, so the team did not know that the first task the participants were supposed to complete was in fact completely outdated, incorrectly formulated and therefore impossible to perform. As the woman frantically tried to complete this first task, everyone but her quickly realised that the task was outdated and impossible to complete, and they started laughing at their own stupidity. Unfortunately, the user thought they were laughing at her and she then started to cry.

This is exactly the kind of situation we want to avoid.

How to respond to these fears

We can counter such fears by repeatedly emphasising, for example, that we are not testing the test person, but the system. This "I'm too stupid to use the computer" is unfortunately based on internalised thinking. Self-blaming is a problem here. Many users of computer systems still think that it's their fault if something doesn't work, that they can't handle the software "properly" and blame themselves rather than the system.

Here is a sentence that we like to repeat like a prayer wheel in this or a modified form during tests:
"We don't test you, we test the system." and:  "Any problems you may have are the fault of the system. And we need your help to track down these problems in order to make the system better and easier to use. You help to assess/evaluate the system as we are too blind to it."

The wording/framing - i.e. how the test itself is labelled - also plays a role here. For example, when we talk about "user testing", this implies that we want to test the user. But this is not the case. We are testing the usability of the system, which is why it is more appropriate to speak of "usability testing / verification" - above all before the users.

Appreciative and respectful behaviour also plays a major role. Sounds obvious, but it is important to remember this time and again, especially when things get hectic.

Therefore, here are some basic and simple ways to reduce psychological stress in test subjects and conduct a usability test according to ethical guidelines.

11 ethical principles for a usability test

1. Language. We argue in favour of avoiding the term "user test". We don't test the user, we test a system with the help of the user.
2. Creating a friendly atmosphere. The atmosphere should be relaxed and free from distractions and interruptions such as questions from others. Drinks and small snacks are always welcome during in-person tests.
3. Let the test person arrive. A welcome and some small talk are important for participants to "arrive" and relax. This should therefore be included in the schedule.
4. Informed consent. The informed consent should be given to the test subjects a few days before the test and signed and returned to the test team on the day of the test at the latest. This consent should contain information about the purpose of the test and the general procedure of the test. It should also clearly and precisely describe how the results will be used and how it will be ensured that the participant's data is treated confidentially. Participants should be informed comprehensively and clearly about how the data will be used. This means that there should be information about who can access the data, where and for how long the data is stored (GDPR) and how long participants can request the deletion of their data or its non-use - as this is usually difficult once data has been anonymised. The voluntary nature of participation and the possibility for participants to cancel the test at any time should also be mentioned. Furthermore, contact options should be offered for any questions.
5. Repeat the clarification directly before the test. Before the test begins, it is best to briefly go through all the points that can also be found in the informed consent with the respondents and make sure once again that everything has been understood and that there are no more unanswered questions in this regard. It is advisable to reiterate the anonymisation of the data and explain to the participants that, for example, quotes that appear in reports or presentations cannot be traced back to the respective persons. It should also be pointed out once again that the test subjects are taking the test completely voluntarily and have the right to cancel the test at any time. This means that the test subjects are in control if, for whatever reason, they start to feel unwell, for example.
6. Warm-up questions/pre-test interview. Before the actual test, where we start working on the tasks, it is a good idea to start with thematically appropriate general warm-up questions. These questions are usually of interest anyway and of a more open nature, e.g. questions about previous use of the product/service in question, questions about frequency of use, questions about specific experiences from the last use, etc. This conversation (pre-test interview) helps the respondents to "wind down" a little, relax a little and familiarise themselves with the environment and the moderators.
7. Control non-verbal cues and facial expressions. While we are sitting together with the test subjects, we should never behave impatiently and also have implicit, unconscious behavioural patterns on screen - such as our facial expressions. This is easier said than done, because a lot of behaviour simply happens unconsciously. It is therefore a good idea to include this in the moderation script or to have the topic on the screen. An eyebrow raised at the wrong moment, breathing too loudly or nervously tapping your fingers can be completely misinterpreted and referred to by the respondent to themselves or their performance - with consequences for the test.
8. Respect participants' time. It is a sign of courtesy if we respect the time of our participants: For us, this means being well prepared and not running over time. This is also where the special role of the so-called pilot test comes into play: the pilot test serves to test our test :) - the test of the test, so to speak. The pilot is very helpful to find out whether our tasks cause confusion or ambiguity or contain errors and whether the test conditions run "smoothly" overall. The pilot test runs exactly like a real test, including test person, consent, script etc... and takes place a few days before the first real test. This gives us enough time to make any corrections or formulate tasks in a more comprehensible way.
9. After the test: Have data usage confirmed again. At the end of the test session, the moderators should again ensure/ask whether the respondent agrees to the use of the data. It is possible that the participants may change their mind during participation. As already mentioned, it is usually difficult or impossible to withdraw at a later date due to the anonymisation of the data records. This should be clearly communicated.
10. For remote tests: offer technical setups the day before. For moderated remote test sessions, it makes sense to offer technical setups a day in advance, for example. This gives participants the opportunity to familiarise themselves with the technology, such as screen sharing, using a test page (not the test product itself!). Positive side effect: The test subjects get to know the test team, which reduces the excitement on the day of the test and any fear of contact.
11. Transparent communication about additional observers. Sometimes managers or developers also want to and should (!) be able to watch the tests. This allows members of the team to experience the problems that users experience when using the system in question at first hand and can be important for establishing an understanding of UX in the company and for improving the user experience. increase the UX maturity level of a company. On the other hand, it can become problematic if too many people observe the test at the same time, as this adds another artificial dimension to the test by making users feel even more "observed" and "tested". This can increase the pressure to perform and the "performance anxiety", which in turn could distort the test results. It is therefore better to have fewer observers and rotate the team per test or test series instead of placing the entire team in the observation room.

If there are other people observing the test in addition to the moderators...

• ... should always be pointed out transparently and it makes sense for these people to be present briefly during the welcome. It should always be communicated openly and honestly who is watching and why.
• ...., we must brief them well on how to behave before, during and after the test. It is important that there are no interruptions due to interposed questions or comments or, as mentioned above, unconscious gestures, facial expressions or noises (snorting, yawning, laughing, finger tapping). We like to tell the story of Jared Spool mentioned above, as it impressively demonstrates how quickly things can unintentionally go wrong.
• ...it is helpful to hand out some sticky notes to the observers, for example, to avoid interruptions in the form of questions from the observers. In this way, the feeling of an "urgent" question that needs to be asked immediately can be neutralised. Here, observers have the opportunity to write down their questions or thoughts directly without interrupting the test. They can then hand them over to us moderators at the end of the test. We briefly review them and then pass these questions on to the respondent if necessary. This should also be scheduled in advance. This procedure ensures that the moderators remain the main point of contact for the test person and there is no restlessness or confusion.

Conclusion: Make "test subject hours"!

There is certainly a lot more to say, but these basic points should suffice for now. We can only recommend to all designers, product managers and developers that, like psychology students, they also complete "test subject hours" and often slip into the role of the test subjects themselves and take part in a usability test from this perspective. This gives you a sense of how it all feels and you can use this experience when designing your own test.

Illustration: Question illustrations by Storyset