SERA Field Study¶
The three iterations of the field study tested different conditions (functionalities) of the equipment which consists of a computer, sensors and a simple robotic device (the Nabaztag) as the front-end for conversational interaction. The scenario of the field study is taken from the general themes of companion technologies and lifestyle reassurance, i.e. health- and fitness related applications. The sample were six subjects out of the target audience for companion technologies of whom three participated in all three iterations. The companion was installed in the subjects’ homes and left there for about ten days in each iteration. The project collected roughly 300 video sequences of interactions in the field study.
Creer et al. 2011: Design decisions for an interactive domestic robot: describing the SERA project set-up This is a preprint of an article submitted for consideration in the Journal Applied Artificial Intelligence 2011 ©Taylor&Francis.
Wallis 2011: From Data to Design This is a preprint of an article submitted for consideration in the Journal Applied Artificial Intelligence 2011 ©Taylor&Francis.
The scenario¶
The SERA field study used a robotic companion that gave advice and encouragement to older people about exercise. The companion had access to the user’s diary of events and the time of day via the system clock. Given this information, the system was aware of what the user might be doing when the system wanted to initiate a conversation. Given that much of the participant’s exercise would take place out of the house (going for a walk, going out with the grandchildren, or going to a tea dance) a useful piece of information is whether or not the user had picked up the house keys. The topic of encouraging exercise provides scope for not only practical use but also the complex issue of the companion’s social status (e.g. persuasion, encouragement, criticism). The idea behind this application area is the study of the use of companions to support patients in rehabilitation after a heart-attack. Therefore, we used concepts and text from a British Heart Foundation pamphlet entitled (at the time) “The Heart Failure Plan: a self-help guide for people with heart-failure, and for their family and friends”. The core idea of this brochure is to encourage, maintain, and increase physical activity while at the same time avoiding the “overactivity-rest-cycle”. The brochure claims that people tend to do too much one day, and then have to rest the next day. This leads to a general downward trend toward being less fit, and so the patient benefits more from taking a smaller amount of exercise on a regular basis. A companion could help by keeping track of exercise and reminding the participant not to do too much.
How the field study was done¶
Initial contact was made between potential participants and the researchers (through the organisation Sheffield 50+). After answering any questions that they had and confirming that they were happy to continue with the research, a visit was organised. One researcher went to the participant’s home and explained the project in more detail, showing pictures of the robot set up and discussing the project and procedure based around the information sheet provided.
If the participant had further questions or required further clarification, these issues were addressed before they were asked to sign the consent form. During the visit, other information was gathered including finding potential sites for the robot set up, taking into account the preferences of the users and practical and privacy issues such as positioning of electrical sockets, what could potentially be videoed by the camera and what position would be walked past frequently without being too intrusive for the participants. The initial site suggested was in the hallway of the house to allow reasonable function of the setup as storage for the participants’ house keys and would provide a more restricted setting for the potential use of speech recognition as input. The participant was asked about access issues that they might have in terms of hearing or visual impairment to make sure that the set up was adjusted to suit their individual needs.
The system as installed in one of the participant’s homes. Components from top to bottom: array microphone, webcam, Nabaztag, key hook, PIR motion sensor, PC (in the pedestal).
At this point, a suitable date was found to install the robot set up, allowing 10 days from a Friday to a Monday. The aim was to capture multiple weekends as it was felt that this would provide more data if the individuals were working, however, it was flexible to ensure convenience for the participants. If the agreed dates were relatively soon after the meeting the participant was asked to fill in an activity plan for that time period. If the dates were agreed and participants wanted to fill in their activities closer to that time, then an empty activity plan was left with them to either email or post back to the researchers a week before installation. Once received, the activity plan was input into the system ready to be installed. On installation, which involved two researchers, an appropriate site for the set up had to be found. Once the participants had seen the physical set up, they were able to make a more informed decision about what would be seen on the video, to maximise privacy for them and any other people living in the house or visitors, and where they would feel happiest to locate the system. Although the researchers provided ideas such as being near to the most used door in the house and somewhere where it would be frequently passed but not too intrusive, the priority was on placing the set up where the participant felt most comfortable having it. The use of the system was demonstrated to the participants and a set of instructions was left with them to look through at any time (the same information sheet mentioned above).
Once described in terms of what could be done, the researchers demonstrated the system, in particular reiterating the information that they were in control of the video recording (if they did not press the video button then the data would not be stored). For the later trials, the demonstrations were more involved highlighting the use of the system as a log to store their activity if they wished to do so. It was restated that the researchers did not want to record the amounts of activity they did but that it aimed to be useful to the participant to feed back information to them. The participants were told that the rabbit would initiate conversations with them in the morning and the evening and when they took their key off the key hook. They were also told that they could initiate their own conversations and instructions were given on how they could do that. They were informed that there might be messages coming through although they were not told where the messages would come from. In some situations the participants did not want to store their house keys on the key hook switch for security or other reasons. Participants were therefore made aware of a choice between using a set of keys or a small rabbit weight or umbrella which could function as a signal to the system that they were going out of the house or coming back. At every stage, the researchers made it clear that the participants were in control and could use the system in any way they wanted. The participants were encouraged to try out the system with the researchers still there. In some cases (e.g. participant 1 in the second iteration), the participant did not want to try using the system with the researchers present and therefore was not pushed into doing so. The robot set up was programmed to contain some information on usage, including information about how to use the tagged cards and placement of the tag for best reception from the tag reader. The system was introduced as a system that would try and help them to maintain their healthy lifestyle. The participants were warned that as part of the procedure they would be asked about weighing themselves and that the researchers were not interested in finding out their weight but if they wanted to write it down, they could do that to keep that information for themselves. Before starting the trial the participants were all asked whether they had access to weighing scales in their home. A notebook was left with the participants which they could either use to store their weight recordings (which was not used by the participants) or they could use it to provide comments on the system or write down any frustrations or experiences that they wanted to remember to pass on to the researchers. This information was aimed to be used during the interviews to remind the participants of their experiences. The robot itself was generally introduced as “the rabbit” although this was not strictly enforced by the researchers, the lead was generally taken by the participant to refer to it as “it”, “he” or “she”, and in one case the name “Harvey”. This was followed by the researchers, although this again was not strictly or explicitly set as part of the original protocol. There were situations where the researchers used a particular pronoun to talk about the system and were corrected by the participant to use the one that fitted their perception of the system. After installation the participants were left with the robot set up installed and told to contact the researchers in case of any problems or questions. After 3 days (usually the Monday following the Friday of installation) the participant was contacted either by telephone or email depending on their preferred mode of contact to see if there were any problems or anything that they wanted to find out about and whether they were happy to continue. If they thought there was a problem, a diagnosis was attempted over the phone and a follow up visit organised if required. If there were no technical problems, the participants were left to continue with the system in their homes without any contact from the researchers for the trial period unless the participants initiated contact. After the trial period, the robot set up was collected by two researchers and follow-up interview appointments were made.
The set-up¶
The hardware consists of a wooden column which contains, hidden at the base, a desktop computer with a broad-band mobile internet connection. The periphery consists of the Nabaztag (a rabbit-like WIFI device marketed as a slightly animated talking Internet interface, see http://www.nabaztag.com/), a passive infrared (PIR) motion detector, a micro-switch on a hook for the house keys and one to three further physical buttons, a webcam (for data collection) and an array microphone.
It was planned to use a speech interface as one of the input modalities, but speech recognition in such conditions turned out to be too problematic for reliable deployment and data collection. Research on speech recognition “in the wild” has been continued throughout the project, while the field study had to rely on other input methods. For iteration 1, physical yes/no buttons were used as input device.
In iteration 2, the Nabaztag’s built-in RFID-reader (Radio Frequency Identification) was used to allow for a broader range of input and more varied interaction. The companion communicated via text-to-speech output, with its LED lights and moving ears as additional means of expression. The method of interaction used was RFID-tagged cards that the Nabaztag can read. The tags were attached to cards with a set of words and symbols on them which represent the different topics of which the robot had some knowledge and interaction items that would allow the user to input appropriate responses to the activity-related conversations. Four groups of cards were prepared:
- “Smiley” icons (smiling, neutral, frowning) for “yes/no/don’t know” as well as for “good/bad/in-between” and similar possible input.
- Numbers from 1 to 5 for rating one’s own well-being.
- Numbers 10, 20 .... 60 for minutes of exercise to add to an activity log.
- Topic cards for user-initiated interactions: requesting the weather report, adding to the activity log, or retrieving the companion’s self-presentation.
Observations in the first two iterations led to the addition of a “shut-up card” with which the user could interrupt the companion’s planned dialog at any point, e.g. to change the topic.
In the pre-study interview, the participants provided their agenda of planned activities for the period of the data collection. When the participant picks up his or her keys, the robot can use the diary to say relevant output, for example, “Have a nice time at the tea dance” or to remind the subject of later appointments, for example, “Going for your walk - don’t forget that Peter is coming at three this afternoon”. The system also needs to be able to say something relevant when the diary does not show any activity, for example, “You’re going out? Have a good time”. If the event counts as exercise, then the corresponding amount of exercise is recorded. When the subject returns home, the robot will ask if he or she had fun (a social politeness) and how he or she is feeling. This information is then used to talk about exercise at the next opportunity. When the user appears in the morning, the companion starts the first dialog of the day. This includes giving a short weather forecast and the question whether the user intends to weigh herself that day. The users can input durations of exercise in minutes that they have done in a day, switch between topics of conversation and rate how he or she was feeling after a day’s exercise.
How data were collected¶
The setup contained a camera which was, in principle, always on, but did not store the recordings. Each time an interaction was started (either by the companion or by the participant) the dialog started with a request to express consent with the recording by pressing the video button. When participants agreed to being recorded, data from that moment minus 30 (?) seconds were stored to the local hard-disk of the PC built into the setup, and the companion announced that the recording was on. The recording (storing) stopped automatically after 60 seconds of inactivity by either companion or participant. We collected approximately 300 video sequences (with audio) of 1 minute length on average. Pre-test questionnaires and post-test semi-structured interviews completed the data collection. For iterations 2 and 3, the log-files of the interactions provided additional information about the participants’ use of the RFID-cards. The six interviews, in length between 30 minutes and over 1 hour, were transcribed and analysed qualitatively. The videos were (partly) transcribed, annotated and analysed using both quantitative and qualitative methods.