Preparations

The development of this showcase was guided by the requirements that it should be transportable, reproducible, easy to use, versatile, and a conduit for interaction and discussion. These goals are reflected in the components of the showcase described here and in the different constellations that single components can be used in.

Note

For offline perusal of this text and in order to run the interactive samples, you can download a complete copy of this showcase, including all interactive software (however, not including videos due to their size - videos can be downloaded separately directly from the website http://showcase.project-sera.eu/).

Components

The main components of the showcase are

• the present text and all linked and referenced additional media,

• SERASystemDemo: an interactive version of the software used in the experimental setup, disentangled from specific hardware requirements (i.e. using ‘virtual’ stand-ins for hardware components such as the Nabaztag rabbit and its RFID-reader, physical switches and buttons, as well as microphone and speakers). However, the software can:

  • optionally use the Nabaztag hardware if present to demonstrate its actual motor behaviour in the field study as well as the use of the RFID-reader as input modality,
  • optionally use a custom-built bridge to the commercial speech recognition software ‘Dragon Naturally Speaking’ to demonstrate the state and usefulness of commercial off-the-shelf (COTS) automatic speech recognition (ASR) in settings similar to the SERA field study.
  

  GUI of the SERA System Demo.

• SERAScriptEditor: a viewer and editor for the state-machine script format used in the development of the setup for the field study.

  

  GUI of the SERA Script Editor.

• Episodes based on the field study:

  • Discussing exercise activities.
  • Wrapping up the day, evaluating progress.

• SERAFloorDemo: A demonstration of the HSM (hierarchical state machine) based prototype implementation of the SERA reference architecture regarding floor management.

Requirements

The hardware and software requirements for running the most basic version of interactive demos (i.e. without speech recognition or voice output and without the Nabaztag hardware) consist only of a computer with a recent version of Java (>= version 1.6 also known as Java SE 6). This allows the showcase, in its basic variant, to be easily transported and distributed and to operate on all major operating system platforms (Linux, Mac OS and Windows).

More complex requirements apply if the demonstrations should interact with the Nabaztag hardware or for the demonstration of ASR capabilities.

Nabaztag

The required hardware consists of:

1. One computer (e.g. a laptop) with wireless network hardware (WLAN)
2. One Nabaztag, see http://www.nabaztag.com/
3. A set of RFID tags for the Nabaztag (marketed as “ztamp:s”)
4. (Optional, but recommended) one WLAN Access Point

The alternative to a dedicated WLAN Access Point is to configure the computer as an access point for the Nabaztag hardware, see below.

ASR and voice output

To use automatic speech recognition and voice output additionally requires:

1. One headset or microphone
2. Speakers (on the computer or using the headset)
3. The software Dragon Naturally Speaking for PC by Nuance, see http://www.nuance.com/for-individuals/by-product/dragon-for-pc/index.htm (the version tested was “Premium Edition English version 11.00.200.049”)

Note that SERAFloorDemo provides voice output independently of the Dragon software and does not use ASR.

Starting Demos

This section describes the steps for setting up the hardware and software for the showcase. To run the basic variant of the demonstrations, no special setup is required. The programs can be found as jar files in the programs sub-directory of the showcase. On most platforms, the demos can be started simply by double-clicking either one of the provided startup files (.bat files for Windows and .sh shell scripts for Linux/Mac OS X) or the jar files. One caveat applies to SERAFloorDemo which uses native libraries: the appropriate startup file for your operating system (platform and architecture type, e.g. win32) has to be used. The startup files also open a console window that allows you to have a look at the debug output of the software. You can also start the demos from the commandline using the following command:

java -jar <program>.jar

replacing <program> with the program’s name, e.g. SERAScriptEditor. See the provided startup-scripts for the more involved invocations for SERAFloorDemo and the showcase episodes.

Setup of the Nabaztag hardware

If correctly configured, the SERASystemDemo, the showcase episodes and any script-sets started using the SERAScriptEditor will automatically use available Nabaztag hardware. In order for this to work the Nabaztag needs to be configured to connect to the server running as part of the demo software and the configuration file of the software itself needs to contain the same IP address that the Nabaztag is connecting to. Here is a short checklist for this setup, useful if you are already familiar with the Nabaztag setup (see below for a more extensive description):

1. Determine the IP-address under which the computers WLAN-hardware will be reachable.
2. Configure the Nabaztag to connect to the same wireless network as the computer. (This can also be the computer itself if it is acting as an access point.)
3. Configure the Nabaztag to connect to port 8765 on this ip-address, i.e. change the setting for Violet Platform at the very bottom of the configuration webpage from the default r.nabaztag.com/vl to <your-ip>:8765/vl replacing <your-ip> with the IP-address determined previously.
4. Change the first setting (jnabserverip) in the configuration files showcase.cfg and editor.cfg in the programs directory to the same IP-address.

The following is a more extensive description of the same four setup steps:

1. Determine the IP-address under which the computers WLAN-hardware will be reachable:

   If you use a separate WLAN-router, this is the IP-address that the WLAN-hardware of your computer gets assigned by this router. How you determine this information depends on your operating system. On Linux and Mac OS X, it is part of the output of the command ifconfig. On Mac OS X, it is also available in the AirPort section of the Network Preferences. On Windows, the comman ipconfig /all does provide this information as well as the status display of the connection settings.

   If you do not have a dedicated WLAN-router, you can configure your computer as a WLAN access point, sometimes known as “sharing its connection”. Please note that in this case the IP-address to be used in the following steps needs to be the IP-address of the WLAN-hardware while it is acting as access point and that this often differs from the IP-address visible while connected to another WLAN, for example the one created by the Nabaztag during its configuration! Setting up a computer as access point differs considerably among operating systems, so we cannot cover all the different options here. As an example, here is an outline of the steps required on Mac OS X:

   • Open System Preferences -> Sharing
   • Note the computer name - this will be the SSID for the Nabaztag configuration.
   • Select (but do not yet activate) Internet Sharing
   • Select Share your connection from Ethernet to computers using AirPort
   • In AirPort Options... select the channel automatic and do not enable encryption. (Encryption would be more secure but does not work in this setup.)
   • Now do enable Internet Sharing (i.e. check the checkbox)
   • Now determine the IP address of your WLAN card, e.g. using ifconfig (normally the first inet ip in the section en1), usually: 10.0.2.1
   • Disable Internet Sharing again in order to perform the following steps. You will need to enable it again after all steps are complete, before you start demo programs.

2. Configure the Nabaztag to connect to the same wireless network as the computer:

   This configuration is also described in the Nabaztag quick start guide:

   • Unplug power from your Nabaztag, hold down the button on top of its head, then plug it back in while holding the button. This will activate configuration mode indicated by all lights turning blue (“Putting your rabbit in Blue”).
   • Connect your computer to the now visible wireless network NabaztagXY (where XY correspond to the final characters of the Nabaztag’s serial number).
   • Open http://192.168.0.1/ in a web browser. You will see the following long webpage for entering configuration settings.
   

   The top section of the Nabaztag’s configuration webpage.

   • Scroll down, and enter the information for your wireless network, such as SSID, encryption settings, and whether or not DHCP is active on the WLAN. If you followed the example above to configure your Mac OS X computer to act as an access point, something similar to the following will be required:

     SSID:manual entry of the name of your computer Encryption:no encryption Authentication (WEP):  Open System (no change) DHCP server:NO (connection sharing does by default not provide DHCP) Local IP:your ip +1 (for example: 10.0.2.2) Local Mask:255.255.255.0 DNS:empty (as it’s not needed if IP-addresses are used directly) Proxy:empty

   You might want to wait for the next step before pressing Update and Start.

3. Configure the Nabaztag to connect to port 8765 on this ip-address:

   Finally, and this differs from the normal Nabaztag setup, enter the following (including the IP-address determined above) at the very bottom of the Nabaztag configuration page.

   Violet Platform:  <your-ip>:8765/vl (originally “r.nabaztag.com/vl”)
   

   The bottom section of the Nabaztag’s configuration webpage including the “Violet Platform” field.

   Now press the button Update and Start. The Nabaztag will try to connect to the wireless network and the three bottom lights on its front should turn green. If this is not the case, please refer to the Nabaztag documentation in order to correct your wireless settings.

4. Change the jnabserverip setting in the demo configuration files:

   Both the files showcase.cfg and editor.cfg in the programs directory of the showcase contain a setting called jnabserverip. Enter the IP-address determined above as the value of this setting using any text-editor.

   Note

   If, during its operation, the Nabaztag starts to continuously spin its ears and to react inconsistently, this usually indicates that the server (part of SERASystemDemo or SERAScriptEditor) was started without the correct jnapserverip config option in the .cfg files.

After performing these steps, and when you start SERASystemDemo, all of the Nabaztag’s lights will turn green, it will spin its ears once and the bottom light will start to glow violet, indicating a successful connection. You can now run SERASystemDemo, one of the two episodes or any script-set using the SERAScriptEditor with the Nabaztag hardware.

Configuration of RFID tags

In order to use RFID tags as input modality for the Nabaztag setup, the demo applications need to know about the mapping of RFID tags to user utterances. This mapping is specified in the file configs/allinone.tagset in the programs directory. It consists of a list of assignments mapping the final 5 characters of an RFID code to a user utterance. As several RFID tags can be mapped to the same utterance, all the mappings used during development and deployment of the field study as well as this showcase are already listed in this file. A simple way of identifying the code of an RFID tag in order to add new mappings is to show the tag to the running SERASystemDemo which will result in debug output containing the code of the RFID tag. The demo application needs to be restarted for a changed mapping to be used.

Setup of Speech Server and Voice Output

The software Dragon Naturally Speaking for PC by Nuance requires a Windows operation system and the same applies to the bridging application developed by SERA: SeraDragonApp.exe located in the directory programs/SeraDragonApp/ inside the SERA showcase. Therefore, in order to demonstrate ASR capabilities, the showcase needs access to a Windows-based machine. The easiest way to achieve this is to run the showcase programs as well as Dragon Naturally Speaking and the bridging application on a Windows machine. (An alternative is to run ASR on a windows virtual machine, see the end of this section for notes regarding this setup.)

Installing Dragon Naturally Speaking involves testing of the microphone equipment as well as a training session for building a user profile. In order to achieve acceptable performance of ASR, both decent microphone equipment and a user profile (specific to the one individual that will be operating the showcase) is needed. Training the user profile consists, at the minimum, of several minutes of reading English text out loud.

Once Dragon Naturally Speaking is running and set up, start the bridge application SeraDragonApp. It will open a single Window and activate the microphone in Dragon Naturally Speaking. You can now run SERASystemDemo, one of the two episodes or any script-set using the SERAScriptEditor and it will automatically use Dragon speech recognition as an additional input source, as an alternative for input using RFID cards, as well as voice output.

The SeraDragonApp running alongside the Dragon Bar.

Running ASR in a virtual machine

You can also run only the speech-server and the bridging application in a virtual machine (e.g. using VirtualBox) on the same machine as the rest of the showcase and only expose the port used by the bridging application on the host machine. By default, this port is 19201 but this can be configured in showcase.cfg, editor.cfg and config.xml (for the bridging application) respectively. You will need to configure port forwarding for the VM with a command similar to this one:

VBoxManage modifyvm "WindowsXPOnDisk0s4" --natpf1 "seraspeech,tcp,127.0.0.1,19201,,19201"

(replacing WindowsXPOnDisk0s4 with the name of your virtual machine).