With interactive e-books it is possible to break free from the one-directional information flow of the printed book and make reading a book a new experience with a seamless interaction with others and the creator much in the same way as ‘online’ gaming has been for computer games. Together with Eaglescience, Noldus and Nijgh&vanDitmar, the COMMIT/ SWELL valorization project VP30, titled “the Quantified Reader” evaluates a Proof-of-concept (PoC) product that enhances the experience for e-book readers by automatically measuring emotions of the reader. These emotions can be used for many things such as input for interactive story telling, or sharing them with other readers and the writer.
The valorization project is part of a larger quantified reading program with additional activities and financing. In this program, the neurophysiology of more than 300 book readers has been measured, with the aim to translate the resulting data into emotions. In a previous phase, the emotions of international bestselling writer Arnon Grunberg have been measured during the e-book writing process.
The overall goal of the program is develop and test a system that can answer questions like: do other readers feel the same emotions in a particular scene? How did the writer feel when writing that scene? The emotion reading and interpretation valorizes and builds upon the sensors and reasoning components for stress and workload that have been used and developed in COMMIT/ SWELL. The automation is an important precondition; the sensors and reasoning algorithms automatically record physical, mental and emotional state and do not distract people from doing their work, in this case reading a book.
The PoC Product collects neuro-physiological data (EEG brain signals from 32 electrodes, breath, skin conductance and ECG / heart beat), data from two Smart Eye camera’s and video. The book text is shown on a tablet, which also asks questions about current emotions after predefined sections. These questions provide for the subjective experience scores, which can later be compared with the neurophysiological results. Before the experiment, twenty sections have been predefined by experts as either “high emotional” or “non-emotional”, which served as a reference. Video is used as input for the Noldus FaceReader only, in order to estimate emotions based on facial expressions.
The PoC product has successfully measured and recorded various neuro-physiological markers from a person reading and linked them to a specific section in the e-book. Analysis of the EEG and physiological data show that it is indeed possible to link these markers to certain emotions. When the data was corrected for time-related effects, models trained and tested on data from individual participants reached around 70% classification accuracy. This suggests that after reading (but not in real time), the PoC product may be used to estimate the emotions that a reader experienced during reading the parts of a novel that are in between the twenty predefined sections.
The PoC product test took place in a special environment. This so-called “Grunberglab” was a featured part of the Arnon Grunberg exposition at the Bijzondere Collecties museum in Amsterdam, in the last quarter of 2014. A special webpage was published to recruit readers for the experiment, but no large promotional campaign was launched. Within two days all 300 available spots were filled, which is a great success. Although part of the attraction was the exhibition of a high-profile writer, the interest in innovative book services was clearly high.
In conclusion it can be stated that the PoC product has proven to be a valuable step in the development of a commercial system based on of- the-shelf consumer electronics (like e-readers or tablets with camera’s,smart watches and affordable EEG headsets), but with unique software and algorithms. Determining emotions is valuable, and the modules that will classify sensor readings are based on the state-of-the-art implementations developed within SWELL. A working prototype of a possible commercial system is being developed in the follow-up COMMIT/ SWELL valorization projects VP35 “Brainwave” and features a 300 euro EEG headset and a 100 euro fitness band with an optical heartrate sensor, Galvanic skin response sensor and temperature sensor.
There has been a lot of interest from the (popular scientific) public media because the project involved Arnon Grunberg and his Grunberglab at the Bijzondere Collecties museum in Amsterdam. Much of the dissemination in the public domain has not only appeared in print but also in the form of a webarticle, a TV or a radio item.