RIOT

Project Summary: Winter 2018

Project Team

RIOT is an immersive, emotionally responsive, live-action film that positions viewers in the middle of a riot in progress. The film responds to participants’ emotional state using artificial intelligence, altering the video story journey in real-time.

An audience member experiencing the RIOT installation
An audience member experiencing the RIOT installation

The RIOT film installation experience reveals to viewers how they might react in the face of imminent danger. A webcam or camera is used to monitor each viewer’s facial characteristics as they watch the film, and the video narrative responds accordingly.

For example, if the viewer becomes agitated, the character in the film responds defensively or impatiently. The same is true for a number of other emotional responses, like calmness or fear.

Karen working with ThoughtWorks developers Sofia Tania and Angelica Perez
Karen working with ThoughtWorks developers Sofia Tania and Angelica Perez

RIOT provides feedback into the complex cultural issues surrounding split second decisions in times of actual crisis. It places viewers at the center of a provocative story for which there are a number of potential live time scenarios which may triggered by their authentic reaction.

Development of a New Prototype

During her time at the ThoughtWorks Arts Residency, Karen worked with ThoughtWorks developers on a new machine learning-based facial expression detection system, and on enhancing the storyline management and RIOT user experience.

As a result, Karen can add new detectable emotions to the experience, and can add new narrative layers to the RIOT prototype via a custom-built user interface.

Karen at one of her regular 'Open Studios'
Karen at one of her regular 'Open Studios'

The RIOT experience works with a deep neural net toolkit for emotion analysis, created by ThoughtWorks, named EmoPy. The system has been made open source in order to provide free access beyond existing closed-box commercial implementations, both widening access and encouraging debate.

Karen’s previous work with Dr. Hongying Meng of Brunel University, London, was refined and developed as part of the piece. This includes his research and implementation of new facial expression analysis techniques.

RIOT exhibited at SPRING/BREAK as part of a ThoughtWorks Arts exhibition during Armory Week
RIOT exhibited at SPRING/BREAK as part of a ThoughtWorks Arts exhibition

During her residency, Karen hosted regular ‘Open Studios’ events at ThoughtWorks New York. Stakeholders from across industry, academia and the arts were invited to discuss the implications of AI technology, using the residency and artwork as a focal point for critical discussion.

Karen undertook research on human emotional understanding with Emily Balcetis, Associate Professor of Psychology at the Social Perception, Action & Motivation Lab at New York University. This research continues as Karen moves forward with new versions of the RIOT system.

Exhibitions and publications

In the fall of 2017 Karen spoke at TED New York, telling the story of her work from conception through to the ThoughtWorks Arts Residency. Karen goes on to explain the next stages of development for RIOT, which will deal with a greater range of emotions, more narratives and a multiplayer experience.

The RIOT film experience was exhibited at the SPRING/BREAK art fair as part of New York’s Armory Week, alongside other ThoughtWorks Arts-incubated artists. It has been exhibited at the Future of Storytelling Festival, the Museum of Modern Art Peru, and Festival of the Mind Sheffield.

The project has also been featured in global publications including Forbes, CBS, The Guardian, Fast Company, NBC, Engadget and the New York Times.

Project Team