Marco Gillies

Movement and Interaction

Marco Gillies, Andrea Kleinsmith, Nina Kov, Max Worgan, Bruno Zamborlin

Movement is, more often than not, about interaction. In many cases the importance of a movement is not so much in the movement itself but in how that movement interacts with the setting for that movement. The meaning of the non-verbal communication, the bodily signals we send out while interacting with other people, is as much in the relationship between two people’s movements as in the movements in the individual. In dance the interaction between dancers and their relationship to the music is critical. Most of our movements are therefore situated within an environment and are aimed at acting within that environment. We follow a line of argument originating with Heidegger and Merleau-Ponty that our behaviour is embodied  in the sense that’s meaning arises through action and interaction within a world. This argument has recently been picked up by thinkers in interactive technology, such as Dourish, who stress the importance of embodiment in our interaction with technology. This point of view raises a question for digital representation of movement: if the meaning of movement arises out of interaction with a world, how do we capture that interaction.

We will explore the technologies and practices that are required to reproduce this type of interaction in digital form, within a number of application contexts where interaction is critical. We will take three case studies of our work:

• Creating animated characters that are able to engage in non-verbal communication with a real person. We base the behaviour of the character on the movements of an actor. How do we capture not just the movement but how the movement responds to the movement of others?
• Creating gestural interaction systems for music, software that enables performers to interact with music via their movements. How are performers able to record the interaction between their movements and the music?
• Creating interactive installations in which members of the public are encouraged to reproduce the movements of a dancer. How do we use interactive technology to “choreograph” the movements of a member of the public and enable them to experience the movements of a dancer not simply by watching them but by performing the movement themselves.

We will present a technological framework we have developed for this based on interactive machine learning; an exploration of artistic practices using this framework and a theoretical framework based on the philosophy of embodiment.