Modeling memory through Interaction

Recent works have shown that the gaze reveals user preferences, and can therefore be used as input for recommender systems and intelligent assistants. The AMI project, co-funded by the European Regional Development Fund (ERDF), seeks to investigate whether our gaze is also a good indicator of our memory capacities, so as to improve interactions with users of online services. In particular, this work will allow us to diagnose early neurodegenerative diseases (Alzheimer's, autism, ...) based on the analysis of the look and the various traces of interaction of the subjects, and to propose a personalized assistant for the Human learning and the cognitive rehabilitation.

Co-funded by ERDF

In addition to this AMI project, we developed several tools related to this research field:

A.M.E.: this software aims at analysing usages (movements, gaze data, clicks, timestamps, …) while passing neuropsychological tests such as TMT, and to automatically detect cognitive and memory disorders. It is compatible with Leap Motion and Tobii X1 Light Eye-Tracker.

Precog: this software has been designed to verify the existence of a link between memory and gaze data. On one hand, we are starting a collaboration with the Nancy University Hospital to study memory troubles among elderly persons, with the goal of automatically diagnosing memory issues based on fixation points, saccades and scanpaths. On the other hand, we are working on predictive models to provide recommendations of educational resources in Intelligent Tutoring Systems, according to what the user looked at and what the system assumes to be memorized. Concretely, Precog allows users to play to the well-known game called “Concentration” (also known as Memory, Pelmanism, Shinkei-suijaku, or Pexeso). In the meantime, the system collects gaze data such as fixation points and durations. Then, it uses these pieces of data to predict which items will be memorized according to primacy and recency effects. Our machine learning algorithm can still be improved, but first results are encouraging. Our application has been developed in C# so as to be compatible with a Tobii X1 Light eye-tracker and Tobii SDK. This prototype has been presented to more than 10,000 persons during the Renaissance Nancy 2013 exhibit. This software is available on demand.