Exploiting Recognition of Natural Visual Stimuli for Brain Computer Interfacing

Visual search is a good illustration about how the brain coordinates a variety of functions such as visual clues extraction from the scene, coordination of the eye-movements, accumu- lation of visual information and visual recognition. It has an important role in many daily life activities of people – the search of a familiar face in a crowd, a book in the library, or a building in a street. In all these scenarios, it represents a recognition of a specific target object in the presence of predominant task-irrelevant visual information. This thesis specifically addresses the visual search in context of brain-computer interaction (BCI) where the accent is on visually rich stimulation, approaching the real-world BCI applications. Different cognitive aspects of visual processing have been extensively studied using the Rapid Serial Visual Presentation (RSVP) protocol – a sequentially presentation of visual stimuli at the same location on a display. The main part of this thesis deals with the visual search under the RSVP protocol. In such a protocol, the electroencephalogram (EEG) reflects whether a subject recognize an image as a target. Our focus is on extracting and exploiting this informa- tion for the BCI application of image retrieval. We designed a framework for EEG-based image search which involves coupling of the EEG decoder with the computer vision technique for image retrieval. These two components of the system are seen as complementary – bringing in the semantic information and large scale computing, respectively. Our design is founded on an iterative coupling, which relaxes a performance requirement of the individual components. We report the results of the experimental evaluation, where natural color images were used as stimuli. This thesis also includes a comprehensive analysis of the EEG correlates of visual recogni- tion in the EEG-based image search task, addressing the effects of the less controllable stimuli to the EEG potentials’ attributes. To this end, our focus is on the spatio-temporal aspects of the EEG potentials, along with the modulation of the EEG-based cortical connectivity estimates. Finally, this thesis presents an exploratory research on active visual search in the context of BCI. An essential part of active searching of the visual scene represents eye movements. Hence, this type of task better reflects our natural behavior and as such extends the possibilities of BCI applications. We designed two experimental protocols to study the active visual search – guided and free-view visual search. We report the results of the combined study of eye tracking data and simultaneously recorded EEG.