Hidden messages

As the solder was led into the television studios he seemed to find the lights blindingly bright. He blinked repeatedly whilst answering questions from his captors, apparently trying to acclimatise to the new environment. The studio, fresh clothes and polite questions were a far cry from the prisoner-of-war cell that he endured for 8 years. It was supposed to be a humiliating event for the US military- a jet pilot shot down, captured and now paraded on international television and compliant with the enemy. But wait- there was something else. As he answered questions about American military policy, were his blinks really due to the bright studio lights? For viewers in the military there was a stark message being transmitted by Commander Jeremiah Denton.

"T-O-R-T-U-R-E"

This was the message being transmitted in Morse code using blinks to the US military. A confirmation that against international law, the prisoners of war were being mistreated. It was a most unusual example of eye interaction and non-verbal communication. The story of the years of abuse endured by Denton and his colleagues has been told in print and in Denton's memoir When Hell is in Season. Another, poignant example of eye interaction is from The Diving Bell and the Butterfly, as story of Elle editor Jean-Dominique Bauby who suffered a stroke that left him with locked in syndrome- only able to communicate with his left eyelid. Non-verbal communication using eye blinks has become an important method of interaction. Patients with quadraplegia, or ALS and a range of disabilities can use computers enabled with eye trackers. 

Eye interaction his the mainstream

Last week Emteq HQ was excited to have finally received our Fove VR headset with integrated eye tracking. There has been considerable interest in eye tracking recently with a number of companies joining the market leaders Tobii and SMI. Eyefluence was recently purchased by Google presumable as part of their plans for augmented and virtual reality. More recently, Israeli startup The Eye Tribe has been acquired by Facebook-owned Oculus.

Emteq can't wait to integrate facial tracking into the Fove

Emteq can't wait to integrate facial tracking into the Fove

By coincidence we also received our research eye tracking glasses from Pupil Labs that will help in our augmented reality product.

 

Project manager Simon modelling the Pupil Labs eye tracker

Project manager Simon modelling the Pupil Labs eye tracker

As an early backer on Kickstarter of Fove, I've waited with baited breath to see how the system performs. One of the things that the company promotes as a potential benefit is the potential for improving human to virtual human interaction

 

Indeed we all know that in face to face interactions, most time is spent looking at the central triangle bounded by the eyes and mouth (see image on the left). The problem with VR is that the most emotionally salient area (around the eyes) is covered by the headset.

Eye tracking (left) shows that the most important area we pay attention to during face to face interaction is the same area largely covered by the HMD (right)

Eye tracking (left) shows that the most important area we pay attention to during face to face interaction is the same area largely covered by the HMD (right)

The second issue is that whilst eye tracking tells us that a person is looking at something, it does not tell us why. Without an assessment of the emotional response together with measuring attention, it becomes difficult to understand the user's motivation or intentions. For example, if a spider walked across your floor you would likely pay attention, but it would be difficult to understand if that was because you were a lover of spiders or arachnophobic. However, someone looking at you would know immediately by looking at your face. Nevertheless, eye-tracking has been very useful for a range of research avenues (for those interested in a deep dive, I've summarised the research below). Adding simultaneous assessment of emotional responses using Emteq's facial expression platform (called FaceTeq) will add further insights- the WHAT and the WHY. We're excited to have had a further order of our system from a leading U.K. university and can't wait to bring emotional connections to the digital world.

 

Eye tracking research overview

Eye tracking has been employed in a range of research studies, from mental health conditions, marketing, product testing, neuropsychology, social psychology and linguistics.Simultaneous measurement of emotional responses complements many of these research areas. As virtual reality is a useful research tool, it's inevitable that there will be a growing need to measure both eye movements and facial expressions....

-      Examination of visual exploration strategies

a.       Patients with brain damage

i.            Visual scanning and matching dysfunction in brain-damaged patients with drawing impairment - Belleza T, Rappaport M, Hopkins HK, Hall K

b.      Patients with neurological diseases

ii.           A new method of visual exploration analysis - Müri RM1, Pflugshaupt T, Nyffeler T, von Wartburg R, Wurtz P.

-      Understanding the control mechanisms of eye movements

a.       In Autism

iii.         Cognitive Mechanisms Underlying Action Prediction in Children and Adults with Autism Spectrum Condition - Schuwerk T, Sodian, Paulus M

b.      In Schizophrenia

iv.         Enhanced top-down control during pursuit eye tracking in schizophrenia - Sprenger A, Trillenberg P, Nagel M, Sweeney JA, Lencer R.

c.       In Alzheimer’s

v.          Visual exploration of facial emotion by healthy older adults and patients with Alzheimer disease - Ogrocki PK, Hills AC, Strauss ME

d.      In Social personality

vi.         Look into my eyes: Investigating joint attention using interactive eye-tracking and fMRI in a developmental sample - Oberwelland E, Schilbach L, Barisic I, Krall SC, Vogeley K, Fink GR, Herpertz-Dahlmann B, Konrad K, Schulte-Rüther M

-      Eye tracking to provide early indicators for diseases

a.       Autism

vii.       Autism spectrum disorder. Contemporary experimental researches review - Luschekina EA, Strelets VB

b.      Schizophrenia

viii.      Vulnerability, Stress, and Support in the Disease Trajectory from Prodrome to Diagnosed Schizophrenia: Diathesis-Stress-Support Model - Cheng SC, Walsh E, Schepp KG

c.       Social personality

ix.         Social anxiety disorder: a critical overview of neurocognitive research - Cremers HR, Roelofs K,

Eye tracking in marketing

-      Brand/product choice

x.          Ideology and brand consumption - Khan R, Misra K, Singh V.

-      Pre-testing and analysing online marketing, print ads, out-of-home media, TV commercials and sports marketing

xi.         Eye movements when viewing advertisements - Emily Higgins, Mallorie Leinenger, and Keith Rayner

-      Evaluation of shopper behaviour

xii.       Narcissism and consumer behaviour: a review and preliminary findings - Sylwia Z. Cisek, Constantine Sedikides, Claire M. Hart, Hayward J. Godwin, Valerie Benson, and Simon P. Liversedge

Eye tracking in Human Computer Interaction

-      Usability of websites, software, apps, mobile devices

xiii.      Human-Computer Interaction in Smart Environments - Gianluca Paravati and Valentina Gatteschi

-      Product testing

xiv.     Improving the human-computer interface: a human factors engineering approach - Salvemini AV

-      Human perception of virtual environments and computer games

xv.       The effect of user's perceived presence and promotion focus on usability for interacting in virtual environments - Sun HM, Li SP, Zhu YQ, Hsiao B    

-      Study cognitive activities in computer supported learning environments

xvi.     Patterns of Interaction in Computer-Supported Learning: A Social Network Analysis - Tuire Palonen & Kai Hakkarainen

-      Understand decision making process in “think aloud” experiments

xvii.    A Think Aloud Study Comparing the Validity and Acceptability of Discrete Choice and Best Worst Scaling Methods - Jennifer A. Whitty, Ruth Walker, Xanthe Golenko and Julie Ratcliffe

-      Identification of which visual elements lead to confusion

xviii.  Brain-Computer Interface Based on Generation of Visual Images - Pavel Bobrov, Alexander Frolov, Charles Cantor, Irina Fedulova, Mikhail Bakhnyan and Alexander Zhavoronkov

Neuropsychology

-      Visual processing

xix.     Visual processing and neuropsychological function in schizophrenia and schizoaffective disorder - Brenner CA, Lysaker PH, Wilt MA, O'Donnell BF

-      Interaction between eye movements, vision and performance

xx.       Neuropsychological correlates of eye movement abnormalities in schizophrenic patients and their unaffected relatives - Zanelli J, MacCabe J, Toulopoulou T, Walshe M, McDonald C, Murray R

-      Object by object search mechanisms in change blindness studies

xxi.     Change blindness, aging, and cognition - Matthew Rizzo, JonDavid Sparks, Sean McEvoy, Sarah Viamonte, Ida Kellison, and Shaun P. Vecera

-      Mechanisms of visual neglect

xxii.    The neurological basis of visual neglect - Milner AD, McIntosh RD

-      Conditions under which external stimuli capture attention

xxiii.  The leading sense: supramodal control of neurophysiological context by attention - P. Lakatos, M.N. O’Connell, A. Barczak, A. Mills, D.C. Javitt, and C.E. Schroeder

-      Perceptual decision making

xxiv.  How mechanisms of perceptual decision-making affect the psychometric function - Gold JI, Ding L

Social Psychology

-      People pay more attention to attractive faces

xxv.   Social Psychological Face Perception: Why Appearance Matters - Leslie A. Zebrowitz and Joann M. Montepare

-      See how decisive people actually are

xxvi.  The influence of emotion regulation on social interactive decision-making - Mascha van ’t Wout, Luke J. Chang and Alan G. Sanfey

-      Scene perception

xxvii.Eye movements and scene perception - Rayner K, Pollatsek A.

-      Perception of art

xxviii.                       Art in the eye of the beholder: the perception of art during monocular viewing - Finney GR, Heilman KM.

-      Perception of film

xxix.  Looking at movies and cartoons: eye-tracking evidence from Williams syndrome and autism - Riby D, Hancock PJ.

-      Perception of the natural environment

xxx.   Eye movements of patients with schizophrenia in a natural environment - Dowiasch S, Backasch B, Einhäuser W, Leube D, Kircher T, Bremmer F

-      People use their gaze to indicate which direction they will travel

xxxi.  Influence of visual path information on human heading perception during rotation - Li L1, Chen J, Peng X. - Li L1, Chen J, Peng X.

Media Psychology

-      Psychological impact of different media formats and messages

xxxii.Perception of visual advertising in different media: from attention to distraction, persuasion, preference and memory - Jaana Simola, Jukka Hyönä and Jarmo Kuisma

-      Human perception of print ads, commercials, shopping shelves, outdoor ads, web pages

xxxiii.                       Eye movements when viewing advertisements - Emily Higgins, Mallorie Leinenger, and Keith Rayner

-      Find out how and which visual elements get and keep attention

xxxiv.                      The development of perceived structure and attention: evidence from divided and selective attention tasks - Shepp BE, Barrett SE

-      Determine impact of placement on ability to cut through cluttered scenes

xxxv.                        Visual crowding: a fundamental limit on conscious perception and object recognition - Whitney D, Levi DM

Industrial Design

-      Optimize design of control panels, prototypes, products, packaging and labelling

xxxvi.                      Evaluate package design performance using eye tracking

Cognitive Psychology

-      Study emotion processing/extent to which emotion processing is conscious

xxxvii.                     Eye movement-related brain activity during perceptual and cognitive processing - Andrey R. Nikolaev, Sebastian Pannasch, Junji Ito and Artem V. Belopolsky

-      Impact of depression/anxiety/other mental health issues

xxxviii.                   Mental health and cognitive function in adults aged 18 to 92 years - Bunce D, Tzur M, Ramchurn A, Gain F, Bond FW

-      Interplay between visual perception and reading and tracking task performance

xxxix.                      Interactions between working memory and visual perception: An ERP/EEG study - Yigal Agam and Robert Sekuler

-      Identify the cause of poor reading skills (eye movements related to comprehension)

xl.         Eye Movements during Silent and Oral Reading in a Regular Orthography: Basic Characteristics and Correlations with Childhood Cognitive Abilities and Adolescent Reading Skills - Krieber M, Bartl-Pokorny KD, Pokorny FB, Zhang D, Landerl K, Körner C, Pernkopf F, Pock T, Einspieler C, Marschik PB

-      Study cognitive processes selectively concentrating on one aspect while inhibiting others

xli.       The role of selective attention on academic foundations: A cognitive neuroscience perspective - Courtney Stevens and Daphne Bavelier

-      Examine human factors in human machine or computer interactions

xlii.      From human-machine interaction to human-machine cooperation - Hoc JM

-      Provide insight into the relationship between attention and performance when skills/abilities are essential e.g. driving, flying

xliii.    The relationship between visual attention and visual working memory encoding: A dissociation between covert and overt orienting - Tas AC, Luck SJ, Hollingworth A

Developmental Psychology

-      Understand progression in children’s allocation of attention and interest

xliv.    Play, attention, and learning: How do play and timing shape the development of attention and influence classroom learning? - James H Hedges, Karen E Adolph, Dima Amso, Daphne Bavelier, Julie A Fiez, Leah Krubitzer, J Devin McAuley, Nora S Newcombe, Susan M Fitzpatrick and Jamshid Ghajar

-      Measure visual perception related to understanding and recall

xlv.     Interactions between working memory and visual perception: An ERP/EEG study - Yigal Agam and Robert Sekuler

-      Measure infants’ ability to recognize motion signals predicting reappearance of objects

xlvi.    How Infants Learn About the Visual World - Scott P. Johnson

-      Investigate relationship between control of eye movements and reading comprehension

xlvii.  The Relation between Reading Skills and Eye Movement Patterns in Adolescent Readers: Evidence from a Regular Orthography - Magdalena Krieber, Katrin D. Bartl-Pokorny, Florian B. Pokorny, Christa Einspieler, Andrea Langmann, Christof Körner, Terje Falck-Ytter, and Peter B. Marschik

-      Development of control of action

xlviii. Evolution, development and intentional control of imitation - Cecilia Heyes

-      Social interaction characteristics -  how children understand what other people feel, how do they understand the actions and intentions of others

xlix.    School and family effects on the ontogeny of children's interests, self-perceptions, and activity choices - Eccles JS

-      Study development of oculomotor functions

l.            Development of eye-movement control - Beatriz Luna, Katerina Velanova, and Charles F. Geier

-      Study language acquisition

li.           Eye-Tracking Measurements of Language Processing: Developmental Differences in Children at High Risk for ASD - Meia Chita-Tegmark, corresponding author Sudha Arunachalam, Charles A. Nelson and Helen Tager-Flusberg

-      Study non-human primates

lii.         Eye-tracking with nonhuman primates is now more accessible than ever before - Machado CJ, Nelson EE

-      Communicate with subjects who cannot communicate verbally

liii.        Eye tracking research to answer questions about augmentative and alternative communication assessment and intervention - Wilkinson KM, Mitchell T

Criminology

-      Role of expertise in deception detection

liv.       Eye Tracking and Pupillometry are Indicators of Dissociable Latent Decision Processes - James F. Cavanagh, Thomas V. Wiecki, Angad Kochar, and Michael J. Frank

-      Eyewitness decision processes in criminal lineup identification

lv.         Predictors of eyewitness identification decisions from video lineups in England: a field study - Horry R, Memon A, Wright DB, Milne R.

Psycholinguistics

-      Reading

lvi.       Opening a Window into Reading Development: Eye Movements’ Role Within a Broader Literacy Research Framework - Brett Miller, PhD and Carol O’Donnell, EdD

-      Help develop efficient learning programs for dyslexic children

lvii.      Developmental Trajectories for Children With Dyslexia and Low IQ Poor Readers - Sarah E. A. Kuppencorresponding author, and Usha Goswamicorresponding

-      Understanding spoken language – what children look at when listening

lviii.    Individual Differences in the Real-Time Comprehension of Children with ASD - Courtney E. Venker, Elizabeth R. Eernisse, Jenny R. Saffran, and Susan Ellis Weismer

-      Cognitive processes related to spoken language

lix.       Speech perception as an active cognitive process - Shannon L. M. Heald and Howard C. Nusbaum

-      Ability to interpret metaphor

lx.         Comprehending spoken metaphoric reference: a real-time analysis - Stewart MT, Heredia RR

-      Body language and lip reading

lxi.       Seeing to hear? Patterns of gaze to speaking faces in children with autism spectrum disorders - Julia R. Irwin and Lawrence Brancazio

-      Turn taking in conversations

lxii.      Influence of Turn-Taking in a Two-Person Conversation on the Gaze of a Viewer - Lotta Hirvenkari, Johanna Ruusuvuori, Veli-Matti Saarinen, Maari Kivioja, Anssi Peräkylä, and Riitta Hari

-      Audio-visual integration

lxiii.    Using eye-tracking to study audio-visual perceptual integration - Xiao M, Wong M, Umali M, Pomplun M.

-      Reading behavior

lxiv.    Examining the online reading behavior and performance of fifth-graders: evidence from eye-movement data - Yao-Ting Sung, Ming-Da Wu, Chun-Kuang Chen, and Kuo-En Chang

-      Scene exploration strategies

lxv.     Perceptual effects of scene context on object identification - De Graef P, Christiaens D, d'Ydewalle G.

Experimental Psychology

-      Scene analysis and visual perception

lxvi.    Eye Movements and Visual Encoding During Scene Perception - Keith Rayner, Tim J. Smith, George L. Malcolm, and John M. Henderson

-      Reaction timing experiments using eye movement responses

lxvii.  The Effects of Incidentally Learned Temporal and Spatial Predictability on Response Times and Visual Fixations during Target Detection and Discrimination - Melissa R. Beck, S. Lee Hong, Amanda E. van Lamsweerde, and Justin M. Ericson

-      Relationship between learning and performance by exploration

lxviii. An exploration of the relationship between knowledge and performance-related variables in high-fidelity simulation: designing instruction that promotes expertise in practice - Hauber RP, Cormier E, Whyte J 4th

-      How visual information is perceived, stored and recalled in memory

lxix.    Visual long-term memory has a massive storage capacity for object details - Timothy F. Brady, Talia Konkle, George A. Alvarez, and Aude Oliva

-      Face to face communication

lxx.     A systematic review and meta-analysis of face-to-face communication of tailored health messages: implications for practice - Wanyonyi KL, Themessl-Huber M, Humphris G, Freeman R.

-      Differences in working memory between healthy and damaged brains

lxxi.    Intact working memory for relational information after medial temporal lobe damage - Jeneson A, Mauldin KN, Squire LR

-      Understand how selections are made between task relevant and task irrelevant information

lxxii.  Configural representations in spatial working memory: modulation by perceptual segregation and voluntary attention - Gmeindl L, Nelson JK, Wiggin T, Reuter-Lorenz PA

-      Determine if selections are made based on spatial representation or perceptual objects

lxxiii. Configural representations in spatial working memory: modulation by perceptual segregation and voluntary attention - Gmeindl L, Nelson JK, Wiggin T, Reuter-Lorenz PA

-      Understand visual attention mechanisms

lxxiv.Mechanisms of visual attention in the human cortex - Kastner S, Ungerleider LG

-      Create gaze contingent paradigms

lxxv.  It’s in your eyes—using gaze-contingent stimuli to create truly interactive paradigms for social cognitive and affective neuroscience - Marcus Wilms,Leonhard Schilbach, Ulrich Pfeiffer, Gary Bente, Gereon R. Fink and Kai Vogeley