EmotiGlass

By: David Prutchi and Jason Meyers

IMPORTANT NOTE:  A complete, single-document whitepaper for the project is available for download at: https://cdn.hackaday.io/files/1606156871752160/EmotiGlass%20-%20v4.1%20PUBLISHED%20-%20.pdf

All augmented reality devices so far provide an interactive experience of the real-world environment that is enhanced by computer-generated perceptual information.  In contrast, the EmotiGlass project explores ways in which a computer can modulate the user’s EMOTIONAL perception of reality.  Our projects aims to develop the first “Modulated-Emotion Reality” device.  EmotiGlass enables completely new applications in the field of augmented reality in which emotional biases can be manipulated by computer applications. Additionally, EmotiGlass has therapeutic applications as an aid to help control stress and anxiety.

Background

Does the sight of a dog make you happy or fearful?  Are you comfortable around people of other races?  We commonly think that the emotions aroused by events around us are the results of our biases and prior experiences.  This is very true, but there is much more than meets the eye when it comes to understanding the way in which our brains attach emotional content to our perceptions.

We develop implantable cardiac medical devices for a living, and a few months ago, we came across an interesting journal article that caught our curiosity.  In this paper [Azevedo, et al. 2017], researchers from the University of London and their colleagues showed that presenting an image at different periods of the cardiac cycle would cause changes in the way that subjects would emotionally perceive that image.

The paper reported that racial biases could clearly be modulated by changing the timing at which an image is presented.  In this study, pictures of dark-skinned or light-skinned individuals holding various objects were presented to coincide with either the heart’s contraction (cardiac systole) or relaxation (diastole).  Results showed that if the image was presented during cardiac systole, subjects significantly misidentified innocuous objects as weapons when they were held by dark-skinned people.  Remarkable!

A group from University College London conducted a related study [Gray, et al., 2009] through which they found that sensory processing depends on when stimuli are experienced in relation to heartbeat timing.  Specifically, the perception of pain – which is strongly biased by emotion—varied depending on when during the cardiac cycle noxious stimuli were delivered to the subjects.

The mechanism behind these effects seems to be that the emotional arousal caused by a stimulus depends on signals being received by the brain from the body’s internal blood pressure sensors (baroreceptors).  Simply timing the delivery of stimuli against the cardiac cycle can change the way in which the brain processes stimuli!

It’s not only the timing component of a stimulus that can change the emotions that it arouses.  The spatial component also has major effects on the emotional perception of visual scenes.  In his exciting book “Of Two Minds: The Revolutionary Science of Dual-Brain Psychology”, Dr. Fredric Schiffer [1998], a Professor of Psychiatry at Harvard Medical School and Attending Psychiatrist at McLean Hospital, showed that very distinct emotions can be evoked by selectively blocking access of the visual field to one of the brain’s hemispheres.

Spatio-temporal changes in the visual field also cause deep changes in emotional response.  EMDR (Eye Movement Desensitization and Reprocessing) is a known psychotherapy technique whereby causing side-to-side eye movements is believed to unlock a memory mechanism that can be used to reprocess traumatic events.  Outside the therapeutic setting, mild side-to-side sweeping of the visual field appear to decrease the emotional impact of distressing events. The objective of the EmotiGlass project is to develop a pair of active glasses that can be controlled to selectively occlude vision with the proper timing and/or spatial distribution to allow a computer to modulate the user’s EMOTIONAL perception of reality.  Unlike augmented-reality devices that limit themselves to enhancing reality with computer-generated content, we aim to develop the first device that allows a computer to modulate the perceived emotional content of reality.

The EmotiGlass Concept

Figure 1 – Conceptual view of the EmotiGlass goggles 

  Figure 2 – Conceptual block diagram for the EmotiGlass goggles

Figure 1 presents a conceptual view of the EmotiGlass goggles, while Figure 2 presnts a possible block diagram.  The concept calls for eyewear frames with multi-pixel light shutters that can be controlled by an on-board microcontroller to selectively occlude the visual field in synchrony with the cardiac cycle as detected by a photoplethysmographic sensor or an ECG channel.  The emotional valence of reality can then be modulated by a computer or by the user through a user interface.  For the sake of convenience, the eyewear can incorporate all of the real-time electronics, while control of operating mode can be done wirelessly via BLE.

The therapeutic applications of EmotiGlass are obvious as an aid to help control stress and anxiety.  However, EmotiGlass enables completely new applications in the field of augmented reality.  For example, the emotional biases of police and other first responders could be dampened to lessen inappropriate use of force.  On the other hand, increased emotional arousal could be used in entertainment applications to enhance movies and gaming applications.

Operating Modalities

Visual Occlusion Synchronized to Baroreceptor Activation

It has been demonstrated that visceral afferent signals to the brain influence thoughts, feelings, and behavior regardless of whether or not these signals reach conscious awareness.  For example, experiments performed by Critchley and his collaborators [Gray et al. 2011, Critchley et al. 2015] at the University of Sussex (Brighton, UK) have shown that the emotional centers of the brain more easily detect fear and assign greater emotional salience to stimuli presented during cardiac systole than when presented at diastole.

In these experiments, subjective and brain responses were measured to pictures of faces showing emotions presented before and during cardiac systole. Facial expressions of disgust were judged as more intense when presented at systole, and rebound heart rate increases were reduced after expressions of disgust and happiness.  Neuroimaging studies showed that activity in the emotional centers of the brain changed with cardiac timing.  As mentioned in the background section, such timing-dependent changes influence emotional judgements, so powerfully that test subjects erred in the identification of items as weapons when held by people with dark skin, but not by people with light skin if the images are presented during cardiac systole.  It is thus indisputable that there exists a close coupling of visceral and emotional processes, and that regions in the body that receive internal information have a strong influence on affective judgment. Selectively occluding vision during parts of the cardiac cycle thus makes it possible to modulate the emotional response to a scene.  As shown in Figure 3, visual blanking may be synchronized to cardiac activity either by using the ECG signal, or the plethysmography signal (which may be detected using the light reflected off the skin as blood enters and leaves the small blood vessels near the pickup sensor).

Figure 3 – Visual occlusion may be applied in synchrony with either the top-of-R of the ECG (left) or when the pulse plethysmography signal crosses a certain threshold.

Photoplethysmography is probably the better alternative because it requires just a LED and a photosensor placed in contact with the skin to detect.  The ear lobe or skull are good points to place the sensor, and calculating timing is not necessary because the pressure signal detected at those locations is virtually the same as that which excites the baroreceptors that innervate the brain.

Lateralized Brain Stimulation

It is well known from the study of split-brain patients that the right brain and left brain are capable of having their own mentation and actions, whereby the right hemisphere can have intact mental faculties, separate from and often beyond the awareness of a person's left-sided mind.    Dr. Fredric Schiffer [1998], a professor of psychiatry at Harvard Medical School and Attending Psychiatrist at McLean Hospital, has demonstrated that this is not only the case in split-brain patients, but rather that everyone has two autonomous minds - one associated with the left brain and one with the right brain - and that in healthy individuals, these two minds cooperate harmoniously towards a balanced personality.

However, troubling behavioral symptoms may happen when a strong imbalance exists between the two minds since an unhealthy person-to-person relationship can develop between these two autonomous minds, whereby the same type of psychological issues and conflicts that would apply between two people may apply to the relationship between the two minds in a single individual.  The dominant hemisphere can thus markedly and negatively affect an individual’s personality.

Dr. Schiffer [1999] uses glasses that selectively occlude vision such that only one side of the brain is stimulated, having static transparent and opaque regions to selectively stimulate only the right brain or the left brain (Figure 4, left).  When used during a psychotherapy session, a patient can be asked to alternate between lateral visual fields and induce both an increase and a decrease in his symptoms to show him that his fears reside only in a part of his mind.  The glasses can also be used outside the context of a psychotherapy session in order to reduce stress and anxiety.  These glasses have fixed occlusion zones and thus need to be specifically constructed for each user. Based on Schiffer’s work, others have designed therapeutic glasses using moveable screens to selectively occlude the patient’s vision.  For example, in U.S. Patent No. 6,141,797 “Opaque goggles having openable window” (Figure 4, right), Robert Buck describes glasses that have moveable opaque windows. These are commercially available under the name “NeuView” (www.neuviewglasses.com).

Figure 4 – Goggles of the type used by Dr. Fredric Schiffer to selectively occlude vision such that only one side of the brain is stimulated.

These eyeglasses for selective lateral stimulation are simple devices through which the temporal and spatial distribution, as well as magnitude of the occlusion cannot be finely controlled.  In addition, these devices do not look like regular eyewear, thus attracting attention, which may be undesirable when treating stress, anxiety, and other mental disturbances.

In contrast, the EmotiGlass goggles can selectively block a subject’s field of vision with controllable spatial distribution and magnitude of the occlusion, all while maintaining the semblance of a normal pair of sunglasses.

The science behind this modality is fascinating!  The basic principle is that the right brain and left brain are capable of having their own mentation and actions, whereby the right hemisphere can have intact mental faculties, separate from and often beyond the awareness of one’s left-sided mind.   As such, a person can have two autonomous minds, one associated with the left brain and one with the right brain.  Schiffer [1998] has demonstrated that a person-to-person relationship can develop between these two autonomous minds, whereby the same type of psychological issues and conflicts that would apply between two people may apply to the relationship between the two minds in a single individual.  The dominant hemisphere can often markedly determine a person's personality.

For example, in a certain person the left mind might “dominate” and “suppress” his right mind. In another, the right mind may be dominant.  In mentally-healthy people, the two minds coexist in harmony with mutual respect and cooperation.  However, troubling behavioral symptoms may happen when a strong imbalance (and hence conflict) between the two minds occurs.

According to Schiffer’s experiments, those who have been affected by emotional trauma lateralize the effects, perhaps in an effort to maintain more-or-less-normal functioning. One hemisphere or the other gets stuck in the past, and acts out through the patient's symptoms.

Schiffer has demonstrated that by restricting vision to a portion of the retina of an eye that is connected to a particular hemisphere of the brain, that hemisphere can be stimulated preferentially, causing changes in the psychological state of a person that may be therapeutically beneficial. This is possible because the eyes are connected to the brain so that vision to the left side of a person goes primarily to the opposite (right) hemisphere and vision to the right side of a person goes primarily to the left hemisphere.  Figure 5 shows that the image of objects are projected by the lenses onto the retinas.  Each retina is divided into a medial and lateral aspect.  Image points projected on the lateral aspect of the retina of the left eye go directly to the brain’s left hemisphere, while points projected on the medial aspect of the retina of the left retina go to the brain’s right hemisphere.  Similarly, image points projected on the lateral aspect of the retina of the right eye go directly to the brain’s right hemisphere, while points projected on the medial aspect of the retina of the left retina go to the brain’s left hemisphere.  Images conveyed to only one hemisphere by the optic nerves can be communicated to the other hemisphere via the corpus callosum.

Figure 5 – Connections between the eyes and the brain.  Left:  Healthy.  Right:  Lateralized vision in split-brain patient

For the lateralized brain stimulation modality, the glasses need to occlude the visual field such that only the parts of the retinas that innervate the left or right brain are illuminated by the scene.  This requires at least three distinct vertical regions as shown in Figure 6.

Figure 6 – Distribution of occlusion for lateralized brain stimulation.  Figure on the left shows the configuration to preferentially stimulate the subject’s right brain hemisphere through both direct innervation (from the right retina’s lateral aspect) and contralateral innervation (from the left retina’s medial aspect).

The front side of the glasses need to be able to create a clear window of around 1cm width, and it should be programmable to allow fine tuning for each individual.  A good programmability range would be between 4 mm and 20 mm with 2mm resolution.  It must be noted that since vision in the far periphery of the visual field has low resolution, the side shutters may be replaced by simple light valves (not pixelized, one for each side) without loss of therapeutic effectiveness.

Swept-Visual Stimulation

Eye Movement Desensitization and Reprocessing (EMDR) therapy was born in 1987 when psychologist Dr. Francine Shapiro when she realized that side-to-side eye movements appeared to decrease the negative emotion associated with her own distressing memories. She hypothesized that eye movements had a desensitizing effect, and went on to test it under controlled clinical conditions [Shapiro, 2017].

The concept has led to a standard procedure that is commonly used to treat victims of psychological trauma.  Although the underlying mechanism is not known, it has been hypothesized that side-to-side movement of the eyes activates the same neuronal networks responsible for REM sleep.  Experimental evidence seems to indicate that actual eye movement is not necessary, and that the same neural activation can be elicited by side-to-side movements in the light patterns of the visual field.

It is thus hypothesized that sweeping a clear window back-and-forth through EmotiGlass will decrease the emotional impact of distressing events.

Figure 7 – Swept visual stimulation with the EmotiGlass goggles.

For this modality, as shown in Figure 7, the clear window should have an approximate width of 1cm, while the opacity of the occluded area should have a programmable range of about 20% of the clear window and up to 100%.  The approximate sweeping frequency to solidify a positive experience is approximately 0.5 to 1Hz, while reprocessing of traumatic memories is done in a therapeutic setting at sweep frequencies of up to 3 Hz.  The most common spatial pattern is simple side-to-side, but diagonal or figure-of-8 sweeps are also used, especially in conjunction with fast movements.

Interoception Enhancement

Interoreceptive awareness is our sensitivity to our own internal sensations.  Receptors throughout our body gather information that enables us to feel hunger, satiety, itch, internal pain, body temperature, nausea, need for the bathroom, tickles, physical exertion, sexual arousal, and other internal states.

Sensations from the interoceptive system alert us that our internal system is out of balance and requires us to take action to restore the balance.  For example, the sensation of thirst compels us to drink in order to restore the body’s electrolyte concentration that is unbalanced by the loss of fluids to sweat and urination.

Introceptive sensations are also important in the way in which we perceive, conceptualize, and remember our affective experiences and other cognitive processes.  These sensations allow us to feel our emotions. 

For example, before speaking in public, we may feel our heart racing, slight nausea, muscle tension, and shakiness in our hands.  Not feeling these sensations would make it difficult to clearly identify the emotions of anxiousness and nervousness.

Not being to interpret our interoceptive signals in itself causes anxiety, and many biofeedback techniques rely in fact on helping people learn how to “read” their own body.  For example, a well-conducted study demonstrated that improving heartbeat perception in patients with medically-unexplained symptoms reduces symptom distress [Schaefer, 2014]. The EmotiGlass goggles are fully equipped to unobtrusively provide direct heartbeat feedback to the user, for example by diming the side panels in synchrony with the plethysmographic signal.  In fact, after the subject is “trained” to perceive the side dimming as a proxy for heartbeat, the rhythm of the dimming can be artificially modulated by a computer to change the emotional state of the user.

Automatic Sunglasses

Although not a feature to modulate the emotional response of the user, automatically-darkening sunglasses can be easily implemented by the addition of a light sensor, so why not…

Licenses

Hardware for the EmotiGlass project is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/4.0/

Software for the EmotiGlass project is supplied under the MIT open-source license. 

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

The following licenses apply to libraries that are used by the software that operates EmotiGlass:

The source code for the Arduino environment is covered by the GPL, which requires any modifications to be open-sourced under the same license. It does not prevent the sale of derivative software or its inclusion in commercial products.   The following libraries fall under this license:

The LCD module library <dog_7565R.h> is open-source released under GNU General Public License version 2 or the GNU Lesser General Public License version 2.1, both as published by the Free Software Foundation.

The following Adafruit libraries are open source released under the BSD License:

We are not a patent attorneys and cannot render a “Freedom to Operate” opinion on the EmotiGlass concept, apparatus, or methods.  We expressly disclaim any liability for the infringement of any intellectual property by the making, using, or selling of devices based on the descriptions provided in this paper, and suggest that anyone interested in such projects seek proper legal counsel.

Disclaimer

The EmotiGlass project is presented solely for educational purposes.  The authors make no medical or healthcare claims for the EmotiGlass device.  The authors do not suggest that the circuits and software presented herein can or should be used in place of or as an adjunct to professional medical treatment or advice.  Sole responsibility for the use of these circuits and/or software or of systems incorporating these circuits and/or software lies with the reader, who must apply for any and all approvals and certifications that the law may require for their use.

Furthermore, the safe operation of these circuits requires the use of batteries, and connection to external signal acquisition/processing/monitoring equipment should be done only through signal isolators with the proper isolation ratings.

The authors do not make any representations as to the completeness or the accuracy of the information contained herein, and disclaim any liability for damages or injuries, whether caused by or arising from the lack of completeness, inaccuracies of the information, misinterpretations of the directions, misapplication of the circuits and information or otherwise.  The authors and publisher expressly disclaim any implied warranties of merchantability and of fitness of use for any particular purpose, even if a particular purpose is indicated in this writing.

References to other manufacturers’ products do not constitute an endorsement of these products, but are included for the purpose of illustration and clarification.  It is not the authors’ intent to make any technical information and interface data presented in this book to supersede information provided by individual manufacturers.

The authors disclaim any liability for the infringement of patents by the making, using, or selling of such equipment or circuitry, and suggest that anyone interested in such commercial projects seek proper legal counsel. Finally, the authors are not responsible to the reader or third parties for any claim of special or consequential damages, in accordance to the previous disclaimer.

References

Azevedo RT, Garfinkel SN, Critchley HD, Tsakiris M, “Cardiac Afferent Activity Modulates the Expression of Racial Stereotypes”, Nature Communications, 8: 13854, 2017; DOI: 10.1038/NCOMMS13854

Buck R, U.S. Patent No. 6,141,797, “Opaque goggles having openable window”, November 7, 2000

Critchley HD, Garfinkel SN, “Interactions between visceral afferent signaling and stimulus processing”, Frontiers in Neuroscience, 9, 2015; DOI: 10.3389/fnins.2015.00286.

Gray MA, Rylander K, Harrison NA, Wallin BG, Critchley HD, “Following One’s Heart: Cardiac Rhythms Gate Central Initiation of Sympathetic Reflexes”, J. Neurosci, 29(6), 1817–1825, 2009; DOI:10.1523/JNEUROSCI.3363-08.2009.

Gray MA, Rylander K, Harrison NA, Wallin BG, Critchley HD, “Cardiac Rhythms Gate Central Initiation of Sympathetic Reflexes”, J Neurosci., 29(6), 1817–1825, 2009; doi:10.1523/JNEUROSCI.3363-08.2009

Gray MA, Beacher FD, Minati L, Nagai Y, Kemp AH, Harrison NA, Critchley H, “Emotional Appraisal Is Influenced by Cardiac Afferent Information”, Emotion D., 2011; DOI: 10.1037/a0025083

Schaefer M, Egloff B, Gerlach AL, Witthöft M, “Improving heartbeat perception in patients with medically unexplained symptoms reduces symptom distress”, Biol Psychol., 101, 69-76, 2014;  DOI: 10.1016/j.biopsycho.2014.05.012.

Schiffer F, Of Two Minds: The Revolutionary Science of Dual-Brain Psychology, Free Press, 1998

Schiffer F, U.S. Patent No. 5,963,294, “Method for using therapeutic glasses for stimulating a change in the psychological state of a subject”, October 5, 1999

Shapiro F, Eye Movement Desensitization and Reprocessing (EMDR) Therapy, Third Edition: Basic Principles, Protocols, and Procedures, The Guilford Press; Third Edition, 2017.