Eye and Brain Analyses Help Stave Off The Dangers of Self-Report

6 min readDec 5, 2017

Have you ever given others the benefit of doubt? If you have, on what grounds? Their facial expression? Their gesture? Their tone?

Limitation of Self-Report

At least for researchers, they usually put their trust in statements and numbers, a self-report survey or questionnaire mostly. Self-report is a classic method of gathering data, but at the same time, it is one of many methodologies that is frequently questioned for its reliability. In fact, there is a quite solid argument for questioning the validity of self-report:

Participants are not always truthful

Imagine you are asked to fill out a questionnaire on your drug use, suicidal impulse or sexual tendency. Would you be 100% honest about it?

Participants may not necessarily have a high introspective ability

Most people find it difficult to assess their feelings and thinking accurately and thoroughly.

Interpretation of rating points varies

Though more insightful than a yes-or-no question, a scale of 0–100 to rate your state of mind, for example, challenges you to “chop” your mental states into exactly 100 pieces and hand in the best representation of yourself. Even worse, everyone “chops” it in different ways.

So what can you do about it then?

More Objective and Quantitative Measures: Physiological Responses

The most desirable solution is to scrutinize every “move” people make that is so subtle to be noticed by an observer as well as the observed. Tracking such subtle “moves”, perhaps seemingly difficult at first sight, is not impossible with the help of physiological measures. In fact, people can hardly control involuntary and spontaneous responses and manipulate their physiological activities at a particular moment. Therefore, compared to a self-report method, physiological responses are more objective and quantitative.

There is a variety of physiological indicators that have been frequently employed in research: electromyography (EMG) — electrical activity produced by skeletal muscles; galvanic skin response (GSR) — changes in electrical properties of the skin; electrocardiogram (ECG) — electrical activity of the heart; etc. They can bring about observations and insights that would have been difficult to capture otherwise, making up for the deficiency in the validity of subjective measures.

Clearly seeing the potential in physiological measures, one study decided to opt for electroencephalography (EEG) and eye-tracking techniques to measure cognitive load and compare self-report and physiological methods.

Measuring Cognitive Load: A Comparison of Self-report and Physiological Methods

This study compared three methods — self-report, EEG, and eye tracking — to measure cognitive load in solving puzzles with four different levels of difficulty (intrinsic cognitive load). The participants were instructed to solve four different puzzles with increasing difficulty from Puzzle 1 to Puzzle 4 and be fitted with an eye tracking device and an EEG headset during the experiment. The experiment was sequenced in the following order:

The operation span task (working memory capacity — recalling the consonant in between mathematical problems — and spatial visualization — paper-folding test)
Participant data survey (demographics, vision issues, prior knowledge, etc.)
Practice Puzzles 1, 2, and 3
Cognitive Load and Puzzle Self-Efficacy Survey (a 9-point response scale for the difficulty level)
Problem-Solving Puzzles 1, 2, 3, and 4, presented in a random order for each participant with Cognitive Load and Puzzle Self-Efficacy Survey in between each puzzle problem.
The exit survey

**Table 1. Self-report Ratings of Cognitive Load (left) vs. Confusion Matrix for EEG Spectral Features (right)**

The study first explored the correlation between the self-report ratings of cognitive load and the difficulty of the puzzles (intrinsic cognitive load). As indicated in Table 1, the participants self-reported higher cognitive load on average as the intrinsic cognitive load increased.

**Figure 1. The process by which the difficulty level of the puzzles and the self-report difficulty ratings for each puzzle are predicted from physiological data.**

How about EEG? Based on the literature in cognitive science indicating that alpha waves decrease and theta waves increase as a task becomes more difficult, the spectral analysis (Figure 1) of EEG was carried out to differentiate the level of tasks. In Table 1, it should be noted that the algorithm did not classify any of Puzzle 1 and 2 samples as Puzzle 4, which shows quite an accurate classification for the first two puzzles. Also, EEG analysis predicted the difficulty level of Puzzle 3 with a high accuracy of 71%. However, the algorithm failed to distinguish the difficulty levels of Puzzle 3 and 4 samples for the observed Puzzle 4.

Let’s compare two results. The EEG data appeared to better distinguish between Puzzle 2 and 3 than did the average self-report cognitive load ratings: there is no significant difference in mean self-report cognitive load ratings between Puzzle 2 (5.19) and 3 (5.28) statistically. However, neither of the two successfully distinguished the difficulty levels of Puzzle 3 and 4. Overall, better distinguishing puzzles from one another, the EEG analyses were more accurate in evaluating the participants’ cognitive load than self-reported cognitive load ratings.

A little digression here. You may wonder why eye tracking techniques are not discussed in the result. The study initially hypothesized that, based on the literature, there is increased pupil dilation for a complex task compared to an easy task. However, the experimenters acknowledged in the end that subtle changes in the pupil and eye movement data were difficult to detect due to the low sampling rate of the eye tracking device and that cognitive load imposed during a puzzle task fluctuated over its duration so capturing changes at every moment is somewhat unnecessary.

Big Opportunities At Stake

Although the study misses out on the opportunity to explore the potential of eye tracking technology, the study successfully demonstrated that physiological measures can possibly serve as an alternative or, if not, a supplement to self-report measures.

Yes. Self-report methods have their shortcomings, but its importance should not be undermined. It is ideal for large sample sizes to observe a trend and is an unobtrusive way of acquiring responses without too much hassle. However, if a research topic demands more objective analysis that is unfathomable through self-report and is confined to a small sample size, then self-report loses its effectiveness. So, it depends on what kind of research it is.

Nonetheless, with respect to evaluating cognitive ability and mental states, physiological methods are unparalleled. For instance, in education, teachers can use physiological measurements to assess and improve students’ learning ability. Companies can acquire clients’ authentic feedback and improve its product and service. Doctors can treat post-traumatic stress disorder patients with comprehensive assessment of recovery. There exists a huge room for application.

On a lighter note, in our Medium, there is a recent post about research in a virtual environment (Virtual Reality: Elevate Your Research From Mediocrity to Greatness). It highlights the advantages of integrating virtual reality into a study. A high ecological validity of virtual reality can feed a lifelike experience that can evoke participants’ genuine reactions such as goosebumps owing to phobia. So, it’s needless to mention the powerful synergy that virtual reality creates with physiological methods. Just something to think about!

Reference

*For further information, contact us at www.looxidlabs.com.