Some thoughts on writing ‘Bayes Glaze’ theoretical papers.

[This was a twitter navel-gazing thread someone ‘unrolled’. I was really surprised that it read basically like a blog post, so I thought why not post it here directly! I’ve made a few edits for readability. So consider this an experiment in micro-blogging ….]

In the past few years, I’ve started and stopped a paper on metacognition, self-inference, and expected precision about a dozen times. I just feel conflicted about the nature of these papers and want to make a very circumspect argument without too much hype. As many of you frequently note, we have way too many ‘Bayes glaze’ review papers in glam mags making a bunch of claims for which there is no clear relationship to data or actual computational mechanisms.

It has gotten so bad, I sometimes see papers or talks where it feels like they took totally unrelated concepts and plastered “prediction” or “prediction error” in random places. This is unfortunate, and it’s largely driven by the fact that these shallow reviews generate a bonkers amount of citations. It is a land rush to publish the same story over and over again just changing the topic labels, planting a flag in an area and then publishing some quasi-related empirical stuff. I know people are excited about predictive processing, and I totally share that. And there is really excellent theoretical work being done, and I guess flag planting in some cases is not totally indefensible for early career researchers. But there is also a lot of cynical stuff, and I worry that this speaks so much more loudly than the good, careful stuff. The danger here is that we’re going to cause a blowback and be ultimately seen as ‘cargo cult computationalists’, which will drag all of our research down both good and otherwise.

In the past my theoretical papers in this area have been super dense and frankly a bit confusing in some aspects. I just wanted to try and really, really do due-diligence and not overstate my case. But I do have some very specific theoretical proposals that I think are unique. I’m not sure why i’m sharing all this, but I think because it is always useful to remind people that we feel imposter syndrome and conflict at all career levels. And I want to try and be more transparent in my own thinking – I feel that the earlier I get feedback the better. And these papers have been living in my head like demons, simultaneously too ashamed to be written and jealous at everyone else getting on with their sexy high impact review papers.

Specifically, I have some fairly straightforward ideas about how interoception and neural gain (precision) inter-relate, and also have a model i’ve been working on for years about how metacognition relates to expected precision. If you’ve seen any of my recent talks, you get the gist of these ideas.

Now, I’m *really* going to force myself to finally write these. I don’t really care where they are published, it doesn’t need to be a glamour review journal (as many have suggested I should aim for). Although at my career stage, I guess that is the thing to do. I think I will probably preprint them on my blog, or at least muse openly about them here, although i’m not sure if this is a great idea for theoretical work.

Further, I will try and hold to three key promises:

  1. Keep it simple. One key hypothesis/proposal per paper. Nothing grandiose.
  2. Specific, falsifiable predictions about behavioral & neurophysiological phenomenon, with no (minimal?) hand-waving
  3. Consider alternative models/views – it really gets my goat when someone slaps ‘prediction error’ on their otherwise straightforward story and then acts like it’s the only game in town. ‘Predictive processing’ tells you almost *nothing* about specific computational architectures, neurobiological mechanisms, or general process theories. I’ve said this until i’m blue in the face: there can be many, many competing models of any phenomenon, all of which utilize prediction errors.

These papers *won’t* be explicitly computational – although we have that work under preparation as well – but will just try to make a single key point that I want to build on. If I achieve my other three aims, it should be reasonably straight-forward to build computational models from these papers.

That is the idea. Now I need to go lock myself in a cabin-in-the-woods for a few weeks and finally get these papers off my plate. Otherwise these Bayesian demons are just gonna keep screaming.

So, where to submit? Don’t say Frontiers…

For whom the bell tolls? A potential death-knell for the heartbeat counting task.

Interoception – the perception of signals arising from the visceral body – is a hot topic in cognitive neuroscience and psychology. And rightly so; a growing body of evidence suggests that brain-body interaction is closely linked to mood1, memory2, and mental health3. In terms of basic science, many theorists argue that the integration of bodily and exteroceptive (i.e., visual) signals underlies the genesis of a subjective, embodied point of view4–6.  However, noninvasively measuring (and even better, manipulating) interoception is inherently difficult. Unlike visual or tactile awareness, where an experimenter can carefully control stimulus strength and detection difficulty, interoceptive signals are inherently spontaneous, uncontrolled processes. As such, prevailing methods for measuring interoception typically involve subjects attending to their heartbeats and reporting how many heartbeats they counted in a given interval. This is known as the heartbeat counting task (or Schandry task, named after its creator)7. Now a new study has cast extreme doubt on what this task actually measures.

The study, published by Zamariola et al in Biological Psychology8, begins by detailing what we already largely know: the heartbeat counting task is inherently problematic. For example, the task is easily confounded by prior knowledge or beliefs about one’s average heart rate. Zamariola et al write:

“Since the original task instruction requires participants to estimate the number of heartbeats, individuals may provide an answer based on beliefs without actually attempting to perceive their heartbeats. Consistent with this view, one study (Windmann, Schonecke, Fröhlig, & Maldener, 1999) showed that changing the heart rate in patients with cardiac pacemaker, setting them to low (50 beats per minute, bpm), medium (75 bpm), or high (110 bpm) heart rate, did not influence their reported number of heartbeats. This suggests that these patients performed the task by relying on previous knowledge instead of perception of their bodily states.”

This raises the question of what exactly the task is measuring. The essence of heartbeat counting tasks is that one must silently count the number of perceived heartbeats over multiple temporal intervals. From this, an “interoceptive accuracy score” (IAcc) is computed using the formula:

1/3 ∑ (1–(|actual heartbeats – reported heartbeats|)/actual heartbeats)

This formula is meant to render over-counting (counting heartbeats that don’t occur) and under-counting (missing actual heartbeats) equivalent, in a score bounded by 0-1. Zamariola et al argue that these scores lack fundamental construct validity on the basis of four core arguments. I summarize each argument below; see the full article for the detailed explanation:

  1. [interoceptive] abilities involved in not missing true heartbeats may differ from abilities involved in not over-interpreting heartbeats-unrelated signals. [this assumption] would be questioned by evidence showing that IAcc scores largely depend on one error type only.
  2. IAcc scores should validly distinguish between respondents. If IAcc scores reflect people’s ability to accurately perceive their inner states, a correlation between actual and reported heartbeats should be observed, and this correlation should linearly increase with higher IAcc scores (i.e., better IAcc scorers should better map actual and reported heartbeats).
  3. a valid measure of interoception accuracy should not be structurally tied to heart condition. This is because heart condition (i.e. actual heartbeats) is not inherent to the definition of the interoceptive accuracy construct. In other words, it is essential for construct validity that people’s accuracy at perceiving their inner life is not structurally bound to their cardiac condition.”
  4. The counting interval [i.e., 10, 15, 30 seconds] should not impact IAcc; a wide range of scores are in fact used and these should be independent of the resultant measure.

Zamariola et al then go on to show that in a sample of 572 healthy individuals (386 female), each of these assumptions are strongly violated; IAcc scores depend largely on under-reporting heartbeats (Fig. 1), that the correlation of actual and perceived heartbeats is extremely low and higher at average than higher IAcc levels (Fig. 2), that IAcc is systematically increased as slower heart rates (Fig. 3), and that longer time intervals lead to substantially worse IAccc (not shown):

Fig. 1

  1. iACC scores are mainly driven by under-reporting; “less than 5% of participants showed overestimation… Hence, IAcc scores essentially inform us of how (un)willing participants are to report they perceived a heartbeat”

Fig. 2

2. Low overall correlation (grey-dashed line) of heartbeats counted and actual heartbeats (r = 0.16, 2.56% shared variance). Further, the correlation varied non-linearly across bins of iACC scores, which in the author’s words demonstrates that “IAcc scores fail to validly differentiate individuals in their ability to accurately perceive their inner states within the top 60% IAcc scorers.”

Fig. 3

3. iACC scores depend negatively on the number of actual heartbeats, suggesting that individuals with lower overall heart-rate will be erroneously characterized as ‘good interoceptive accuracy’.

Overall, the authors draw the conclusion the heartbeat counting task is nigh-useless, lacking both face and construct validity. What should we measure instead? The authors offer that, if one can have very many trials, than the mere correlation of counted and actual heartbeats may be a (slightly) better measure. However, given the massive bias present in under-reporting heartbeats, they suggest that the task measures only the willingness to report a heartbeat at all. As such, they highlight the need for true psychophysical tasks which can distinguish participant reporting bias (i.e., criterion) from the true sensitivity to heart beats. A potentially robust alternative may be the multi-interval heartbeat discrimination task9, in which a method of constant stimuli is used to compare heartbeats to multiple intervals of temporal stimuli. However, this task is substantially more difficult to administer; it requires some knowledge of psychophysics and as much as 45 minutes to complete. As many (myself included) are interesting in measuring interoception in sensitive patient populations, it’s not a given that this task will be widely adopted.

I’m curious what my readers think. For me, this paper proffers a final nail in the coffin of heartbeat counting tasks. Nearly every interoception researcher I’ve spoken to has expressed concerns about what the task actually measures. Worse, large intrasubject variance and the fact that many subjects perform incredibly poorly on the task seems to undermine the idea that it is anything like a measure of cardiac perception. At best, it seems to be a measure of interoceptive attention and report-bias. The study by Zamariola and colleagues is well-powered, sensibly conducted, and seems to provide unambiguous evidence against the task’s basic validity. Heart-beart counting; the bell tolls for thee.

References

  1. Foster, J. A. & McVey Neufeld, K.-A. Gut–brain axis: how the microbiome influences anxiety and depression. Trends Neurosci. 36, 305–312 (2013).
  2. Zelano, C. et al. Nasal Respiration Entrains Human Limbic Oscillations and Modulates Cognitive Function. J. Neurosci. 36, 12448–12467 (2016).
  3. Khalsa, S. S. et al. Interoception and Mental Health: a Roadmap. Biol. Psychiatry Cogn. Neurosci. Neuroimaging (2017). doi:10.1016/j.bpsc.2017.12.004
  4. Park, H.-D. & Tallon-Baudry, C. The neural subjective frame: from bodily signals to perceptual consciousness. Philos. Trans. R. Soc. Lond. B Biol. Sci. 369, 20130208 (2014).
  5. Seth, A. K. Interoceptive inference, emotion, and the embodied self. Trends Cogn. Sci. 17, 565–573 (2013).
  6. Barrett, L. F. & Simmons, W. K. Interoceptive predictions in the brain. Nat. Rev. Neurosci. 16, 419–429 (2015).
  7. Schandry, R., Sparrer, B. & Weitkunat, R. From the heart to the brain: A study of heartbeat contingent scalp potentials. Int. J. Neurosci. 30, 261–275 (1986).
  8. Zamariola, G., Maurage, P., Luminet, O. & Corneille, O. Interoceptive Accuracy Scores from the Heartbeat Counting Task are Problematic: Evidence from Simple Bivariate Correlations. Biol. Psychol. doi:10.1016/j.biopsycho.2018.06.006
  9. Brener, J. & Ring, C. Towards a psychophysics of interoceptive processes: the measurement of heartbeat detection. Philos. Trans. R. Soc. B Biol. Sci. 371, 20160015 (2016).

 

Unexpected arousal shapes confidence – blog and news coverage

For those looking for a good summary of our recent publication, several outlets gave us solid coverage for expert and non-expert alike. Here is a short summary of the most useful write-ups:

The eLife digest itself was excellent – make sure to fill out the survey at the end to let eLife know what you think of the digests  (I love them).

via Arousing confidence – Brains and Behaviour – Medium

As you read the words on this page, you might also notice a growing feeling of confidence that you understand their meaning. Every day we make decisions based on ambiguous information and in response to factors over which we have little or no control. Yet rather than being constantly paralysed by doubt, we generally feel reasonably confident about our choices. So where does this feeling of confidence come from?

Computational models of human decision-making assume that our confidence depends on the quality of the information available to us: the less ambiguous this information, the more confident we should feel. According to this idea, the information on which we base our decisions is also the information that determines how confident we are that those decisions are correct. However, recent experiments suggest that this is not the whole story. Instead, our internal states — specifically how our heart is beating and how alert we are — may influence our confidence in our decisions without affecting the decisions themselves.

To test this possibility, Micah Allen and co-workers asked volunteers to decide whether dots on a screen were moving to the left or to the right, and to indicate how confident they were in their choice. As the task became objectively more difficult, the volunteers became less confident about their decisions. However, increasing the volunteers’ alertness or “arousal” levels immediately before a trial countered this effect, showing that task difficulty is not the only factor that determines confidence. Measures of arousal — specifically heart rate and pupil dilation — were also related to how confident the volunteers felt on each trial. These results suggest that unconscious processes might exert a subtle influence on our conscious, reflective decisions, independently of the accuracy of the decisions themselves.

The next step will be to develop more refined mathematical models of perception and decision-making to quantify the exact impact of arousal and other bodily sensations on confidence. The results may also be relevant to understanding clinical disorders, such as anxiety and depression, where changes in arousal might lock sufferers into an unrealistically certain or uncertain world.

The PNAS journal club also published a useful summary, including some great quotes from Phil Corlett and Rebecca Todd:

via Journal Club: How your body feels influences your confidence levels | National Academy of Sciences

… Allen’s findings are “relevant to anyone whose job is to make difficult perceptual judgments trying to see signal in a lot of noise,” such as radiologists or baggage inspectors, says cognitive neuroscientist Rebecca Todd at the University of British Columbia in Vancouver, who did not take part in the research. Todd suggests that people who apply decision-making models to real world problems need to better account for the influence of internal or emotional states on confidence.

The fact that bodily states can influence confidence may even shed light on mental disorders, which often involve blunted or heightened signals from the body. Symptoms could result from how changes in sensory input affect perceptual decision-making, says cognitive neuroscientist and schizophrenia researcher Phil Corlett at Yale University, who did not participate in this study.

Corlett notes that some of the same ion channels involved in regulating heart rate are implicated in schizophrenia as well. “Maybe boosting heart rate might lead people with schizophrenia to see or hear things that aren’t present,” he speculates, adding that future work could analyze how people with mental disorders perform on these tasks…

I also wrote a blog post summarizing the article for The Conversation:

via How subtle changes in our bodies affect conscious awareness and decision confidence

How do we become aware of our own thoughts and feelings? And what enables us to know when we’ve made a good or bad decision? Every day we are confronted with ambiguous situations. If we want to learn from our mistakes, it is important that we sometimes reflect on our decisions. Did I make the right choice when I leveraged my house mortgage against the market? Was that stop light green or red? Did I really hear a footstep in the attic, or was it just the wind?

When events are more uncertain, for example if our windscreen fogs up while driving, we are typically less confident in what we’ve seen or decided. This ability to consciously examine our own experiences, sometimes called introspection, is thought to depend on the brain appraising how reliable or “noisy” the information driving those experiences is. Some scientists and philosophers believe that this capacity for introspection is a necessary feature of consciousness itself, forging the crucial link between sensation and awareness.

One important theory is that the brain acts as a kind of statistician, weighting options by their reliability, to produce a feeling of confidence more or less in line with what we’ve actually seen, felt or done. And although this theory does a reasonably good job of explaining our confidence in a variety of settings, it neglects an important fact about our brains – they are situated within our bodies. Even now, as you read the words on this page, you might have some passing awareness of how your socks sit on your feet, how fast your heart is beating or if the room is the right temperature.

Even if you were not fully aware of these things, the body is always shaping how we experience ourselves and the world around us. That is to say experience is always from somewhere, embodied within a particular perspective. Indeed, recent research suggests that our conscious awareness of the world is very much dependent on exactly these kinds of internal bodily states. But what about confidence? Is it possible that when I reflect on what I’ve just seen or felt, my body is acting behind the scenes? …

The New Scientist took an interesting angle not as explored in the other write-ups, and also included a good response from Ariel Zylberberg:

via A bit of disgust can change how confident you feel | New Scientist

“We were tricking the brain and changing the body in a way that had nothing to do with the task,” Allen says. In doing so, they showed that a person’s sense of confidence relies on internal as well as external signals – and the balance can be shifted by increasing your alertness.

Allen thinks the reaction to disgust suppressed the “noise” created by the more varied movement of the dots during the more difficult versions of the task. “They’re taking their own confidence as a cue and ignoring the stimulus in the world.”

“It’s surprising that they show that confidence can be motivated by processes inside a person, instead of what we tend to believe, which is that confidence should be motivated by external things that affect a decision,” says Ariel Zylberberg at Columbia University in New York. “Disgust leads to aversion. If you try a food and it’s disgusting, you walk away from it,” says Zylberberg. “Here, if you induce disgust, high confidence becomes lower and low confidence becomes higher. It could be that disgust is generating this repulsion.”

It is not clear whether it is the feeling of disgust that changes a person’s confidence in this way, or whether inducing alertness with a different emotion, such as anger or fear, would have the same effect.

You can find all the coverage for our article using these excellent services, altmetric & ImpactStory.

https://www.altmetric.com/details/12986857

https://impactstory.org/u/0000-0001-9399-4179/p/mfatd6ZhpW

Thanks to everyone who shared, enjoyed, and interacted with our research!