The Wild West of Publication Reform Is Now

It’s been a while since I’ve tried out my publication reform revolutionary hat (it comes in red!), but tonight as I was winding down I came across a post I simply could not resist. Titled “Post-publication peer review and the problem of privilege” by evolutionary ecologist Stephen Heard, the post argues that we should be cautious of post-publication review schemes insofar as they may bring about a new era of privilege in research consumption. Stephen writes:

“The packaging of papers into conventional journals, following pre-publication peer review, provides an important but under-recognized service: a signalling system that conveys information about quality and breath of relevance. I know, for instance, that I’ll be interested in almost any paper in The American Naturalist*. That the paper was judged (by peer reviewers and editors) suitable for that journal tells me two things: that it’s very good, and that it has broad implications beyond its particular topic (so I might want to read it even if it isn’t exactly in my own sub-sub-discipline). Take away that peer-review-provided signalling, and what’s left? A firehose of undifferentiated preprints, thousands of them, that are all equal candidates for my limited reading time (such that it exists). I can’t read them all (nobody can), so I have just two options: identify things to read by keyword alerts (which work only if very narrowly focused**), or identify them by author alerts. In other words, in the absence of other signals, I’ll read papers authored by people who I already know write interesting and important papers.”

In a nutshell, Stephen turns the entire argument for PPPR and publishing reform on its head. High impact[1] journals don’t represent elitism; rather they provide the no name rising young scientist a chance to have their work read and cited. This argument really made me pause for a second as it represents the polar opposite of almost my entire worldview on the scientific game and academic publishing. In my view, top-tier journals represent an entrenched system of elitism masquerading as meritocracy. They make arbitrary, journalistic decisions that exert intense power over career advancement. If anything the self-publication revolution represents the ability of a ‘nobody’ to shake the field with a powerful argument or study.

Needless to say I was at first shocked to see this argument supported by a number of other scientists on Twitter, who felt that it represented “everything wrong with the anti-journal rhetoric” spouted by loons such as myself. But then I remembered that in fact this is a version of an argument I hear almost weekly when similar discussions come up with colleagues. Ever since I wrote my pie-in-the sky self-publishing manifesto (don’t call it a manifesto!), I’ve been subjected (and rightly so!) to a kind of trial-by-peers as a de facto representative of the ‘revolution’. Most recently I was even cornered at a holiday party by a large and intimidating physicist who yelled at me that I was naïve and that “my system” would never work, for almost the exact reasons raised in Stephen’s post. So lets take a look at what these common worries are.

The Filter Problem

Bar none the first, most common complaint I hear when talking about various forms of publication reform is the “filter problem”. Stephen describes the fear quite succinctly; how will we ever find the stuff worth reading when the data deluge hits? How can we sort the wheat from the chaff, if journals don’t do it for us?

I used to take this problem seriously, and try to dream up all kinds of neato reddit-like schemes to solve it. But the truth is, it just represents a way of thinking that is rapidly becoming irrelevant. Journal based indexing isn’t a useful way to find papers. It is one signal in a sea of information and it isn’t at all clear what it actually represents. I feel like people who worry about the filter bubble tend to be more senior scientists who already struggle to keep up with the literature. For one thing, science is marked by an incessant march towards specialization. The notion that hundreds of people must read and cite our work for it to be meaningful is largely poppycock. The average paper is mostly technical, incremental, and obvious in nature. This is absolutely fine and necessary – not everything can be ground breaking and even the breakthroughs must be vetted in projects that are by definition less so. For the average paper then, being regularly cited by 20-50 people is damn good and likely represents the total target audience in that topic area. If you network to those people using social media and traditional conferences, it really isn’t hard to get your paper in their hands.

Moreover, the truly ground breaking stuff will find its audience no matter where it is published. We solve the filter problem every single day, by publically sharing and discussing papers that interest us. Arguing that we need journals to solve this problem ignores the fact that they obscure good papers behind meaningless brands, and more importantly, that scientists are perfectly capable of identifying excellent papers from content alone. You can smell a relevant paper from a mile away – regardless of where it is published! We don’t need to wait for some pie in the sky centralised service to solve this ‘problem’ (although someday once the dust settles i’m sure such things will be useful). Just go out and read some papers that interest you! Follow some interesting people on twitter. Develop a professional network worth having! And don’t buy into the idea that the whole world must read your paper for it to be worth it.

The Privilege Problem 

Ok, so lets say you agree with me to this point. Using some combination of email, social media, alerts, and RSS you feel fully capable of finding relevant stuff for your research (I do!). But your worried about this brave new world where people archive any old rubbish they like and embittered post-docs descend to sneer gleefully at it from the dark recesses of pubpeer. Won’t the new system be subject to favouritism, cults of personality, and the privilege of the elite? As Stephen says, isn’t it likely that popular persons will have their papers reviewed and promoted and all the rest will fade to the back?

The answer is yes and no. As I’ve said many times, there is no utopia. We can and must fight for a better system, but cheaters will always find away[2]. No matter how much transparency and rigor we implement, someone is going to find a loophole. And the oldest of all loopholes is good old human-corruption and hero worship. I’ve personally advocated for a day when data, code, and interpretation are all separate publishable, citable items that each contribute to ones CV. In this brave new world PPPRs would be performed by ‘review cliques’ who build up their reputation as reliable reviewers by consistently giving high marks to science objects that go on to garner acclaim, are rarely retracted, and perform well on various meta-analytic robustness indices (reproducibility, transparency, documentation, novelty, etc). They won’t replace or supplant pre-publication peer review. Rather we can ‘let a million flowers bloom’. I am all for a continuum of rigor, ranging from preregistered, confirmatory research with pre and post peer review, to fully exploratory, data driven science that is simply uploaded to a repository with a ‘use at your peril’ warning’. We don’t need to pit one reform tool against another; the brave new world will be a hybrid mixture of every tool we have at our disposal. Such a system would be massively transparent, but of course not perfect. We’d gain a cornucopia of new metrics by which to weight and reward scientists, but assuredly some clever folks would benefit more than others. We need to be ready when that day comes, aware of whatever pitfalls may bely our brave new science.

Welcome to the Wild West

Honestly though, all this kind of talk is just pointless. We all have our own opinions of what will be the best way to do science, or what will happen. For my own part I am sure some version of this sci-fi depiction is inevitable. But it doesn’t matter because the revolution is here, it’s now, it’s changing the way we consume and produce science right before our very eyes. Every day a new preprint lands on twitter with a massive splash. Just last week in my own field of cognitive neuroscience a preprint on problems in cluster inference for fMRI rocked the field, threatening to undermine thousands of existing papers while generating heated discussion in the majority of labs around the world. The week before that #cingulategate erupted when PNAS published a paper which was met with instant outcry and roundly debunked by an incredibly series of thorough post-publication reviews. A multitude of high-profile fraud cases have been exposed, and careers ended, via anonymous comments on pubpeer. People are out there, right now finding and sharing papers, discussing the ones that matter, and arguing about the ones that don’t. The future is now and we have almost no idea what shape it is taking, who the players are, or what it means for the future of funding and training. We need to stop acting like this is some fantasy future 10 years from now; we have entered the wild west and it is time to discuss what that means for science.

Authors note: In case it isn’t clear, i’m quite glad that Stephen raised the important issue of privilege. I am sure that there are problems to be rooted out and discussed along these lines, particularly in terms of the way PPPR and filtering is accomplished now in our wild west. What I object to is the idea that the future will look like it does now; we must imagine a future where science is radically improved!

[1] I’m not sure if Stephen meant high impact as I don’t know the IF of American Naturalist, maybe he just meant ‘journals I like’.

[2] Honestly this is where we need to discuss changing the hyper-capitalist system of funding and incentives surrounding publication but that is another post entirely! Maybe people wouldn’t cheat so much if we didn’t pit them against a thousand other scientists in a no-holds-barred cage match to the death.

How useful is twitter for academics, really?

Recently I was intrigued by a post on twitter conversion rates (e.g. the likelihood that a view on your tweet results in a click on the link) by journalist Derek Thompson at the Atlantic. Derek writes that although using twitter gives him great joy, he’s not sure it results in the kinds of readership his employers would feel merits the time spent on the service. Derek found that even his most viral tweets only resulted in a conversion rate of about 3% – on par with the click-through rate of east asian display ads (i.e. quite poorly in the media world). Using the recently released twitter metrics, Derek found an average conversion of around 1.5% with the best posts hitting the 3% ceiling. Ultimately he concludes that twitter seems to be great at generating buzz within the twitter-sphere but performs poorly at translating that buzz into external influence.

This struck my curiosity, as it definitely reflected my own experience tweeting out papers and tracking the resultant clicks on the actual paper itself. However, the demands of academia are quite different than that of corporate media. In my experience ‘good’ posts do exactly result in a 2-3% conversion rate, or about 30 clicks on the DOI link for every 1000 views. A typical post I consider ‘successful’ will net about 5-8k views and thus 150-200 clicks. Below are some samples of my most ‘successful’ paper tweets this month, with screen grabs of the twitter analytics for each:

Screenshot 2015-12-04 11.42.45

Screenshot 2015-12-04 11.44.00

Screenshot 2015-12-04 11.44.41

Screenshot 2015-12-04 11.45.29

Sharing each of these papers resulted in a conversion rate of about 2%, roughly in line with Derek’s experience. These are all what I would consider ‘successful’ shares, at least for me, with > 100 engagements each. You can also see that in total, external engagement (i.e., clicking the link to the paper) is below that of ‘internal’ engagement (likes, RTs, expands etc). So it does appear that on the whole twitter shares may generate a lot of internal ‘buzz’ but not necessarily reach very far beyond twitter.

For a corporate sponsor, these conversion rates are unacceptable. But for academics, I would argue the ceiling of the actually interested audience is somewhere around 200 people, which corresponds pretty well with the average paper clicks generated by successful posts. Academics are so highly specialized that i’d wager citation success is really more about making sure your paper falls into the right hands, rather than that of people working in totally different areas. I’d suggest that even for landmark ‘high impact’ papers eventual success will still be predicted more by the adoption of your select core peer group (i.e. other scientists who study vegetarian dinosaur poop in the Himalayan range). In my anecdotal experience, I would say that I more regularly find papers that grab my interest on twitter than any other experience. Moreover, unlike ‘self finds’, it seems to me papers found on twitter are more likely to be outside my immediate wheelhouse – statistics publications, genetics, etc. This is an important, if difficult to quantify type of impact.

In general, we have  to ask what exactly is a 2-3% conversion rate worth? If 200 people click my paper link, are any of them actually reading it? To probe this a bit further I used twitters new survey tool, which recently added support for multiple options, to ask my followers about how often they read papers found on twitter:

Screenshot 2015-12-04 10.58.42.png

As you can see, out of 145 responses more than 50% in total said they read papers found on twitter “Occasionally” (52%) or “Frequently” (30%). If these numbers are at all representative, I think they are pretty reassuring for the academic twitter user. As many as ~45 out of ~150 respondents say they read papers on twitter “frequently” suggesting the service has become a major source for finding interesting papers, at least among its users. All together my take away is that while you shouldn’t expect to beat the general 3% curve, the ability to get your published work on the desks of as many as 50-100 members of your core audience is pretty powerful. This is a more tangible result than ‘engagements’ or conversion rates.

Finally, it’s worth noting that the picture on how this all relates to citation behavior is murky at best. A quick surf of the scientific literature on correlating citation rate and social media exposure is inconclusive at best. Two papers found by Neurocritic are examplars of my impression of this literature, with one claiming a large effect size and the other claiming none at all. In the end I suspect how useful twitter is for sharing research really depends on several factors including your field (e.g. probably more useful for machine learning than organic chemistry) and something i’d vaguely define as ‘network quality’. Ultimately I suspect the rule is quality of followers over quantity; if your end goal is to get your papers in the hands of a round 200 engaged readers (which twitter can do for you) then having a network that actually includes those people is probably worth more than being a ‘Kardashian’ of social media.

 

We the Kardashians are Democratizing Science

I had a good laugh this weekend at a paper published to Genome Biology. Neil Hall, the author of the paper and well-established Liverpool biologist, writes that in the brave new era of social media, there “is a danger that this form of communication is gaining too high a value and that we are losing sight of key metrics of scientific value, such as citation indices.” Wow, what a punchline! According to Neil, we’re in danger of forgetting that tweets and blogposts are, according to him, the worthless gossip of academia. After all, who reads Nature and Science these days?? I know so many colleagues getting big grants and tenure track jobs just over their tweets! Never mind that Neil himself has about 11 papers published in Nature journals – or perhaps we are left to sympathize with the poor, untweeted author? Outside of bitter sarcasm, the article is a fun bit of satire, and I’d like to think charitably that it was aimed not only at ‘altmetrics’, but at the metric enterprise in general. Still, I agree totally with Kathryn Clancy that the joke fails insofar as it seems to be ‘punching down’ at those of us with less established CVs than Neil, who take to social media in order to network and advance our own fledgling research profiles. I think it also belies a critical misapprehension of how social media fits into the research ecosystem common among established scholars. This sentiment is expressed rather precisely by Neil when discussing his Kardashian index:

The Kardashian Index
The Kardashian Index

“In an age dominated by the cult of celebrity we, as scientists, need to protect ourselves from mindlessly lauding shallow popularity and take an informed and critical view of the value we place on the opinion of our peers. Social media makes it very easy for people to build a seemingly impressive persona by essentially ‘shouting louder’ than others. Having an opinion on something does not make one an expert.”

So there you have it. Twitter equals shallow popularity. Never mind the endless possibilities of having seamless networked interactions with peers from around the world. Never mind sharing the latest results, discussing them, and branching these interactions into blog posts that themselves evolve into papers. Forget entirely that without this infosphere of interaction, we’d be left totally at the whims of Impact Factor to find interesting papers among the thousands published daily. What it’s really all about is building a “seemingly impressive persona” by “shouting louder than others”. What then does constitute effective scientific output, Neil? The answer it seems – more high impact papers:

“I propose that all scientists calculate their own K-index on an annual basis and include it in their Twitter profile. Not only does this help others decide how much weight they should give to someone’s 140 character wisdom, it can also be an incentive – if your K-index gets above 5, then it’s time to get off Twitter and write those papers.”

Well then, I’m glad we covered that. I’m sure there were many scientists or scholars out there who amid the endless cycle of insane job pressure, publish or perish horse-racing, and blood feuding for grants thought, ‘gee I’d better just stop this publishing thing entirely and tweet instead’. And likewise, I’m sure every young scientist looks at ‘Kardashians’ and thinks, ‘hey I’d better suspend all critical thinking, forget all my training, and believe everything this person says’. I hope you can feel me rolling my eyes.  Seriously though – this represents a fundamental and common misunderstanding of the point of all this faffing about on the internet. Followers, impact, and notoriety are all poorly understood side-effects of this process; they are neither the means nor goal. And never mind those less concrete (and misleading) contributions like freely shared code, data, or thoughts – the point here is to blather and gossip!

While a (sorta) funny joke, it is this point that is done the most disservice by Neil’s article. We (the Kardashians) are democratizing science. We are filtering the literally unending deluge of papers to try and find the most outrageous, the most interesting, and the most forgotten, so that they can see the light of day beyond wherever they were published and forgotten. We seek these papers to generate discussion and to garner attention where it is needed most. We are the academy’s newest, first line of defense, contextualizing results when the media runs wild with them. We tweet often because there is a lot to tweet, and we gain followers because the things we tweet are interesting. And we do all of this without the comfort of a lofty CV or high impact track record, with little concrete assurance that it will even benefit us, all while still trying to produce the standard signs of success. And it may not seem like it now – but in time it will be clear that what we do is just as much a part of the scientific process as those lofty Nature papers. Are we perfect? No. Do we sometimes fall victim to sensationalism or crowd mentality? Of course – we are only fallible human beings, trying to find and create utility within a new frontier. We may not be the filter science deserves – but we are the one it needs. Wear your Kardshian index with pride.

Twitter Follow-up: Can MVPA Invalidate Simulation Theory?

Thanks to the wonders of social media, while I was out grocery shopping I received several interesting and useful responses to my previous post on the relationship between multivariate pattern analysis and simulation theory. Rather than try and fit my responses into 140 characters, I figured i’d take a bit more space here to hash them out. I think the idea is really enhanced by these responses, which point to several findings and features of which I was not aware. The short answer seems to be, no MVPA does not invalidate simulation theory (ST) and may even provide evidence for it in the realm of motor intentions, but that we might be able to point towards a better standard of evidence for more exploratory applications of ST (e.g. empathy-for-pain). An important point to come out of these responses as one might expect, is that the interpretation of these methodologies is not always straightforward.

I’ll start with Antonia Hamilton’s question, as it points to a bit of literature that speaks directly to the issue:

antonio_reply

Antonia is referring to this paper by Oosterhof and colleagues, where they directly compare passive viewing and active performance of the same paradigm using decoding techniques. I don’t read nearly as much social cognition literature as I used to, and wasn’t previously aware of this paper. It’s really a fascinating project and I suggest anyone interested in this issue read it at once (it’s open access, yay!). In the introduction the authors point out that spatial overlap alone cannot demonstrate equivalent mechanisms for viewing and performing the same action:

Numerous functional neuroimaging studies have identified brain regions that are active during both the observation and the execution of actions (e.g., Etzel et al. 2008; Iacoboni et al. 1999). Although these studies show spatial overlap of frontal and parietal activations elicited by action observation and execution, they do not demonstrate representational overlap between visual and motor action representations. That is, spatially overlapping activations could reflect different neural populations in the same broad brain regions (Gazzola and Keysers 2009; Morrison and Downing 2007; Peelen and Downing 2007b). Spatial overlap of activations per se cannot establish whether the patterns of neural response are similar for a given action (whether it is seen or performed) but different for different actions, an essential property of the “mirror system” hypothesis.”

They then go on to explain that while MVPA could conceivably demonstrate a simulation-like mechanism (i.e. a common neural representation for viewing/doing), several previous papers attempting to show just that failed to do so. The authors suggest that this may be due to a variety of methodological limitations, which they set out to correct for in their JNPhys publication. Oosterhof et al show that clusters of voxels located primarily in the intraparietal and superior temporal sulci encode cross-modal information, that is code similar information both when viewing and doing:

Click to go to PDF.
From Oosterhof et al, showing combined classification accuray for (train see, test do; train do, test see).

Essentially Oosterhof et al trained their classifier on one modality (see or do) , tested the classifier on the opposite modality in another session, and then repeated this procedure for all possible combinations of session and modality (while appropriately correcting for multiple comparisons). The map above represents the combined classification accuracy from both train-test combinations; interestingly in the supplementary info they show that the maps do slightly differ depend on what was trained:

Click to go to SI.
From supplementary info, A shows classifier trained on see, tested on do, B shows the opposite.

Oosterhof and colleagues also investigate the specificity of information for particular gestures in a second experiment, but for our purposes lets focus on just the first. My first thought is that this does actually provide some evidence for a simulation theory of understanding motor intentions. Clearly there is enough information in each modality to accurately decode the opposite modality: there are populations of neurons encoding similar information both for action execution and perception. Realistically I think this has to be the minimal burden of proof needed to consider an imaging finding to be evidence for simulation theory. So the results of Oosterhof et al do provide supporting evidence for simulation theory in the domain of motor intentions.

Nonetheless, the results also strengthen the argument that more exploratory extentions of ST (like empathy-for-pain) must be held to a similar burden of proof before generalization in these domains is supported. Simply showing spatial overlap is not evidence of simulation, as Oosterhof themselves argue. I think it is interesting to note the slight spatial divergence between the two train-test maps (see on do, do on see). While we can obviously identify voxels encoding cross-modality information, it is interesting that those voxels do not subsume the entirety of whatever neural computation relates these two modalities; each has something unique to predict in the other. I don’t think that observation invalidates simulation theory, but it might suggest an interesting mechanism not specified in the ‘vanilla’ flavor of ST. To be extra boring, it would be really nice to see an independent replication of this finding, since as Oosterhof themselves point out, the evidence for cross-modal information is inconsistent across studies. Even though the classifier performs well above chance in this study,  it is also worth noting that the majority of surviving voxels in their study show somewhere around 40-50% classification accuracy, not exactly gangbusters. It would be interesting to see if they could identify voxels within these regions that selectively encode only viewing or performing; this might be evidence for a hybrid-theory account of motor intentions.

leoreply

Leonhard’s question is an interesting one that I don’t have a ready response for. As I understand it, the idea is that demonstrating no difference of patterns between a self and other-related condition (e.g. performing an action vs watching someone else do it) might actually be an argument for simulation, since this could be caused by that region using isomorphic computations for both conditions. This an interesting point – i’m not sure what the status of null findings is in the decoding literature, but this merits further thought.

The next two came from James Kilner and Tal Yarkoni. I’ve put them together as I think they fall under a more methodological class of questions/comments and I don’t feel quite experienced enough to answer them- but i’d love to hear from someone with more experience in multivariate/multivoxel techniques:

kilner_reply

talreply

James Kilner asks about the performance of MVPA in the case that the pattern might be spatially overlapping but not identical for two conditions. This is an interesting question and i’m not sure I know the correct answer; my intuition is that you could accurately discriminate both conditions using the same voxels and that this would be strong evidence against a simple simulation theory account (spatial overlap but representational heterogeneity).

Here is more precise answer to James’ question from Sam Schwarzkopf, posted in the comments of the original post:

2. The multivariate aspect obviously adds sensitivity by looking at pattern information, or generally any information of more than one variable (e.g. voxels in a region). As such it is more sensitive to the information content in a region than just looking at the average response from that region. Such an approach can reveal that region A contains some diagnostic information about an experimental variable while region B does not, even though they both show the same mean activation. This is certainly useful knowledge that can help us advance our understanding of the brain – but in the end it is still only one small piece in the puzzle. And as both Tal and James pointed out (in their own ways) and as you discussed as well, you can’t really tell what the diagnostic information actually represents.
Conversely, you can’t be sure that just because MVPA does not pick up diagnostic information from a region that it therefore doesn’t contain any information about the variable of interest. MVPA can only work as long as there is a pattern of information within the features you used.

This last point is most relevant to James’ comment. Say you are using voxels as features to decode some experimental variable. If all the neurons with different tuning characteristics in an area are completely intermingled (like orientation-preference in mouse visual cortex for instance) you should not really see any decoding – even if the neurons in that area are demonstrably selective to the experimental variable.

In general it is clear that the interpretation of decoded patterns is not straightforward- it isn’t clear precisely what information they reflect, and it seems like if a region contained a totally heterogeneous population of neurons you wouldn’t pick up any decoding at all. With respect to ST,  I don’t know if this completely invalidates our ability to test predictions- I don’t think one would expect such radical heterogeneity in a region like STS, but rather a few sub-populations responding selectively to self and other, which MVPA might be able to reveal. It’s an important point to consider though.

Tal’s point is an important one regarding the different sources of information that GLM and MVPA techniques pick up. The paper he refers to by Jimura and Poldrack set out to investigate exactly this by comparing the spatial conjunction and divergent sensitivity of each method. Importantly they subtracted the mean of each beta-coefficient from the multivariate analysis to insure that the analysis contained only information not in the GLM:

pold_mvpa

As you can see in the above, Jimura and Poldrack show that MVPA picks up a large number of voxels not found in the GLM analysis. Their interpretation is that the GLM is designed to pick up regions responding globally or in most cases to stimulation, whereas MVPA likely picks up globally distributed responses that show variance in their response. This is a bit like the difference between functional integration and localization; both are complementary to the understanding of some cognitive function. I take Tal’s point to be that the MVPA and GLM are sensitive to different sources of information and that this blurs the ability of the technique to evaluate simulation theory- you might observe differences between the two that would resemble evidence against ST (different information in different areas) when in reality you would be modelling altogether different aspects of the cognition. edit: after more discussion with Tal on Twitter, it’s clear that he meant to point out the ambiguity inherent in interpreting the predictive power of MVPA; by nature these analyses will pick up a lot of confounding a causal noise- arousal, reaction time, respiration, etc, which would be excluded in a GLM analysis. So these are not necessarily or even likely to be “direct read-outs” of representations, particularly to the extent that such confounds correlate with the task. See this helpful post by neuroskeptic for an overview of one recent paper examining this issue. See here for a study investigating the complex neurovascular origins of MVPA for fMRI. 

Thanks sincerely for these responses, as it’s been really interesting and instructive for me to go through these papers and think about their implications. I’m still new to these techniques and it is exciting to gain a deeper appreciation of the subtleties involved in their interpretation. On that note, I must direct you to check out Sam Schwarzkopf’s excellent reply to my original post. Sam points out some common misunderstandings (of which I am perhaps guilty of several) regarding the interpretation of MVPA/decoding versus GLM techniques, arguing essentially that they pick up much of the same information and can both be considered ‘decoding’ in some sense, further muddying their ability to resolves debates like that surrounding simulation theory.

My response to Carr and Pinker on Media Plasticity

Our ongoing discussion regarding the moral panic surrounding Nicolas Carr’s book The Shallows continues over at Carr’s blog today, with his recent response to Pinker’s slamming the book. I maintain that there are good and bad (frightening!!) things in both accounts. Namely, Pinker’s stolid refusal to acknowledge the research I’ve based my entire PhD on, and Carr’s endless fanning of the one-sided moral panic.

Excerpt from Carr’s Blog:

Steven Pinker and the Internet

And then there’s this: “It’s not as if habits of deep reflection, thorough research and rigorous reasoning ever came naturally to people.” Exactly. And that’s another cause for concern. Our most valuable mental habits – the habits of deep and focused thought – must be learned, and the way we learn them is by practicing them, regularly and attentively. And that’s what our continuously connected, constantly distracted lives are stealing from us: the encouragement and the opportunity to practice reflection, introspection, and other contemplative modes of thought. Even formal research is increasingly taking the form of “power browsing,” according to a 2008 University College London study, rather than attentive and thorough study. Patricia Greenfield, a professor of developmental psychology at UCLA, warned in a Science article last year that our growing use of screen-based media appears to be weakening our “higher-order cognitive processes,” including “abstract vocabulary, mindfulness, reflection, inductive problem solving, critical thinking, and imagination.”

As someone who has enjoyed and learned a lot from Steven Pinker’s books about language and cognition, I was disappointed to see the Harvard psychologist write, in Friday’s New York Times, a cursory op-ed column about people’s very real concerns over the Internet’s influence on their minds and their intellectual lives. Pinker seems to dismiss out of hand the evidence indicating that our intensifying use of the Net and related digital media may be reducing the depth and rigor of our thoughts. He goes so far as to assert that such media “are the only things that will keep us smart.” And yet the evidence he offers to support his sweeping claim consists largely of opinions and anecdotes, along with one very good Woody Allen joke.

Right here I would like to point out the kind of leap Carr is making. I’d really like a closer look at the supposed evidence demonstrating  “our intensifying use of the Net and related digital media may be reducing the depth and rigor of our thoughts.” This is a huge claim! How does one define the ‘depth’ and ‘rigor’ of our thoughts? I know of exactly one peer-reviewed high impact paper demonstrating a loss of specifically executive function in heavy-media multi-taskers. While there is evidence that generally speaking, multi-tasking can interfere with some forms of goal-directed activity, I am aware of no papers directly linking specific forms of internet behavior to a drop in executive function. Furthermore, the HMM paper included in it’s measure of multi-tasking ‘watching tv’, ‘viewing funny videos’, and ‘playing videogames’. I don’t know about you, but for me there is definitely a difference between ‘work’ multitasking, in which I focus and work through multiple streams, and ‘play’ multitasking, in which I might casually surf the net while watching TV. The second claim is worse- what exactly is ‘depth’? And how do we link it to executive functioning?

Is Carr claiming people with executive function deficits are incapable or impaired in thinking creatively? If it takes me 10 years to publish a magnum opus, have I thought less deeply than the author that cranks out a feature length popular novel every 2 years? Depth involves a normative judgment of what separates ‘good’ thinking from ‘bad’ thinking, and to imply there is some kind of peer-reviewed consensus here is patently false. In fact, here is a recent review paper on fmri creativity research (is this depth?) indicating that the existing research is so incredibly disparate and poorly defined as to be untenable. That’s the problem with Carr’s claims- he oversimplifies both the diversity of internet usage and the existing research on executive and creative function. To be fair to Carr, he does go on to do a fair job of dismantling Pinker’s frighteningly dogmatic rejection of generalizable brain plasticity research:

One thing that didn’t surprise me was Pinker’s attempt to downplay the importance of neuroplasticity. While he acknowledges that our brains adapt to shifts in the environment, including (one infers) our use of media and other tools, he implies that we need not concern ourselves with the effects of those adaptations. Because all sorts of things influence the brain, he oddly argues, we don’t have to care about how any one thing influences the brain. Pinker, it’s important to point out, has an axe to grind here. The growing body of research on the adult brain’s remarkable ability to adapt, even at the cellular level, to changing circumstances and new experiences poses a challenge to Pinker’s faith in evolutionary psychology and behavioral genetics. The more adaptable the brain is, the less we’re merely playing out ancient patterns of behavior imposed on us by our genetic heritage.

Here is my response, posted on Nick’s blog:

Hi Nick,

As you know from our discussion at my blog, I’m not really a fan of the extreme views given by either you or Pinker. However, I applaud the thorough rebuttal you’ve given here to Stephen’s poorly researched response. As someone doing my PhD in neuroplasticity and cognitive technology, it absolutely infuriated me to see Stephen completely handwave away a decade of solid research showing generalizable cognitive gains from various forms of media-practice. To simply ignore findings from, for example the Bavalier lab, that demonstrate reliable and highly generalizable cognitive and visual gains and plasticity is to border on the unethically dogmatic.

Pinker isn’t well known for being flexible within cognitive science however; he’s probably the only person even more dogmatic about nativist modularism than Fodor. Unfortunately, Stephen enjoys a large public following and his work has really been embraced by the anti-religion ‘brights’ movement. While on some levels I appreciate this movement’s desire to promote rationality, I cringe at how great scholars like Dennett and Pinker seem totally unwilling to engage with the expanding body of research that casts a great deal of doubt on the 1980’s era cogsci they built their careers on.

So I give you kudos there. I close as usual, by saying that you’re presenting a ‘sexy’ and somewhat sensationalistic account that while sure to sell books and generate controversy, is probably based more in moral panic than sound theory. I have no doubt that the evidence you’ve marshaled demonstrates the cognitive potency of new media. Further, I’m sure you are aware of the heavy-media multitasking paper demonstrating a drop in executive functioning in HMMs.

However, you neglect in the posts I’ve seen to emphasize what those authors clearly did: that these findings are not likely to represent a true loss of function but rather are indicators of a shift in cognitive style. Your unwillingness to declare the normative element in your thesis regarding ‘deep thought’ is almost as chilling as Pinker’s total refusal to acknowledge the growing body of plasticity research. Simply put, I think you are aware that you’ve conflated executive processing with ‘deep thinking’, and are not really making the case that we know to be true.

Media is a tool like any other. It’s outcome measures are completely dependent on how we use it and our individual differences. You could make this case quite well with your evidence, but you seem to embrace the moral panic surrounding your work. It’s obvious that certain patterns, including the ones probably driving your collected research, will play on our plasticity to create cognitive differences. Plasticity is limited however, and you really don’t play on the most common theme in mental training literature: balance and trade-off. Your failure to acknowledge the economical and often conservative nature of the brain forces me to lump your work in with the decade that preceded your book, in which it was proclaimed that violent video games and heavy metal music would rot our collective minds. These things didn’t happen, except in those who where already at high risk, and furthermore they produced unanticipated cognitive gains. I think if you want to be on the ‘not wrong’ side of history, you may want to introduce a little flexibility to your argument. I guess if it makes you feel better, for many in the next generation of cognition researchers, it’s already too late for a dogmatic thinker like Pinker.

Final thoughts?

Snorkeling ’the shallows’: what’s the cognitive trade-off in internet behavior?

I am quite eager to comment on the recent explosion of e-commentary regarding Nicolas Carr’s new book. Bloggers have already done an excellent job summarizing the response to Carr’s argument. Further, Clay Shirky and Jonah Lehrer have both argued convincingly that there’s not much new about this sort of reasoning. I’ve also argued along these lines, using the example of language itself as a radical departure from pre-linguistic living. Did our predecessors worry about their brains as they learned to represent the world with odd noises and symbols?

Surely they did not. And yet we can also be sure that the brain underwent a massive revolution following the acquisition of language. Chomsky’s linguistics would of course obscure this fact, preferring us to believe that our linguistic abilities are the amalgation of things we already possessed: vision, problem solving, auditory and acoustic control. I’m not going to spend too much time arguing against the modularist view of cognition however; chances are if you are here reading this, you are already pretty convinced that the brain changes in response to cultural adaptations.

It is worth sketching out a stock Chomskyian response however. Strict nativists, like Chomsky, hold that our language abilities are the product of an innate grammar module. Although typically agnostic about the exact source of this module (it could have been a genetic mutation for example), nativists argue that plasticity of the brain has no potential other than slightly enhancing or decreasing our existing abilities. You get a language module, a cognition module, and so on, and you don’t have much choice as to how you use that schema or what it does. The development of anguage on this view wasn’t something radically new that changed the brain of its users but rather a novel adaptation of things we already and still have.

To drive home the point, it’s not suprising that notable nativist Stephen Pinker is quoted as simply not buying the ‘changing our brains’ hypothesis:

“As someone who believes both in human nature and in timeless standards of logic and evidence, I’m skeptical of the common claim that the Internet is changing the way we think. Electronic media aren’t going to revamp the brain’s mechanisms of information processing, nor will they supersede modus ponens or Bayes’ theorem. Claims that the Internet is changing human thought are propelled by a number of forces: the pressure on pundits to announce that this or that “changes everything”; a superficial conception of what “thinking” is that conflates content with process; the neophobic mindset that “if young people do something that I don’t do, the culture is declining.” But I don’t think the claims stand up to scrutiny.”

Pinker makes some good points- I agree that a lot of hype is driven by the kinds of thinking he mentions. Yet, I do not at all agree that electronic media cannot and will not revamp our mechanisms for information processing. In contrast to the nativist account, I think we’ve better reason than ever to suspect that the relation between brain and cognition is not 1:1 but rather dynamic, evolving with us as we develop new tools that stimulate our brains in unique and interesting ways.

The development of language massively altered the functioning of our brain. Given the ability to represent the world externally, we no longer needed to rely on perceptual mechanisms in the same way. Our ability to discriminate amongst various types of plant, or sounds, is clearly sub-par to that of our non-linguistic brethren. And so we come full circle. The things we do change our brains. And it is the case that our brains are incredibly economical. We know for example that only hours after limb amputation, our somatosensory neurons invade the dormant cells, reassigning them rather than letting them die off. The brain is quite massively plastic- Nicolas Carr certainly gets that much right.

Perhaps the best way to approach this question is with an excerpt from social media. I recently asked of my fellow tweeps,

To which an astute follower replied:

Now, I do realize that this is really the central question in the ‘shallows’ debate. Moving from the basic fact that our brains are quite plastic, we all readily accept that we’re becoming the subject of some very intense stimulation. Most social media, or general internet users, shift rapidly from task to task, tweet to tweet. In my own work flow, I may open dozens and dozens of tabs, searching for that one paper or quote that can propel me to a new insight. Sometimes I get confused and forget what I was doing. Yet none of this interferes at all with my ‘deep thinking’. Eventually I go home and read a fantastic sci-fi book like Snowcrash. My imagination of the book is just as good as ever; and I can’t wait to get online and start discussing it. So where is the trade-off?

So there must be a trade-off, right? Tape a kitten’s eyes shut and its visual cortex is re-assigned to other sensory modalities. The brain is a nasty economist, and if we’re stimulating one new thing we must be losing something old. Yet what did we lose with language? Perhaps we lost some vestigial abilities to sense and smell. Yet we gained the power of the sonnet, the persuasion of rhetoric, the imagination of narrative, the ability to travel to the moon and murder the earth.

In the end, I’m just not sure it’s the right kind of stimulation. We’re not going to lose our ability to read in fact, I think I can make an extremely tight argument against the specific hypothesis that the internet robs us of our ability to deep-think. Deep thinking is itself a controversial topic. What exactly do we mean by it? Am I deep thinking if I spend all day shifting between 9 million tasks? Nicolas Carr says no, but how can he be sure those 9 million tasks are not converging around a central creative point?

I believe, contrary to Carr, that internet and social media surfing is a unique form of self stimulation and expression. By interacting together in the millions through networks like twitter and facebook, we’re building a cognitive apparatus that, like language, does not function entirely within the brain. By increasing access to information and the customizability of that access, we’re ensuring that millions of users have access to all kinds of thought-provoking information. In his book, Carr says things like ‘on the internet, there’s no time for deep thought. it’s go go go’. But that is only one particular usage pattern, and it ignores ample research suggesting that posts online may in fact be more reflective and honest than in-person utterances (I promise, I am going to do a lit review post soon!)

Today’s internet user doesn’t have to conform to whatever Carr thinks is the right kind of deep-thought. Rather, we can ‘skim the shallows’ of twitter and facebook for impressions, interactions, and opinions. When I read a researcher, I no longer have to spend years attending conferences to get a personal feel for them. I can instead look at their wikipedia, read the discussion page, see what’s being said on twitter. In short, skimming the shallows makes me better able to choose the topics I want to investigate deeply, and lets me learn about them in whatever temporal pattern I like. Youtube with a side of wikipedia and blog posts? Yes please. It’s a multi-modal whole brain experience that isn’t likely to conform to ‘on/off’ dichotomies. Sure, something may be sacrificed, but it may not be. It might be that digital technology has enough of the old (language, vision, motivation) plus enough of the new that it just might constitute or bring about radically new forms of cognition. These will undoubtably change or cognitive style, perhaps obsoleting Pinker’s Bayesian mechanisms in favor of new digitally referential ones.

So I don’t have an answer for you yet ToddStark. I do know however, that we’re going to have to take a long hard look at the research review by Carr. Further, it seems quite clear that there can be no one-sided view of digital media. It’s not anymore intrinsically good or bad than language. Language can be used to destroy nations just as it can tell a little girl a thoughtful bed time story. If we’re to quick to make up our minds about what internet-cognition is doing to our plastic little brains, we might miss the forest for the trees. The digital media revolution gives us the chance to learn just what happens in the brain when its’ got a shiny new tool. We don’t know the exact nature of the stimulation, and finding out is going to require a look at all the evidence, for and against. Further, it’s a gross oversimplification to talk about internet behavior as ‘shallow’ or ‘deep’. Research on usage and usability tells us this; there are many ways to use the internet, and some of them probably get us thinking much deeper than others.