On not noticing the major problems with a study – and what it may say about the current incentive structure in Sciences.
I had an …interesting…experience the other day. I regularly go to the neuro journal club. Not because I do brains, but because it has lots of interesting members, and is quite active. (So, I felt lonely and under stimulated. Can even happen to introverts).
I had decided to present a paper I didn’t get to in the spring, so I had it all prepared, although as I read it I wasn’t sure what possessed me to present an fMRI study, as I’m not at all up on the methodological intricacies.
But, it was an interesting topic. They were comparing responses to real objects with responses to photographs of the same real objects. We spend a lot of time in psychology – from the neuro level up to the social level, and certainly across the developmental span – exposing people to images. For lots of good reasons, of course, like control, or ease, or well we don’t know shit about this, so let’s start simply. Pictures are alsopsychologically potent. Just look at the popularity of films, photographs, pornography. So, it is certainly not wrong to use pictures to probe minds.
But, pictures have some odd features also. We readily read them as 3-D and get fooled in fun and interesting ways that would not quite be possible (at least not for very long) in a 3-D world.
I began the talk with a picture of the Penrose triangle, and what Nicholas Humphrey call the Gregunddrum (after Richard Gregory who constructed it). (Go here to the Wikipedia page, and you’ll see both)
Much papers have been published discussing visual illusions (they do tell us something about the visual system), but, perhaps they are illusions because our visual systems have developed to deal with a 3-dimensional world. (My buddy Andrew has lots to say about that)
The researchers made use of new fMRI technology developed for research on haptic perception – a way in which you can tilt the head so you can actually interact with real objects. What they tesedt is called fMRA – an evident adaptation of the BOLD signal when you repeat a stimulus. You repeat a stimulus, the BOLD response gets attenuated. This has been replicated in a lot of ways, with all sorts of stimuli – all 2D, but with shadings, and implied textures, etc, and has been used to probe the visual system. In their experiment, they present participants with the real object, or a picture of that exact same object. Sometimes repeated (once) and sometimes not.
What they found, from my amateur understanding of the whole thing, was that the 2D pictures generated the expected fMRA, but the 3D items did not. And, then they proceeded to discuss why this may be so.
My friends, who actually do brains, and actually do studies with fMRI ripped it to pieces. They had collected their 13 subjects using different coils – but aggregated across (there were only 3 in the other coil, but still). The objects were of metal – most likely fMRI safe, but still metal, which is known to influence the recording. Why did they only do one repeat, and not multiple repeats? How come they did not control better where the participants kept their eyes (they claimed they could not use an eye tracker in that coil – fair enough – but where they kept their eyes may be very important). Was it really a good idea to only analyze the visual streams when maybe what is an interesting difference between pictures and objects may be in some motor area, indicating some kind of affordance (in the Gibsoninan way – the drinkability, the grabability, the interactability etc), and couldn’t they have come up with more interesting objects that would be more interesting to look at or perhaps grab. And, their conclusions, their discussion about what it could be was just implausible.
Lots of this I could not have noticed. I don’t do brains, as I said. And, the critique was of methodology; a background knowledge that you have if you do this research, but is invisible to you if you don’t. I’m highly aware of how important knowledge of methodology is, of course, and have insisted in my teaching that students actually do hands on work, because the problems – oh those devils – are in the details, not in the narratives. You can claim you have evidence for something, but you have to scrutinize the measures.
And, yes, it was a bit uncomfortable. I had – unwittingly – picked a paper that had an interesting topic, but was not very well executed and reasoned. But it was also very very very educational. For me and for the other participants in the journal club who may not yet be as sophisticated when it comes to this type of methodology.
Another interesting observation came up – which is in part why I put this story on this blog. I had originally prepared this paper together with another paper that discussed ideas about how to fix science (focus there was on how to deal with replicability and the file-drawer) – which resulted in a very interesting discussion on the issues facing science today.
As one of the researchers suggested, this paper may very well be a good illustration of the problems with the current incentive system. The speculation (and, of course, this is speculative, because we did not have the scientists present) is that this was a study that didn’t quite work out as expected – we all have tons of those: A pilot, that raised interesting questions, but where you really needed to fix the problems, and proceed with a new version probing the phenomenon further. Instead, a story is spun that could possibly be consistent with their weak results – that may not even have been the initial story, who knows. But, one that follows the current science narrative – looky here at our surprising result – published in a channel that Nature has open for kind of…lesser research. Because you must publish, and you must follow the narrative.
There would be a couple of paths that, for the purpose of science, would have been much more interesting. Either going back to the drawing board, learn from the failures (which we do all the time anyway), and come up with improved research. Or, possibly, if it must be published (and why not), publish it as a work in progress, with an enumeration of the problems, to be discussed and improved upon.
As it was, an unwitting person like me (and, of course, I’m a highly trained unwitting person, but just in a sister area – but with an interest in incorporating results from the brain sciences) takes the story to mean something that, well, it didn’t quite do. I needed the input from 4 of my more knowledgeable colleagues to understand the shortcomings. This is a problem, because it isn’t always that you have access to knowledgeable colleagues to help you go through and point out the issues with published results.