Björn Brembs tweeted in this nature article on six red flags for problematic work. The article regards the irreproducibility of a large number of results in cancer research, and is very much directly aimed at their specific methodologies at least for some of the six flags. (I don’t even know what a western blot looks like. I keep imagining cacti and canyons).
Might this recentarticle from Neurobonkers (covering work from Greg Francis) suggest similar key points to look for?
And, just for that, I give you my remote associate: Six Flags Magic Mountain (out west)
Remember that Nature article with the tabloid type headline?
Well, they added a clarification. (Via Rolf Zwaan on twitter). Scroll down. It is at the bottom of the articler, but before the comments. (Comments have grown, and are worth a read.)
Wanted to link in a post by Stephen Curry from his “Reciprocal Space” blog. Impact factors. I won’t summarize, as I’d rather people just click over there, read the post, as well as Drug Monkey’s different stance on Stephen’s position. (I’m not quite sure where I would fall, as I figure it needs a model to decide which option has more, um, impact. Oh, never mind).
But, I will link in the San Francisco Declaration on Research Assessment (which is on Stephen’s post too). Just so there will be multiple routes towards it.
(On Edit: Adding this other commentary on the Declaration on Research Assessment, from Science Mag)
I completely resonate with Bobbie Spellmans enthusiasm! And, there’s been a lot of tweets about the mechanics (not as easy as you would think).
Here is a summary of the concerns, and answers, from Daniel Simons.
I am very interested in the outcome of this. I haven’t used the results of verbal overshadowing, but some of my old research buds did, and it shows up in a lot of reasoning in other research at my faculty.
How should we fix science? Oh, I have no idea, really. In fact, I’m not sure if it is so much fixing as it is some kind of reset, heating up for simulated annealing to take place. Finding the nudge to push it into another strange attractor. Start a revolution, or threaten to. Go disruptive in Clay Christensen’s sense.
A few posts and articles I came across the past few days:
Scholarly kitchens “are we in a rut?” post suggests that science is just becoming more of the same. Like Goldman Sachs we have become too big to fail (but not good enough to wring good money out of others hands into our own coffers). We are bloated, cowardly, staid, boring, and simply working in some kind of markov process fashion to keep at a fast paced but increasingly dull equilibrium.
Daniel Simons had a question out on Google + setting up a hypothetical decision about who, of two scientists, should be considered having more impact? The one with the grants, the one without, or neither one?
Sanjay Srivastava (among others) suggested 3, for the reason that science really shouldn’t be about that kind of production (and pulling in the cash), but about investigation, which is really a risky area, with no guarantee of pay-off ever (albeit it can be big when you have it). Which I agree with (of course, disclosure, I’m in Sweden, I am crappy at bringing in money, they seem to tolerate my very low frequency of publication and my tenacious wresting of the curriculum so it will fit my purposes).
Another summary article by Bruce Bower of the problems with psychology as a science was tweeted in (first in my stream) by Keith Laws – the toothbrush problem. (You know, nobody wants to share. Eeeeew, your germs are all over my theory!).
Speaking of Keith, can just as well link in his post on Whether or not psychology is a science or is just seductively posing as one. (Now, why do I suddenly think of Kink’s Lola?).
Another interesting angle was brought up in a guest-blog by Patric Rabbit on Dorothy Bishop’s blog. Some researchers claimed that intelligence must have gone down since Victorian times. The evidence, they said, was that reaction times (RT’s Pre Twitter) have become longer, and supposedly RT is related to intelligence.Except, of course, this is a case of apparatus improving. The earlier RT’s just had that much more uncertainty around them, and in fact kind of hard cuts, so it is just not on a comparable scale. But, that is only clear because of someone sharing their long laboratory experience. All that tacit knowledge which is not published. That makes it very difficult to be cumulative in any way. (I’d say there’s other reasons to doubt the conclusion anyway, but, then, my forays into intelligence are just slightly beyond what is minimally required for a psychology lecturer, though I won’t make the mistake of claiming that Cozma Shalizis lovely grumpy critique of the statistics behind the g-construct invalidates the entire notion of cognitive capacity.).
And, this just makes me reflect on the article I linked as kind of a last throwaway (scroll to bottom) in in this earlier post, on game theory musings about science: Paula Stephans analysis of the perverse incentives in Science. At least in the US,
I’m going through Nassim Taleb’s Anti Fragile* right now. On audio, thru the night, so I acquire it in a mosaic, non-linear fashion (perhaps fitting for a book on non-linearity). The ideas that I think could be useful to consider for science (although, could we get the leaders and the funders in on this?), is optionality and the bar-bell strategy. Research, if you want to make it well (focusing on actually getting to understand nature better) is more akin to start-ups or to the various types of arts, from writing, music, painting, etc. High risk, high failure, possibly really big pay-offs. And, you can’t change that. Sure, there are spaces of ‘normal science’ where there is something that you can focus on and explore. But, lots of times, you have no idea. Will the next song be a hit? Is the book you write a Harry Potter, or something milder? Are you cracking natures code open a bit more with the next set of experiments, or will nature just say (metaphorically) Nope?
But, the thing is, if you want to get in on it, don’t try to forecast which one will pay off. Bet on all of them. A little bit. Let scientists tinker, see if the codes crack open. That is what we know how to do anyway. Then, if something opens big, throw money at it. That is just fine. But, keep the budget for the tinkering.
And, allow the tinkering. The bar-bell. Security on one side, wild risk on the other. Don’t hold the scientists, or authors, or writers, or musicians to earning their keep through results, because you just don’t know (that will just make people nervous, and go after the sure bet, and it leads to all those control things, and impact factors and all that which impedes creativity and curiosity). Let them (me!) have a kind of sinecure. A job. Something to do which is half way productive, but not too taxing. I actually don’t think teaching is too bad, would it just be kept to the kind of level where you also have time to think about the real big things. And, consider that not everybody can be like EO Wilson, who advocates a weekly schedule that effectively precludes being an active parent. (Of course, I think all this active parenting is really status signaling, because the kids really don’t need that much interference, or else we would have died out as a species long ago).
Really. If you want to make room for the possibility of interesting science done right, allow for scientists to do science right, and realize that you cannot judge them on publication and positive results only, but perhaps also on how well they produced those null results. (Well, closing off large areas of parameter space is important, although few people recognize that hidden heroism). The wildest, most enduring art and science are flukes. You can prepare for the flukes. Just don’t expect them.
* People bitch about Taleb, but most of the critique I’ve seen radically misses the point and seems more to be upset that he is pissing on their prediction parade, and go all ad hominem. I think he basically has a very important point, and the quibbles are about unimportant details. Plus, I seem to have a really soft spot for curmudgeons, as long as they are intellectually interesting. In fact, I think I have linked a number of curmudgeons here. Just complements my agreeable, but mildly subversive persona.
The morning after the Monkey House post chronicling science skirmishes, I got a tweet reply from D Budding, which I glanced at on the bus, seeing it had a reference to David Lynch’s cartoon “The angriest dog in the world“. A main stay of LA weekly, which I religiously got every week when I lived there. I didn’t read it carefully at the time, so I immediately thought it was a commentary on the recent blowup. Well, odd are the associative paths, but kinda apt.
It really was in reference to something completely different – a comment about Groening’s “Life in Hell” – an equal mainstay (and, which made me want to use “Jeff” and “Akbar” as my name stimuli in my IAT set, except that Jeff is not exactly a swedish name).
But, I’d thought I share. Here is the Angriest Dog In The World
Daniël Lakens continued his English Blogging with this piece on statistics and psychology. It is a yearning piece. Statistics, why are you so coy? And, then, my favorite Statistician gets a hold of it and answers. May this be the beginning of a beautiful friendship!
This time in Nijmegen. I so want to go. But, I want to go to Stockholm too. And, I’m self-financed (no damned granting agencies wants to send me on my wild goose chases, when I’m not presenting).
If you can, do.
My Twitter-bud Daniël Lakens has a blog. WHY WASN’T I INFORMED? Ahem. (as you were).
And, here is his excellent fight-club titled post on effect sizes. Niftily demystifying it for the rest of us!
Having ADHD-like gone back and forth between papers, web-sites, and text-books to unpack this very important concept (with some, but not great progress), I’m very grateful he took one for the team!
Well, they let psychologists loose on twitter, where they came up with amusing titles, in the tradition of Psychological Science.
For when you need a good laugh.
And, just to remind you: Rolf Zwaan provided empirical work on the impact of said type of titles.
