Another thought on publishing reviews: a resource for data mining

Following my previous post about the relevance of publishing reviews and reviewer names as discussed at PLOS One, here is another contribution of mine to the discussion:

If all reviews were made publicly available, that would allow for some interesting data mining which might uncover trends about the reviewing process, which could in turn help us improve the process. How often are reviewers completely inconsistent? (see http://dynamicecology.wordpress.com/2012/11/25/how-random-are-referee-decisions/) Are reviewers more stringeant or less detailed for authors from specific backgrounds (gender, country, etc.)? What is the overall balance in reviews between correcting style and correcting substance? Which proportion of reviewer recommendations are accompagnied by a specific reference? Etc.

What is the actual work of a PI? Lots of stuff.

Recently Michael Eisen (lab, blog, twitter) tweeted:

are there any studies of how PIs apportion time between grants, teaching, papers, writing recs, meetings and actual work?

To which I reacted:

.@mbeisen With due respect, I think you are mistaken about what your "real job" is. Teaching major responsability. & planning advising etc.

A small dialogue ensued:

@marc_rr and you know what my real job is because...?
@mbeisen let's say the job of a PI who's a university prof. 140 char lack nuance but excluding teaching from real work seems shocking to me
@marc_rr i am paid to do research, my teaching is voluntary
@marc_rr and that is not me disrespecting teaching - into which i put a shit-ton of time - just a fact of where my money comes from
@mbeisen OK I see. The original tweet said "PI" not "Mike Eisen". I understand your position now. Still not 100% clear about "real work".
@mbeisen and meetings are often an important real work of a PI IMO: share info, make decisions etc

So first a clarification: Michael Eisen is an HHMI investigator. Which as I understand it, means that he doesn't need to teach. So super cool that he does!

Second, I still take issue with the exclusion of "grants, teaching, papers, writing recs, meetings" from "actual work".

The question is, what is the actual work of a PI? Situations can varry widely, so I'll take a common case for now of a Professor who's also a PI, has a lab with grad students (known in Europe as PhD students) and postdocs. Presumably, if you're a Professor you are supposed to teach (but see above), and presumably if you have a lab with students and postdocs you have soft money grants. So what should you be doing?

• Advising and managing your students and postdocs. Depending on their number and degree of autonomy, this can take quite a bit of your time. In my opinion, a very legitimate part of your work.
• Often related to the above, writing and correcting papers, managing the review process (what revisions do you do? which ones do you try to argue away? where do you (re)submit?). Also legitimate.
• Since the students and postdocs are paid, you need to get grants, which means that you need to write them. The legitimacy of this taking up your time is for me a question of degree. I don't think that we would be justified giving grants without asking for a project and reviewing it, and actually thinking about your plans for the next years, how they relate to the competition, how feasible they are, and how they make for not-too-bad PhD projects is important and useful. On the other hand, I can understand the frustration of my US colleagues who need to write an awful number of grants to get a few funded. Moreover, various agencies from the NIH to the EU FP programs ask for all kind of non scientific stuff which is mostly bureaucratic and of dubious use. So I'd say that in principle writing grants is a legitimate part of a PI's work, but that if it takes too much of our time the system is broken (success rate too low and/or too many useless requirements).
• Writing recommendation letters and generally following up on people who worked with you. Seems legit to me.
• Keep up with the literature. Obviously.
• Also obviously (I hope) reviewing other peoples papers and grants.
• Teaching. If you're a Professor at a University, it seems quite legitimate to spend some time on what you're paid for. But further than that, I think that teaching, from undergraduate (Bachelor) to summer schools for PhD students, is a very important responsability, and that we should try to do it well. Which means spending more time preparing classes and interacting with students, and can also mean reading up or going to lectures about pedagogy. Very legitimate.
• Meetings. Ah-ha, no one likes meetings, right? Yet every meeting was called by someone, and others agreed to come. Fact is, meetings can be quite useful and frankly indispensable. You want to coordinate the teaching between Professors to provide a consistent learning experience which meets your goals? You want to decide on the future directions of your department? You want to launch a new interdisciplinary effort? You want to organize a conference? All these things need meetings. So it's true that a mis-managed or mis-called meeting is a waste of time, but so is debugging code. You don't get to do it 100% right. Like grant writing, a question of degree then.
• If you have time, as a Professor you can even do some hands-on research yourself! But in my experience, it would be unrealistic to expect this to be more than 5-20% of your time, unless you've decided on a very small group strategy (i.e., you and one student).
• OK, there's one thing which I have to do and which would not be part of my job in an ideal world: dealing with the ever changing rules of my central administration concerning refunds of expenses and human resources. For this, you need a full time competent admin (disclaimer: I have a competent admin whom I share with 7 or 8 other group leaders).

In summary, I think that there are many aspects to a Professor / PI / group leader's job, and that most of these can be considered "actual work".

We are no longer postdocs (see also this interesting discussion)...

My 2 cents on open peer review

There's a discussion about the relevance of publishing reviews and reviewer names on the restricted discussion forum of PLOS One editors. I'm miroring my own contribution here:

My 2 cents:

• I would really like to see the reviews published next to the paper. This way the work done by the reviewers would not be lost to the community, the readers would be able to see which points were contentious or not, the reader could also see whether a given methodological point (dear to the readers heart let's assume) was covered, everyone could see how much the review process improves many papers submitted to PLOS One (in some cases I feel the reviewers contributed more than some authors). I fail to see any drawbacks whatsover.
• BUT I don't believe in publishing the reviewers' names, at least in the present state of things. Too many possibilities of pressure, retaliation, etc.
• I am also favorable to double-blind. I've read that in quite a few cases the reviewer can guess the authors. I fail to see how it's a problem. Similar to publishing the reviews, I don't see any potential drawbacks.

How open is #openaccess? A few comments

Major actors of open access publishing have taken an excellent initiative, namely establishing a grid detailing how open different publishers are.



They took into account different aspects of open access, and also asks for comments. So here are my suggestions of additional criteria to take into account for publication openess:

• Are the peer reviews publicly available next to the article, anonymously or not?
• Is it possible to freely comment on the publisher's website, next to the article (as in PLOS and BMC)?
• Does the publisher have a systematic policy of embargo towards the press, conference presentations, etc?
• When relevant, must authors provide Open Source source code? (for PLOS Computational Biology you have to, but not for OUP Bioinformatics.)
• When relevant, must authors deposit raw data in independent reliable databanks à la GenBank?

Did I miss anything? Will they really take into account such annoying suggestions? Time will tell.

And now for something completely the same

Those who know me, might know that I have an active blog in French. This allows me to reach out to people who are interested in science, but would not necessarily read about it in English.

But for some topics, I would like to reach out to my colleagues, irrespective of language. I.e., in English. Hence this other blog, where I will post from time to time on topics of interest to other scientists.