7 Habits of the Open Scientist: #2 -- Reproducible Research

Note: this post is part of a series on habits of the open scientist.  Here I discuss the second habit, reproducible research.  The previous post was on open scientific publishing.

Reproducible research

Reproducibility is part of the definition of science: if the results of your experiment cannot be replicated by different people in a different location, then you're not doing science.  Far from being a mere philosophic concern, reproducible research has been a key issue in prominent controversies like climategate and cancer research clinical trials.

Especially disconcerting is the typical irreproducibility of scientific work involving computer code:

“Computational science is facing a credibility crisis: it’s impossible to verify most of the computational results presented at conferences and in papers today.” (LeVeque, Mitchell, Stodden, CiSE 2012)

Frankly, I used to find that I was often unable to reproduce my own computational results after a few months, because I had not maintained sufficiently detailed notes about my code and my computing environment.

The open scientist ensures that the entire research compendium -- including not only the paper but the data, source code, parameters, post-processing, and computing environment -- is made freely available, preferably in a way that facilitates its reuse by others.

I won't spend more time motivating reproducible research, since others have done that much better than I could.  Instead, let me focus on the relatively easy first steps you can take to make your research more reproducible.

The bare minimum: publish your code and data

If you wish to set an example of good reproducible computational research practices, I have good news for you: the bar is very low at the moment.  The reason why "it's impossible to verify most of the computational results" is that most researchers don't release their code and data.  The first step toward working reproducibly is simply to put the code and data that is used in your published research out in the open.

If you don't want to release your code to the public, please read about why you should and why you can.  Once you're convinced, go endorse the Science Code Manifesto.

Releasing your code and data can be as simple as posting a tarball on your website with a reference to the paper it pertains to.  Or you may wish to start putting all your code out in the open on Bitbucket or Github, like I do.  I don't claim that these are the best solutions possible, but they are a big step forward from keeping everything on your own hard drive.

When you release your code and data, it is important to use an appropriate license.  Victoria Stodden, a leader in the reproducible research movement, recommends the use of a permissive license like modified BSD for code and Science Commons Database Protocol for data.  Together with the Creative Commons BY license for media (that I mentioned in my last post), these comprise the Reproducible Research Standard, a convenient amalgamation of licenses for open science.

Be sure to include a mention of reproducibility in your paper, along with links to the code and data.  If you release your work under the RRS, I suggest using this citation.

Real benefits

The open scientist may adopt reproducible research practices for philosophical reasons, but he soon finds that they bring more direct benefits.  Because he writes code and prepares data with the expectation that it will be seen by others, the open scientist finds it much easier for himself, students, and colleagues to build on past work.  New collaborations are formed when others discover his work through openly released code and data.  And (as in the case of this paper, for example) the code itself may be the main subject of publications in journals that have come to recognize the importance of scientific software.

Taking it further

Like free and open scientific publishing, reproducible research has become a very large movement, and only a book could hope to cover it all.  Here I've merely distilled some basic practical suggestions.

Openly releasing code and data is only the first step.  Open scientists may wish to adopt tools that track code provenance and ensure a fully reproducible workflow, such as

• Madagascar
• VisTrails
• knitr
• Sumatra
• VCR

Next up: Habit #3 -- Pre-publication dissemination of research.

Book Review: Reinventing Discovery

I believe that the process of science—how discoveries are made—will change more in the next twenty years than it has in the past 300 years. --Michael Nielsen, Reinventing Discovery

I appreciate an author who's not afraid to make bold claims, and Michael Nielsen certainly fits that description.  He goes on to say even that

To historians looking back a hundred years from now, there will be two eras of science: pre-network science, and networked science.  We are living in the time of transition to the second era of science.

I grew up feeling that the golden age of science was the first half of the twentieth century, which gave us marvelous advances like relativity and quantum mechanics.  According to Nielsen, though, I'm witnessing the most transformative period of scientific development since the invention of the scholarly journal in the 1700's.  Although I'm a firm believer in the power of the internet to accelerate scientific advances, I was skeptical.
I downloaded Michael Nielsen's Reinventing Discovery on Tuesday and read it in less than 48 hours (between shopping trips while on vacation in Dubai).  Although I was familiar with much of the material in the book, it was an engaging and highly thought-provoking read that I think both scientists and laypersons will enjoy.  I'll focus here on the ideas that struck me as especially insightful.
Nielsen gives several examples to illustrate the beginnings of his foretold revolution; some are scientific (the Polymath project, GalaxyZoo, FoldIt) while others simply illustrate the power of our new networked world (Kasparov versus the World, Innocentive).  These examples are used extensively and lend a convincing empricism to a book that claims to predict the future.  They also allow Nielsen to dive into actual science, adding to the fun.
Many scientific advances are the result of combinations of knowledge from different fields, communities, or traditions that are brought together by fortuitous encounters among different people.  In a well-networked world, these encounters can be made to happen by giving individuals enough accessible information and communication. Nielsen refers to this as "designed serendipity".

The reason designed serendipity is important is because in creative work, most of us...spend much of our time blocked by problems that would be routine, if only we could find the right expert to help us. As recently as 20 years ago, finding that right expert was likely to be difficult. But, as examples such as InnoCentive and Kasparov versus the World show, we can now design systems that make it routine.

Offline, it can take months to track down a new collaborator with expertise that complements your own in just the right way. But that changes when you can ask a question in an online forum and get a response ten minutes later from one of the world’s leading experts on the topic you asked about.

The trouble is, of course, that the forum in question doesn't exist -- and if it did, who would have time to read all the messages?  Nielsen delves into this question, discussing how to design an "architecture of attention" that allows individuals to focus on the bits most relevant to them, so that large groups of people can work on a single problem in a way that allows each of them to exercise his particular expertise.  Taking the idea of designed serendipity to its logical yet astounding conclusion, Nielsen presents a science fiction (pun intended) portrayal of a future network that connects all researchers across disciplines to the collaborations they are most aptly suited for.  I found this imaginary future world both fascinating and believable.
The second part of the book explores the powers that are being unleashed as torrents of data are made accessible and analyzable.  Here Nielsen draws examples from Medline, Google Flu Trends, and GalaxyZoo.  While the importance of "data science" is already widely recognized, Nielsen expresses it nicely:

Confronted by such a wealth of data, in many ways we are not so much knowledge-limited as we are question-limited...the questions you can answer are actually an emergent property of complex systems of knowledge: the number of questions you can answer grows much faster than your knowledge.

In my opinion, he gets a bit carried away, suggesting that huge, complex models generated by analyzing mountains of data "might...contain more truth than our conventional theories" and arguing that "in the history of science the distinction between models and explanations is blurred to the point of non-existence", using Planck's study of thermal radiation as an example.  Planck's "model" was trying to explain a tiny amount of data and came up with terse mathematical equations to do so.  The suggestion that such a model is similar to linguistic models based on fitting terabytes (or more!) of data, and that the latter hold some kind of "truth" surprised me -- I suspect rather that models informed by so much data are accurate because they never need to do more than interpolate between nearby known values.  Nevertheless, it was interesting to see Nielsen's different and audacious perspective well-defended.
A question of more practical importance is how to get all those terabytes of data out in the open, and Nielsen brings an interesting point of view to this discussion as well, comparing the current situation to that of the pre-journal scientific era, when figures like Galileo and Newton communicated their discoveries by anagrams, in order to ensure the discoverer could claim credit later but also that his competitors couldn't read the discovery until then.  The solution then was imposed top-down: wealthy patrons demanded that the discoveries they funded be published openly, which meant that one had to publish in order to get and maintain a job.
The logical conclusion is that policies (from governments and granting agencies) should now be used to urge researchers to release their data and code publicly.  Employment decisions should give preference to researchers who follow this approach.  At present, the current of incentives rather discourages such "open science", but like Nielsen I am hopeful that the tide will soon turn.  I was left pondering what I could do to help; Nielsen provides numerous suggestions.  I'll conclude with some of the most relevant for computational scientists like myself.

...a lot of scientific knowledge is far better expressed as code than in the form of a scientific paper. But today, that knowledge often either remains hidden, or else is shoehorned into papers, because there’s no incentive to do otherwise. But if we got a citation-measurement-reward cycle going for code, then writing and sharing code would start to help rather than hurt scientists’ careers. This would have many positive consequences, but it would have one particularly crucial consequence: it would give scientists a strong motivation to create new tools for doing science.
...
Work in cahoots with your scientist programmer friends to establish shared norms for citation, and for sharing of code. And then work together to gradually ratchet up the pressure on other scientists to follow those norms. Don’t just promote your own work, but also insist more broadly on the value of code as a scientific contribution in its own right, every bit as valuable as more traditional forms.

Searching the scientific literature

Many of the fundamental skills of a scientist are seldom taught. Instead, one is expected to pick them up through intuition, informal conversations, or trial-and-error. One of these essential skills is how to search the literature for journal articles related to a particular topic.

This is a challenging task with severe consequences for failure. Just ask any Ph.D. student who discovered that his thesis was focused on a problem that had already been solved. Or anybody who left grad school because of the overwhelming task of grasping and keeping up with the scientific literature related to his thesis topic.

 

Why

The purpose of a literature search is not merely to become aware of what results are already known. Rather, a good literature search provides a map of the scientific terrain, indicating the general layout of a research area:

 

• What are the main goals of research in this area?
• What kind of advances are considered significant, and why?
• What are the recognized open questions, and what impact would their answers have?
• What other research areas are most closely connected to this one?
• Are there other research areas with connections to this one that have not been recognized?
• How is this research area viewed by those who focus on related, competing topics?

How

With these goals in mind, how does one conduct an effective literature search? Here are some techniques that have served me well:

• Ask for help. You have a network of collaborators (or at least an advisor!) who each know some part of the literature much better than you. If you're starting research in a new topic where they have expertise, ask them for the most significant work on that topic. Ask their opinion of new papers that seem significant to you. Ask them for the right keywords, authors, and review articles to start with. Because they can make connections that a search engine never would, they are your most valuable resource.
• Use Google Scholar. Yes, there are countless databases and search tools out there for looking at a articles from a particular discipline or publisher. But I have yet to find one as effective as Scholar. I'm convinced that its coverage is much broader than any of the commercial academic databases available. For instance, few other databases cover the ArXiv, which is an essential source in some fields.
• Link forward through the literature. Every paper has a list of references to the works that it cites. But since you're mostly interested in learning about the state-of-the-art, it's usually more helpful to obtain a list of papers that cite the one you have. This is another major advantage of Google Scholar, which allows you to do so easily. Each search result includes a link to a list of all the articles that cite it.
• Learn how to do effective keyword searches. This skill has become incredibly valuable in the internet age, and nowhere more so than in searching for journal articles. When learning about a new topic, it can be hard to know which keywords to search for, and you should ask for help (see above). Once you know the right words, it can be very important whether search for A and B, A or B, A since year X, B authored by Y, and so forth. Learn how to refine your searches in this way.
• Learn to rapidly evaluate article titles and abstracts. You can't hope to read all the articles, or even all the abstracts published in your field. Your ability to find the most relevant ones is directly proportional to how quickly you can eliminate the irrelevant. I'm convinced that this skill can only be obtained by experience, but you can accelerate it by noticing articles that you thought would be useful but turned out not to be, as well as becoming aware of who the key authors are in an area.
• Check for articles in review journals. Most fields have some journals that publish only review articles. Such articles provide a broad overview of a topic along with a detailed bibliography; they are invaluable when starting research on a new topic. In my field, the most relevant are Acta Numerica and SIAM Review. Review articles tend to rank high in search engines because they are heavily cited, but it can be worth searching for them specifically or even browsing review journals that publish a low volume (like the two just mentioned).
• Check the websites of key authors. You can often find their preprints there long before the published article becomes available. Of course, you don't have time to do this on a large scale, so you have to be selective.

I'm planning a future post that will discuss what to do with all the relevant and significant articles you find.