9 Code review

Code review means checking and evaluating each other’s code, either formally, as part of peer review, or informally, as co-authors or colleagues. It helps catch conceptual, programmatic and syntactic errors, and it can happen at any stage of the research cycle.

This chapter follows Ivimey-Cook et al. (2023), Implementing code review in the scientific workflow: Insights from ecology and evolutionary biology (Journal of Evolutionary Biology 36: 1347–1356, https://doi.org/10.1111/jeb.14230).

9.1 Why it is missing, and why it matters

You probably already use open-source software such as R or Python to clean, manipulate, visualise and analyse your data. Code-based software is more transparent, reproducible and flexible than point-and-click software, because analyses can be tailored to what a study needs.

That flexibility has a cost. Code is more error-prone (Budd et al. 1998), and several high-profile retractions have followed coding mistakes (Bolnick & Paull 2009; Huijgen et al. 2012; Ma & Chang 2007; Miller 2006; Williams & Bürkner 2020).

In computer science and software development, code review is routine. In the academic disciplines that rely on code to make inferences and predictions, it is largely absent from the research and publication process (Indriasari et al. 2020). We argue for a shift in research culture that brings code review into all stages of the research process (Ivimey-Cook et al. 2023).

Part of the reason is structural. Informal training, together with too little time and too few incentives (Touchon & McCoy 2016), means coding and analysis are often the responsibility of a single member of a team for the whole project, unlike methodology and experimental design, which the whole team usually develops. Because research code is so often one person’s work, pre-publication code review is unlikely to happen, and important when it does.

9.2 The four Rs

The core priorities are the same at every stage. We frame them as four questions (Ivimey-Cook et al. 2023).

R Reported Does the code match the description of what is reported in the methods section?

R Run Can the code actually be executed?

R Reliable Does the code give the correct result, not just a result, on every run?

R Reproducible Do the outputs match the results reported in the paper?

Reported

Code is a key research output and part of the scientific method. Open code to accompany written methods is becoming more common, following similar pushes for Open and FAIR data (Lamprecht et al. 2020). The first check is therefore consistency between the code and the manuscript, which catches conceptual errors.

Does the code match what the methods section says was done? If the methods describe a generalised linear model with Poisson error and the code fits a Gaussian one, the review has to pick that up.

Software and package versions. At minimum, list the packages central to the analysis or visualisation, with version numbers, somewhere in the manuscript. In R you can get these with citation(); in Python, with setuptools. A full list of everything used, including packages for cleaning and tidying, can go elsewhere: the README, or the session_info.txt described in Documented, is the natural place.

This is more than bookkeeping. If a package version is later found to contain a bug, you can identify the papers that depend on it.

Environment management also helps. It is beyond TADA’s minimum standard (see Beyond TADA), but worth using where you can. Packages such as renv (Ushey & Wickham 2023), which replaces packrat (Ushey et al. 2022), and groundhog (Simonsohn & Gruson 2023) in R, or poetry (Eustace 2023) and pipenv (Pipenv Maintainer Team 2023) in Python, help make the environment reproducible and let you load specific package versions. Containerisation with Docker (Boettiger 2015) is another option.

Run

Code that matches the methods in a paper does not necessarily run. Programmatic and syntactic errors can stop it re-running.

Common causes are listed below. The first two are from our code review paper (Ivimey-Cook et al. 2023) and the rest from Stage 5 of the SORTEE guidelines (Pick et al. 2026), which lists the same failures from the data editor’s side:

• calls to libraries or modules that are not installed in the current environment
• spelling mistakes
• local or absolute file paths that do not exist on the reviewer’s machine
• missing files, or filenames in the code that do not match the archived files
• variable names in the code that do not match those in the data files
• a missing code chunk

This is the easiest of the four checks, yet it fails often: 74% of over 9,000 archived R files failed to complete without error (Trisovic et al. 2022).

Run it in a fresh session, from a fresh download, in a new folder. Anything that only works because of what is already loaded in your environment will fail, and that is what you are trying to find.

Reliable

This is the hardest of the four checks, and it needs a reviewer who knows the study.

Code can run perfectly well, produce an output and still be wrong, because code can produce incorrect results reproducibly, giving the same wrong answer every time. For example, if code selects or modifies the wrong column in a dataset, the analysis is wrong but nothing errors (Ivimey-Cook et al. 2023).

Unit testing helps, with testthat (Wickham 2011) in R and pytest (Okken 2022) in Python. But this type of error remains common and is extremely difficult to pick up without deep familiarity with the dataset and code. It is also thought to increase with the number of lines and the complexity of the code (Lipow 1982).

This is the strongest argument for co-author review over external review. A reviewer who does not know your data cannot tell that mass should have been mass_g. A co-author can.

Reproducible

The basic question is whether the code can produce the output, and so support the conclusions, given in the paper.

The final step of code review is to check that the outputs, when the code is re-run, match those reported in the analysis and results sections, including the relevant figures and the narrative text.

When exact matching is not possible. Apply some tolerance, especially with stochastic methods, where parameter estimates change between runs, or with computationally demanding and slow techniques. A mismatch happens, for example, when set.seed() (R) or random.seed() (Python) was not used before stochastic sampling.

Providing model outputs helps. Save them with saveRDS() in R or pickle.dump() in Python, though a reviewer then cannot run the code themselves to see whether they get similar results. Where the code cannot be re-run in reasonable time, compare the newly generated results with those originally reported.

The literature gives two criteria for judging a match.

Conclusion and numbers (Archmiller et al. 2020). Compare the conclusion, meaning direction and significance level, and the numbers, meaning intervals matching within one significant figure. Their worked example:

Original	Reproduced	Verdict
mean 4.00, interval 3.30–5.00	mean 4.12, interval 3.45–4.91	Conclusion and numbers reproduced
mean 4.00, interval 3.30–5.00	mean 6.5, interval 6.0–7.0	Same conclusion; numbers not quantitatively reproduced
mean 4.00, interval 3.30–5.00	mean 4.10, interval −1.00–8.40	Not reproduced, because the interval now overlaps zero

Percentage error (Hardwicke et al. 2021; see also Kambouris et al. 2024). Comparison at one significant figure depends heavily on scale, whereas percentage error does not. Using

\[\text{PE} = \frac{\text{new} - \text{original}}{\text{original}} \times 100\]

error is classified as non-existent when this value is 0, minor between 0 and 10, and major (and not reproducible) at 10 or above.

The formula as published is signed. In practice, it is the absolute value that is compared against those thresholds.

Whether the interval overlaps zero (or the null value) should be the same in the reported and reproduced results, and the two sets of estimates and intervals should not differ significantly from each other. With stochastic MCMC methods, variation between chains is expected to be much lower than variation within a chain, so the reproduced value should sit well within the reported credible interval.

9.3 Writing reviewable code

Code review depends on readable code. We suggest four solutions (Ivimey-Cook et al. 2023):

1. Call packages explicitly via the namespace: package::function() in R, package.module.function() in Python.
2. Use relative file paths, for example the here package (Müller & Bryan 2020), preferably with an R project file if you use R and RStudio, or a virtual environment in Python.
3. Remove redundant packages.
4. Write analysis code with clear subheadings, intuitive comments and easy-to-understand object names.

For style, styler (Müller & Walthert 2020) reformats R code to a standard style in one command, while pycodestyle (Rocholl 2022) checks Python code against PEP 8, with black (Python Software Foundation n.d.) or autopep8 (autopep8 contributors n.d.) to do the reformatting. Supported styles include Google and tidyverse in R, and PEP 8 in Python.

Literate formats (R and Python Markdown, Quarto, and Jupyter Notebook or JupyterLab) show code in chunks that, with descriptive comments, are much easier to read. See Annotated.

Structure

A simple folder and file structure helps researchers at any skill level follow the order and flow of an analysis:

• Sequentially labelled subfolders and scripts, so someone following the code knows the order to run things in, with files beginning 01…, 02…
• Folders named for their purpose, such as data, scripts and functions
• Raw data kept separate, so cleaning or wrangling code cannot modify or overwrite it by accident
• A README and any other metadata a reader needs to re-run the code: packages and version numbers, descriptions of the data files, the project aim, author contact details and any licences

You should also explain any data cleaning or curation that happened before the analysis code: what manipulation or pre-processing produced the data in its current state, or where an intermediate data file was used.

Metadata tooling. Packages such as codemeta (Boettiger 2017) and dataReporter (Petersen & Ekstrøm 2019) in R, or CodeMetaPy (van Gompel 2023) and cookiecutter (Feldroy 2022) in Python, can help produce consistent metadata and project templates. Pipeline tools automate the steps from raw data to results: targets (Landau 2021) and workflowr (Blischak et al. 2019) in R, and luigi (The Luigi Authors 2023) in Python.

9.4 When code review happens

Code review can start when a project starts, and it is useful beyond publication.

Pre-submission

This is where most of the value is. It is also where a reviewer can catch Reliable errors, because they know the study.

Collaborate on code continuously, at every stage of a manuscript. Platforms such as GitHub help, since users can submit and comment on pull requests. At SORTEE, we used GitHub issues to summarise the discussion of an individual’s code.

The SORTEE guidelines give authors the same advice, under “Perform a pre-submission code review” (Pick et al. 2026); see Advice for authors for what that review should cover.

During peer review

Peer review is one of the most important places for code review, because reviewers can follow the logic of the analyses much as they follow the argument from introduction to discussion.

Some journals already request code and data at submission, including The Royal Society, Behavioral Ecology and Sociobiology and The American Naturalist, though in our experience this is not enforced (Ivimey-Cook et al. 2023). In some cases, such as the Journal of Open Source Software, the entire formal peer-review process, code and manuscript alike, is hosted on GitHub.

Both sides benefit (Fernández-Juricic 2021). For authors, providing code during peer review could improve the manuscript. For reviewers, available code gives much more insight, because the link between experimental methods and statistical analysis is clearer. That is the first R, code as reported.

There are two practical obstacles.

Finding reviewers. Willing reviewers are already in short supply. It is reasonable to expect a reviewer to check that code is as reported, but anything deeper takes time from overworked academics who may not have the right expertise. A workable compromise is for reviewers to say whether they can check that the code is as reported, and for the journal to make sure at least one reviewer has checked the code.

Double-blind review. Anonymised data and code can be submitted directly to the journal, for example as a zip file, or uploaded to Figshare, OSF, GitHub or Dryad, which allow authors to provide an anonymised link for peer review.

Post-publication

Post-publication review is discussed much less. It cannot stop incorrect results being published, but it does let you check the four Rs afterwards.

The first question is whether all the code used to produce the results has been made available. If not, the main option is to ask the corresponding author or the journal, much as data are requested “upon reasonable request”.

If you find that the code does not adhere to one of the Rs:

1. Contact the corresponding author, or, if the paper uses the MeRIT system (Nakagawa et al. 2023), the person who conducted the analysis. A GitHub issue works well if there is a repository. Otherwise, email.
2. Rule out harmless causes first, such as differences in R or package versions, or the stochasticity inherent in simulations and MCMC sampling.
3. Let the authors go to the journal themselves. If there is a genuine error, the authors should get the chance to contact the journal and correct their own mistake.

Journal culture matters here. Under The American Naturalist’s stance (Bolnick 2022), authors who contact the journal to correct code or data errors are not penalised, and corrections are encouraged where warranted. Where updated results would change the narrative of a published paper, corrections are harder, and publication versioning or “living” documents may be a first step (Kane & Amin 2023).

9.5 Running a code club

A code club is a good way to set up peer code review within a lab or among colleagues. This section draws on our experience building a code review club with SORTEE (Ivimey-Cook et al. 2023).

The club is designed around the barriers it addresses (Gomes et al. 2022): lack of time and incentive, lack of technical knowledge and unclear workflows, and social pressures and the fear of being judged by peers.

Set a clear goal for each session

Is the focus general learning and improving readability, or is it to error-check and scrutinise reproducibility? A clear structure and goal for each session keeps comments and advice focused on the reason for the review.

Unless the aim is specifically to evaluate different analytical options, it is better to leave methodological questions aside. A session that drifts into the choice of model is no longer reviewing the code.

Make it a judgement-free environment

Volunteers often feel anxious about showing code that may have errors. Normalise errors and stress that perfection is never possible. It also helps to say that there is no such thing as bad code (Barnes 2010), and that there are usually several ways to approach the same problem.

Think about group size

A smaller group is a friendlier place to start: people feel more comfortable speaking up, and it suits peer-to-peer learning and focused review. Larger groups work better for general discussion of themes and tips.

Decide on credit

Code review outside formal peer review can have a large impact on a project, from error checking to confirming that the statistical analyses are appropriate. So what should reviewers get?

• Acknowledgement via MeRIT (Method Reporting with Initials for Transparency; Nakagawa et al. 2023), for example: “…mixed model with a Gaussian error distribution. Code was checked by E.I.C.”
• Co-authorship, where the review substantially changed the project and the paper, for instance where a reviewer finds a major coding error that, once fixed, changes the results and conclusions.

We think credit should match the reviewer’s impact on the project (Ivimey-Cook et al. 2023).

9.6 A reviewer’s script

A sequence for a single review session, built on the four Rs. The full checklist is at templates/code-review-checklist.md.

Before the session (reviewer, 20 minutes)

1. Download the project into a new, empty folder.
2. Open the README. Can you tell what to run, in what order, and what you need installed? Note anything you had to guess.
3. Try to run it, in a fresh session. Record every error word for word, and stop at the first one you cannot resolve in two minutes.

In the session

4. Reported. Walk the methods section against the code, section by section. Flag anything described in the paper you cannot find in the code, and anything in the code not described in the paper.
5. Run. Go through the errors you hit. Most will be environment or path problems, and most are quick to fix.
6. Reliable. Pick the two or three steps where a silent error would matter most, such as the filtering step, a join or the model specification, and read them closely with the author. Ask what each column is.
7. Reproducible. Take three numbers from the results section and find them in the output.

Afterwards

8. Write it up as a short list, ordered by severity, separating must fix from would be nice.
9. Agree how the reviewer will be credited.

Give feedback on the code, not the coder. “This line reads mass, is that the right column?” is easier to act on than “you used the wrong column”. Fear of being judged is one of the barriers we identified (Ivimey-Cook et al. 2023; Gomes et al. 2022), so the way you phrase comments affects whether people keep volunteering.

9.7 Checklist

Try doing all of these to make the dog happy.

• Reported. The code does what the methods section says it does
• Software and key packages, with versions, are stated
• Run. The code executes from a fresh download in a fresh session
• All required libraries are installed and loaded within the code
• File paths are relative and resolve
• Reliable. Filtering, joins and model specifications have been read closely by someone who knows the data
• Seeds are set before stochastic steps
• Reproducible. Spot-checked numbers from the results section match the output
• Mismatches are explained (version, stochasticity, post-processing) and documented
• Slow or proprietary steps have saved outputs archived
• The reviewer’s contribution is credited (MeRIT, acknowledgement, or authorship)