Scientific Workflows on the Eve of the Butlerian Jihad

Notes on the tools I use to keep research code reproducible and manageable
Author

W. H. W. Thompson

Published

February 28, 2026

Why I Changed My Workflow

Computers got faster for a long time, and we got used to treating slow code as a temporary problem. Moore’s observation about transistor density [2] was a decent guide for decades. It is less useful now.

More speed now has to come from the code: better algorithms, better compilers, and better use of parallel hardware [3]. Julia [4] addresses part of this. But in my own work, the algorithm is rarely where I lose the most time. I lose it getting an environment running, finding the right output file, or remembering how I made a figure.

We are not usually taught how to manage that part of research [5]. I have inherited notebooks that only run in one particular cell order and folders full of CSVs with no record of how they were made. Clusters add another layer of shell scripts and SLURM settings. I have seen a project’s analysis get smaller simply because the workflow could not handle the version we wanted to run.

The meme is exaggerated, but I have spent plenty of time doing the things it makes fun of: rebuilding an old run, moving files by hand, and debugging a batch script instead of looking at results.

It is easier to learn these tools now. I use LLMs when I need a first draft of a configuration file or help finding the right command. I still read the result and run it myself. That has made me more willing to set up tools I used to put off.

The Dune title is a joke, mostly. The useful point is simpler: take advantage of these tools while keeping the code understandable to you and your collaborators.

This post grew out of a talk for students in my department. The companion repository has the working examples. Here is what I use and why.

How Much Structure?

I never know how much structure a new project needs. If I build a framework on day one, I may spend a week on features the project never uses. If I keep writing one-off scripts, I eventually cannot reproduce my own results.

  • If you take the time to build software the right way, you worry you will never reuse it.
  • If you write sloppy code, it will eventually bite you.
  • Damned if you do. Damned if you don’t.

Most of the repetitive work is around the model: loading data, choosing parameters, launching jobs, and collecting outputs. That is the part I try to automate first.

What I Imagined

The Reality

Is It Worth The Time? (XKCD 1205)

The xkcd chart is a useful check on my urge to automate everything. A task I do once is probably fine by hand. A task I repeat for every experiment deserves a script.

A Running Example

Clear function signatures and schemas help me understand a project when I come back to it. They also give an LLM enough context to make a useful suggestion without guessing what the data looks like.

I am going to use one small example for the rest of the post.

The Process

  • You have data or simulation parameters.
  • You have a model: ABM, statistical, or ML.
  • You want to run sweeps over parameter space with fixed logic.
  • You want to collect and visualize results without manual copy-paste.
  • You want to turn plots into a publication or shareable artifact.

The Goal

I want to be able to change a parameter, run the analysis, and know which files produced the figure I am looking at.

The example fine-tunes BERT [6] on AG News, tries a few adapter settings, and gathers the results. The repo has the full code.

  1. Fine-tune BERT on AG News (4 classes: World, Sports, Business, Sci/Tech).
  2. Train a lightweight adapter/MLP on extracted embeddings.
  3. Run parameter sweeps over adapter dimension and learning rate.
  4. Compare results and publish figures automatically.
/Users/willthompson/Documents/CSDS/side_projects/PythonForScientificDevelopment/.venv/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Sample from the AG News dataset: 4 categories of news article headlines. We use this as a reference pipeline throughout; the full code is in the companion repo.

Here are the tools I used. You can swap out most of them; the point is that each one has a specific job.

Problem Tooling Choice
Reproducible environment uv + lockfiles
Data validation Pydantic schemas
Hardware acceleration PyTorch / JAX
Code reliability pytest + CI
Workflow automation Snakemake
Experiment tracking Weights & Biases
Publication and dashboards Quarto

Code and Environment

I think of this in two parts: the code and environment that run an experiment, and the tools that connect runs to results. Start with git if you are not already using it. I rely on it throughout.

1. Keep Core Logic in Python Source

I keep reusable code in Python files and import it into notebooks when I need to explore. Then I can test the same functions I use for the analysis.

# src/core/model.py — importable, testable, versionable
def simulate_trajectories(
    n_agents: int,
    time_steps: int,
    params: dict[str, float],
) -> list:
    """Simulates N trajectories over T steps."""
    ...

Type annotations make the inputs and outputs easier to see. A type checker can catch some mistakes before I spend an afternoon tracking them through a notebook.

In the example, src/core/trainer.py handles training and src/core/model.py defines the adapter. The sweep configuration and plotting code live elsewhere.

2. Reproducible Environments with uv

Getting someone else’s Python project to run can still be painful. A list of top-level packages does not tell me which versions were actually used. I use uv to create a lockfile and install from it. It is fast, and uv sync gives a collaborator the same dependency versions I used.

# pyproject.toml
[project]
name = "scientific_dev"
requires-python = ">=3.12"
dependencies = [
    "jax>=0.9.0",
    "snakemake>=8.4.0",
    "pydantic>=2.12.5",
]

The example locks PyTorch, Transformers, Pydantic, and the rest of its dependencies. A collaborator can run uv sync before trying the pipeline.

3. Hardware Acceleration with JAX / PyTorch

A GPU helps when you can do the same array operation over a lot of data. It does not make every Python loop fast. I try to profile a slow step before moving it to JAX or PyTorch.

JAX is handy for vectorized numerical work and automatic differentiation. vmap lets me apply a function to a batch without writing the batch loop myself:

import jax.numpy as jnp
from jax import vmap

def compute_score(x):
    return jnp.sum(jnp.sin(x) ** 2)

large_input_matrix = jnp.ones((1000, 128))  # 1000 samples, 128 features each

# Run over all rows in parallel on GPU
batch_score = vmap(compute_score)
scores = batch_score(large_input_matrix)

Moving NumPy code to JAX sometimes takes only a small change, but I still check that the work is large enough to justify compilation and data transfer. I use PyTorch for neural networks and JAX for some simulation and inference code.

4. Validation with Pydantic

I use Pydantic at the edges of a pipeline, where a config or data file first comes in. If an adapter dimension is negative or a field has the wrong type, I want the job to fail there, before it spends an hour on a cluster.

# src/core/schema.py
from pydantic import BaseModel, Field, field_validator

class NLPModelConfig(BaseModel):
    model_name: str = Field(default="distilbert-base-uncased")
    adapter_dim: int = Field(default=256)

    @field_validator("adapter_dim")
    def validate_adapter_dim(cls, v: int):
        if v <= 0:
            raise ValueError("adapter_dim must be positive")
        return v

View Schema →

In the example, NLPModelConfig checks each sweep configuration before training starts.

Running the Pipeline

1. Command Center with just

I forget command flags. A justfile gives the commands I run often short names: just all, just plot, or just sync. It is also a useful place to point an LLM, so it uses the project’s existing commands.

# Default config
config := "configs/nlp_baseline.yaml"

# Run full pipeline
all p="local" c=config:
    uv run snakemake --workflow-profile ./workflow/profiles/{{p}} --configfile {{c}}

# Run training only
train p="local" c=config:
    uv run snakemake train_all --workflow-profile ./workflow/profiles/{{p}} --configfile {{c}}

# Run evaluation only
test p="local" c=config:
    uv run snakemake test_all --workflow-profile ./workflow/profiles/{{p}} --configfile {{c}}

# Sync results from cluster
sync:
    ./tools/sync_vacc.sh

# Regenerate figures from synced data
plot:
    uv run snakemake plots/fig1_roc_sweep.png -j1

# Unlock snakemake after a crash
unlock:
    uv run snakemake --unlock

# Preview Quarto site locally
preview:
    cd notebooks/quarto && quarto preview . --no-browser

View Justfile →

In the example, just all runs the pipeline from input data to figures.

2. Orchestration with Snakemake

Snakemake [7] keeps track of how files are made. I tell it that a figure depends on analysis output, which depends on training runs. It works backward from the figure and runs only the missing or outdated steps. I no longer have to remember which scripts to rerun after changing a parameter.

It can submit those jobs through SLURM too. That replaces a pile of batch scripts with a workflow I can run locally or on the cluster. When I return to a paper months later, I can start from the figure target and see the chain of work that produced it.

In the example, one rule launches the adapter sweep: one job for each combination of dimension and learning rate.

Snakemake infers the full dependency graph from your rules and only re-runs what’s out of date. This DAG was generated automatically from the BERT pipeline rules.
# workflow/rules/train.smk
rule train_bert_classifier:
    output:
        weights = "results/{cfg_name}/{params_path}/model.pt",
        stats   = "results/{cfg_name}/{params_path}/training_stats.parquet"
    resources:
        gpu = 1
    shell:
        "PYTHONPATH=src python scripts/run_training.py "
        "--config configs/{wildcards.cfg_name}.yaml "
        "--output {output.weights}"

View Full Rule →

3. Testing with pytest and CI

Tests catch small mistakes before they spread through a project. I run them locally with pytest and in CI when I push a change.

# tests/test_nlp_logic.py
def test_validation():
    with pytest.raises(ValueError):
        NLPModelConfig(adapter_dim=-1)
# .github/workflows/ci.yml
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: uv run pytest tests/

View CI Workflow →

The example’s CI job runs the tests on each push. It is much easier to deal with a failed test then than with a bad figure during the final writeup.

4. Experiment Tracking with Weights & Biases

For long runs, I use Weights & Biases to watch loss curves and compare configurations without opening each output file. In the BERT sweep, I can check progress from a browser while jobs are still on the cluster. I do not need it for every project.

# src/core/trainer.py
def init_logger(self):
    wandb.init(
        project="neural-adapter-research",
        config=self.config.model_dump(),
        name=self.config.full_run_name,
    )

wandb.log({
    "train/step_loss": loss.item(),
    "train/learning_rate": self.optimizer.param_groups[0]["lr"],
    "train/global_step": epoch * len(self.train_loader) + step,
})

View Trainer →

5. Storage and Querying with Parquet + DuckDB

CSV files are easy to make and easy to lose track of. For larger sweeps I save results as Parquet and query them with DuckDB. I get compressed files and SQL queries without setting up a database server.

SELECT s.*, m."test/auroc"
FROM read_parquet('results/**/training_stats.parquet') s
LEFT JOIN read_parquet('results/**/test_metrics.parquet') m
  ON s.full_run_name = m.full_run_name

The example writes training and test metrics to Parquet. One DuckDB query joins the runs for the final analysis.

View Query Example →

Writing Up the Results with Quarto

I wrote this post in Quarto.

Quarto turns a .qmd file into a webpage, PDF, or slides. I can keep prose, code, equations, and figures in one source file and render it again when the analysis changes.

That matters because I have broken figures by copying them into a draft and forgetting to update them. Here, the code that makes a figure lives with the text that discusses it. Re-rendering is a better habit than manual copy and paste.

Quarto also runs Observable JavaScript. The small Vicsek simulation below is an example; it works in the page without a separate server.

## Theoretical Loss
$$
\mathcal{L}(\theta) = \frac{1}{N} \sum_{i=1}^N \text{CrossEntropy}(f_\theta(x_i), y_i)
$$

```python
import seaborn as sns
sns.lineplot(data=df, x='epoch', y='loss')
```

A Small Interactive Model

This is a Vicsek model [8]. Agents move in a plane and tend to align with nearby agents. I built it in Observable JavaScript inside the same Quarto document as the rest of this post.

Lower the noise and the agents start moving together. Raise it and they scatter. The simulation is here because it is nice to be able to show a model next to the explanation, with the code in the same source file.

A Few Habits Worth Keeping

When I help students with research code, I keep coming back to a few habits:

  1. Get one result to reproduce first. Then automate and scale it.
  2. Leave a way back to old results. Save the config, code version, and command that made each figure.
  3. Run tests when you change code. A failure on Tuesday afternoon is easier to fix than one the night before a deadline.
  4. Read generated code. LLMs help with boilerplate and unfamiliar syntax. They cannot check whether the experiment answers the question you meant to ask.

Where to Start

I would not install every tool here on day one. Start with the step you keep doing by hand, make it repeatable, and add the next tool when you need it.

LLMs have made some of the setup easier for me. The reason to keep tests, schemas, and version control is the same as ever: I need to know what ran and whether I trust the result.

The companion repository contains working examples you can adapt to your own projects.

Acknowledgments

Nicholas Landry taught me a lot of this while working on XGI. He spent many patient hours showing me how to make pull requests, use linters, and write code other people can work with. Phil Chodrow and Daniel Kaiser also introduced me to Quarto, Snakemake, and several of the ideas in this post. Thanks to all three.

References

[1]
R. P. Feynman, “There’s plenty of room at the bottom,” Engineering and Science, vol. 23, no. 5, pp. 22–36, 1960.
[2]
G. E. Moore, “Cramming more components onto integrated circuits,” Electronics, vol. 38, no. 8, pp. 114–117, 1965.
[3]
C. E. Leiserson et al., “There’s plenty of room at the top: What will drive computer performance after Moore’s law?” Science, vol. 368, no. 6495, p. eaam9744, 2020, doi: 10.1126/science.aam9744.
[4]
J. Bezanson, A. Edelman, S. Karpinski, and V. B. Shah, “Julia: A fresh approach to numerical computing,” SIAM Review, vol. 59, no. 1, pp. 65–98, 2017, doi: 10.1137/141000671.
[5]
G. Wilson, J. Bryan, K. Cranston, J. Kitzes, L. Nederbragt, and T. K. Teal, “Good enough practices in scientific computing,” PLOS Computational Biology, vol. 13, no. 6, p. e1005510, 2017, doi: 10.1371/journal.pcbi.1005510.
[6]
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of NAACL-HLT, 2019, pp. 4171–4186.
[7]
F. Mölder et al., Sustainable data analysis with Snakemake. (2021). doi: 10.12688/f1000research.29032.2.
[8]
T. Vicsek, A. Czirók, E. Ben-Jacob, I. Cohen, and O. Shochet, “Novel type of phase transition in a system of self-driven particles,” Physical Review Letters, vol. 75, no. 6, pp. 1226–1229, 1995, doi: 10.1103/PhysRevLett.75.1226.