Projects · Infrastructure

A reproducible research environment

Every project should run the same way on my machine and on a server, so a result still holds up when it moves from my laptop into a paper. This page describes the setup that makes it work.

The problem it solves

Empirical work has a common failure mode. A result lives on one laptop, tied to whatever package versions happened to be installed when it was made. A year later the figure won't regenerate, and even the author can't reconstruct how it was produced.

The fix is to give every project its own environment. Dependencies are pinned, and a single command rebuilds every figure from the raw data. Nothing important is left sitting on the host machine, so reproducibility is built into the setup.

Architecture

The whole environment is containerized. Each service is defined in a single file and isolated from the others, and the same definition runs in both development and production. There's no “works on my machine” gap to fall through.

Containerized services

Docker Compose defines the web server, language runtime, and database as isolated, version-pinned services. The whole stack comes up from one file with one up.

Reproducible pipelines

Each analysis repo has a single entry point that rebuilds every output from source. No manual steps and no hidden state: raw data goes in, the figures come out.

Self-hosted

All of it runs on my own hardware, a small cluster of machines I administer myself. TLS, database backups, and deploys are set up and under my control, with no hosting platform between me and the hardware.

Automation

The repetitive work of regenerating widgets and rerunning pipelines is scripted rather than done by hand.

Every project follows the same shape. This site's own stack is a fair example: three version-pinned services that talk only to each other over a private network, with the application built from a local image and every secret kept out of the repo.

compose.yml · this site's stack

services:
  db:                     # MariaDB, pinned
    image: mariadb:11
    env_file: .env        # secrets, never in git
    volumes: [db_data:/var/lib/mysql]
    networks: [internal]  # private to the stack

  php:                    # built from a local image
    build: { context: ., dockerfile: Dockerfile.php }
    networks: [internal]
    healthcheck:          # auto-restart if it stalls
      test: ["CMD", "php", "-r", "echo 'ok';"]

  web:                    # nginx, the only exposed service
    image: nginx:alpine
    depends_on: [php]
    networks: [internal]

volumes:  { db_data: {} } # survives a full rebuild
networks: { internal: { driver: bridge } }

What runs on it

The same environment hosts this site and the research codebases behind the papers, along with the automation that keeps them in sync, regenerating chart widgets and rerunning pipelines as the data changes. I maintain the research and the platform it runs on as one system.

Why it matters

The stack exists for the research. A result can be rebuilt on demand, and a reviewer can run the code instead of taking a figure on trust.

See the research codebases for the pipelines this runs, or this site for the platform itself.