Skip to content

CI Pipelines and Repository Mirrors

OpenLatch is hosted on Codeberg (Forgejo). The CI runs natively there. Mirrors on GitLab and GitHub carry their own CI configuration so that organisations who fork or mirror the repository can run pipelines without depending on the upstream platform.

CI architecture

All meaningful CI logic lives in ci/*.sh shell scripts at the repository root. The platform-specific YAML files are thin wrappers that set up the environment and call those scripts.

ci/
  backend-lint.sh      # ruff check + ruff format --check
  backend-test.sh      # pytest --cov
  firmware-test.sh     # pio test -e native
  firmware-build.sh    # pio run -e esp32dev
  security-scan.sh     # pip-audit

When you need to change what a pipeline step does — adjust the ci/*.sh script. The platform YAML files rarely need to change, which minimises merge conflicts when pulling upstream updates.

Platform overview

Platform Config file Docs deploy Pages URL
Codeberg (Forgejo) .forgejo/workflows/ci.yaml pages branch https://openlatch.codeberg.page/
GitHub .github/workflows/ci.yml GitHub Pages (Actions) https://<org>.github.io/openlatch/
GitLab .gitlab-ci.yml GitLab Pages (public/) https://<group>.gitlab.io/openlatch/

Pipeline stages

All three platforms run the same stages:

Stage Jobs Trigger
lint backend-lint every push / PR
test backend-test, firmware-test:native every push / PR
build firmware-build (ESP32), docs-build main branch only for binary; always for docs
pages/deploy docs published to platform pages main branch only
security pip-audit (non-blocking) every push / PR

What the firmware native tests need

The firmware-test job runs the host (native) test suite — no ESP32 hardware required. It does require libsodium development headers, installed via the package manager of the runner OS:

  • Ubuntu/Debian: apt-get install -y libsodium-dev
  • Alpine: apk add libsodium-dev
  • macOS: brew install libsodium

All three CI configurations install libsodium automatically on their runners.

What the ESP32 build needs

firmware-build downloads the Espressif32 toolchain via PlatformIO on first run (~500 MB). This is cached under ~/.platformio in the Forgejo and GitHub configurations and under .platformio/ (project-relative) in the GitLab configuration.

Setting up a Codeberg → GitLab mirror (IN-Berlin setup)

1. Mirror the repository

In your GitLab instance:

  1. New Project → Import Project → Repository by URL
  2. Enter https://codeberg.org/OpenLatch/OpenLatch.git
  3. Enable Mirror repository (pull)
  4. Set a sync interval (e.g., every hour)

GitLab will pull all branches and tags from Codeberg automatically.

2. Enable CI on the mirror

.gitlab-ci.yml is already in the repository. GitLab detects it and runs pipelines on every push — including mirror syncs.

If your GitLab instance uses tagged runners (not shared runners), add the tag as a CI/CD variable:

Settings → CI/CD → Variables
  RUNNER_TAG = <your runner tag>

Then add tags: [$RUNNER_TAG] to jobs in .gitlab-ci.yml for your local deployment, or configure a default runner tag in your project settings.

3. Override CI/CD variables for your instance

Do not edit .gitlab-ci.yml to change database credentials or other local settings. Instead, set CI/CD variables in Settings → CI/CD → Variables:

Variable Default Purpose
POSTGRES_PASSWORD postgres PostgreSQL password for test database
POSTGRES_DB openlatch_test Test database name
OPENLATCH_DATABASE_URL constructed from above Full async DB URL

Variables set in the GitLab UI override the defaults in .gitlab-ci.yml without touching the file — no merge conflicts.

4. GitLab Pages for docs

The pages job publishes documentation to your GitLab Pages URL (https://<group>.gitlab.io/openlatch/). It runs on main branch pushes automatically.

If your instance has GitLab Pages disabled or uses a custom domain, set:

Settings → Pages → Force HTTPS, custom domain, etc.

5. Keeping the mirror up to date

GitLab mirrors are read-only pull mirrors by default. Do not commit directly to the mirrored repository — you will diverge from upstream and create conflicts.

If you need local customisations (runner tags, custom build flags, etc.):

  1. Create a branch local/<your-org> from main
  2. Commit your customisations there
  3. Keep main clean (mirror only)
  4. Rebase local/<your-org> onto main after each mirror sync
git fetch origin
git checkout local/in-berlin
git rebase origin/main
# resolve any conflicts (rare — see below)
git push --force-with-lease origin local/in-berlin

Because platform YAML files call ci/*.sh scripts and rarely change structure, rebasing the local branch is almost always conflict-free.

Setting up a Codeberg → GitHub mirror

1. Mirror the repository

Option A — GitHub Importer: github.com/new/import, paste Codeberg URL.

Option B — Manual push mirror from Codeberg: 1. Codeberg repository → Settings → Git Hooks / Mirrors → Push mirrors 2. Add https://github.com/<org>/openlatch.git with a GitHub personal access token

Option B keeps Codeberg as the authoritative source and pushes to GitHub automatically.

2. Enable GitHub Actions

.github/workflows/ci.yml is already in the repository. GitHub detects it automatically.

3. Enable GitHub Pages

For the docs deploy step to work:

  1. Settings → Pages → Source: set to GitHub Actions
  2. The docs job will deploy on every push to main

4. GitHub Actions permissions

The docs job requires the pages: write and id-token: write permissions, which are already set in .github/workflows/ci.yml. No additional setup is needed unless your organisation has restricted workflow permissions.

Platform-specific limitations

Forgejo (Codeberg)

  • Action marketplace: not all GitHub Actions marketplace actions are available. The CI uses only well-known actions (actions/checkout, actions/setup-python, actions/cache, actions/upload-artifact) which are mirrored on Codeberg.
  • Codeberg Pages: served from the pages branch at https://<org>.codeberg.page/<repo>/. The mkdocs gh-deploy --remote-branch pages command creates and updates this branch automatically.
  • Self-hosted runners: Codeberg provides shared runners. For the ESP32 build you may want a self-hosted runner with a cached PlatformIO environment to avoid the ~500 MB toolchain download.

GitHub Actions

  • Pages deployment: uses the newer actions/upload-pages-artifact + actions/deploy-pages approach (requires the Pages source set to "GitHub Actions" in repository settings).
  • Free tier limits: public repositories have unlimited minutes. Private repositories have a monthly quota.

GitLab CI

  • GitLab Pages URL: https://<group>.gitlab.io/<project>/. If the project is in a subgroup, the URL includes all subgroup names.
  • Container registry: the firmware:build job produces firmware.bin as a job artifact, not a container image. There is intentionally no docker build step — a backend Dockerfile will be added when containerised deployment is implemented.
  • Job names: the pages job name is reserved by GitLab — renaming it will break Pages deployment.
  • Mirror sync triggers: when GitLab syncs a mirror, it triggers a pipeline on the main branch. This is normal behaviour and expected.

Running CI scripts locally

All ci/*.sh scripts can be run locally for debugging:

# Backend lint (from repo root)
./ci/backend-lint.sh

# Backend tests (requires PostgreSQL running and DATABASE_URL set)
export OPENLATCH_DATABASE_URL="postgresql+asyncpg://postgres:postgres@localhost/openlatch_test"
cd backend && pip install -e ".[dev]" && cd ..
./ci/backend-test.sh

# Firmware native tests (requires libsodium-dev and platformio)
pip install platformio
./ci/firmware-test.sh

# Firmware ESP32 build
./ci/firmware-build.sh

# Security scan
./ci/security-scan.sh

Use the tools/docker-compose.yml to start a local PostgreSQL for the backend tests:

cd tools && docker compose up -d