ci: isole la base Postgres du job test par run
Deux jobs `test` qui se chevauchent (push de branche peu après un merge sur `main`) se connectaient au *même* conteneur `--name postgres` sur le réseau partagé du runner Forgejo : leurs `prisma migrate deploy` + `seedReferenceData` (beforeAll racine Mocha) se télescopaient — P2002 sur `key`, « No pending migrations » à tort, décalage des ids de `tech_step`/`diet` cassant des assertions sans rapport. Vu 3× en une session (PR #47, #53, #54), à chaque fois re-déclenché par un commit vide sur un runner au repos. `concurrency:` serait le correctif naturel mais ce Forgejo (9.0.3) est antérieur à son support côté Actions (arrivé en Forgejo v14 / Gitea 1.26, et workflow-level uniquement même là) : la clé est ignorée silencieusement. À la place, le conteneur prend un nom unique par run (`postgres-${GITHUB_RUN_ID}`), poussé dans `DATABASE_URL` via `$GITHUB_ENV` pour les étapes suivantes — deux runs ne partagent plus jamais de base. Nettoyage : étape de teardown `if: always()` + purge des conteneurs abandonnés (bornée par l'âge) au démarrage, pour un job tué net avant son teardown. `timeout-minutes: 40` borne aussi le cas « job zombie ~1h » observé cette session (Mocha sans `--exit`). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
713d6c520d
commit
721a1ecf81
1 changed files with 81 additions and 15 deletions
96
.github/workflows/ci.yml
vendored
96
.github/workflows/ci.yml
vendored
|
|
@ -12,13 +12,18 @@ on:
|
||||||
push:
|
push:
|
||||||
|
|
||||||
env:
|
env:
|
||||||
# Hostname `postgres` = the container the `test` job starts itself (see
|
# Only the `test` job ever opens a database connection, and it *overwrites*
|
||||||
# that job's "Start Postgres" step) — a plain `docker run`, not a
|
# this value at runtime (see its "Start Postgres" step) with a per-run
|
||||||
# `services:` block, for the same reason the intent-service below can't be
|
# hostname. This entry stays for one reason: `build`'s `pnpm install` runs
|
||||||
# one: this repo's Forgejo `act` runner does not keep a `services:`
|
# `prisma generate` (apps/api `postinstall`), which wants DATABASE_URL to
|
||||||
# container as the *same* instance across a job's steps, so `migrate
|
# be present and syntactically valid — it never dials the host, so the
|
||||||
# deploy` would land the schema in one Postgres and `pnpm --filter api
|
# literal `postgres` here need not resolve anywhere.
|
||||||
# test`'s `beforeAll` would then hit an empty one ("table … does not
|
#
|
||||||
|
# Why the `test` job runs Postgres as a plain `docker run` and not a
|
||||||
|
# `services:` block: this repo's Forgejo `act` runner does not keep a
|
||||||
|
# `services:` container as the *same* instance across a job's steps, so
|
||||||
|
# `migrate deploy` would land the schema in one Postgres and `pnpm --filter
|
||||||
|
# api test`'s `beforeAll` would then hit an empty one ("table … does not
|
||||||
# exist" right after "migrations successfully applied"). One long-lived
|
# exist" right after "migrations successfully applied"). One long-lived
|
||||||
# container in the job, shared by every step, removes that failure mode.
|
# container in the job, shared by every step, removes that failure mode.
|
||||||
DATABASE_URL: "postgresql://ci:ci@postgres:5432/batchcooking_ci?schema=public"
|
DATABASE_URL: "postgresql://ci:ci@postgres:5432/batchcooking_ci?schema=public"
|
||||||
|
|
@ -61,6 +66,13 @@ jobs:
|
||||||
|
|
||||||
test:
|
test:
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
|
# Runaway ceiling, not a target: the job takes ~16-19 min normally (runs
|
||||||
|
# 117-121). Mocha runs with no `--exit` (`apps/api/.mocharc.json`), so a
|
||||||
|
# single leaked handle would otherwise hang this job until the runner's
|
||||||
|
# global timeout — a `test` job zombied for ~1h once before being
|
||||||
|
# cancelled by hand. 40 min leaves roughly 2x headroom for a slow /
|
||||||
|
# contended runner while still capping a hang at well under that hour.
|
||||||
|
timeout-minutes: 40
|
||||||
steps:
|
steps:
|
||||||
- name: Install curl and the docker CLI
|
- name: Install curl and the docker CLI
|
||||||
run: |
|
run: |
|
||||||
|
|
@ -68,26 +80,70 @@ jobs:
|
||||||
|
|
||||||
# Postgres as a plain background container instead of a `services:`
|
# Postgres as a plain background container instead of a `services:`
|
||||||
# block — see this workflow's top-level `env.DATABASE_URL` comment for
|
# block — see this workflow's top-level `env.DATABASE_URL` comment for
|
||||||
# why. Reachable as `postgres` from this job (the runner mounts the
|
# why. The runner mounts the docker socket and runs every job on one
|
||||||
# docker socket and runs jobs on a shared user-defined network, so a
|
# shared user-defined network, so a `--name` sibling resolves by name;
|
||||||
# `--name postgres` sibling resolves by name); the network is
|
# the network is discovered from the job container itself rather than
|
||||||
# discovered from the job container itself rather than hardcoded.
|
# hardcoded.
|
||||||
|
#
|
||||||
|
# The container name carries the run id (`postgres-${GITHUB_RUN_ID}`)
|
||||||
|
# rather than a bare `postgres` *because* that network is shared by
|
||||||
|
# every job on this runner, across all branches. With a fixed name,
|
||||||
|
# two `test` jobs overlapping — a feature push landing within minutes
|
||||||
|
# of a `main` merge — both connected to the *same* container, and
|
||||||
|
# their `prisma migrate deploy` + `seedReferenceData` (root Mocha
|
||||||
|
# `beforeAll`) raced: `Unique constraint failed on the fields: (key)`
|
||||||
|
# in `seedReferenceData`, "No pending migrations to apply" when there
|
||||||
|
# should be some, `tech_step`/`diet` reference-row id shifts (+5)
|
||||||
|
# breaking unrelated assertions. Hit three times in one session (PRs
|
||||||
|
# #47, #53, #54), each needing an empty-commit re-trigger once the
|
||||||
|
# runner was idle. A GitHub-style `concurrency:` group on this job is
|
||||||
|
# the natural fix, but this Forgejo (9.0.3) predates Actions
|
||||||
|
# `concurrency` support entirely — it arrived in Forgejo v14 / Gitea
|
||||||
|
# 1.26 and the key is silently ignored on anything older. A per-run
|
||||||
|
# container name reaches the same end (no two runs share a database)
|
||||||
|
# with zero runner support required; the `if: always()` teardown step
|
||||||
|
# at the end of the job, plus the age-gated reap below, keep these
|
||||||
|
# per-run containers from piling up on the runner host.
|
||||||
- name: Start Postgres in the background
|
- name: Start Postgres in the background
|
||||||
run: |
|
run: |
|
||||||
|
pg_host="postgres-${GITHUB_RUN_ID}"
|
||||||
|
|
||||||
|
# Best-effort reap of per-run Postgres containers left behind by a
|
||||||
|
# `test` job that was force-killed before its teardown step could
|
||||||
|
# run (SIGKILL skips `if: always()` steps). Age-gated well above
|
||||||
|
# this job's `timeout-minutes`, so a slower *concurrent* `test`
|
||||||
|
# job's container is never in range.
|
||||||
|
reap_before=$(( $(date +%s) - 7200 ))
|
||||||
|
for c in $(docker ps -aq --filter "name=postgres-"); do
|
||||||
|
started="$(docker inspect -f '{{.State.StartedAt}}' "$c" 2>/dev/null)" || continue
|
||||||
|
started_epoch="$(date -d "$started" +%s 2>/dev/null)" || continue
|
||||||
|
if [ "$started_epoch" -lt "$reap_before" ]; then
|
||||||
|
echo "reaping abandoned Postgres container $c (started $started)"
|
||||||
|
docker rm -f "$c" >/dev/null 2>&1 || true
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
net="$(docker inspect "$(hostname)" \
|
net="$(docker inspect "$(hostname)" \
|
||||||
--format '{{range $k, $_ := .NetworkSettings.Networks}}{{$k}} {{end}}' \
|
--format '{{range $k, $_ := .NetworkSettings.Networks}}{{$k}} {{end}}' \
|
||||||
| awk '{print $1}')"
|
| awk '{print $1}')"
|
||||||
echo "job network: ${net:?could not determine the job container network}"
|
echo "job network: ${net:?could not determine the job container network}"
|
||||||
docker rm -f postgres >/dev/null 2>&1 || true
|
docker rm -f "$pg_host" >/dev/null 2>&1 || true
|
||||||
docker run -d --name postgres --network "$net" \
|
docker run -d --name "$pg_host" --network "$net" \
|
||||||
-e POSTGRES_USER=ci -e POSTGRES_PASSWORD=ci -e POSTGRES_DB=batchcooking_ci \
|
-e POSTGRES_USER=ci -e POSTGRES_PASSWORD=ci -e POSTGRES_DB=batchcooking_ci \
|
||||||
postgres:16-alpine
|
postgres:16-alpine
|
||||||
|
|
||||||
|
# Hand this run's hostname to every later step (prisma migrate
|
||||||
|
# deploy, pnpm --filter api test) — this line overrides the
|
||||||
|
# workflow-level env.DATABASE_URL, whose `postgres` host no longer
|
||||||
|
# resolves.
|
||||||
|
echo "DATABASE_URL=postgresql://ci:ci@${pg_host}:5432/batchcooking_ci?schema=public" >> "$GITHUB_ENV"
|
||||||
|
|
||||||
for _ in $(seq 1 60); do
|
for _ in $(seq 1 60); do
|
||||||
docker exec postgres pg_isready -U ci -d batchcooking_ci && exit 0
|
docker exec "$pg_host" pg_isready -U ci -d batchcooking_ci && exit 0
|
||||||
sleep 2
|
sleep 2
|
||||||
done
|
done
|
||||||
echo "Postgres did not become ready in time" >&2
|
echo "Postgres did not become ready in time" >&2
|
||||||
docker logs postgres || true
|
docker logs "$pg_host" || true
|
||||||
exit 1
|
exit 1
|
||||||
|
|
||||||
- uses: https://github.com/actions/checkout@v4
|
- uses: https://github.com/actions/checkout@v4
|
||||||
|
|
@ -134,6 +190,16 @@ jobs:
|
||||||
- run: pnpm --filter api exec prisma migrate deploy
|
- run: pnpm --filter api exec prisma migrate deploy
|
||||||
- run: pnpm --filter api test
|
- run: pnpm --filter api test
|
||||||
|
|
||||||
|
# Remove this run's Postgres even when a step above failed or the job
|
||||||
|
# hit its `timeout-minutes` — with a per-run name nothing else ever
|
||||||
|
# reuses or cleans it, and a bare `docker run` sibling outlives the job
|
||||||
|
# container. `|| true`: teardown must never itself fail the job. A job
|
||||||
|
# killed outright (SIGKILL, no `always()` step) is caught by the
|
||||||
|
# age-gated reap in "Start Postgres" on a later run.
|
||||||
|
- name: Stop this run's Postgres
|
||||||
|
if: always()
|
||||||
|
run: docker rm -f "postgres-${GITHUB_RUN_ID}" >/dev/null 2>&1 || true
|
||||||
|
|
||||||
intent-service-test:
|
intent-service-test:
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
steps:
|
steps:
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue