batchCooking/.github/workflows/ci.yml
Nicolas 393300205d
Some checks failed
CI / build (push) Successful in 1m27s
CI / e2e (push) Successful in 6m39s
CI / lint (push) Successful in 1m41s
CI / intent-service-test (push) Successful in 3m4s
CI / test (push) Failing after 12m43s
perf(tech-steps): corrige la largeur du textcat, entrainement 32x plus rapide
Le service mettait ~930 s (15-16 min) a devenir `healthy`, ce que le README
presentait comme "un compromis assume, pas un defaut de configuration a
corriger". Mauvais diagnostic : le cout ne venait ni des iterations ni de la
taille du corpus, mais du parametre `length` de `spacy.TextCatBOW.v3`, laisse
a son defaut spaCy de 262144 faute d'etre passe explicitement.

Avec `no_output_layer: False` et 74 techniques, la couche de sortie faisait
19 398 730 parametres mis a jour par Adam a chaque pas de gradient, pour une
table de hachage vide a 99 % (le corpus tient dans quelques milliers de
tokens distincts).

Mesure contre le vrai corpus (fr, 25 iterations, jeu de dev = derniere
utterance de chaque technique tenue hors entrainement) :

  length=262144 ngram=1  dev 97.3%  conf 0.809  394.7 s   <- avant
  length=2048   ngram=2  dev 97.3%  conf 0.919   12.3 s   <- apres

Meme accuracy, meilleure confiance, 32x plus rapide. Les bigrammes profitent
des locutions du corpus ("faire fondre", "laisser mijoter") qu'un sac
d'unigrammes ne distingue pas de leurs constituants isoles.

PipelineRegistry.initialize() (fr+en, chargement + entrainement) mesure a
28,5 s contre ~930 s. Suite pytest complete : 46 passed en 40,6 s.

- locale_pipeline.py : `_TEXTCAT_MODEL_LENGTH` / `_TEXTCAT_NGRAM_SIZE`
  explicites et commentes avec les mesures ; historique de calibration de
  `_TRAINING_ITERATIONS` corrige (il attribuait le cout aux iterations)
- README : sections "Temps de demarrage" et "Limitations connues" reecrites
  (la couverture d'evaluation, 26/74 techniques, y devient la limitation
  bloquante ; ajout des collisions de classes observees)
- docker-compose.yml / ci.yml : commentaires citant ~930 s corriges,
  start_period et timeout 1200s -> 300s (plafond pour runner lent, pas cible)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-03 18:01:06 +02:00

205 lines
8.6 KiB
YAML

name: CI
# Every push, on every branch — not just main — so build breakage shows up
# on the first commit of a branch, not only once a PR targets main. No
# separate pull_request trigger: for a PR from a branch on this same repo,
# push already fires on that branch, and GitHub attaches the run to the PR
# by commit SHA regardless of which event triggered it — adding
# pull_request too would just run every job twice per push. (Re-add it,
# scoped to forks, only if this repo starts accepting fork PRs — push
# events from a fork never reach here.)
on:
push:
env:
# Hostname `postgres` = the container the `test` job starts itself (see
# that job's "Start Postgres" step) — a plain `docker run`, not a
# `services:` block, for the same reason the intent-service below can't be
# one: this repo's Forgejo `act` runner does not keep a `services:`
# container as the *same* instance across a job's steps, so `migrate
# deploy` would land the schema in one Postgres and `pnpm --filter api
# test`'s `beforeAll` would then hit an empty one ("table … does not
# exist" right after "migrations successfully applied"). One long-lived
# container in the job, shared by every step, removes that failure mode.
DATABASE_URL: "postgresql://ci:ci@postgres:5432/batchcooking_ci?schema=public"
# Test-only secret, never used outside CI — real deployments must set their own.
JWT_SECRET: "ci-only-secret-not-used-anywhere-else-32chars+"
# Same reasoning as JWT_SECRET above — lets tech-step-worker.routes.test.ts
# exercise the success path (matching secret), not just the "unset"
# rejection every environment that doesn't set this gets by default.
INTERNAL_WORKER_SECRET: "ci-only-worker-secret-not-used-anywhere-else-32chars+"
# Same reasoning — lets admin-auth.test.ts exercise the admin login
# success path + the requireAdmin-guarded routes, not just the "no admin
# secret configured -> 401" path.
ADMIN_JWT_SECRET: "ci-only-admin-secret-not-used-anywhere-else-32chars+"
# Shared between the `test` job's own uvicorn step (below) and apps/api's
# IntentServiceClient — that step runs `uvicorn` as a plain background
# process (not a `services:` block or even a container) because CI can
# only pull a published image, not build
# services/tech-step-intent-service/Dockerfile from this checkout.
INTENT_SERVICE_BASE_URL: "http://localhost:8000"
INTENT_SERVICE_SECRET: "ci-only-intent-secret-not-used-anywhere-else-32chars+"
jobs:
# Five independent jobs, no needs: between them — each starts in parallel
# and reports as its own check, instead of the previous single chained
# "lint-and-test then e2e" pipeline.
lint:
runs-on: ubuntu-latest
steps:
- uses: https://github.com/actions/checkout@v4
- uses: pnpm/action-setup@v4
- uses: https://github.com/actions/setup-node@v4
with:
node-version: 22
cache: pnpm
- run: pnpm install --frozen-lockfile
- run: pnpm lint
test:
runs-on: ubuntu-latest
steps:
- name: Install curl and the docker CLI
run: |
apt-get update && apt-get install -y curl docker.io
# Postgres as a plain background container instead of a `services:`
# block — see this workflow's top-level `env.DATABASE_URL` comment for
# why. Reachable as `postgres` from this job (the runner mounts the
# docker socket and runs jobs on a shared user-defined network, so a
# `--name postgres` sibling resolves by name); the network is
# discovered from the job container itself rather than hardcoded.
- name: Start Postgres in the background
run: |
net="$(docker inspect "$(hostname)" \
--format '{{range $k, $_ := .NetworkSettings.Networks}}{{$k}} {{end}}' \
| awk '{print $1}')"
echo "job network: ${net:?could not determine the job container network}"
docker rm -f postgres >/dev/null 2>&1 || true
docker run -d --name postgres --network "$net" \
-e POSTGRES_USER=ci -e POSTGRES_PASSWORD=ci -e POSTGRES_DB=batchcooking_ci \
postgres:16-alpine
for _ in $(seq 1 60); do
docker exec postgres pg_isready -U ci -d batchcooking_ci && exit 0
sleep 2
done
echo "Postgres did not become ready in time" >&2
docker logs postgres || true
exit 1
- uses: https://github.com/actions/checkout@v4
- uses: pnpm/action-setup@v4
- uses: https://github.com/actions/setup-node@v4
with:
node-version: 22
cache: pnpm
- uses: https://github.com/astral-sh/setup-uv@v5
with:
python-version: "3.12"
enable-cache: true
# A `services:` block can only pull an already-published image — it
# can't build services/tech-step-intent-service/Dockerfile from this
# checkout. Running `uvicorn` as a plain background step instead: it
# keeps running for the rest of this job (steps in one job share the
# same process tree), and `pnpm --filter api test` below needs a real
# instance to talk to per this repo's "never mock an internal service"
# test convention — same "real dependency, started by the job itself"
# posture as the Postgres step above, not a mock HTTP server.
- name: Install services/tech-step-intent-service
working-directory: services/tech-step-intent-service
run: uv sync --frozen
- name: Start services/tech-step-intent-service in the background
working-directory: services/tech-step-intent-service
run: |
uv run uvicorn intent_service.main:app --host 0.0.0.0 --port 8000 &
# `/health` only returns 200 once this service has finished
# training itself from scratch (no model ever persisted to disk —
# see its own README) — measured at ~30s combined (fr+en) against
# the current ~74-technique corpus, down from ~930s once
# `spacy.TextCatBOW.v3`'s `length` stopped being left at its
# 262144 default (see `_TEXTCAT_MODEL_LENGTH` in
# `locale_pipeline.py`, and docker-compose.yml's healthcheck for
# the same reasoning). The timeout stays well above that: it's a
# ceiling for a slow/contended runner, not a target.
timeout 300 bash -c 'until curl -sf http://localhost:8000/health > /dev/null; do sleep 2; done'
- run: pnpm install --frozen-lockfile
- run: pnpm --filter api exec prisma migrate deploy
- run: pnpm --filter api test
intent-service-test:
runs-on: ubuntu-latest
steps:
- uses: https://github.com/actions/checkout@v4
- uses: https://github.com/astral-sh/setup-uv@v5
with:
python-version: "3.12"
enable-cache: true
- name: Install services/tech-step-intent-service
working-directory: services/tech-step-intent-service
run: uv sync --frozen
- name: Run pytest
working-directory: services/tech-step-intent-service
run: uv run pytest -q
build:
runs-on: ubuntu-latest
steps:
- uses: https://github.com/actions/checkout@v4
- uses: pnpm/action-setup@v4
- uses: https://github.com/actions/setup-node@v4
with:
node-version: 22
cache: pnpm
- run: pnpm install --frozen-lockfile
- run: pnpm build
e2e:
runs-on: ubuntu-latest
steps:
- uses: https://github.com/actions/checkout@v4
- name: Install Xvfb and Cypress dependencies
run: |
apt-get update
apt-get install -y xvfb libgtk2.0-0 libgtk-3-0 libgbm-dev libnotify-dev libnss3 libxss1 libasound2 libxtst6 xauth dbus-x11
- uses: pnpm/action-setup@v4
- uses: https://github.com/actions/setup-node@v4
with:
node-version: 22
cache: pnpm
- name: Cache Cypress binary
uses: https://github.com/actions/cache@v4
with:
path: ~/.cache/Cypress
key: cypress-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml') }}
- run: pnpm install --frozen-lockfile
# pnpm install doesn't reliably trigger Cypress's postinstall binary
# download (see apps/web's cypress caveat in the README) — install it
# explicitly so `cypress run` finds it.
- run: pnpm --filter web exec cypress install
- name: Run E2E tests
env:
HOST: "0.0.0.0"
run: pnpm --filter web e2e
# No dev server needed here — Cypress spins up its own Vite dev
# server internally for component testing (see cypress.config.ts's
# `component.devServer`), unlike `e2e` above which needs the real app
# running first.
- run: pnpm --filter web cy:run:component