Nicolas
53d415fddb
Une seule feature livree en une seule PR, en 5 phases : - Phase 1 : enrichit le corpus NLP (tech-step-training-data.ts) et ajoute un harness d'evaluation (precision/rappel/F1) avec un jeu de test etiquete - la premiere metrique objective de qualite pour ce classifieur. - Phase 2 : schema Prisma (StepTechStepCorrection, TechStepTrainingSuggestion) + endpoints utilisateur (POST/GET corrections, ouverts a tout viewer, pas seulement l'auteur) + endpoints internes /internal/tech-steps/* proteges par secret partage (requireInternalWorker). - Phase 3 : UI de highlight/correction cote web (selection de texte -> association a une technique, ou clic sur un highlight existant pour le corriger/supprimer) - verifiee via Cypress (component + e2e, en Chrome reel). - Phase 4 : worker LLM autonome (services/tech-step-llm-worker, hors du monorepo pnpm comme experiments/llm-tech-step-poc) qui audite les clauses a faible confiance et transforme les corrections utilisateur en suggestions d'entrainement, sans jamais toucher le chemin interactif. - Phase 5 : script retrain-tech-steps.ts (gate de regression F1 + backfill) et list-pending-training-suggestions.ts pour la revue humaine avant application au corpus. Verification effectuee cette session : tsc/biome sur l'ensemble du repo, build complet (pnpm build), suite Cypress complete (component 39/39, e2e 75/76 - le seul echec est preexistant et sans rapport, cote recipe-form.feature/ingredient-picker), tests unitaires du worker (6/6) et son install/typecheck reels contre node-llama-cpp. Les tests Mocha d'apps/api (Phases 1 et 2) n'ont pas pu etre executes dans cette session (pas de Postgres local disponible) - a lancer avant merge. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
47 lines
4.4 KiB
Markdown
47 lines
4.4 KiB
Markdown
# tech-step-llm-worker
|
|
|
|
Standalone scheduled worker for the tech-step detection reliability feature (see the repo root's feature plan). Periodically:
|
|
|
|
1. **`audit-low-confidence`** — samples clauses `apps/api`'s NLP classifier (`tech-step-matcher.ts`) itself scored below its own confidence threshold, asks a local LLM for a second opinion, and proposes a new training utterance whenever the LLM disagrees with what the NLP anchor already implied.
|
|
2. **`transform-corrections`** — drains user-submitted tech-step corrections (`StepDescription.tsx`'s editable mode, `apps/web`) not yet processed, and asks the LLM to propose new synonyms/example utterances from each one.
|
|
|
|
Both jobs only ever **propose** `TechStepTrainingSuggestion` rows for a maintainer to review — nothing here edits `tech-step-training-data.ts` automatically. See `apps/api/src/scripts/retrain-tech-steps.ts` for the maintainer-driven step that actually applies reviewed suggestions.
|
|
|
|
## Why this lives outside the pnpm workspace
|
|
|
|
Same reasoning as `experiments/llm-tech-step-poc`: `node-llama-cpp`'s native binding must never end up compiled into `apps/api`'s own install/Docker build. This package has its own `package.json`/lockfile-less install, entirely separate from `pnpm-workspace.yaml` (which only covers `apps/*`/`packages/*`).
|
|
|
|
It also has **no Prisma client and no direct database access** — every read/write goes through `apps/api`'s `/internal/tech-steps/*` routes (`api-client.ts`), authenticated with a shared secret (`INTERNAL_WORKER_SECRET`, must match `apps/api`'s own). This keeps `apps/api` the single owner of the schema, and keeps this worker a simple "read some text over HTTP, run local inference, POST a suggestion" process with nothing to keep in sync if the schema changes shape.
|
|
|
|
## Setup
|
|
|
|
```bash
|
|
cd services/tech-step-llm-worker
|
|
pnpm install --ignore-workspace
|
|
cp .env.example .env
|
|
# edit .env: set INTERNAL_WORKER_SECRET to match apps/api's own
|
|
pnpm start # runs the cron loop
|
|
# or:
|
|
RUN_ONCE=true pnpm start # runs both jobs once and exits
|
|
```
|
|
|
|
The GGUF model (`qwen2.5-1.5b` by default, `Q4_K_M`, ~1GB) downloads on first run into `./models/` (gitignored) and is cached there for subsequent runs — expect the very first run to take noticeably longer than later ones. See `src/config.ts` for every environment variable this reads, including `TECH_STEP_LLM_MODEL_PATH` to point at an already-downloaded GGUF file instead (useful offline, or when a mid-deploy network download isn't wanted).
|
|
|
|
## Running via Docker Compose
|
|
|
|
`docker-compose.yml` (repo root) defines a `tech-step-llm-worker` service alongside `app`/`postgres` — it's optional: set `INTERNAL_WORKER_SECRET` in the root `.env` to enable it, leave it unset and the service simply won't start (its `environment:` block fails loudly if referenced without a value, same posture as the other required secrets in that file).
|
|
|
|
## Testing
|
|
|
|
```bash
|
|
pnpm test
|
|
```
|
|
|
|
Unit tests (`test/jobs/*.test.ts`) mock `api-client.ts`'s HTTP calls and a fake `TechStepLlmService`-shaped object directly — no real network calls, no real model loaded, no real `apps/api` needed. There is currently no integration test exercising a real model against a real `apps/api` instance; that would need to be run manually (see "Setup" above) before merging any future change to the prompts/schemas in `llm-verdict.ts`.
|
|
|
|
## Known limitations (first version of this feature)
|
|
|
|
- **Scheduler cadence** (`TECH_STEP_WORKER_CRON`, default weekly) is a provisional floor, not a calibrated value — see the feature's plan document for what it should be tuned against (recipe/correction volume, server resources).
|
|
- **Sampling in `audit-low-confidence`** only looks at the `AUDIT_SAMPLE_SIZE` (`apps/api`'s `tech-step-worker.service.ts`) most-recently-created steps, not the whole recipe catalog — a smarter sampling strategy (e.g. weighted by how often a recipe is actually viewed/planned) is future work.
|
|
- **No per-key technique definitions** are sent to the LLM today — just the bare `TechStep.key` list (`GET /reference/tech-steps`, e.g. `"panFry"`, `"foldIn"`). Adding a short human-readable gloss per technique (a new `TechStepView.description` field) would likely improve `judgeClause`'s accuracy but is out of scope for this version.
|
|
- **Locale is always assumed `"fr"`** in `transform-corrections` — no recipe/step in the app carries its own locale field yet (see `recipe.service.ts`'s `DEFAULT_TECH_STEP_LOCALE` comment on the API side).
|