feat(recipes): rapatrie le corpus NLP cote Python et l'enrichit de 48 techniques
Changement d'architecture demande par l'utilisateur : le dataset d'entrainement (TECH_STEP_TRAINING_DATA) quitte apps/api pour vivre entierement dans services/tech-step-intent-service (intent_service/training_data.py). Ce service est desormais autonome : il s'entraine lui-meme une seule fois, a son propre demarrage (PipelineRegistry.initialize, dans le lifespan FastAPI), sans plus dependre d'un POST /v1/train pousse par apps/api (route supprimee). apps/api ne connait plus aucune technique/synonyme, uniquement le resultat de POST /v1/process. Corpus enrichi avec les 48 techniques du lexique fourni (Arroser, Appertiser, Braiser, Caraméliser, Confire, Julienne/Brunoise/Mirepoix/ Paysanne, Cuire à blanc/au bain-marie/à l'étouffée, Déglacer variantes, Emulsionner, Glacer, Pocher, Réduire, Suer, Zester, etc.), soit 74 techniques au total (26 + 48). Integration complete bout en bout : - reference-seed-data.ts : 48 nouvelles entrees TECH_STEPS - apps/web/locales/fr/translation.json : libelles francais correspondants - "Mitonner" fondu comme synonyme de simmer (pas une technique distincte, sa propre definition le dit) - "Blanchir un oeuf" (whiskPale) distingue de "Blanchir un legume" (blanch, existant) via des synonymes en phrase complete plutot qu'au mot nu — filter_spans (deja en place) resout la collision par specificite Impact performance mesure : le corpus elargi (74 classes vs 26) rend l'entrainement bien plus lent a nombre d'iterations egal (150 iterations depassait 17 minutes par run de test) — reduit a 40 iterations apres mesures repetees en local (~200s/locale, ~400s pour fr+en combines). docker-compose.yml (healthcheck start_period 600s), CI (timeout curl 600s) et le README du service documentent ce nouveau temps de demarrage. CONFIDENCE_THRESHOLD recalibre a 0.2 par verification manuelle (0.75 puis 0.45 ne tenaient plus compte tenu du nombre de classes) — marque explicitement comme placeholder en attendant une vraie repasse de calibrate-tech-step-threshold.ts (necessite Postgres, indisponible dans cet environnement). Verifie : 28/28 tests pytest du service (suite complete re-ecrite pour s'entrainer une seule fois par session sur le vrai corpus, fixture partagee dans conftest.py), lint + build complets du monorepo. La suite Mocha d'apps/api reste a confirmer via CI (le root hook mocha n'attend plus l'entrainement, seulement CI's propre attente sur /health). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
74a0052431
commit
065ef2a31a
33 changed files with 2199 additions and 1393 deletions
10
.github/workflows/ci.yml
vendored
10
.github/workflows/ci.yml
vendored
|
|
@ -94,7 +94,15 @@ jobs:
|
||||||
working-directory: services/tech-step-intent-service
|
working-directory: services/tech-step-intent-service
|
||||||
run: |
|
run: |
|
||||||
uv run uvicorn intent_service.main:app --host 0.0.0.0 --port 8000 &
|
uv run uvicorn intent_service.main:app --host 0.0.0.0 --port 8000 &
|
||||||
timeout 60 bash -c 'until curl -sf http://localhost:8000/health > /dev/null; do sleep 1; done'
|
# `/health` only returns 200 once this service has finished
|
||||||
|
# training itself from scratch (no model ever persisted to disk —
|
||||||
|
# see its own README) — measured at ~200s per locale (~400s for
|
||||||
|
# fr+en combined) against the current ~74-technique corpus, so
|
||||||
|
# this wait is generous rather than the fast "base models only"
|
||||||
|
# check it used to be before that service trained itself at
|
||||||
|
# startup (see docker-compose.yml's healthcheck for the same
|
||||||
|
# reasoning).
|
||||||
|
timeout 600 bash -c 'until curl -sf http://localhost:8000/health > /dev/null; do sleep 2; done'
|
||||||
|
|
||||||
- run: pnpm install --frozen-lockfile
|
- run: pnpm install --frozen-lockfile
|
||||||
- run: pnpm --filter api exec prisma migrate deploy
|
- run: pnpm --filter api exec prisma migrate deploy
|
||||||
|
|
|
||||||
|
|
@ -103,8 +103,10 @@ pnpm --filter api exec prisma migrate dev
|
||||||
pnpm --filter api prisma:seed
|
pnpm --filter api prisma:seed
|
||||||
|
|
||||||
# Microservice de détection des techniques (spaCy) — requis, `pnpm dev:api`
|
# Microservice de détection des techniques (spaCy) — requis, `pnpm dev:api`
|
||||||
# ne peut plus détecter aucune technique de cuisine sans lui (voir son
|
# ne peut plus détecter aucune technique de cuisine sans lui. Lance-le en
|
||||||
# propre README pour le détail)
|
# premier et laisse-le tourner : il s'entraîne lui-même à chaque démarrage
|
||||||
|
# (~7 minutes pour le corpus actuel, voir son propre README) avant de
|
||||||
|
# répondre quoi que ce soit sur /health.
|
||||||
cd services/tech-step-intent-service
|
cd services/tech-step-intent-service
|
||||||
uv sync
|
uv sync
|
||||||
cp .env.example .env # édite-le : même INTENT_SERVICE_SECRET que apps/api/.env
|
cp .env.example .env # édite-le : même INTENT_SERVICE_SECRET que apps/api/.env
|
||||||
|
|
|
||||||
|
|
@ -62,11 +62,12 @@ export const UNITS: Array<{ uid: string; type: UnitType; toBaseFactor: number }>
|
||||||
//
|
//
|
||||||
// Just a flat list of stable ids here — the actual matching data (per-
|
// Just a flat list of stable ids here — the actual matching data (per-
|
||||||
// locale synonym lists + example phrasings the classifier trains on) lives
|
// locale synonym lists + example phrasings the classifier trains on) lives
|
||||||
// in `lib/recipe-matching/tech-step-training-data.ts`'s
|
// in `services/tech-step-intent-service/intent_service/training_data.py`'s
|
||||||
// `TECH_STEP_TRAINING_DATA`, not here: unlike this list, it's read by
|
// `TECH_STEP_TRAINING_DATA`, not here: it's owned and trained entirely by
|
||||||
// `TechStepClassifierService`'s training pass, not the seed script, so it
|
// that separate Python service (see its own README), not read by this
|
||||||
// doesn't belong alongside the rest of this file's DB-seeded reference
|
// seed script at all, so it doesn't belong alongside the rest of this
|
||||||
// data. Every entry here must have a matching entry there.
|
// file's DB-seeded reference data. Every entry here must have a matching
|
||||||
|
// entry there.
|
||||||
export const TECH_STEPS: string[] = [
|
export const TECH_STEPS: string[] = [
|
||||||
"cook",
|
"cook",
|
||||||
"fry",
|
"fry",
|
||||||
|
|
@ -94,6 +95,57 @@ export const TECH_STEPS: string[] = [
|
||||||
"bake",
|
"bake",
|
||||||
"plate",
|
"plate",
|
||||||
"coat",
|
"coat",
|
||||||
|
// Lexique de techniques ajouté par la suite — voir
|
||||||
|
// `services/tech-step-intent-service/intent_service/training_data.py`
|
||||||
|
// pour les synonymes/phrases d'exemple de chacune.
|
||||||
|
"baste",
|
||||||
|
"appertize",
|
||||||
|
"whiskPale",
|
||||||
|
"goldenBrown",
|
||||||
|
"braise",
|
||||||
|
"truss",
|
||||||
|
"caramelize",
|
||||||
|
"score",
|
||||||
|
"lineMold",
|
||||||
|
"clarify",
|
||||||
|
"compote",
|
||||||
|
"concasse",
|
||||||
|
"confit",
|
||||||
|
"julienne",
|
||||||
|
"brunoise",
|
||||||
|
"mirepoix",
|
||||||
|
"paysanne",
|
||||||
|
"blindBake",
|
||||||
|
"bainMarie",
|
||||||
|
"smother",
|
||||||
|
"decant",
|
||||||
|
"dilute",
|
||||||
|
"punchDown",
|
||||||
|
"disgorge",
|
||||||
|
"loosen",
|
||||||
|
"shellEgg",
|
||||||
|
"scald",
|
||||||
|
"pod",
|
||||||
|
"emulsify",
|
||||||
|
"hollowOut",
|
||||||
|
"shock",
|
||||||
|
"setGel",
|
||||||
|
"glaze",
|
||||||
|
"thicken",
|
||||||
|
"filet",
|
||||||
|
"proof",
|
||||||
|
"peelBlanch",
|
||||||
|
"whipUp",
|
||||||
|
"moisten",
|
||||||
|
"pasteurize",
|
||||||
|
"poach",
|
||||||
|
"reduce",
|
||||||
|
"rubIn",
|
||||||
|
"dustWithFlour",
|
||||||
|
"sweat",
|
||||||
|
"sift",
|
||||||
|
"toast",
|
||||||
|
"zest",
|
||||||
];
|
];
|
||||||
|
|
||||||
// The 14 allergens EU Regulation 1169/2011 (Annex II) requires food
|
// The 14 allergens EU Regulation 1169/2011 (Annex II) requires food
|
||||||
|
|
|
||||||
|
|
@ -2,11 +2,14 @@ import { env } from "../../config/env.js";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Thin fetch wrapper around `services/tech-step-intent-service`'s HTTP
|
* Thin fetch wrapper around `services/tech-step-intent-service`'s HTTP
|
||||||
* contract (`/v1/train`, `/v1/process`) — the microservice
|
* contract (`POST /v1/process`) — the microservice
|
||||||
* {@link TechStepClassifierService} (`tech-step-matcher.ts`) delegates NER +
|
* {@link TechStepClassifierService} (`tech-step-matcher.ts`) delegates NER +
|
||||||
* intent classification to, in place of the `node-nlp` `NlpManager` it used
|
* intent classification to, in place of the `node-nlp` `NlpManager` it used
|
||||||
* to own directly. See that service's own README for the full contract and
|
* to own directly. See that service's own README for the full contract and
|
||||||
* why it never touches Postgres itself.
|
* why it never touches Postgres itself — it also owns its own training
|
||||||
|
* corpus now (`training_data.py`), trained once at its own startup, so
|
||||||
|
* `apps/api` never pushes anything to it; `process()` below is this
|
||||||
|
* client's only method.
|
||||||
*
|
*
|
||||||
* Authenticated with `INTENT_SERVICE_SECRET` — the inverse direction of
|
* Authenticated with `INTENT_SERVICE_SECRET` — the inverse direction of
|
||||||
* `requireInternalWorker`'s `INTERNAL_WORKER_SECRET` (this time `apps/api`
|
* `requireInternalWorker`'s `INTERNAL_WORKER_SECRET` (this time `apps/api`
|
||||||
|
|
@ -21,20 +24,13 @@ export interface IntentServiceEntity {
|
||||||
end: number;
|
end: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** The full result of a `POST /v1/process` call — mirrors `ProcessResponse` (Python `schemas.py`). `intent` is `null` only when `locale` was never trained or `text` is blank; otherwise always a real `uid` (the Python service's `textcat` has no "None" sentinel, unlike node-nlp — see that service's README). */
|
/** The full result of a `POST /v1/process` call — mirrors `ProcessResponse` (Python `schemas.py`). `intent` is `null` only when `locale` isn't one this service trains for, or `text` is blank; otherwise always a real `uid` (the Python service's `textcat` has no "None" sentinel, unlike node-nlp — see that service's README). */
|
||||||
export interface IntentServiceProcessResult {
|
export interface IntentServiceProcessResult {
|
||||||
entities: IntentServiceEntity[];
|
entities: IntentServiceEntity[];
|
||||||
intent: string | null;
|
intent: string | null;
|
||||||
score: number;
|
score: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** One technique's training data for one locale, as sent to `POST /v1/train` — mirrors `TrainEntryPayload` (Python `schemas.py`), itself shaped after `TechStepLocaleTrainingData` (`tech-step-training-data.ts`). */
|
|
||||||
export interface IntentServiceTrainEntry {
|
|
||||||
uid: string;
|
|
||||||
synonyms: string[];
|
|
||||||
utterances: string[];
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Client for `services/tech-step-intent-service` — a real class (not a
|
* Client for `services/tech-step-intent-service` — a real class (not a
|
||||||
* plain object of functions) per this repo's service-style-logic
|
* plain object of functions) per this repo's service-style-logic
|
||||||
|
|
@ -81,25 +77,6 @@ export class IntentServiceClient {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* (Re)trains the intent service's pipeline for `locale` from `entries` —
|
|
||||||
* called once per locale by `TechStepClassifierService._train`, itself
|
|
||||||
* memoized so this only ever runs once per server process (see that
|
|
||||||
* method's own doc comment). Reconstructs the whole pipeline server-side,
|
|
||||||
* never a partial/incremental update — same "always retrains fresh from
|
|
||||||
* the one source of truth" posture the old in-process `NlpManager` had.
|
|
||||||
*/
|
|
||||||
public async train(locale: string, entries: IntentServiceTrainEntry[]): Promise<void> {
|
|
||||||
try {
|
|
||||||
await this._request("/v1/train", {
|
|
||||||
method: "POST",
|
|
||||||
body: JSON.stringify({ locale, entries }),
|
|
||||||
});
|
|
||||||
} catch (err) {
|
|
||||||
throw err;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Equivalent to the old `NlpManager.process(locale, text)` — returns every
|
* Equivalent to the old `NlpManager.process(locale, text)` — returns every
|
||||||
* candidate technique mention (NER) plus the intent classifier's verdict
|
* candidate technique mention (NER) plus the intent classifier's verdict
|
||||||
|
|
|
||||||
|
|
@ -2,8 +2,8 @@
|
||||||
* Hand-labeled evaluation set for {@link techStepClassifier} — what
|
* Hand-labeled evaluation set for {@link techStepClassifier} — what
|
||||||
* `tech-step-eval.test.ts` runs the real classifier against to compute
|
* `tech-step-eval.test.ts` runs the real classifier against to compute
|
||||||
* precision/recall/F1 (`tech-step-evaluator.ts`), the objective gate any
|
* precision/recall/F1 (`tech-step-evaluator.ts`), the objective gate any
|
||||||
* future change to `tech-step-training-data.ts` must clear (see that
|
* future change to `services/tech-step-intent-service`'s `training_data.py`
|
||||||
* module's own doc comment).
|
* must clear (see that module's own doc comment).
|
||||||
*
|
*
|
||||||
* Deliberately *not* reusing `TECH_STEP_TRAINING_DATA`'s own `utterances`
|
* Deliberately *not* reusing `TECH_STEP_TRAINING_DATA`'s own `utterances`
|
||||||
* verbatim — scoring the classifier against the exact sentences it was
|
* verbatim — scoring the classifier against the exact sentences it was
|
||||||
|
|
@ -246,7 +246,7 @@ export const TECH_STEP_EVAL_DATASET: TechStepEvalCase[] = [
|
||||||
// --- Documented false-positive traps, re-verified with fresh wording ---
|
// --- Documented false-positive traps, re-verified with fresh wording ---
|
||||||
// `brown`'s EN synonyms are verb forms only ("browned"/"browning"), not
|
// `brown`'s EN synonyms are verb forms only ("browned"/"browning"), not
|
||||||
// bare "brown" — precisely so this doesn't false-positive (see that
|
// bare "brown" — precisely so this doesn't false-positive (see that
|
||||||
// entry's own comment in tech-step-training-data.ts).
|
// entry's own comment in training_data.py).
|
||||||
{
|
{
|
||||||
description: "This recipe calls for two tablespoons of brown sugar.",
|
description: "This recipe calls for two tablespoons of brown sugar.",
|
||||||
locale: "en",
|
locale: "en",
|
||||||
|
|
|
||||||
|
|
@ -3,8 +3,9 @@
|
||||||
* hand-labeled evaluation set (`tech-step-eval-dataset.ts`) — the objective
|
* hand-labeled evaluation set (`tech-step-eval-dataset.ts`) — the objective
|
||||||
* counterpart to the "inspected by eye" verdict every corpus change used to
|
* counterpart to the "inspected by eye" verdict every corpus change used to
|
||||||
* get before this module existed. Every future edit to
|
* get before this module existed. Every future edit to
|
||||||
* `tech-step-training-data.ts` (including the LLM-assisted suggestions the
|
* `services/tech-step-intent-service`'s `training_data.py` (including the
|
||||||
* worker in `services/tech-step-llm-worker` proposes) is expected to run
|
* LLM-assisted suggestions the worker in `services/tech-step-llm-worker`
|
||||||
|
* proposes) is expected to run
|
||||||
* through `tech-step-eval.test.ts`'s regression gate, which calls
|
* through `tech-step-eval.test.ts`'s regression gate, which calls
|
||||||
* {@link computeTechStepMetrics} — a corpus change that raises recall on one
|
* {@link computeTechStepMetrics} — a corpus change that raises recall on one
|
||||||
* technique but silently tanks another's precision should fail loudly here,
|
* technique but silently tanks another's precision should fail loudly here,
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,5 @@
|
||||||
import { prisma } from "../../db/prisma.js";
|
import { prisma } from "../../db/prisma.js";
|
||||||
import { intentServiceClient } from "./intent-service-client.js";
|
import { intentServiceClient } from "./intent-service-client.js";
|
||||||
import { TECH_STEP_TRAINING_DATA } from "./tech-step-training-data.js";
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Auto-detects which cooking techniques (`TechStep`) a free-text recipe
|
* Auto-detects which cooking techniques (`TechStep`) a free-text recipe
|
||||||
|
|
@ -15,13 +14,14 @@ import { TECH_STEP_TRAINING_DATA } from "./tech-step-training-data.js";
|
||||||
* generalize past its own vocabulary — a step describing melting butter as
|
* generalize past its own vocabulary — a step describing melting butter as
|
||||||
* "jusqu'à ce que le beurre ait disparu dans la poêle" mentions no verb any
|
* "jusqu'à ce que le beurre ait disparu dans la poêle" mentions no verb any
|
||||||
* regex could anchor on, yet unmistakably *means* `melt`. Replaced with a
|
* regex could anchor on, yet unmistakably *means* `melt`. Replaced with a
|
||||||
* small hybrid pipeline (originally built on `node-nlp`, now delegated to
|
* small hybrid pipeline (originally built on `node-nlp`, now entirely
|
||||||
* `services/tech-step-intent-service` — a spaCy-based microservice, see
|
* delegated to `services/tech-step-intent-service` — a spaCy-based
|
||||||
* {@link IntentServiceClient} and that service's own README):
|
* microservice, see {@link IntentServiceClient} and that service's own
|
||||||
|
* README):
|
||||||
*
|
*
|
||||||
* 1. **NER** (the intent service's `PhraseMatcher`, built from `synonyms` in
|
* 1. **NER** (the intent service's `PhraseMatcher`, built from its own
|
||||||
* `TECH_STEP_TRAINING_DATA`) finds every *candidate* technique mention in
|
* `training_data.py`'s `synonyms`) finds every *candidate* technique
|
||||||
* the whole description, each with its exact character span —
|
* mention in the whole description, each with its exact character span —
|
||||||
* mechanically the same job the old regexes did, just as flat synonym
|
* mechanically the same job the old regexes did, just as flat synonym
|
||||||
* lists instead of hand-written patterns. This step alone is *not* the
|
* lists instead of hand-written patterns. This step alone is *not* the
|
||||||
* final answer — see step 3.
|
* final answer — see step 3.
|
||||||
|
|
@ -31,7 +31,7 @@ import { TECH_STEP_TRAINING_DATA } from "./tech-step-training-data.js";
|
||||||
* and `melt`) needs each judged on its own surrounding context, not the
|
* and `melt`) needs each judged on its own surrounding context, not the
|
||||||
* whole step lumped into one classification.
|
* whole step lumped into one classification.
|
||||||
* 3. **NLP intent classification** (the intent service's `textcat`, trained
|
* 3. **NLP intent classification** (the intent service's `textcat`, trained
|
||||||
* on `TECH_STEP_TRAINING_DATA`'s `utterances`) then classifies each
|
* on its own `training_data.py`'s `utterances`) then classifies each
|
||||||
* clause on its own — this is what actually delivers "meaning, not
|
* clause on its own — this is what actually delivers "meaning, not
|
||||||
* keywords": the classifier was deliberately trained on paraphrases that
|
* keywords": the classifier was deliberately trained on paraphrases that
|
||||||
* never use the technique's own verb (e.g. "jusqu'à ce que le beurre ait
|
* never use the technique's own verb (e.g. "jusqu'à ce que le beurre ait
|
||||||
|
|
@ -50,12 +50,12 @@ import { TECH_STEP_TRAINING_DATA } from "./tech-step-training-data.js";
|
||||||
*
|
*
|
||||||
* `normalizeText` and {@link splitIntoClauses} are pure (no DB/model
|
* `normalizeText` and {@link splitIntoClauses} are pure (no DB/model
|
||||||
* access) so they stay unit-testable in isolation (see
|
* access) so they stay unit-testable in isolation (see
|
||||||
* `test/tech-step-matcher.test.ts`); the classifier itself needs a one-time
|
* `test/tech-step-matcher.test.ts`); this class only ever needs a
|
||||||
* training pass (`_ensureTrained`, a `POST /v1/train` call per locale to the
|
* `TechStep.key -> id` lookup from the DB, memoized on the shared
|
||||||
* intent service) plus a `TechStep.key -> id` lookup from the DB, both
|
* {@link techStepClassifier} singleton rather than repeated per call — the
|
||||||
* memoized on the shared {@link techStepClassifier} singleton rather than
|
* NLP model itself trains once, inside `services/tech-step-intent-service`'s
|
||||||
* repeated per call — training is the expensive part, never worth redoing
|
* own startup, entirely independently of this class (see that service's
|
||||||
* per request let alone per step.
|
* README — this repo no longer pushes any corpus to it over HTTP).
|
||||||
*/
|
*/
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
@ -242,32 +242,29 @@ export function splitIntoClauses(
|
||||||
* `TECH_STEP_TRAINING_DATA` — see `test/tech-step-matcher.test.ts` for the
|
* `TECH_STEP_TRAINING_DATA` — see `test/tech-step-matcher.test.ts` for the
|
||||||
* cases this threshold was picked to pass.
|
* cases this threshold was picked to pass.
|
||||||
*
|
*
|
||||||
* Recalibrated to `0.45` for the migration off `node-nlp` to
|
* Recalibrated for the migration off `node-nlp` to
|
||||||
* `services/tech-step-intent-service` (spaCy `textcat`, exclusive classes
|
* `services/tech-step-intent-service` (spaCy `textcat`, exclusive classes)
|
||||||
* over ~26 techniques) — its score distribution is meaningfully different
|
* — its score distribution is meaningfully different from node-nlp's own
|
||||||
* from node-nlp's own classifier. With that service's training tuned for
|
* classifier, and shifts again every time the corpus' technique count
|
||||||
* real confidence rather than just correct argmax (see
|
* changes (more exclusive classes generally means a *lower* natural
|
||||||
* `_TRAINING_ITERATIONS`/`_TRAINING_DROPOUT` in
|
* confidence ceiling, softmax mass spread thinner).
|
||||||
* `services/tech-step-intent-service/intent_service/locale_pipeline.py`),
|
*
|
||||||
* genuine matches score comfortably above `0.45` with real margin: `melt`
|
* Currently `0.2`, set against the corpus as expanded to ~74 techniques
|
||||||
* scores `~0.95` on "jusqu'à ce que le beurre ait disparu dans la poêle"
|
* (`services/tech-step-intent-service/intent_service/training_data.py`,
|
||||||
* (the exact motivating no-keyword case this pipeline exists for, no NER
|
* `_TRAINING_ITERATIONS = 40`) from manual spot-checks, not yet a real
|
||||||
* anchor to fall back to), `preheat` `~0.90` on "mettre la poêle sur feu
|
* `calibrate-tech-step-threshold.ts` sweep against
|
||||||
* vif", down to `~0.51` for the weakest real case seen (`bake`, anchored).
|
* `TECH_STEP_EVAL_DATASET` (needs Postgres — see that script's own doc
|
||||||
* The noise floor stays far below all of them: English recipe text run
|
* comment): observed real-case scores ranged `0.25`-`0.89` (`simmer`
|
||||||
* through the French classifier (must find *nothing*, confirmed by
|
* lowest, still correct and anchored anyway; `melt` highest, the
|
||||||
* `recipe-translation.test.ts`'s own locale-isolation test) scores `~0.05`
|
* motivating anchor-less case), against a noise floor around `0.02`
|
||||||
* for every technique — indistinguishable from the ~1/26 uniform baseline
|
* (English text through the French classifier). `0.2` sits with margin
|
||||||
* over this many exclusive classes. Cross-checked against
|
* above the noise floor and below every real case seen so far, but **this
|
||||||
* `apps/api/src/scripts/calibrate-tech-step-threshold.ts`'s sweep over
|
* is a placeholder pending the real eval-dataset sweep** — do not treat it
|
||||||
* `TECH_STEP_EVAL_DATASET`: aggregate F1 climbs to its plateau (`0.987`)
|
* as load-bearing precision the way the previous `0.45` (calibrated
|
||||||
* starting at `0.45` and stays flat through `0.95`, so this is the lowest
|
* against the ~26-technique corpus, `TECH_STEP_EVAL_DATASET` F1 plateauing
|
||||||
* threshold that already captures every gain available from trusting the
|
* exactly there) was.
|
||||||
* classifier more — a higher value would only ever risk discarding a
|
|
||||||
* genuine anchor-less match like the two above, never buy back any
|
|
||||||
* precision.
|
|
||||||
*/
|
*/
|
||||||
export const CONFIDENCE_THRESHOLD = 0.45;
|
export const CONFIDENCE_THRESHOLD = 0.2;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* One clause's full classification detail — the finer-grained sibling of
|
* One clause's full classification detail — the finer-grained sibling of
|
||||||
|
|
@ -295,29 +292,34 @@ export interface TechStepClauseClassification {
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Owns the trained state behind {@link matchTechStepSpans} — a real class
|
* Owns the `TechStep.key -> id` lookup behind {@link matchTechStepSpans} —
|
||||||
* (not a plain object of functions) per this repo's service-style-logic
|
* a real class (not a plain object of functions) per this repo's
|
||||||
* convention, even though it's only ever used as the one shared
|
* service-style-logic convention, even though it's only ever used as the
|
||||||
* {@link techStepClassifier} singleton below: it holds real state (the
|
* one shared {@link techStepClassifier} singleton below: it holds real
|
||||||
* memoized training/lookup promises), not just grouped stateless helpers.
|
* state (the memoized lookup promise), not just grouped stateless helpers.
|
||||||
* The actual NER/intent-classification model lives in
|
* The actual NER/intent-classification model lives entirely in
|
||||||
* `services/tech-step-intent-service` (a separate process) — this class's
|
* `services/tech-step-intent-service` (a separate process, trained from
|
||||||
* own state is just what it needs to talk to that service correctly
|
* its own `training_data.py` at its own startup) — this class never
|
||||||
* (whether training has been kicked off yet, and the `TechStep.key -> id`
|
* trains or pushes anything to it, it only calls `POST /v1/process` and
|
||||||
* lookup that service's `uid`s must still be resolved through).
|
* resolves whatever `uid` comes back to a local DB id.
|
||||||
*/
|
*/
|
||||||
export class TechStepClassifierService {
|
export class TechStepClassifierService {
|
||||||
/** Memoized training pass — `undefined` until the first call starts it, after which every caller (concurrent or not) awaits the same promise rather than retraining. */
|
/** Memoized `TechStep.key -> id` lookup — resolved from the DB once, reused by every call rather than queried per request. `undefined` until the first call starts loading it, after which every caller (concurrent or not) awaits the same promise. */
|
||||||
private _trained: Promise<void> | undefined;
|
private _techStepIdsLoaded: Promise<void> | undefined;
|
||||||
/** Memoized `TechStep.key -> id` lookup — training data only knows techniques by their stable `uid`/`key`, resolved to the real DB id once, alongside training. */
|
|
||||||
private _techStepIdByUid: Map<string, number> | undefined;
|
private _techStepIdByUid: Map<string, number> | undefined;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Forces training (a `POST /v1/train` call per locale to
|
* Forces the `TechStep.key -> id` lookup to load now, synchronously with
|
||||||
* `services/tech-step-intent-service`) to happen now, synchronously with
|
|
||||||
* server startup (see `server.ts`, which also retries this against a
|
* server startup (see `server.ts`, which also retries this against a
|
||||||
* not-yet-ready intent service), rather than stalling whichever request
|
* not-yet-reachable intent service), rather than stalling whichever
|
||||||
* happens to be first to save/preview a recipe.
|
* request happens to be first to save/preview a recipe. Doesn't wait on
|
||||||
|
* `services/tech-step-intent-service` finishing its own training — that
|
||||||
|
* service is only ever considered "up" by Docker Compose/CI once it
|
||||||
|
* already is (see that service's `GET /health`), so by the time this
|
||||||
|
* runs in a real deployment it's already trained; a request racing an
|
||||||
|
* intent service that's genuinely still starting just gets an empty
|
||||||
|
* match list back (see `IntentServiceProcessResult`'s own doc comment),
|
||||||
|
* not an error.
|
||||||
*/
|
*/
|
||||||
public async warmUp(): Promise<void> {
|
public async warmUp(): Promise<void> {
|
||||||
try {
|
try {
|
||||||
|
|
@ -341,7 +343,7 @@ export class TechStepClassifierService {
|
||||||
*/
|
*/
|
||||||
public async matchTechStepSpans(description: string, locale: string): Promise<TechStepMatch[]> {
|
public async matchTechStepSpans(description: string, locale: string): Promise<TechStepMatch[]> {
|
||||||
try {
|
try {
|
||||||
await this._ensureTrained();
|
await this._ensureTechStepIdsLoaded();
|
||||||
if (description.trim().length === 0) return [];
|
if (description.trim().length === 0) return [];
|
||||||
|
|
||||||
// The intent service only ever returns enum-style candidates (its own
|
// The intent service only ever returns enum-style candidates (its own
|
||||||
|
|
@ -405,7 +407,7 @@ export class TechStepClassifierService {
|
||||||
locale: string,
|
locale: string,
|
||||||
): Promise<TechStepClauseClassification[]> {
|
): Promise<TechStepClauseClassification[]> {
|
||||||
try {
|
try {
|
||||||
await this._ensureTrained();
|
await this._ensureTechStepIdsLoaded();
|
||||||
if (description.trim().length === 0) return [];
|
if (description.trim().length === 0) return [];
|
||||||
|
|
||||||
const nerResult = await intentServiceClient.process(locale, description);
|
const nerResult = await intentServiceClient.process(locale, description);
|
||||||
|
|
@ -491,46 +493,34 @@ export class TechStepClassifierService {
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Trains `services/tech-step-intent-service` from
|
* Resolves the `uid -> TechStep.id` lookup exactly once — memoized on
|
||||||
* {@link TECH_STEP_TRAINING_DATA} and resolves the `uid -> TechStep.id`
|
* `_techStepIdsLoaded` so a burst of concurrent calls (several steps of
|
||||||
* lookup, both exactly once — memoized on `_trained` so a burst of
|
* the same recipe save, awaited via the same event loop tick) all await
|
||||||
* concurrent calls (several steps of the same recipe save, awaited via
|
* the one in-flight DB query rather than each firing their own.
|
||||||
* the same event loop tick) all await the one in-flight training pass
|
|
||||||
* rather than each kicking off their own.
|
|
||||||
*/
|
*/
|
||||||
private async _ensureTrained(): Promise<void> {
|
private async _ensureTechStepIdsLoaded(): Promise<void> {
|
||||||
if (this._trained === undefined) {
|
if (this._techStepIdsLoaded === undefined) {
|
||||||
this._trained = this._train();
|
this._techStepIdsLoaded = this._loadTechStepIds();
|
||||||
}
|
}
|
||||||
try {
|
try {
|
||||||
await this._trained;
|
await this._techStepIdsLoaded;
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
// A failed training pass must be retried by the *next* call, not
|
// A failed load must be retried by the *next* call, not leave every
|
||||||
// leave every future call permanently rejecting against a stale
|
// future call permanently rejecting against a stale failed promise.
|
||||||
// failed promise.
|
this._techStepIdsLoaded = undefined;
|
||||||
this._trained = undefined;
|
|
||||||
throw err;
|
throw err;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private async _train(): Promise<void> {
|
private async _loadTechStepIds(): Promise<void> {
|
||||||
try {
|
try {
|
||||||
const techSteps = await prisma.techStep.findMany({ select: { id: true, key: true } });
|
const techSteps = await prisma.techStep.findMany({ select: { id: true, key: true } });
|
||||||
this._techStepIdByUid = new Map(techSteps.map((techStep) => [techStep.key, techStep.id]));
|
this._techStepIdByUid = new Map(techSteps.map((techStep) => [techStep.key, techStep.id]));
|
||||||
|
|
||||||
for (const locale of ["fr", "en"] as const) {
|
|
||||||
const entries = TECH_STEP_TRAINING_DATA.map((entry) => ({
|
|
||||||
uid: entry.uid,
|
|
||||||
synonyms: entry[locale].synonyms,
|
|
||||||
utterances: entry[locale].utterances,
|
|
||||||
}));
|
|
||||||
await intentServiceClient.train(locale, entries);
|
|
||||||
}
|
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
throw err; // see matchTechStepSpans()'s catch comment above
|
throw err; // see matchTechStepSpans()'s catch comment above
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Single shared instance — training is expensive enough (a couple of minutes total, both locales combined — see `services/tech-step-intent-service`'s own `_TRAINING_ITERATIONS`) that every caller must reuse the one already-trained model, never spin up their own. */
|
/** Single shared instance — every caller reuses the one memoized `TechStep.key -> id` lookup rather than re-querying the DB. The actual model training (expensive — a couple of minutes, both locales combined) happens entirely inside `services/tech-step-intent-service`'s own startup, not here — see that service's `_TRAINING_ITERATIONS`. */
|
||||||
export const techStepClassifier = new TechStepClassifierService();
|
export const techStepClassifier = new TechStepClassifierService();
|
||||||
|
|
|
||||||
|
|
@ -1,911 +0,0 @@
|
||||||
/**
|
|
||||||
* Training corpus for {@link TechStepClassifierService} (`tech-step-matcher.ts`)
|
|
||||||
* — one entry per `TechStep` (`uid` matches `reference-seed-data.ts`'s
|
|
||||||
* `TECH_STEPS`, which still owns the reference `TechStep` rows themselves;
|
|
||||||
* this file replaces `TECH_STEPS[].mappings`' regex expressions as the
|
|
||||||
* *matching* data source).
|
|
||||||
*
|
|
||||||
* Two distinct kinds of content per technique/locale, feeding two distinct
|
|
||||||
* mechanisms of the classifier (see that file's doc comment for why both
|
|
||||||
* are needed):
|
|
||||||
*
|
|
||||||
* - `synonyms` — short literal words/set phrases, fed to node-nlp's NER
|
|
||||||
* (enum entities). Mechanically equivalent to the old regexes' verb-form
|
|
||||||
* alternations, just spelled out as plain words instead of a pattern
|
|
||||||
* (node-nlp's own stemmer/fuzzy matching already covers minor
|
|
||||||
* conjugation/typo variance that the regexes had to enumerate by hand).
|
|
||||||
* Used only to find *candidate* technique mentions and cut a step into
|
|
||||||
* clauses around them — never the final answer on their own.
|
|
||||||
* - `utterances` — full example clauses, fed to node-nlp's NLP Manager as
|
|
||||||
* training documents for the intent classifier. Deliberately mixes
|
|
||||||
* keyword-anchored phrasings (reinforces the obvious case) with
|
|
||||||
* paraphrases that never use the technique's own verb at all (e.g.
|
|
||||||
* "jusqu'à ce que le beurre ait disparu" for `melt`) — this second kind
|
|
||||||
* is what actually delivers on "comprendre le sens, pas juste les mots
|
|
||||||
* clés" (see the PR this file was introduced in): a clause reaching the
|
|
||||||
* classifier gets labeled by what it's trained to recognize as *meaning*
|
|
||||||
* this technique, not by which literal word triggered its extraction.
|
|
||||||
*
|
|
||||||
* Kept as static in-code data (not DB rows, unlike the old
|
|
||||||
* `TechStepMapping` table) because nothing needs to query/edit it at
|
|
||||||
* runtime — it only ever feeds one thing, the classifier's one-time
|
|
||||||
* training pass (see `TechStepClassifierService._ensureTrained`) — same
|
|
||||||
* reasoning `INGREDIENT_LABELS_EN` (`packages/shared`) is a plain object,
|
|
||||||
* not a database table.
|
|
||||||
*/
|
|
||||||
|
|
||||||
/** One technique's matching data for one locale — see this file's doc comment for what each list feeds. */
|
|
||||||
export interface TechStepLocaleTrainingData {
|
|
||||||
synonyms: string[];
|
|
||||||
utterances: string[];
|
|
||||||
}
|
|
||||||
|
|
||||||
/** One technique's full training entry — `uid` must match a `TECH_STEPS[].uid` in `reference-seed-data.ts`. */
|
|
||||||
export interface TechStepTrainingEntry {
|
|
||||||
uid: string;
|
|
||||||
fr: TechStepLocaleTrainingData;
|
|
||||||
en: TechStepLocaleTrainingData;
|
|
||||||
}
|
|
||||||
|
|
||||||
export const TECH_STEP_TRAINING_DATA: TechStepTrainingEntry[] = [
|
|
||||||
{
|
|
||||||
uid: "cook",
|
|
||||||
fr: {
|
|
||||||
synonyms: [
|
|
||||||
"cuire",
|
|
||||||
"cuisez",
|
|
||||||
"cuisant",
|
|
||||||
"cuisson",
|
|
||||||
"cuit",
|
|
||||||
"cuite",
|
|
||||||
"cuites",
|
|
||||||
"cuits",
|
|
||||||
"cuisiner",
|
|
||||||
"cuisinez",
|
|
||||||
"cuisiné",
|
|
||||||
"cuisinée",
|
|
||||||
"faire cuire",
|
|
||||||
"laisser cuire",
|
|
||||||
],
|
|
||||||
utterances: [
|
|
||||||
"faire cuire à feu moyen",
|
|
||||||
"laisser cuire jusqu'à ce que ce soit prêt",
|
|
||||||
"la cuisson dure environ dix minutes",
|
|
||||||
"jusqu'à ce que la viande ne soit plus rose au centre",
|
|
||||||
"poursuivre la cuisson à couvert",
|
|
||||||
// Two real recipe clauses found misclassified (as `preheat` and
|
|
||||||
// `panFry` respectively, both above the confidence threshold) once
|
|
||||||
// real, longer, comma-heavy sentences started reaching the
|
|
||||||
// classifier — neither error came from a missing keyword (both
|
|
||||||
// clauses' own NER anchor, "laisser cuire"/"faire cuire", was
|
|
||||||
// already right), just the classifier's low-heat/occasional-
|
|
||||||
// stirring phrasing not resembling anything short and clean-cut it
|
|
||||||
// had actually been trained on.
|
|
||||||
"baisser le feu et laisser cuire à découvert encore un quart d'heure",
|
|
||||||
"faire cuire à feu doux en remuant de temps en temps",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
// NOT "cooked through"/"cooking through" — both are word-prefix
|
|
||||||
// extensions of "cooked"/"cooking" above, so any text containing them
|
|
||||||
// matches BOTH the short and long form as separate overlapping NER
|
|
||||||
// candidates, corrupting clause-splitting (confirmed via "It should
|
|
||||||
// be cooking through evenly", which spuriously grew a second,
|
|
||||||
// wrongly-classified `roast` candidate). See this pattern flagged
|
|
||||||
// throughout the file wherever it was found — the fix is always to
|
|
||||||
// drop the longer, redundant form rather than keep both.
|
|
||||||
synonyms: ["cook", "cooks", "cooked", "cooking"],
|
|
||||||
utterances: [
|
|
||||||
"cook over medium heat",
|
|
||||||
"cook until done",
|
|
||||||
"cooking takes about ten minutes",
|
|
||||||
"until no longer pink in the middle",
|
|
||||||
"continue cooking covered",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "fry",
|
|
||||||
fr: {
|
|
||||||
synonyms: [
|
|
||||||
"frire",
|
|
||||||
"frit",
|
|
||||||
"frite",
|
|
||||||
"frites",
|
|
||||||
"friture",
|
|
||||||
"faire frire",
|
|
||||||
"faites frire",
|
|
||||||
"bain de friture",
|
|
||||||
"huile de friture",
|
|
||||||
],
|
|
||||||
utterances: [
|
|
||||||
"faire frire dans l'huile chaude",
|
|
||||||
"plonger dans la friture",
|
|
||||||
"jusqu'à ce que ce soit doré et croustillant à l'extérieur",
|
|
||||||
"l'huile doit être bien chaude avant d'y plonger les morceaux",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
// NOT "frying oil" — a word-prefix extension of "frying" above (see
|
|
||||||
// the `cook` entry's comment for why that duplicates/corrupts NER
|
|
||||||
// candidates; here it was even worse, misclassifying as `preheat`).
|
|
||||||
synonyms: ["fry", "fries", "fried", "frying", "deep fry", "deep-fried", "deep frying"],
|
|
||||||
utterances: [
|
|
||||||
"fry in hot oil",
|
|
||||||
"deep fry until golden",
|
|
||||||
"until crisp and golden on the outside",
|
|
||||||
"the oil should be very hot before adding the pieces",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "melt",
|
|
||||||
fr: {
|
|
||||||
synonyms: [
|
|
||||||
"fondre",
|
|
||||||
"fondu",
|
|
||||||
"fondue",
|
|
||||||
"fondues",
|
|
||||||
"faire fondre",
|
|
||||||
"faites fondre",
|
|
||||||
// Also a plausible way to say "melt" (heating something — usually
|
|
||||||
// a fat — until it liquefies), not just a `preheat` phrasing —
|
|
||||||
// restores what the regex-based system anchored on before this
|
|
||||||
// pipeline replaced it.
|
|
||||||
"faire chauffer",
|
|
||||||
"faites chauffer",
|
|
||||||
"liquéfier",
|
|
||||||
"liquéfiez",
|
|
||||||
"liquéfié",
|
|
||||||
"faire liquéfier",
|
|
||||||
],
|
|
||||||
utterances: [
|
|
||||||
"faire fondre le beurre",
|
|
||||||
"jusqu'à ce que le beurre ait disparu dans la poêle",
|
|
||||||
"le beurre doit être complètement liquide",
|
|
||||||
"laisser le fromage devenir tout liquide sur feu doux",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
synonyms: ["melt", "melts", "melted", "melting", "liquefy", "liquefied"],
|
|
||||||
utterances: [
|
|
||||||
"melt the butter",
|
|
||||||
"until the butter has completely disappeared into the pan",
|
|
||||||
"the butter should be fully liquid",
|
|
||||||
"let the cheese turn completely liquid over low heat",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "deglaze",
|
|
||||||
fr: {
|
|
||||||
// NOT "déglacer la poêle"/"déglacer le fond de cuisson" — both are
|
|
||||||
// word-prefix extensions of "déglacer" above (see `cook`'s comment
|
|
||||||
// for why that duplicates NER candidates).
|
|
||||||
synonyms: ["déglacer", "déglacez", "déglacé", "déglacée", "déglaçage"],
|
|
||||||
utterances: [
|
|
||||||
"déglacer avec le vin blanc",
|
|
||||||
"verser le vin dans la poêle chaude pour décoller les sucs",
|
|
||||||
"gratter les sucs de cuisson au fond de la casserole avec un peu de bouillon",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
// NOT "deglaze the pan" — a word-prefix extension of "deglaze" above
|
|
||||||
// (see `cook`'s comment for why that duplicates NER candidates).
|
|
||||||
synonyms: ["deglaze", "deglazes", "deglazed", "deglazing", "lift the browned bits"],
|
|
||||||
utterances: [
|
|
||||||
"deglaze with white wine",
|
|
||||||
"pour the wine into the hot pan to lift the browned bits",
|
|
||||||
"scrape up the browned bits at the bottom of the pan with a splash of stock",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "simmer",
|
|
||||||
fr: {
|
|
||||||
synonyms: [
|
|
||||||
"mijoter",
|
|
||||||
"mijotez",
|
|
||||||
"mijote",
|
|
||||||
"mijotant",
|
|
||||||
"mijoté",
|
|
||||||
"frémir",
|
|
||||||
"frémissant",
|
|
||||||
"frémissante",
|
|
||||||
"à petit feu",
|
|
||||||
],
|
|
||||||
utterances: [
|
|
||||||
"laisser mijoter à feu doux",
|
|
||||||
"faire mijoter pendant une heure",
|
|
||||||
"de petites bulles doivent remonter doucement à la surface",
|
|
||||||
"laisser cuire tout doucement à couvert pendant longtemps",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
// NOT "simmering gently" — a word-prefix extension of "simmering"
|
|
||||||
// above (see `cook`'s comment for why that duplicates NER candidates).
|
|
||||||
synonyms: ["simmer", "simmers", "simmered", "simmering", "gentle simmer", "low simmer"],
|
|
||||||
utterances: [
|
|
||||||
"let it simmer over low heat",
|
|
||||||
"simmer for one hour",
|
|
||||||
"small bubbles should gently rise to the surface",
|
|
||||||
"let it cook very gently, covered, for a long time",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "boil",
|
|
||||||
fr: {
|
|
||||||
synonyms: [
|
|
||||||
"bouillir",
|
|
||||||
"bouillant",
|
|
||||||
"bouillie",
|
|
||||||
"bouillies",
|
|
||||||
"ébullition",
|
|
||||||
"porter à ébullition",
|
|
||||||
"gros bouillons",
|
|
||||||
],
|
|
||||||
utterances: [
|
|
||||||
"porter à ébullition",
|
|
||||||
"faire bouillir l'eau",
|
|
||||||
"de grosses bulles doivent agiter la surface avec force",
|
|
||||||
"jusqu'à ce que ça bouillonne franchement",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
// NOT "boiling point" — a word-prefix extension of "boiling" above
|
|
||||||
// (see `cook`'s comment for why that duplicates NER candidates).
|
|
||||||
synonyms: ["boil", "boils", "boiled", "boiling", "rolling boil"],
|
|
||||||
utterances: [
|
|
||||||
"bring to a boil",
|
|
||||||
"boil the water",
|
|
||||||
"large bubbles should be vigorously breaking the surface",
|
|
||||||
"until it's rolling vigorously",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "roast",
|
|
||||||
fr: {
|
|
||||||
// NOT "rôti au four" — a word-prefix extension of "rôti" above (see
|
|
||||||
// `cook`'s comment for why that duplicates NER candidates).
|
|
||||||
synonyms: ["rôtir", "rôti", "rôtie", "rôties", "rôtis", "rôtissage"],
|
|
||||||
utterances: [
|
|
||||||
"faire rôtir la volaille entière",
|
|
||||||
"le rôti doit dorer uniformément de tous les côtés",
|
|
||||||
"cuire la pièce de viande entière au four à chaleur sèche",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
synonyms: ["roast", "roasts", "roasted", "roasting", "oven-roast", "oven roasted"],
|
|
||||||
utterances: [
|
|
||||||
"roast the whole bird",
|
|
||||||
"it should brown evenly on every side",
|
|
||||||
"cook the whole piece of meat in dry oven heat",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "grill",
|
|
||||||
fr: {
|
|
||||||
synonyms: [
|
|
||||||
"griller",
|
|
||||||
"grillez",
|
|
||||||
"grillé",
|
|
||||||
"grillée",
|
|
||||||
"grillées",
|
|
||||||
"grillade",
|
|
||||||
"grillades",
|
|
||||||
"barbecue",
|
|
||||||
"au barbecue",
|
|
||||||
],
|
|
||||||
utterances: [
|
|
||||||
"faire griller sur la grille du barbecue",
|
|
||||||
"marquer les steaks sur une plaque brûlante",
|
|
||||||
"des traces de quadrillage doivent apparaître à la cuisson",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
synonyms: ["grill", "grills", "grilled", "grilling", "barbecue", "char-grill", "charbroiled"],
|
|
||||||
utterances: [
|
|
||||||
"grill on the barbecue rack",
|
|
||||||
"sear the steaks on a scorching-hot plate",
|
|
||||||
"char marks should appear as it cooks",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "panFry",
|
|
||||||
fr: {
|
|
||||||
// Deliberately NOT "poêlé"/"poêlée"/"poêlés" here, despite reading
|
|
||||||
// like natural panFry vocabulary: node-nlp's French stemmer reduces
|
|
||||||
// them to the same root as the bare noun "poêle" (a pan), so
|
|
||||||
// registering them made every plain mention of "poêle" — e.g.
|
|
||||||
// `preheat`'s own "la poêle" — a false-positive panFry candidate too.
|
|
||||||
// Found via the "jusqu'à ce que le beurre ait disparu dans la poêle"
|
|
||||||
// regression test, which unexpectedly grew a spurious panFry match.
|
|
||||||
synonyms: ["sauter", "sautez", "sauté", "sautée", "sautées", "sautant", "à la poêle"],
|
|
||||||
utterances: [
|
|
||||||
"faire sauter les légumes à la poêle",
|
|
||||||
"saisir rapidement à feu vif en remuant sans cesse",
|
|
||||||
"faire revenir en remuant vivement dans une poêle très chaude",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
synonyms: [
|
|
||||||
"sauté",
|
|
||||||
"sauteed",
|
|
||||||
"sautéed",
|
|
||||||
"sauteing",
|
|
||||||
"pan-fry",
|
|
||||||
"pan fried",
|
|
||||||
"pan-fried",
|
|
||||||
"stir-fry",
|
|
||||||
"pan searing",
|
|
||||||
"seared in a pan",
|
|
||||||
],
|
|
||||||
utterances: [
|
|
||||||
"sauté the vegetables in a pan",
|
|
||||||
"quickly sear over high heat, stirring constantly",
|
|
||||||
"cook briskly, stirring, in a very hot pan",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "blanch",
|
|
||||||
fr: {
|
|
||||||
synonyms: ["blanchir", "blanchissez", "blanchi", "blanchie", "blanchies", "blanchiment"],
|
|
||||||
utterances: [
|
|
||||||
"faire blanchir les légumes deux minutes dans l'eau bouillante",
|
|
||||||
"plonger brièvement dans l'eau bouillante puis directement dans l'eau glacée",
|
|
||||||
"cuire très rapidement à l'eau bouillante avant de stopper la cuisson au froid",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
// "parboil" is folded in here rather than kept a separate technique —
|
|
||||||
// in home-cooking usage (as opposed to professional usage, where they
|
|
||||||
// can differ) it names the same "briefly pre-cook in boiling water"
|
|
||||||
// move blanching does.
|
|
||||||
synonyms: [
|
|
||||||
"blanch",
|
|
||||||
"blanches",
|
|
||||||
"blanched",
|
|
||||||
"blanching",
|
|
||||||
"parboil",
|
|
||||||
"parboiled",
|
|
||||||
"parboiling",
|
|
||||||
],
|
|
||||||
utterances: [
|
|
||||||
"blanch the vegetables for two minutes in boiling water",
|
|
||||||
"briefly plunge into boiling water then straight into ice water",
|
|
||||||
"cook very quickly in boiling water before stopping it cold",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "marinate",
|
|
||||||
fr: {
|
|
||||||
synonyms: [
|
|
||||||
"mariner",
|
|
||||||
"marinez",
|
|
||||||
"mariné",
|
|
||||||
"marinée",
|
|
||||||
"marinées",
|
|
||||||
"marinade",
|
|
||||||
"macérer",
|
|
||||||
"macérez",
|
|
||||||
"macération",
|
|
||||||
"faire mariner",
|
|
||||||
],
|
|
||||||
utterances: [
|
|
||||||
"laisser mariner la viande toute la nuit au réfrigérateur",
|
|
||||||
"faire tremper dans la sauce plusieurs heures avant cuisson pour parfumer",
|
|
||||||
"laisser reposer dans le mélange d'huile et d'épices avant de cuisiner",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
// NOT "marinating for" — a word-prefix extension of "marinating"
|
|
||||||
// above (see `cook`'s comment for why that duplicates NER candidates
|
|
||||||
// — here it was even worse, misclassifying as `simmer`).
|
|
||||||
synonyms: [
|
|
||||||
"marinate",
|
|
||||||
"marinates",
|
|
||||||
"marinated",
|
|
||||||
"marinating",
|
|
||||||
"marinade",
|
|
||||||
"soak in the marinade",
|
|
||||||
],
|
|
||||||
utterances: [
|
|
||||||
"let the meat marinate overnight in the fridge",
|
|
||||||
"soak in the sauce for several hours before cooking to flavor it",
|
|
||||||
"let it sit in the oil and spice mixture before cooking",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "chop",
|
|
||||||
fr: {
|
|
||||||
// NOT "hacher grossièrement" — a word-prefix extension of "hacher"
|
|
||||||
// above (see `cook`'s comment for why that duplicates NER candidates).
|
|
||||||
synonyms: [
|
|
||||||
"hacher",
|
|
||||||
"hachez",
|
|
||||||
"haché",
|
|
||||||
"hachée",
|
|
||||||
"hachées",
|
|
||||||
"hachis",
|
|
||||||
"couper en morceaux",
|
|
||||||
"tailler en morceaux",
|
|
||||||
],
|
|
||||||
utterances: [
|
|
||||||
"hacher finement les oignons",
|
|
||||||
"couper en tout petits morceaux irréguliers au couteau",
|
|
||||||
"réduire les herbes en petits fragments avant de les ajouter",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
// NOT "chop coarsely" — a word-prefix extension of "chop" above (see
|
|
||||||
// `cook`'s comment for why that duplicates NER candidates).
|
|
||||||
synonyms: ["chop", "chops", "chopped", "chopping", "roughly chop", "coarsely chopped"],
|
|
||||||
utterances: [
|
|
||||||
"finely chop the onions",
|
|
||||||
"cut into small, uneven pieces with a knife",
|
|
||||||
"break the herbs down into small bits before adding them",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "peel",
|
|
||||||
fr: {
|
|
||||||
synonyms: [
|
|
||||||
"éplucher",
|
|
||||||
"épluchez",
|
|
||||||
"épluché",
|
|
||||||
"épluchée",
|
|
||||||
"épluchées",
|
|
||||||
"épluchage",
|
|
||||||
"peler",
|
|
||||||
"pelez",
|
|
||||||
"pelé",
|
|
||||||
"pelée",
|
|
||||||
"pelées",
|
|
||||||
],
|
|
||||||
utterances: [
|
|
||||||
"éplucher les pommes de terre",
|
|
||||||
"retirer la peau des carottes avec un économe",
|
|
||||||
"ôter la pelure du fruit avant de le couper",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
synonyms: ["peel", "peels", "peeled", "peeling", "pare", "pared", "paring"],
|
|
||||||
utterances: [
|
|
||||||
"peel the potatoes",
|
|
||||||
"remove the skin from the carrots with a peeler",
|
|
||||||
"take the skin off the fruit before cutting it",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "mince",
|
|
||||||
fr: {
|
|
||||||
synonyms: [
|
|
||||||
"émincer",
|
|
||||||
"émincez",
|
|
||||||
"émincé",
|
|
||||||
"émincée",
|
|
||||||
"émincées",
|
|
||||||
"ciseler",
|
|
||||||
"ciselez",
|
|
||||||
"ciselé",
|
|
||||||
"ciselée",
|
|
||||||
"ciselées",
|
|
||||||
],
|
|
||||||
utterances: [
|
|
||||||
"émincer l'oignon en fines lamelles",
|
|
||||||
"couper en très fines tranches régulières",
|
|
||||||
"détailler en lamelles aussi fines que possible",
|
|
||||||
// Without this, a short clause naming a different vegetable —
|
|
||||||
// "Émincer les tomates" — scored just above `melt`'s confidence
|
|
||||||
// threshold instead (a training-set-composition side effect of
|
|
||||||
// adding utterances elsewhere in this same pass, found by the full
|
|
||||||
// regression suite). A second example anchored on a different noun
|
|
||||||
// widens `mince`'s own region enough to reclaim it.
|
|
||||||
"émincer les tomates en fines rondelles",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
// NOT "mince finely" — a word-prefix extension of "mince" above (see
|
|
||||||
// `cook`'s comment for why that duplicates NER candidates).
|
|
||||||
synonyms: ["mince", "minces", "minced", "mincing", "thinly slice", "finely mince"],
|
|
||||||
utterances: [
|
|
||||||
"mince the onion into thin strips",
|
|
||||||
"cut into very thin, even slices",
|
|
||||||
"slice into strips as thin as possible",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "mix",
|
|
||||||
fr: {
|
|
||||||
synonyms: [
|
|
||||||
"mélanger",
|
|
||||||
"mélangez",
|
|
||||||
"mélangé",
|
|
||||||
"mélangée",
|
|
||||||
"mélangées",
|
|
||||||
"mélange",
|
|
||||||
"brasser",
|
|
||||||
"brassez",
|
|
||||||
"amalgamer",
|
|
||||||
"amalgamez",
|
|
||||||
],
|
|
||||||
utterances: [
|
|
||||||
"mélanger tous les ingrédients dans un saladier",
|
|
||||||
"combiner le sucre et la farine ensemble",
|
|
||||||
"remuer jusqu'à obtenir une préparation homogène",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
synonyms: [
|
|
||||||
"mix",
|
|
||||||
"mixes",
|
|
||||||
"mixed",
|
|
||||||
"mixing",
|
|
||||||
"combine",
|
|
||||||
"combined",
|
|
||||||
"blend",
|
|
||||||
"blended",
|
|
||||||
"blending",
|
|
||||||
"stir together",
|
|
||||||
],
|
|
||||||
utterances: [
|
|
||||||
"mix all the ingredients in a bowl",
|
|
||||||
"combine the sugar and flour together",
|
|
||||||
"stir until the mixture is smooth and even",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "whisk",
|
|
||||||
fr: {
|
|
||||||
synonyms: [
|
|
||||||
"fouetter",
|
|
||||||
"fouettez",
|
|
||||||
"fouetté",
|
|
||||||
"fouettée",
|
|
||||||
"fouettées",
|
|
||||||
"au fouet",
|
|
||||||
"battre au fouet",
|
|
||||||
"monter au fouet",
|
|
||||||
],
|
|
||||||
utterances: [
|
|
||||||
"fouetter les œufs et le sucre",
|
|
||||||
"battre vigoureusement au fouet jusqu'à ce que ça blanchisse",
|
|
||||||
"travailler énergiquement pour incorporer de l'air au mélange",
|
|
||||||
// Without these, "Fouetter les blancs en neige" misclassified as
|
|
||||||
// `foldIn` — its own training utterance below also happens to say
|
|
||||||
// "les blancs en neige", and node-nlp's intent classifier leaned on
|
|
||||||
// that shared noun phrase over the actual verb. The exact phrase
|
|
||||||
// itself is needed (not just a paraphrase of it) — a longer,
|
|
||||||
// differently-worded utterance alone wasn't enough to outweigh
|
|
||||||
// `foldIn`'s own close phrasing.
|
|
||||||
"fouetter les blancs en neige",
|
|
||||||
"fouetter les blancs en neige jusqu'à ce qu'ils soient fermes",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
synonyms: ["whisk", "whisks", "whisked", "whisking", "beat", "whip", "whipped", "whipping"],
|
|
||||||
utterances: [
|
|
||||||
"whisk the eggs and sugar",
|
|
||||||
"beat vigorously with a whisk until pale",
|
|
||||||
"work it briskly to whip air into the mixture",
|
|
||||||
"whisk the egg whites until stiff peaks form",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "foldIn",
|
|
||||||
fr: {
|
|
||||||
synonyms: [
|
|
||||||
"incorporer",
|
|
||||||
"incorporez",
|
|
||||||
"incorporé",
|
|
||||||
"incorporée",
|
|
||||||
"incorporées",
|
|
||||||
// NOT "incorporer délicatement" — it's a superstring of "incorporer"
|
|
||||||
// above, so both would match the same text and hand
|
|
||||||
// `splitIntoClauses` two overlapping candidates for one mention
|
|
||||||
// (found via "Incorporer délicatement la farine" producing two
|
|
||||||
// duplicate matches instead of one).
|
|
||||||
"mélanger délicatement",
|
|
||||||
],
|
|
||||||
utterances: [
|
|
||||||
"incorporer délicatement les blancs en neige",
|
|
||||||
"ajouter en soulevant doucement la masse pour ne pas casser les bulles",
|
|
||||||
"mélanger tout doucement de bas en haut pour garder l'air emprisonné",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
synonyms: ["fold in", "folds in", "folded in", "folding in", "gently fold", "fold gently"],
|
|
||||||
utterances: [
|
|
||||||
"gently fold in the beaten egg whites",
|
|
||||||
"add by gently lifting the batter so you don't knock the air out",
|
|
||||||
"very gently stir from the bottom up to keep the air trapped in",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "setAside",
|
|
||||||
fr: {
|
|
||||||
synonyms: [
|
|
||||||
"réserver",
|
|
||||||
"réservez",
|
|
||||||
"réservé",
|
|
||||||
"réservée",
|
|
||||||
"réservées",
|
|
||||||
"mettre de côté",
|
|
||||||
"laisser de côté",
|
|
||||||
],
|
|
||||||
utterances: [
|
|
||||||
"réserver au frais en attendant",
|
|
||||||
"mettre de côté pour plus tard",
|
|
||||||
"laisser attendre sur le plan de travail pendant la préparation du reste",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
synonyms: ["set aside", "sets aside", "setting aside", "set it aside", "reserve", "reserved"],
|
|
||||||
utterances: [
|
|
||||||
"set aside in the fridge for now",
|
|
||||||
"put it aside for later",
|
|
||||||
"let it wait on the counter while you prepare the rest",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "season",
|
|
||||||
fr: {
|
|
||||||
synonyms: [
|
|
||||||
"assaisonner",
|
|
||||||
"assaisonnez",
|
|
||||||
"assaisonné",
|
|
||||||
"assaisonnée",
|
|
||||||
"assaisonnement",
|
|
||||||
"relever",
|
|
||||||
"relevez",
|
|
||||||
"épicer",
|
|
||||||
"épicez",
|
|
||||||
],
|
|
||||||
utterances: [
|
|
||||||
"assaisonner avec du sel et du poivre",
|
|
||||||
"rectifier le goût en ajoutant des épices",
|
|
||||||
"ajouter du sel selon votre goût avant de servir",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
synonyms: ["season", "seasons", "seasoned", "seasoning", "spice it up", "add seasoning"],
|
|
||||||
utterances: [
|
|
||||||
"season with salt and pepper",
|
|
||||||
"adjust the taste by adding spices",
|
|
||||||
"add salt to taste before serving",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "drain",
|
|
||||||
fr: {
|
|
||||||
synonyms: [
|
|
||||||
"égoutter",
|
|
||||||
"égouttez",
|
|
||||||
"égoutté",
|
|
||||||
"égouttée",
|
|
||||||
"égouttées",
|
|
||||||
"essorer",
|
|
||||||
"essorez",
|
|
||||||
"essoré",
|
|
||||||
"essorée",
|
|
||||||
],
|
|
||||||
utterances: [
|
|
||||||
"égoutter les pâtes dans une passoire",
|
|
||||||
"verser dans une passoire pour retirer l'eau de cuisson",
|
|
||||||
"laisser l'excédent d'eau s'écouler avant de servir",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
synonyms: ["drain", "drains", "drained", "draining", "strain", "strained", "straining"],
|
|
||||||
utterances: [
|
|
||||||
"drain the pasta in a colander",
|
|
||||||
"pour into a colander to remove the cooking water",
|
|
||||||
"let the excess water run off before serving",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "brown",
|
|
||||||
fr: {
|
|
||||||
synonyms: [
|
|
||||||
"faire revenir",
|
|
||||||
"faites revenir",
|
|
||||||
"faire dorer",
|
|
||||||
"faites dorer",
|
|
||||||
"colorer",
|
|
||||||
"colorez",
|
|
||||||
"faire colorer",
|
|
||||||
],
|
|
||||||
utterances: [
|
|
||||||
"faire revenir les oignons dans l'huile chaude",
|
|
||||||
"faire dorer la viande sur toutes les faces",
|
|
||||||
"saisir jusqu'à ce que la surface prenne une belle couleur caramel",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
// Verb forms only (not bare "brown"), same reasoning the old regex
|
|
||||||
// doc comment gave — a bare "brown" false-positives on ingredient
|
|
||||||
// descriptions like "brown sugar"/"brown rice", which never get to
|
|
||||||
// the classifier since they're not step text, but keeping the
|
|
||||||
// synonym itself anchored costs nothing and stays consistent.
|
|
||||||
synonyms: ["browned", "browning"],
|
|
||||||
utterances: [
|
|
||||||
"brown the onions in hot oil",
|
|
||||||
"brown the meat on every side",
|
|
||||||
"sear until the surface turns a deep caramel color",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "rest",
|
|
||||||
fr: {
|
|
||||||
synonyms: ["reposer", "laisser reposer", "laissez reposer", "temps de repos"],
|
|
||||||
utterances: [
|
|
||||||
"laisser reposer la pâte trente minutes",
|
|
||||||
"laisser la viande se détendre hors du four avant de la découper",
|
|
||||||
"attendre quelques minutes avant de servir pour que les jus se répartissent",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
// Anchored to "let ... rest"/"rest for" rather than bare "rest",
|
|
||||||
// same false-positive reasoning as `brown` above ("the rest of the").
|
|
||||||
synonyms: ["let it rest", "let them rest", "resting for", "rested for", "resting time"],
|
|
||||||
utterances: [
|
|
||||||
"let the dough rest for thirty minutes",
|
|
||||||
"let the meat relax outside the oven before carving it",
|
|
||||||
"wait a few minutes before serving so the juices redistribute",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "preheat",
|
|
||||||
fr: {
|
|
||||||
synonyms: [
|
|
||||||
"préchauffer",
|
|
||||||
"préchauffez",
|
|
||||||
"préchauffé",
|
|
||||||
"préchauffée",
|
|
||||||
// A pan already described as hot ("poêle chaude") implies it's
|
|
||||||
// been preheated, without the verb itself — the classic "Dans une
|
|
||||||
// poêle chaude, faire chauffer une noix de beurre" case (both
|
|
||||||
// `preheat` and `melt` in one instruction).
|
|
||||||
"poêle chaude",
|
|
||||||
"préchauffage",
|
|
||||||
],
|
|
||||||
utterances: [
|
|
||||||
"préchauffer le four à 180 degrés",
|
|
||||||
"mettre le four à chauffer avant d'y placer le plat",
|
|
||||||
"allumer le four à l'avance pour qu'il soit à température",
|
|
||||||
// A pan gets preheated too, not just an oven — without an example
|
|
||||||
// like this, "poêle" (which also appears throughout `panFry`'s own
|
|
||||||
// training utterances) biased the classifier toward `panFry` for
|
|
||||||
// any preheating clause that happens to mention a pan, found while
|
|
||||||
// testing against the classic "Préchauffer la poêle, puis faire
|
|
||||||
// fondre le beurre" case.
|
|
||||||
"préchauffer la poêle avant d'y verser l'huile",
|
|
||||||
"faire chauffer la poêle à vide quelques minutes",
|
|
||||||
// "poêle" + "feu vif" together still read as `panFry` (the act of
|
|
||||||
// actually cooking something in it) rather than `preheat` (getting
|
|
||||||
// it hot beforehand, nothing in it yet) without an example this
|
|
||||||
// close to that exact wording — found via "mettre la poêle sur feu
|
|
||||||
// vif" (no food mentioned at all) still classifying as panFry.
|
|
||||||
"mettre la poêle vide sur feu vif avant d'ajouter quoi que ce soit",
|
|
||||||
"mettre la poêle sur feu vif",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
// NOT "preheating time" — a word-prefix extension of "preheating"
|
|
||||||
// above (see `cook`'s comment for why that duplicates NER candidates).
|
|
||||||
synonyms: ["preheat", "preheats", "preheated", "preheating", "hot pan"],
|
|
||||||
utterances: [
|
|
||||||
"preheat the oven to 180 degrees",
|
|
||||||
"turn the oven on to heat up before putting the dish in",
|
|
||||||
"switch the oven on ahead of time so it's up to temperature",
|
|
||||||
"preheat the pan before adding the oil",
|
|
||||||
"heat the empty pan for a few minutes first",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "bake",
|
|
||||||
fr: {
|
|
||||||
synonyms: [
|
|
||||||
"cuire au four",
|
|
||||||
"cuisson au four",
|
|
||||||
"enfourner",
|
|
||||||
"enfournez",
|
|
||||||
"au four",
|
|
||||||
"enfourné",
|
|
||||||
"enfournée",
|
|
||||||
],
|
|
||||||
utterances: [
|
|
||||||
"enfourner pendant quarante-cinq minutes",
|
|
||||||
"mettre au four jusqu'à ce que ce soit doré",
|
|
||||||
"cuire dans le four préchauffé jusqu'à ce que la surface soit ferme",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
// NOT "baked in the oven" — a word-prefix extension of "baked" above
|
|
||||||
// (see `cook`'s comment for why that duplicates NER candidates).
|
|
||||||
synonyms: ["bake", "bakes", "baked", "baking", "in the oven", "oven-baked"],
|
|
||||||
utterances: [
|
|
||||||
"bake for forty-five minutes",
|
|
||||||
"put it in the oven until golden",
|
|
||||||
"cook in the preheated oven until the surface is firm",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "plate",
|
|
||||||
fr: {
|
|
||||||
// NOT "dressage de l'assiette" — a word-prefix extension of
|
|
||||||
// "dressage" above (see `cook`'s comment for why that duplicates NER
|
|
||||||
// candidates).
|
|
||||||
synonyms: ["dresser", "dressez", "dressage", "disposer dans l'assiette"],
|
|
||||||
utterances: [
|
|
||||||
"dresser harmonieusement dans les assiettes",
|
|
||||||
"disposer joliment sur l'assiette avant de servir",
|
|
||||||
"présenter avec soin au centre de l'assiette",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
// NOT "plate up"/"plated nicely" — both are word-prefix extensions of
|
|
||||||
// "plate"/"plated" above (see `cook`'s comment for why that
|
|
||||||
// duplicates NER candidates).
|
|
||||||
synonyms: ["plate", "plates", "plated", "plating"],
|
|
||||||
utterances: [
|
|
||||||
"plate it up nicely",
|
|
||||||
"arrange it neatly on the plate before serving",
|
|
||||||
"present it carefully in the center of the plate",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
uid: "coat",
|
|
||||||
fr: {
|
|
||||||
synonyms: [
|
|
||||||
"napper",
|
|
||||||
"nappez",
|
|
||||||
"nappé",
|
|
||||||
"nappée",
|
|
||||||
"nappées",
|
|
||||||
"nappage",
|
|
||||||
"enrober",
|
|
||||||
"enrobez",
|
|
||||||
"enrobé",
|
|
||||||
"enrobée",
|
|
||||||
"enrobées",
|
|
||||||
],
|
|
||||||
utterances: [
|
|
||||||
"napper le gâteau de chocolat fondu",
|
|
||||||
"recouvrir uniformément d'une fine couche de sauce",
|
|
||||||
"verser la sauce par-dessus pour bien enrober",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
en: {
|
|
||||||
// NOT "coat evenly" — a word-prefix extension of "coat" above (see
|
|
||||||
// `cook`'s comment for why that duplicates NER candidates).
|
|
||||||
synonyms: ["coat", "coats", "coated", "coating", "dredge", "dredged", "dredging"],
|
|
||||||
utterances: [
|
|
||||||
"coat the cake with melted chocolate",
|
|
||||||
"cover evenly with a thin layer of sauce",
|
|
||||||
"pour the sauce over it so it's well covered",
|
|
||||||
],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
];
|
|
||||||
|
|
@ -6,8 +6,8 @@ import { renumberStepTechSteps } from "../modules/recipe/recipe-tech-step-correc
|
||||||
/**
|
/**
|
||||||
* Recomputes every existing `Step`'s `"auto"`-sourced `StepTechStep`
|
* Recomputes every existing `Step`'s `"auto"`-sourced `StepTechStep`
|
||||||
* entries against the *current* classifier
|
* entries against the *current* classifier
|
||||||
* (`tech-step-matcher.ts`/`tech-step-training-data.ts`), the same way
|
* (`tech-step-matcher.ts`, delegating to `services/tech-step-intent-service`),
|
||||||
* `updateRecipe` does when a user resaves a recipe through the UI —
|
* the same way `updateRecipe` does when a user resaves a recipe through the UI —
|
||||||
* always `"fr"` (`DEFAULT_TECH_STEP_LOCALE` in `recipe.service.ts`; there's
|
* always `"fr"` (`DEFAULT_TECH_STEP_LOCALE` in `recipe.service.ts`; there's
|
||||||
* no persisted per-recipe locale to recover for a step that already
|
* no persisted per-recipe locale to recover for a step that already
|
||||||
* exists, so this matches real resave behavior exactly rather than
|
* exists, so this matches real resave behavior exactly rather than
|
||||||
|
|
|
||||||
|
|
@ -6,14 +6,15 @@ import { prisma } from "../db/prisma.js";
|
||||||
* comment) — generated by `services/tech-step-llm-worker`'s scheduled
|
* comment) — generated by `services/tech-step-llm-worker`'s scheduled
|
||||||
* jobs, from either a user correction or the worker's own low-confidence
|
* jobs, from either a user correction or the worker's own low-confidence
|
||||||
* audit (`sourceType`). What a maintainer reads *before* hand-editing
|
* audit (`sourceType`). What a maintainer reads *before* hand-editing
|
||||||
* `tech-step-training-data.ts` and running `retrain-tech-steps.ts` — this
|
* `services/tech-step-intent-service/intent_service/training_data.py` and
|
||||||
* script never writes anything, purely a read-only report to stdout:
|
* running `retrain-tech-steps.ts` — this script never writes anything,
|
||||||
|
* purely a read-only report to stdout:
|
||||||
*
|
*
|
||||||
* pnpm --filter api exec tsx src/scripts/list-pending-training-suggestions.ts
|
* pnpm --filter api exec tsx src/scripts/list-pending-training-suggestions.ts
|
||||||
*
|
*
|
||||||
* Grouped by technique key so every suggestion for the same entry in
|
* Grouped by technique key so every suggestion for the same entry in
|
||||||
* `TECH_STEP_TRAINING_DATA` is read together, matching how that file
|
* `training_data.py`'s `TECH_STEP_TRAINING_DATA` is read together, matching
|
||||||
* itself is organized (one block per technique).
|
* how that file itself is organized (one block per technique).
|
||||||
*/
|
*/
|
||||||
async function listPendingTrainingSuggestions(): Promise<void> {
|
async function listPendingTrainingSuggestions(): Promise<void> {
|
||||||
const suggestions = await prisma.techStepTrainingSuggestion.findMany({
|
const suggestions = await prisma.techStepTrainingSuggestion.findMany({
|
||||||
|
|
|
||||||
|
|
@ -28,10 +28,15 @@ function parseSuggestionIds(flag: "applied" | "rejected"): number[] {
|
||||||
* Maintainer workflow closing the loop on a training-corpus change (see
|
* Maintainer workflow closing the loop on a training-corpus change (see
|
||||||
* this feature's plan document):
|
* this feature's plan document):
|
||||||
*
|
*
|
||||||
* 1. A maintainer has already hand-edited `tech-step-training-data.ts`
|
* 1. A maintainer has already hand-edited
|
||||||
* (informed by `list-pending-training-suggestions.ts`'s report), and
|
* `services/tech-step-intent-service/intent_service/training_data.py`
|
||||||
|
* (informed by `list-pending-training-suggestions.ts`'s report),
|
||||||
* decided which `TechStepTrainingSuggestion` ids they incorporated
|
* decided which `TechStepTrainingSuggestion` ids they incorporated
|
||||||
* (`--applied=`) or explicitly discarded (`--rejected=`).
|
* (`--applied=`) or explicitly discarded (`--rejected=`), **and
|
||||||
|
* restarted `tech-step-intent-service`** so it retrains from the
|
||||||
|
* edited corpus — that service only ever trains once, at its own
|
||||||
|
* startup (see its README), so this script's eval gate below is
|
||||||
|
* meaningless against a service still running the old corpus.
|
||||||
* 2. This script re-runs the F1 regression gate
|
* 2. This script re-runs the F1 regression gate
|
||||||
* ({@link runTechStepEvalSuite} against {@link MIN_OVERALL_F1}) —
|
* ({@link runTechStepEvalSuite} against {@link MIN_OVERALL_F1}) —
|
||||||
* refuses to backfill at all if the edited corpus scores worse than
|
* refuses to backfill at all if the edited corpus scores worse than
|
||||||
|
|
|
||||||
|
|
@ -5,27 +5,23 @@ import { resetDatabase } from "./reset-db.js";
|
||||||
* Mocha root hook plugin (see `.mocharc.json`'s `require`) — runs once
|
* Mocha root hook plugin (see `.mocharc.json`'s `require`) — runs once
|
||||||
* before every test file's own suites, regardless of load order.
|
* before every test file's own suites, regardless of load order.
|
||||||
*
|
*
|
||||||
* Warms up `techStepClassifier` here, with its own generous timeout,
|
* Warms up `techStepClassifier` here — resolving the `TechStep.key -> id`
|
||||||
|
* lookup from the DB (see `TechStepClassifierService._loadTechStepIds`) —
|
||||||
* instead of leaving it to happen lazily on whichever test file Mocha
|
* instead of leaving it to happen lazily on whichever test file Mocha
|
||||||
* happens to load first. In production this one-time cost (a `POST
|
* happens to load first, same as `server.ts` does before the real server
|
||||||
* /v1/train` round-trip per locale to `services/tech-step-intent-service`,
|
* ever accepts traffic. Fast by itself (one DB query, one HTTP call to
|
||||||
* training a real `textcat` on the full `TECH_STEP_TRAINING_DATA` corpus)
|
* `services/tech-step-intent-service`): that service now trains itself
|
||||||
* is paid by `server.ts`'s own `techStepClassifier.warmUp()` before the
|
* entirely at its own process startup (see its own README), so unlike
|
||||||
* server ever accepts traffic — but this test suite builds its `app`
|
* before this migration, nothing here waits on a slow training pass — CI's
|
||||||
* directly via `createApp()` (see e.g. `tech-step-worker.routes.test.ts`),
|
* own "wait for `/health`" step (`.github/workflows/ci.yml`) is what
|
||||||
* never running `server.ts` at all. Without this hook, that cost instead
|
* ensures that service is already fully trained before `pnpm --filter api
|
||||||
* landed inside whichever test's own call happened to trigger
|
* test` even starts.
|
||||||
* `_ensureTrained()` first — found the hard way in CI, where training the
|
|
||||||
* full corpus took longer than a single test's default 10s timeout
|
|
||||||
* (`.mocharc.json`) and failed an otherwise-unrelated test purely because
|
|
||||||
* Mocha loaded its file first alphabetically.
|
|
||||||
*
|
*
|
||||||
* `resetDatabase()` runs first, deliberately: `_train()`
|
* `resetDatabase()` runs first, deliberately: id resolution needs
|
||||||
* (`tech-step-matcher.ts`) resolves `TechStep.key -> id` from the database
|
* `TechStep` rows, and a freshly-migrated (never-seeded) test database has
|
||||||
* alongside training, and a freshly-migrated (never-seeded) test database
|
* none yet. Every per-test `beforeEach` in this suite already calls
|
||||||
* has no `TechStep` rows yet — every per-test `beforeEach` in this suite
|
* `resetDatabase()` again before its own test, which is a no-op
|
||||||
* already calls `resetDatabase()` again before its own test, which is a
|
* duplication of effort but not a correctness problem: `TRUNCATE ...
|
||||||
* no-op duplication of effort but not a correctness problem: `TRUNCATE ...
|
|
||||||
* RESTART IDENTITY` plus deterministic re-seeding (`seedReferenceData`)
|
* RESTART IDENTITY` plus deterministic re-seeding (`seedReferenceData`)
|
||||||
* assigns the exact same ids every time, so the `uid -> id` map memoized
|
* assigns the exact same ids every time, so the `uid -> id` map memoized
|
||||||
* here from this first reset stays valid for every reset after it.
|
* here from this first reset stays valid for every reset after it.
|
||||||
|
|
@ -33,13 +29,11 @@ import { resetDatabase } from "./reset-db.js";
|
||||||
export const mochaHooks = {
|
export const mochaHooks = {
|
||||||
// biome-ignore lint/suspicious/noExplicitAny: Mocha's root hook `this` (a Context with `.timeout()`) isn't typed without @types/mocha (not a dependency here) — same untyped-`this` shape already used in tech-step-worker.routes.test.ts.
|
// biome-ignore lint/suspicious/noExplicitAny: Mocha's root hook `this` (a Context with `.timeout()`) isn't typed without @types/mocha (not a dependency here) — same untyped-`this` shape already used in tech-step-worker.routes.test.ts.
|
||||||
async beforeAll(this: any): Promise<void> {
|
async beforeAll(this: any): Promise<void> {
|
||||||
// Generous on purpose: training both locales' `textcat` on the full
|
// A little more generous than Mocha's normal 10s per-test default
|
||||||
// corpus takes on the order of a couple of minutes combined (see
|
// (`.mocharc.json`) purely for a slower/contended CI runner's first
|
||||||
// `_TRAINING_ITERATIONS` in `services/tech-step-intent-service`'s
|
// network round-trip to `services/tech-step-intent-service` — not
|
||||||
// `locale_pipeline.py`) — comfortably under 10 minutes even on a
|
// because anything here waits on training anymore.
|
||||||
// slower/contended CI runner, but nowhere near Mocha's normal 10s
|
this.timeout(30000);
|
||||||
// per-test default (`.mocharc.json`).
|
|
||||||
this.timeout(600000);
|
|
||||||
await resetDatabase();
|
await resetDatabase();
|
||||||
await techStepClassifier.warmUp();
|
await techStepClassifier.warmUp();
|
||||||
},
|
},
|
||||||
|
|
|
||||||
|
|
@ -38,8 +38,9 @@ describe("recipe-translation", () => {
|
||||||
// `translateRecipeSteps` now goes through `techStepClassifier` (a
|
// `translateRecipeSteps` now goes through `techStepClassifier` (a
|
||||||
// trained model, not a pure regex test against a caller-supplied
|
// trained model, not a pure regex test against a caller-supplied
|
||||||
// mapping list — see `tech-step-matcher.ts`), so these tests exercise
|
// mapping list — see `tech-step-matcher.ts`), so these tests exercise
|
||||||
// the real training corpus (`tech-step-training-data.ts`) against a real
|
// the real training corpus (`services/tech-step-intent-service`'s
|
||||||
// `TechStep` catalog rather than synthetic fixtures — same posture
|
// `training_data.py`) against a real `TechStep` catalog rather than
|
||||||
|
// synthetic fixtures — same posture
|
||||||
// `tech-step-matcher.test.ts`'s own `techStepClassifier` describe block
|
// `tech-step-matcher.test.ts`'s own `techStepClassifier` describe block
|
||||||
// takes, for the same reason.
|
// takes, for the same reason.
|
||||||
describe("translateRecipeSteps", () => {
|
describe("translateRecipeSteps", () => {
|
||||||
|
|
|
||||||
|
|
@ -118,15 +118,16 @@ describe("tech-step-matcher", () => {
|
||||||
// `techStepClassifier` is the one shared singleton (see
|
// `techStepClassifier` is the one shared singleton (see
|
||||||
// tech-step-matcher.ts's own doc comment on why) — these tests
|
// tech-step-matcher.ts's own doc comment on why) — these tests
|
||||||
// exercise it against the real training corpus
|
// exercise it against the real training corpus
|
||||||
// (`tech-step-training-data.ts`) and the real seeded `TechStep`
|
// (`services/tech-step-intent-service`'s `training_data.py`) and the
|
||||||
// catalog, rather than synthetic injectable fixtures the old
|
// real seeded `TechStep` catalog, rather than synthetic injectable
|
||||||
// regex-based `matchTechStepSpans(description, mappings)` allowed.
|
// fixtures the old regex-based `matchTechStepSpans(description,
|
||||||
// Training now round-trips over HTTP to a real, locally running
|
// mappings)` allowed. Every call round-trips over HTTP to a real,
|
||||||
// `services/tech-step-intent-service` (see that service's own README
|
// locally running `services/tech-step-intent-service` (see that
|
||||||
// and `apps/api/.env.test`) — the very first call in the whole suite
|
// service's own README and `apps/api/.env.test`) — that service trains
|
||||||
// pays for that plus the service's own spaCy pipeline setup (subsequent
|
// itself once at its own startup (`test-support/mocha-root-hooks.ts`'s
|
||||||
// calls reuse the already-trained pipeline and are fast) — comfortably
|
// root hook doesn't wait on it, CI's own "wait for /health" step
|
||||||
// inside this suite's default 10s timeout (.mocharc.json).
|
// already does), so calls here are just a normal HTTP round-trip,
|
||||||
|
// comfortably inside this suite's default 10s timeout (.mocharc.json).
|
||||||
let simmerId: number;
|
let simmerId: number;
|
||||||
let cookId: number;
|
let cookId: number;
|
||||||
let bakeId: number;
|
let bakeId: number;
|
||||||
|
|
|
||||||
|
|
@ -79,7 +79,7 @@ describe("Recipe tech-step corrections", () => {
|
||||||
const { agent, profileId } = await signup();
|
const { agent, profileId } = await signup();
|
||||||
// "Faire mijoter la sauce." names no technique the classifier itself
|
// "Faire mijoter la sauce." names no technique the classifier itself
|
||||||
// registers a bare-word anchor for at this exact span in isolation
|
// registers a bare-word anchor for at this exact span in isolation
|
||||||
// (see tech-step-training-data.ts) — irrelevant here either way,
|
// (see services/tech-step-intent-service's training_data.py) — irrelevant here either way,
|
||||||
// since this test's whole point is the *manual* addition, not
|
// since this test's whole point is the *manual* addition, not
|
||||||
// whatever the classifier does or doesn't auto-detect for it.
|
// whatever the classifier does or doesn't auto-detect for it.
|
||||||
const { recipeId, stepId } = await createPublicRecipeWithStep(profileId);
|
const { recipeId, stepId } = await createPublicRecipeWithStep(profileId);
|
||||||
|
|
|
||||||
|
|
@ -425,7 +425,55 @@
|
||||||
"preheat": "Préchauffer",
|
"preheat": "Préchauffer",
|
||||||
"bake": "Cuire au four",
|
"bake": "Cuire au four",
|
||||||
"plate": "Dresser",
|
"plate": "Dresser",
|
||||||
"coat": "Napper"
|
"coat": "Napper",
|
||||||
|
"baste": "Arroser",
|
||||||
|
"appertize": "Appertiser",
|
||||||
|
"whiskPale": "Blanchir (jaunes d'œufs)",
|
||||||
|
"goldenBrown": "Blondir",
|
||||||
|
"braise": "Braiser",
|
||||||
|
"truss": "Brider",
|
||||||
|
"caramelize": "Caraméliser",
|
||||||
|
"score": "Cerner",
|
||||||
|
"lineMold": "Chemiser",
|
||||||
|
"clarify": "Clarifier",
|
||||||
|
"compote": "Compoter",
|
||||||
|
"concasse": "Concasser",
|
||||||
|
"confit": "Confire",
|
||||||
|
"julienne": "Couper en julienne",
|
||||||
|
"brunoise": "Couper en brunoise",
|
||||||
|
"mirepoix": "Couper en mirepoix",
|
||||||
|
"paysanne": "Couper en paysanne",
|
||||||
|
"blindBake": "Cuire à blanc",
|
||||||
|
"bainMarie": "Cuire au bain-marie",
|
||||||
|
"smother": "Cuire à l'étouffée",
|
||||||
|
"decant": "Décanter",
|
||||||
|
"dilute": "Délayer",
|
||||||
|
"punchDown": "Dégazer",
|
||||||
|
"disgorge": "Dégorger",
|
||||||
|
"loosen": "Détendre",
|
||||||
|
"shellEgg": "Écaler",
|
||||||
|
"scald": "Échauder",
|
||||||
|
"pod": "Écosser",
|
||||||
|
"emulsify": "Émulsionner",
|
||||||
|
"hollowOut": "Évider",
|
||||||
|
"shock": "Frapper",
|
||||||
|
"setGel": "Gélifier",
|
||||||
|
"glaze": "Glacer",
|
||||||
|
"thicken": "Lier",
|
||||||
|
"filet": "Lever les filets",
|
||||||
|
"proof": "Laisser pousser",
|
||||||
|
"peelBlanch": "Monder",
|
||||||
|
"whipUp": "Monter",
|
||||||
|
"moisten": "Mouiller",
|
||||||
|
"pasteurize": "Pasteuriser",
|
||||||
|
"poach": "Pocher",
|
||||||
|
"reduce": "Réduire",
|
||||||
|
"rubIn": "Sabler",
|
||||||
|
"dustWithFlour": "Singer",
|
||||||
|
"sweat": "Suer",
|
||||||
|
"sift": "Tamiser",
|
||||||
|
"toast": "Torréfier",
|
||||||
|
"zest": "Zester"
|
||||||
},
|
},
|
||||||
"allergens": {
|
"allergens": {
|
||||||
"gluten": "Gluten",
|
"gluten": "Gluten",
|
||||||
|
|
|
||||||
|
|
@ -88,10 +88,20 @@ services:
|
||||||
"-c",
|
"-c",
|
||||||
"import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=2)",
|
"import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=2)",
|
||||||
]
|
]
|
||||||
interval: 10s
|
interval: 15s
|
||||||
timeout: 3s
|
timeout: 3s
|
||||||
retries: 10
|
retries: 5
|
||||||
start_period: 15s
|
# This service trains itself from scratch on every start (no model
|
||||||
|
# ever persisted to disk, see its own README) — `/health` only
|
||||||
|
# returns 200 once that's done, not just once the base spaCy models
|
||||||
|
# are loaded. Measured at ~200s per locale (~400s for fr+en combined)
|
||||||
|
# against the current ~74-technique corpus
|
||||||
|
# (`intent_service/training_data.py`) — `start_period` generous
|
||||||
|
# enough that failing checks during that whole window never count
|
||||||
|
# against `retries` (which would otherwise flip this container to
|
||||||
|
# "unhealthy" mid-training, blocking `app`'s own
|
||||||
|
# `depends_on: condition: service_healthy` indefinitely).
|
||||||
|
start_period: 600s
|
||||||
|
|
||||||
# Deliberately its own image, not built into `app`'s (see
|
# Deliberately its own image, not built into `app`'s (see
|
||||||
# services/tech-step-llm-worker/Dockerfile's own doc comment) — a
|
# services/tech-step-llm-worker/Dockerfile's own doc comment) — a
|
||||||
|
|
|
||||||
|
|
@ -19,18 +19,29 @@ ce service en l'état) de pouvoir aussi absorber ce que fait aujourd'hui
|
||||||
passer (les modèles `md`, avec vecteurs de mots, sont conservés dans ce but,
|
passer (les modèles `md`, avec vecteurs de mots, sont conservés dans ce but,
|
||||||
même si rien ici ne s'en sert encore).
|
même si rien ici ne s'en sert encore).
|
||||||
|
|
||||||
## Pourquoi ce service ne possède aucune donnée d'entraînement
|
## Ce service est entièrement autonome
|
||||||
|
|
||||||
Contrairement à un service NLP habituel, **ce service ne connaît aucune
|
Contrairement à sa toute première version, **ce service possède désormais
|
||||||
technique par lui-même** — `apps/api` reste l'unique source de vérité du
|
son propre corpus** — `intent_service/training_data.py`
|
||||||
corpus (`TECH_STEP_TRAINING_DATA`,
|
(`TECH_STEP_TRAINING_DATA`), revu par PR comme le reste du code. Il
|
||||||
`apps/api/src/lib/recipe-matching/tech-step-training-data.ts`, revu par PR
|
s'entraîne lui-même une seule fois, à son propre démarrage
|
||||||
comme le reste du code). Il pousse l'intégralité du corpus ici via
|
(`PipelineRegistry.initialize()`, appelé par `main.py`'s `lifespan`), et ne
|
||||||
`POST /v1/train` à chaque warm-up serveur (`TechStepClassifierService._train`)
|
persiste jamais rien sur disque — un redémarrage du process réentraîne
|
||||||
— ce service (re)construit alors son pipeline en mémoire, sans jamais rien
|
toujours from scratch depuis ce fichier. `apps/api` ne connaît plus aucune
|
||||||
persister sur disque. Le workflow mainteneur existant
|
technique ni aucun synonyme : il n'appelle plus que `POST /v1/process` (plus
|
||||||
(`apps/api/src/scripts/retrain-tech-steps.ts`, édition manuelle du corpus)
|
de `POST /v1/train`, supprimé).
|
||||||
n'a pas changé.
|
|
||||||
|
Workflow mainteneur pour changer le corpus :
|
||||||
|
|
||||||
|
1. Éditer `intent_service/training_data.py` à la main (informé par le
|
||||||
|
rapport de `apps/api/src/scripts/list-pending-training-suggestions.ts`).
|
||||||
|
2. **Redémarrer ce service** (`docker compose restart tech-step-intent-service`,
|
||||||
|
ou simplement redéployer) — le nouveau corpus n'a d'effet qu'une fois
|
||||||
|
réentraîné au démarrage, contrairement à l'ancienne version qui pouvait
|
||||||
|
être réentraînée à chaud via `POST /v1/train`.
|
||||||
|
3. Depuis `apps/api`, lancer `pnpm --filter api exec tsx
|
||||||
|
src/scripts/retrain-tech-steps.ts` — vérifie le F1 contre
|
||||||
|
`TECH_STEP_EVAL_DATASET` avant de backfiller les recettes existantes.
|
||||||
|
|
||||||
## Pourquoi ce service vit hors du workspace pnpm
|
## Pourquoi ce service vit hors du workspace pnpm
|
||||||
|
|
||||||
|
|
@ -39,31 +50,48 @@ n'a rien à faire dans `pnpm-workspace.yaml` (qui ne couvre que
|
||||||
`apps/*`/`packages/*`), et ses dépendances (spaCy, ses modèles) ne doivent
|
`apps/*`/`packages/*`), et ses dépendances (spaCy, ses modèles) ne doivent
|
||||||
jamais se retrouver dans l'image `apps/api`. **Aucun accès direct à
|
jamais se retrouver dans l'image `apps/api`. **Aucun accès direct à
|
||||||
Postgres** non plus — la résolution `TechStep.key -> id` reste entièrement
|
Postgres** non plus — la résolution `TechStep.key -> id` reste entièrement
|
||||||
côté `apps/api` (`TechStepClassifierService._train`), ce service ne
|
côté `apps/api` (`TechStepClassifierService`), ce service ne manipule que
|
||||||
manipule que des `uid` (chaînes opaques) tout du long.
|
des `uid` (chaînes opaques) tout du long.
|
||||||
|
|
||||||
## Contrat HTTP
|
## Contrat HTTP
|
||||||
|
|
||||||
Voir `intent_service/schemas.py` pour le détail exact. En résumé :
|
Voir `intent_service/schemas.py` pour le détail exact. En résumé :
|
||||||
|
|
||||||
- `GET /health` — sans authentification, `200` une fois les modèles spaCy
|
- `GET /health` — sans authentification, `200` une fois ce service
|
||||||
de base chargés (pas de lazy-load, voir `intent_service/main.py`).
|
entièrement prêt : modèles spaCy de base chargés **et** les deux locales
|
||||||
- `POST /v1/train` — `{ locale, entries: [{ uid, synonyms, utterances }] }`
|
entraînées (pas de lazy-load, voir `intent_service/main.py`) — voir
|
||||||
→ reconstruit le pipeline de `locale` à neuf.
|
"Temps de démarrage" plus bas pour ce que ça implique en pratique.
|
||||||
- `POST /v1/process` — `{ locale, text }` → `{ entities: [{ uid, start, end }], intent, score }`.
|
- `POST /v1/process` — `{ locale, text }` → `{ entities: [{ uid, start, end }], intent, score }`.
|
||||||
|
|
||||||
`/v1/train` et `/v1/process` exigent le header `X-Intent-Service-Secret`
|
`/v1/process` exige le header `X-Intent-Service-Secret` (voir
|
||||||
(voir `intent_service/security.py`), qui doit matcher `INTENT_SERVICE_SECRET`
|
`intent_service/security.py`), qui doit matcher `INTENT_SERVICE_SECRET`
|
||||||
côté `apps/api`.
|
côté `apps/api`.
|
||||||
|
|
||||||
|
## Temps de démarrage
|
||||||
|
|
||||||
|
**Ce service met plusieurs minutes à devenir `healthy`** — contrairement à
|
||||||
|
node-nlp (entraînement quasi instantané), entraîner le `textcat` sur le
|
||||||
|
corpus réel (~74 techniques) prend de l'ordre de 200 secondes par locale
|
||||||
|
(mesuré localement, sans GPU), donc environ 400 secondes (~7 minutes) pour
|
||||||
|
`fr`+`en` combinés à chaque démarrage du process. `docker-compose.yml` et
|
||||||
|
`.github/workflows/ci.yml` ont un `start_period`/timeout d'attente
|
||||||
|
généreux pour ça — voir leurs propres commentaires. C'est un compromis
|
||||||
|
assumé, pas un défaut de configuration à corriger : moins d'itérations
|
||||||
|
entraîne plus vite mais laisse des verdicts corrects sous
|
||||||
|
`CONFIDENCE_THRESHOLD` (voir le commentaire de cette constante,
|
||||||
|
`apps/api/src/lib/recipe-matching/tech-step-matcher.ts`, et celui de
|
||||||
|
`_TRAINING_ITERATIONS`/`_TRAINING_BATCH_SIZE` dans `locale_pipeline.py`
|
||||||
|
pour le détail du compromis).
|
||||||
|
|
||||||
## Logs
|
## Logs
|
||||||
|
|
||||||
`intent_service/logging_config.py` branche un format JSON structuré (une
|
`intent_service/logging_config.py` branche un format JSON structuré (une
|
||||||
ligne par évènement — `timestamp`/`level`/`message` + champs métier fusionnés
|
ligne par évènement — `timestamp`/`level`/`message` + champs métier fusionnés
|
||||||
— même convention que `LoggerService` côté `apps/api`) sur toute la
|
— même convention que `LoggerService` côté `apps/api`) sur toute la
|
||||||
journalisation de ce service, niveau `LOG_LEVEL` (`INFO` par défaut, voir
|
journalisation de ce service, niveau `LOG_LEVEL` (`INFO` par défaut, voir
|
||||||
`.env.example`). `routes/process.py` et `routes/train.py` journalisent
|
`.env.example`). `routes/process.py` journalise chaque appel avec son input
|
||||||
chaque appel avec son input et son output complets :
|
et son output complets, `pipeline_registry.py` journalise le déroulement de
|
||||||
|
l'entraînement au démarrage :
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{"timestamp": "...", "level": "info", "message": "tech-step NLP process", "locale": "fr", "text": "faire fondre le beurre", "entities": [{"uid": "melt", "start": 6, "end": 13}], "intent": "melt", "score": 0.93}
|
{"timestamp": "...", "level": "info", "message": "tech-step NLP process", "locale": "fr", "text": "faire fondre le beurre", "entities": [{"uid": "melt", "start": 6, "end": 13}], "intent": "melt", "score": 0.93}
|
||||||
|
|
@ -95,7 +123,8 @@ uv run uvicorn intent_service.main:app --reload --port 8000
|
||||||
aux côtés de `postgres`/`app`/`tech-step-llm-worker` — **pas optionnel**,
|
aux côtés de `postgres`/`app`/`tech-step-llm-worker` — **pas optionnel**,
|
||||||
contrairement au worker LLM : sans lui, `apps/api` ne peut plus détecter
|
contrairement au worker LLM : sans lui, `apps/api` ne peut plus détecter
|
||||||
aucune technique de cuisine. `app` attend qu'il soit `healthy`
|
aucune technique de cuisine. `app` attend qu'il soit `healthy`
|
||||||
(`depends_on: condition: service_healthy`) avant de démarrer.
|
(`depends_on: condition: service_healthy`) avant de démarrer — voir "Temps
|
||||||
|
de démarrage" ci-dessus pour combien de temps ça prend en pratique.
|
||||||
|
|
||||||
## Testing
|
## Testing
|
||||||
|
|
||||||
|
|
@ -107,6 +136,10 @@ uv run pytest
|
||||||
exacts et d'insensibilité accents/casse de
|
exacts et d'insensibilité accents/casse de
|
||||||
`apps/api/test/recipe-matching/tech-step-matcher.test.ts` — le point de
|
`apps/api/test/recipe-matching/tech-step-matcher.test.ts` — le point de
|
||||||
fidélité le plus critique de ce service (voir le plan de migration).
|
fidélité le plus critique de ce service (voir le plan de migration).
|
||||||
|
`tests/conftest.py`'s fixture `client` (scope "session") ne s'entraîne
|
||||||
|
qu'une seule fois pour toute la suite — c'est *le vrai corpus complet*,
|
||||||
|
pas un jeu jouet, donc la première utilisation de cette fixture prend le
|
||||||
|
même temps qu'un vrai démarrage (voir "Temps de démarrage" ci-dessus).
|
||||||
|
|
||||||
Aucun test ici ne dépend d'une vraie base Postgres ni d'`apps/api` en
|
Aucun test ici ne dépend d'une vraie base Postgres ni d'`apps/api` en
|
||||||
service — à l'inverse, la suite Mocha d'`apps/api`
|
service — à l'inverse, la suite Mocha d'`apps/api`
|
||||||
|
|
@ -114,24 +147,24 @@ service — à l'inverse, la suite Mocha d'`apps/api`
|
||||||
vraie instance de ce service tournant (voir `apps/api/.env.test`), conforme
|
vraie instance de ce service tournant (voir `apps/api/.env.test`), conforme
|
||||||
à la convention du repo de ne jamais mocker un service interne.
|
à la convention du repo de ne jamais mocker un service interne.
|
||||||
|
|
||||||
## Limitations connues (première version)
|
## Limitations connues
|
||||||
|
|
||||||
- **`/v1/train` prend de l'ordre de la minute par locale** (~110s mesuré en
|
- **Démarrage lent** (~7 minutes) — voir "Temps de démarrage" ci-dessus.
|
||||||
CI avec `_TRAINING_ITERATIONS`/`_TRAINING_BATCH_SIZE` actuels, voir
|
Une optimisation possible non explorée : parallélisation de
|
||||||
`locale_pipeline.py`) — `apps/api` l'appelle deux fois au warm-up
|
l'entraînement `fr`/`en` (actuellement séquentiel,
|
||||||
(`fr`/`en`), donc un redémarrage prend quelques minutes avant qu'une
|
`PipelineRegistry.initialize`).
|
||||||
recette puisse voir ses techniques détectées. Contrairement à node-nlp
|
- **`CONFIDENCE_THRESHOLD` côté `apps/api` est un placeholder** depuis
|
||||||
(entraînement quasi instantané), c'est un vrai compromis assumé : moins
|
l'élargissement du corpus à ~74 techniques (calibré à la main, pas via
|
||||||
d'itérations entraînait plus vite mais laissait des verdicts corrects
|
une vraie repasse de `calibrate-tech-step-threshold.ts` contre
|
||||||
sous `CONFIDENCE_THRESHOLD` (voir le commentaire de cette constante,
|
`TECH_STEP_EVAL_DATASET` — voir le commentaire de cette constante).
|
||||||
`apps/api/src/lib/recipe-matching/tech-step-matcher.ts`).
|
|
||||||
- **Textcat bag-of-words** (`spacy.TextCatBOW.v3`) — suffisant pour le
|
- **Textcat bag-of-words** (`spacy.TextCatBOW.v3`) — suffisant pour le
|
||||||
corpus actuel une fois correctement entraîné, mais n'exploite pas les
|
corpus actuel une fois correctement entraîné, mais n'exploite pas les
|
||||||
vecteurs de mots des modèles `md` chargés. Migrable vers une architecture
|
vecteurs de mots des modèles `md` chargés. Migrable vers une architecture
|
||||||
tok2vec/similarité sans changer le contrat HTTP, si le F1 mesuré par
|
tok2vec/similarité sans changer le contrat HTTP, si le F1 mesuré par
|
||||||
`apps/api/src/scripts/calibrate-tech-step-threshold.ts` le justifie un
|
`apps/api/src/scripts/calibrate-tech-step-threshold.ts` le justifie un
|
||||||
jour.
|
jour.
|
||||||
- **Reconstruit tout le pipeline à chaque `/v1/train`** (pas de fusion
|
- **Reconstruit tout le pipeline à chaque démarrage** (pas de persistance,
|
||||||
incrémentale) — un choix délibéré (voir `LocalePipeline.train`), pas une
|
pas de fusion incrémentale) — un choix délibéré (voir
|
||||||
limitation à lever : `TECH_STEP_TRAINING_DATA` doit toujours rester
|
`LocalePipeline.train`), pas une limitation à lever : `training_data.py`
|
||||||
l'unique source de vérité, jamais un état local qui dérive.
|
doit toujours rester l'unique source de vérité, jamais un état sur disque
|
||||||
|
qui pourrait dériver.
|
||||||
|
|
|
||||||
|
|
@ -42,8 +42,9 @@ class Settings(BaseSettings):
|
||||||
|
|
||||||
# Niveau du logging structuré (`logging_config.py`) — voir ce module pour
|
# Niveau du logging structuré (`logging_config.py`) — voir ce module pour
|
||||||
# le format. `INFO` par défaut : c'est à ce niveau que `routes/process.py`
|
# le format. `INFO` par défaut : c'est à ce niveau que `routes/process.py`
|
||||||
# et `routes/train.py` journalisent chaque input/output du pipeline NLP,
|
# journalise chaque input/output du pipeline NLP, et que
|
||||||
# pour qu'un déploiement par défaut les voie sans configuration
|
# `pipeline_registry.py` journalise l'entraînement au démarrage, pour
|
||||||
|
# qu'un déploiement par défaut les voie sans configuration
|
||||||
# supplémentaire (`docker logs`/Portainer).
|
# supplémentaire (`docker logs`/Portainer).
|
||||||
log_level: str = "INFO"
|
log_level: str = "INFO"
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -2,8 +2,9 @@
|
||||||
`node-nlp`'s `NlpManager` faisait pour cette locale dans
|
`node-nlp`'s `NlpManager` faisait pour cette locale dans
|
||||||
`TechStepClassifierService` (`apps/api/src/lib/recipe-matching/tech-step-matcher.ts`) :
|
`TechStepClassifierService` (`apps/api/src/lib/recipe-matching/tech-step-matcher.ts`) :
|
||||||
NER par entités enum (ici un `PhraseMatcher`) + classification d'intention
|
NER par entités enum (ici un `PhraseMatcher`) + classification d'intention
|
||||||
(ici un `textcat`), les deux entraînés à partir du même corpus
|
(ici un `textcat`), les deux entraînés à partir du corpus possédé par ce
|
||||||
(`TECH_STEP_TRAINING_DATA`, poussé par `apps/api` via `POST /v1/train`).
|
service lui-même (`training_data.TECH_STEP_TRAINING_DATA` — plus poussé par
|
||||||
|
`apps/api` via HTTP, voir `pipeline_registry.py`).
|
||||||
|
|
||||||
Le modèle de base spaCy (tokenizer + vecteurs + le composant
|
Le modèle de base spaCy (tokenizer + vecteurs + le composant
|
||||||
`diacritics_normalizer` défini plus bas) est chargé une seule fois
|
`diacritics_normalizer` défini plus bas) est chargé une seule fois
|
||||||
|
|
@ -12,9 +13,8 @@ paresseusement au premier `train()`, pour que `GET /health` ne devienne
|
||||||
`200` qu'une fois ce coût payé) puis réutilisé à chaque `train()` : seul le
|
`200` qu'une fois ce coût payé) puis réutilisé à chaque `train()` : seul le
|
||||||
`textcat` (retiré puis rajouté à neuf) et le `PhraseMatcher` (remplacé) sont
|
`textcat` (retiré puis rajouté à neuf) et le `PhraseMatcher` (remplacé) sont
|
||||||
reconstruits à chaque appel, jamais le tokenizer/les vecteurs. Rien n'est
|
reconstruits à chaque appel, jamais le tokenizer/les vecteurs. Rien n'est
|
||||||
jamais persisté sur disque — même posture que `autoSave`/`autoLoad: false`
|
jamais persisté sur disque — `training_data.py` reste l'unique source de
|
||||||
sur l'ancien `NlpManager` : `TECH_STEP_TRAINING_DATA` (côté `apps/api`) reste
|
vérité, reconstruite en mémoire depuis zéro à chaque démarrage du process.
|
||||||
l'unique source de vérité.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
@ -50,21 +50,32 @@ _EXCLUDED_COMPONENTS = ["parser", "ner", "tagger", "morphologizer", "attribute_r
|
||||||
_TEXTCAT_PIPE_NAME = "textcat"
|
_TEXTCAT_PIPE_NAME = "textcat"
|
||||||
|
|
||||||
# Nombre d'itérations d'entraînement du textcat et taille de minibatch —
|
# Nombre d'itérations d'entraînement du textcat et taille de minibatch —
|
||||||
# calibrés empiriquement contre le corpus réel (`TECH_STEP_TRAINING_DATA`,
|
# calibrés empiriquement contre le corpus réel (`training_data.py`), pas
|
||||||
# ~26 techniques/locale), pas seulement contre les petits corpus jouets des
|
# seulement contre les petits corpus jouets des tests de ce fichier. Trop
|
||||||
# tests de ce fichier. Une première valeur plus basse (30 itérations, lot de
|
# peu d'itérations laisse des clauses correctement classifiées (bon argmax)
|
||||||
# 8) convergeait mal sur le vrai corpus : des clauses correctement
|
# mais avec une confiance dérisoire (`0.02`-`0.08` observé à 5-15
|
||||||
# classifiées mais sans ancre NER (le cas motivant tout ce pipeline, voir
|
# itérations) — bien en dessous de tout seuil raisonnable pour
|
||||||
# `TechStepClassifierService`'s doc comment côté apps/api) scoraient à peine
|
# `CONFIDENCE_THRESHOLD` (`tech-step-matcher.ts`).
|
||||||
# 0.5-0.7, et des clauses *avec* ancre à peine 0.2-0.3 — bien en dessous de
|
#
|
||||||
# `CONFIDENCE_THRESHOLD` (`tech-step-matcher.ts`), un CI réel l'a confirmé
|
# `150` convenait au corpus original (~26 techniques) mais ne passe plus à
|
||||||
# avant que ces valeurs ne soient relevées. `150`/`16` entraîne en ~110s par
|
# l'échelle une fois le corpus élargi à ~74 : le temps d'entraînement croît
|
||||||
# locale sur un runner GitHub Actions standard (donc ~220s pour fr+en
|
# avec le nombre de classes autant qu'avec les itérations (mesuré :
|
||||||
# combinés au warm-up — voir `server.ts`'s propre commentaire sur le retry
|
# ~150s pour seulement 30 itérations sur 74 classes, contre ~110s pour 150
|
||||||
# côté apps/api) et pousse les mêmes scores nettement au-dessus du seuil
|
# itérations sur 26 classes) — `150` sur 74 classes dépassait 17 minutes
|
||||||
# (melt ~0.95, preheat ~0.90, bake ~0.51) sans dégrader le rejet du bruit
|
# rien que pour une locale, constaté en CI. `40` est le meilleur compromis
|
||||||
# (texte anglais via le classifieur français reste ~0.05, inchangé).
|
# trouvé empiriquement sur ce corpus élargi : ~200s par locale (~400s pour
|
||||||
_TRAINING_ITERATIONS = 150
|
# fr+en combinés au démarrage), avec des scores exploitables sur tous les
|
||||||
|
# cas testés à la main (melt ~0.89, preheat ~0.76, compote ~0.76, zest
|
||||||
|
# ~0.64, julienne ~0.56, cook/bake ~0.33, simmer ~0.25 — le plus faible
|
||||||
|
# observé, toujours correct en argmax et de toute façon ancré par NER) et
|
||||||
|
# un bruit hors-vocabulaire qui reste négligeable (anglais via le
|
||||||
|
# classifieur français : `~0.02`). Une vraie repasse de
|
||||||
|
# `calibrate-tech-step-threshold.ts` contre `TECH_STEP_EVAL_DATASET` reste
|
||||||
|
# nécessaire pour confirmer/affiner ces deux valeurs (voir
|
||||||
|
# `CONFIDENCE_THRESHOLD`'s propre commentaire, `tech-step-matcher.ts`) — ce
|
||||||
|
# qui suit est une mesure manuelle ponctuelle, pas un remplacement de cette
|
||||||
|
# calibration.
|
||||||
|
_TRAINING_ITERATIONS = 40
|
||||||
_TRAINING_BATCH_SIZE = 16
|
_TRAINING_BATCH_SIZE = 16
|
||||||
# Abaissé de `0.2` avec le reste de cette recalibration — `0.1` régularise
|
# Abaissé de `0.2` avec le reste de cette recalibration — `0.1` régularise
|
||||||
# encore contre la petite taille du corpus par technique tout en laissant
|
# encore contre la petite taille du corpus par technique tout en laissant
|
||||||
|
|
@ -107,8 +118,8 @@ class _DiacriticsNormalizer:
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class TrainEntry:
|
class TrainEntry:
|
||||||
"""Une technique à entraîner pour une locale — miroir de
|
"""Une technique à entraîner pour une locale — construit par
|
||||||
`TrainEntryPayload` (`schemas.py`)."""
|
`PipelineRegistry.initialize()` depuis `training_data.entries_for_locale`."""
|
||||||
|
|
||||||
uid: str
|
uid: str
|
||||||
synonyms: list[str] = field(default_factory=list)
|
synonyms: list[str] = field(default_factory=list)
|
||||||
|
|
|
||||||
|
|
@ -6,7 +6,7 @@ Portainer ou un agrégateur de logs — cohérent avec le reste du repo plutôt
|
||||||
qu'un format propre à ce seul service.
|
qu'un format propre à ce seul service.
|
||||||
|
|
||||||
Configuré une fois au démarrage (`main.py`) plutôt que par un `print()` ad
|
Configuré une fois au démarrage (`main.py`) plutôt que par un `print()` ad
|
||||||
hoc dans chaque route — `routes/process.py`/`routes/train.py` appellent
|
hoc dans chaque route — `routes/process.py`/`pipeline_registry.py` appellent
|
||||||
`logging.getLogger(__name__)` normalement, ce module ne fait que brancher le
|
`logging.getLogger(__name__)` normalement, ce module ne fait que brancher le
|
||||||
formateur JSON sur la racine du logging Python.
|
formateur JSON sur la racine du logging Python.
|
||||||
"""
|
"""
|
||||||
|
|
@ -69,7 +69,8 @@ def configure_logging(level: str) -> None:
|
||||||
# spaCy/thinc journalisent leur propre chatter interne ("Created
|
# spaCy/thinc journalisent leur propre chatter interne ("Created
|
||||||
# vocabulary", "Finished initializing nlp object"...) sur le logger
|
# vocabulary", "Finished initializing nlp object"...) sur le logger
|
||||||
# `"spacy"`, qui propage jusqu'à la racine et se retrouverait donc
|
# `"spacy"`, qui propage jusqu'à la racine et se retrouverait donc
|
||||||
# mélangé aux lignes input/output de `routes/process.py`/`routes/train.py`
|
# mélangé aux lignes input/output de `routes/process.py`/l'entraînement
|
||||||
|
# journalisé par `pipeline_registry.py`
|
||||||
# — ce sont ces dernières que ce service existe pour rendre visibles, pas
|
# — ce sont ces dernières que ce service existe pour rendre visibles, pas
|
||||||
# le détail interne de spaCy. `WARNING` laisse quand même remonter un
|
# le détail interne de spaCy. `WARNING` laisse quand même remonter un
|
||||||
# vrai problème (dépréciation, échec partiel) sans le bruit `INFO`.
|
# vrai problème (dépréciation, échec partiel) sans le bruit `INFO`.
|
||||||
|
|
|
||||||
|
|
@ -1,11 +1,15 @@
|
||||||
"""Point d'entrée FastAPI — `uv run uvicorn intent_service.main:app` (voir
|
"""Point d'entrée FastAPI — `uv run uvicorn intent_service.main:app` (voir
|
||||||
le Dockerfile et le README de ce service).
|
le Dockerfile et le README de ce service).
|
||||||
|
|
||||||
Le chargement des modèles spaCy de base (`PipelineRegistry.preload_all`) se
|
Le chargement des modèles spaCy de base *et* l'entraînement de chaque
|
||||||
fait dans le handler `lifespan` ci-dessous, *avant* qu'uvicorn n'accepte de
|
locale (`PipelineRegistry.initialize`) se font dans le handler `lifespan`
|
||||||
requêtes — `GET /health` ne répond donc `200` qu'une fois ce coût payé,
|
ci-dessous, *avant* qu'uvicorn n'accepte de requêtes — `GET /health` ne
|
||||||
jamais pendant un chargement encore en cours (uvicorn ne sert aucune requête
|
répond donc `200` qu'une fois ce coût payé (chargement + entraînement),
|
||||||
tant que le `lifespan` de démarrage n'est pas terminé).
|
jamais pendant qu'il est encore en cours (uvicorn ne sert aucune requête
|
||||||
|
tant que le `lifespan` de démarrage n'est pas terminé). Ce service est
|
||||||
|
autonome : `training_data.TECH_STEP_TRAINING_DATA` vit dans ce module,
|
||||||
|
`apps/api` ne pousse plus rien via HTTP (voir `pipeline_registry.py` pour
|
||||||
|
le détail de ce que ça change par rapport à la version précédente).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from contextlib import asynccontextmanager
|
from contextlib import asynccontextmanager
|
||||||
|
|
@ -15,22 +19,21 @@ from fastapi import FastAPI
|
||||||
from .config import settings
|
from .config import settings
|
||||||
from .logging_config import configure_logging
|
from .logging_config import configure_logging
|
||||||
from .pipeline_registry import registry
|
from .pipeline_registry import registry
|
||||||
from .routes import health, process, train
|
from .routes import health, process
|
||||||
|
|
||||||
# Avant tout le reste : `routes/process.py`/`routes/train.py` journalisent
|
# Avant tout le reste : `routes/process.py` journalise dès la première
|
||||||
# dès la première requête, `preload_all()` ci-dessous journalise aussi (voir
|
# requête, `initialize()` ci-dessous journalise aussi (voir
|
||||||
# `pipeline_registry.py`) — le formateur JSON doit déjà être en place.
|
# `pipeline_registry.py`) — le formateur JSON doit déjà être en place.
|
||||||
configure_logging(settings.log_level)
|
configure_logging(settings.log_level)
|
||||||
|
|
||||||
|
|
||||||
@asynccontextmanager
|
@asynccontextmanager
|
||||||
async def lifespan(app: FastAPI):
|
async def lifespan(app: FastAPI):
|
||||||
registry.preload_all()
|
registry.initialize()
|
||||||
yield
|
yield
|
||||||
|
|
||||||
|
|
||||||
app = FastAPI(title="tech-step-intent-service", lifespan=lifespan)
|
app = FastAPI(title="tech-step-intent-service", lifespan=lifespan)
|
||||||
|
|
||||||
app.include_router(health.router)
|
app.include_router(health.router)
|
||||||
app.include_router(train.router)
|
|
||||||
app.include_router(process.router)
|
app.include_router(process.router)
|
||||||
|
|
|
||||||
|
|
@ -3,16 +3,17 @@ partagé du process (une instance vit pour toute la durée de vie d'`uvicorn`,
|
||||||
montée sur `app.state`, voir `main.py`).
|
montée sur `app.state`, voir `main.py`).
|
||||||
|
|
||||||
Volontairement une classe "registre" séparée de `LocalePipeline` lui-même :
|
Volontairement une classe "registre" séparée de `LocalePipeline` lui-même :
|
||||||
`LocalePipeline` ne connaît qu'une seule locale, ce module route
|
`LocalePipeline` ne connaît qu'une seule locale, ce module route `process`
|
||||||
`train`/`process` vers la bonne instance selon le `locale` reçu dans la
|
vers la bonne instance selon le `locale` reçu dans la requête — même
|
||||||
requête — même séparation de responsabilité que `TechStepClassifierService`
|
séparation de responsabilité que `TechStepClassifierService` (une seule
|
||||||
(une seule instance, un seul `NlpManager` multi-langues) avait implicitement
|
instance, un seul `NlpManager` multi-langues) avait implicitement via
|
||||||
via node-nlp, explicitée ici puisque spaCy charge un modèle par langue.
|
node-nlp, explicitée ici puisque spaCy charge un modèle par langue.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import logging
|
import logging
|
||||||
|
|
||||||
from .locale_pipeline import SUPPORTED_LOCALES, LocalePipeline, ProcessResult, TrainEntry, UnsupportedLocaleError
|
from .locale_pipeline import SUPPORTED_LOCALES, LocalePipeline, ProcessResult, TrainEntry
|
||||||
|
from .training_data import entries_for_locale
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
@ -23,21 +24,38 @@ class PipelineRegistry:
|
||||||
locale: LocalePipeline(locale) for locale in SUPPORTED_LOCALES
|
locale: LocalePipeline(locale) for locale in SUPPORTED_LOCALES
|
||||||
}
|
}
|
||||||
|
|
||||||
def preload_all(self) -> None:
|
def initialize(self) -> None:
|
||||||
"""Charge le modèle spaCy de base de chaque locale connue — appelé
|
"""Charge le modèle spaCy de base *et* entraîne chaque locale connue
|
||||||
une fois au démarrage du process (`main.py`), pas paresseusement au
|
depuis `training_data.TECH_STEP_TRAINING_DATA` — appelé une fois au
|
||||||
premier appel, pour que `GET /health` ne réponde `200` qu'une fois
|
démarrage du process (`main.py`'s `lifespan`), avant que `uvicorn`
|
||||||
ce coût payé (voir `LocalePipeline.preload`)."""
|
n'accepte de requêtes.
|
||||||
logger.info("tech-step NLP preloading base pipelines", extra={"locales": list(self._pipelines)})
|
|
||||||
for pipeline in self._pipelines.values():
|
|
||||||
pipeline.preload()
|
|
||||||
logger.info("tech-step NLP base pipelines ready", extra={"locales": list(self._pipelines)})
|
|
||||||
|
|
||||||
def train(self, locale: str, entries: list[TrainEntry]) -> tuple[int, int, int]:
|
Contrairement à la version précédente de ce service (où `apps/api`
|
||||||
pipeline = self._pipelines.get(locale)
|
poussait le corpus via `POST /v1/train` à son propre warm-up), ce
|
||||||
if pipeline is None:
|
service est maintenant entièrement autonome : `apps/api` ne connaît
|
||||||
raise UnsupportedLocaleError(f"Unsupported locale: {locale!r}")
|
plus aucune technique, seulement le résultat de
|
||||||
return pipeline.train(entries)
|
`POST /v1/process`. `GET /health` ne répond `200` qu'une fois cette
|
||||||
|
méthode terminée (chargement *et* entraînement) — pas seulement le
|
||||||
|
chargement — pour que `docker-compose.yml`'s `depends_on: ...
|
||||||
|
condition: service_healthy` (et la boucle d'attente équivalente en
|
||||||
|
CI) ne laisse jamais `apps/api` démarrer face à un service qui
|
||||||
|
répondrait mais ne saurait encore rien détecter.
|
||||||
|
"""
|
||||||
|
logger.info("tech-step NLP initializing pipelines", extra={"locales": list(self._pipelines)})
|
||||||
|
for locale, pipeline in self._pipelines.items():
|
||||||
|
pipeline.preload()
|
||||||
|
entries = [TrainEntry(**entry) for entry in entries_for_locale(locale)]
|
||||||
|
label_count, utterance_count, synonym_count = pipeline.train(entries)
|
||||||
|
logger.info(
|
||||||
|
"tech-step NLP pipeline trained",
|
||||||
|
extra={
|
||||||
|
"locale": locale,
|
||||||
|
"labelCount": label_count,
|
||||||
|
"utteranceCount": utterance_count,
|
||||||
|
"synonymCount": synonym_count,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
logger.info("tech-step NLP pipelines ready", extra={"locales": list(self._pipelines)})
|
||||||
|
|
||||||
def process(self, locale: str, text: str) -> ProcessResult:
|
def process(self, locale: str, text: str) -> ProcessResult:
|
||||||
pipeline = self._pipelines.get(locale)
|
pipeline = self._pipelines.get(locale)
|
||||||
|
|
|
||||||
|
|
@ -1,58 +0,0 @@
|
||||||
"""`POST /v1/train` — appelé par `apps/api` (`IntentServiceClient.train`,
|
|
||||||
`TechStepClassifierService._train`) une fois par locale à chaque warm-up
|
|
||||||
serveur, avec l'intégralité de `TECH_STEP_TRAINING_DATA` filtrée pour cette
|
|
||||||
locale. Voir `LocalePipeline.train` pour ce que "reconstruit à neuf" signifie
|
|
||||||
concrètement.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import logging
|
|
||||||
|
|
||||||
from fastapi import APIRouter, Depends, HTTPException, status
|
|
||||||
|
|
||||||
from ..locale_pipeline import TrainEntry, UnsupportedLocaleError
|
|
||||||
from ..pipeline_registry import registry
|
|
||||||
from ..schemas import TrainRequest, TrainResponse
|
|
||||||
from ..security import require_valid_secret
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
router = APIRouter(dependencies=[Depends(require_valid_secret)])
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/v1/train", response_model=TrainResponse)
|
|
||||||
def train(request: TrainRequest) -> TrainResponse:
|
|
||||||
entries = [
|
|
||||||
TrainEntry(uid=entry.uid, synonyms=entry.synonyms, utterances=entry.utterances)
|
|
||||||
for entry in request.entries
|
|
||||||
]
|
|
||||||
|
|
||||||
logger.info(
|
|
||||||
"tech-step NLP train starting",
|
|
||||||
extra={"locale": request.locale, "uids": [entry.uid for entry in entries]},
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
label_count, utterance_count, synonym_count = registry.train(request.locale, entries)
|
|
||||||
except UnsupportedLocaleError as err:
|
|
||||||
logger.warning("tech-step NLP train rejected", extra={"locale": request.locale, "error": str(err)})
|
|
||||||
# 422, pas 500 : une locale non supportée dans une requête de
|
|
||||||
# `apps/api` est une erreur de configuration/version-skew entre les
|
|
||||||
# deux services (voir le contrat documenté dans le plan de
|
|
||||||
# migration), pas un échec inattendu du service lui-même.
|
|
||||||
raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, detail=str(err)) from err
|
|
||||||
|
|
||||||
logger.info(
|
|
||||||
"tech-step NLP train done",
|
|
||||||
extra={
|
|
||||||
"locale": request.locale,
|
|
||||||
"labelCount": label_count,
|
|
||||||
"utteranceCount": utterance_count,
|
|
||||||
"synonymCount": synonym_count,
|
|
||||||
},
|
|
||||||
)
|
|
||||||
|
|
||||||
return TrainResponse(
|
|
||||||
locale=request.locale,
|
|
||||||
label_count=label_count,
|
|
||||||
utterance_count=utterance_count,
|
|
||||||
synonym_count=synonym_count,
|
|
||||||
)
|
|
||||||
|
|
@ -1,43 +1,13 @@
|
||||||
"""Modèles Pydantic du contrat HTTP — voir le plan de migration pour le
|
"""Modèles Pydantic du contrat HTTP — voir le plan de migration pour le
|
||||||
contrat exact attendu côté `apps/api` (`IntentServiceClient`,
|
contrat exact attendu côté `apps/api` (`IntentServiceClient`,
|
||||||
`apps/api/src/lib/recipe-matching/intent-service-client.ts`).
|
`apps/api/src/lib/recipe-matching/intent-service-client.ts`).
|
||||||
|
|
||||||
|
Pas de `POST /v1/train` ici — ce service s'entraîne lui-même au démarrage
|
||||||
|
depuis `training_data.py` (voir `pipeline_registry.py`/`main.py`), plus
|
||||||
|
besoin d'un contrat HTTP pour ça.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from pydantic import BaseModel, ConfigDict, Field
|
from pydantic import BaseModel
|
||||||
from pydantic.alias_generators import to_camel
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# POST /v1/train
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
class TrainEntryPayload(BaseModel):
|
|
||||||
"""Une technique — mêmes champs qu'une entrée de `TECH_STEP_TRAINING_DATA`
|
|
||||||
(`apps/api/src/lib/recipe-matching/tech-step-training-data.ts`) pour une
|
|
||||||
locale donnée."""
|
|
||||||
|
|
||||||
uid: str
|
|
||||||
synonyms: list[str] = Field(default_factory=list)
|
|
||||||
utterances: list[str] = Field(default_factory=list)
|
|
||||||
|
|
||||||
|
|
||||||
class TrainRequest(BaseModel):
|
|
||||||
locale: str
|
|
||||||
entries: list[TrainEntryPayload]
|
|
||||||
|
|
||||||
|
|
||||||
class TrainResponse(BaseModel):
|
|
||||||
# camelCase en sortie (`labelCount`, pas `label_count`) — cohérent avec
|
|
||||||
# la convention JSON déjà en place côté `apps/api`/`packages/shared`
|
|
||||||
# (voir `TechStepAuditClauseView` etc.), même si le code Python interne
|
|
||||||
# reste en snake_case (convention PEP 8).
|
|
||||||
model_config = ConfigDict(alias_generator=to_camel, populate_by_name=True)
|
|
||||||
|
|
||||||
locale: str
|
|
||||||
label_count: int
|
|
||||||
utterance_count: int
|
|
||||||
synonym_count: int
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# POST /v1/process
|
# POST /v1/process
|
||||||
|
|
|
||||||
1692
services/tech-step-intent-service/intent_service/training_data.py
Normal file
1692
services/tech-step-intent-service/intent_service/training_data.py
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -2,9 +2,30 @@
|
||||||
`INTENT_SERVICE_SECRET` est absent — cette variable doit donc être définie
|
`INTENT_SERVICE_SECRET` est absent — cette variable doit donc être définie
|
||||||
avant le tout premier `import intent_service...` de la session pytest.
|
avant le tout premier `import intent_service...` de la session pytest.
|
||||||
`conftest.py` est chargé par pytest avant la collecte des modules de test,
|
`conftest.py` est chargé par pytest avant la collecte des modules de test,
|
||||||
donc avant que `test_routes_*.py`/`test_security.py` n'importent
|
donc avant que `test_routes_process.py`/`test_security.py` n'importent
|
||||||
`intent_service.main`."""
|
`intent_service.main`.
|
||||||
|
"""
|
||||||
|
|
||||||
import os
|
import os
|
||||||
|
|
||||||
os.environ.setdefault("INTENT_SERVICE_SECRET", "pytest-only-secret-not-used-anywhere-else-32ch")
|
os.environ.setdefault("INTENT_SERVICE_SECRET", "pytest-only-secret-not-used-anywhere-else-32ch")
|
||||||
|
|
||||||
|
import pytest # noqa: E402 — après le `setdefault` ci-dessus, voir le docstring.
|
||||||
|
from fastapi.testclient import TestClient # noqa: E402
|
||||||
|
|
||||||
|
from intent_service.main import app # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="session")
|
||||||
|
def client():
|
||||||
|
"""`TestClient(app)` utilisé comme gestionnaire de contexte déclenche le
|
||||||
|
vrai `lifespan` — puisque `main.py`'s `lifespan` entraîne maintenant
|
||||||
|
l'intégralité du vrai corpus `training_data.TECH_STEP_TRAINING_DATA`
|
||||||
|
(pas un jeu jouet, voir `PipelineRegistry.initialize`), refaire ça une
|
||||||
|
fois par fichier de test (ou pire, une fois par test) multiplierait un
|
||||||
|
entraînement non négligeable sur toute la suite pour rien — scope
|
||||||
|
"session" pour que chaque test ayant besoin d'une vraie app en cours
|
||||||
|
d'exécution partage la même instance déjà entraînée.
|
||||||
|
"""
|
||||||
|
with TestClient(app) as test_client:
|
||||||
|
yield test_client
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,7 @@
|
||||||
"""Vérifie le format des lignes de log produites par
|
"""Vérifie le format des lignes de log produites par
|
||||||
`logging_config._JsonFormatter` — ce que `routes/process.py`/`routes/train.py`
|
`logging_config._JsonFormatter` — ce que `routes/process.py` et
|
||||||
utilisent pour journaliser l'input/l'output de chaque appel NLP."""
|
`pipeline_registry.py` utilisent pour journaliser l'input/l'output de
|
||||||
|
chaque appel NLP et le déroulement de l'entraînement au démarrage."""
|
||||||
|
|
||||||
import json
|
import json
|
||||||
import logging
|
import logging
|
||||||
|
|
|
||||||
|
|
@ -1,52 +1,31 @@
|
||||||
"""Contrat JSON de `POST /v1/process` — voir `schemas.py`/`routes/process.py`."""
|
"""Contrat JSON de `POST /v1/process` — voir `schemas.py`/`routes/process.py`.
|
||||||
|
|
||||||
|
Le service s'entraîne désormais lui-même au démarrage sur le vrai corpus
|
||||||
|
(`training_data.TECH_STEP_TRAINING_DATA`, voir `conftest.py`'s fixture
|
||||||
|
`client` partagée) — ces tests vérifient donc le contrat HTTP contre des
|
||||||
|
phrases réelles du corpus, plus besoin d'un `POST /v1/train` préalable avec
|
||||||
|
des données jouets.
|
||||||
|
"""
|
||||||
|
|
||||||
import pytest
|
|
||||||
from fastapi.testclient import TestClient
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
from intent_service.config import settings
|
from intent_service.config import settings
|
||||||
from intent_service.main import app
|
|
||||||
|
|
||||||
_HEADERS = {"X-Intent-Service-Secret": settings.intent_service_secret}
|
_HEADERS = {"X-Intent-Service-Secret": settings.intent_service_secret}
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
def test_process_against_an_unsupported_locale_returns_empty_result(client: TestClient):
|
||||||
def client():
|
# "de" n'a aucun modèle spaCy connu (`SUPPORTED_LOCALES`) — se comporte
|
||||||
with TestClient(app) as test_client:
|
# comme "jamais entraîné" côté `/v1/process`, jamais une erreur (voir
|
||||||
yield test_client
|
# `PipelineRegistry.process`).
|
||||||
|
|
||||||
|
|
||||||
def test_process_against_an_untrained_locale_returns_empty_result(client: TestClient):
|
|
||||||
response = client.post(
|
response = client.post(
|
||||||
"/v1/process", headers=_HEADERS, json={"locale": "fr", "text": "faire mijoter à feu doux"}
|
"/v1/process", headers=_HEADERS, json={"locale": "de", "text": "faire mijoter à feu doux"}
|
||||||
)
|
)
|
||||||
assert response.status_code == 200
|
assert response.status_code == 200
|
||||||
assert response.json() == {"entities": [], "intent": None, "score": 0.0}
|
assert response.json() == {"entities": [], "intent": None, "score": 0.0}
|
||||||
|
|
||||||
|
|
||||||
def test_process_after_train_returns_entities_and_intent(client: TestClient):
|
def test_process_returns_entities_and_intent_for_a_real_corpus_sentence(client: TestClient):
|
||||||
client.post(
|
|
||||||
"/v1/train",
|
|
||||||
headers=_HEADERS,
|
|
||||||
json={
|
|
||||||
"locale": "fr",
|
|
||||||
"entries": [
|
|
||||||
# `textcat` (exclusive_classes) exige >= 2 labels (voir
|
|
||||||
# LocalePipeline.train) — un second label est nécessaire
|
|
||||||
# même si ce test ne vérifie que celui de "simmer".
|
|
||||||
{
|
|
||||||
"uid": "simmer",
|
|
||||||
"synonyms": ["mijoter"],
|
|
||||||
"utterances": ["faire mijoter à feu doux", "laisser mijoter à couvert"],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"uid": "boil",
|
|
||||||
"synonyms": ["bouillir"],
|
|
||||||
"utterances": ["faire bouillir l'eau", "porter à ébullition"],
|
|
||||||
},
|
|
||||||
],
|
|
||||||
},
|
|
||||||
)
|
|
||||||
|
|
||||||
response = client.post(
|
response = client.post(
|
||||||
"/v1/process", headers=_HEADERS, json={"locale": "fr", "text": "Faire mijoter à feu doux"}
|
"/v1/process", headers=_HEADERS, json={"locale": "fr", "text": "Faire mijoter à feu doux"}
|
||||||
)
|
)
|
||||||
|
|
@ -57,15 +36,16 @@ def test_process_after_train_returns_entities_and_intent(client: TestClient):
|
||||||
assert [entity["uid"] for entity in body["entities"]] == ["simmer"]
|
assert [entity["uid"] for entity in body["entities"]] == ["simmer"]
|
||||||
|
|
||||||
|
|
||||||
def test_process_with_blank_text_returns_empty_result(client: TestClient):
|
def test_process_matches_english_text_against_the_english_trained_vocabulary(client: TestClient):
|
||||||
client.post(
|
response = client.post(
|
||||||
"/v1/train",
|
"/v1/process", headers=_HEADERS, json={"locale": "en", "text": "Chop the onions finely"}
|
||||||
headers=_HEADERS,
|
|
||||||
json={
|
|
||||||
"locale": "en",
|
|
||||||
"entries": [{"uid": "boil", "synonyms": ["boil"], "utterances": ["bring to the boil"]}],
|
|
||||||
},
|
|
||||||
)
|
)
|
||||||
response = client.post("/v1/process", headers=_HEADERS, json={"locale": "en", "text": " "})
|
assert response.status_code == 200
|
||||||
|
body = response.json()
|
||||||
|
assert [entity["uid"] for entity in body["entities"]] == ["chop"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_process_with_blank_text_returns_empty_result(client: TestClient):
|
||||||
|
response = client.post("/v1/process", headers=_HEADERS, json={"locale": "fr", "text": " "})
|
||||||
assert response.status_code == 200
|
assert response.status_code == 200
|
||||||
assert response.json() == {"entities": [], "intent": None, "score": 0.0}
|
assert response.json() == {"entities": [], "intent": None, "score": 0.0}
|
||||||
|
|
|
||||||
|
|
@ -1,47 +0,0 @@
|
||||||
"""Contrat JSON de `POST /v1/train` — voir `schemas.py`/`routes/train.py`."""
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from fastapi.testclient import TestClient
|
|
||||||
|
|
||||||
from intent_service.config import settings
|
|
||||||
from intent_service.main import app
|
|
||||||
|
|
||||||
_HEADERS = {"X-Intent-Service-Secret": settings.intent_service_secret}
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def client():
|
|
||||||
with TestClient(app) as test_client:
|
|
||||||
yield test_client
|
|
||||||
|
|
||||||
|
|
||||||
def test_train_returns_counts(client: TestClient):
|
|
||||||
response = client.post(
|
|
||||||
"/v1/train",
|
|
||||||
headers=_HEADERS,
|
|
||||||
json={
|
|
||||||
"locale": "fr",
|
|
||||||
"entries": [
|
|
||||||
{"uid": "melt", "synonyms": ["faire fondre"], "utterances": ["faire fondre le beurre"]},
|
|
||||||
{"uid": "boil", "synonyms": ["bouillir"], "utterances": ["faire bouillir l'eau"]},
|
|
||||||
],
|
|
||||||
},
|
|
||||||
)
|
|
||||||
assert response.status_code == 200
|
|
||||||
body = response.json()
|
|
||||||
assert body == {"locale": "fr", "labelCount": 2, "utteranceCount": 2, "synonymCount": 2}
|
|
||||||
|
|
||||||
|
|
||||||
def test_train_with_unsupported_locale_returns_422(client: TestClient):
|
|
||||||
response = client.post(
|
|
||||||
"/v1/train",
|
|
||||||
headers=_HEADERS,
|
|
||||||
json={"locale": "de", "entries": [{"uid": "melt", "synonyms": [], "utterances": []}]},
|
|
||||||
)
|
|
||||||
assert response.status_code == 422
|
|
||||||
|
|
||||||
|
|
||||||
def test_train_with_empty_entries_returns_zero_counts(client: TestClient):
|
|
||||||
response = client.post("/v1/train", headers=_HEADERS, json={"locale": "en", "entries": []})
|
|
||||||
assert response.status_code == 200
|
|
||||||
assert response.json() == {"locale": "en", "labelCount": 0, "utteranceCount": 0, "synonymCount": 0}
|
|
||||||
|
|
@ -1,17 +1,12 @@
|
||||||
"""`require_valid_secret` — miroir inversé de
|
"""`require_valid_secret` — miroir inversé de
|
||||||
`require-internal-worker.test.ts` côté `apps/api`."""
|
`require-internal-worker.test.ts` côté `apps/api`. Utilise la fixture
|
||||||
|
`client` partagée (`conftest.py`) — pas besoin d'une app entraînée
|
||||||
|
séparément juste pour tester l'authentification.
|
||||||
|
"""
|
||||||
|
|
||||||
import pytest
|
|
||||||
from fastapi.testclient import TestClient
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
from intent_service.config import settings
|
from intent_service.config import settings
|
||||||
from intent_service.main import app
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def client():
|
|
||||||
with TestClient(app) as test_client:
|
|
||||||
yield test_client
|
|
||||||
|
|
||||||
|
|
||||||
def test_rejects_a_missing_secret(client: TestClient):
|
def test_rejects_a_missing_secret(client: TestClient):
|
||||||
|
|
|
||||||
|
|
@ -479,10 +479,13 @@ appelé en HTTP par `TechStepClassifierService` via `IntentServiceClient`
|
||||||
(`intent-service-client.ts`) — `node-nlp` était peu maintenu et tournait
|
(`intent-service-client.ts`) — `node-nlp` était peu maintenu et tournait
|
||||||
in-process dans l'event loop Node ; spaCy offre un écosystème NLP plus
|
in-process dans l'event loop Node ; spaCy offre un écosystème NLP plus
|
||||||
robuste, dans un processus séparé, avec l'ambition à terme de pouvoir aussi
|
robuste, dans un processus séparé, avec l'ambition à terme de pouvoir aussi
|
||||||
absorber ce que fait `services/tech-step-llm-worker`. `TECH_STEP_TRAINING_DATA`
|
absorber ce que fait `services/tech-step-llm-worker`. Ce service est
|
||||||
reste possédé par `apps/api` (revu par PR comme le reste du code) et poussé
|
entièrement autonome : `TECH_STEP_TRAINING_DATA` (~74 techniques) vit
|
||||||
intégralement à ce service via `POST /v1/train` à chaque warm-up — ce service
|
désormais dans son propre `training_data.py`, revu par PR comme le reste du
|
||||||
ne touche jamais Postgres lui-même (voir son propre README).
|
code mais plus poussé par `apps/api` via HTTP — le service s'entraîne
|
||||||
|
lui-même une seule fois, à son propre démarrage, et ne touche jamais
|
||||||
|
Postgres (voir son propre README, y compris pour le temps de démarrage —
|
||||||
|
plusieurs minutes, l'entraînement n'étant jamais persisté sur disque).
|
||||||
|
|
||||||
`normalizeText` (décomposition NFD + suppression des diacritiques + minuscule)
|
`normalizeText` (décomposition NFD + suppression des diacritiques + minuscule)
|
||||||
reste utilisée par `ingredient-matcher.ts`, mais n'intervient plus dans la
|
reste utilisée par `ingredient-matcher.ts`, mais n'intervient plus dans la
|
||||||
|
|
@ -516,12 +519,15 @@ normalisation du pipeline spaCy côté service.
|
||||||
un match clairement ancré sur un mot-clé juste parce que le modèle n'est
|
un match clairement ancré sur un mot-clé juste parce que le modèle n'est
|
||||||
pas assez confiant.
|
pas assez confiant.
|
||||||
|
|
||||||
Entraînement (`_train`) et résolution `TechStep.key -> id` sont mémoïsés une
|
Résolution `TechStep.key -> id` mémoïsée une seule fois sur le singleton
|
||||||
seule fois sur le singleton partagé `techStepClassifier` (jamais par requête).
|
partagé `techStepClassifier` (jamais par requête) — c'est tout ce
|
||||||
`server.ts` appelle `techStepClassifier.warmUp()` avant d'accepter du trafic,
|
qu'`apps/api` a encore à mémoïser, l'entraînement du modèle lui-même vivant
|
||||||
avec retry/backoff si `services/tech-step-intent-service` n'est pas encore
|
entièrement côté `services/tech-step-intent-service`. `server.ts` appelle
|
||||||
prêt (le cas normal en Docker Compose, où `app` attend qu'il soit `healthy`
|
`techStepClassifier.warmUp()` avant d'accepter du trafic, avec retry/backoff
|
||||||
avant même de démarrer — voir `docker-compose.yml`).
|
si `services/tech-step-intent-service` n'est pas encore joignable (le cas
|
||||||
|
normal en Docker Compose, où `app` attend qu'il soit `healthy` avant même de
|
||||||
|
démarrer — voir `docker-compose.yml`, et le README de ce service pour
|
||||||
|
combien de temps ça prend).
|
||||||
|
|
||||||
**Pièges rencontrés en construisant ce pipeline**, tous corrigés dans le code
|
**Pièges rencontrés en construisant ce pipeline**, tous corrigés dans le code
|
||||||
(pas juste contournés) :
|
(pas juste contournés) :
|
||||||
|
|
@ -538,10 +544,10 @@ avant même de démarrer — voir `docker-compose.yml`).
|
||||||
défaut — persistait le modèle entraîné dans un fichier `model.nlp` (cwd du
|
défaut — persistait le modèle entraîné dans un fichier `model.nlp` (cwd du
|
||||||
process) et le rechargeait *au lieu de* ré-entraîner au prochain démarrage
|
process) et le rechargeait *au lieu de* ré-entraîner au prochain démarrage
|
||||||
s'il existait déjà. Un modèle obsolète sur disque aurait masqué
|
s'il existait déjà. Un modèle obsolète sur disque aurait masqué
|
||||||
silencieusement toute mise à jour de `TECH_STEP_TRAINING_DATA`. Non
|
silencieusement toute mise à jour du corpus. Non applicable au service
|
||||||
applicable au service Python actuel : `POST /v1/train` reconstruit tout en
|
Python actuel : il réentraîne tout en mémoire à chaque démarrage du
|
||||||
mémoire à chaque appel, sans jamais rien persister sur disque (voir ce
|
process, sans jamais rien persister sur disque (voir ce service's own
|
||||||
service's own README).
|
README).
|
||||||
- `nlp.make_doc()` (spaCy) ne fait tourner que le tokenizer, pas les
|
- `nlp.make_doc()` (spaCy) ne fait tourner que le tokenizer, pas les
|
||||||
composants du pipeline — un piège trouvé en construisant le `PhraseMatcher`
|
composants du pipeline — un piège trouvé en construisant le `PhraseMatcher`
|
||||||
du nouveau service : les patterns de synonymes doivent explicitement
|
du nouveau service : les patterns de synonymes doivent explicitement
|
||||||
|
|
|
||||||
|
|
@ -353,9 +353,9 @@ fiable.
|
||||||
`tech_step` (`TechStep`, `key` unique, ex. `"simmer"`) est le catalogue des
|
`tech_step` (`TechStep`, `key` unique, ex. `"simmer"`) est le catalogue des
|
||||||
techniques (mijoter, préchauffer…) — juste un id/clé stable référencé par
|
techniques (mijoter, préchauffer…) — juste un id/clé stable référencé par
|
||||||
`step_tech_step`. Les données de détection elles-mêmes (synonymes + phrases
|
`step_tech_step`. Les données de détection elles-mêmes (synonymes + phrases
|
||||||
d'exemple par langue, entraînant le microservice spaCy
|
d'exemple par langue) vivent en code dans le microservice spaCy lui-même
|
||||||
`services/tech-step-intent-service`) vivent en code
|
(`services/tech-step-intent-service/intent_service/training_data.py`), pas
|
||||||
(`tech-step-training-data.ts`), pas dans une table — l'ancienne
|
dans une table ni côté `apps/api` — l'ancienne
|
||||||
`tech_step_mapping` (`TechStepMapping`, une regex par technique/locale) a
|
`tech_step_mapping` (`TechStepMapping`, une regex par technique/locale) a
|
||||||
été supprimée une fois constaté que les regex ne généralisaient jamais
|
été supprimée une fois constaté que les regex ne généralisaient jamais
|
||||||
au-delà de leur propre vocabulaire — voir
|
au-delà de leur propre vocabulaire — voir
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue