Compare commits
10 commits
main
...
feat/tech-
| Author | SHA1 | Date | |
|---|---|---|---|
| f2fd3b961a | |||
| 19e0507852 | |||
| c6b6b790c8 | |||
| 065ef2a31a | |||
| 74a0052431 | |||
| 1bf97ce6de | |||
| 9590569f3f | |||
| 6128400414 | |||
| 690125bec3 | |||
| 18abae7b6a |
58 changed files with 5592 additions and 1908 deletions
|
|
@ -22,6 +22,14 @@ JWT_SECRET=changeme-generate-a-real-random-secret-at-least-32-chars
|
|||
# browser, so login "succeeds" but every subsequent request 401s.
|
||||
# COOKIE_SECURE=false
|
||||
|
||||
# Required — secret shared between "app" and "tech-step-intent-service"
|
||||
# (docker-compose.yml, apps/api/src/config/env.ts). Unlike
|
||||
# INTERNAL_WORKER_SECRET below, there's no "leave it unset" escape hatch:
|
||||
# tech-step-intent-service is a core dependency, not an optional background
|
||||
# job — without it, no recipe step can have its techniques detected at all.
|
||||
# Generate your own the same way as JWT_SECRET above.
|
||||
INTENT_SERVICE_SECRET=changeme-generate-a-real-random-secret-at-least-32-chars
|
||||
|
||||
# Only needed to run the optional `tech-step-llm-worker` service — shared
|
||||
# between it and "app" (docker-compose.yml). Generate your own the same
|
||||
# way as JWT_SECRET above; leave both this and the service commented
|
||||
|
|
|
|||
61
.github/workflows/ci.yml
vendored
61
.github/workflows/ci.yml
vendored
|
|
@ -19,9 +19,15 @@ env:
|
|||
# exercise the success path (matching secret), not just the "unset"
|
||||
# rejection every environment that doesn't set this gets by default.
|
||||
INTERNAL_WORKER_SECRET: "ci-only-worker-secret-not-used-anywhere-else-32chars+"
|
||||
# Shared between the `test` job's own uvicorn step (below) and apps/api's
|
||||
# IntentServiceClient — see the `test` job for why this can't be a
|
||||
# `services:` container like postgres above (GitHub Actions can only pull
|
||||
# a published image, not build services/tech-step-intent-service/Dockerfile).
|
||||
INTENT_SERVICE_BASE_URL: "http://localhost:8000"
|
||||
INTENT_SERVICE_SECRET: "ci-only-intent-secret-not-used-anywhere-else-32chars+"
|
||||
|
||||
jobs:
|
||||
# Four independent jobs, no needs: between them — each starts in parallel
|
||||
# Five independent jobs, no needs: between them — each starts in parallel
|
||||
# and reports as its own check, instead of the previous single chained
|
||||
# "lint-and-test then e2e" pipeline.
|
||||
lint:
|
||||
|
|
@ -65,10 +71,63 @@ jobs:
|
|||
node-version: 22
|
||||
cache: pnpm
|
||||
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.12"
|
||||
- uses: astral-sh/setup-uv@v3
|
||||
with:
|
||||
enable-cache: true
|
||||
|
||||
# `services:` (like the `postgres` container above) can only pull an
|
||||
# already-published image — it can't build
|
||||
# services/tech-step-intent-service/Dockerfile from this checkout.
|
||||
# Running `uvicorn` as a plain background step instead: it keeps
|
||||
# running for the rest of this job (GitHub Actions steps in one job
|
||||
# share the same runner process tree), and `pnpm --filter api test`
|
||||
# below needs a real instance to talk to per this repo's "never mock
|
||||
# an internal service" test convention — same reasoning as the real
|
||||
# `postgres` container just above, not a mock HTTP server.
|
||||
- name: Install services/tech-step-intent-service
|
||||
working-directory: services/tech-step-intent-service
|
||||
run: uv sync --frozen
|
||||
- name: Start services/tech-step-intent-service in the background
|
||||
working-directory: services/tech-step-intent-service
|
||||
run: |
|
||||
uv run uvicorn intent_service.main:app --host 0.0.0.0 --port 8000 &
|
||||
# `/health` only returns 200 once this service has finished
|
||||
# training itself from scratch (no model ever persisted to disk —
|
||||
# see its own README) — measured at ~335s per locale (~670s for
|
||||
# fr+en combined) against the current ~74-technique corpus,
|
||||
# trained on each technique's own synonyms in addition to its
|
||||
# example phrases, so this wait is generous rather than the fast
|
||||
# "base models only" check it used to be before that service
|
||||
# trained itself at startup (see docker-compose.yml's healthcheck
|
||||
# for the same reasoning).
|
||||
timeout 900 bash -c 'until curl -sf http://localhost:8000/health > /dev/null; do sleep 2; done'
|
||||
|
||||
- run: pnpm install --frozen-lockfile
|
||||
- run: pnpm --filter api exec prisma migrate deploy
|
||||
- run: pnpm --filter api test
|
||||
|
||||
intent-service-test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.12"
|
||||
- uses: astral-sh/setup-uv@v3
|
||||
with:
|
||||
enable-cache: true
|
||||
|
||||
- name: Install services/tech-step-intent-service
|
||||
working-directory: services/tech-step-intent-service
|
||||
run: uv sync --frozen
|
||||
- name: Run pytest
|
||||
working-directory: services/tech-step-intent-service
|
||||
run: uv run pytest -q
|
||||
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
|
|
|
|||
11
.gitignore
vendored
11
.gitignore
vendored
|
|
@ -71,11 +71,12 @@ web_modules/
|
|||
!.env.example
|
||||
!.env.test.example
|
||||
|
||||
# node-nlp's default auto-save file (apps/api/src/lib/recipe-matching/
|
||||
# tech-step-matcher.ts explicitly disables autoSave/autoLoad, but this is a
|
||||
# belt-and-suspenders guard against it ever reappearing — a stale trained
|
||||
# model on disk must never silently shadow TECH_STEP_TRAINING_DATA).
|
||||
model.nlp
|
||||
# Python virtualenvs/caches for services/tech-step-intent-service (this repo
|
||||
# is otherwise all-Node — see that service's own .gitignore for the rest;
|
||||
# duplicated here too since some tooling only honors the repo-root file).
|
||||
services/tech-step-intent-service/.venv/
|
||||
services/tech-step-intent-service/__pycache__/
|
||||
services/tech-step-intent-service/.pytest_cache/
|
||||
|
||||
# parcel-bundler cache (https://parceljs.org/)
|
||||
.cache
|
||||
|
|
|
|||
32
README.md
32
README.md
|
|
@ -44,6 +44,8 @@ runtime Node pur (Docker, pas de transpilation à la volée), voir la note dans
|
|||
- Node.js 22 (voir `.nvmrc`)
|
||||
- pnpm 10 (`corepack enable` puis `corepack use pnpm@10.12.4`, ou installation manuelle)
|
||||
- Docker (pour Postgres en local)
|
||||
- Python 3.12+ et [`uv`](https://docs.astral.sh/uv/) (pour
|
||||
`services/tech-step-intent-service` en dev natif — requis, voir plus bas)
|
||||
|
||||
## Installation
|
||||
|
||||
|
|
@ -100,6 +102,17 @@ pnpm --filter api exec prisma migrate dev
|
|||
# techniques...) — automatique après `prisma migrate reset`, sinon à la main :
|
||||
pnpm --filter api prisma:seed
|
||||
|
||||
# Microservice de détection des techniques (spaCy) — requis, `pnpm dev:api`
|
||||
# ne peut plus détecter aucune technique de cuisine sans lui. Lance-le en
|
||||
# premier et laisse-le tourner : il s'entraîne lui-même à chaque démarrage
|
||||
# (~11 minutes pour le corpus actuel, voir son propre README) avant de
|
||||
# répondre quoi que ce soit sur /health.
|
||||
cd services/tech-step-intent-service
|
||||
uv sync
|
||||
cp .env.example .env # édite-le : même INTENT_SERVICE_SECRET que apps/api/.env
|
||||
uv run uvicorn intent_service.main:app --reload --port 8000
|
||||
cd ../..
|
||||
|
||||
# Backend (http://localhost:3000)
|
||||
pnpm dev:api
|
||||
|
||||
|
|
@ -137,7 +150,7 @@ pnpm --filter web cy:run:component # tests de composant UI isolés (Cypress com
|
|||
pnpm build # build de tous les workspaces
|
||||
```
|
||||
|
||||
La CI GitHub Actions (`.github/workflows/ci.yml`) exécute quatre jobs indépendants (`lint`, `test`, `build`, `e2e` — ce dernier lance aussi `cy:run:component`) en parallèle, sur chaque push (toutes branches) et sur chaque PR vers `main` — pas de chaînage entre eux, chacun apparaît comme son propre check. Voir aussi [Déploiement](#déploiement) pour le pipeline de release (`.github/workflows/release.yml`).
|
||||
La CI GitHub Actions (`.github/workflows/ci.yml`) exécute cinq jobs indépendants (`lint`, `test`, `intent-service-test`, `build`, `e2e` — ce dernier lance aussi `cy:run:component`) en parallèle, sur chaque push (toutes branches) et sur chaque PR vers `main` — pas de chaînage entre eux, chacun apparaît comme son propre check. `test` démarre `services/tech-step-intent-service` en arrière-plan (voir ce fichier) puisque la suite Mocha ne mocke jamais un service interne. Voir aussi [Déploiement](#déploiement) pour le pipeline de release (`.github/workflows/release.yml`).
|
||||
|
||||
### Base de test isolée de la base de dev (`apps/api`)
|
||||
|
||||
|
|
@ -158,6 +171,13 @@ Un garde-fou (`assertRunningAgainstTestDatabase()`) refuse d'exécuter
|
|||
`resetDatabase()` si `DATABASE_URL` ne contient ni `"test"` ni `"ci"` — la
|
||||
seule base qu'il doit rejeter est ta vraie base de dev.
|
||||
|
||||
`services/tech-step-intent-service` doit aussi tourner en local avant
|
||||
`pnpm --filter api test` — les tests touchant `tech-step-matcher.ts` passent
|
||||
par le vrai service (jamais un mock, voir
|
||||
[specs/dev-conventions.md](specs/dev-conventions.md)) et échouent avec une
|
||||
erreur de connexion, pas une assertion utile, s'il n'est pas démarré. Voir la
|
||||
section [Développement](#développement) ci-dessus.
|
||||
|
||||
## Déploiement
|
||||
|
||||
Une seule image Docker (`apps/api/Dockerfile`) sert à la fois l'API et le frontend
|
||||
|
|
@ -180,10 +200,12 @@ synchronisation de la table `sources` depuis le registre d'adaptateurs de code
|
|||
puis `node dist/server.js`. Les trois étapes sont sûres/idempotentes à
|
||||
répéter à chaque redémarrage du conteneur.
|
||||
|
||||
`docker-compose.yml` ne définit donc que deux services : `postgres` et `app` (un
|
||||
seul port, `APP_PORT`, défaut `3000` — plus de `WEB_PORT`/`CORS_ORIGIN` à
|
||||
coordonner entre deux origines, le frontend et l'API sont désormais servis depuis
|
||||
la même origine).
|
||||
Le duo `postgres`/`app` de `docker-compose.yml` n'expose donc qu'un seul port
|
||||
applicatif, `APP_PORT` (défaut `3000`) — plus de `WEB_PORT`/`CORS_ORIGIN` à
|
||||
coordonner entre deux origines, le frontend et l'API sont désormais servis
|
||||
depuis la même origine. Les deux autres services du fichier
|
||||
(`tech-step-intent-service`, `tech-step-llm-worker`) n'exposent eux aucun port
|
||||
au host — voir leurs propres README pour leur rôle.
|
||||
|
||||
**Pas de registre d'image** dans cette configuration : l'instance **Portainer** de
|
||||
production est reliée directement au dépôt Git et reconstruit elle-même
|
||||
|
|
|
|||
|
|
@ -12,6 +12,13 @@ JWT_SECRET=changeme-generate-a-real-random-secret-at-least-32-chars
|
|||
# JWT_EXPIRES_IN=7d
|
||||
# AUTH_COOKIE_NAME=session
|
||||
# CORS_ORIGIN=http://localhost:5173
|
||||
# INTENT_SERVICE_BASE_URL=http://localhost:8000
|
||||
|
||||
# Required — services/tech-step-intent-service must be running locally (see
|
||||
# that service's own README) for any recipe save/preview to detect
|
||||
# techniques at all. Must match that service's own INTENT_SERVICE_SECRET.
|
||||
# Generate your own the same way as JWT_SECRET above.
|
||||
INTENT_SERVICE_SECRET=changeme-generate-a-real-random-secret-at-least-32-chars
|
||||
|
||||
# Only needed if you're running services/tech-step-llm-worker locally —
|
||||
# every /internal/tech-steps/* request is rejected outright while unset.
|
||||
|
|
|
|||
|
|
@ -14,6 +14,15 @@ DATABASE_URL="postgresql://changeme:changeme@localhost:5432/batchcooking_test?sc
|
|||
# node -e "console.log(require('crypto').randomBytes(48).toString('hex'))"
|
||||
JWT_SECRET=changeme-generate-a-real-random-secret-at-least-32-chars
|
||||
|
||||
# Required — the Mocha suite exercises the real techStepClassifier, which
|
||||
# now round-trips over HTTP to services/tech-step-intent-service (no mocks
|
||||
# of internal services, per this repo's test conventions). Start that
|
||||
# service locally first (see its own README) with a matching
|
||||
# INTENT_SERVICE_SECRET, or every test touching tech-step-matcher.ts fails
|
||||
# with a connection error rather than a useful assertion failure.
|
||||
INTENT_SERVICE_BASE_URL=http://localhost:8000
|
||||
INTENT_SERVICE_SECRET=changeme-generate-a-real-random-secret-at-least-32-chars
|
||||
|
||||
# Optional — only needed to exercise tech-step-worker.routes.test.ts's
|
||||
# success path (a request with a matching secret); every other test runs
|
||||
# fine without it. Any value at least 32 chars works locally.
|
||||
|
|
|
|||
|
|
@ -2,5 +2,6 @@
|
|||
"extension": ["ts"],
|
||||
"spec": "test/**/*.test.ts",
|
||||
"node-option": ["import=tsx"],
|
||||
"timeout": 10000
|
||||
"timeout": 10000,
|
||||
"require": ["test-support/mocha-root-hooks.ts"]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -26,7 +26,6 @@
|
|||
"dotenv": "^16.4.5",
|
||||
"express": "^4.21.1",
|
||||
"jsonwebtoken": "^9.0.3",
|
||||
"node-nlp": "4.27.0",
|
||||
"prisma": "^5.22.0",
|
||||
"zod": "^3.23.8"
|
||||
},
|
||||
|
|
|
|||
|
|
@ -627,13 +627,13 @@ model RecipeIngredient {
|
|||
/// Matching a step's free text against these (`tech-step-matcher.ts`'s
|
||||
/// `TechStepClassifierService`) used to go through a DB-backed
|
||||
/// `TechStepMapping` table of per-locale regex expressions — replaced with
|
||||
/// a node-nlp model trained from in-code data
|
||||
/// (`tech-step-training-data.ts`) once regexes turned out unable to
|
||||
/// generalize past their own literal vocabulary. Nothing queries/edits
|
||||
/// that matching data at runtime anymore (it only ever feeds the
|
||||
/// classifier's one-time training pass), so it no longer needs a table of
|
||||
/// its own — this row now only exists to be a stable id/key other tables
|
||||
/// (`StepTechStep`) reference.
|
||||
/// a spaCy-based model (`services/tech-step-intent-service`) trained from
|
||||
/// in-code data (`tech-step-training-data.ts`) once regexes turned out
|
||||
/// unable to generalize past their own literal vocabulary. Nothing
|
||||
/// queries/edits that matching data at runtime anymore (it only ever feeds
|
||||
/// that service's one-time training pass), so it no longer needs a table
|
||||
/// of its own — this row now only exists to be a stable id/key other
|
||||
/// tables (`StepTechStep`) reference.
|
||||
model TechStep {
|
||||
id Int @id @default(autoincrement())
|
||||
key String @unique
|
||||
|
|
|
|||
|
|
@ -71,6 +71,25 @@ const envSchema = z.object({
|
|||
* fails closed rather than open if a real deployment forgets to set it.
|
||||
*/
|
||||
INTERNAL_WORKER_SECRET: z.string().min(32).optional(),
|
||||
/**
|
||||
* Base URL of `services/tech-step-intent-service` (the spaCy-based
|
||||
* microservice `TechStepClassifierService` delegates NER + intent
|
||||
* classification to, see `lib/recipe-matching/intent-service-client.ts`).
|
||||
* Has a default (unlike `DATABASE_URL`/secrets below) since it isn't
|
||||
* secret and dev natively runs it on a fixed local port — Docker Compose
|
||||
* overrides it to the compose network's service name.
|
||||
*/
|
||||
INTENT_SERVICE_BASE_URL: z.string().url().default("http://localhost:8000"),
|
||||
/**
|
||||
* Shared secret sent as an `X-Intent-Service-Secret` header on every call
|
||||
* to `services/tech-step-intent-service`. Unlike `INTERNAL_WORKER_SECRET`
|
||||
* above, **required, no `.optional()`** — that service is a core
|
||||
* dependency (recipe save/preview can no longer detect any technique
|
||||
* without it), not an optional background job; an environment that
|
||||
* forgets to set this must fail loudly at startup, not silently run with
|
||||
* every technique detection request failing one at a time.
|
||||
*/
|
||||
INTENT_SERVICE_SECRET: z.string().min(32, "INTENT_SERVICE_SECRET must be at least 32 characters"),
|
||||
});
|
||||
|
||||
/** Parsed, validated environment — import this instead of reading `process.env` directly anywhere else. */
|
||||
|
|
|
|||
|
|
@ -62,11 +62,12 @@ export const UNITS: Array<{ uid: string; type: UnitType; toBaseFactor: number }>
|
|||
//
|
||||
// Just a flat list of stable ids here — the actual matching data (per-
|
||||
// locale synonym lists + example phrasings the classifier trains on) lives
|
||||
// in `lib/recipe-matching/tech-step-training-data.ts`'s
|
||||
// `TECH_STEP_TRAINING_DATA`, not here: unlike this list, it's read by
|
||||
// `TechStepClassifierService`'s training pass, not the seed script, so it
|
||||
// doesn't belong alongside the rest of this file's DB-seeded reference
|
||||
// data. Every entry here must have a matching entry there.
|
||||
// in `services/tech-step-intent-service/intent_service/training_data.py`'s
|
||||
// `TECH_STEP_TRAINING_DATA`, not here: it's owned and trained entirely by
|
||||
// that separate Python service (see its own README), not read by this
|
||||
// seed script at all, so it doesn't belong alongside the rest of this
|
||||
// file's DB-seeded reference data. Every entry here must have a matching
|
||||
// entry there.
|
||||
export const TECH_STEPS: string[] = [
|
||||
"cook",
|
||||
"fry",
|
||||
|
|
@ -94,6 +95,57 @@ export const TECH_STEPS: string[] = [
|
|||
"bake",
|
||||
"plate",
|
||||
"coat",
|
||||
// Lexique de techniques ajouté par la suite — voir
|
||||
// `services/tech-step-intent-service/intent_service/training_data.py`
|
||||
// pour les synonymes/phrases d'exemple de chacune.
|
||||
"baste",
|
||||
"appertize",
|
||||
"whiskPale",
|
||||
"goldenBrown",
|
||||
"braise",
|
||||
"truss",
|
||||
"caramelize",
|
||||
"score",
|
||||
"lineMold",
|
||||
"clarify",
|
||||
"compote",
|
||||
"concasse",
|
||||
"confit",
|
||||
"julienne",
|
||||
"brunoise",
|
||||
"mirepoix",
|
||||
"paysanne",
|
||||
"blindBake",
|
||||
"bainMarie",
|
||||
"smother",
|
||||
"decant",
|
||||
"dilute",
|
||||
"punchDown",
|
||||
"disgorge",
|
||||
"loosen",
|
||||
"shellEgg",
|
||||
"scald",
|
||||
"pod",
|
||||
"emulsify",
|
||||
"hollowOut",
|
||||
"shock",
|
||||
"setGel",
|
||||
"glaze",
|
||||
"thicken",
|
||||
"filet",
|
||||
"proof",
|
||||
"peelBlanch",
|
||||
"whipUp",
|
||||
"moisten",
|
||||
"pasteurize",
|
||||
"poach",
|
||||
"reduce",
|
||||
"rubIn",
|
||||
"dustWithFlour",
|
||||
"sweat",
|
||||
"sift",
|
||||
"toast",
|
||||
"zest",
|
||||
];
|
||||
|
||||
// The 14 allergens EU Regulation 1169/2011 (Annex II) requires food
|
||||
|
|
|
|||
101
apps/api/src/lib/recipe-matching/intent-service-client.ts
Normal file
101
apps/api/src/lib/recipe-matching/intent-service-client.ts
Normal file
|
|
@ -0,0 +1,101 @@
|
|||
import { env } from "../../config/env.js";
|
||||
|
||||
/**
|
||||
* Thin fetch wrapper around `services/tech-step-intent-service`'s HTTP
|
||||
* contract (`POST /v1/process`) — the microservice
|
||||
* {@link TechStepClassifierService} (`tech-step-matcher.ts`) delegates NER +
|
||||
* intent classification to, in place of the `node-nlp` `NlpManager` it used
|
||||
* to own directly. See that service's own README for the full contract and
|
||||
* why it never touches Postgres itself — it also owns its own training
|
||||
* corpus now (`training_data.py`), trained once at its own startup, so
|
||||
* `apps/api` never pushes anything to it; `process()` below is this
|
||||
* client's only method.
|
||||
*
|
||||
* Authenticated with `INTENT_SERVICE_SECRET` — the inverse direction of
|
||||
* `requireInternalWorker`'s `INTERNAL_WORKER_SECRET` (this time `apps/api`
|
||||
* is the caller, not the callee), but the same "one flat shared secret"
|
||||
* shape.
|
||||
*/
|
||||
|
||||
/** One candidate technique mention the service's `PhraseMatcher` found — offsets `[start, end)`, same convention as `String.prototype.slice`. Mirrors `EntityPayload` (Python `schemas.py`). */
|
||||
export interface IntentServiceEntity {
|
||||
uid: string;
|
||||
start: number;
|
||||
end: number;
|
||||
}
|
||||
|
||||
/** The full result of a `POST /v1/process` call — mirrors `ProcessResponse` (Python `schemas.py`). `intent` is `null` only when `locale` isn't one this service trains for, or `text` is blank; otherwise always a real `uid` (the Python service's `textcat` has no "None" sentinel, unlike node-nlp — see that service's README). */
|
||||
export interface IntentServiceProcessResult {
|
||||
entities: IntentServiceEntity[];
|
||||
intent: string | null;
|
||||
score: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Client for `services/tech-step-intent-service` — a real class (not a
|
||||
* plain object of functions) per this repo's service-style-logic
|
||||
* convention, even though it holds no state of its own: it's used as the
|
||||
* one shared {@link intentServiceClient} singleton below, same reasoning as
|
||||
* `TechStepClassifierService` itself.
|
||||
*/
|
||||
export class IntentServiceClient {
|
||||
/**
|
||||
* Performs a JSON request against the intent service and returns the
|
||||
* parsed body.
|
||||
*
|
||||
* @throws {Error} if the response status is not in the 2xx range, or the
|
||||
* request itself fails (network error, service down) — left as a plain
|
||||
* `Error` rather than a typed `HttpError`: this is an internal
|
||||
* service-to-service call, not a request `apps/api`'s own HTTP layer
|
||||
* needs to map to a client-facing status code (see
|
||||
* `TechStepClassifierService.warmUp`'s retry in `server.ts` for how a
|
||||
* failure here is actually handled).
|
||||
*/
|
||||
private async _request<TResponseBody>(
|
||||
path: string,
|
||||
init: RequestInit = {},
|
||||
): Promise<TResponseBody> {
|
||||
try {
|
||||
const response = await fetch(`${env.INTENT_SERVICE_BASE_URL}${path}`, {
|
||||
...init,
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
"X-Intent-Service-Secret": env.INTENT_SERVICE_SECRET,
|
||||
...init.headers,
|
||||
},
|
||||
});
|
||||
if (!response.ok) {
|
||||
const body = await response.text().catch(() => "");
|
||||
throw new Error(`${init.method ?? "GET"} ${path} failed: ${response.status} ${body}`);
|
||||
}
|
||||
return (await response.json()) as TResponseBody;
|
||||
} catch (err) {
|
||||
// Rethrown as-is — every caller (`TechStepClassifierService`) already
|
||||
// wraps its own `await`s per the repo's try/catch convention; this is
|
||||
// just where the `await` itself has to sit inside one.
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Equivalent to the old `NlpManager.process(locale, text)` — returns every
|
||||
* candidate technique mention (NER) plus the intent classifier's verdict
|
||||
* for `text` as a whole, whether `text` is a full step description or a
|
||||
* single clause `TechStepClassifierService` already cut out of one (this
|
||||
* service doesn't know or care which, exactly like `NlpManager` before
|
||||
* it).
|
||||
*/
|
||||
public async process(locale: string, text: string): Promise<IntentServiceProcessResult> {
|
||||
try {
|
||||
return await this._request("/v1/process", {
|
||||
method: "POST",
|
||||
body: JSON.stringify({ locale, text }),
|
||||
});
|
||||
} catch (err) {
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Single shared instance — stateless, no reason for more than one (same reasoning as `techStepClassifier`/`prisma`). */
|
||||
export const intentServiceClient = new IntentServiceClient();
|
||||
|
|
@ -2,8 +2,8 @@
|
|||
* Hand-labeled evaluation set for {@link techStepClassifier} — what
|
||||
* `tech-step-eval.test.ts` runs the real classifier against to compute
|
||||
* precision/recall/F1 (`tech-step-evaluator.ts`), the objective gate any
|
||||
* future change to `tech-step-training-data.ts` must clear (see that
|
||||
* module's own doc comment).
|
||||
* future change to `services/tech-step-intent-service`'s `training_data.py`
|
||||
* must clear (see that module's own doc comment).
|
||||
*
|
||||
* Deliberately *not* reusing `TECH_STEP_TRAINING_DATA`'s own `utterances`
|
||||
* verbatim — scoring the classifier against the exact sentences it was
|
||||
|
|
@ -246,7 +246,7 @@ export const TECH_STEP_EVAL_DATASET: TechStepEvalCase[] = [
|
|||
// --- Documented false-positive traps, re-verified with fresh wording ---
|
||||
// `brown`'s EN synonyms are verb forms only ("browned"/"browning"), not
|
||||
// bare "brown" — precisely so this doesn't false-positive (see that
|
||||
// entry's own comment in tech-step-training-data.ts).
|
||||
// entry's own comment in training_data.py).
|
||||
{
|
||||
description: "This recipe calls for two tablespoons of brown sugar.",
|
||||
locale: "en",
|
||||
|
|
|
|||
|
|
@ -3,8 +3,9 @@
|
|||
* hand-labeled evaluation set (`tech-step-eval-dataset.ts`) — the objective
|
||||
* counterpart to the "inspected by eye" verdict every corpus change used to
|
||||
* get before this module existed. Every future edit to
|
||||
* `tech-step-training-data.ts` (including the LLM-assisted suggestions the
|
||||
* worker in `services/tech-step-llm-worker` proposes) is expected to run
|
||||
* `services/tech-step-intent-service`'s `training_data.py` (including the
|
||||
* LLM-assisted suggestions the worker in `services/tech-step-llm-worker`
|
||||
* proposes) is expected to run
|
||||
* through `tech-step-eval.test.ts`'s regression gate, which calls
|
||||
* {@link computeTechStepMetrics} — a corpus change that raises recall on one
|
||||
* technique but silently tanks another's precision should fail loudly here,
|
||||
|
|
|
|||
|
|
@ -1,6 +1,5 @@
|
|||
import { NlpManager } from "node-nlp";
|
||||
import { prisma } from "../../db/prisma.js";
|
||||
import { TECH_STEP_TRAINING_DATA } from "./tech-step-training-data.js";
|
||||
import { intentServiceClient } from "./intent-service-client.js";
|
||||
|
||||
/**
|
||||
* Auto-detects which cooking techniques (`TechStep`) a free-text recipe
|
||||
|
|
@ -15,25 +14,27 @@ import { TECH_STEP_TRAINING_DATA } from "./tech-step-training-data.js";
|
|||
* generalize past its own vocabulary — a step describing melting butter as
|
||||
* "jusqu'à ce que le beurre ait disparu dans la poêle" mentions no verb any
|
||||
* regex could anchor on, yet unmistakably *means* `melt`. Replaced with a
|
||||
* small hybrid pipeline built on `node-nlp` ({@link TechStepClassifierService}):
|
||||
* small hybrid pipeline (originally built on `node-nlp`, now entirely
|
||||
* delegated to `services/tech-step-intent-service` — a spaCy-based
|
||||
* microservice, see {@link IntentServiceClient} and that service's own
|
||||
* README):
|
||||
*
|
||||
* 1. **NER** (node-nlp enum entities, `synonyms` in `TECH_STEP_TRAINING_DATA`)
|
||||
* finds every *candidate* technique mention in the whole
|
||||
* description, each with its exact character span — mechanically the
|
||||
* same job the old regexes did, just as flat synonym lists instead of
|
||||
* hand-written patterns (node-nlp's own stemmer/fuzzy matching already
|
||||
* covers minor conjugation/typo variance the regexes had to enumerate
|
||||
* by hand). This step alone is *not* the final answer — see step 3.
|
||||
* 1. **NER** (the intent service's `PhraseMatcher`, built from its own
|
||||
* `training_data.py`'s `synonyms`) finds every *candidate* technique
|
||||
* mention in the whole description, each with its exact character span —
|
||||
* mechanically the same job the old regexes did, just as flat synonym
|
||||
* lists instead of hand-written patterns. This step alone is *not* the
|
||||
* final answer — see step 3.
|
||||
* 2. The description is cut into clauses around those candidate spans
|
||||
* ({@link splitIntoClauses}) — a step naming two techniques ("Dans une
|
||||
* poêle chaude, faire chauffer une noix de beurre" is both `preheat`
|
||||
* and `melt`) needs each judged on its own surrounding context, not the
|
||||
* whole step lumped into one classification.
|
||||
* 3. **NLP intent classification** (node-nlp's `NlpManager`, trained on
|
||||
* `TECH_STEP_TRAINING_DATA`'s `utterances`) then classifies each clause
|
||||
* on its own — this is what actually delivers "meaning, not keywords":
|
||||
* the classifier was deliberately trained on paraphrases that never use
|
||||
* the technique's own verb (e.g. "jusqu'à ce que le beurre ait
|
||||
* 3. **NLP intent classification** (the intent service's `textcat`, trained
|
||||
* on its own `training_data.py`'s `utterances`) then classifies each
|
||||
* clause on its own — this is what actually delivers "meaning, not
|
||||
* keywords": the classifier was deliberately trained on paraphrases that
|
||||
* never use the technique's own verb (e.g. "jusqu'à ce que le beurre ait
|
||||
* disparu" for `melt`), so a clause reaching it gets labeled by what it
|
||||
* was trained to recognize as *meaning* a technique, not by which
|
||||
* literal word the NER step happened to anchor on. The NER-implied
|
||||
|
|
@ -49,12 +50,12 @@ import { TECH_STEP_TRAINING_DATA } from "./tech-step-training-data.js";
|
|||
*
|
||||
* `normalizeText` and {@link splitIntoClauses} are pure (no DB/model
|
||||
* access) so they stay unit-testable in isolation (see
|
||||
* `test/tech-step-matcher.test.ts`); the classifier itself needs a one-time
|
||||
* training pass (`_ensureTrained`, node-nlp's `NlpManager.train()`) plus a
|
||||
* `TechStep.key -> id` lookup from the DB, both memoized on the shared
|
||||
* {@link techStepClassifier} singleton rather than repeated per call —
|
||||
* training is the expensive part (a few hundred ms for this corpus), never
|
||||
* worth redoing per request let alone per step.
|
||||
* `test/tech-step-matcher.test.ts`); this class only ever needs a
|
||||
* `TechStep.key -> id` lookup from the DB, memoized on the shared
|
||||
* {@link techStepClassifier} singleton rather than repeated per call — the
|
||||
* NLP model itself trains once, inside `services/tech-step-intent-service`'s
|
||||
* own startup, entirely independently of this class (see that service's
|
||||
* README — this repo no longer pushes any corpus to it over HTTP).
|
||||
*/
|
||||
|
||||
/**
|
||||
|
|
@ -241,18 +242,31 @@ export function splitIntoClauses(
|
|||
* `TECH_STEP_TRAINING_DATA` — see `test/tech-step-matcher.test.ts` for the
|
||||
* cases this threshold was picked to pass.
|
||||
*
|
||||
* Raised from `0.65` after finding real (non-adversarial) misclassified
|
||||
* clauses that scored just above the old threshold — e.g. English recipe
|
||||
* text run through the French classifier (which must find *nothing*,
|
||||
* confirmed by `recipe-translation.test.ts`'s own locale-isolation test)
|
||||
* scored `0.69` for `boil`, essentially classifier noise on
|
||||
* out-of-vocabulary input rather than a real, confident verdict. The
|
||||
* clauses this threshold exists to actually trust score far higher in
|
||||
* practice (`0.91`–`1.0` for the real corrected cases found this session)
|
||||
* — `0.75` sits comfortably above the noise floor and below every genuine
|
||||
* match seen so far.
|
||||
* Recalibrated for the migration off `node-nlp` to
|
||||
* `services/tech-step-intent-service` (spaCy `textcat`, exclusive classes)
|
||||
* — its score distribution is meaningfully different from node-nlp's own
|
||||
* classifier, and shifts again every time the corpus' technique count
|
||||
* changes (more exclusive classes generally means a *lower* natural
|
||||
* confidence ceiling, softmax mass spread thinner).
|
||||
*
|
||||
* Currently `0.25`, set against the corpus as expanded to ~74 techniques
|
||||
* (`services/tech-step-intent-service/intent_service/training_data.py`,
|
||||
* `_TRAINING_ITERATIONS = 25`, `textcat` trained on each technique's own
|
||||
* `synonyms` in addition to its `utterances` — see that constant's own
|
||||
* comment for the calibration history) from manual spot-checks, not yet a
|
||||
* real `calibrate-tech-step-threshold.ts` sweep against
|
||||
* `TECH_STEP_EVAL_DATASET` (needs Postgres — see that script's own doc
|
||||
* comment): observed real-case scores ranged `0.31`-`0.89` (`simmer`
|
||||
* lowest, still correct in argmax and anchored anyway; `melt` highest, the
|
||||
* motivating anchor-less case), against a noise floor around `0.02`
|
||||
* (English text through the French classifier). `0.25` sits with real
|
||||
* margin above the noise floor and below every real case seen so far, but
|
||||
* **this is a placeholder pending the real eval-dataset sweep** — do not
|
||||
* treat it as load-bearing precision the way the original `0.45`
|
||||
* (calibrated against the ~26-technique corpus, `TECH_STEP_EVAL_DATASET`
|
||||
* F1 plateauing exactly there) was.
|
||||
*/
|
||||
export const CONFIDENCE_THRESHOLD = 0.75;
|
||||
export const CONFIDENCE_THRESHOLD = 0.25;
|
||||
|
||||
/**
|
||||
* One clause's full classification detail — the finer-grained sibling of
|
||||
|
|
@ -273,70 +287,41 @@ export interface TechStepClauseClassification {
|
|||
end: number;
|
||||
/** The clause's NER anchor's own implied technique `uid`, if it had one — same as `TechStepClause.anchor.uid`. */
|
||||
anchorUid: string | null;
|
||||
/** The intent classifier's own top guess for this clause, whatever its score — `null` only when it returned node-nlp's `"None"` sentinel. Unlike {@link TechStepMatch}, never silently replaced by the anchor's uid — the whole point of this type is to expose the classifier's raw opinion, confident or not. */
|
||||
/** The intent classifier's own top guess for this clause, whatever its score — `null` only when the intent service had nothing trained for `locale`, or the clause text was blank. Unlike {@link TechStepMatch}, never silently replaced by the anchor's uid — the whole point of this type is to expose the classifier's raw opinion, confident or not. */
|
||||
intentUid: string | null;
|
||||
/** The intent classifier's own confidence for `intentUid` — `0` when `intentUid` is `null` (nothing to have a score about). */
|
||||
score: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Trains and owns the `node-nlp` model behind {@link matchTechStepSpans} —
|
||||
* Owns the `TechStep.key -> id` lookup behind {@link matchTechStepSpans} —
|
||||
* a real class (not a plain object of functions) per this repo's
|
||||
* service-style-logic convention, even though it's only ever used as the
|
||||
* one shared {@link techStepClassifier} singleton below: it holds real
|
||||
* state (the trained model, the memoized training/lookup promises), not
|
||||
* just grouped stateless helpers.
|
||||
* state (the memoized lookup promise), not just grouped stateless helpers.
|
||||
* The actual NER/intent-classification model lives entirely in
|
||||
* `services/tech-step-intent-service` (a separate process, trained from
|
||||
* its own `training_data.py` at its own startup) — this class never
|
||||
* trains or pushes anything to it, it only calls `POST /v1/process` and
|
||||
* resolves whatever `uid` comes back to a local DB id.
|
||||
*/
|
||||
export class TechStepClassifierService {
|
||||
/** node-nlp's manager — both NER (enum entities) and NLP (intent classification) live on the same instance, trained together. */
|
||||
private readonly _manager: NlpManager;
|
||||
/** Memoized training pass — `undefined` until the first call starts it, after which every caller (concurrent or not) awaits the same promise rather than retraining. */
|
||||
private _trained: Promise<void> | undefined;
|
||||
/** Memoized `TechStep.key -> id` lookup — training data only knows techniques by their stable `uid`/`key`, resolved to the real DB id once, alongside training. */
|
||||
/** Memoized `TechStep.key -> id` lookup — resolved from the DB once, reused by every call rather than queried per request. `undefined` until the first call starts loading it, after which every caller (concurrent or not) awaits the same promise. */
|
||||
private _techStepIdsLoaded: Promise<void> | undefined;
|
||||
private _techStepIdByUid: Map<string, number> | undefined;
|
||||
|
||||
public constructor() {
|
||||
this._manager = new NlpManager({
|
||||
languages: ["fr", "en"],
|
||||
forceNER: true,
|
||||
nlu: { log: false },
|
||||
// node-nlp's enum-entity NER defaults to a fuzzy (Levenshtein-based)
|
||||
// 0.8 accuracy threshold — loose enough that e.g. "faire" (the
|
||||
// generic French helper verb in almost every recipe step) fuzzy-
|
||||
// matches `fry`'s synonym "frire" at 0.80, a false positive found
|
||||
// while tuning this against the real training corpus. `1` (exact,
|
||||
// after node-nlp's own case/accent/stemming normalization — real
|
||||
// conjugation variance is still covered by listing each form in
|
||||
// `tech-step-training-data.ts`) removed it without losing any real
|
||||
// match. Precision matters more than recall for this stage — NER
|
||||
// only proposes candidate split points, `_classifyClause`'s trained
|
||||
// model (not fuzzy string distance) is what actually has to be
|
||||
// right.
|
||||
ner: { threshold: 1 },
|
||||
// node-nlp defaults to `autoSave`/`autoLoad: true` — silently
|
||||
// persisting the trained model to a `model.nlp` file in the process's
|
||||
// cwd, and *loading from that file instead of retraining* the next
|
||||
// time a manager is constructed, if the file already exists. Found
|
||||
// this the hard way: a stray `model.nlp` appeared at the repo root
|
||||
// after running this locally. That's the opposite of what this
|
||||
// service wants — `TECH_STEP_TRAINING_DATA` in code is the single
|
||||
// source of truth this always trains fresh from (see this file's own
|
||||
// doc comment) — a stale on-disk model silently shadowing a
|
||||
// corpus/threshold update would be a nasty, hard-to-notice class of
|
||||
// bug. Both off; nothing here should ever touch disk.
|
||||
autoSave: false,
|
||||
autoLoad: false,
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Forces training plus node-nlp's own one-time lazy setup (loading its
|
||||
* bundled per-language stemmers/tokenizers on the *first* real
|
||||
* `NlpManager.process()` call takes a few seconds by itself, separate
|
||||
* from and much slower than the ~40ms `train()` pass — measured against
|
||||
* this corpus while tuning the pipeline) to happen now, synchronously
|
||||
* with server startup (see `server.ts`), rather than stalling whichever
|
||||
* request happens to be first to save/preview a recipe.
|
||||
* Forces the `TechStep.key -> id` lookup to load now, synchronously with
|
||||
* server startup (see `server.ts`, which also retries this against a
|
||||
* not-yet-reachable intent service), rather than stalling whichever
|
||||
* request happens to be first to save/preview a recipe. Doesn't wait on
|
||||
* `services/tech-step-intent-service` finishing its own training — that
|
||||
* service is only ever considered "up" by Docker Compose/CI once it
|
||||
* already is (see that service's `GET /health`), so by the time this
|
||||
* runs in a real deployment it's already trained; a request racing an
|
||||
* intent service that's genuinely still starting just gets an empty
|
||||
* match list back (see `IntentServiceProcessResult`'s own doc comment),
|
||||
* not an error.
|
||||
*/
|
||||
public async warmUp(): Promise<void> {
|
||||
try {
|
||||
|
|
@ -360,25 +345,21 @@ export class TechStepClassifierService {
|
|||
*/
|
||||
public async matchTechStepSpans(description: string, locale: string): Promise<TechStepMatch[]> {
|
||||
try {
|
||||
await this._ensureTrained();
|
||||
await this._ensureTechStepIdsLoaded();
|
||||
if (description.trim().length === 0) return [];
|
||||
|
||||
const nerResult = await this._manager.process(locale, description);
|
||||
const candidates: TechniqueCandidate[] = nerResult.entities
|
||||
// node-nlp's language plugins also auto-extract their own built-in
|
||||
// entities (numbers, durations, dates…) alongside the enum
|
||||
// entities `_train` registered from `TECH_STEP_TRAINING_DATA` —
|
||||
// `type === "enum"` is what tells the two apart; without this
|
||||
// filter a step like "10 minutes" would hand `splitIntoClauses` a
|
||||
// bogus "duration" candidate that resolves to no real technique.
|
||||
.filter((entity) => entity.type === "enum")
|
||||
.map((entity) => ({
|
||||
uid: entity.entity,
|
||||
// The intent service only ever returns enum-style candidates (its own
|
||||
// `PhraseMatcher`, built solely from `TECH_STEP_TRAINING_DATA`'s
|
||||
// `synonyms`) — unlike node-nlp, it never mixes in built-in
|
||||
// numbers/durations/dates entities, so no `type === "enum"` filter is
|
||||
// needed here anymore. Its `start`/`end` are already `[start, end)`
|
||||
// (matching `String.prototype.slice`), unlike node-nlp's inclusive
|
||||
// `end` — no `+ 1` needed either.
|
||||
const nerResult = await intentServiceClient.process(locale, description);
|
||||
const candidates: TechniqueCandidate[] = nerResult.entities.map((entity) => ({
|
||||
uid: entity.uid,
|
||||
start: entity.start,
|
||||
// node-nlp's own `end` is inclusive (verified against a real
|
||||
// trained model) — `+ 1` converts to this module's `[start, end)`
|
||||
// convention, matching `String.prototype.slice`.
|
||||
end: entity.end + 1,
|
||||
end: entity.end,
|
||||
}));
|
||||
|
||||
const clauses = splitIntoClauses(description, candidates);
|
||||
|
|
@ -428,16 +409,14 @@ export class TechStepClassifierService {
|
|||
locale: string,
|
||||
): Promise<TechStepClauseClassification[]> {
|
||||
try {
|
||||
await this._ensureTrained();
|
||||
await this._ensureTechStepIdsLoaded();
|
||||
if (description.trim().length === 0) return [];
|
||||
|
||||
const nerResult = await this._manager.process(locale, description);
|
||||
const candidates: TechniqueCandidate[] = nerResult.entities
|
||||
.filter((entity) => entity.type === "enum")
|
||||
.map((entity) => ({
|
||||
uid: entity.entity,
|
||||
const nerResult = await intentServiceClient.process(locale, description);
|
||||
const candidates: TechniqueCandidate[] = nerResult.entities.map((entity) => ({
|
||||
uid: entity.uid,
|
||||
start: entity.start,
|
||||
end: entity.end + 1,
|
||||
end: entity.end,
|
||||
}));
|
||||
|
||||
const clauses = splitIntoClauses(description, candidates);
|
||||
|
|
@ -456,15 +435,14 @@ export class TechStepClassifierService {
|
|||
});
|
||||
continue;
|
||||
}
|
||||
const result = await this._manager.process(locale, clauseText);
|
||||
const intentUid = result.intent !== "None" ? result.intent : null;
|
||||
const result = await intentServiceClient.process(locale, clauseText);
|
||||
results.push({
|
||||
clauseText,
|
||||
start: clause.start,
|
||||
end: clause.end,
|
||||
anchorUid,
|
||||
intentUid,
|
||||
score: intentUid === null ? 0 : result.score,
|
||||
intentUid: result.intent,
|
||||
score: result.intent === null ? 0 : result.score,
|
||||
});
|
||||
}
|
||||
return results;
|
||||
|
|
@ -506,8 +484,8 @@ export class TechStepClassifierService {
|
|||
const clauseText = description.slice(clause.start, clause.end).trim();
|
||||
if (clauseText.length === 0) return clause.anchor?.uid ?? null;
|
||||
|
||||
const result = await this._manager.process(locale, clauseText);
|
||||
if (result.intent !== "None" && result.score >= CONFIDENCE_THRESHOLD) {
|
||||
const result = await intentServiceClient.process(locale, clauseText);
|
||||
if (result.intent !== null && result.score >= CONFIDENCE_THRESHOLD) {
|
||||
return result.intent;
|
||||
}
|
||||
return clause.anchor?.uid ?? null;
|
||||
|
|
@ -517,52 +495,34 @@ export class TechStepClassifierService {
|
|||
}
|
||||
|
||||
/**
|
||||
* Trains `_manager` from {@link TECH_STEP_TRAINING_DATA} and resolves the
|
||||
* `uid -> TechStep.id` lookup, both exactly once — memoized on
|
||||
* `_trained` so a burst of concurrent calls (several steps of the same
|
||||
* recipe save, awaited via the same event loop tick) all await the one
|
||||
* in-flight training pass rather than each kicking off their own.
|
||||
* Resolves the `uid -> TechStep.id` lookup exactly once — memoized on
|
||||
* `_techStepIdsLoaded` so a burst of concurrent calls (several steps of
|
||||
* the same recipe save, awaited via the same event loop tick) all await
|
||||
* the one in-flight DB query rather than each firing their own.
|
||||
*/
|
||||
private async _ensureTrained(): Promise<void> {
|
||||
if (this._trained === undefined) {
|
||||
this._trained = this._train();
|
||||
private async _ensureTechStepIdsLoaded(): Promise<void> {
|
||||
if (this._techStepIdsLoaded === undefined) {
|
||||
this._techStepIdsLoaded = this._loadTechStepIds();
|
||||
}
|
||||
try {
|
||||
await this._trained;
|
||||
await this._techStepIdsLoaded;
|
||||
} catch (err) {
|
||||
// A failed training pass must be retried by the *next* call, not
|
||||
// leave every future call permanently rejecting against a stale
|
||||
// failed promise.
|
||||
this._trained = undefined;
|
||||
// A failed load must be retried by the *next* call, not leave every
|
||||
// future call permanently rejecting against a stale failed promise.
|
||||
this._techStepIdsLoaded = undefined;
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
private async _train(): Promise<void> {
|
||||
private async _loadTechStepIds(): Promise<void> {
|
||||
try {
|
||||
const techSteps = await prisma.techStep.findMany({ select: { id: true, key: true } });
|
||||
this._techStepIdByUid = new Map(techSteps.map((techStep) => [techStep.key, techStep.id]));
|
||||
|
||||
for (const entry of TECH_STEP_TRAINING_DATA) {
|
||||
for (const [locale, data] of [
|
||||
["fr", entry.fr],
|
||||
["en", entry.en],
|
||||
] as const) {
|
||||
if (data.synonyms.length > 0) {
|
||||
this._manager.addNamedEntityText(entry.uid, entry.uid, [locale], data.synonyms);
|
||||
}
|
||||
for (const utterance of data.utterances) {
|
||||
this._manager.addDocument(locale, utterance, entry.uid);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
await this._manager.train();
|
||||
} catch (err) {
|
||||
throw err; // see matchTechStepSpans()'s catch comment above
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Single shared instance — training is expensive enough (a few hundred ms) that every caller must reuse the one already-trained model, never spin up their own. */
|
||||
/** Single shared instance — every caller reuses the one memoized `TechStep.key -> id` lookup rather than re-querying the DB. The actual model training (expensive — a couple of minutes, both locales combined) happens entirely inside `services/tech-step-intent-service`'s own startup, not here — see that service's `_TRAINING_ITERATIONS`. */
|
||||
export const techStepClassifier = new TechStepClassifierService();
|
||||
|
|
|
|||
|
|
@ -1,911 +0,0 @@
|
|||
/**
|
||||
* Training corpus for {@link TechStepClassifierService} (`tech-step-matcher.ts`)
|
||||
* — one entry per `TechStep` (`uid` matches `reference-seed-data.ts`'s
|
||||
* `TECH_STEPS`, which still owns the reference `TechStep` rows themselves;
|
||||
* this file replaces `TECH_STEPS[].mappings`' regex expressions as the
|
||||
* *matching* data source).
|
||||
*
|
||||
* Two distinct kinds of content per technique/locale, feeding two distinct
|
||||
* mechanisms of the classifier (see that file's doc comment for why both
|
||||
* are needed):
|
||||
*
|
||||
* - `synonyms` — short literal words/set phrases, fed to node-nlp's NER
|
||||
* (enum entities). Mechanically equivalent to the old regexes' verb-form
|
||||
* alternations, just spelled out as plain words instead of a pattern
|
||||
* (node-nlp's own stemmer/fuzzy matching already covers minor
|
||||
* conjugation/typo variance that the regexes had to enumerate by hand).
|
||||
* Used only to find *candidate* technique mentions and cut a step into
|
||||
* clauses around them — never the final answer on their own.
|
||||
* - `utterances` — full example clauses, fed to node-nlp's NLP Manager as
|
||||
* training documents for the intent classifier. Deliberately mixes
|
||||
* keyword-anchored phrasings (reinforces the obvious case) with
|
||||
* paraphrases that never use the technique's own verb at all (e.g.
|
||||
* "jusqu'à ce que le beurre ait disparu" for `melt`) — this second kind
|
||||
* is what actually delivers on "comprendre le sens, pas juste les mots
|
||||
* clés" (see the PR this file was introduced in): a clause reaching the
|
||||
* classifier gets labeled by what it's trained to recognize as *meaning*
|
||||
* this technique, not by which literal word triggered its extraction.
|
||||
*
|
||||
* Kept as static in-code data (not DB rows, unlike the old
|
||||
* `TechStepMapping` table) because nothing needs to query/edit it at
|
||||
* runtime — it only ever feeds one thing, the classifier's one-time
|
||||
* training pass (see `TechStepClassifierService._ensureTrained`) — same
|
||||
* reasoning `INGREDIENT_LABELS_EN` (`packages/shared`) is a plain object,
|
||||
* not a database table.
|
||||
*/
|
||||
|
||||
/** One technique's matching data for one locale — see this file's doc comment for what each list feeds. */
|
||||
export interface TechStepLocaleTrainingData {
|
||||
synonyms: string[];
|
||||
utterances: string[];
|
||||
}
|
||||
|
||||
/** One technique's full training entry — `uid` must match a `TECH_STEPS[].uid` in `reference-seed-data.ts`. */
|
||||
export interface TechStepTrainingEntry {
|
||||
uid: string;
|
||||
fr: TechStepLocaleTrainingData;
|
||||
en: TechStepLocaleTrainingData;
|
||||
}
|
||||
|
||||
export const TECH_STEP_TRAINING_DATA: TechStepTrainingEntry[] = [
|
||||
{
|
||||
uid: "cook",
|
||||
fr: {
|
||||
synonyms: [
|
||||
"cuire",
|
||||
"cuisez",
|
||||
"cuisant",
|
||||
"cuisson",
|
||||
"cuit",
|
||||
"cuite",
|
||||
"cuites",
|
||||
"cuits",
|
||||
"cuisiner",
|
||||
"cuisinez",
|
||||
"cuisiné",
|
||||
"cuisinée",
|
||||
"faire cuire",
|
||||
"laisser cuire",
|
||||
],
|
||||
utterances: [
|
||||
"faire cuire à feu moyen",
|
||||
"laisser cuire jusqu'à ce que ce soit prêt",
|
||||
"la cuisson dure environ dix minutes",
|
||||
"jusqu'à ce que la viande ne soit plus rose au centre",
|
||||
"poursuivre la cuisson à couvert",
|
||||
// Two real recipe clauses found misclassified (as `preheat` and
|
||||
// `panFry` respectively, both above the confidence threshold) once
|
||||
// real, longer, comma-heavy sentences started reaching the
|
||||
// classifier — neither error came from a missing keyword (both
|
||||
// clauses' own NER anchor, "laisser cuire"/"faire cuire", was
|
||||
// already right), just the classifier's low-heat/occasional-
|
||||
// stirring phrasing not resembling anything short and clean-cut it
|
||||
// had actually been trained on.
|
||||
"baisser le feu et laisser cuire à découvert encore un quart d'heure",
|
||||
"faire cuire à feu doux en remuant de temps en temps",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
// NOT "cooked through"/"cooking through" — both are word-prefix
|
||||
// extensions of "cooked"/"cooking" above, so any text containing them
|
||||
// matches BOTH the short and long form as separate overlapping NER
|
||||
// candidates, corrupting clause-splitting (confirmed via "It should
|
||||
// be cooking through evenly", which spuriously grew a second,
|
||||
// wrongly-classified `roast` candidate). See this pattern flagged
|
||||
// throughout the file wherever it was found — the fix is always to
|
||||
// drop the longer, redundant form rather than keep both.
|
||||
synonyms: ["cook", "cooks", "cooked", "cooking"],
|
||||
utterances: [
|
||||
"cook over medium heat",
|
||||
"cook until done",
|
||||
"cooking takes about ten minutes",
|
||||
"until no longer pink in the middle",
|
||||
"continue cooking covered",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "fry",
|
||||
fr: {
|
||||
synonyms: [
|
||||
"frire",
|
||||
"frit",
|
||||
"frite",
|
||||
"frites",
|
||||
"friture",
|
||||
"faire frire",
|
||||
"faites frire",
|
||||
"bain de friture",
|
||||
"huile de friture",
|
||||
],
|
||||
utterances: [
|
||||
"faire frire dans l'huile chaude",
|
||||
"plonger dans la friture",
|
||||
"jusqu'à ce que ce soit doré et croustillant à l'extérieur",
|
||||
"l'huile doit être bien chaude avant d'y plonger les morceaux",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
// NOT "frying oil" — a word-prefix extension of "frying" above (see
|
||||
// the `cook` entry's comment for why that duplicates/corrupts NER
|
||||
// candidates; here it was even worse, misclassifying as `preheat`).
|
||||
synonyms: ["fry", "fries", "fried", "frying", "deep fry", "deep-fried", "deep frying"],
|
||||
utterances: [
|
||||
"fry in hot oil",
|
||||
"deep fry until golden",
|
||||
"until crisp and golden on the outside",
|
||||
"the oil should be very hot before adding the pieces",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "melt",
|
||||
fr: {
|
||||
synonyms: [
|
||||
"fondre",
|
||||
"fondu",
|
||||
"fondue",
|
||||
"fondues",
|
||||
"faire fondre",
|
||||
"faites fondre",
|
||||
// Also a plausible way to say "melt" (heating something — usually
|
||||
// a fat — until it liquefies), not just a `preheat` phrasing —
|
||||
// restores what the regex-based system anchored on before this
|
||||
// pipeline replaced it.
|
||||
"faire chauffer",
|
||||
"faites chauffer",
|
||||
"liquéfier",
|
||||
"liquéfiez",
|
||||
"liquéfié",
|
||||
"faire liquéfier",
|
||||
],
|
||||
utterances: [
|
||||
"faire fondre le beurre",
|
||||
"jusqu'à ce que le beurre ait disparu dans la poêle",
|
||||
"le beurre doit être complètement liquide",
|
||||
"laisser le fromage devenir tout liquide sur feu doux",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
synonyms: ["melt", "melts", "melted", "melting", "liquefy", "liquefied"],
|
||||
utterances: [
|
||||
"melt the butter",
|
||||
"until the butter has completely disappeared into the pan",
|
||||
"the butter should be fully liquid",
|
||||
"let the cheese turn completely liquid over low heat",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "deglaze",
|
||||
fr: {
|
||||
// NOT "déglacer la poêle"/"déglacer le fond de cuisson" — both are
|
||||
// word-prefix extensions of "déglacer" above (see `cook`'s comment
|
||||
// for why that duplicates NER candidates).
|
||||
synonyms: ["déglacer", "déglacez", "déglacé", "déglacée", "déglaçage"],
|
||||
utterances: [
|
||||
"déglacer avec le vin blanc",
|
||||
"verser le vin dans la poêle chaude pour décoller les sucs",
|
||||
"gratter les sucs de cuisson au fond de la casserole avec un peu de bouillon",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
// NOT "deglaze the pan" — a word-prefix extension of "deglaze" above
|
||||
// (see `cook`'s comment for why that duplicates NER candidates).
|
||||
synonyms: ["deglaze", "deglazes", "deglazed", "deglazing", "lift the browned bits"],
|
||||
utterances: [
|
||||
"deglaze with white wine",
|
||||
"pour the wine into the hot pan to lift the browned bits",
|
||||
"scrape up the browned bits at the bottom of the pan with a splash of stock",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "simmer",
|
||||
fr: {
|
||||
synonyms: [
|
||||
"mijoter",
|
||||
"mijotez",
|
||||
"mijote",
|
||||
"mijotant",
|
||||
"mijoté",
|
||||
"frémir",
|
||||
"frémissant",
|
||||
"frémissante",
|
||||
"à petit feu",
|
||||
],
|
||||
utterances: [
|
||||
"laisser mijoter à feu doux",
|
||||
"faire mijoter pendant une heure",
|
||||
"de petites bulles doivent remonter doucement à la surface",
|
||||
"laisser cuire tout doucement à couvert pendant longtemps",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
// NOT "simmering gently" — a word-prefix extension of "simmering"
|
||||
// above (see `cook`'s comment for why that duplicates NER candidates).
|
||||
synonyms: ["simmer", "simmers", "simmered", "simmering", "gentle simmer", "low simmer"],
|
||||
utterances: [
|
||||
"let it simmer over low heat",
|
||||
"simmer for one hour",
|
||||
"small bubbles should gently rise to the surface",
|
||||
"let it cook very gently, covered, for a long time",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "boil",
|
||||
fr: {
|
||||
synonyms: [
|
||||
"bouillir",
|
||||
"bouillant",
|
||||
"bouillie",
|
||||
"bouillies",
|
||||
"ébullition",
|
||||
"porter à ébullition",
|
||||
"gros bouillons",
|
||||
],
|
||||
utterances: [
|
||||
"porter à ébullition",
|
||||
"faire bouillir l'eau",
|
||||
"de grosses bulles doivent agiter la surface avec force",
|
||||
"jusqu'à ce que ça bouillonne franchement",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
// NOT "boiling point" — a word-prefix extension of "boiling" above
|
||||
// (see `cook`'s comment for why that duplicates NER candidates).
|
||||
synonyms: ["boil", "boils", "boiled", "boiling", "rolling boil"],
|
||||
utterances: [
|
||||
"bring to a boil",
|
||||
"boil the water",
|
||||
"large bubbles should be vigorously breaking the surface",
|
||||
"until it's rolling vigorously",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "roast",
|
||||
fr: {
|
||||
// NOT "rôti au four" — a word-prefix extension of "rôti" above (see
|
||||
// `cook`'s comment for why that duplicates NER candidates).
|
||||
synonyms: ["rôtir", "rôti", "rôtie", "rôties", "rôtis", "rôtissage"],
|
||||
utterances: [
|
||||
"faire rôtir la volaille entière",
|
||||
"le rôti doit dorer uniformément de tous les côtés",
|
||||
"cuire la pièce de viande entière au four à chaleur sèche",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
synonyms: ["roast", "roasts", "roasted", "roasting", "oven-roast", "oven roasted"],
|
||||
utterances: [
|
||||
"roast the whole bird",
|
||||
"it should brown evenly on every side",
|
||||
"cook the whole piece of meat in dry oven heat",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "grill",
|
||||
fr: {
|
||||
synonyms: [
|
||||
"griller",
|
||||
"grillez",
|
||||
"grillé",
|
||||
"grillée",
|
||||
"grillées",
|
||||
"grillade",
|
||||
"grillades",
|
||||
"barbecue",
|
||||
"au barbecue",
|
||||
],
|
||||
utterances: [
|
||||
"faire griller sur la grille du barbecue",
|
||||
"marquer les steaks sur une plaque brûlante",
|
||||
"des traces de quadrillage doivent apparaître à la cuisson",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
synonyms: ["grill", "grills", "grilled", "grilling", "barbecue", "char-grill", "charbroiled"],
|
||||
utterances: [
|
||||
"grill on the barbecue rack",
|
||||
"sear the steaks on a scorching-hot plate",
|
||||
"char marks should appear as it cooks",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "panFry",
|
||||
fr: {
|
||||
// Deliberately NOT "poêlé"/"poêlée"/"poêlés" here, despite reading
|
||||
// like natural panFry vocabulary: node-nlp's French stemmer reduces
|
||||
// them to the same root as the bare noun "poêle" (a pan), so
|
||||
// registering them made every plain mention of "poêle" — e.g.
|
||||
// `preheat`'s own "la poêle" — a false-positive panFry candidate too.
|
||||
// Found via the "jusqu'à ce que le beurre ait disparu dans la poêle"
|
||||
// regression test, which unexpectedly grew a spurious panFry match.
|
||||
synonyms: ["sauter", "sautez", "sauté", "sautée", "sautées", "sautant", "à la poêle"],
|
||||
utterances: [
|
||||
"faire sauter les légumes à la poêle",
|
||||
"saisir rapidement à feu vif en remuant sans cesse",
|
||||
"faire revenir en remuant vivement dans une poêle très chaude",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
synonyms: [
|
||||
"sauté",
|
||||
"sauteed",
|
||||
"sautéed",
|
||||
"sauteing",
|
||||
"pan-fry",
|
||||
"pan fried",
|
||||
"pan-fried",
|
||||
"stir-fry",
|
||||
"pan searing",
|
||||
"seared in a pan",
|
||||
],
|
||||
utterances: [
|
||||
"sauté the vegetables in a pan",
|
||||
"quickly sear over high heat, stirring constantly",
|
||||
"cook briskly, stirring, in a very hot pan",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "blanch",
|
||||
fr: {
|
||||
synonyms: ["blanchir", "blanchissez", "blanchi", "blanchie", "blanchies", "blanchiment"],
|
||||
utterances: [
|
||||
"faire blanchir les légumes deux minutes dans l'eau bouillante",
|
||||
"plonger brièvement dans l'eau bouillante puis directement dans l'eau glacée",
|
||||
"cuire très rapidement à l'eau bouillante avant de stopper la cuisson au froid",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
// "parboil" is folded in here rather than kept a separate technique —
|
||||
// in home-cooking usage (as opposed to professional usage, where they
|
||||
// can differ) it names the same "briefly pre-cook in boiling water"
|
||||
// move blanching does.
|
||||
synonyms: [
|
||||
"blanch",
|
||||
"blanches",
|
||||
"blanched",
|
||||
"blanching",
|
||||
"parboil",
|
||||
"parboiled",
|
||||
"parboiling",
|
||||
],
|
||||
utterances: [
|
||||
"blanch the vegetables for two minutes in boiling water",
|
||||
"briefly plunge into boiling water then straight into ice water",
|
||||
"cook very quickly in boiling water before stopping it cold",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "marinate",
|
||||
fr: {
|
||||
synonyms: [
|
||||
"mariner",
|
||||
"marinez",
|
||||
"mariné",
|
||||
"marinée",
|
||||
"marinées",
|
||||
"marinade",
|
||||
"macérer",
|
||||
"macérez",
|
||||
"macération",
|
||||
"faire mariner",
|
||||
],
|
||||
utterances: [
|
||||
"laisser mariner la viande toute la nuit au réfrigérateur",
|
||||
"faire tremper dans la sauce plusieurs heures avant cuisson pour parfumer",
|
||||
"laisser reposer dans le mélange d'huile et d'épices avant de cuisiner",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
// NOT "marinating for" — a word-prefix extension of "marinating"
|
||||
// above (see `cook`'s comment for why that duplicates NER candidates
|
||||
// — here it was even worse, misclassifying as `simmer`).
|
||||
synonyms: [
|
||||
"marinate",
|
||||
"marinates",
|
||||
"marinated",
|
||||
"marinating",
|
||||
"marinade",
|
||||
"soak in the marinade",
|
||||
],
|
||||
utterances: [
|
||||
"let the meat marinate overnight in the fridge",
|
||||
"soak in the sauce for several hours before cooking to flavor it",
|
||||
"let it sit in the oil and spice mixture before cooking",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "chop",
|
||||
fr: {
|
||||
// NOT "hacher grossièrement" — a word-prefix extension of "hacher"
|
||||
// above (see `cook`'s comment for why that duplicates NER candidates).
|
||||
synonyms: [
|
||||
"hacher",
|
||||
"hachez",
|
||||
"haché",
|
||||
"hachée",
|
||||
"hachées",
|
||||
"hachis",
|
||||
"couper en morceaux",
|
||||
"tailler en morceaux",
|
||||
],
|
||||
utterances: [
|
||||
"hacher finement les oignons",
|
||||
"couper en tout petits morceaux irréguliers au couteau",
|
||||
"réduire les herbes en petits fragments avant de les ajouter",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
// NOT "chop coarsely" — a word-prefix extension of "chop" above (see
|
||||
// `cook`'s comment for why that duplicates NER candidates).
|
||||
synonyms: ["chop", "chops", "chopped", "chopping", "roughly chop", "coarsely chopped"],
|
||||
utterances: [
|
||||
"finely chop the onions",
|
||||
"cut into small, uneven pieces with a knife",
|
||||
"break the herbs down into small bits before adding them",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "peel",
|
||||
fr: {
|
||||
synonyms: [
|
||||
"éplucher",
|
||||
"épluchez",
|
||||
"épluché",
|
||||
"épluchée",
|
||||
"épluchées",
|
||||
"épluchage",
|
||||
"peler",
|
||||
"pelez",
|
||||
"pelé",
|
||||
"pelée",
|
||||
"pelées",
|
||||
],
|
||||
utterances: [
|
||||
"éplucher les pommes de terre",
|
||||
"retirer la peau des carottes avec un économe",
|
||||
"ôter la pelure du fruit avant de le couper",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
synonyms: ["peel", "peels", "peeled", "peeling", "pare", "pared", "paring"],
|
||||
utterances: [
|
||||
"peel the potatoes",
|
||||
"remove the skin from the carrots with a peeler",
|
||||
"take the skin off the fruit before cutting it",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "mince",
|
||||
fr: {
|
||||
synonyms: [
|
||||
"émincer",
|
||||
"émincez",
|
||||
"émincé",
|
||||
"émincée",
|
||||
"émincées",
|
||||
"ciseler",
|
||||
"ciselez",
|
||||
"ciselé",
|
||||
"ciselée",
|
||||
"ciselées",
|
||||
],
|
||||
utterances: [
|
||||
"émincer l'oignon en fines lamelles",
|
||||
"couper en très fines tranches régulières",
|
||||
"détailler en lamelles aussi fines que possible",
|
||||
// Without this, a short clause naming a different vegetable —
|
||||
// "Émincer les tomates" — scored just above `melt`'s confidence
|
||||
// threshold instead (a training-set-composition side effect of
|
||||
// adding utterances elsewhere in this same pass, found by the full
|
||||
// regression suite). A second example anchored on a different noun
|
||||
// widens `mince`'s own region enough to reclaim it.
|
||||
"émincer les tomates en fines rondelles",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
// NOT "mince finely" — a word-prefix extension of "mince" above (see
|
||||
// `cook`'s comment for why that duplicates NER candidates).
|
||||
synonyms: ["mince", "minces", "minced", "mincing", "thinly slice", "finely mince"],
|
||||
utterances: [
|
||||
"mince the onion into thin strips",
|
||||
"cut into very thin, even slices",
|
||||
"slice into strips as thin as possible",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "mix",
|
||||
fr: {
|
||||
synonyms: [
|
||||
"mélanger",
|
||||
"mélangez",
|
||||
"mélangé",
|
||||
"mélangée",
|
||||
"mélangées",
|
||||
"mélange",
|
||||
"brasser",
|
||||
"brassez",
|
||||
"amalgamer",
|
||||
"amalgamez",
|
||||
],
|
||||
utterances: [
|
||||
"mélanger tous les ingrédients dans un saladier",
|
||||
"combiner le sucre et la farine ensemble",
|
||||
"remuer jusqu'à obtenir une préparation homogène",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
synonyms: [
|
||||
"mix",
|
||||
"mixes",
|
||||
"mixed",
|
||||
"mixing",
|
||||
"combine",
|
||||
"combined",
|
||||
"blend",
|
||||
"blended",
|
||||
"blending",
|
||||
"stir together",
|
||||
],
|
||||
utterances: [
|
||||
"mix all the ingredients in a bowl",
|
||||
"combine the sugar and flour together",
|
||||
"stir until the mixture is smooth and even",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "whisk",
|
||||
fr: {
|
||||
synonyms: [
|
||||
"fouetter",
|
||||
"fouettez",
|
||||
"fouetté",
|
||||
"fouettée",
|
||||
"fouettées",
|
||||
"au fouet",
|
||||
"battre au fouet",
|
||||
"monter au fouet",
|
||||
],
|
||||
utterances: [
|
||||
"fouetter les œufs et le sucre",
|
||||
"battre vigoureusement au fouet jusqu'à ce que ça blanchisse",
|
||||
"travailler énergiquement pour incorporer de l'air au mélange",
|
||||
// Without these, "Fouetter les blancs en neige" misclassified as
|
||||
// `foldIn` — its own training utterance below also happens to say
|
||||
// "les blancs en neige", and node-nlp's intent classifier leaned on
|
||||
// that shared noun phrase over the actual verb. The exact phrase
|
||||
// itself is needed (not just a paraphrase of it) — a longer,
|
||||
// differently-worded utterance alone wasn't enough to outweigh
|
||||
// `foldIn`'s own close phrasing.
|
||||
"fouetter les blancs en neige",
|
||||
"fouetter les blancs en neige jusqu'à ce qu'ils soient fermes",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
synonyms: ["whisk", "whisks", "whisked", "whisking", "beat", "whip", "whipped", "whipping"],
|
||||
utterances: [
|
||||
"whisk the eggs and sugar",
|
||||
"beat vigorously with a whisk until pale",
|
||||
"work it briskly to whip air into the mixture",
|
||||
"whisk the egg whites until stiff peaks form",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "foldIn",
|
||||
fr: {
|
||||
synonyms: [
|
||||
"incorporer",
|
||||
"incorporez",
|
||||
"incorporé",
|
||||
"incorporée",
|
||||
"incorporées",
|
||||
// NOT "incorporer délicatement" — it's a superstring of "incorporer"
|
||||
// above, so both would match the same text and hand
|
||||
// `splitIntoClauses` two overlapping candidates for one mention
|
||||
// (found via "Incorporer délicatement la farine" producing two
|
||||
// duplicate matches instead of one).
|
||||
"mélanger délicatement",
|
||||
],
|
||||
utterances: [
|
||||
"incorporer délicatement les blancs en neige",
|
||||
"ajouter en soulevant doucement la masse pour ne pas casser les bulles",
|
||||
"mélanger tout doucement de bas en haut pour garder l'air emprisonné",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
synonyms: ["fold in", "folds in", "folded in", "folding in", "gently fold", "fold gently"],
|
||||
utterances: [
|
||||
"gently fold in the beaten egg whites",
|
||||
"add by gently lifting the batter so you don't knock the air out",
|
||||
"very gently stir from the bottom up to keep the air trapped in",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "setAside",
|
||||
fr: {
|
||||
synonyms: [
|
||||
"réserver",
|
||||
"réservez",
|
||||
"réservé",
|
||||
"réservée",
|
||||
"réservées",
|
||||
"mettre de côté",
|
||||
"laisser de côté",
|
||||
],
|
||||
utterances: [
|
||||
"réserver au frais en attendant",
|
||||
"mettre de côté pour plus tard",
|
||||
"laisser attendre sur le plan de travail pendant la préparation du reste",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
synonyms: ["set aside", "sets aside", "setting aside", "set it aside", "reserve", "reserved"],
|
||||
utterances: [
|
||||
"set aside in the fridge for now",
|
||||
"put it aside for later",
|
||||
"let it wait on the counter while you prepare the rest",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "season",
|
||||
fr: {
|
||||
synonyms: [
|
||||
"assaisonner",
|
||||
"assaisonnez",
|
||||
"assaisonné",
|
||||
"assaisonnée",
|
||||
"assaisonnement",
|
||||
"relever",
|
||||
"relevez",
|
||||
"épicer",
|
||||
"épicez",
|
||||
],
|
||||
utterances: [
|
||||
"assaisonner avec du sel et du poivre",
|
||||
"rectifier le goût en ajoutant des épices",
|
||||
"ajouter du sel selon votre goût avant de servir",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
synonyms: ["season", "seasons", "seasoned", "seasoning", "spice it up", "add seasoning"],
|
||||
utterances: [
|
||||
"season with salt and pepper",
|
||||
"adjust the taste by adding spices",
|
||||
"add salt to taste before serving",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "drain",
|
||||
fr: {
|
||||
synonyms: [
|
||||
"égoutter",
|
||||
"égouttez",
|
||||
"égoutté",
|
||||
"égouttée",
|
||||
"égouttées",
|
||||
"essorer",
|
||||
"essorez",
|
||||
"essoré",
|
||||
"essorée",
|
||||
],
|
||||
utterances: [
|
||||
"égoutter les pâtes dans une passoire",
|
||||
"verser dans une passoire pour retirer l'eau de cuisson",
|
||||
"laisser l'excédent d'eau s'écouler avant de servir",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
synonyms: ["drain", "drains", "drained", "draining", "strain", "strained", "straining"],
|
||||
utterances: [
|
||||
"drain the pasta in a colander",
|
||||
"pour into a colander to remove the cooking water",
|
||||
"let the excess water run off before serving",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "brown",
|
||||
fr: {
|
||||
synonyms: [
|
||||
"faire revenir",
|
||||
"faites revenir",
|
||||
"faire dorer",
|
||||
"faites dorer",
|
||||
"colorer",
|
||||
"colorez",
|
||||
"faire colorer",
|
||||
],
|
||||
utterances: [
|
||||
"faire revenir les oignons dans l'huile chaude",
|
||||
"faire dorer la viande sur toutes les faces",
|
||||
"saisir jusqu'à ce que la surface prenne une belle couleur caramel",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
// Verb forms only (not bare "brown"), same reasoning the old regex
|
||||
// doc comment gave — a bare "brown" false-positives on ingredient
|
||||
// descriptions like "brown sugar"/"brown rice", which never get to
|
||||
// the classifier since they're not step text, but keeping the
|
||||
// synonym itself anchored costs nothing and stays consistent.
|
||||
synonyms: ["browned", "browning"],
|
||||
utterances: [
|
||||
"brown the onions in hot oil",
|
||||
"brown the meat on every side",
|
||||
"sear until the surface turns a deep caramel color",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "rest",
|
||||
fr: {
|
||||
synonyms: ["reposer", "laisser reposer", "laissez reposer", "temps de repos"],
|
||||
utterances: [
|
||||
"laisser reposer la pâte trente minutes",
|
||||
"laisser la viande se détendre hors du four avant de la découper",
|
||||
"attendre quelques minutes avant de servir pour que les jus se répartissent",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
// Anchored to "let ... rest"/"rest for" rather than bare "rest",
|
||||
// same false-positive reasoning as `brown` above ("the rest of the").
|
||||
synonyms: ["let it rest", "let them rest", "resting for", "rested for", "resting time"],
|
||||
utterances: [
|
||||
"let the dough rest for thirty minutes",
|
||||
"let the meat relax outside the oven before carving it",
|
||||
"wait a few minutes before serving so the juices redistribute",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "preheat",
|
||||
fr: {
|
||||
synonyms: [
|
||||
"préchauffer",
|
||||
"préchauffez",
|
||||
"préchauffé",
|
||||
"préchauffée",
|
||||
// A pan already described as hot ("poêle chaude") implies it's
|
||||
// been preheated, without the verb itself — the classic "Dans une
|
||||
// poêle chaude, faire chauffer une noix de beurre" case (both
|
||||
// `preheat` and `melt` in one instruction).
|
||||
"poêle chaude",
|
||||
"préchauffage",
|
||||
],
|
||||
utterances: [
|
||||
"préchauffer le four à 180 degrés",
|
||||
"mettre le four à chauffer avant d'y placer le plat",
|
||||
"allumer le four à l'avance pour qu'il soit à température",
|
||||
// A pan gets preheated too, not just an oven — without an example
|
||||
// like this, "poêle" (which also appears throughout `panFry`'s own
|
||||
// training utterances) biased the classifier toward `panFry` for
|
||||
// any preheating clause that happens to mention a pan, found while
|
||||
// testing against the classic "Préchauffer la poêle, puis faire
|
||||
// fondre le beurre" case.
|
||||
"préchauffer la poêle avant d'y verser l'huile",
|
||||
"faire chauffer la poêle à vide quelques minutes",
|
||||
// "poêle" + "feu vif" together still read as `panFry` (the act of
|
||||
// actually cooking something in it) rather than `preheat` (getting
|
||||
// it hot beforehand, nothing in it yet) without an example this
|
||||
// close to that exact wording — found via "mettre la poêle sur feu
|
||||
// vif" (no food mentioned at all) still classifying as panFry.
|
||||
"mettre la poêle vide sur feu vif avant d'ajouter quoi que ce soit",
|
||||
"mettre la poêle sur feu vif",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
// NOT "preheating time" — a word-prefix extension of "preheating"
|
||||
// above (see `cook`'s comment for why that duplicates NER candidates).
|
||||
synonyms: ["preheat", "preheats", "preheated", "preheating", "hot pan"],
|
||||
utterances: [
|
||||
"preheat the oven to 180 degrees",
|
||||
"turn the oven on to heat up before putting the dish in",
|
||||
"switch the oven on ahead of time so it's up to temperature",
|
||||
"preheat the pan before adding the oil",
|
||||
"heat the empty pan for a few minutes first",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "bake",
|
||||
fr: {
|
||||
synonyms: [
|
||||
"cuire au four",
|
||||
"cuisson au four",
|
||||
"enfourner",
|
||||
"enfournez",
|
||||
"au four",
|
||||
"enfourné",
|
||||
"enfournée",
|
||||
],
|
||||
utterances: [
|
||||
"enfourner pendant quarante-cinq minutes",
|
||||
"mettre au four jusqu'à ce que ce soit doré",
|
||||
"cuire dans le four préchauffé jusqu'à ce que la surface soit ferme",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
// NOT "baked in the oven" — a word-prefix extension of "baked" above
|
||||
// (see `cook`'s comment for why that duplicates NER candidates).
|
||||
synonyms: ["bake", "bakes", "baked", "baking", "in the oven", "oven-baked"],
|
||||
utterances: [
|
||||
"bake for forty-five minutes",
|
||||
"put it in the oven until golden",
|
||||
"cook in the preheated oven until the surface is firm",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "plate",
|
||||
fr: {
|
||||
// NOT "dressage de l'assiette" — a word-prefix extension of
|
||||
// "dressage" above (see `cook`'s comment for why that duplicates NER
|
||||
// candidates).
|
||||
synonyms: ["dresser", "dressez", "dressage", "disposer dans l'assiette"],
|
||||
utterances: [
|
||||
"dresser harmonieusement dans les assiettes",
|
||||
"disposer joliment sur l'assiette avant de servir",
|
||||
"présenter avec soin au centre de l'assiette",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
// NOT "plate up"/"plated nicely" — both are word-prefix extensions of
|
||||
// "plate"/"plated" above (see `cook`'s comment for why that
|
||||
// duplicates NER candidates).
|
||||
synonyms: ["plate", "plates", "plated", "plating"],
|
||||
utterances: [
|
||||
"plate it up nicely",
|
||||
"arrange it neatly on the plate before serving",
|
||||
"present it carefully in the center of the plate",
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
uid: "coat",
|
||||
fr: {
|
||||
synonyms: [
|
||||
"napper",
|
||||
"nappez",
|
||||
"nappé",
|
||||
"nappée",
|
||||
"nappées",
|
||||
"nappage",
|
||||
"enrober",
|
||||
"enrobez",
|
||||
"enrobé",
|
||||
"enrobée",
|
||||
"enrobées",
|
||||
],
|
||||
utterances: [
|
||||
"napper le gâteau de chocolat fondu",
|
||||
"recouvrir uniformément d'une fine couche de sauce",
|
||||
"verser la sauce par-dessus pour bien enrober",
|
||||
],
|
||||
},
|
||||
en: {
|
||||
// NOT "coat evenly" — a word-prefix extension of "coat" above (see
|
||||
// `cook`'s comment for why that duplicates NER candidates).
|
||||
synonyms: ["coat", "coats", "coated", "coating", "dredge", "dredged", "dredging"],
|
||||
utterances: [
|
||||
"coat the cake with melted chocolate",
|
||||
"cover evenly with a thin layer of sauce",
|
||||
"pour the sauce over it so it's well covered",
|
||||
],
|
||||
},
|
||||
},
|
||||
];
|
||||
|
|
@ -6,8 +6,8 @@ import { renumberStepTechSteps } from "../modules/recipe/recipe-tech-step-correc
|
|||
/**
|
||||
* Recomputes every existing `Step`'s `"auto"`-sourced `StepTechStep`
|
||||
* entries against the *current* classifier
|
||||
* (`tech-step-matcher.ts`/`tech-step-training-data.ts`), the same way
|
||||
* `updateRecipe` does when a user resaves a recipe through the UI —
|
||||
* (`tech-step-matcher.ts`, delegating to `services/tech-step-intent-service`),
|
||||
* the same way `updateRecipe` does when a user resaves a recipe through the UI —
|
||||
* always `"fr"` (`DEFAULT_TECH_STEP_LOCALE` in `recipe.service.ts`; there's
|
||||
* no persisted per-recipe locale to recover for a step that already
|
||||
* exists, so this matches real resave behavior exactly rather than
|
||||
|
|
|
|||
101
apps/api/src/scripts/calibrate-tech-step-threshold.ts
Normal file
101
apps/api/src/scripts/calibrate-tech-step-threshold.ts
Normal file
|
|
@ -0,0 +1,101 @@
|
|||
import { prisma } from "../db/prisma.js";
|
||||
import { TECH_STEP_EVAL_DATASET } from "../lib/recipe-matching/tech-step-eval-dataset.js";
|
||||
import {
|
||||
computeTechStepMetrics,
|
||||
type TechStepEvalOutcome,
|
||||
} from "../lib/recipe-matching/tech-step-evaluator.js";
|
||||
import { techStepClassifier } from "../lib/recipe-matching/tech-step-matcher.js";
|
||||
|
||||
/**
|
||||
* Candidate thresholds to sweep, `0.05` to `0.95` in `0.05` steps — fine
|
||||
* enough to find a good value without an unreasonable number of full
|
||||
* `TECH_STEP_EVAL_DATASET` passes (each threshold only needs one
|
||||
* {@link techStepClassifier.classifyClauses} call per eval case, not a
|
||||
* retrain — see this file's own doc comment for why).
|
||||
*/
|
||||
const CANDIDATE_THRESHOLDS = Array.from({ length: 19 }, (_, i) => Math.round((i + 1) * 5) / 100);
|
||||
|
||||
/**
|
||||
* One-off maintainer tool for recalibrating `CONFIDENCE_THRESHOLD`
|
||||
* (`tech-step-matcher.ts`) after a change to the underlying intent
|
||||
* classifier — most notably, the migration from `node-nlp` to
|
||||
* `services/tech-step-intent-service` (spaCy): a different model produces a
|
||||
* differently-shaped confidence score distribution, so a threshold tuned
|
||||
* against the old classifier has no reason to still be the right cutoff for
|
||||
* the new one.
|
||||
*
|
||||
* Reuses `techStepClassifier.classifyClauses` — already public, and
|
||||
* deliberately *not* threshold-applied (see that method's own doc comment)
|
||||
* — to get every eval case's raw `{anchorUid, intentUid, score}` per clause
|
||||
* exactly once, then replays `_classifyClause`'s own decision rule
|
||||
* (`intentUid` if confident enough, `anchorUid` otherwise) locally in this
|
||||
* script for every candidate threshold. This is what makes a full sweep
|
||||
* cheap: one classifier pass per eval case regardless of how many
|
||||
* thresholds are being compared, rather than one full pass *per threshold*.
|
||||
*
|
||||
* Prints a threshold -> precision/recall/F1 table and the threshold that
|
||||
* maximizes aggregate F1 — does **not** edit `tech-step-matcher.ts` itself.
|
||||
* A maintainer reads the table, updates `CONFIDENCE_THRESHOLD` by hand (with
|
||||
* an updated doc comment recording what run/F1 the new value was calibrated
|
||||
* against, same as the existing comment's own format), then re-runs
|
||||
* `retrain-tech-steps.ts` to confirm the change clears `MIN_OVERALL_F1`.
|
||||
*
|
||||
* Usage:
|
||||
*
|
||||
* pnpm --filter api exec tsx src/scripts/calibrate-tech-step-threshold.ts
|
||||
*/
|
||||
async function calibrateTechStepThreshold(): Promise<void> {
|
||||
console.info(`Classifying ${TECH_STEP_EVAL_DATASET.length} eval case(s)...`);
|
||||
|
||||
// One classifier pass per eval case, all clauses' raw verdicts kept
|
||||
// alongside the case's own `expectedKeys` — reused for every candidate
|
||||
// threshold in the loop below.
|
||||
const casesWithClauses = await Promise.all(
|
||||
TECH_STEP_EVAL_DATASET.map(async (evalCase) => ({
|
||||
expectedKeys: evalCase.expectedKeys,
|
||||
clauses: await techStepClassifier.classifyClauses(evalCase.description, evalCase.locale),
|
||||
})),
|
||||
);
|
||||
|
||||
console.info("\nthreshold precision recall f1");
|
||||
let bestThreshold = CANDIDATE_THRESHOLDS[0] ?? 0;
|
||||
let bestF1 = -1;
|
||||
|
||||
for (const threshold of CANDIDATE_THRESHOLDS) {
|
||||
const outcomes: TechStepEvalOutcome[] = casesWithClauses.map(({ expectedKeys, clauses }) => {
|
||||
const actualKeys = clauses
|
||||
// Mirrors `_classifyClause`'s own decision rule exactly (see that
|
||||
// method, `tech-step-matcher.ts`) — the classifier's own verdict
|
||||
// when confident enough, otherwise its clause's NER anchor, `null`
|
||||
// when neither applies (no keyword, no confident classification).
|
||||
.map((clause) =>
|
||||
clause.intentUid !== null && clause.score >= threshold
|
||||
? clause.intentUid
|
||||
: clause.anchorUid,
|
||||
)
|
||||
.filter((key): key is string => key !== null);
|
||||
return { expectedKeys, actualKeys };
|
||||
});
|
||||
|
||||
const { overall } = computeTechStepMetrics(outcomes);
|
||||
console.info(
|
||||
`${threshold.toFixed(2)} ${overall.precision.toFixed(3)} ${overall.recall.toFixed(3)} ${overall.f1.toFixed(3)}`,
|
||||
);
|
||||
if (overall.f1 > bestF1) {
|
||||
bestF1 = overall.f1;
|
||||
bestThreshold = threshold;
|
||||
}
|
||||
}
|
||||
|
||||
console.info(
|
||||
`\nBest aggregate F1 ${bestF1.toFixed(3)} at threshold ${bestThreshold.toFixed(2)} — update CONFIDENCE_THRESHOLD in tech-step-matcher.ts by hand if this differs from the current value.`,
|
||||
);
|
||||
}
|
||||
|
||||
calibrateTechStepThreshold()
|
||||
.then(() => prisma.$disconnect())
|
||||
.catch(async (err) => {
|
||||
console.error(err);
|
||||
await prisma.$disconnect();
|
||||
process.exit(1);
|
||||
});
|
||||
|
|
@ -6,14 +6,15 @@ import { prisma } from "../db/prisma.js";
|
|||
* comment) — generated by `services/tech-step-llm-worker`'s scheduled
|
||||
* jobs, from either a user correction or the worker's own low-confidence
|
||||
* audit (`sourceType`). What a maintainer reads *before* hand-editing
|
||||
* `tech-step-training-data.ts` and running `retrain-tech-steps.ts` — this
|
||||
* script never writes anything, purely a read-only report to stdout:
|
||||
* `services/tech-step-intent-service/intent_service/training_data.py` and
|
||||
* running `retrain-tech-steps.ts` — this script never writes anything,
|
||||
* purely a read-only report to stdout:
|
||||
*
|
||||
* pnpm --filter api exec tsx src/scripts/list-pending-training-suggestions.ts
|
||||
*
|
||||
* Grouped by technique key so every suggestion for the same entry in
|
||||
* `TECH_STEP_TRAINING_DATA` is read together, matching how that file
|
||||
* itself is organized (one block per technique).
|
||||
* `training_data.py`'s `TECH_STEP_TRAINING_DATA` is read together, matching
|
||||
* how that file itself is organized (one block per technique).
|
||||
*/
|
||||
async function listPendingTrainingSuggestions(): Promise<void> {
|
||||
const suggestions = await prisma.techStepTrainingSuggestion.findMany({
|
||||
|
|
|
|||
|
|
@ -28,10 +28,15 @@ function parseSuggestionIds(flag: "applied" | "rejected"): number[] {
|
|||
* Maintainer workflow closing the loop on a training-corpus change (see
|
||||
* this feature's plan document):
|
||||
*
|
||||
* 1. A maintainer has already hand-edited `tech-step-training-data.ts`
|
||||
* (informed by `list-pending-training-suggestions.ts`'s report), and
|
||||
* 1. A maintainer has already hand-edited
|
||||
* `services/tech-step-intent-service/intent_service/training_data.py`
|
||||
* (informed by `list-pending-training-suggestions.ts`'s report),
|
||||
* decided which `TechStepTrainingSuggestion` ids they incorporated
|
||||
* (`--applied=`) or explicitly discarded (`--rejected=`).
|
||||
* (`--applied=`) or explicitly discarded (`--rejected=`), **and
|
||||
* restarted `tech-step-intent-service`** so it retrains from the
|
||||
* edited corpus — that service only ever trains once, at its own
|
||||
* startup (see its README), so this script's eval gate below is
|
||||
* meaningless against a service still running the old corpus.
|
||||
* 2. This script re-runs the F1 regression gate
|
||||
* ({@link runTechStepEvalSuite} against {@link MIN_OVERALL_F1}) —
|
||||
* refuses to backfill at all if the edited corpus scores worse than
|
||||
|
|
|
|||
|
|
@ -9,21 +9,44 @@ import { registerAllRecipeSources } from "./sources/index.js";
|
|||
// doesn't happen inside app.ts/createServer() itself.
|
||||
registerAllRecipeSources();
|
||||
|
||||
// Trains the tech-step classifier (and pays node-nlp's own one-time lazy
|
||||
// setup cost — see `TechStepClassifierService.warmUp`) before accepting
|
||||
// any traffic, so the first real recipe save/preview isn't the one stuck
|
||||
// waiting several seconds for it.
|
||||
/**
|
||||
* Trains the tech-step classifier (a `POST /v1/train` round-trip per locale
|
||||
* to `services/tech-step-intent-service` — see
|
||||
* `TechStepClassifierService.warmUp`) before accepting any traffic, so the
|
||||
* first real recipe save/preview isn't the one stuck waiting for it.
|
||||
*
|
||||
* Retried with exponential backoff: in Docker Compose, `app`'s own
|
||||
* `depends_on: tech-step-intent-service: condition: service_healthy`
|
||||
* (`docker-compose.yml`) already means that service is up by the time this
|
||||
* runs, but native dev (`pnpm dev:api`, no Compose ordering at all) can
|
||||
* easily start this before the intent service has finished loading its
|
||||
* spaCy models — a transient connection failure here shouldn't need a
|
||||
* manual restart. Still non-fatal after every attempt is exhausted: the
|
||||
* *next* real call retries training itself (see `_ensureTrained`'s own
|
||||
* retry-on-failure comment), same graceful-degrade posture as before this
|
||||
* retry loop existed.
|
||||
*/
|
||||
async function warmUpTechStepClassifier(): Promise<void> {
|
||||
const maxAttempts = 5;
|
||||
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
|
||||
try {
|
||||
await techStepClassifier.warmUp();
|
||||
return;
|
||||
} catch (err) {
|
||||
// Not fatal to startup — a failed warm-up just means the *next* call
|
||||
// retries training itself (see `_ensureTrained`'s own retry-on-failure
|
||||
// comment), same graceful-degrade posture as everywhere else training
|
||||
// failures surface. Still worth a loud log: this shouldn't normally fail.
|
||||
logger.error("Tech-step classifier warm-up failed", {
|
||||
if (attempt === maxAttempts) {
|
||||
logger.error("Tech-step classifier warm-up failed after retries", {
|
||||
error: err instanceof Error ? err.message : String(err),
|
||||
attempts: attempt,
|
||||
});
|
||||
return;
|
||||
}
|
||||
const delayMs = 1000 * 2 ** (attempt - 1);
|
||||
await new Promise((resolve) => setTimeout(resolve, delayMs));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
await warmUpTechStepClassifier();
|
||||
|
||||
const server = createServer();
|
||||
|
||||
|
|
|
|||
50
apps/api/src/types/node-nlp.d.ts
vendored
50
apps/api/src/types/node-nlp.d.ts
vendored
|
|
@ -1,50 +0,0 @@
|
|||
/**
|
||||
* Minimal ambient typing for `node-nlp` (no official/DefinitelyTyped types
|
||||
* exist for it) — declares only the `NlpManager` surface
|
||||
* `tech-step-matcher.ts` actually calls, verified against the real
|
||||
* package (v4.27.0) rather than the library's full documented API, which
|
||||
* this repo doesn't use the rest of.
|
||||
*/
|
||||
declare module "node-nlp" {
|
||||
/** Constructor options this repo passes — `NlpManager` accepts more, only what's used here is typed. */
|
||||
export interface NlpManagerOptions {
|
||||
languages?: string[];
|
||||
forceNER?: boolean;
|
||||
nlu?: { log?: boolean };
|
||||
ner?: { threshold?: number };
|
||||
/** Defaults to `true` — persists the trained model to `modelFileName` (default `model.nlp`, in `process.cwd()`). See `tech-step-matcher.ts`'s own constructor comment for why this repo always sets it `false`. */
|
||||
autoSave?: boolean;
|
||||
/** Defaults to `true` — loads from `modelFileName` instead of training fresh if that file already exists. Always `false` here, same reasoning as `autoSave`. */
|
||||
autoLoad?: boolean;
|
||||
}
|
||||
|
||||
/** One entity `NlpManager.process`'s result reports — see `tech-step-matcher.ts`'s own `NerEntity` for the subset this repo reads. */
|
||||
export interface NlpEntity {
|
||||
entity: string;
|
||||
start: number;
|
||||
end: number;
|
||||
type: string;
|
||||
accuracy?: number;
|
||||
sourceText?: string;
|
||||
}
|
||||
|
||||
/** `NlpManager.process`'s result — trimmed to the fields this repo reads (the real object carries many more). */
|
||||
export interface NlpProcessResult {
|
||||
intent: string;
|
||||
score: number;
|
||||
entities: NlpEntity[];
|
||||
}
|
||||
|
||||
export class NlpManager {
|
||||
public constructor(options?: NlpManagerOptions);
|
||||
public addNamedEntityText(
|
||||
entityName: string,
|
||||
optionName: string,
|
||||
languages: string[],
|
||||
texts: string[],
|
||||
): void;
|
||||
public addDocument(locale: string, utterance: string, intent: string): void;
|
||||
public train(): Promise<void>;
|
||||
public process(locale: string, text: string): Promise<NlpProcessResult>;
|
||||
}
|
||||
}
|
||||
40
apps/api/test-support/mocha-root-hooks.ts
Normal file
40
apps/api/test-support/mocha-root-hooks.ts
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
import { techStepClassifier } from "../src/lib/recipe-matching/tech-step-matcher.js";
|
||||
import { resetDatabase } from "./reset-db.js";
|
||||
|
||||
/**
|
||||
* Mocha root hook plugin (see `.mocharc.json`'s `require`) — runs once
|
||||
* before every test file's own suites, regardless of load order.
|
||||
*
|
||||
* Warms up `techStepClassifier` here — resolving the `TechStep.key -> id`
|
||||
* lookup from the DB (see `TechStepClassifierService._loadTechStepIds`) —
|
||||
* instead of leaving it to happen lazily on whichever test file Mocha
|
||||
* happens to load first, same as `server.ts` does before the real server
|
||||
* ever accepts traffic. Fast by itself (one DB query, one HTTP call to
|
||||
* `services/tech-step-intent-service`): that service now trains itself
|
||||
* entirely at its own process startup (see its own README), so unlike
|
||||
* before this migration, nothing here waits on a slow training pass — CI's
|
||||
* own "wait for `/health`" step (`.github/workflows/ci.yml`) is what
|
||||
* ensures that service is already fully trained before `pnpm --filter api
|
||||
* test` even starts.
|
||||
*
|
||||
* `resetDatabase()` runs first, deliberately: id resolution needs
|
||||
* `TechStep` rows, and a freshly-migrated (never-seeded) test database has
|
||||
* none yet. Every per-test `beforeEach` in this suite already calls
|
||||
* `resetDatabase()` again before its own test, which is a no-op
|
||||
* duplication of effort but not a correctness problem: `TRUNCATE ...
|
||||
* RESTART IDENTITY` plus deterministic re-seeding (`seedReferenceData`)
|
||||
* assigns the exact same ids every time, so the `uid -> id` map memoized
|
||||
* here from this first reset stays valid for every reset after it.
|
||||
*/
|
||||
export const mochaHooks = {
|
||||
// biome-ignore lint/suspicious/noExplicitAny: Mocha's root hook `this` (a Context with `.timeout()`) isn't typed without @types/mocha (not a dependency here) — same untyped-`this` shape already used in tech-step-worker.routes.test.ts.
|
||||
async beforeAll(this: any): Promise<void> {
|
||||
// A little more generous than Mocha's normal 10s per-test default
|
||||
// (`.mocharc.json`) purely for a slower/contended CI runner's first
|
||||
// network round-trip to `services/tech-step-intent-service` — not
|
||||
// because anything here waits on training anymore.
|
||||
this.timeout(30000);
|
||||
await resetDatabase();
|
||||
await techStepClassifier.warmUp();
|
||||
},
|
||||
};
|
||||
|
|
@ -38,8 +38,9 @@ describe("recipe-translation", () => {
|
|||
// `translateRecipeSteps` now goes through `techStepClassifier` (a
|
||||
// trained model, not a pure regex test against a caller-supplied
|
||||
// mapping list — see `tech-step-matcher.ts`), so these tests exercise
|
||||
// the real training corpus (`tech-step-training-data.ts`) against a real
|
||||
// `TechStep` catalog rather than synthetic fixtures — same posture
|
||||
// the real training corpus (`services/tech-step-intent-service`'s
|
||||
// `training_data.py`) against a real `TechStep` catalog rather than
|
||||
// synthetic fixtures — same posture
|
||||
// `tech-step-matcher.test.ts`'s own `techStepClassifier` describe block
|
||||
// takes, for the same reason.
|
||||
describe("translateRecipeSteps", () => {
|
||||
|
|
|
|||
|
|
@ -118,13 +118,16 @@ describe("tech-step-matcher", () => {
|
|||
// `techStepClassifier` is the one shared singleton (see
|
||||
// tech-step-matcher.ts's own doc comment on why) — these tests
|
||||
// exercise it against the real training corpus
|
||||
// (`tech-step-training-data.ts`) and the real seeded `TechStep`
|
||||
// catalog, rather than synthetic injectable fixtures the old
|
||||
// regex-based `matchTechStepSpans(description, mappings)` allowed.
|
||||
// Training + node-nlp's own one-time per-language setup can take a
|
||||
// few seconds on the very first call in the whole suite (subsequent
|
||||
// calls reuse the same trained model and are fast) — comfortably
|
||||
// inside this suite's default 10s timeout (.mocharc.json).
|
||||
// (`services/tech-step-intent-service`'s `training_data.py`) and the
|
||||
// real seeded `TechStep` catalog, rather than synthetic injectable
|
||||
// fixtures the old regex-based `matchTechStepSpans(description,
|
||||
// mappings)` allowed. Every call round-trips over HTTP to a real,
|
||||
// locally running `services/tech-step-intent-service` (see that
|
||||
// service's own README and `apps/api/.env.test`) — that service trains
|
||||
// itself once at its own startup (`test-support/mocha-root-hooks.ts`'s
|
||||
// root hook doesn't wait on it, CI's own "wait for /health" step
|
||||
// already does), so calls here are just a normal HTTP round-trip,
|
||||
// comfortably inside this suite's default 10s timeout (.mocharc.json).
|
||||
let simmerId: number;
|
||||
let cookId: number;
|
||||
let bakeId: number;
|
||||
|
|
|
|||
|
|
@ -79,7 +79,7 @@ describe("Recipe tech-step corrections", () => {
|
|||
const { agent, profileId } = await signup();
|
||||
// "Faire mijoter la sauce." names no technique the classifier itself
|
||||
// registers a bare-word anchor for at this exact span in isolation
|
||||
// (see tech-step-training-data.ts) — irrelevant here either way,
|
||||
// (see services/tech-step-intent-service's training_data.py) — irrelevant here either way,
|
||||
// since this test's whole point is the *manual* addition, not
|
||||
// whatever the classifier does or doesn't auto-detect for it.
|
||||
const { recipeId, stepId } = await createPublicRecipeWithStep(profileId);
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ import request from "supertest";
|
|||
import { createApp } from "../src/app.js";
|
||||
import { prisma } from "../src/db/prisma.js";
|
||||
import { syncRecipeSources } from "../src/db/recipe-source-sync.js";
|
||||
import { seedReferenceData } from "../src/db/reference-seed-data.js";
|
||||
import { seedReferenceData, TECH_STEPS } from "../src/db/reference-seed-data.js";
|
||||
import type { RecipeSourceAdapter } from "../src/lib/recipe-sources/recipe-source-adapter.js";
|
||||
import {
|
||||
clearRecipeSources,
|
||||
|
|
@ -137,7 +137,9 @@ describe("Reference data", () => {
|
|||
const res = await request(app).get("/reference/tech-steps");
|
||||
|
||||
expect(res.status).to.equal(200);
|
||||
expect(res.body).to.have.length(26);
|
||||
// `TECH_STEPS.length` (reference-seed-data.ts), not a hardcoded
|
||||
// number — this catalog has grown since (26 -> 74) and will again.
|
||||
expect(res.body).to.have.length(TECH_STEPS.length);
|
||||
expect(res.body.map((t: { key: string }) => t.key)).to.include("simmer");
|
||||
expect(res.body[0]).to.have.keys(["id", "key"]);
|
||||
});
|
||||
|
|
@ -155,7 +157,7 @@ describe("Reference data", () => {
|
|||
await seedReferenceData(prisma);
|
||||
|
||||
const res = await request(app).get("/reference/tech-steps");
|
||||
expect(res.body).to.have.length(26);
|
||||
expect(res.body).to.have.length(TECH_STEPS.length);
|
||||
});
|
||||
});
|
||||
|
||||
|
|
|
|||
|
|
@ -425,7 +425,55 @@
|
|||
"preheat": "Préchauffer",
|
||||
"bake": "Cuire au four",
|
||||
"plate": "Dresser",
|
||||
"coat": "Napper"
|
||||
"coat": "Napper",
|
||||
"baste": "Arroser",
|
||||
"appertize": "Appertiser",
|
||||
"whiskPale": "Blanchir (jaunes d'œufs)",
|
||||
"goldenBrown": "Blondir",
|
||||
"braise": "Braiser",
|
||||
"truss": "Brider",
|
||||
"caramelize": "Caraméliser",
|
||||
"score": "Cerner",
|
||||
"lineMold": "Chemiser",
|
||||
"clarify": "Clarifier",
|
||||
"compote": "Compoter",
|
||||
"concasse": "Concasser",
|
||||
"confit": "Confire",
|
||||
"julienne": "Couper en julienne",
|
||||
"brunoise": "Couper en brunoise",
|
||||
"mirepoix": "Couper en mirepoix",
|
||||
"paysanne": "Couper en paysanne",
|
||||
"blindBake": "Cuire à blanc",
|
||||
"bainMarie": "Cuire au bain-marie",
|
||||
"smother": "Cuire à l'étouffée",
|
||||
"decant": "Décanter",
|
||||
"dilute": "Délayer",
|
||||
"punchDown": "Dégazer",
|
||||
"disgorge": "Dégorger",
|
||||
"loosen": "Détendre",
|
||||
"shellEgg": "Écaler",
|
||||
"scald": "Échauder",
|
||||
"pod": "Écosser",
|
||||
"emulsify": "Émulsionner",
|
||||
"hollowOut": "Évider",
|
||||
"shock": "Frapper",
|
||||
"setGel": "Gélifier",
|
||||
"glaze": "Glacer",
|
||||
"thicken": "Lier",
|
||||
"filet": "Lever les filets",
|
||||
"proof": "Laisser pousser",
|
||||
"peelBlanch": "Monder",
|
||||
"whipUp": "Monter",
|
||||
"moisten": "Mouiller",
|
||||
"pasteurize": "Pasteuriser",
|
||||
"poach": "Pocher",
|
||||
"reduce": "Réduire",
|
||||
"rubIn": "Sabler",
|
||||
"dustWithFlour": "Singer",
|
||||
"sweat": "Suer",
|
||||
"sift": "Tamiser",
|
||||
"toast": "Torréfier",
|
||||
"zest": "Zester"
|
||||
},
|
||||
"allergens": {
|
||||
"gluten": "Gluten",
|
||||
|
|
|
|||
|
|
@ -27,9 +27,6 @@ services:
|
|||
context: .
|
||||
dockerfile: apps/api/Dockerfile
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
NODE_ENV: production
|
||||
PORT: 3000
|
||||
|
|
@ -51,8 +48,61 @@ services:
|
|||
# default: `/internal/tech-steps/*` fails closed rather than open
|
||||
# for a deployment that doesn't run the worker at all.
|
||||
INTERNAL_WORKER_SECRET: ${INTERNAL_WORKER_SECRET:-}
|
||||
# Compose network service name, not localhost — same reasoning as
|
||||
# DATABASE_URL above. Unlike INTERNAL_WORKER_SECRET, no `:-` fallback:
|
||||
# tech-step-intent-service is a core dependency (see its own entry
|
||||
# below), not an optional background job.
|
||||
INTENT_SERVICE_BASE_URL: "http://tech-step-intent-service:8000"
|
||||
INTENT_SERVICE_SECRET: ${INTENT_SERVICE_SECRET:?set INTENT_SERVICE_SECRET in .env}
|
||||
ports:
|
||||
- "${APP_PORT:-3000}:3000"
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
tech-step-intent-service:
|
||||
condition: service_healthy
|
||||
|
||||
# spaCy-based NER + intent classification microservice
|
||||
# (services/tech-step-intent-service) — `app` delegates all tech-step
|
||||
# detection to it over HTTP (see `IntentServiceClient`,
|
||||
# apps/api/src/lib/recipe-matching/intent-service-client.ts). Unlike
|
||||
# `tech-step-llm-worker` below, **not optional**: without it, `app` can no
|
||||
# longer detect any cooking technique in a recipe step at all. No exposed
|
||||
# port — reachable only from `app` on the compose network, nothing ever
|
||||
# calls into it from outside.
|
||||
tech-step-intent-service:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: services/tech-step-intent-service/Dockerfile
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
INTENT_SERVICE_SECRET: ${INTENT_SERVICE_SECRET:?set INTENT_SERVICE_SECRET in .env}
|
||||
healthcheck:
|
||||
# No curl/wget in the python:3.12-slim base image — a one-line Python
|
||||
# request is the healthcheck for a service that's already guaranteed
|
||||
# to have Python (see this service's Dockerfile).
|
||||
test:
|
||||
[
|
||||
"CMD",
|
||||
"python",
|
||||
"-c",
|
||||
"import urllib.request; urllib.request.urlopen('http://localhost:8000/health', timeout=2)",
|
||||
]
|
||||
interval: 15s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
# This service trains itself from scratch on every start (no model
|
||||
# ever persisted to disk, see its own README) — `/health` only
|
||||
# returns 200 once that's done, not just once the base spaCy models
|
||||
# are loaded. Measured at ~335s per locale (~670s for fr+en combined)
|
||||
# against the current ~74-technique corpus, trained on each
|
||||
# technique's own synonyms in addition to its example phrases
|
||||
# (`intent_service/locale_pipeline.py`'s `_TRAINING_ITERATIONS`) —
|
||||
# `start_period` generous enough that failing checks during that
|
||||
# whole window never count against `retries` (which would otherwise
|
||||
# flip this container to "unhealthy" mid-training, blocking `app`'s
|
||||
# own `depends_on: condition: service_healthy` indefinitely).
|
||||
start_period: 900s
|
||||
|
||||
# Deliberately its own image, not built into `app`'s (see
|
||||
# services/tech-step-llm-worker/Dockerfile's own doc comment) — a
|
||||
|
|
|
|||
696
pnpm-lock.yaml
696
pnpm-lock.yaml
|
|
@ -44,9 +44,6 @@ importers:
|
|||
jsonwebtoken:
|
||||
specifier: ^9.0.3
|
||||
version: 9.0.3
|
||||
node-nlp:
|
||||
specifier: 4.27.0
|
||||
version: 4.27.0
|
||||
prisma:
|
||||
specifier: ^5.22.0
|
||||
version: 5.22.0
|
||||
|
|
@ -853,224 +850,12 @@ packages:
|
|||
resolution: {integrity: sha512-Yhlar6v9WQgUp/He7BdgzOz8lqMQ8sU+jkCq7Wx8Myc5YFJLbEe7lgui/V7G1qB1DJykHSGwreceSaD60Y0PUQ==, tarball: https://registry.npmjs.org/@mapbox/node-pre-gyp/-/node-pre-gyp-1.0.11.tgz}
|
||||
hasBin: true
|
||||
|
||||
'@microsoft/recognizers-text-choice@1.3.1':
|
||||
resolution: {integrity: sha512-HubunMJVq/OetmdvcAmBh5skMlg+yiScm3V2wNyNZIVvLgli4+8nzbg/W/fI9dpaf6wv9ZQ7d2IYvn8swJBo3A==, tarball: https://registry.npmjs.org/@microsoft/recognizers-text-choice/-/recognizers-text-choice-1.3.1.tgz}
|
||||
engines: {node: '>=10.3.0'}
|
||||
|
||||
'@microsoft/recognizers-text-data-types-timex-expression@1.3.1':
|
||||
resolution: {integrity: sha512-jarJIFIJZBqeofy3hh0vdQo1yOmTM+jCjj6/zmo9JunsQ6LO750eZHCg9eLptQhsvq321XCt5xdRNLCwU8YeNA==, tarball: https://registry.npmjs.org/@microsoft/recognizers-text-data-types-timex-expression/-/recognizers-text-data-types-timex-expression-1.3.1.tgz}
|
||||
engines: {node: '>=10.3.0'}
|
||||
|
||||
'@microsoft/recognizers-text-date-time@1.3.2':
|
||||
resolution: {integrity: sha512-fUEGOTccS55ZY0erzjS1bunJYA9lGXjcZoru5oPOlnxbJS4Lk0ylgdH2Ub2EjAyqr8DIJhdLNOEesCdAXMvlNg==, tarball: https://registry.npmjs.org/@microsoft/recognizers-text-date-time/-/recognizers-text-date-time-1.3.2.tgz}
|
||||
engines: {node: '>=10.3.0'}
|
||||
|
||||
'@microsoft/recognizers-text-number-with-unit@1.3.1':
|
||||
resolution: {integrity: sha512-gzCpPP4zQ5Vb+RHaWjzP2t1c+mj6GYOsFoI2NyJkm8OZ52XI+x9SJCgrrD2ujzjOd5/CQVC46rE22rfGwXLDkA==, tarball: https://registry.npmjs.org/@microsoft/recognizers-text-number-with-unit/-/recognizers-text-number-with-unit-1.3.1.tgz}
|
||||
engines: {node: '>=10.3.0'}
|
||||
|
||||
'@microsoft/recognizers-text-number@1.3.1':
|
||||
resolution: {integrity: sha512-JBxhSdihdQLQilCtqISEBw5kM+CNGTXzy5j5hNoZECNUEvBUPkAGNEJAeQPMP5abrYks29aSklnSvSyLObXaNQ==, tarball: https://registry.npmjs.org/@microsoft/recognizers-text-number/-/recognizers-text-number-1.3.1.tgz}
|
||||
engines: {node: '>=10.3.0'}
|
||||
|
||||
'@microsoft/recognizers-text-sequence@1.3.1':
|
||||
resolution: {integrity: sha512-J7Kg35hpm0NcFHmu69Bb4q7DPDiSpCd8ApUZqNm59itIjrQJHpSdl9HF6JxuQQz0Ftc/li5ZLqSuupJAmA/sgg==, tarball: https://registry.npmjs.org/@microsoft/recognizers-text-sequence/-/recognizers-text-sequence-1.3.1.tgz}
|
||||
engines: {node: '>=10.3.0'}
|
||||
|
||||
'@microsoft/recognizers-text-suite@1.3.0':
|
||||
resolution: {integrity: sha512-uqG4vzy5N2CmBaeINny0bLdnGp0jDbT1moNoLC+Yim3G8kHOU9lpDfwA6VN6HTYaDM5854SNMEzLjJdS1TPFTw==, tarball: https://registry.npmjs.org/@microsoft/recognizers-text-suite/-/recognizers-text-suite-1.3.0.tgz}
|
||||
engines: {node: '>=10.3.0'}
|
||||
|
||||
'@microsoft/recognizers-text@1.3.1':
|
||||
resolution: {integrity: sha512-HikLoRUgSzM4OKP3JVBzUUp3Q7L4wgI17p/3rERF01HVmopcujY3i6wgx8PenCwbenyTNxjr1AwSDSVuFlYedQ==, tarball: https://registry.npmjs.org/@microsoft/recognizers-text/-/recognizers-text-1.3.1.tgz}
|
||||
engines: {node: '>=10.3.0'}
|
||||
|
||||
'@napi-rs/lzma-linux-x64-gnu@1.5.1':
|
||||
resolution: {integrity: sha512-oTXEIha4SsuXdTA4Iyskj0kpdx2yVXdhd75c2v3xGrHFfVMsbhTPZU/nMPL4sWKo4pBHm3aucLaqGlF696dTyQ==, tarball: https://registry.npmjs.org/@napi-rs/lzma-linux-x64-gnu/-/lzma-linux-x64-gnu-1.5.1.tgz}
|
||||
engines: {node: ^22.20 || ^24.12 || >=25}
|
||||
cpu: [x64]
|
||||
os: [linux]
|
||||
|
||||
'@nlpjs/builtin-duckling@4.26.1':
|
||||
resolution: {integrity: sha512-3qkH955X2g5MXV1EqT3fTAT/lLEdiqqe5IgBDyr+MQB7FOV9R3YhqGIn3DFOl+TSm/tP5n/BAEptkTNn/TOpmQ==, tarball: https://registry.npmjs.org/@nlpjs/builtin-duckling/-/builtin-duckling-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/builtin-microsoft@4.26.1':
|
||||
resolution: {integrity: sha512-AODgzTcfYUf5Ozm00aQnHImDum7Idtl0F9dSPoaXpfj7rZqP8hPZ7iWwdGTAvISH/da2YhjPOU65QSYk2YpjFA==, tarball: https://registry.npmjs.org/@nlpjs/builtin-microsoft/-/builtin-microsoft-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/core-loader@4.26.1':
|
||||
resolution: {integrity: sha512-IiRtn65bdiUSQHy2kusco2fmhk39u2Mc2c5Fsm9+9EVG6BtJCmVEFU/btAzGDAmxEA/E4qKecaAT4LvcW6TPbA==, tarball: https://registry.npmjs.org/@nlpjs/core-loader/-/core-loader-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/core@4.26.1':
|
||||
resolution: {integrity: sha512-M/PeFddsi3y7Z1piFJxsLGm5/xdMhcrpOsml7s6CTEgYo8iduaT30HDd61tZxDyvvJseU6uFqlXSn7XKkAcC1g==, tarball: https://registry.npmjs.org/@nlpjs/core/-/core-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/emoji@4.26.1':
|
||||
resolution: {integrity: sha512-Q0PoXwIvaB1bnRXK4U/YD7mrqaz29Yfed3s2au0iXl1bffUgoG+hs4GORCvyy7DFCCLlc9d5yDM3oLIX/ggZ+Q==, tarball: https://registry.npmjs.org/@nlpjs/emoji/-/emoji-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/evaluator@4.26.1':
|
||||
resolution: {integrity: sha512-WeUrC8qq7+V8Jhkkjc2yiXdzy9V0wbETv8/qasQmL0QmEuwBDJF+fvfl4z2vWpBb0vW07A8aNrFElKELzbpkdg==, tarball: https://registry.npmjs.org/@nlpjs/evaluator/-/evaluator-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-all@4.26.1':
|
||||
resolution: {integrity: sha512-UzRm1JRRAyQqilEOxQ2ySMOitKbhPk5iKYbjD8FREDcPjreUvDxVuQsYUOvYucmEyFcZU2U/TdJx+fX9/bcaKQ==, tarball: https://registry.npmjs.org/@nlpjs/lang-all/-/lang-all-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-ar@4.26.1':
|
||||
resolution: {integrity: sha512-MUlVtabt9ltG7WyzCQpFJymLJlnEqp3mxhgN9JHyFH7oZMK3REvMovFfvEUAbfiYrJEv/BN5KKLL7yrvUeaHtg==, tarball: https://registry.npmjs.org/@nlpjs/lang-ar/-/lang-ar-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-bn@4.26.1':
|
||||
resolution: {integrity: sha512-sim1iZKBDdehi/yBUKrLW51QvS9uB+sXW7lj+THVqBy5UsnEQvt4gzE0NsC873uJMh66vt2AlHkhzgPH0qH/nQ==, tarball: https://registry.npmjs.org/@nlpjs/lang-bn/-/lang-bn-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-ca@4.26.1':
|
||||
resolution: {integrity: sha512-fD4R5tcAB0uYtNxSEF20b1KmF6nUQSbiJqrIUJI5yis4ObjCYRQnSh4bjVDKUKxyONjbD6L8EaK5GrY1/jkwFQ==, tarball: https://registry.npmjs.org/@nlpjs/lang-ca/-/lang-ca-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-cs@4.26.1':
|
||||
resolution: {integrity: sha512-CqI6VB8toaJ/MlP1D4K9BctA6GpZJhMKyEy+OX9xavDe4r4ao/SxlSaIYK3izK0k+J38lJWC5lXYGazfCdTGjA==, tarball: https://registry.npmjs.org/@nlpjs/lang-cs/-/lang-cs-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-da@4.26.1':
|
||||
resolution: {integrity: sha512-krI/ojeDSi329ENM/hLIsbUh1x4XRTKAbtPcbFxAY6XVhcSVoWPO7L77jFTL1NQeE1oGRFzGHaeC9hZJ8phVbA==, tarball: https://registry.npmjs.org/@nlpjs/lang-da/-/lang-da-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-de@4.26.1':
|
||||
resolution: {integrity: sha512-HfZQwsE5FICq9taVZDiyktmdAePVF5948NM80et0d9mx43RWDFhHKQYgtJPwfQXtdCoQtOM5TOJ2FanGwzPeaA==, tarball: https://registry.npmjs.org/@nlpjs/lang-de/-/lang-de-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-el@4.26.1':
|
||||
resolution: {integrity: sha512-pcOvuSwPCXxI+2xNZZzM4V5pTRDntYoJi0SP/ic2nV4IPQ0nU2j16dYfg1HlvET/E6iN1VTqghrCaf10SMkDGA==, tarball: https://registry.npmjs.org/@nlpjs/lang-el/-/lang-el-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-en-min@4.26.1':
|
||||
resolution: {integrity: sha512-1sJZ7dy7ysqzbsB8IklguvB88J8EPIv4XGVkZCcwecKtOw+fp5LAsZ3TJVmEf18iK1gD4cEGr7qZg5fpPxTpWQ==, tarball: https://registry.npmjs.org/@nlpjs/lang-en-min/-/lang-en-min-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-en@4.26.1':
|
||||
resolution: {integrity: sha512-GVoJpOjyk5TtBAqo/fxsiuuH7jXycyakGT0gw5f01u9lOmUnpJegvXyGff/Nb0j14pXcGHXOhmpWrcTrG2B0LQ==, tarball: https://registry.npmjs.org/@nlpjs/lang-en/-/lang-en-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-es@4.26.1':
|
||||
resolution: {integrity: sha512-fIPQt+WPcNdyxZOCMkOPlMb4Y1iE585QxjB9IAdFz8ZtVg7mc4dlv5f46ud7ppdMh84iLOuOdo6pzu2Cqm14lw==, tarball: https://registry.npmjs.org/@nlpjs/lang-es/-/lang-es-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-eu@4.26.1':
|
||||
resolution: {integrity: sha512-Ha8GHTbgQYd7dwHM8aWHDyxmbUNUcyu/5xlBKqqBOPxysDyZ6Ad0tvj0FmJBy6mYhqmFTPBnEAo69cfuFSqWIQ==, tarball: https://registry.npmjs.org/@nlpjs/lang-eu/-/lang-eu-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-fa@4.26.1':
|
||||
resolution: {integrity: sha512-qJCmNXgJZnfNXUnKnxvEGEzSFBdQT4XU7/rMxuFmSJqmQY7fH/Vsmi5CKF94VRBPOIV4ULlEJuLpUWHXRmOnVQ==, tarball: https://registry.npmjs.org/@nlpjs/lang-fa/-/lang-fa-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-fi@4.26.1':
|
||||
resolution: {integrity: sha512-W/rUcrzSh3KE07q2vOsssTpU1sbX32gbBzKPZfRJ2ZUF4afO+eHxmAywikXubP4kiU3JxVNLvXXEjuGD3SBUbA==, tarball: https://registry.npmjs.org/@nlpjs/lang-fi/-/lang-fi-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-fr@4.26.1':
|
||||
resolution: {integrity: sha512-LTA852atCJnHtKDmtjx/ui5AnvEIkrPx+MJQ2mB3gn8ko6i2UITnJgPmJE9Kej5bLasVZOAJvU/SrfXEmnPGOw==, tarball: https://registry.npmjs.org/@nlpjs/lang-fr/-/lang-fr-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-ga@4.26.1':
|
||||
resolution: {integrity: sha512-JsP1CZ8r3Jd6o/Az7cN3exz0HDP3FNYLzh4Vi6ksEkdKF0yCjJ9G5dXZYqS9qFIN5ffemWn29G4WRELY6QH/cQ==, tarball: https://registry.npmjs.org/@nlpjs/lang-ga/-/lang-ga-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-gl@4.26.1':
|
||||
resolution: {integrity: sha512-y1NNu6NVy/6o5UNfihgg0WkSlVr4IvKA5W193CpRLZWS4FccQDmnFFhyYWRkshyDbgEsfsZ0Rs3BoE82+T2Ubg==, tarball: https://registry.npmjs.org/@nlpjs/lang-gl/-/lang-gl-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-hi@4.26.1':
|
||||
resolution: {integrity: sha512-Fw9rXqF5l8q9etJG5uOlEFpnMVjQEWMaCIgQfEcA1yTvieSV8mpoSvQkEZl+DFhww+azareoJ7ZCkx0gJ9UDuQ==, tarball: https://registry.npmjs.org/@nlpjs/lang-hi/-/lang-hi-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-hu@4.26.1':
|
||||
resolution: {integrity: sha512-7dPUn5/ZpLZmsdRwO+dtORuMIiIpnsWbgSLIKdOLh8irhgUR+M2bYTfkdnKcrEcHzHPP8Svn7pU0xk7OKSUA1w==, tarball: https://registry.npmjs.org/@nlpjs/lang-hu/-/lang-hu-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-hy@4.26.1':
|
||||
resolution: {integrity: sha512-T2brpLGDJryAwWmjtnmY8Ot6ZUkCz+/nRR9/QM1PybvZIqOVLjJqA49bqjJfT5DMN89HbwC7I/15NTT0y09i1Q==, tarball: https://registry.npmjs.org/@nlpjs/lang-hy/-/lang-hy-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-id@4.26.1':
|
||||
resolution: {integrity: sha512-rVuIkYFKdltFhMT/a2ZxD9ovoZSVZF7OPuqYjTXW9xKd3Ff32yUrzcf/pHXlqmZOSltqOH3E5jZRRDkHvgUOjQ==, tarball: https://registry.npmjs.org/@nlpjs/lang-id/-/lang-id-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-it@4.26.1':
|
||||
resolution: {integrity: sha512-BZA3QnfQGW91gYaybRmHnCAPBvQggtmHZJrAmuBZUKUS12HoQm8uybjw2fZO+vahEeUQceKNDISRcT1eLLijog==, tarball: https://registry.npmjs.org/@nlpjs/lang-it/-/lang-it-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-ja@4.26.1':
|
||||
resolution: {integrity: sha512-QgkuJOkHguRFyfnckH2It5/Kg8zecnOMJsHxYeuDC4tBF7jL/5xqWis+679lYLsXtAkrG8+fjVcBbjyopP0KHg==, tarball: https://registry.npmjs.org/@nlpjs/lang-ja/-/lang-ja-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-ko@4.26.1':
|
||||
resolution: {integrity: sha512-Q0N8bLJJ829ILWCKH1UQWPSNyuLaEURAXCawkDju4pt33DBLcpqz9IzO9dnqiFc+fjSgVzZ7WMaLT18hXZQ9vg==, tarball: https://registry.npmjs.org/@nlpjs/lang-ko/-/lang-ko-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-lt@4.26.1':
|
||||
resolution: {integrity: sha512-SeYZxRhdCy+ClQNnF/u0MAtcDui/ocdk4NtgNOCuwNTNuzhN3t3rfGeArfBGmZeg1SIeBLUDE9dsTxYCv5AOEg==, tarball: https://registry.npmjs.org/@nlpjs/lang-lt/-/lang-lt-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-ms@4.26.1':
|
||||
resolution: {integrity: sha512-KxWBS+tFY2U8z9UrjQIqMM40npGDOskP5DcWhaEE3zuhzf3RTDYjy8sdz34jVd0fBdbPihX133h3bFibg2Cm7w==, tarball: https://registry.npmjs.org/@nlpjs/lang-ms/-/lang-ms-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-ne@4.26.1':
|
||||
resolution: {integrity: sha512-K3E2l+0LTESv+dO+ZTIdvNa+zwMJvvnMiFYYkKvJst6lhc8JgvGOsPxGsjJn6PDhI3wyfQu+dg3b+bnVPu4FDA==, tarball: https://registry.npmjs.org/@nlpjs/lang-ne/-/lang-ne-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-nl@4.26.1':
|
||||
resolution: {integrity: sha512-I/mP1RRbUN4BQ+8NXAl2FKaLHbb7f6S8JVjxHQ0sKHT4BgQ3+r0yO+DVcEsHg+vWRiY1Fyzh0gq0PhLVnF6HnA==, tarball: https://registry.npmjs.org/@nlpjs/lang-nl/-/lang-nl-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-no@4.26.1':
|
||||
resolution: {integrity: sha512-a0CLL2c/OCzbg7J7ugyrsAksI96XhkQ3IeBbbx60o5o/9wsFNik6cPWrkpoE5xNtw7gLlAJWabwDiZXkl8Zrcw==, tarball: https://registry.npmjs.org/@nlpjs/lang-no/-/lang-no-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-pl@4.26.1':
|
||||
resolution: {integrity: sha512-nrDXlq+TzQLE5IpXPIlFMzd8OpquvApWsouh6fmLsD9HZLZI4O3w1M4sXXLzE+9Ggu9Cy1m1QJ0/i7XCcv115g==, tarball: https://registry.npmjs.org/@nlpjs/lang-pl/-/lang-pl-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-pt@4.26.1':
|
||||
resolution: {integrity: sha512-p6yZHaJ0e+n0avMHpdDw5PMk4HkKXjPbOMbrlg0dF+VRqChjxfH478Q423rDyzu/4MzDsIYB+p6KzL9AARKXpg==, tarball: https://registry.npmjs.org/@nlpjs/lang-pt/-/lang-pt-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-ro@4.26.1':
|
||||
resolution: {integrity: sha512-baUdTA0DWpDR0Tn6fxo+RDN/6gbuINLCARtHwap2UR/HKQWP2XoH/DIvcjZpwUTalr5MQjso31epcdeRRapczA==, tarball: https://registry.npmjs.org/@nlpjs/lang-ro/-/lang-ro-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-ru@4.26.1':
|
||||
resolution: {integrity: sha512-NaZ2DAOGxWG2Us9IyIDs3m6vhGpUaUJRVgzzHHyX3LO3xEYjZmtnA0jEpBaTOe2PuNHThv0WCZUNn9BSurV3PA==, tarball: https://registry.npmjs.org/@nlpjs/lang-ru/-/lang-ru-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-sl@4.26.1':
|
||||
resolution: {integrity: sha512-QBJwcJt+oKUpAnHKNJkLkx9Xm1n4dUPC5GPYfAXTnJZf0hNWJSY21GicdWi7Vu/qFJ3ghIqtSP8D7KIPLnibNw==, tarball: https://registry.npmjs.org/@nlpjs/lang-sl/-/lang-sl-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-sr@4.26.1':
|
||||
resolution: {integrity: sha512-drH3+UqTW637uLWsnLrcp8jEKUGxV61ZgCBjNkVQNEv1/jbpSg6IqgynSY2JyhtnlV0f870KS0HvSbyo5AD4Ng==, tarball: https://registry.npmjs.org/@nlpjs/lang-sr/-/lang-sr-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-sv@4.26.1':
|
||||
resolution: {integrity: sha512-2axkrYFC02tAlxCWeiEKISbe4dSteciP1CIggO/dZglnnLWgdF+g7kOeYMn7abCfFVSnh5vLqfDkrwnyIqt7Ag==, tarball: https://registry.npmjs.org/@nlpjs/lang-sv/-/lang-sv-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-ta@4.26.1':
|
||||
resolution: {integrity: sha512-keeh+croa1TAirV9Fd3OQMo5IkAlTGNWTNweHbi/htYMX0MKOPYxyqg+VH2bml+57VY2aUj/WYgV/p3ATx9EfQ==, tarball: https://registry.npmjs.org/@nlpjs/lang-ta/-/lang-ta-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-th@4.26.1':
|
||||
resolution: {integrity: sha512-2SWZhrln3rMw8/DsRc9yS5bi3qEdGfw2pq9Uejx/UYED5zvvL6kh9AiCJZT4k0wMBGEwWUV6HxJ0Pq/jOTHogg==, tarball: https://registry.npmjs.org/@nlpjs/lang-th/-/lang-th-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-tl@4.26.1':
|
||||
resolution: {integrity: sha512-AzmLtg28tm0VXCm0Q0EY3OtA3m4oYxaqh4VX6uhB4J+PoEsIkm0py12SJxMNIsh/r98pobCumH8KH9bvHQoCAg==, tarball: https://registry.npmjs.org/@nlpjs/lang-tl/-/lang-tl-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-tr@4.26.1':
|
||||
resolution: {integrity: sha512-p30uuXvE9pZeU/5XkrQfvxRgiAOBmP3EyBFGV/+P05PEogaqbsmmtVCgCnR63yeRvVnGbToPBPjRK3OO1y4AEQ==, tarball: https://registry.npmjs.org/@nlpjs/lang-tr/-/lang-tr-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-uk@4.26.1':
|
||||
resolution: {integrity: sha512-PVEvmlhvl6BL3e/Q4qjMPsnwON3cWEYvDh9dg+Si+sjD2Edu9tajolJKcQ6ZA4I8dXrld5xuXx+DEBH/uB4uWQ==, tarball: https://registry.npmjs.org/@nlpjs/lang-uk/-/lang-uk-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/lang-zh@4.26.1':
|
||||
resolution: {integrity: sha512-kwqeqeEgMAMvucVX9HNE1p6s/2APP23ZsS8Um/lNvtswb4gL5jjYF9kyCvRfqlPBQSWWdRv7wwcnNXOvXYkxcQ==, tarball: https://registry.npmjs.org/@nlpjs/lang-zh/-/lang-zh-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/language-min@4.25.0':
|
||||
resolution: {integrity: sha512-g8jtbDbqtRm+dlD/1Vnb4VWfKbKteApEGVTqIMxYkk6N/HMhvLZ5J2svrxzrB98a/HZ0fb//YBfFgymnz9Oukg==, tarball: https://registry.npmjs.org/@nlpjs/language-min/-/language-min-4.25.0.tgz}
|
||||
|
||||
'@nlpjs/language@4.25.0':
|
||||
resolution: {integrity: sha512-tUF6QENoUQ/E26RYc32IgsttStSF9cNO4ySN+BQECn8VpjukWdwbMw073MlOLXzjfeobxa+3hCVrmPPcW+V3UA==, tarball: https://registry.npmjs.org/@nlpjs/language/-/language-4.25.0.tgz}
|
||||
|
||||
'@nlpjs/ner@4.27.0':
|
||||
resolution: {integrity: sha512-ptwkxriJdmgHSH9TfP10JQ1jviaSl2SupSFGUvTuWkuJhobQd3hbnlSq40V6XYvJNmqh9M9zEab/AKeghxYOTA==, tarball: https://registry.npmjs.org/@nlpjs/ner/-/ner-4.27.0.tgz}
|
||||
|
||||
'@nlpjs/neural@4.25.0':
|
||||
resolution: {integrity: sha512-Oz20denGiBe0DlQsS7lN4TNrATN1nXlHKc/HB6jJPegjVmgJVCugDaHwIGoV7qOWyA6F2fRRwOgD+quNT2gVpg==, tarball: https://registry.npmjs.org/@nlpjs/neural/-/neural-4.25.0.tgz}
|
||||
|
||||
'@nlpjs/nlg@4.26.1':
|
||||
resolution: {integrity: sha512-PCJWiZ7464ChXXUGvjBZIFtoqkC24Oy6X63HgQrSv+63svz22Y5Cmu1MYLk77Nb+4keWv+hKhFJKDkvJoOpBVg==, tarball: https://registry.npmjs.org/@nlpjs/nlg/-/nlg-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/nlp@4.27.0':
|
||||
resolution: {integrity: sha512-q6X7sY6TYVnQRZJKF/6mfLFlNA5oRYLhgQ5k3i1IBqH9lbWTAZJr31w/dCf97HXaYaj+vJp3h0ucfNumme9EIw==, tarball: https://registry.npmjs.org/@nlpjs/nlp/-/nlp-4.27.0.tgz}
|
||||
|
||||
'@nlpjs/nlu@4.27.0':
|
||||
resolution: {integrity: sha512-j4DUdoXS/y/Xag6ysYXx7Ve8NBmUVViUSCJhj3r49+zGyYtyVAHuVcqSej5q0tJjn0JSMT+6+ip8klON1q8ixw==, tarball: https://registry.npmjs.org/@nlpjs/nlu/-/nlu-4.27.0.tgz}
|
||||
|
||||
'@nlpjs/request@4.25.0':
|
||||
resolution: {integrity: sha512-MPVYWfFZY03WyFL7GWkUkv8tw968OXsdxFSJEvjXHzhiCe/vAlPCWbvoR+VnoQTgzLHxs/KIF6sIF2s9AzsLmQ==, tarball: https://registry.npmjs.org/@nlpjs/request/-/request-4.25.0.tgz}
|
||||
|
||||
'@nlpjs/sentiment@4.26.1':
|
||||
resolution: {integrity: sha512-U2WmcW3w6yDDO45+Y7v5e6DPQj8e0x+RUUePPyRu2uIZmUtIKG+qCPMWnNLMmYQZoSQEFxmMMlLcGDC7tN7o3w==, tarball: https://registry.npmjs.org/@nlpjs/sentiment/-/sentiment-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/similarity@4.26.1':
|
||||
resolution: {integrity: sha512-QutSBFGo/huNuz60PgqCjub0oBd9S8MLrjme33U5GzxuSvToQzXtn9/ynIia8qDm009D09VXV+LPeNE4h7yuSg==, tarball: https://registry.npmjs.org/@nlpjs/similarity/-/similarity-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/slot@4.26.1':
|
||||
resolution: {integrity: sha512-mK8EEy5O+mRGne822PIKMxHSFh8j+iC7hGJ6T31XdFsNhFEYXLI/0dmeBstZgTSKBTe27HNFgCCwuGb77u0o9w==, tarball: https://registry.npmjs.org/@nlpjs/slot/-/slot-4.26.1.tgz}
|
||||
|
||||
'@nlpjs/xtables@4.25.0':
|
||||
resolution: {integrity: sha512-+baCtMZIp+aDqODLQs8Wyyke5qUqQkL8AGWsZzwYuJV8S7xdW2+XklRnHnkFc3p3foC248TkzG5L8j9r6INOtg==, tarball: https://registry.npmjs.org/@nlpjs/xtables/-/xtables-4.25.0.tgz}
|
||||
|
||||
'@noble/hashes@1.8.0':
|
||||
resolution: {integrity: sha512-jCs9ldd7NwzpgXDIf6P3+NrHh9/sD6CQdxHyjQI+h/6rDNo88ypBxxz45UDuZHz9r3tNz7N/VInSVoVdtXEI4A==, tarball: https://registry.npmjs.org/@noble/hashes/-/hashes-1.8.0.tgz}
|
||||
engines: {node: ^14.21.3 || >=16}
|
||||
|
|
@ -1333,10 +1118,6 @@ packages:
|
|||
resolution: {integrity: sha512-ID7fosbc50TbT0MK0EG12O+gAP3W3Aa/Pz4DaTtQtEvlc9Odaqi0de+xuZ7Li2GtK4HzEX7IuRWS/JmZLksR3Q==, tarball: https://registry.npmjs.org/@teppeis/multimaps/-/multimaps-3.0.0.tgz}
|
||||
engines: {node: '>=14'}
|
||||
|
||||
'@tootallnate/once@2.0.1':
|
||||
resolution: {integrity: sha512-HqmEUIGRJ5fSXchkVgR5F7qn48bDBzv0kWj/Kfu5e6uci4UlEeng4331LnBkWffb++Ei3FOVLxo8JJWMFBDMeQ==, tarball: https://registry.npmjs.org/@tootallnate/once/-/once-2.0.1.tgz}
|
||||
engines: {node: '>= 10'}
|
||||
|
||||
'@types/babel__core@7.20.5':
|
||||
resolution: {integrity: sha512-qoQprZvz5wQFJwMDqeseRXWv3rqMvhgpbXFfVyWhbx9X47POIA6i/+dXefEmZKoAgOaTdaIgNSMqMIU61yRyzA==, tarball: https://registry.npmjs.org/@types/babel__core/-/babel__core-7.20.5.tgz}
|
||||
|
||||
|
|
@ -1506,10 +1287,6 @@ packages:
|
|||
engines: {node: '>=0.4.0'}
|
||||
hasBin: true
|
||||
|
||||
adler-32@1.3.1:
|
||||
resolution: {integrity: sha512-ynZ4w/nUUv5rrsR8UUGoe1VC9hZj6V5hU9Qw1HlMDJGEJw5S7TfTErWTjMys6M7vr0YWcPqs3qAr4ss0nDfP+A==, tarball: https://registry.npmjs.org/adler-32/-/adler-32-1.3.1.tgz}
|
||||
engines: {node: '>=0.8'}
|
||||
|
||||
agent-base@6.0.2:
|
||||
resolution: {integrity: sha512-RZNwNclF7+MS/8bDg70amg32dyeZGZxiDuQmZxKLAlQjr3jGyLx+4Kkk58UO7D2QdgFIQCovuSuZESne6RG6XQ==, tarball: https://registry.npmjs.org/agent-base/-/agent-base-6.0.2.tgz}
|
||||
engines: {node: '>= 6.0.0'}
|
||||
|
|
@ -1613,9 +1390,6 @@ packages:
|
|||
resolution: {integrity: sha512-Z7tMw1ytTXt5jqMcOP+OQteU1VuNK9Y02uuJtKQ1Sv69jXQKKg5cibLwGJow8yzZP+eAc18EmLGPal0bp36rvQ==, tarball: https://registry.npmjs.org/astral-regex/-/astral-regex-2.0.0.tgz}
|
||||
engines: {node: '>=8'}
|
||||
|
||||
async@2.6.4:
|
||||
resolution: {integrity: sha512-mzo5dfJYwAn29PeiJ0zvwTo04zj8HDJj0Mn8TD7sno7q12prdbnasKJHhkm2c1LgrhlJ0teaea8860oxi51mGA==, tarball: https://registry.npmjs.org/async/-/async-2.6.4.tgz}
|
||||
|
||||
async@3.2.6:
|
||||
resolution: {integrity: sha512-htCUDlxyyCLMgaM3xXg0C0LW2xqfuQ6p05pCEIsXuyQ+a1koYKTuBMzRNwmybfLgvJDMd0r1LTn4+E0Ti6C2AA==, tarball: https://registry.npmjs.org/async/-/async-3.2.6.tgz}
|
||||
|
||||
|
|
@ -1657,9 +1431,6 @@ packages:
|
|||
bcrypt-pbkdf@1.0.2:
|
||||
resolution: {integrity: sha512-qeFIXtP4MSoi6NLqO12WfqARWWuCKi2Rn/9hJLEmtB5yTNr9DqFWkJRCf2qShWzPeAMRnOgCrq0sg/KLv5ES9w==, tarball: https://registry.npmjs.org/bcrypt-pbkdf/-/bcrypt-pbkdf-1.0.2.tgz}
|
||||
|
||||
bignumber.js@7.2.1:
|
||||
resolution: {integrity: sha512-S4XzBk5sMB+Rcb/LNcpzXr57VRTxgAvaAEDAl1AwRx27j00hT84O6OkteE7u8UB3NuaaygCRrEpqox4uDOrbdQ==, tarball: https://registry.npmjs.org/bignumber.js/-/bignumber.js-7.2.1.tgz}
|
||||
|
||||
binary-extensions@2.3.0:
|
||||
resolution: {integrity: sha512-Ceh+7ox5qe7LJuLHoY0feh3pHuUDHAcRUeyL2VYghZwfpkNIy/+8Ocg0a3UuSoYzavmylwuLWQOf3hl0jjMMIw==, tarball: https://registry.npmjs.org/binary-extensions/-/binary-extensions-2.3.0.tgz}
|
||||
engines: {node: '>=8'}
|
||||
|
|
@ -1748,10 +1519,6 @@ packages:
|
|||
caseless@0.12.0:
|
||||
resolution: {integrity: sha512-4tYFyifaFfGacoiObjJegolkwSU4xQNGbVgUiNYVUxbQ2x2lUsFvY4hVgVzGiIe6WLOPqycWXA40l+PWsxthUw==, tarball: https://registry.npmjs.org/caseless/-/caseless-0.12.0.tgz}
|
||||
|
||||
cfb@1.2.2:
|
||||
resolution: {integrity: sha512-KfdUZsSOw19/ObEWasvBP/Ac4reZvAGauZhs6S/gqNhXhI7cKwvlH7ulj+dOEYnca4bm4SGo8C1bTAQvnTjgQA==, tarball: https://registry.npmjs.org/cfb/-/cfb-1.2.2.tgz}
|
||||
engines: {node: '>=0.8'}
|
||||
|
||||
chai@5.3.3:
|
||||
resolution: {integrity: sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw==, tarball: https://registry.npmjs.org/chai/-/chai-5.3.3.tgz}
|
||||
engines: {node: '>=18'}
|
||||
|
|
@ -1822,10 +1589,6 @@ packages:
|
|||
resolution: {integrity: sha512-JQHZ2QMW6l3aH/j6xCqQThY/9OH4D/9ls34cgkUBiEeocRTU04tHfKPBsUK1PqZCUQM7GiA0IIXJSuXHI64Kbg==, tarball: https://registry.npmjs.org/clone/-/clone-1.0.4.tgz}
|
||||
engines: {node: '>=0.8'}
|
||||
|
||||
codepage@1.15.0:
|
||||
resolution: {integrity: sha512-3g6NUTPd/YtuuGrhMnOMRjFc+LJw/bnMp3+0r/Wcz3IXUuCosKRJvMphm5+Q+bvTVGcJJuRvVLuYba+WojaFaA==, tarball: https://registry.npmjs.org/codepage/-/codepage-1.15.0.tgz}
|
||||
engines: {node: '>=0.8'}
|
||||
|
||||
color-convert@2.0.1:
|
||||
resolution: {integrity: sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==, tarball: https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz}
|
||||
engines: {node: '>=7.0.0'}
|
||||
|
|
@ -1940,11 +1703,6 @@ packages:
|
|||
typescript:
|
||||
optional: true
|
||||
|
||||
crc-32@1.2.2:
|
||||
resolution: {integrity: sha512-ROmzCKrTnOwybPcJApAA6WBWij23HVfGVNKqqrZpuyZOHqK2CwHSvpGuyt/UNNvaIjEd8X5IFGp4Mh+Ie1IHJQ==, tarball: https://registry.npmjs.org/crc-32/-/crc-32-1.2.2.tgz}
|
||||
engines: {node: '>=0.8'}
|
||||
hasBin: true
|
||||
|
||||
cross-env@10.1.0:
|
||||
resolution: {integrity: sha512-GsYosgnACZTADcmEyJctkJIoqAhHjttw7RsFrVoJNXbsWWqaq6Ym+7kZjq6mS45O0jij6vtiReppKQEtqWy6Dw==, tarball: https://registry.npmjs.org/cross-env/-/cross-env-10.1.0.tgz}
|
||||
engines: {node: '>=20'}
|
||||
|
|
@ -2132,9 +1890,6 @@ packages:
|
|||
resolution: {integrity: sha512-uBq4egWHTcTt33a72vpSG0z3HnPuIl6NqYcTrKEg2azoEyl2hpW0zqlxysq2pK9HlDIHyHyakeYaYnSAwd8bow==, tarball: https://registry.npmjs.org/dotenv/-/dotenv-16.6.1.tgz}
|
||||
engines: {node: '>=12'}
|
||||
|
||||
doublearray@0.0.2:
|
||||
resolution: {integrity: sha512-aw55FtZzT6AmiamEj2kvmR6BuFqvYgKZUkfQ7teqVRNqD5UE0rw8IeW/3gieHNKQ5sPuDKlljWEn4bzv5+1bHw==, tarball: https://registry.npmjs.org/doublearray/-/doublearray-0.0.2.tgz}
|
||||
|
||||
dunder-proto@1.0.1:
|
||||
resolution: {integrity: sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==, tarball: https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz}
|
||||
engines: {node: '>= 0.4'}
|
||||
|
|
@ -2401,10 +2156,6 @@ packages:
|
|||
resolution: {integrity: sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow==, tarball: https://registry.npmjs.org/forwarded/-/forwarded-0.2.0.tgz}
|
||||
engines: {node: '>= 0.6'}
|
||||
|
||||
frac@1.1.2:
|
||||
resolution: {integrity: sha512-w/XBfkibaTl3YDqASwfDUqkna4Z2p9cFSr1aHDt0WoMTECnRfBOv2WArlZILlqgWlmdIlALXGpM2AOhEk5W3IA==, tarball: https://registry.npmjs.org/frac/-/frac-1.1.2.tgz}
|
||||
engines: {node: '>=0.8'}
|
||||
|
||||
fresh@0.5.2:
|
||||
resolution: {integrity: sha512-zJ2mQYM18rEFOudeV4GShTGIQ7RbzA7ozbU9I/XBpm7kqgMywgmylMwXHxZJmkVoYkna9d2pVXVXPdYTP9ej8Q==, tarball: https://registry.npmjs.org/fresh/-/fresh-0.5.2.tgz}
|
||||
engines: {node: '>= 0.6'}
|
||||
|
|
@ -2510,9 +2261,6 @@ packages:
|
|||
graceful-fs@4.2.11:
|
||||
resolution: {integrity: sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ==, tarball: https://registry.npmjs.org/graceful-fs/-/graceful-fs-4.2.11.tgz}
|
||||
|
||||
grapheme-splitter@1.0.4:
|
||||
resolution: {integrity: sha512-bzh50DW9kTPM00T8y4o8vQg89Di9oLJVLW/KaOGIXJWP/iqCN6WKYkbNOF04vFLJhwcpYUh9ydh/+5vpOqV4YQ==, tarball: https://registry.npmjs.org/grapheme-splitter/-/grapheme-splitter-1.0.4.tgz}
|
||||
|
||||
has-ansi@4.0.1:
|
||||
resolution: {integrity: sha512-Qr4RtTm30xvEdqUXbSBVWDu+PrTokJOwe/FU+VdfJPk+MXAPoeOzKpRyrDTnZIJwAkQ4oBLTU53nu0HrkF/Z2A==, tarball: https://registry.npmjs.org/has-ansi/-/has-ansi-4.0.1.tgz}
|
||||
engines: {node: '>=8'}
|
||||
|
|
@ -2558,10 +2306,6 @@ packages:
|
|||
resolution: {integrity: sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ==, tarball: https://registry.npmjs.org/http-errors/-/http-errors-2.0.1.tgz}
|
||||
engines: {node: '>= 0.8'}
|
||||
|
||||
http-proxy-agent@5.0.0:
|
||||
resolution: {integrity: sha512-n2hY8YdoRE1i7r6M0w9DIw5GgZN0G25P8zLCRQ8rjXtTU3vsNFBI/vWK/UIeE6g5MUUz6avwAPXmL6Fy9D/90w==, tarball: https://registry.npmjs.org/http-proxy-agent/-/http-proxy-agent-5.0.0.tgz}
|
||||
engines: {node: '>= 6'}
|
||||
|
||||
http-signature@1.4.0:
|
||||
resolution: {integrity: sha512-G5akfn7eKbpDN+8nPS/cb57YeA1jLTVxjpCj7tmm3QKPdyDy7T+qSC40e9ptydSWvkwjSXw1VbkpyEm39ukeAg==, tarball: https://registry.npmjs.org/http-signature/-/http-signature-1.4.0.tgz}
|
||||
engines: {node: '>=0.10'}
|
||||
|
|
@ -2814,9 +2558,6 @@ packages:
|
|||
knuth-shuffle-seeded@1.0.6:
|
||||
resolution: {integrity: sha512-9pFH0SplrfyKyojCLxZfMcvkhf5hH0d+UwR9nTVJ/DDQJGuzcXjTwB7TP7sDfehSudlGGaOLblmEWqv04ERVWg==, tarball: https://registry.npmjs.org/knuth-shuffle-seeded/-/knuth-shuffle-seeded-1.0.6.tgz}
|
||||
|
||||
kuromoji@0.1.2:
|
||||
resolution: {integrity: sha512-V0dUf+C2LpcPEXhoHLMAop/bOht16Dyr+mDiIE39yX3vqau7p80De/koFqpiTcL1zzdZlc3xuHZ8u5gjYRfFaQ==, tarball: https://registry.npmjs.org/kuromoji/-/kuromoji-0.1.2.tgz}
|
||||
|
||||
lazy-ass@1.6.0:
|
||||
resolution: {integrity: sha512-cc8oEVoctTvsFZ/Oje/kGnHbpWHYBe8IAJe4C0QNc3t8uM/0Y8+erSz/7Y1ALuXTEZTMvxXwO6YbX1ey3ujiZw==, tarball: https://registry.npmjs.org/lazy-ass/-/lazy-ass-1.6.0.tgz}
|
||||
engines: {node: '> 0.8'}
|
||||
|
|
@ -3081,9 +2822,6 @@ packages:
|
|||
node-html-parser@5.3.3:
|
||||
resolution: {integrity: sha512-ncg1033CaX9UexbyA7e1N0aAoAYRDiV8jkTvzEnfd1GDvzFdrsXLzR4p4ik8mwLgnaKP/jyUFWDy9q3jvRT2Jw==, tarball: https://registry.npmjs.org/node-html-parser/-/node-html-parser-5.3.3.tgz}
|
||||
|
||||
node-nlp@4.27.0:
|
||||
resolution: {integrity: sha512-LnkhOUPXX0CMFbSzJ1gHI+7Yb3ULLip5gRsqedXb6pryjcRCbNzPgHXcH/6G9B1vSbDfO+y3X2B4QZpfP12OyQ==, tarball: https://registry.npmjs.org/node-nlp/-/node-nlp-4.27.0.tgz}
|
||||
|
||||
node-releases@2.0.53:
|
||||
resolution: {integrity: sha512-D9UOmYG3UH1V+ENW56t5QXBwJw1YEY18ruVeus89Rw+SyIgjPkCO84bRzO3uNIYosJbNwiabWVn48o3uJLjxFQ==, tarball: https://registry.npmjs.org/node-releases/-/node-releases-2.0.53.tgz}
|
||||
engines: {node: '>=18'}
|
||||
|
|
@ -3633,10 +3371,6 @@ packages:
|
|||
split@1.0.1:
|
||||
resolution: {integrity: sha512-mTyOoPbrivtXnwnIxZRFYRrPNtEFKlpB2fvjSnCQUiAA6qAZzqwna5envK4uk6OIeP17CsdF3rSBGYVBsU0Tkg==, tarball: https://registry.npmjs.org/split/-/split-1.0.1.tgz}
|
||||
|
||||
ssf@0.11.2:
|
||||
resolution: {integrity: sha512-+idbmIXoYET47hH+d7dfm2epdOMUDjqcB4648sTZ+t2JwoyBFL/insLfB/racrDmsKB3diwsDA696pZMieAC5g==, tarball: https://registry.npmjs.org/ssf/-/ssf-0.11.2.tgz}
|
||||
engines: {node: '>=0.8'}
|
||||
|
||||
sshpk@1.18.0:
|
||||
resolution: {integrity: sha512-2p2KJZTSqQ/I3+HX42EpYOa2l3f8Erv8MWKsy2I9uf4wA7yFIkXRffYdsx86y6z4vHtV8u7g+pPlr8/4ouAxsQ==, tarball: https://registry.npmjs.org/sshpk/-/sshpk-1.18.0.tgz}
|
||||
engines: {node: '>=0.10.0'}
|
||||
|
|
@ -3977,14 +3711,6 @@ packages:
|
|||
wide-align@1.1.5:
|
||||
resolution: {integrity: sha512-eDMORYaPNZ4sQIuuYPDHdQvf4gyCF9rEEV/yPxGfwPkRodwEgiMUUXTx/dex+Me0wxx53S+NgUHaP7y3MGlDmg==, tarball: https://registry.npmjs.org/wide-align/-/wide-align-1.1.5.tgz}
|
||||
|
||||
wmf@1.0.2:
|
||||
resolution: {integrity: sha512-/p9K7bEh0Dj6WbXg4JG0xvLQmIadrner1bi45VMJTfnbVHsc7yIajZyoSoK60/dtVBs12Fm6WkUI5/3WAVsNMw==, tarball: https://registry.npmjs.org/wmf/-/wmf-1.0.2.tgz}
|
||||
engines: {node: '>=0.8'}
|
||||
|
||||
word@0.3.0:
|
||||
resolution: {integrity: sha512-OELeY0Q61OXpdUfTp+oweA/vtLVg5VDOXh+3he3PNzLGG/y0oylSOC1xRVj0+l4vQ3tj/bB1HVHv1ocXkQceFA==, tarball: https://registry.npmjs.org/word/-/word-0.3.0.tgz}
|
||||
engines: {node: '>=0.8'}
|
||||
|
||||
workerpool@6.5.1:
|
||||
resolution: {integrity: sha512-Fs4dNYcsdpYSAfVxhnl1L5zTksjvOJxtC5hzMNl+1t9B8hTJTdKDyZ5ju7ztgPy+ft9tBFXoOlDNiOT9WUXZlA==, tarball: https://registry.npmjs.org/workerpool/-/workerpool-6.5.1.tgz}
|
||||
|
||||
|
|
@ -4006,11 +3732,6 @@ packages:
|
|||
wrappy@1.0.2:
|
||||
resolution: {integrity: sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==, tarball: https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz}
|
||||
|
||||
xlsx@0.18.5:
|
||||
resolution: {integrity: sha512-dmg3LCjBPHZnQp5/F/+nnTa+miPJxUXB6vtk42YjBBKayDNagxGEeIdWApkYPOf3Z3pm3k62Knjzp7lMeTEtFQ==, tarball: https://registry.npmjs.org/xlsx/-/xlsx-0.18.5.tgz}
|
||||
engines: {node: '>=0.8'}
|
||||
hasBin: true
|
||||
|
||||
xmlbuilder@15.1.1:
|
||||
resolution: {integrity: sha512-yMqGBqtXyeN1e3TGYvgNgDVZ3j84W4cwkOXQswghol6APgZWaff9lnbvN7MHYJOiXsvGPXtjTYJEiC9J2wv9Eg==, tarball: https://registry.npmjs.org/xmlbuilder/-/xmlbuilder-15.1.1.tgz}
|
||||
engines: {node: '>=8.0'}
|
||||
|
|
@ -4064,9 +3785,6 @@ packages:
|
|||
yup@1.6.1:
|
||||
resolution: {integrity: sha512-JED8pB50qbA4FOkDol0bYF/p60qSEDQqBD0/qeIrUCG1KbPBIQ776fCUNb9ldbPcSTxA69g/47XTo4TqWiuXOA==, tarball: https://registry.npmjs.org/yup/-/yup-1.6.1.tgz}
|
||||
|
||||
zlibjs@0.3.1:
|
||||
resolution: {integrity: sha512-+J9RrgTKOmlxFSDHo0pI1xM6BLVUv+o0ZT9ANtCxGkjIVCCUdx9alUF8Gm+dGLKbkkkidWIHFDZHDMpfITt4+w==, tarball: https://registry.npmjs.org/zlibjs/-/zlibjs-0.3.1.tgz}
|
||||
|
||||
zod@3.25.76:
|
||||
resolution: {integrity: sha512-gzUt/qt81nXsFGKIFcC3YnfEAx5NkunCfnDlvuBSSFS02bcXu4Lmea0AFIUwbLWxWPx3d9p8S5QoaujKcNQxcQ==, tarball: https://registry.npmjs.org/zod/-/zod-3.25.76.tgz}
|
||||
|
||||
|
|
@ -4675,344 +4393,9 @@ snapshots:
|
|||
- encoding
|
||||
- supports-color
|
||||
|
||||
'@microsoft/recognizers-text-choice@1.3.1':
|
||||
dependencies:
|
||||
'@microsoft/recognizers-text': 1.3.1
|
||||
grapheme-splitter: 1.0.4
|
||||
|
||||
'@microsoft/recognizers-text-data-types-timex-expression@1.3.1': {}
|
||||
|
||||
'@microsoft/recognizers-text-date-time@1.3.2':
|
||||
dependencies:
|
||||
'@microsoft/recognizers-text': 1.3.1
|
||||
'@microsoft/recognizers-text-number': 1.3.1
|
||||
'@microsoft/recognizers-text-number-with-unit': 1.3.1
|
||||
lodash: 4.18.1
|
||||
|
||||
'@microsoft/recognizers-text-number-with-unit@1.3.1':
|
||||
dependencies:
|
||||
'@microsoft/recognizers-text': 1.3.1
|
||||
'@microsoft/recognizers-text-number': 1.3.1
|
||||
lodash: 4.18.1
|
||||
|
||||
'@microsoft/recognizers-text-number@1.3.1':
|
||||
dependencies:
|
||||
'@microsoft/recognizers-text': 1.3.1
|
||||
bignumber.js: 7.2.1
|
||||
lodash: 4.18.1
|
||||
|
||||
'@microsoft/recognizers-text-sequence@1.3.1':
|
||||
dependencies:
|
||||
'@microsoft/recognizers-text': 1.3.1
|
||||
grapheme-splitter: 1.0.4
|
||||
|
||||
'@microsoft/recognizers-text-suite@1.3.0':
|
||||
dependencies:
|
||||
'@microsoft/recognizers-text': 1.3.1
|
||||
'@microsoft/recognizers-text-choice': 1.3.1
|
||||
'@microsoft/recognizers-text-data-types-timex-expression': 1.3.1
|
||||
'@microsoft/recognizers-text-date-time': 1.3.2
|
||||
'@microsoft/recognizers-text-number': 1.3.1
|
||||
'@microsoft/recognizers-text-number-with-unit': 1.3.1
|
||||
'@microsoft/recognizers-text-sequence': 1.3.1
|
||||
|
||||
'@microsoft/recognizers-text@1.3.1': {}
|
||||
|
||||
'@napi-rs/lzma-linux-x64-gnu@1.5.1':
|
||||
optional: true
|
||||
|
||||
'@nlpjs/builtin-duckling@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/builtin-microsoft@4.26.1':
|
||||
dependencies:
|
||||
'@microsoft/recognizers-text-suite': 1.3.0
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/core-loader@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
'@nlpjs/request': 4.25.0
|
||||
transitivePeerDependencies:
|
||||
- supports-color
|
||||
|
||||
'@nlpjs/core@4.26.1': {}
|
||||
|
||||
'@nlpjs/emoji@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/evaluator@4.26.1':
|
||||
dependencies:
|
||||
escodegen: 2.1.0
|
||||
esprima: 4.0.1
|
||||
|
||||
'@nlpjs/lang-all@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
'@nlpjs/lang-ar': 4.26.1
|
||||
'@nlpjs/lang-bn': 4.26.1
|
||||
'@nlpjs/lang-ca': 4.26.1
|
||||
'@nlpjs/lang-cs': 4.26.1
|
||||
'@nlpjs/lang-da': 4.26.1
|
||||
'@nlpjs/lang-de': 4.26.1
|
||||
'@nlpjs/lang-el': 4.26.1
|
||||
'@nlpjs/lang-en': 4.26.1
|
||||
'@nlpjs/lang-es': 4.26.1
|
||||
'@nlpjs/lang-eu': 4.26.1
|
||||
'@nlpjs/lang-fa': 4.26.1
|
||||
'@nlpjs/lang-fi': 4.26.1
|
||||
'@nlpjs/lang-fr': 4.26.1
|
||||
'@nlpjs/lang-ga': 4.26.1
|
||||
'@nlpjs/lang-gl': 4.26.1
|
||||
'@nlpjs/lang-hi': 4.26.1
|
||||
'@nlpjs/lang-hu': 4.26.1
|
||||
'@nlpjs/lang-hy': 4.26.1
|
||||
'@nlpjs/lang-id': 4.26.1
|
||||
'@nlpjs/lang-it': 4.26.1
|
||||
'@nlpjs/lang-ja': 4.26.1
|
||||
'@nlpjs/lang-ko': 4.26.1
|
||||
'@nlpjs/lang-lt': 4.26.1
|
||||
'@nlpjs/lang-ms': 4.26.1
|
||||
'@nlpjs/lang-ne': 4.26.1
|
||||
'@nlpjs/lang-nl': 4.26.1
|
||||
'@nlpjs/lang-no': 4.26.1
|
||||
'@nlpjs/lang-pl': 4.26.1
|
||||
'@nlpjs/lang-pt': 4.26.1
|
||||
'@nlpjs/lang-ro': 4.26.1
|
||||
'@nlpjs/lang-ru': 4.26.1
|
||||
'@nlpjs/lang-sl': 4.26.1
|
||||
'@nlpjs/lang-sr': 4.26.1
|
||||
'@nlpjs/lang-sv': 4.26.1
|
||||
'@nlpjs/lang-ta': 4.26.1
|
||||
'@nlpjs/lang-th': 4.26.1
|
||||
'@nlpjs/lang-tl': 4.26.1
|
||||
'@nlpjs/lang-tr': 4.26.1
|
||||
'@nlpjs/lang-uk': 4.26.1
|
||||
'@nlpjs/lang-zh': 4.26.1
|
||||
'@nlpjs/language': 4.25.0
|
||||
|
||||
'@nlpjs/lang-ar@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-bn@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-ca@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-cs@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-da@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-de@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-el@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-en-min@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-en@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
'@nlpjs/lang-en-min': 4.26.1
|
||||
|
||||
'@nlpjs/lang-es@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-eu@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-fa@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-fi@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-fr@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-ga@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-gl@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-hi@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-hu@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-hy@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-id@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-it@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-ja@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
kuromoji: 0.1.2
|
||||
|
||||
'@nlpjs/lang-ko@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-lt@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-ms@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
'@nlpjs/lang-id': 4.26.1
|
||||
|
||||
'@nlpjs/lang-ne@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-nl@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-no@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-pl@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-pt@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-ro@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-ru@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-sl@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-sr@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-sv@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-ta@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-th@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-tl@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-tr@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-uk@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/lang-zh@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/language-min@4.25.0': {}
|
||||
|
||||
'@nlpjs/language@4.25.0': {}
|
||||
|
||||
'@nlpjs/ner@4.27.0':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
'@nlpjs/language-min': 4.25.0
|
||||
'@nlpjs/similarity': 4.26.1
|
||||
|
||||
'@nlpjs/neural@4.25.0': {}
|
||||
|
||||
'@nlpjs/nlg@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
|
||||
'@nlpjs/nlp@4.27.0':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
'@nlpjs/ner': 4.27.0
|
||||
'@nlpjs/nlg': 4.26.1
|
||||
'@nlpjs/nlu': 4.27.0
|
||||
'@nlpjs/sentiment': 4.26.1
|
||||
'@nlpjs/slot': 4.26.1
|
||||
|
||||
'@nlpjs/nlu@4.27.0':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
'@nlpjs/language-min': 4.25.0
|
||||
'@nlpjs/neural': 4.25.0
|
||||
'@nlpjs/similarity': 4.26.1
|
||||
|
||||
'@nlpjs/request@4.25.0':
|
||||
dependencies:
|
||||
http-proxy-agent: 5.0.0
|
||||
https-proxy-agent: 5.0.1
|
||||
transitivePeerDependencies:
|
||||
- supports-color
|
||||
|
||||
'@nlpjs/sentiment@4.26.1':
|
||||
dependencies:
|
||||
'@nlpjs/core': 4.26.1
|
||||
'@nlpjs/language-min': 4.25.0
|
||||
'@nlpjs/neural': 4.25.0
|
||||
|
||||
'@nlpjs/similarity@4.26.1': {}
|
||||
|
||||
'@nlpjs/slot@4.26.1': {}
|
||||
|
||||
'@nlpjs/xtables@4.25.0':
|
||||
dependencies:
|
||||
xlsx: 0.18.5
|
||||
|
||||
'@noble/hashes@1.8.0': {}
|
||||
|
||||
'@nodelib/fs.scandir@2.1.5':
|
||||
|
|
@ -5199,8 +4582,6 @@ snapshots:
|
|||
|
||||
'@teppeis/multimaps@3.0.0': {}
|
||||
|
||||
'@tootallnate/once@2.0.1': {}
|
||||
|
||||
'@types/babel__core@7.20.5':
|
||||
dependencies:
|
||||
'@babel/parser': 7.29.8
|
||||
|
|
@ -5423,8 +4804,6 @@ snapshots:
|
|||
|
||||
acorn@8.18.0: {}
|
||||
|
||||
adler-32@1.3.1: {}
|
||||
|
||||
agent-base@6.0.2:
|
||||
dependencies:
|
||||
debug: 4.4.3(supports-color@8.1.1)
|
||||
|
|
@ -5514,10 +4893,6 @@ snapshots:
|
|||
|
||||
astral-regex@2.0.0: {}
|
||||
|
||||
async@2.6.4:
|
||||
dependencies:
|
||||
lodash: 4.18.1
|
||||
|
||||
async@3.2.6: {}
|
||||
|
||||
asynckit@0.4.0: {}
|
||||
|
|
@ -5554,8 +4929,6 @@ snapshots:
|
|||
dependencies:
|
||||
tweetnacl: 0.14.5
|
||||
|
||||
bignumber.js@7.2.1: {}
|
||||
|
||||
binary-extensions@2.3.0: {}
|
||||
|
||||
blob-util@2.0.2: {}
|
||||
|
|
@ -5654,11 +5027,6 @@ snapshots:
|
|||
|
||||
caseless@0.12.0: {}
|
||||
|
||||
cfb@1.2.2:
|
||||
dependencies:
|
||||
adler-32: 1.3.1
|
||||
crc-32: 1.2.2
|
||||
|
||||
chai@5.3.3:
|
||||
dependencies:
|
||||
assertion-error: 2.0.1
|
||||
|
|
@ -5738,8 +5106,6 @@ snapshots:
|
|||
clone@1.0.4:
|
||||
optional: true
|
||||
|
||||
codepage@1.15.0: {}
|
||||
|
||||
color-convert@2.0.1:
|
||||
dependencies:
|
||||
color-name: 1.1.4
|
||||
|
|
@ -5821,8 +5187,6 @@ snapshots:
|
|||
optionalDependencies:
|
||||
typescript: 5.9.3
|
||||
|
||||
crc-32@1.2.2: {}
|
||||
|
||||
cross-env@10.1.0:
|
||||
dependencies:
|
||||
'@epic-web/invariant': 1.0.0
|
||||
|
|
@ -6067,8 +5431,6 @@ snapshots:
|
|||
|
||||
dotenv@16.6.1: {}
|
||||
|
||||
doublearray@0.0.2: {}
|
||||
|
||||
dunder-proto@1.0.1:
|
||||
dependencies:
|
||||
call-bind-apply-helpers: 1.0.2
|
||||
|
|
@ -6461,8 +5823,6 @@ snapshots:
|
|||
|
||||
forwarded@0.2.0: {}
|
||||
|
||||
frac@1.1.2: {}
|
||||
|
||||
fresh@0.5.2: {}
|
||||
|
||||
from@0.1.7: {}
|
||||
|
|
@ -6584,8 +5944,6 @@ snapshots:
|
|||
|
||||
graceful-fs@4.2.11: {}
|
||||
|
||||
grapheme-splitter@1.0.4: {}
|
||||
|
||||
has-ansi@4.0.1:
|
||||
dependencies:
|
||||
ansi-regex: 4.1.1
|
||||
|
|
@ -6626,14 +5984,6 @@ snapshots:
|
|||
statuses: 2.0.2
|
||||
toidentifier: 1.0.1
|
||||
|
||||
http-proxy-agent@5.0.0:
|
||||
dependencies:
|
||||
'@tootallnate/once': 2.0.1
|
||||
agent-base: 6.0.2
|
||||
debug: 4.4.3(supports-color@8.1.1)
|
||||
transitivePeerDependencies:
|
||||
- supports-color
|
||||
|
||||
http-signature@1.4.0:
|
||||
dependencies:
|
||||
assert-plus: 1.0.0
|
||||
|
|
@ -6876,12 +6226,6 @@ snapshots:
|
|||
dependencies:
|
||||
seed-random: 2.2.0
|
||||
|
||||
kuromoji@0.1.2:
|
||||
dependencies:
|
||||
async: 2.6.4
|
||||
doublearray: 0.0.2
|
||||
zlibjs: 0.3.1
|
||||
|
||||
lazy-ass@1.6.0: {}
|
||||
|
||||
lazy-ass@2.0.3: {}
|
||||
|
|
@ -7133,26 +6477,6 @@ snapshots:
|
|||
css-select: 4.3.0
|
||||
he: 1.2.0
|
||||
|
||||
node-nlp@4.27.0:
|
||||
dependencies:
|
||||
'@nlpjs/builtin-duckling': 4.26.1
|
||||
'@nlpjs/builtin-microsoft': 4.26.1
|
||||
'@nlpjs/core-loader': 4.26.1
|
||||
'@nlpjs/emoji': 4.26.1
|
||||
'@nlpjs/evaluator': 4.26.1
|
||||
'@nlpjs/lang-all': 4.26.1
|
||||
'@nlpjs/language': 4.25.0
|
||||
'@nlpjs/neural': 4.25.0
|
||||
'@nlpjs/nlg': 4.26.1
|
||||
'@nlpjs/nlp': 4.27.0
|
||||
'@nlpjs/nlu': 4.27.0
|
||||
'@nlpjs/request': 4.25.0
|
||||
'@nlpjs/sentiment': 4.26.1
|
||||
'@nlpjs/similarity': 4.26.1
|
||||
'@nlpjs/xtables': 4.25.0
|
||||
transitivePeerDependencies:
|
||||
- supports-color
|
||||
|
||||
node-releases@2.0.53: {}
|
||||
|
||||
node-source-walk@7.0.2:
|
||||
|
|
@ -7751,10 +7075,6 @@ snapshots:
|
|||
dependencies:
|
||||
through: 2.3.8
|
||||
|
||||
ssf@0.11.2:
|
||||
dependencies:
|
||||
frac: 1.1.2
|
||||
|
||||
sshpk@1.18.0:
|
||||
dependencies:
|
||||
asn1: 0.2.6
|
||||
|
|
@ -8079,10 +7399,6 @@ snapshots:
|
|||
dependencies:
|
||||
string-width: 4.2.3
|
||||
|
||||
wmf@1.0.2: {}
|
||||
|
||||
word@0.3.0: {}
|
||||
|
||||
workerpool@6.5.1: {}
|
||||
|
||||
workerpool@9.3.4: {}
|
||||
|
|
@ -8107,16 +7423,6 @@ snapshots:
|
|||
|
||||
wrappy@1.0.2: {}
|
||||
|
||||
xlsx@0.18.5:
|
||||
dependencies:
|
||||
adler-32: 1.3.1
|
||||
cfb: 1.2.2
|
||||
codepage: 1.15.0
|
||||
crc-32: 1.2.2
|
||||
ssf: 0.11.2
|
||||
wmf: 1.0.2
|
||||
word: 0.3.0
|
||||
|
||||
xmlbuilder@15.1.1: {}
|
||||
|
||||
y18n@5.0.8: {}
|
||||
|
|
@ -8174,6 +7480,4 @@ snapshots:
|
|||
toposort: 2.0.2
|
||||
type-fest: 2.19.0
|
||||
|
||||
zlibjs@0.3.1: {}
|
||||
|
||||
zod@3.25.76: {}
|
||||
|
|
|
|||
11
services/tech-step-intent-service/.env.example
Normal file
11
services/tech-step-intent-service/.env.example
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
# Secret partagé attendu sur le header `X-Intent-Service-Secret` de chaque
|
||||
# requête (sauf `GET /health`) — doit matcher `INTENT_SERVICE_SECRET` côté
|
||||
# apps/api/.env (voir apps/api/src/config/env.ts). Requis, pas de valeur par
|
||||
# défaut : `Settings` (intent_service/config.py) refuse de démarrer sans.
|
||||
INTENT_SERVICE_SECRET=changeme-generate-a-real-random-secret-at-least-32-chars
|
||||
|
||||
# Optionnel — niveau du logging JSON structuré (intent_service/logging_config.py).
|
||||
# INFO par défaut : chaque appel /v1/process et /v1/train journalise son
|
||||
# input (locale/texte, entrées d'entraînement) et son output (entités,
|
||||
# intent, score) à ce niveau.
|
||||
# LOG_LEVEL=INFO
|
||||
5
services/tech-step-intent-service/.gitignore
vendored
Normal file
5
services/tech-step-intent-service/.gitignore
vendored
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
.venv/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.pytest_cache/
|
||||
.env
|
||||
34
services/tech-step-intent-service/Dockerfile
Normal file
34
services/tech-step-intent-service/Dockerfile
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
# Standalone image for services/tech-step-intent-service — hors du build
|
||||
# apps/api (voir services/tech-step-llm-worker/Dockerfile pour le précédent
|
||||
# direct : un service Python/spaCy n'a rien à faire dans l'image Node de
|
||||
# l'API, et inversement). Rien n'est persisté sur disque (pas de VOLUME,
|
||||
# contrairement au worker LLM) : tout l'état (textcat/matcher entraînés)
|
||||
# vit en mémoire, reconstruit à chaque `/v1/train` depuis un corpus que ce
|
||||
# service ne possède pas lui-même (voir intent_service/README.md).
|
||||
FROM python:3.12-slim AS base
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates && rm -rf /var/lib/apt/lists/*
|
||||
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
|
||||
WORKDIR /service
|
||||
|
||||
FROM base AS build
|
||||
# `uv.lock` est commité pour ce service (même rigueur que
|
||||
# `pnpm-lock.yaml`/`--frozen-lockfile` pour apps/api et
|
||||
# services/tech-step-llm-worker) — `--frozen` échoue bruyamment si
|
||||
# `pyproject.toml` a dérivé du lock plutôt que de re-résoudre en silence.
|
||||
# `--no-install-project` sépare l'installation des dépendances (dont les
|
||||
# wheels de modèles spaCy, pinnés par URL dans pyproject.toml) de la copie
|
||||
# du code applicatif, pour que le cache de layer Docker survive à un
|
||||
# changement dans intent_service/ sans retélécharger ~80 Mo de modèles.
|
||||
# Chemins préfixés par `services/tech-step-intent-service/` : le contexte
|
||||
# de build est la racine du repo (`docker-compose.yml`'s `build.context: .`),
|
||||
# même convention que `services/tech-step-llm-worker/Dockerfile`.
|
||||
COPY services/tech-step-intent-service/pyproject.toml services/tech-step-intent-service/uv.lock ./
|
||||
RUN uv sync --frozen --no-install-project --no-dev
|
||||
COPY services/tech-step-intent-service/intent_service ./intent_service
|
||||
RUN uv sync --frozen --no-dev
|
||||
|
||||
FROM base AS runtime
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
COPY --from=build /service /service
|
||||
EXPOSE 8000
|
||||
CMD ["uv", "run", "uvicorn", "intent_service.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
172
services/tech-step-intent-service/README.md
Normal file
172
services/tech-step-intent-service/README.md
Normal file
|
|
@ -0,0 +1,172 @@
|
|||
# tech-step-intent-service
|
||||
|
||||
Microservice de détection d'intention (technique de cuisine) — remplace le
|
||||
pipeline `node-nlp` qui vivait dans `apps/api`
|
||||
(`TechStepClassifierService`, `apps/api/src/lib/recipe-matching/tech-step-matcher.ts`) :
|
||||
|
||||
1. **NER par phrases** (`spacy.matcher.PhraseMatcher`) — trouve les mentions
|
||||
candidates d'une technique dans un texte, à partir des `synonyms` de
|
||||
chaque technique.
|
||||
2. **Classification d'intention** (`textcat` spaCy, bag-of-words) — verdict
|
||||
de la technique qu'une clause de texte *signifie*, entraîné sur les
|
||||
`utterances` de chaque technique (y compris des paraphrases n'utilisant
|
||||
jamais le mot-clé lui-même).
|
||||
|
||||
Basé sur **spaCy** (`fr_core_news_md`/`en_core_web_md`) plutôt que node-nlp —
|
||||
écosystème NLP plus robuste/maintenu, avec l'ambition à terme (hors scope de
|
||||
ce service en l'état) de pouvoir aussi absorber ce que fait aujourd'hui
|
||||
`services/tech-step-llm-worker` une fois ce pipeline assez riche pour s'en
|
||||
passer (les modèles `md`, avec vecteurs de mots, sont conservés dans ce but,
|
||||
même si rien ici ne s'en sert encore).
|
||||
|
||||
## Ce service est entièrement autonome
|
||||
|
||||
Contrairement à sa toute première version, **ce service possède désormais
|
||||
son propre corpus** — `intent_service/training_data.py`
|
||||
(`TECH_STEP_TRAINING_DATA`), revu par PR comme le reste du code. Il
|
||||
s'entraîne lui-même une seule fois, à son propre démarrage
|
||||
(`PipelineRegistry.initialize()`, appelé par `main.py`'s `lifespan`), et ne
|
||||
persiste jamais rien sur disque — un redémarrage du process réentraîne
|
||||
toujours from scratch depuis ce fichier. `apps/api` ne connaît plus aucune
|
||||
technique ni aucun synonyme : il n'appelle plus que `POST /v1/process` (plus
|
||||
de `POST /v1/train`, supprimé).
|
||||
|
||||
Workflow mainteneur pour changer le corpus :
|
||||
|
||||
1. Éditer `intent_service/training_data.py` à la main (informé par le
|
||||
rapport de `apps/api/src/scripts/list-pending-training-suggestions.ts`).
|
||||
2. **Redémarrer ce service** (`docker compose restart tech-step-intent-service`,
|
||||
ou simplement redéployer) — le nouveau corpus n'a d'effet qu'une fois
|
||||
réentraîné au démarrage, contrairement à l'ancienne version qui pouvait
|
||||
être réentraînée à chaud via `POST /v1/train`.
|
||||
3. Depuis `apps/api`, lancer `pnpm --filter api exec tsx
|
||||
src/scripts/retrain-tech-steps.ts` — vérifie le F1 contre
|
||||
`TECH_STEP_EVAL_DATASET` avant de backfiller les recettes existantes.
|
||||
|
||||
## Pourquoi ce service vit hors du workspace pnpm
|
||||
|
||||
Même raisonnement que `services/tech-step-llm-worker` : un service Python
|
||||
n'a rien à faire dans `pnpm-workspace.yaml` (qui ne couvre que
|
||||
`apps/*`/`packages/*`), et ses dépendances (spaCy, ses modèles) ne doivent
|
||||
jamais se retrouver dans l'image `apps/api`. **Aucun accès direct à
|
||||
Postgres** non plus — la résolution `TechStep.key -> id` reste entièrement
|
||||
côté `apps/api` (`TechStepClassifierService`), ce service ne manipule que
|
||||
des `uid` (chaînes opaques) tout du long.
|
||||
|
||||
## Contrat HTTP
|
||||
|
||||
Voir `intent_service/schemas.py` pour le détail exact. En résumé :
|
||||
|
||||
- `GET /health` — sans authentification, `200` une fois ce service
|
||||
entièrement prêt : modèles spaCy de base chargés **et** les deux locales
|
||||
entraînées (pas de lazy-load, voir `intent_service/main.py`) — voir
|
||||
"Temps de démarrage" plus bas pour ce que ça implique en pratique.
|
||||
- `POST /v1/process` — `{ locale, text }` → `{ entities: [{ uid, start, end }], intent, score }`.
|
||||
|
||||
`/v1/process` exige le header `X-Intent-Service-Secret` (voir
|
||||
`intent_service/security.py`), qui doit matcher `INTENT_SERVICE_SECRET`
|
||||
côté `apps/api`.
|
||||
|
||||
## Temps de démarrage
|
||||
|
||||
**Ce service met plusieurs minutes à devenir `healthy`** — contrairement à
|
||||
node-nlp (entraînement quasi instantané), entraîner le `textcat` sur le
|
||||
corpus réel (~74 techniques, chaque technique entraînée sur ses `synonyms`
|
||||
en plus de ses `utterances` — voir `locale_pipeline.py`) prend de l'ordre
|
||||
de 335 secondes par locale (mesuré localement, sans GPU), donc environ 670
|
||||
secondes (~11 minutes) pour `fr`+`en` combinés à chaque démarrage du
|
||||
process. `docker-compose.yml` et
|
||||
`.github/workflows/ci.yml` ont un `start_period`/timeout d'attente
|
||||
généreux pour ça — voir leurs propres commentaires. C'est un compromis
|
||||
assumé, pas un défaut de configuration à corriger : moins d'itérations
|
||||
entraîne plus vite mais laisse des verdicts corrects sous
|
||||
`CONFIDENCE_THRESHOLD` (voir le commentaire de cette constante,
|
||||
`apps/api/src/lib/recipe-matching/tech-step-matcher.ts`, et celui de
|
||||
`_TRAINING_ITERATIONS`/`_TRAINING_BATCH_SIZE` dans `locale_pipeline.py`
|
||||
pour le détail du compromis).
|
||||
|
||||
## Logs
|
||||
|
||||
`intent_service/logging_config.py` branche un format JSON structuré (une
|
||||
ligne par évènement — `timestamp`/`level`/`message` + champs métier fusionnés
|
||||
— même convention que `LoggerService` côté `apps/api`) sur toute la
|
||||
journalisation de ce service, niveau `LOG_LEVEL` (`INFO` par défaut, voir
|
||||
`.env.example`). `routes/process.py` journalise chaque appel avec son input
|
||||
et son output complets, `pipeline_registry.py` journalise le déroulement de
|
||||
l'entraînement au démarrage :
|
||||
|
||||
```json
|
||||
{"timestamp": "...", "level": "info", "message": "tech-step NLP process", "locale": "fr", "text": "faire fondre le beurre", "entities": [{"uid": "melt", "start": 6, "end": 13}], "intent": "melt", "score": 0.93}
|
||||
```
|
||||
|
||||
Le chatter interne de spaCy (`"spacy"` logger — chargement de vocabulaire,
|
||||
etc.) est explicitement mis à `WARNING` pour ne pas noyer ces lignes.
|
||||
|
||||
## Setup
|
||||
|
||||
Ce service utilise [`uv`](https://docs.astral.sh/uv/) pour ses dépendances
|
||||
(`uv.lock` committé, `uv sync --frozen` partout — Dockerfile, CI, dev).
|
||||
|
||||
```bash
|
||||
cd services/tech-step-intent-service
|
||||
uv sync
|
||||
cp .env.example .env
|
||||
# édite .env : génère un INTENT_SERVICE_SECRET, identique à celui d'apps/api
|
||||
uv run uvicorn intent_service.main:app --reload --port 8000
|
||||
```
|
||||
|
||||
`apps/api` (natif, `pnpm dev:api`, ou sa suite Mocha) doit pointer
|
||||
`INTENT_SERVICE_BASE_URL=http://localhost:8000` et le même
|
||||
`INTENT_SERVICE_SECRET` (voir `apps/api/.env.example`).
|
||||
|
||||
## Running via Docker Compose
|
||||
|
||||
`docker-compose.yml` (racine) définit un service `tech-step-intent-service`
|
||||
aux côtés de `postgres`/`app`/`tech-step-llm-worker` — **pas optionnel**,
|
||||
contrairement au worker LLM : sans lui, `apps/api` ne peut plus détecter
|
||||
aucune technique de cuisine. `app` attend qu'il soit `healthy`
|
||||
(`depends_on: condition: service_healthy`) avant de démarrer — voir "Temps
|
||||
de démarrage" ci-dessus pour combien de temps ça prend en pratique.
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
uv run pytest
|
||||
```
|
||||
|
||||
`tests/test_locale_pipeline_entities.py` rejoue les cas d'offsets caractère
|
||||
exacts et d'insensibilité accents/casse de
|
||||
`apps/api/test/recipe-matching/tech-step-matcher.test.ts` — le point de
|
||||
fidélité le plus critique de ce service (voir le plan de migration).
|
||||
`tests/conftest.py`'s fixture `client` (scope "session") ne s'entraîne
|
||||
qu'une seule fois pour toute la suite — c'est *le vrai corpus complet*,
|
||||
pas un jeu jouet, donc la première utilisation de cette fixture prend le
|
||||
même temps qu'un vrai démarrage (voir "Temps de démarrage" ci-dessus).
|
||||
|
||||
Aucun test ici ne dépend d'une vraie base Postgres ni d'`apps/api` en
|
||||
service — à l'inverse, la suite Mocha d'`apps/api`
|
||||
(`tech-step-matcher.test.ts`/`recipe-translation.test.ts`) exige elle une
|
||||
vraie instance de ce service tournant (voir `apps/api/.env.test`), conforme
|
||||
à la convention du repo de ne jamais mocker un service interne.
|
||||
|
||||
## Limitations connues
|
||||
|
||||
- **Démarrage lent** (~11 minutes) — voir "Temps de démarrage" ci-dessus.
|
||||
Une optimisation possible non explorée : parallélisation de
|
||||
l'entraînement `fr`/`en` (actuellement séquentiel,
|
||||
`PipelineRegistry.initialize`).
|
||||
- **`CONFIDENCE_THRESHOLD` côté `apps/api` est un placeholder** depuis
|
||||
l'élargissement du corpus à ~74 techniques (calibré à la main, pas via
|
||||
une vraie repasse de `calibrate-tech-step-threshold.ts` contre
|
||||
`TECH_STEP_EVAL_DATASET` — voir le commentaire de cette constante).
|
||||
- **Textcat bag-of-words** (`spacy.TextCatBOW.v3`) — suffisant pour le
|
||||
corpus actuel une fois correctement entraîné, mais n'exploite pas les
|
||||
vecteurs de mots des modèles `md` chargés. Migrable vers une architecture
|
||||
tok2vec/similarité sans changer le contrat HTTP, si le F1 mesuré par
|
||||
`apps/api/src/scripts/calibrate-tech-step-threshold.ts` le justifie un
|
||||
jour.
|
||||
- **Reconstruit tout le pipeline à chaque démarrage** (pas de persistance,
|
||||
pas de fusion incrémentale) — un choix délibéré (voir
|
||||
`LocalePipeline.train`), pas une limitation à lever : `training_data.py`
|
||||
doit toujours rester l'unique source de vérité, jamais un état sur disque
|
||||
qui pourrait dériver.
|
||||
12
services/tech-step-intent-service/intent_service/__init__.py
Normal file
12
services/tech-step-intent-service/intent_service/__init__.py
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
"""Microservice de détection d'intention (technique de cuisine).
|
||||
|
||||
Remplace le pipeline `node-nlp` qui vivait dans `apps/api`
|
||||
(`TechStepClassifierService`, `apps/api/src/lib/recipe-matching/tech-step-matcher.ts`) :
|
||||
NER par phrases (synonymes) + classification d'intention (textcat), les deux
|
||||
entraînés à la demande depuis un corpus qui reste possédé par `apps/api`
|
||||
(`TECH_STEP_TRAINING_DATA`) et poussé ici via `POST /v1/train`.
|
||||
|
||||
Ce service ne touche jamais Postgres — voir `services/tech-step-llm-worker`
|
||||
pour le précédent architectural (même posture : aucun accès DB direct,
|
||||
tout passe par HTTP, la résolution `TechStep.key -> id` reste côté `apps/api`).
|
||||
"""
|
||||
52
services/tech-step-intent-service/intent_service/config.py
Normal file
52
services/tech-step-intent-service/intent_service/config.py
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
"""Configuration du service, lue depuis l'environnement (`pydantic-settings`).
|
||||
|
||||
Contrairement à `requireInternalWorker` côté `apps/api`
|
||||
(`apps/api/src/middlewares/require-internal-worker.ts`), qui tolère un
|
||||
`INTERNAL_WORKER_SECRET` absent (le worker LLM est un job de fond
|
||||
optionnel) et échoue "juste" requête par requête dans ce cas, ce service est
|
||||
une dépendance coeur : `INTENT_SERVICE_SECRET` absent doit empêcher
|
||||
`uvicorn` de démarrer du tout plutôt que de démarrer dans un état où chaque
|
||||
requête échouerait silencieusement en boucle — `Settings` n'a donc aucune
|
||||
valeur par défaut ni type optionnel pour ce champ, la validation Pydantic
|
||||
lève dès l'import de ce module si la variable manque.
|
||||
"""
|
||||
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
# `env_file=".env"` : lu uniquement en dev natif (`cp .env.example .env`,
|
||||
# voir le README de ce service) — sans effet en Docker, où
|
||||
# docker-compose.yml passe les variables directement en `environment:`
|
||||
# et où aucun `.env` n'est copié dans l'image. Un `.env` absent n'est pas
|
||||
# une erreur ici (pydantic-settings ignore silencieusement un fichier
|
||||
# manquant) ; c'est bien `intent_service_secret` ci-dessous, sans valeur
|
||||
# par défaut, qui fait échouer le démarrage si la variable n'est
|
||||
# disponible par aucune des deux voies.
|
||||
#
|
||||
# `case_sensitive` par défaut (False) : `INTENT_SERVICE_SECRET` (la
|
||||
# convention majuscule utilisée partout ailleurs dans le repo, cf.
|
||||
# `docker-compose.yml`/`.env.example`) matche bien le champ
|
||||
# `intent_service_secret` ci-dessous.
|
||||
model_config = SettingsConfigDict(env_file=".env")
|
||||
|
||||
# Secret partagé attendu sur le header `X-Intent-Service-Secret` de
|
||||
# chaque requête (sauf `GET /health`) — voir `security.py`. Doit matcher
|
||||
# `INTENT_SERVICE_SECRET` côté `apps/api/src/config/env.ts`.
|
||||
intent_service_secret: str
|
||||
|
||||
# Pas de `port` ici : `uvicorn` prend son port en argument de ligne de
|
||||
# commande (`--port`, voir le Dockerfile et le README de ce service),
|
||||
# jamais lu depuis `Settings` — une variable d'env dupliquant ce que la
|
||||
# commande de démarrage fixe déjà explicitement n'aurait aucun lecteur.
|
||||
|
||||
# Niveau du logging structuré (`logging_config.py`) — voir ce module pour
|
||||
# le format. `INFO` par défaut : c'est à ce niveau que `routes/process.py`
|
||||
# journalise chaque input/output du pipeline NLP, et que
|
||||
# `pipeline_registry.py` journalise l'entraînement au démarrage, pour
|
||||
# qu'un déploiement par défaut les voie sans configuration
|
||||
# supplémentaire (`docker logs`/Portainer).
|
||||
log_level: str = "INFO"
|
||||
|
||||
|
||||
settings = Settings()
|
||||
|
|
@ -0,0 +1,424 @@
|
|||
"""Pipeline spaCy pour UNE locale — l'équivalent Python de ce que
|
||||
`node-nlp`'s `NlpManager` faisait pour cette locale dans
|
||||
`TechStepClassifierService` (`apps/api/src/lib/recipe-matching/tech-step-matcher.ts`) :
|
||||
NER par entités enum (ici un `PhraseMatcher`) + classification d'intention
|
||||
(ici un `textcat`), les deux entraînés à partir du corpus possédé par ce
|
||||
service lui-même (`training_data.TECH_STEP_TRAINING_DATA` — plus poussé par
|
||||
`apps/api` via HTTP, voir `pipeline_registry.py`).
|
||||
|
||||
Le modèle de base spaCy (tokenizer + vecteurs + le composant
|
||||
`diacritics_normalizer` défini plus bas) est chargé une seule fois
|
||||
(`preload()`, appelé au démarrage du process — voir `main.py` — pas
|
||||
paresseusement au premier `train()`, pour que `GET /health` ne devienne
|
||||
`200` qu'une fois ce coût payé) puis réutilisé à chaque `train()` : seul le
|
||||
`textcat` (retiré puis rajouté à neuf) et le `PhraseMatcher` (remplacé) sont
|
||||
reconstruits à chaque appel, jamais le tokenizer/les vecteurs. Rien n'est
|
||||
jamais persisté sur disque — `training_data.py` reste l'unique source de
|
||||
vérité, reconstruite en mémoire depuis zéro à chaque démarrage du process.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import random
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
import spacy
|
||||
from spacy.language import Language
|
||||
from spacy.matcher import PhraseMatcher
|
||||
from spacy.tokens import Doc, Span
|
||||
from spacy.training import Example
|
||||
from spacy.util import filter_spans, fix_random_seed, minibatch
|
||||
|
||||
from .text_normalization import normalize_text
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Modèle spaCy de base par locale — voir pyproject.toml pour la version
|
||||
# pinnée exacte. `md` (pas `sm`) : conserve les vecteurs de mots, inutilisés
|
||||
# par le pipeline v1 (textcat bag-of-words) mais retenus pour l'ambition
|
||||
# future de similarité sémantique (voir le README de ce service).
|
||||
SUPPORTED_LOCALES = {
|
||||
"fr": "fr_core_news_md",
|
||||
"en": "en_core_web_md",
|
||||
}
|
||||
|
||||
# Composants du modèle de base non utilisés par ce pipeline (on ne s'appuie
|
||||
# ni sur le NER générique de spaCy, ni sur l'analyse syntaxique/morphologique
|
||||
# — seuls le tokenizer et les vecteurs de mots restent nécessaires) : les
|
||||
# exclure au chargement évite le coût mémoire/CPU de composants qui ne
|
||||
# tourneraient jamais.
|
||||
_EXCLUDED_COMPONENTS = ["parser", "ner", "tagger", "morphologizer", "attribute_ruler", "lemmatizer"]
|
||||
|
||||
_TEXTCAT_PIPE_NAME = "textcat"
|
||||
|
||||
# Nombre d'itérations d'entraînement du textcat et taille de minibatch —
|
||||
# calibrés empiriquement contre le corpus réel (`training_data.py`), pas
|
||||
# seulement contre les petits corpus jouets des tests de ce fichier. Trop
|
||||
# peu d'itérations laisse des clauses correctement classifiées (bon argmax)
|
||||
# mais avec une confiance dérisoire — bien en dessous de tout seuil
|
||||
# raisonnable pour `CONFIDENCE_THRESHOLD` (`tech-step-matcher.ts`).
|
||||
#
|
||||
# Trois passes de calibration successives, toutes mesurées contre le
|
||||
# corpus réel (74 techniques) :
|
||||
# 1. `150` itérations (calibré pour le corpus original, ~26 techniques) ne
|
||||
# passe plus à l'échelle une fois élargi : `150` sur 74 classes
|
||||
# dépassait 17 minutes pour une seule locale, constaté en CI.
|
||||
# 2. `40` itérations, `examples` limité aux `utterances` (pas les
|
||||
# `synonyms`) : ~200s/locale, mais confiance faible sur les clauses
|
||||
# ancrées sans paraphrase entraînée (`simmer`/`cook`/`bake` ~0.25-0.34).
|
||||
# 3. **Configuration actuelle** : les `synonyms` de chaque technique sont
|
||||
# désormais aussi des exemples d'entraînement du textcat (voir plus bas
|
||||
# dans `train()`) — un signal "mot-clé isolé -> sa propre technique"
|
||||
# qui manquait complètement avant. À `_TRAINING_ITERATIONS` inchangé
|
||||
# (40), le nombre d'exemples par époque grimpe de ~286 à ~749 et le
|
||||
# temps d'entraînement suit (~535s/locale) ; réduire à `25` retrouve un
|
||||
# temps proche de l'étape 2 (~336s/locale, ~670s pour fr+en combinés)
|
||||
# tout en gardant l'essentiel du gain de confiance apporté par les
|
||||
# synonymes : melt ~0.89, preheat ~0.77, compote ~0.78, julienne ~0.76,
|
||||
# zest ~0.66, bake ~0.62, cook ~0.38, simmer ~0.31 — le plus faible
|
||||
# observé, mais désormais nettement au-dessus du seuil de confiance
|
||||
# (contre ~0.25, sous le seuil d'alors, à l'étape 2). Bruit
|
||||
# hors-vocabulaire toujours négligeable (anglais via le classifieur
|
||||
# français : `~0.02`). Une vraie repasse de
|
||||
# `calibrate-tech-step-threshold.ts` contre `TECH_STEP_EVAL_DATASET`
|
||||
# reste nécessaire pour confirmer/affiner ces valeurs (voir
|
||||
# `CONFIDENCE_THRESHOLD`'s propre commentaire, `tech-step-matcher.ts`)
|
||||
# — ce qui précède est une mesure manuelle ponctuelle, pas un
|
||||
# remplacement de cette calibration.
|
||||
_TRAINING_ITERATIONS = 25
|
||||
_TRAINING_BATCH_SIZE = 16
|
||||
# Arrêt anticipé : `_TRAINING_ITERATIONS` reste le plafond (le pire cas ne
|
||||
# change pas), un corpus/locale qui converge plus vite n'a pas à payer les
|
||||
# itérations restantes pour rien. Une époque compte comme "sans progrès"
|
||||
# quand sa perte totale ne descend pas d'au moins `_EARLY_STOPPING_MIN_DELTA`
|
||||
# sous la meilleure perte vue jusqu'ici ; `_EARLY_STOPPING_PATIENCE` époques
|
||||
# consécutives sans progrès arrêtent l'entraînement.
|
||||
#
|
||||
# Mesuré contre le corpus réel (74 techniques, budget de 40 itérations,
|
||||
# avant le passage à 25) : ne s'est jamais déclenché — la perte continuait
|
||||
# de baisser significativement sur toute la plage (cohérent avec la
|
||||
# confiance qui grimpait encore nettement entre 15 et 40 itérations, voir
|
||||
# le commentaire de `_TRAINING_ITERATIONS`). Ce n'est donc pas un gain de
|
||||
# temps aujourd'hui, mais un filet de sécurité peu coûteux pour la suite : si
|
||||
# `_TRAINING_ITERATIONS` est un jour augmenté pour une meilleure confiance,
|
||||
# ceci évite de payer des itérations supplémentaires une fois la
|
||||
# convergence réellement atteinte, sans qu'il faille retrouver le bon
|
||||
# plafond à la main à chaque changement du corpus.
|
||||
_EARLY_STOPPING_PATIENCE = 3
|
||||
_EARLY_STOPPING_MIN_DELTA = 0.001
|
||||
# Abaissé de `0.2` avec le reste de cette recalibration — `0.1` régularise
|
||||
# encore contre la petite taille du corpus par technique tout en laissant
|
||||
# plus de signal passer à chaque pas, ce qui a mesurablement aidé la
|
||||
# confiance finale sans signe de sur-ajustement (le bruit hors-vocabulaire
|
||||
# reste aussi bas qu'avant, voir ci-dessus).
|
||||
_TRAINING_DROPOUT = 0.1
|
||||
# Seed fixe — un warm-up reproductible d'un redémarrage à l'autre (même
|
||||
# corpus en entrée) est préférable à un score qui varie légèrement à chaque
|
||||
# déploiement pour la même donnée, en particulier pendant la calibration du
|
||||
# seuil de confiance côté apps/api.
|
||||
_TRAINING_SEED = 0
|
||||
|
||||
|
||||
class _DiacriticsNormalizer:
|
||||
"""Composant de pipeline réécrivant `token.norm_` avec `normalize_text()`
|
||||
(le port Python de `normalizeText()` côté `apps/api`) pour chaque token.
|
||||
|
||||
Point clé : ce composant tourne aussi bien sur les `Doc` construits pour
|
||||
les *patterns* du `PhraseMatcher` (voir `LocalePipeline.train`) que sur
|
||||
le *texte cible* passé à `process()` — les deux passent donc par
|
||||
exactement la même normalisation, ce qui garantit qu'un synonyme comme
|
||||
"mijoter" matche indifféremment "MIJOTER"/"mijoté"/"Mijotée" dans le
|
||||
texte, reproduisant le comportement `ner.threshold: 1` (exact après
|
||||
normalisation, sans tolérance floue Levenshtein) de l'ancien `NlpManager`.
|
||||
Indépendant des `entries` entraînées — ajouté une seule fois par
|
||||
`preload()`, jamais retiré/rajouté par `train()`.
|
||||
|
||||
Opère token par token, sur du texte déjà tokenisé — `normalize_text()`
|
||||
ne fait que réécrire la forme d'un token existant (minuscule, sans
|
||||
diacritique), jamais fusionner/scinder des tokens : les patterns
|
||||
(`nlp.make_doc(synonym)` + ce composant appliqué à la main, voir
|
||||
`LocalePipeline.train`) et le texte cible (`nlp(text)`, pipeline
|
||||
complet) passent donc toujours par le *même* découpage en tokens que
|
||||
le tokenizer du modèle de base leur donne, avant que ce composant n'y
|
||||
touche — pas de risque de désalignement entre les deux.
|
||||
"""
|
||||
|
||||
def __call__(self, doc: Doc) -> Doc:
|
||||
for token in doc:
|
||||
token.norm_ = normalize_text(token.text)
|
||||
return doc
|
||||
|
||||
|
||||
@Language.factory("diacritics_normalizer")
|
||||
def _create_diacritics_normalizer(nlp: Language, name: str) -> _DiacriticsNormalizer:
|
||||
return _DiacriticsNormalizer()
|
||||
|
||||
|
||||
@dataclass
|
||||
class TrainEntry:
|
||||
"""Une technique à entraîner pour une locale — construit par
|
||||
`PipelineRegistry.initialize()` depuis `training_data.entries_for_locale`."""
|
||||
|
||||
uid: str
|
||||
synonyms: list[str] = field(default_factory=list)
|
||||
utterances: list[str] = field(default_factory=list)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Entity:
|
||||
"""Une mention candidate trouvée par le `PhraseMatcher` — offsets
|
||||
caractère `[start, end)`, miroir de `EntityPayload` (`schemas.py`)."""
|
||||
|
||||
uid: str
|
||||
start: int
|
||||
end: int
|
||||
|
||||
|
||||
@dataclass
|
||||
class ProcessResult:
|
||||
"""Résultat complet d'un `process()` — miroir de `ProcessResponse`
|
||||
(`schemas.py`)."""
|
||||
|
||||
entities: list[Entity]
|
||||
intent: str | None
|
||||
score: float
|
||||
|
||||
|
||||
class UnsupportedLocaleError(ValueError):
|
||||
"""`locale` ne correspond à aucun modèle spaCy connu (voir
|
||||
`SUPPORTED_LOCALES`) — distinct d'une locale simplement "pas encore
|
||||
entraînée" (`LocalePipeline.is_trained is False`), qui n'est pas une
|
||||
erreur (voir `process()`)."""
|
||||
|
||||
|
||||
class LocalePipeline:
|
||||
"""Pipeline spaCy (NER par phrases + textcat) pour une locale donnée.
|
||||
Un `PipelineRegistry` (voir `pipeline_registry.py`) en détient une
|
||||
instance par locale supportée.
|
||||
"""
|
||||
|
||||
def __init__(self, locale: str) -> None:
|
||||
if locale not in SUPPORTED_LOCALES:
|
||||
raise UnsupportedLocaleError(f"Unsupported locale: {locale!r}")
|
||||
self._locale = locale
|
||||
self._model_name = SUPPORTED_LOCALES[locale]
|
||||
# `None` tant que `preload()` n'a pas tourné.
|
||||
self._base_nlp: Language | None = None
|
||||
# `None` tant qu'aucun `train()` n'a réussi — `process()` traite ça
|
||||
# comme "rien à trouver" plutôt qu'une erreur, exactement le
|
||||
# comportement testé côté `apps/api` pour "une locale jamais
|
||||
# entraînée".
|
||||
self._matcher: PhraseMatcher | None = None
|
||||
self._trained = False
|
||||
|
||||
@property
|
||||
def is_trained(self) -> bool:
|
||||
return self._trained
|
||||
|
||||
def preload(self) -> None:
|
||||
"""Charge le modèle spaCy de base (tokenizer + vecteurs) et le
|
||||
composant `diacritics_normalizer` — idempotent, sans effet si déjà
|
||||
chargé. Appelé au démarrage du process pour les deux locales
|
||||
connues (voir `main.py`), pas paresseusement au premier `train()`.
|
||||
"""
|
||||
if self._base_nlp is not None:
|
||||
return
|
||||
nlp = spacy.load(self._model_name, exclude=_EXCLUDED_COMPONENTS)
|
||||
nlp.add_pipe("diacritics_normalizer", first=True)
|
||||
self._base_nlp = nlp
|
||||
|
||||
def train(self, entries: list[TrainEntry]) -> tuple[int, int, int]:
|
||||
"""Reconstruit le `textcat` et le `PhraseMatcher` de ce pipeline à
|
||||
partir de `entries` (le tokenizer/les vecteurs restent ceux chargés
|
||||
par `preload()`). Retourne `(label_count, example_count,
|
||||
synonym_count)` pour la journalisation (`pipeline_registry.py`) —
|
||||
`example_count` est le nombre réel d'exemples donnés au `textcat`
|
||||
(`utterances` *et* `synonyms` combinés, voir plus bas), pas
|
||||
seulement `entry.utterances`.
|
||||
|
||||
`entries` vide retombe à `is_trained == False` plutôt que de lever —
|
||||
un appelant qui n'a rien à entraîner pour cette locale obtient le
|
||||
même comportement que "jamais entraîné", pas une erreur 500.
|
||||
"""
|
||||
self.preload()
|
||||
assert self._base_nlp is not None # garanti par preload() ci-dessus
|
||||
|
||||
if _TEXTCAT_PIPE_NAME in self._base_nlp.pipe_names:
|
||||
self._base_nlp.remove_pipe(_TEXTCAT_PIPE_NAME)
|
||||
|
||||
if not entries:
|
||||
self._matcher = None
|
||||
self._trained = False
|
||||
return (0, 0, 0)
|
||||
|
||||
nlp = self._base_nlp
|
||||
# `nlp.make_doc()` ne fait tourner *que* le tokenizer, pas les
|
||||
# composants du pipeline — le `diacritics_normalizer` ajouté par
|
||||
# `preload()` ne tournerait donc jamais sur les `Doc` de patterns
|
||||
# s'ils n'étaient construits qu'avec `make_doc()`, alors que
|
||||
# `process()` appelle `nlp(text)` (le pipeline complet) sur le texte
|
||||
# cible. Sans ce correctif, un synonyme accentué comme "préchauffer"
|
||||
# n'aurait jamais matché "PRÉCHAUFFER"/"Préchauffer" : trouvé en
|
||||
# calibrant contre les cas exacts de `tech-step-matcher.test.ts`
|
||||
# (fr, la locale la plus concernée par les accents) — un synonyme
|
||||
# sans diacritique comme "faire fondre" masquait le bug en semblant
|
||||
# fonctionner par coïncidence. Appliquer explicitement le même
|
||||
# composant aux deux côtés garantit qu'ils passent par la même
|
||||
# normalisation.
|
||||
diacritics_normalizer = nlp.get_pipe("diacritics_normalizer")
|
||||
|
||||
matcher = PhraseMatcher(nlp.vocab, attr="NORM")
|
||||
synonym_count = 0
|
||||
for entry in entries:
|
||||
if not entry.synonyms:
|
||||
continue
|
||||
patterns = [diacritics_normalizer(nlp.make_doc(synonym)) for synonym in entry.synonyms]
|
||||
matcher.add(entry.uid, patterns)
|
||||
synonym_count += len(entry.synonyms)
|
||||
|
||||
# `textcat` (exclusive_classes) exige au moins deux labels (voir
|
||||
# spaCy's error E867) — jamais un problème avec le vrai corpus
|
||||
# (`TECH_STEP_TRAINING_DATA` a ~74 techniques), mais un `entries` à
|
||||
# un seul élément resterait structurellement valide pour le NER
|
||||
# seul : ne pas planter, juste ne pas construire de textcat du tout
|
||||
# (`process()` retombe alors sur `intent: null` via son garde
|
||||
# `if not cats`, exactement comme "rien à classifier"). Journalisé
|
||||
# explicitement — sans ça, "pourquoi cette locale ne classifie
|
||||
# jamais rien" ne serait visible qu'en déduisant `labelCount < 2`
|
||||
# de la ligne "tech-step NLP pipeline trained" (`pipeline_registry.py`).
|
||||
examples: list[Example] = []
|
||||
if len(entries) < 2:
|
||||
logger.warning(
|
||||
"tech-step NLP textcat skipped: fewer than 2 labels, intent classification disabled for this locale",
|
||||
extra={"locale": self._locale, "labelCount": len(entries)},
|
||||
)
|
||||
else:
|
||||
textcat = nlp.add_pipe(
|
||||
_TEXTCAT_PIPE_NAME,
|
||||
config={
|
||||
"model": {
|
||||
"@architectures": "spacy.TextCatBOW.v3",
|
||||
"exclusive_classes": True,
|
||||
"ngram_size": 1,
|
||||
"no_output_layer": False,
|
||||
},
|
||||
},
|
||||
)
|
||||
for entry in entries:
|
||||
textcat.add_label(entry.uid)
|
||||
|
||||
for entry in entries:
|
||||
cats = {other.uid: 0.0 for other in entries}
|
||||
cats[entry.uid] = 1.0
|
||||
# `synonyms` (déjà utilisés pour le `PhraseMatcher` ci-dessus)
|
||||
# sont aussi de bonnes phrases d'entraînement pour le
|
||||
# `textcat` — un texte réduit au mot-clé lui-même ("fondre",
|
||||
# "faire fondre") est le cas le plus net qui soit pour sa
|
||||
# propre technique, et n'était auparavant vu par le textcat
|
||||
# que noyé dans le contexte plus riche des `utterances`.
|
||||
for text in (*entry.synonyms, *entry.utterances):
|
||||
doc = nlp.make_doc(text)
|
||||
examples.append(Example.from_dict(doc, {"cats": cats}))
|
||||
|
||||
# Graine le RNG Python *et* celui de numpy/thinc sous-jacent à
|
||||
# `nlp.update()` (initialisation des poids, masque de dropout) —
|
||||
# `random.Random(_TRAINING_SEED)` ci-dessous ne couvre que l'ordre
|
||||
# de mélange des exemples choisi par ce module, pas ce que spaCy
|
||||
# fait en interne à chaque pas de gradient.
|
||||
fix_random_seed(_TRAINING_SEED)
|
||||
rng = random.Random(_TRAINING_SEED)
|
||||
if examples:
|
||||
optimizer = nlp.initialize(lambda: examples)
|
||||
best_loss = float("inf")
|
||||
epochs_without_improvement = 0
|
||||
for iteration in range(_TRAINING_ITERATIONS):
|
||||
rng.shuffle(examples)
|
||||
losses: dict[str, float] = {}
|
||||
for batch in minibatch(examples, size=_TRAINING_BATCH_SIZE):
|
||||
nlp.update(batch, sgd=optimizer, drop=_TRAINING_DROPOUT, losses=losses)
|
||||
epoch_loss = losses.get(_TEXTCAT_PIPE_NAME, 0.0)
|
||||
# Arrêt anticipé — voir `_EARLY_STOPPING_PATIENCE`'s propre
|
||||
# commentaire. `_TRAINING_ITERATIONS` reste le plafond
|
||||
# (pire cas inchangé), ceci ne fait que raccourcir les
|
||||
# cas qui convergent plus vite.
|
||||
if epoch_loss < best_loss - _EARLY_STOPPING_MIN_DELTA:
|
||||
best_loss = epoch_loss
|
||||
epochs_without_improvement = 0
|
||||
else:
|
||||
epochs_without_improvement += 1
|
||||
if epochs_without_improvement >= _EARLY_STOPPING_PATIENCE:
|
||||
logger.info(
|
||||
"tech-step NLP textcat training stopped early",
|
||||
extra={
|
||||
"locale": self._locale,
|
||||
"iteration": iteration + 1,
|
||||
"maxIterations": _TRAINING_ITERATIONS,
|
||||
"finalLoss": epoch_loss,
|
||||
},
|
||||
)
|
||||
break
|
||||
else:
|
||||
# Des `entries` avec des `uid` mais aucune `utterance` nulle
|
||||
# part (corpus incomplet) : le textcat a des labels mais rien
|
||||
# pour apprendre à les distinguer — toujours initialisé pour
|
||||
# rester un pipeline valide ; `process()` renverra alors un
|
||||
# score ~uniforme entre labels. Ce n'est pas ce module qui doit
|
||||
# juger la qualité du corpus reçu (voir `tech-step-eval-runner.ts`
|
||||
# côté apps/api pour ce rôle).
|
||||
nlp.initialize()
|
||||
|
||||
self._matcher = matcher
|
||||
self._trained = True
|
||||
return (len(entries), len(examples), synonym_count)
|
||||
|
||||
def process(self, text: str) -> ProcessResult:
|
||||
"""Reproduit la forme de `NlpManager.process(locale, text)` : les
|
||||
entités candidates (NER) et le verdict du classifieur d'intention
|
||||
sur `text` tel quel — que ce soit la description complète ou une
|
||||
clause déjà découpée côté `apps/api`, ce module ne le sait pas et ne
|
||||
s'en soucie pas, exactement comme l'ancien `NlpManager`.
|
||||
|
||||
`intent` vaut `None` dans deux cas distincts, tous deux silencieux
|
||||
côté retour (voir le log d'avertissement de `train()` pour repérer
|
||||
le second en amont) : `text` vide/blanc, ou `doc.cats` vide parce
|
||||
que `train()` a reçu moins de deux labels pour cette locale (le
|
||||
textcat n'a alors jamais été construit — voir son propre
|
||||
commentaire).
|
||||
"""
|
||||
if not self._trained or self._base_nlp is None or self._matcher is None or not text.strip():
|
||||
return ProcessResult(entities=[], intent=None, score=0.0)
|
||||
|
||||
doc = self._base_nlp(text)
|
||||
|
||||
# A technique's own synonym list can legitimately contain one phrase
|
||||
# nested inside another (`melt`'s "fondre" is a literal substring of
|
||||
# its own "faire fondre") — the `PhraseMatcher` reports *both* as
|
||||
# separate matches at overlapping positions, which without
|
||||
# resolution would hand `splitIntoClauses` (apps/api) two candidates
|
||||
# for what a human reads as one mention, producing the same
|
||||
# techStepId twice in the final result. `filter_spans` keeps only
|
||||
# the longest match at each position (so "faire fondre" wins over
|
||||
# the "fondre" it contains) — found by a real regression in
|
||||
# `tech-step-matcher.test.ts`'s "detects several distinct
|
||||
# techniques..." case once this service replaced node-nlp (which
|
||||
# apparently resolved this internally; nothing here recreates that
|
||||
# by choice, `filter_spans` is spaCy's own documented tool for
|
||||
# exactly this "one span per position" problem, e.g. as used for
|
||||
# NER-style outputs).
|
||||
matched_spans = [
|
||||
Span(doc, start, end, label=match_id) for match_id, start, end in self._matcher(doc)
|
||||
]
|
||||
entities = sorted(
|
||||
(
|
||||
Entity(uid=self._base_nlp.vocab.strings[span.label], start=span.start_char, end=span.end_char)
|
||||
for span in filter_spans(matched_spans)
|
||||
),
|
||||
key=lambda entity: entity.start,
|
||||
)
|
||||
|
||||
cats = doc.cats
|
||||
if not cats:
|
||||
return ProcessResult(entities=entities, intent=None, score=0.0)
|
||||
intent = max(cats, key=cats.get)
|
||||
return ProcessResult(entities=entities, intent=intent, score=cats[intent])
|
||||
|
|
@ -0,0 +1,77 @@
|
|||
"""Logging structuré — même convention que `LoggerService` côté `apps/api`
|
||||
(`apps/api/src/lib/logger.service.ts`) : une ligne JSON par évènement
|
||||
(`timestamp`, `level`, `message`, + le reste des champs fournis fusionné),
|
||||
jamais du texte libre, pour rester grep/parse-able par `docker logs`/
|
||||
Portainer ou un agrégateur de logs — cohérent avec le reste du repo plutôt
|
||||
qu'un format propre à ce seul service.
|
||||
|
||||
Configuré une fois au démarrage (`main.py`) plutôt que par un `print()` ad
|
||||
hoc dans chaque route — `routes/process.py`/`pipeline_registry.py` appellent
|
||||
`logging.getLogger(__name__)` normalement, ce module ne fait que brancher le
|
||||
formateur JSON sur la racine du logging Python.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import sys
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
|
||||
class _JsonFormatter(logging.Formatter):
|
||||
"""Sérialise chaque `LogRecord` en une ligne JSON. Les champs
|
||||
supplémentaires passés via `logger.info(msg, extra={...})` sont fusionnés
|
||||
tels quels dans l'objet — c'est ce que `routes/process.py` utilise pour
|
||||
joindre `locale`/`text`/`entities`/`intent`/`score` à la ligne."""
|
||||
|
||||
# Attributs standards de `LogRecord` — tout le reste posé sur le record
|
||||
# (via `extra=`) est un champ métier ajouté par l'appelant, à fusionner
|
||||
# dans la sortie JSON.
|
||||
_STANDARD_ATTRS = frozenset(logging.LogRecord("", 0, "", 0, "", None, None).__dict__.keys())
|
||||
|
||||
def format(self, record: logging.LogRecord) -> str:
|
||||
payload: dict[str, Any] = {
|
||||
"timestamp": datetime.fromtimestamp(record.created, tz=UTC).isoformat(),
|
||||
"level": record.levelname.lower(),
|
||||
"message": record.getMessage(),
|
||||
}
|
||||
extra_fields = {
|
||||
key: value for key, value in record.__dict__.items() if key not in self._STANDARD_ATTRS
|
||||
}
|
||||
payload.update(extra_fields)
|
||||
if record.exc_info:
|
||||
payload["error"] = self.formatException(record.exc_info)
|
||||
return json.dumps(payload, ensure_ascii=False, default=str)
|
||||
|
||||
|
||||
def configure_logging(level: str) -> None:
|
||||
"""Branche le formateur JSON sur la racine du logging Python — appelé
|
||||
une fois au démarrage (`main.py`), avant que `routes/*` ne journalisent
|
||||
quoi que ce soit."""
|
||||
# L'encodage par défaut de `sys.stdout` suit la locale de l'OS/console,
|
||||
# pas forcément UTF-8 — sur Windows en particulier, garder ce défaut
|
||||
# produit de vrais octets invalides (pas juste un affichage terminal
|
||||
# trompeur) pour tout texte accentué journalisé par `routes/process.py`
|
||||
# (le texte réel des étapes de recette, en français) — trouvé en
|
||||
# vérifiant les octets bruts d'un log réel, pas juste son affichage.
|
||||
# `reconfigure` existe sur `sys.stdout` dans toute exécution Python
|
||||
# normale (pas dans certains contextes embarqués/redirigés exotiques) —
|
||||
# protégé par `hasattr` pour ne jamais faire planter le démarrage pour un
|
||||
# souci de confort d'affichage.
|
||||
if hasattr(sys.stdout, "reconfigure"):
|
||||
sys.stdout.reconfigure(encoding="utf-8")
|
||||
handler = logging.StreamHandler(sys.stdout)
|
||||
handler.setFormatter(_JsonFormatter())
|
||||
root = logging.getLogger()
|
||||
root.handlers = [handler]
|
||||
root.setLevel(level)
|
||||
|
||||
# spaCy/thinc journalisent leur propre chatter interne ("Created
|
||||
# vocabulary", "Finished initializing nlp object"...) sur le logger
|
||||
# `"spacy"`, qui propage jusqu'à la racine et se retrouverait donc
|
||||
# mélangé aux lignes input/output de `routes/process.py`/l'entraînement
|
||||
# journalisé par `pipeline_registry.py`
|
||||
# — ce sont ces dernières que ce service existe pour rendre visibles, pas
|
||||
# le détail interne de spaCy. `WARNING` laisse quand même remonter un
|
||||
# vrai problème (dépréciation, échec partiel) sans le bruit `INFO`.
|
||||
logging.getLogger("spacy").setLevel(logging.WARNING)
|
||||
39
services/tech-step-intent-service/intent_service/main.py
Normal file
39
services/tech-step-intent-service/intent_service/main.py
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
"""Point d'entrée FastAPI — `uv run uvicorn intent_service.main:app` (voir
|
||||
le Dockerfile et le README de ce service).
|
||||
|
||||
Le chargement des modèles spaCy de base *et* l'entraînement de chaque
|
||||
locale (`PipelineRegistry.initialize`) se font dans le handler `lifespan`
|
||||
ci-dessous, *avant* qu'uvicorn n'accepte de requêtes — `GET /health` ne
|
||||
répond donc `200` qu'une fois ce coût payé (chargement + entraînement),
|
||||
jamais pendant qu'il est encore en cours (uvicorn ne sert aucune requête
|
||||
tant que le `lifespan` de démarrage n'est pas terminé). Ce service est
|
||||
autonome : `training_data.TECH_STEP_TRAINING_DATA` vit dans ce module,
|
||||
`apps/api` ne pousse plus rien via HTTP (voir `pipeline_registry.py` pour
|
||||
le détail de ce que ça change par rapport à la version précédente).
|
||||
"""
|
||||
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from fastapi import FastAPI
|
||||
|
||||
from .config import settings
|
||||
from .logging_config import configure_logging
|
||||
from .pipeline_registry import registry
|
||||
from .routes import health, process
|
||||
|
||||
# Avant tout le reste : `routes/process.py` journalise dès la première
|
||||
# requête, `initialize()` ci-dessous journalise aussi (voir
|
||||
# `pipeline_registry.py`) — le formateur JSON doit déjà être en place.
|
||||
configure_logging(settings.log_level)
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
registry.initialize()
|
||||
yield
|
||||
|
||||
|
||||
app = FastAPI(title="tech-step-intent-service", lifespan=lifespan)
|
||||
|
||||
app.include_router(health.router)
|
||||
app.include_router(process.router)
|
||||
|
|
@ -0,0 +1,75 @@
|
|||
"""Détient un `LocalePipeline` par locale supportée — le seul état mutable
|
||||
partagé du process (une instance vit pour toute la durée de vie d'`uvicorn`,
|
||||
montée sur `app.state`, voir `main.py`).
|
||||
|
||||
Volontairement une classe "registre" séparée de `LocalePipeline` lui-même :
|
||||
`LocalePipeline` ne connaît qu'une seule locale, ce module route `process`
|
||||
vers la bonne instance selon le `locale` reçu dans la requête — même
|
||||
séparation de responsabilité que `TechStepClassifierService` (une seule
|
||||
instance, un seul `NlpManager` multi-langues) avait implicitement via
|
||||
node-nlp, explicitée ici puisque spaCy charge un modèle par langue.
|
||||
"""
|
||||
|
||||
import logging
|
||||
|
||||
from .locale_pipeline import SUPPORTED_LOCALES, LocalePipeline, ProcessResult, TrainEntry
|
||||
from .training_data import entries_for_locale
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class PipelineRegistry:
|
||||
def __init__(self) -> None:
|
||||
self._pipelines: dict[str, LocalePipeline] = {
|
||||
locale: LocalePipeline(locale) for locale in SUPPORTED_LOCALES
|
||||
}
|
||||
|
||||
def initialize(self) -> None:
|
||||
"""Charge le modèle spaCy de base *et* entraîne chaque locale connue
|
||||
depuis `training_data.TECH_STEP_TRAINING_DATA` — appelé une fois au
|
||||
démarrage du process (`main.py`'s `lifespan`), avant que `uvicorn`
|
||||
n'accepte de requêtes.
|
||||
|
||||
Contrairement à la version précédente de ce service (où `apps/api`
|
||||
poussait le corpus via `POST /v1/train` à son propre warm-up), ce
|
||||
service est maintenant entièrement autonome : `apps/api` ne connaît
|
||||
plus aucune technique, seulement le résultat de
|
||||
`POST /v1/process`. `GET /health` ne répond `200` qu'une fois cette
|
||||
méthode terminée (chargement *et* entraînement) — pas seulement le
|
||||
chargement — pour que `docker-compose.yml`'s `depends_on: ...
|
||||
condition: service_healthy` (et la boucle d'attente équivalente en
|
||||
CI) ne laisse jamais `apps/api` démarrer face à un service qui
|
||||
répondrait mais ne saurait encore rien détecter.
|
||||
"""
|
||||
logger.info("tech-step NLP initializing pipelines", extra={"locales": list(self._pipelines)})
|
||||
for locale, pipeline in self._pipelines.items():
|
||||
pipeline.preload()
|
||||
entries = [TrainEntry(**entry) for entry in entries_for_locale(locale)]
|
||||
label_count, example_count, synonym_count = pipeline.train(entries)
|
||||
logger.info(
|
||||
"tech-step NLP pipeline trained",
|
||||
extra={
|
||||
"locale": locale,
|
||||
"labelCount": label_count,
|
||||
# Nombre réel d'exemples donnés au textcat (utterances
|
||||
# *et* synonyms combinés — voir `LocalePipeline.train`),
|
||||
# pas seulement le compte d'`utterances` du corpus.
|
||||
"exampleCount": example_count,
|
||||
"synonymCount": synonym_count,
|
||||
},
|
||||
)
|
||||
logger.info("tech-step NLP pipelines ready", extra={"locales": list(self._pipelines)})
|
||||
|
||||
def process(self, locale: str, text: str) -> ProcessResult:
|
||||
pipeline = self._pipelines.get(locale)
|
||||
if pipeline is None:
|
||||
# Une locale que ce service ne sait structurellement pas
|
||||
# charger (pas de modèle spaCy connu) se comporte comme une
|
||||
# locale "jamais entraînée" côté `process` — reproduit le test
|
||||
# `apps/api` existant ("returns an empty sequence for a locale
|
||||
# nothing was trained on"), qui ne distingue pas les deux cas.
|
||||
return ProcessResult(entities=[], intent=None, score=0.0)
|
||||
return pipeline.process(text)
|
||||
|
||||
|
||||
registry = PipelineRegistry()
|
||||
|
|
@ -0,0 +1,18 @@
|
|||
"""`GET /health` — sondé par le `healthcheck` Docker (`docker-compose.yml`)
|
||||
et par l'étape CI qui attend que ce service soit prêt avant de lancer la
|
||||
suite Mocha de `apps/api` (voir `.github/workflows/ci.yml`). Volontairement
|
||||
sans authentification, même posture que le `GET /health` existant côté
|
||||
`apps/api` (`app.ts`) — un healthcheck qui exigerait un secret compliquerait
|
||||
sa configuration pour un gain de sécurité nul (il ne renvoie aucune donnée).
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter
|
||||
|
||||
from ..schemas import HealthResponse
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/health", response_model=HealthResponse)
|
||||
def health() -> HealthResponse:
|
||||
return HealthResponse(status="ok")
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
"""`POST /v1/process` — appelé par `apps/api` (`IntentServiceClient.process`)
|
||||
en remplacement direct de l'ancien `NlpManager.process(locale, text)`. Voir
|
||||
`LocalePipeline.process` pour la sémantique exacte (locale non entraînée ou
|
||||
`text` vide -> résultat vide, jamais une erreur).
|
||||
"""
|
||||
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
|
||||
from ..pipeline_registry import registry
|
||||
from ..schemas import EntityPayload, ProcessRequest, ProcessResponse
|
||||
from ..security import require_valid_secret
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(dependencies=[Depends(require_valid_secret)])
|
||||
|
||||
|
||||
@router.post("/v1/process", response_model=ProcessResponse)
|
||||
def process(request: ProcessRequest) -> ProcessResponse:
|
||||
result = registry.process(request.locale, request.text)
|
||||
|
||||
# Une ligne par appel — input (`locale`/`text`) et output (`entities`/
|
||||
# `intent`/`score`) réunis dans la même ligne JSON, pour pouvoir suivre
|
||||
# exactement ce que le pipeline a décidé pour un texte donné (voir
|
||||
# `logging_config.py` pour le format).
|
||||
logger.info(
|
||||
"tech-step NLP process",
|
||||
extra={
|
||||
"locale": request.locale,
|
||||
"text": request.text,
|
||||
"entities": [{"uid": entity.uid, "start": entity.start, "end": entity.end} for entity in result.entities],
|
||||
"intent": result.intent,
|
||||
"score": result.score,
|
||||
},
|
||||
)
|
||||
|
||||
return ProcessResponse(
|
||||
entities=[EntityPayload(uid=entity.uid, start=entity.start, end=entity.end) for entity in result.entities],
|
||||
intent=result.intent,
|
||||
score=result.score,
|
||||
)
|
||||
45
services/tech-step-intent-service/intent_service/schemas.py
Normal file
45
services/tech-step-intent-service/intent_service/schemas.py
Normal file
|
|
@ -0,0 +1,45 @@
|
|||
"""Modèles Pydantic du contrat HTTP — voir le plan de migration pour le
|
||||
contrat exact attendu côté `apps/api` (`IntentServiceClient`,
|
||||
`apps/api/src/lib/recipe-matching/intent-service-client.ts`).
|
||||
|
||||
Pas de `POST /v1/train` ici — ce service s'entraîne lui-même au démarrage
|
||||
depuis `training_data.py` (voir `pipeline_registry.py`/`main.py`), plus
|
||||
besoin d'un contrat HTTP pour ça.
|
||||
"""
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# POST /v1/process
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class ProcessRequest(BaseModel):
|
||||
locale: str
|
||||
text: str
|
||||
|
||||
|
||||
class EntityPayload(BaseModel):
|
||||
"""Une mention candidate d'une technique — offsets caractère `[start, end)`
|
||||
dans `text`, convention identique à `String.prototype.slice` côté
|
||||
`apps/api` (pas de décalage `+1` à appliquer côté Node, contrairement à
|
||||
l'ancien `NlpManager` de node-nlp)."""
|
||||
|
||||
uid: str
|
||||
start: int
|
||||
end: int
|
||||
|
||||
|
||||
class ProcessResponse(BaseModel):
|
||||
entities: list[EntityPayload]
|
||||
intent: str | None
|
||||
score: float
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# GET /health
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class HealthResponse(BaseModel):
|
||||
status: str
|
||||
35
services/tech-step-intent-service/intent_service/security.py
Normal file
35
services/tech-step-intent-service/intent_service/security.py
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
"""Authentification des appels entrants — miroir inversé de `requireInternalWorker`
|
||||
(`apps/api/src/middlewares/require-internal-worker.ts`) : ici c'est
|
||||
`apps/api` qui appelle *ce* service, donc c'est ce service qui vérifie le
|
||||
secret plutôt que de l'envoyer.
|
||||
|
||||
Comparaison à temps constant (`hmac.compare_digest`, l'équivalent Python du
|
||||
`timingSafeEqual` de Node utilisé côté `apps/api`) — même raisonnement :
|
||||
un attaquant ne doit rien apprendre de la durée de la comparaison au-delà de
|
||||
ce qu'une différence de longueur révèle déjà.
|
||||
"""
|
||||
|
||||
import hmac
|
||||
|
||||
from fastapi import Header, HTTPException, status
|
||||
|
||||
from .config import settings
|
||||
|
||||
_SECRET_HEADER_NAME = "x-intent-service-secret"
|
||||
|
||||
|
||||
def require_valid_secret(
|
||||
x_intent_service_secret: str | None = Header(default=None, alias=_SECRET_HEADER_NAME),
|
||||
) -> None:
|
||||
"""Dépendance FastAPI montée sur chaque route protégée (`/v1/*`) — pas
|
||||
`GET /health`, sondé par le healthcheck Docker sans configuration
|
||||
d'auth propre.
|
||||
|
||||
`settings.intent_service_secret` est garanti non vide par `config.py`
|
||||
(pas de valeur par défaut dans `Settings`) — le seul cas à traiter ici
|
||||
est un header manquant ou incorrect côté appelant.
|
||||
"""
|
||||
if x_intent_service_secret is None or not hmac.compare_digest(
|
||||
x_intent_service_secret, settings.intent_service_secret
|
||||
):
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Not authenticated")
|
||||
|
|
@ -0,0 +1,32 @@
|
|||
"""Port Python de `normalizeText` (`apps/api/src/lib/recipe-matching/tech-step-matcher.ts`).
|
||||
|
||||
Doit rester bit-pour-bit équivalent à sa contrepartie TypeScript — c'est ce
|
||||
qui garantit qu'un synonyme matché ici tombe exactement sur les mêmes
|
||||
positions caractère que ce que `apps/api` attendait de node-nlp (voir
|
||||
`LocalePipeline`'s `diacritics_normalizer`, qui applique cette fonction aux
|
||||
patterns *et* au texte cible pour les faire matcher identiquement).
|
||||
|
||||
TypeScript original :
|
||||
|
||||
const COMBINING_DIACRITICS_PATTERN = /\\p{Diacritic}/gu;
|
||||
export function normalizeText(text: string): string {
|
||||
return text.normalize("NFD").replace(COMBINING_DIACRITICS_PATTERN, "").toLowerCase();
|
||||
}
|
||||
|
||||
`unicodedata.combining(ch) != 0` (catégories Unicode Mn/Mc, la classe de
|
||||
combinaison canonique) est l'idiome Python standard pour "strip accents
|
||||
after NFD" — légèrement plus étroit que `\\p{Diacritic}` en théorie (qui
|
||||
couvre aussi quelques diacritiques autonomes hors caractères combinants),
|
||||
mais strictement équivalent pour tout caractère latin accentué usuel
|
||||
(français/anglais) une fois décomposé en NFD, ce qui est le seul cas
|
||||
réellement exercé par ce corpus.
|
||||
"""
|
||||
|
||||
import unicodedata
|
||||
|
||||
|
||||
def normalize_text(text: str) -> str:
|
||||
"""Décompose en NFD, retire les marques combinantes (accents), met en minuscule."""
|
||||
decomposed = unicodedata.normalize("NFD", text)
|
||||
stripped = "".join(char for char in decomposed if not unicodedata.combining(char))
|
||||
return stripped.lower()
|
||||
1692
services/tech-step-intent-service/intent_service/training_data.py
Normal file
1692
services/tech-step-intent-service/intent_service/training_data.py
Normal file
File diff suppressed because it is too large
Load diff
53
services/tech-step-intent-service/pyproject.toml
Normal file
53
services/tech-step-intent-service/pyproject.toml
Normal file
|
|
@ -0,0 +1,53 @@
|
|||
[project]
|
||||
name = "tech-step-intent-service"
|
||||
version = "0.1.0"
|
||||
description = "Microservice de détection d'intention (technique de cuisine) — remplace node-nlp côté apps/api."
|
||||
requires-python = ">=3.12"
|
||||
dependencies = [
|
||||
"fastapi>=0.115,<0.116",
|
||||
"uvicorn[standard]>=0.32,<0.33",
|
||||
"spacy>=3.8,<3.9",
|
||||
# Fournit les tables de lookup ("lexeme_norm" notamment) que
|
||||
# `nlp.initialize()` réclame pour l'anglais lors de l'entraînement du
|
||||
# textcat (`en_core_web_md` ne les embarque pas lui-même, contrairement à
|
||||
# `fr_core_news_md`) — sans ce paquet, entraîner un pipeline "en" lève
|
||||
# `E955`.
|
||||
"spacy-lookups-data>=1.0,<1.1",
|
||||
"pydantic-settings>=2.6,<3",
|
||||
# Modèles spaCy installés comme des dépendances pip normales, pinnées par
|
||||
# URL de release GitHub (pas via `python -m spacy download`, qui résout
|
||||
# "la dernière version compatible" et n'est pas verrouillable par
|
||||
# `uv.lock`). `uv sync --frozen` installe donc déjà les modèles — aucune
|
||||
# étape `spacy download` séparée, ni au Dockerfile ni en CI. Version
|
||||
# 3.8.0 choisie pour matcher la ligne spaCy 3.8 pinnée ci-dessus (voir
|
||||
# https://github.com/explosion/spacy-models/releases).
|
||||
"fr_core_news_md @ https://github.com/explosion/spacy-models/releases/download/fr_core_news_md-3.8.0/fr_core_news_md-3.8.0-py3-none-any.whl",
|
||||
"en_core_web_md @ https://github.com/explosion/spacy-models/releases/download/en_core_web_md-3.8.0/en_core_web_md-3.8.0-py3-none-any.whl",
|
||||
]
|
||||
|
||||
[dependency-groups]
|
||||
dev = [
|
||||
"pytest>=8,<9",
|
||||
# Requis par fastapi.testclient.TestClient (httpx en interne depuis FastAPI 0.110+).
|
||||
"httpx>=0.27,<0.28",
|
||||
]
|
||||
|
||||
[tool.uv]
|
||||
# Les deux modèles ci-dessus sont publiés comme des builds "any" universels
|
||||
# (pas de wheel spécifique par plateforme) — rien à déclarer de plus ici,
|
||||
# contrairement à un paquet avec des extras natifs par OS/arch.
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["intent_service"]
|
||||
|
||||
[tool.hatch.metadata]
|
||||
# Requis par hatchling pour accepter des dépendances pinnées par URL directe
|
||||
# (les wheels de modèles spaCy ci-dessus) plutôt qu'un nom+version résolu
|
||||
# depuis un index PyPI — voir la note sur `pyproject.toml` dans le plan de
|
||||
# migration pour pourquoi ces modèles sont déclarés ainsi plutôt que via
|
||||
# `python -m spacy download`.
|
||||
allow-direct-references = true
|
||||
31
services/tech-step-intent-service/tests/conftest.py
Normal file
31
services/tech-step-intent-service/tests/conftest.py
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
"""`Settings` (`intent_service/config.py`) lève dès l'import si
|
||||
`INTENT_SERVICE_SECRET` est absent — cette variable doit donc être définie
|
||||
avant le tout premier `import intent_service...` de la session pytest.
|
||||
`conftest.py` est chargé par pytest avant la collecte des modules de test,
|
||||
donc avant que `test_routes_process.py`/`test_security.py` n'importent
|
||||
`intent_service.main`.
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
os.environ.setdefault("INTENT_SERVICE_SECRET", "pytest-only-secret-not-used-anywhere-else-32ch")
|
||||
|
||||
import pytest # noqa: E402 — après le `setdefault` ci-dessus, voir le docstring.
|
||||
from fastapi.testclient import TestClient # noqa: E402
|
||||
|
||||
from intent_service.main import app # noqa: E402
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def client():
|
||||
"""`TestClient(app)` utilisé comme gestionnaire de contexte déclenche le
|
||||
vrai `lifespan` — puisque `main.py`'s `lifespan` entraîne maintenant
|
||||
l'intégralité du vrai corpus `training_data.TECH_STEP_TRAINING_DATA`
|
||||
(pas un jeu jouet, voir `PipelineRegistry.initialize`), refaire ça une
|
||||
fois par fichier de test (ou pire, une fois par test) multiplierait un
|
||||
entraînement non négligeable sur toute la suite pour rien — scope
|
||||
"session" pour que chaque test ayant besoin d'une vraie app en cours
|
||||
d'exécution partage la même instance déjà entraînée.
|
||||
"""
|
||||
with TestClient(app) as test_client:
|
||||
yield test_client
|
||||
|
|
@ -0,0 +1,148 @@
|
|||
"""Rejoue les cas d'offsets caractère exacts et d'insensibilité accents/casse
|
||||
de `tech-step-matcher.test.ts` (`apps/api/test/recipe-matching/tech-step-matcher.test.ts`)
|
||||
contre le `PhraseMatcher`/`diacritics_normalizer` de `LocalePipeline` — le
|
||||
point de fidélité le plus critique de cette migration (voir le plan). Doit
|
||||
être vert *avant* de brancher `apps/api` dessus.
|
||||
|
||||
Ces tests entraînent un pipeline minimal (pas le corpus complet
|
||||
`TECH_STEP_TRAINING_DATA`, propriété de `apps/api`) avec juste assez de
|
||||
`synonyms`/`utterances` pour reproduire chaque cas — le textcat n'est pas ce
|
||||
qui est vérifié ici (voir `test_locale_pipeline_intent.py`).
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
from intent_service.locale_pipeline import LocalePipeline, TrainEntry
|
||||
|
||||
# Un jeu d'entrées minimal mais réaliste, reprenant les synonymes réels de
|
||||
# `tech-step-training-data.ts` pour "preheat"/"melt" qui rendent les cas
|
||||
# `tech-step-matcher.test.ts` exacts (voir ce fichier, lignes 155/787).
|
||||
_FR_ENTRIES = [
|
||||
TrainEntry(
|
||||
uid="preheat",
|
||||
synonyms=["préchauffer", "poêle chaude"],
|
||||
utterances=["préchauffer le four à 180 degrés", "mettre la poêle sur feu vif"],
|
||||
),
|
||||
TrainEntry(
|
||||
# `synonyms` deliberately includes both "fondre" (standalone) and
|
||||
# "faire fondre" (containing it) — mirrors the real corpus
|
||||
# (`tech-step-training-data.ts`) exactly, and is what
|
||||
# `test_does_not_double_match_a_synonym_nested_in_a_longer_one`
|
||||
# below exists to guard: the `PhraseMatcher` reports both as
|
||||
# separate overlapping matches, `LocalePipeline.process` must
|
||||
# collapse them into one.
|
||||
uid="melt",
|
||||
synonyms=["fondre", "fondu", "faire fondre", "faire chauffer"],
|
||||
utterances=["faire fondre le beurre", "faire chauffer une noix de beurre"],
|
||||
),
|
||||
TrainEntry(
|
||||
uid="simmer",
|
||||
synonyms=["mijoter"],
|
||||
utterances=["faire mijoter à feu doux", "laisser mijoter à couvert"],
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def fr_pipeline() -> LocalePipeline:
|
||||
pipeline = LocalePipeline("fr")
|
||||
pipeline.train(_FR_ENTRIES)
|
||||
return pipeline
|
||||
|
||||
|
||||
def test_matches_an_exact_expression(fr_pipeline: LocalePipeline):
|
||||
result = fr_pipeline.process("Faire mijoter à feu doux")
|
||||
assert [entity.uid for entity in result.entities] == ["simmer"]
|
||||
entity = result.entities[0]
|
||||
text = "Faire mijoter à feu doux"
|
||||
assert text[entity.start : entity.end].lower() == "mijoter"
|
||||
|
||||
|
||||
def test_is_case_and_accent_insensitive(fr_pipeline: LocalePipeline):
|
||||
result = fr_pipeline.process("FAIRE MIJOTER")
|
||||
assert [entity.uid for entity in result.entities] == ["simmer"]
|
||||
|
||||
|
||||
def test_returns_no_entities_when_nothing_matches(fr_pipeline: LocalePipeline):
|
||||
result = fr_pipeline.process("Ranger les couverts dans le tiroir")
|
||||
assert result.entities == []
|
||||
|
||||
|
||||
def test_returns_empty_for_an_empty_text(fr_pipeline: LocalePipeline):
|
||||
result = fr_pipeline.process("")
|
||||
assert result.entities == []
|
||||
assert result.intent is None
|
||||
assert result.score == 0.0
|
||||
|
||||
|
||||
def test_untrained_locale_returns_empty_without_error():
|
||||
pipeline = LocalePipeline("en")
|
||||
result = pipeline.process("melt the butter")
|
||||
assert result.entities == []
|
||||
assert result.intent is None
|
||||
assert result.score == 0.0
|
||||
|
||||
|
||||
def test_detects_two_techniques_with_exact_tight_spans_reading_order(fr_pipeline: LocalePipeline):
|
||||
text = "Préchauffer la poêle, puis faire fondre le beurre"
|
||||
result = fr_pipeline.process(text)
|
||||
|
||||
uids_by_start = sorted(((entity.start, entity.uid) for entity in result.entities))
|
||||
assert [uid for _, uid in uids_by_start] == ["preheat", "melt"]
|
||||
|
||||
preheat_entity = next(e for e in result.entities if e.uid == "preheat")
|
||||
melt_entity = next(e for e in result.entities if e.uid == "melt")
|
||||
assert text[preheat_entity.start : preheat_entity.end].lower() == "préchauffer"
|
||||
assert text[melt_entity.start : melt_entity.end].lower() == "faire fondre"
|
||||
|
||||
|
||||
def test_does_not_double_match_a_synonym_nested_in_a_longer_one(fr_pipeline: LocalePipeline):
|
||||
# Regression: "fondre" is itself a substring of "faire fondre" — both
|
||||
# are registered as `melt` synonyms (like the real corpus). Without
|
||||
# `filter_spans` in `LocalePipeline.process`, the `PhraseMatcher`
|
||||
# reports *both* overlapping matches, producing `melt` twice in
|
||||
# apps/api's final `matchTechSteps` output instead of once (caught by a
|
||||
# real CI failure in `tech-step-matcher.test.ts` once this service
|
||||
# replaced node-nlp).
|
||||
text = "faire fondre le beurre"
|
||||
result = fr_pipeline.process(text)
|
||||
assert [entity.uid for entity in result.entities] == ["melt"]
|
||||
entity = result.entities[0]
|
||||
assert text[entity.start : entity.end] == "faire fondre"
|
||||
|
||||
|
||||
def test_matches_the_classic_poele_chaude_example_with_exact_offsets(fr_pipeline: LocalePipeline):
|
||||
# Le cas motivant les context spans côté apps/api (tech-step-matcher.test.ts) :
|
||||
# le mot-clé de `preheat` est un groupe nominal ("poêle chaude"), pas un
|
||||
# verbe. Offsets attendus IDENTIQUES à ceux du test TS d'origine :
|
||||
# preheat -> [9, 21) ("poêle chaude"), melt -> [23, 37) ("faire chauffer").
|
||||
text = "Dans une poêle chaude, faire chauffer une noix de beurre"
|
||||
result = fr_pipeline.process(text)
|
||||
|
||||
preheat_entity = next(e for e in result.entities if e.uid == "preheat")
|
||||
melt_entity = next(e for e in result.entities if e.uid == "melt")
|
||||
|
||||
assert (preheat_entity.start, preheat_entity.end) == (9, 21)
|
||||
assert text[preheat_entity.start : preheat_entity.end] == "poêle chaude"
|
||||
|
||||
assert (melt_entity.start, melt_entity.end) == (23, 37)
|
||||
assert text[melt_entity.start : melt_entity.end] == "faire chauffer"
|
||||
|
||||
|
||||
def test_chop_matches_english_text_tight_span():
|
||||
pipeline = LocalePipeline("en")
|
||||
pipeline.train(
|
||||
[
|
||||
TrainEntry(
|
||||
uid="chop",
|
||||
synonyms=["chop"],
|
||||
utterances=["chop the onions finely", "finely chop the garlic"],
|
||||
),
|
||||
TrainEntry(uid="boil", synonyms=["boil"], utterances=["bring to the boil", "boil the water"]),
|
||||
]
|
||||
)
|
||||
text = "Chop the onions finely"
|
||||
result = pipeline.process(text)
|
||||
chop_entity = next(e for e in result.entities if e.uid == "chop")
|
||||
assert (chop_entity.start, chop_entity.end) == (0, 4)
|
||||
assert text[chop_entity.start : chop_entity.end] == "Chop"
|
||||
|
|
@ -0,0 +1,80 @@
|
|||
"""Vérifie le round-trip entraînement -> prédiction du `textcat` (la partie
|
||||
"comprendre le sens, pas juste les mots clés" du pipeline — voir le
|
||||
commentaire de `tech-step-matcher.ts` côté apps/api pour la motivation
|
||||
d'origine)."""
|
||||
|
||||
from intent_service.locale_pipeline import LocalePipeline, TrainEntry
|
||||
|
||||
_ENTRIES = [
|
||||
TrainEntry(
|
||||
uid="melt",
|
||||
synonyms=["faire fondre"],
|
||||
utterances=[
|
||||
"faire fondre le beurre à feu doux",
|
||||
"laisser fondre le beurre dans la poêle",
|
||||
"jusqu'à ce que le beurre ait disparu",
|
||||
"jusqu'à ce que le beurre ait complètement disparu dans la poêle",
|
||||
],
|
||||
),
|
||||
TrainEntry(
|
||||
uid="boil",
|
||||
synonyms=["bouillir"],
|
||||
utterances=[
|
||||
"porter l'eau à ébullition",
|
||||
"faire bouillir l'eau salée",
|
||||
"laisser bouillir quelques minutes",
|
||||
"porter à ébullition puis baisser le feu",
|
||||
],
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
def test_train_returns_label_example_and_synonym_counts():
|
||||
pipeline = LocalePipeline("fr")
|
||||
label_count, example_count, synonym_count = pipeline.train(_ENTRIES)
|
||||
assert label_count == 2
|
||||
# `example_count` couvre les utterances *et* les synonyms (voir
|
||||
# LocalePipeline.train — les synonymes sont aussi des exemples
|
||||
# d'entraînement pour le textcat, pas seulement pour le PhraseMatcher).
|
||||
expected_examples = sum(len(entry.utterances) + len(entry.synonyms) for entry in _ENTRIES)
|
||||
assert example_count == expected_examples
|
||||
assert synonym_count == sum(len(entry.synonyms) for entry in _ENTRIES)
|
||||
assert pipeline.is_trained is True
|
||||
|
||||
|
||||
def test_classifies_a_paraphrase_never_using_the_techniques_own_verb():
|
||||
# Le cas motivant tout le pipeline (voir tech-step-matcher.ts) : aucune
|
||||
# forme de "fondre" dans cette phrase, mais elle ne peut raisonnablement
|
||||
# signifier que `melt` une fois le textcat entraîné sur les paraphrases
|
||||
# ci-dessus.
|
||||
pipeline = LocalePipeline("fr")
|
||||
pipeline.train(_ENTRIES)
|
||||
|
||||
result = pipeline.process("jusqu'à ce que le beurre ait disparu dans la poêle")
|
||||
assert result.intent == "melt"
|
||||
assert result.score > 0.5
|
||||
|
||||
|
||||
def test_empty_entries_leaves_the_pipeline_untrained():
|
||||
pipeline = LocalePipeline("fr")
|
||||
pipeline.train([])
|
||||
assert pipeline.is_trained is False
|
||||
result = pipeline.process("faire fondre le beurre")
|
||||
assert result.intent is None
|
||||
assert result.entities == []
|
||||
|
||||
|
||||
def test_retraining_replaces_the_previous_textcat_rather_than_accumulating():
|
||||
# `textcat` (exclusive_classes) exige >= 2 labels (voir la note dans
|
||||
# LocalePipeline.train) — le second entraînement garde donc 2 entrées,
|
||||
# mais remplace "boil" par une technique différente ("chop"), pour
|
||||
# vérifier que "boil" ne peut plus jamais ressortir après coup (pas de
|
||||
# fusion incrémentale — voir la doc de `LocalePipeline.train`).
|
||||
pipeline = LocalePipeline("fr")
|
||||
pipeline.train(_ENTRIES)
|
||||
|
||||
chop_entry = TrainEntry(uid="chop", synonyms=["couper"], utterances=["couper les légumes en dés"])
|
||||
pipeline.train([_ENTRIES[0], chop_entry])
|
||||
|
||||
result = pipeline.process("porter l'eau à ébullition")
|
||||
assert result.intent != "boil"
|
||||
|
|
@ -0,0 +1,57 @@
|
|||
"""Vérifie le format des lignes de log produites par
|
||||
`logging_config._JsonFormatter` — ce que `routes/process.py` et
|
||||
`pipeline_registry.py` utilisent pour journaliser l'input/l'output de
|
||||
chaque appel NLP et le déroulement de l'entraînement au démarrage."""
|
||||
|
||||
import json
|
||||
import logging
|
||||
|
||||
from intent_service.logging_config import _JsonFormatter
|
||||
|
||||
|
||||
def _make_record(**extra: object) -> logging.LogRecord:
|
||||
record = logging.LogRecord(
|
||||
name="intent_service.routes.process",
|
||||
level=logging.INFO,
|
||||
pathname=__file__,
|
||||
lineno=1,
|
||||
msg="tech-step NLP process",
|
||||
args=(),
|
||||
exc_info=None,
|
||||
)
|
||||
for key, value in extra.items():
|
||||
setattr(record, key, value)
|
||||
return record
|
||||
|
||||
|
||||
def test_formats_a_record_as_json_with_timestamp_level_and_message():
|
||||
record = _make_record()
|
||||
payload = json.loads(_JsonFormatter().format(record))
|
||||
assert payload["message"] == "tech-step NLP process"
|
||||
assert payload["level"] == "info"
|
||||
assert "timestamp" in payload
|
||||
|
||||
|
||||
def test_merges_extra_fields_into_the_top_level_payload():
|
||||
record = _make_record(
|
||||
locale="fr",
|
||||
text="faire fondre le beurre",
|
||||
entities=[{"uid": "melt", "start": 0, "end": 12}],
|
||||
intent="melt",
|
||||
score=0.93,
|
||||
)
|
||||
payload = json.loads(_JsonFormatter().format(record))
|
||||
assert payload["locale"] == "fr"
|
||||
assert payload["text"] == "faire fondre le beurre"
|
||||
assert payload["entities"] == [{"uid": "melt", "start": 0, "end": 12}]
|
||||
assert payload["intent"] == "melt"
|
||||
assert payload["score"] == 0.93
|
||||
|
||||
|
||||
def test_preserves_accented_characters_literally_not_escaped():
|
||||
# `ensure_ascii=False` — un `docker logs` humain doit pouvoir lire
|
||||
# directement "poêle", pas "poêle".
|
||||
record = _make_record(text="Dans une poêle chaude")
|
||||
line = _JsonFormatter().format(record)
|
||||
assert "poêle" in line
|
||||
assert "\\u00ea" not in line
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
"""Contrat JSON de `POST /v1/process` — voir `schemas.py`/`routes/process.py`.
|
||||
|
||||
Le service s'entraîne désormais lui-même au démarrage sur le vrai corpus
|
||||
(`training_data.TECH_STEP_TRAINING_DATA`, voir `conftest.py`'s fixture
|
||||
`client` partagée) — ces tests vérifient donc le contrat HTTP contre des
|
||||
phrases réelles du corpus, plus besoin d'un `POST /v1/train` préalable avec
|
||||
des données jouets.
|
||||
"""
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from intent_service.config import settings
|
||||
|
||||
_HEADERS = {"X-Intent-Service-Secret": settings.intent_service_secret}
|
||||
|
||||
|
||||
def test_process_against_an_unsupported_locale_returns_empty_result(client: TestClient):
|
||||
# "de" n'a aucun modèle spaCy connu (`SUPPORTED_LOCALES`) — se comporte
|
||||
# comme "jamais entraîné" côté `/v1/process`, jamais une erreur (voir
|
||||
# `PipelineRegistry.process`).
|
||||
response = client.post(
|
||||
"/v1/process", headers=_HEADERS, json={"locale": "de", "text": "faire mijoter à feu doux"}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert response.json() == {"entities": [], "intent": None, "score": 0.0}
|
||||
|
||||
|
||||
def test_process_returns_entities_and_intent_for_a_real_corpus_sentence(client: TestClient):
|
||||
response = client.post(
|
||||
"/v1/process", headers=_HEADERS, json={"locale": "fr", "text": "Faire mijoter à feu doux"}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert body["intent"] == "simmer"
|
||||
assert body["score"] > 0
|
||||
assert [entity["uid"] for entity in body["entities"]] == ["simmer"]
|
||||
|
||||
|
||||
def test_process_matches_english_text_against_the_english_trained_vocabulary(client: TestClient):
|
||||
response = client.post(
|
||||
"/v1/process", headers=_HEADERS, json={"locale": "en", "text": "Chop the onions finely"}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
body = response.json()
|
||||
assert [entity["uid"] for entity in body["entities"]] == ["chop"]
|
||||
|
||||
|
||||
def test_process_with_blank_text_returns_empty_result(client: TestClient):
|
||||
response = client.post("/v1/process", headers=_HEADERS, json={"locale": "fr", "text": " "})
|
||||
assert response.status_code == 200
|
||||
assert response.json() == {"entities": [], "intent": None, "score": 0.0}
|
||||
37
services/tech-step-intent-service/tests/test_security.py
Normal file
37
services/tech-step-intent-service/tests/test_security.py
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
"""`require_valid_secret` — miroir inversé de
|
||||
`require-internal-worker.test.ts` côté `apps/api`. Utilise la fixture
|
||||
`client` partagée (`conftest.py`) — pas besoin d'une app entraînée
|
||||
séparément juste pour tester l'authentification.
|
||||
"""
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from intent_service.config import settings
|
||||
|
||||
|
||||
def test_rejects_a_missing_secret(client: TestClient):
|
||||
response = client.post("/v1/process", json={"locale": "fr", "text": "faire fondre"})
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
def test_rejects_a_wrong_secret(client: TestClient):
|
||||
response = client.post(
|
||||
"/v1/process",
|
||||
json={"locale": "fr", "text": "faire fondre"},
|
||||
headers={"X-Intent-Service-Secret": "not-the-right-secret"},
|
||||
)
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
def test_accepts_the_configured_secret(client: TestClient):
|
||||
response = client.post(
|
||||
"/v1/process",
|
||||
json={"locale": "fr", "text": "faire fondre"},
|
||||
headers={"X-Intent-Service-Secret": settings.intent_service_secret},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
|
||||
def test_health_requires_no_secret(client: TestClient):
|
||||
response = client.get("/health")
|
||||
assert response.status_code == 200
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
"""Réplique les cas de `normalizeText` de `tech-step-matcher.test.ts`
|
||||
(`apps/api/test/recipe-matching/tech-step-matcher.test.ts`) contre le port
|
||||
Python — les deux fonctions doivent rester bit-pour-bit équivalentes."""
|
||||
|
||||
from intent_service.text_normalization import normalize_text
|
||||
|
||||
|
||||
def test_lowercases_and_strips_accents():
|
||||
assert normalize_text("Déglacer AU FOUR") == "deglacer au four"
|
||||
|
||||
|
||||
def test_strips_a_variety_of_diacritics_including_cedilla():
|
||||
assert normalize_text("Façon Œuf à l'Étouffée") == "facon œuf a l'etouffee"
|
||||
|
||||
|
||||
def test_leaves_already_plain_text_unchanged_aside_from_casing():
|
||||
assert normalize_text("Mix everything") == "mix everything"
|
||||
|
||||
|
||||
def test_returns_an_empty_string_for_an_empty_input():
|
||||
assert normalize_text("") == ""
|
||||
1560
services/tech-step-intent-service/uv.lock
Normal file
1560
services/tech-step-intent-service/uv.lock
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -473,49 +473,64 @@ sens est sans ambiguïté. `TechStepMapping` a été supprimée (migration
|
|||
interrogée/éditée à l'exécution, les données de matching vivent en code
|
||||
(`tech-step-training-data.ts`).
|
||||
|
||||
`node-nlp` a ensuite été remplacé à son tour par un microservice Python dédié,
|
||||
`services/tech-step-intent-service` (spaCy — `PhraseMatcher` + `textcat`),
|
||||
appelé en HTTP par `TechStepClassifierService` via `IntentServiceClient`
|
||||
(`intent-service-client.ts`) — `node-nlp` était peu maintenu et tournait
|
||||
in-process dans l'event loop Node ; spaCy offre un écosystème NLP plus
|
||||
robuste, dans un processus séparé, avec l'ambition à terme de pouvoir aussi
|
||||
absorber ce que fait `services/tech-step-llm-worker`. Ce service est
|
||||
entièrement autonome : `TECH_STEP_TRAINING_DATA` (~74 techniques) vit
|
||||
désormais dans son propre `training_data.py`, revu par PR comme le reste du
|
||||
code mais plus poussé par `apps/api` via HTTP — le service s'entraîne
|
||||
lui-même une seule fois, à son propre démarrage, et ne touche jamais
|
||||
Postgres (voir son propre README, y compris pour le temps de démarrage —
|
||||
plusieurs minutes, l'entraînement n'étant jamais persisté sur disque).
|
||||
|
||||
`normalizeText` (décomposition NFD + suppression des diacritiques + minuscule)
|
||||
reste utilisée par `ingredient-matcher.ts`, mais n'intervient plus dans la
|
||||
détection des techniques elle-même — node-nlp gère sa propre normalisation
|
||||
par langue.
|
||||
détection des techniques elle-même — un port Python de cette même fonction
|
||||
(`intent_service/text_normalization.py`) alimente le composant de
|
||||
normalisation du pipeline spaCy côté service.
|
||||
|
||||
**Pipeline en 3 étapes** (`TechStepClassifierService.matchTechStepSpans`) :
|
||||
1. **NER** (entités enum node-nlp, `synonyms` de `TECH_STEP_TRAINING_DATA`)
|
||||
trouve chaque mention *candidate* d'une technique dans la description
|
||||
entière, avec sa position exacte — équivalent mécanique des anciennes
|
||||
regex, en listes de synonymes plutôt qu'en patterns écrits à la main.
|
||||
`ner.threshold: 1` (exact après normalisation, pas de tolérance floue
|
||||
Levenshtein) — le défaut à 0.8 faisait matcher "faire" (verbe auxiliaire
|
||||
omniprésent en français) contre le synonyme "frire" de `fry` par pure
|
||||
proximité de chaîne, un faux positif détecté en calibrant contre le
|
||||
corpus réel.
|
||||
1. **NER** (le `PhraseMatcher` du service, construit depuis les `synonyms` de
|
||||
`TECH_STEP_TRAINING_DATA`) trouve chaque mention *candidate* d'une
|
||||
technique dans la description entière, avec sa position exacte —
|
||||
équivalent mécanique des anciennes regex, en listes de synonymes plutôt
|
||||
qu'en patterns écrits à la main. Le matching se fait sur une normalisation
|
||||
stricte (accents/casse) sans tolérance floue de type Levenshtein — voir
|
||||
`services/tech-step-intent-service/intent_service/locale_pipeline.py`.
|
||||
2. La description est découpée en clauses autour de ces candidats
|
||||
(`splitIntoClauses`, pure/testable sans modèle) — une étape nommant deux
|
||||
techniques a besoin que chacune soit jugée sur son propre contexte, pas
|
||||
la phrase entière classée d'un bloc.
|
||||
3. **Classification d'intention NLP** (le même `NlpManager`, entraîné sur les
|
||||
`utterances` de `TECH_STEP_TRAINING_DATA`) classe chaque clause
|
||||
3. **Classification d'intention NLP** (le `textcat` du service, entraîné sur
|
||||
les `utterances` de `TECH_STEP_TRAINING_DATA`) classe chaque clause
|
||||
individuellement — c'est ce qui apporte la compréhension du **sens** :
|
||||
le corpus d'entraînement mélange volontairement des tournures ancrées sur
|
||||
le mot-clé et des paraphrases qui ne l'emploient jamais (ex. "jusqu'à ce
|
||||
que le beurre ait disparu" pour `melt`), donc le verdict final d'une
|
||||
clause vient de ce que le modèle reconnaît comme *signifiant* la
|
||||
technique, pas du mot littéral qui a déclenché son découpage. En dessous
|
||||
de `CONFIDENCE_THRESHOLD` (0.65 — ajusté empiriquement contre le corpus
|
||||
réel, voir `test/tech-step-matcher.test.ts`), retombe sur la technique
|
||||
impliquée par l'ancre NER de la clause plutôt que d'abandonner un match
|
||||
clairement ancré sur un mot-clé juste parce qu'un petit modèle n'est pas
|
||||
assez confiant.
|
||||
de `CONFIDENCE_THRESHOLD` (voir la constante dans `tech-step-matcher.ts`
|
||||
pour la valeur courante et comment elle a été calibrée), retombe sur la
|
||||
technique impliquée par l'ancre NER de la clause plutôt que d'abandonner
|
||||
un match clairement ancré sur un mot-clé juste parce que le modèle n'est
|
||||
pas assez confiant.
|
||||
|
||||
Entraînement (`_train`) et résolution `TechStep.key -> id` sont mémoïsés une
|
||||
seule fois sur le singleton partagé `techStepClassifier` (jamais par requête).
|
||||
Le tout premier appel réel à `NlpManager.process()` déclenche aussi le
|
||||
chargement paresseux des ressources par langue de node-nlp (plusieurs
|
||||
secondes, mesuré) — `server.ts` appelle `techStepClassifier.warmUp()` avant
|
||||
d'accepter du trafic pour que ce ne soit jamais la première vraie requête qui
|
||||
attend.
|
||||
Résolution `TechStep.key -> id` mémoïsée une seule fois sur le singleton
|
||||
partagé `techStepClassifier` (jamais par requête) — c'est tout ce
|
||||
qu'`apps/api` a encore à mémoïser, l'entraînement du modèle lui-même vivant
|
||||
entièrement côté `services/tech-step-intent-service`. `server.ts` appelle
|
||||
`techStepClassifier.warmUp()` avant d'accepter du trafic, avec retry/backoff
|
||||
si `services/tech-step-intent-service` n'est pas encore joignable (le cas
|
||||
normal en Docker Compose, où `app` attend qu'il soit `healthy` avant même de
|
||||
démarrer — voir `docker-compose.yml`, et le README de ce service pour
|
||||
combien de temps ça prend).
|
||||
|
||||
**Deux pièges rencontrés en construisant ce pipeline**, tous deux corrigés
|
||||
dans le code (pas juste contournés) :
|
||||
**Pièges rencontrés en construisant ce pipeline**, tous corrigés dans le code
|
||||
(pas juste contournés) :
|
||||
- `db/prisma.ts` construisait `new PrismaClient()` sans jamais importer
|
||||
`config/env.ts` — dans le run de test complet, un *autre* fichier
|
||||
chargeait toujours `config/env.ts` (donc `.env.test`) en premier par pur
|
||||
|
|
@ -525,12 +540,20 @@ dans le code (pas juste contournés) :
|
|||
tant que `resetDatabase()` ne throw pas (heureusement son garde-fou le
|
||||
fait). Fixé en import `config/env.js` pour effet de bord tout en haut de
|
||||
`prisma.ts`, avant `new PrismaClient()`.
|
||||
- `NlpManager` a `autoSave`/`autoLoad: true` par défaut — persiste le
|
||||
modèle entraîné dans un fichier `model.nlp` (cwd du process) et le
|
||||
recharge *au lieu de* ré-entraîner au prochain démarrage s'il existe déjà.
|
||||
Un modèle obsolète sur disque masquerait silencieusement toute mise à
|
||||
jour de `TECH_STEP_TRAINING_DATA`/`CONFIDENCE_THRESHOLD`. Les deux sont
|
||||
explicitement à `false` dans le constructeur de `TechStepClassifierService`.
|
||||
- (historique, node-nlp) `NlpManager` avait `autoSave`/`autoLoad: true` par
|
||||
défaut — persistait le modèle entraîné dans un fichier `model.nlp` (cwd du
|
||||
process) et le rechargeait *au lieu de* ré-entraîner au prochain démarrage
|
||||
s'il existait déjà. Un modèle obsolète sur disque aurait masqué
|
||||
silencieusement toute mise à jour du corpus. Non applicable au service
|
||||
Python actuel : il réentraîne tout en mémoire à chaque démarrage du
|
||||
process, sans jamais rien persister sur disque (voir ce service's own
|
||||
README).
|
||||
- `nlp.make_doc()` (spaCy) ne fait tourner que le tokenizer, pas les
|
||||
composants du pipeline — un piège trouvé en construisant le `PhraseMatcher`
|
||||
du nouveau service : les patterns de synonymes doivent explicitement
|
||||
repasser par le composant de normalisation, sinon un synonyme accentué
|
||||
("préchauffer") ne matche jamais sa forme normalisée dans le texte cible
|
||||
(voir le commentaire dans `locale_pipeline.py`'s `train()`).
|
||||
|
||||
### Résolution ingrédients/unités — `ingredient-matcher.ts`
|
||||
|
||||
|
|
|
|||
|
|
@ -353,8 +353,9 @@ fiable.
|
|||
`tech_step` (`TechStep`, `key` unique, ex. `"simmer"`) est le catalogue des
|
||||
techniques (mijoter, préchauffer…) — juste un id/clé stable référencé par
|
||||
`step_tech_step`. Les données de détection elles-mêmes (synonymes + phrases
|
||||
d'exemple par langue, entraînant un classifieur `node-nlp`) vivent en code
|
||||
(`tech-step-training-data.ts`), pas dans une table — l'ancienne
|
||||
d'exemple par langue) vivent en code dans le microservice spaCy lui-même
|
||||
(`services/tech-step-intent-service/intent_service/training_data.py`), pas
|
||||
dans une table ni côté `apps/api` — l'ancienne
|
||||
`tech_step_mapping` (`TechStepMapping`, une regex par technique/locale) a
|
||||
été supprimée une fois constaté que les regex ne généralisaient jamais
|
||||
au-delà de leur propre vocabulaire — voir
|
||||
|
|
|
|||
Loading…
Reference in a new issue