revert(recipes): annule le reequilibrage du corpus d'entrainement du textcat
Trois strategies de generation differentes (tournures modales generiques, tournures reduites + substitution de synonyme, substitution de synonyme en priorite) ont ete tentees pour porter chaque technique a 20 utterances par locale. Les trois degradent mesurablement le F1 agrege contre TECH_STEP_EVAL_DATASET (tech-step-eval.test.ts) en dessous du seuil 0.8 : 0.7999 -> 0.791 -> 0.744 (chaque tentative pire que la precedente). tech-step-eval-runner.ts documente explicitement ce seuil comme calibre avec une marge deja tres etroite (0.8 pour un score mesure a 0.815) et previent contre le fait de l'assouplir pour accommoder un classifieur plus faible plutot que de corriger le probleme de fond - assouplir le seuil ou le jeu d'evaluation pour faire passer cette PR irait a l'encontre de cette convention documentee du projet. Revient a l'etat d'avant tout reequilibrage (corpus a 3-7 utterances/ technique, _TRAINING_ITERATIONS=25, timeouts a 900s) - le dernier etat confirme vert en CI sur cette branche. Ameliorer reellement l'equilibre du corpus necessite du contenu redige a la main et verifie technique par technique contre ce meme F1, pas une generation programmatique en bloc. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
e2ffa7d103
commit
74cd14c0a5
7 changed files with 30 additions and 2744 deletions
17
.github/workflows/ci.yml
vendored
17
.github/workflows/ci.yml
vendored
|
|
@ -96,15 +96,14 @@ jobs:
|
||||||
uv run uvicorn intent_service.main:app --host 0.0.0.0 --port 8000 &
|
uv run uvicorn intent_service.main:app --host 0.0.0.0 --port 8000 &
|
||||||
# `/health` only returns 200 once this service has finished
|
# `/health` only returns 200 once this service has finished
|
||||||
# training itself from scratch (no model ever persisted to disk —
|
# training itself from scratch (no model ever persisted to disk —
|
||||||
# see its own README) — measured at ~690s per locale (~1340s for
|
# see its own README) — measured at ~335s per locale (~670s for
|
||||||
# fr+en combined) against the current ~74-technique corpus, each
|
# fr+en combined) against the current ~74-technique corpus,
|
||||||
# now rebalanced toward 20 `utterances` in addition to its
|
# trained on each technique's own synonyms in addition to its
|
||||||
# `synonyms` (see `intent_service/locale_pipeline.py`'s
|
# example phrases, so this wait is generous rather than the fast
|
||||||
# `_TRAINING_ITERATIONS` for why it's `20`, not a smaller value
|
# "base models only" check it used to be before that service
|
||||||
# that trains faster but measurably fails this repo's own F1
|
# trained itself at startup (see docker-compose.yml's healthcheck
|
||||||
# quality gate — see docker-compose.yml's healthcheck for the
|
# for the same reasoning).
|
||||||
# same reasoning).
|
timeout 900 bash -c 'until curl -sf http://localhost:8000/health > /dev/null; do sleep 2; done'
|
||||||
timeout 1800 bash -c 'until curl -sf http://localhost:8000/health > /dev/null; do sleep 2; done'
|
|
||||||
|
|
||||||
- run: pnpm install --frozen-lockfile
|
- run: pnpm install --frozen-lockfile
|
||||||
- run: pnpm --filter api exec prisma migrate deploy
|
- run: pnpm --filter api exec prisma migrate deploy
|
||||||
|
|
|
||||||
|
|
@ -94,17 +94,15 @@ services:
|
||||||
# This service trains itself from scratch on every start (no model
|
# This service trains itself from scratch on every start (no model
|
||||||
# ever persisted to disk, see its own README) — `/health` only
|
# ever persisted to disk, see its own README) — `/health` only
|
||||||
# returns 200 once that's done, not just once the base spaCy models
|
# returns 200 once that's done, not just once the base spaCy models
|
||||||
# are loaded. Measured at ~690s per locale (~1340s for fr+en
|
# are loaded. Measured at ~335s per locale (~670s for fr+en combined)
|
||||||
# combined) against the current ~74-technique corpus, each now
|
# against the current ~74-technique corpus, trained on each
|
||||||
# rebalanced toward 20 `utterances` in addition to its `synonyms`
|
# technique's own synonyms in addition to its example phrases
|
||||||
# (`intent_service/locale_pipeline.py`'s `_TRAINING_ITERATIONS`,
|
# (`intent_service/locale_pipeline.py`'s `_TRAINING_ITERATIONS`) —
|
||||||
# raised from `10` after a smaller value measurably failed this
|
|
||||||
# repo's own F1 quality gate — see that constant's own comment) —
|
|
||||||
# `start_period` generous enough that failing checks during that
|
# `start_period` generous enough that failing checks during that
|
||||||
# whole window never count against `retries` (which would otherwise
|
# whole window never count against `retries` (which would otherwise
|
||||||
# flip this container to "unhealthy" mid-training, blocking `app`'s
|
# flip this container to "unhealthy" mid-training, blocking `app`'s
|
||||||
# own `depends_on: condition: service_healthy` indefinitely).
|
# own `depends_on: condition: service_healthy` indefinitely).
|
||||||
start_period: 1800s
|
start_period: 900s
|
||||||
|
|
||||||
# Deliberately its own image, not built into `app`'s (see
|
# Deliberately its own image, not built into `app`'s (see
|
||||||
# services/tech-step-llm-worker/Dockerfile's own doc comment) — a
|
# services/tech-step-llm-worker/Dockerfile's own doc comment) — a
|
||||||
|
|
|
||||||
|
|
@ -43,13 +43,7 @@ Workflow mainteneur pour changer le corpus :
|
||||||
rapport de `apps/api/src/scripts/list-pending-training-suggestions.ts`)
|
rapport de `apps/api/src/scripts/list-pending-training-suggestions.ts`)
|
||||||
pour une technique, ou `intent_service/utensil_vocabulary.py` pour un
|
pour une technique, ou `intent_service/utensil_vocabulary.py` pour un
|
||||||
ustensile (pas de rapport équivalent pour ce dernier — pas de mécanisme
|
ustensile (pas de rapport équivalent pour ce dernier — pas de mécanisme
|
||||||
de correction utilisateur sur les ustensiles aujourd'hui). Vise 20
|
de correction utilisateur sur les ustensiles aujourd'hui).
|
||||||
`utterances` par locale (voir `training_data.py`'s own doc comment) —
|
|
||||||
une technique ajoutée/éditée avec moins que ça, exécuter
|
|
||||||
`augment_utterances.py` (racine de ce service) pour la remettre à
|
|
||||||
niveau (`tests/test_training_data_balance.py` fait respecter un
|
|
||||||
plancher de 12 en CI — 20 est visé, pas garanti pour une technique au
|
|
||||||
vocabulaire propre trop pauvre, voir ce script's own doc comment).
|
|
||||||
2. **Redémarrer ce service** (`docker compose restart tech-step-intent-service`,
|
2. **Redémarrer ce service** (`docker compose restart tech-step-intent-service`,
|
||||||
ou simplement redéployer) — le nouveau corpus n'a d'effet qu'une fois
|
ou simplement redéployer) — le nouveau corpus n'a d'effet qu'une fois
|
||||||
réentraîné au démarrage, contrairement à l'ancienne version qui pouvait
|
réentraîné au démarrage, contrairement à l'ancienne version qui pouvait
|
||||||
|
|
@ -89,20 +83,19 @@ côté `apps/api`.
|
||||||
|
|
||||||
**Ce service met plusieurs minutes à devenir `healthy`** — contrairement à
|
**Ce service met plusieurs minutes à devenir `healthy`** — contrairement à
|
||||||
node-nlp (entraînement quasi instantané), entraîner le `textcat` sur le
|
node-nlp (entraînement quasi instantané), entraîner le `textcat` sur le
|
||||||
corpus réel (~74 techniques, chacune rééquilibrée vers 20 `utterances` en
|
corpus réel (~74 techniques, chaque technique entraînée sur ses `synonyms`
|
||||||
plus de ses `synonyms` — voir `training_data.py`/`locale_pipeline.py`)
|
en plus de ses `utterances` — voir `locale_pipeline.py`) prend de l'ordre
|
||||||
prend de l'ordre de 690 secondes par locale (mesuré localement, sans GPU),
|
de 335 secondes par locale (mesuré localement, sans GPU), donc environ 670
|
||||||
donc environ 1340 secondes (~22 minutes) pour `fr`+`en` combinés à chaque
|
secondes (~11 minutes) pour `fr`+`en` combinés à chaque démarrage du
|
||||||
démarrage du process. `docker-compose.yml` et
|
process. `docker-compose.yml` et
|
||||||
`.github/workflows/ci.yml` ont un `start_period`/timeout d'attente
|
`.github/workflows/ci.yml` ont un `start_period`/timeout d'attente
|
||||||
généreux pour ça (`1800s`) — voir leurs propres commentaires. C'est un
|
généreux pour ça — voir leurs propres commentaires. C'est un compromis
|
||||||
compromis assumé, pas un défaut de configuration à corriger : moins
|
assumé, pas un défaut de configuration à corriger : moins d'itérations
|
||||||
d'itérations entraîne plus vite mais laisse des verdicts corrects sous
|
entraîne plus vite mais laisse des verdicts corrects sous
|
||||||
`CONFIDENCE_THRESHOLD`, voire fait chuter le F1 agrégé sous le seuil de
|
`CONFIDENCE_THRESHOLD` (voir le commentaire de cette constante,
|
||||||
`test/recipe-matching/tech-step-eval.test.ts` (constaté concrètement en
|
`apps/api/src/lib/recipe-matching/tech-step-matcher.ts`, et celui de
|
||||||
CI — voir le commentaire de `_TRAINING_ITERATIONS`/`_TRAINING_BATCH_SIZE`
|
`_TRAINING_ITERATIONS`/`_TRAINING_BATCH_SIZE` dans `locale_pipeline.py`
|
||||||
dans `locale_pipeline.py` pour le détail de cette calibration, et celui de
|
pour le détail du compromis).
|
||||||
`CONFIDENCE_THRESHOLD`, `apps/api/src/lib/recipe-matching/tech-step-matcher.ts`).
|
|
||||||
|
|
||||||
## Logs
|
## Logs
|
||||||
|
|
||||||
|
|
@ -173,12 +166,10 @@ vraie instance de ce service tournant (voir `apps/api/.env.test`), conforme
|
||||||
|
|
||||||
## Limitations connues
|
## Limitations connues
|
||||||
|
|
||||||
- **Démarrage lent** (~22 minutes) — voir "Temps de démarrage" ci-dessus.
|
- **Démarrage lent** (~11 minutes) — voir "Temps de démarrage" ci-dessus.
|
||||||
Une optimisation possible non explorée : parallélisation de
|
Une optimisation possible non explorée : parallélisation de
|
||||||
l'entraînement `fr`/`en` (actuellement séquentiel,
|
l'entraînement `fr`/`en` (actuellement séquentiel,
|
||||||
`PipelineRegistry.initialize`) — diviserait potentiellement ce temps par
|
`PipelineRegistry.initialize`).
|
||||||
deux, contrairement à réduire `_TRAINING_ITERATIONS` qui dégrade
|
|
||||||
directement la qualité (voir cette constante's own comment).
|
|
||||||
- **`CONFIDENCE_THRESHOLD` côté `apps/api` est un placeholder** depuis
|
- **`CONFIDENCE_THRESHOLD` côté `apps/api` est un placeholder** depuis
|
||||||
l'élargissement du corpus à ~74 techniques (calibré à la main, pas via
|
l'élargissement du corpus à ~74 techniques (calibré à la main, pas via
|
||||||
une vraie repasse de `calibrate-tech-step-threshold.ts` contre
|
une vraie repasse de `calibrate-tech-step-threshold.ts` contre
|
||||||
|
|
|
||||||
|
|
@ -1,271 +0,0 @@
|
||||||
"""Maintainer script — tops up every technique's `utterances` (both locales)
|
|
||||||
to a minimum of 20 each, preserving all existing utterances/synonyms/comments
|
|
||||||
verbatim. Re-run this whenever a technique is added/edited with fewer than
|
|
||||||
20 `utterances` per locale — see `training_data.py`'s own module doc comment
|
|
||||||
for why 20 is the target (a textcat class starved of examples relative to
|
|
||||||
its siblings is a real source of confidently-wrong classifications, not
|
|
||||||
just a theoretical concern — this is what motivated the rebalance in the
|
|
||||||
first place).
|
|
||||||
|
|
||||||
**Generation strategy, in priority order** — this matters, see the
|
|
||||||
regression this script's own history records:
|
|
||||||
|
|
||||||
1. **Synonym substitution** (`_synonym_variants`) — for every existing
|
|
||||||
utterance whose leading phrase exactly matches one of the technique's
|
|
||||||
own `synonyms` (e.g. `melt`'s "faire fondre le beurre" starts with the
|
|
||||||
synonym "faire fondre"), swap in every *other* synonym from the same
|
|
||||||
list ("liquéfier le beurre", "faire chauffer le beurre", ...). This is
|
|
||||||
the primary source precisely because it injects genuinely
|
|
||||||
technique-*distinguishing* vocabulary (the corpus's own hand-picked
|
|
||||||
synonym list) rather than filler shared across every class.
|
|
||||||
2. **Modal-frame wrapping** (`_frame_variants`) — only used to fill
|
|
||||||
whatever's still missing after (1) is exhausted, and deliberately kept
|
|
||||||
to a *small* frame pool (3 per locale, not the dozen tried in an
|
|
||||||
earlier attempt at this script). A first version of this script relied
|
|
||||||
on frame-wrapping as the *primary* mechanism with 12/10 frames per
|
|
||||||
locale: it reached 20 utterances everywhere, but measurably **hurt**
|
|
||||||
`test/recipe-matching/tech-step-eval.test.ts`'s aggregate F1 (0.80 ->
|
|
||||||
0.79, confirmed twice in CI, once even after doubling
|
|
||||||
`_TRAINING_ITERATIONS`) — every one of the 74 classes ended up sharing
|
|
||||||
the same handful of high-frequency connector words ("il", "faut",
|
|
||||||
"de", "à", "veillez"...), which a bag-of-words classifier reads as
|
|
||||||
*reduced* inter-class separability, not neutral padding. Frame-wrapping
|
|
||||||
is grammatically safe but structurally low-value; kept only as a
|
|
||||||
fallback for techniques whose synonym list is too short to reach 20 on
|
|
||||||
its own (e.g. `julienne`, 4 synonyms).
|
|
||||||
|
|
||||||
Declarative/result-state utterances ("le beurre doit être liquide") are
|
|
||||||
never used as a source for either strategy (would be ungrammatical once
|
|
||||||
wrapped/substituted) — `is_fr_infinitive_led`/`is_en_imperative_led` decide
|
|
||||||
which existing utterances are safe sources for (2); (1) has its own,
|
|
||||||
stricter "starts with a known synonym" check that already excludes them.
|
|
||||||
|
|
||||||
Run from `services/tech-step-intent-service/` (this directory):
|
|
||||||
`./.venv/Scripts/python.exe augment_utterances.py` (Windows) or
|
|
||||||
`.venv/bin/python augment_utterances.py` (Linux/macOS) — needs the service's
|
|
||||||
own `uv sync`'d virtualenv, see this service's README. Rewrites
|
|
||||||
`training_data.py` in place by textual splicing (AST only to *locate* each
|
|
||||||
`utterances=[...]` list's line range — never to regenerate the file), so
|
|
||||||
every existing comment, `synonyms` list, and hand-written utterance survives
|
|
||||||
untouched. A technique already at/above 20 for a locale is left untouched —
|
|
||||||
re-running this script is always safe, never re-pads an already-balanced
|
|
||||||
entry (see `top_up`).
|
|
||||||
"""
|
|
||||||
|
|
||||||
import ast
|
|
||||||
import sys
|
|
||||||
|
|
||||||
SRC_PATH = "intent_service/training_data.py"
|
|
||||||
|
|
||||||
# Small fallback frame pool — see this module's own doc comment for why it's
|
|
||||||
# deliberately short (3 per locale, not a dozen) and only ever a fallback
|
|
||||||
# behind synonym substitution.
|
|
||||||
FR_FRAMES = [
|
|
||||||
"il faut {u}",
|
|
||||||
"veillez à {u}",
|
|
||||||
"pensez à {u}",
|
|
||||||
"n'oubliez pas de {u}",
|
|
||||||
"assurez-vous de {u}",
|
|
||||||
]
|
|
||||||
EN_FRAMES = [
|
|
||||||
"make sure to {u}",
|
|
||||||
"remember to {u}",
|
|
||||||
"be sure to {u}",
|
|
||||||
"don't forget to {u}",
|
|
||||||
"take care to {u}",
|
|
||||||
]
|
|
||||||
|
|
||||||
EN_VERB_WHITELIST = {
|
|
||||||
"make", "add", "pour", "mix", "stir", "cut", "place", "cover", "remove", "heat", "let",
|
|
||||||
"keep", "turn", "cook", "bake", "roast", "grill", "fry", "boil", "simmer", "whisk", "fold",
|
|
||||||
"chop", "mince", "peel", "drain", "season", "rest", "plate", "coat", "melt", "sauté", "saute",
|
|
||||||
"braise", "blanch", "marinate", "brown", "glaze", "thicken", "reduce", "dilute", "loosen",
|
|
||||||
"moisten", "sift", "toast", "zest", "scald", "pod", "shell", "hollow", "shock", "emulsify",
|
|
||||||
"decant", "dust", "sweat", "rub", "punch", "confit", "caramelize", "score", "line", "clarify",
|
|
||||||
"stew", "dice", "fillet", "proof", "poach", "pasteurize", "sterilize", "can", "preserve",
|
|
||||||
"tie", "truss", "baste", "spoon", "brush", "whip", "beat", "work", "sear", "flatten", "press",
|
|
||||||
"knead", "run", "cool", "warm", "combine", "blend", "arrange", "present", "sprinkle", "strain",
|
|
||||||
"separate", "bring", "grate", "continue", "deglaze", "scrape", "char", "break", "slice", "set",
|
|
||||||
"adjust", "switch", "sterilize", "secure", "mark", "butter", "crush", "julienne", "reheat",
|
|
||||||
"smother", "build", "scoop", "plunge", "increase", "pass", "collect", "have", "adjust",
|
|
||||||
"dry-toast", "dry-roast", "heat-treat", "pre-bake", "salt", "soak",
|
|
||||||
}
|
|
||||||
EN_ADVERB_SKIP = {
|
|
||||||
"coarsely", "roughly", "finely", "quickly", "lightly", "briefly", "gently", "carefully",
|
|
||||||
"gradually", "very", "thoroughly", "evenly", "generously", "slowly", "thinly", "deep", "blind",
|
|
||||||
"dry",
|
|
||||||
}
|
|
||||||
|
|
||||||
_TARGET = 20
|
|
||||||
|
|
||||||
|
|
||||||
def is_fr_infinitive_led(u: str) -> bool:
|
|
||||||
first = u.split(" ", 1)[0].lower()
|
|
||||||
return first.endswith(("er", "ir", "re")) and len(first) > 2
|
|
||||||
|
|
||||||
|
|
||||||
def is_en_imperative_led(u: str) -> bool:
|
|
||||||
words = u.lower().replace(",", "").split()
|
|
||||||
if not words:
|
|
||||||
return False
|
|
||||||
first = words[0]
|
|
||||||
if first in EN_VERB_WHITELIST:
|
|
||||||
return True
|
|
||||||
if first in EN_ADVERB_SKIP and len(words) > 1:
|
|
||||||
return words[1] in EN_VERB_WHITELIST
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def _synonym_variants(existing: list[str], synonyms: list[str]) -> list[str]:
|
|
||||||
"""Substitutes every *other* synonym in place of whichever synonym an
|
|
||||||
existing utterance's leading phrase exactly matches — see this module's
|
|
||||||
own doc comment for why this is the primary generation strategy."""
|
|
||||||
if len(synonyms) < 2:
|
|
||||||
return []
|
|
||||||
seen = set(existing)
|
|
||||||
sorted_synonyms = sorted(set(synonyms), key=len, reverse=True)
|
|
||||||
out: list[str] = []
|
|
||||||
for u in existing:
|
|
||||||
lower_u = u.lower()
|
|
||||||
matched = next(
|
|
||||||
(
|
|
||||||
syn
|
|
||||||
for syn in sorted_synonyms
|
|
||||||
if lower_u == syn.lower() or lower_u.startswith(f"{syn.lower()} ")
|
|
||||||
),
|
|
||||||
None,
|
|
||||||
)
|
|
||||||
if matched is None:
|
|
||||||
continue
|
|
||||||
rest = u[len(matched) :]
|
|
||||||
for syn in sorted_synonyms:
|
|
||||||
if syn == matched:
|
|
||||||
continue
|
|
||||||
candidate = f"{syn}{rest}"
|
|
||||||
if candidate in seen:
|
|
||||||
continue
|
|
||||||
seen.add(candidate)
|
|
||||||
out.append(candidate)
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def _frame_variants(existing: list[str], frames: list[str], is_led) -> list[str]:
|
|
||||||
sources = [u for u in existing if is_led(u)]
|
|
||||||
if not sources:
|
|
||||||
return []
|
|
||||||
seen = set(existing)
|
|
||||||
out: list[str] = []
|
|
||||||
for frame in frames:
|
|
||||||
for u in sources:
|
|
||||||
candidate = frame.format(u=u)
|
|
||||||
if candidate in seen:
|
|
||||||
continue
|
|
||||||
seen.add(candidate)
|
|
||||||
out.append(candidate)
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def top_up(existing: list[str], synonyms: list[str], locale: str) -> list[str]:
|
|
||||||
if len(existing) >= _TARGET:
|
|
||||||
return []
|
|
||||||
needed = _TARGET - len(existing)
|
|
||||||
pool = _synonym_variants(existing, synonyms)
|
|
||||||
if len(pool) < needed:
|
|
||||||
frames = FR_FRAMES if locale == "fr" else EN_FRAMES
|
|
||||||
is_led = is_fr_infinitive_led if locale == "fr" else is_en_imperative_led
|
|
||||||
# Frame variants must also dedupe against the synonym-substitution
|
|
||||||
# pool already chosen, not just `existing` — otherwise the two
|
|
||||||
# sources could independently produce the same string.
|
|
||||||
already = set(existing) | set(pool)
|
|
||||||
for candidate in _frame_variants(existing, frames, is_led):
|
|
||||||
if candidate in already:
|
|
||||||
continue
|
|
||||||
pool.append(candidate)
|
|
||||||
already.add(candidate)
|
|
||||||
return pool[:needed]
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
with open(SRC_PATH, encoding="utf-8") as f:
|
|
||||||
source = f.read()
|
|
||||||
tree = ast.parse(source)
|
|
||||||
lines = source.splitlines(keepends=True)
|
|
||||||
|
|
||||||
module_body = tree.body
|
|
||||||
training_data_list = None
|
|
||||||
for node in module_body:
|
|
||||||
if isinstance(node, ast.AnnAssign) and isinstance(node.target, ast.Name):
|
|
||||||
if node.target.id == "TECH_STEP_TRAINING_DATA":
|
|
||||||
training_data_list = node.value
|
|
||||||
break
|
|
||||||
if training_data_list is None or not isinstance(training_data_list, ast.List):
|
|
||||||
print("Could not locate TECH_STEP_TRAINING_DATA list", file=sys.stderr)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
insertions: list[tuple[int, str, list[str]]] = []
|
|
||||||
total_added = 0
|
|
||||||
shortfalls: list[tuple[str, str, int]] = []
|
|
||||||
|
|
||||||
for entry_call in training_data_list.elts:
|
|
||||||
assert isinstance(entry_call, ast.Call)
|
|
||||||
uid = None
|
|
||||||
for kw in entry_call.keywords:
|
|
||||||
if kw.arg == "uid":
|
|
||||||
assert isinstance(kw.value, ast.Constant)
|
|
||||||
uid = kw.value.value
|
|
||||||
for kw in entry_call.keywords:
|
|
||||||
if kw.arg not in ("fr", "en"):
|
|
||||||
continue
|
|
||||||
locale = kw.arg
|
|
||||||
locale_call = kw.value
|
|
||||||
assert isinstance(locale_call, ast.Call)
|
|
||||||
utterances_list_node = None
|
|
||||||
synonyms_list_node = None
|
|
||||||
for inner_kw in locale_call.keywords:
|
|
||||||
if inner_kw.arg == "utterances":
|
|
||||||
utterances_list_node = inner_kw.value
|
|
||||||
elif inner_kw.arg == "synonyms":
|
|
||||||
synonyms_list_node = inner_kw.value
|
|
||||||
if utterances_list_node is None:
|
|
||||||
continue
|
|
||||||
assert isinstance(utterances_list_node, ast.List)
|
|
||||||
existing = [
|
|
||||||
elt.value for elt in utterances_list_node.elts if isinstance(elt, ast.Constant)
|
|
||||||
]
|
|
||||||
synonyms = (
|
|
||||||
[elt.value for elt in synonyms_list_node.elts if isinstance(elt, ast.Constant)]
|
|
||||||
if isinstance(synonyms_list_node, ast.List)
|
|
||||||
else []
|
|
||||||
)
|
|
||||||
new_ones = top_up(existing, synonyms, locale)
|
|
||||||
final_count = len(existing) + len(new_ones)
|
|
||||||
if final_count < _TARGET:
|
|
||||||
shortfalls.append((uid, locale, final_count))
|
|
||||||
if not new_ones:
|
|
||||||
continue
|
|
||||||
last_elt = utterances_list_node.elts[-1]
|
|
||||||
insert_after_line = last_elt.end_lineno - 1
|
|
||||||
indent = lines[insert_after_line][
|
|
||||||
: len(lines[insert_after_line]) - len(lines[insert_after_line].lstrip())
|
|
||||||
]
|
|
||||||
new_lines = [f'{indent}"{s}",\n' for s in new_ones]
|
|
||||||
insertions.append((insert_after_line, uid, new_lines))
|
|
||||||
total_added += len(new_ones)
|
|
||||||
|
|
||||||
insertions.sort(key=lambda t: t[0], reverse=True)
|
|
||||||
for line_idx, uid, new_lines in insertions:
|
|
||||||
lines[line_idx + 1 : line_idx + 1] = new_lines
|
|
||||||
|
|
||||||
with open(SRC_PATH, "w", encoding="utf-8", newline="\n") as f:
|
|
||||||
f.writelines(lines)
|
|
||||||
|
|
||||||
print(f"Added {total_added} new utterances across {len(insertions)} (technique, locale) pairs.")
|
|
||||||
if shortfalls:
|
|
||||||
print(f"{len(shortfalls)} (uid, locale) pair(s) still below {_TARGET} — not enough synonym")
|
|
||||||
print("variety to reach the target without falling back to more generic frames:")
|
|
||||||
for uid, locale, count in shortfalls:
|
|
||||||
print(f" {uid} ({locale}): {count}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
|
|
@ -60,7 +60,7 @@ _TEXTCAT_PIPE_NAME = "textcat"
|
||||||
# mais avec une confiance dérisoire — bien en dessous de tout seuil
|
# mais avec une confiance dérisoire — bien en dessous de tout seuil
|
||||||
# raisonnable pour `CONFIDENCE_THRESHOLD` (`tech-step-matcher.ts`).
|
# raisonnable pour `CONFIDENCE_THRESHOLD` (`tech-step-matcher.ts`).
|
||||||
#
|
#
|
||||||
# Quatre passes de calibration successives, toutes mesurées contre le
|
# Trois passes de calibration successives, toutes mesurées contre le
|
||||||
# corpus réel (74 techniques) :
|
# corpus réel (74 techniques) :
|
||||||
# 1. `150` itérations (calibré pour le corpus original, ~26 techniques) ne
|
# 1. `150` itérations (calibré pour le corpus original, ~26 techniques) ne
|
||||||
# passe plus à l'échelle une fois élargi : `150` sur 74 classes
|
# passe plus à l'échelle une fois élargi : `150` sur 74 classes
|
||||||
|
|
@ -87,29 +87,7 @@ _TEXTCAT_PIPE_NAME = "textcat"
|
||||||
# `CONFIDENCE_THRESHOLD`'s propre commentaire, `tech-step-matcher.ts`)
|
# `CONFIDENCE_THRESHOLD`'s propre commentaire, `tech-step-matcher.ts`)
|
||||||
# — ce qui précède est une mesure manuelle ponctuelle, pas un
|
# — ce qui précède est une mesure manuelle ponctuelle, pas un
|
||||||
# remplacement de cette calibration.
|
# remplacement de cette calibration.
|
||||||
# 4. Le corpus a ensuite été rééquilibré vers 20 `utterances` par technique
|
_TRAINING_ITERATIONS = 25
|
||||||
# et par locale (voir `training_data.py`'s propre commentaire de tête
|
|
||||||
# pour l'algorithme exact et pourquoi certaines techniques restent
|
|
||||||
# volontairement en dessous de 20). Deux tentatives ont mesurablement
|
|
||||||
# échoué avant celle-ci : `_TRAINING_ITERATIONS` inchangé (25) avec un
|
|
||||||
# générateur reposant principalement sur des tournures modales
|
|
||||||
# génériques a fait chuter le F1 agrégé
|
|
||||||
# (`test/recipe-matching/tech-step-eval.test.ts`) à `0.79`, sous le
|
|
||||||
# seuil `0.8` ; doubler `_TRAINING_ITERATIONS` (`10` -> `20`) sur ce
|
|
||||||
# même corpus n'a pas aidé (`0.791`, toujours sous le seuil) — la cause
|
|
||||||
# n'était pas un manque d'itérations mais un générateur qui diluait la
|
|
||||||
# séparabilité entre classes (voir `training_data.py`). Le générateur a
|
|
||||||
# donc été revu (substitution de synonyme en priorité, tournures
|
|
||||||
# modales seulement en complément limité) et `_TRAINING_ITERATIONS`
|
|
||||||
# remonté à `20` sur ce nouveau corpus — mesuré : ~691s (fr, 1923
|
|
||||||
# exemples) / ~645s (en, 1807 exemples), ~1336s pour fr+en combinés.
|
|
||||||
# Confiance nettement rétablie sur les techniques auparavant en échec
|
|
||||||
# (`sweat` ~0.99, `bainMarie` ~0.98, `julienne` ~0.88). À
|
|
||||||
# confirmer/affiner par une vraie repasse de
|
|
||||||
# `calibrate-tech-step-threshold.ts` comme aux étapes précédentes — ce
|
|
||||||
# qui précède reste une mesure ponctuelle plus la gate F1 de CI, pas un
|
|
||||||
# remplacement de cette calibration.
|
|
||||||
_TRAINING_ITERATIONS = 20
|
|
||||||
_TRAINING_BATCH_SIZE = 16
|
_TRAINING_BATCH_SIZE = 16
|
||||||
# Arrêt anticipé : `_TRAINING_ITERATIONS` reste le plafond (le pire cas ne
|
# Arrêt anticipé : `_TRAINING_ITERATIONS` reste le plafond (le pire cas ne
|
||||||
# change pas), un corpus/locale qui converge plus vite n'a pas à payer les
|
# change pas), un corpus/locale qui converge plus vite n'a pas à payer les
|
||||||
|
|
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,34 +0,0 @@
|
||||||
"""Garde-fou de non-régression pour l'équilibrage du corpus (voir
|
|
||||||
`training_data.py`'s propre commentaire de tête) : un textcat entraîné sur
|
|
||||||
des classes très inégales en nombre d'exemples est une source réelle de
|
|
||||||
classifications confiantes mais fausses sur une phrase jamais vue (constaté
|
|
||||||
en pratique — voir l'historique Git de ce fichier).
|
|
||||||
|
|
||||||
`_MIN_UTTERANCES_PER_LOCALE` est volontairement `12`, pas `20` : la
|
|
||||||
génération vise `20` (`augment_utterances.py`'s `_TARGET`) mais s'arrête
|
|
||||||
avant si la technique n'a pas assez de vocabulaire distinctif propre
|
|
||||||
(`synonyms`) pour l'atteindre sans retomber massivement sur des tournures
|
|
||||||
génériques partagées par toutes les classes — une première version de ce
|
|
||||||
script forçait `20` partout via ce mécanisme et a mesurablement *dégradé*
|
|
||||||
`test/recipe-matching/tech-step-eval.test.ts` (F1 agrégé) plutôt que de
|
|
||||||
l'améliorer, en réduisant la séparabilité entre classes plus qu'en ajoutant
|
|
||||||
un vrai signal. `12` reste très au-dessus du plancher d'origine (3-7) tout
|
|
||||||
en laissant `augment_utterances.py` s'arrêter honnêtement plutôt que de
|
|
||||||
forcer un compte rond au prix de la qualité."""
|
|
||||||
|
|
||||||
from intent_service.training_data import TECH_STEP_TRAINING_DATA
|
|
||||||
|
|
||||||
_MIN_UTTERANCES_PER_LOCALE = 12
|
|
||||||
|
|
||||||
|
|
||||||
def test_every_technique_has_at_least_the_minimum_utterances_per_locale():
|
|
||||||
short = [
|
|
||||||
(entry.uid, locale, len(getattr(entry, locale).utterances))
|
|
||||||
for entry in TECH_STEP_TRAINING_DATA
|
|
||||||
for locale in ("fr", "en")
|
|
||||||
if len(getattr(entry, locale).utterances) < _MIN_UTTERANCES_PER_LOCALE
|
|
||||||
]
|
|
||||||
assert short == [], (
|
|
||||||
f"{len(short)} (uid, locale) pair(s) below the {_MIN_UTTERANCES_PER_LOCALE}-utterance "
|
|
||||||
f"floor: {short}"
|
|
||||||
)
|
|
||||||
Loading…
Reference in a new issue