revert(recipes): annule le reequilibrage du corpus d'entrainement du textcat
Trois strategies de generation differentes (tournures modales generiques, tournures reduites + substitution de synonyme, substitution de synonyme en priorite) ont ete tentees pour porter chaque technique a 20 utterances par locale. Les trois degradent mesurablement le F1 agrege contre TECH_STEP_EVAL_DATASET (tech-step-eval.test.ts) en dessous du seuil 0.8 : 0.7999 -> 0.791 -> 0.744 (chaque tentative pire que la precedente). tech-step-eval-runner.ts documente explicitement ce seuil comme calibre avec une marge deja tres etroite (0.8 pour un score mesure a 0.815) et previent contre le fait de l'assouplir pour accommoder un classifieur plus faible plutot que de corriger le probleme de fond - assouplir le seuil ou le jeu d'evaluation pour faire passer cette PR irait a l'encontre de cette convention documentee du projet. Revient a l'etat d'avant tout reequilibrage (corpus a 3-7 utterances/ technique, _TRAINING_ITERATIONS=25, timeouts a 900s) - le dernier etat confirme vert en CI sur cette branche. Ameliorer reellement l'equilibre du corpus necessite du contenu redige a la main et verifie technique par technique contre ce meme F1, pas une generation programmatique en bloc. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
e2ffa7d103
commit
74cd14c0a5
7 changed files with 30 additions and 2744 deletions
17
.github/workflows/ci.yml
vendored
17
.github/workflows/ci.yml
vendored
|
|
@ -96,15 +96,14 @@ jobs:
|
|||
uv run uvicorn intent_service.main:app --host 0.0.0.0 --port 8000 &
|
||||
# `/health` only returns 200 once this service has finished
|
||||
# training itself from scratch (no model ever persisted to disk —
|
||||
# see its own README) — measured at ~690s per locale (~1340s for
|
||||
# fr+en combined) against the current ~74-technique corpus, each
|
||||
# now rebalanced toward 20 `utterances` in addition to its
|
||||
# `synonyms` (see `intent_service/locale_pipeline.py`'s
|
||||
# `_TRAINING_ITERATIONS` for why it's `20`, not a smaller value
|
||||
# that trains faster but measurably fails this repo's own F1
|
||||
# quality gate — see docker-compose.yml's healthcheck for the
|
||||
# same reasoning).
|
||||
timeout 1800 bash -c 'until curl -sf http://localhost:8000/health > /dev/null; do sleep 2; done'
|
||||
# see its own README) — measured at ~335s per locale (~670s for
|
||||
# fr+en combined) against the current ~74-technique corpus,
|
||||
# trained on each technique's own synonyms in addition to its
|
||||
# example phrases, so this wait is generous rather than the fast
|
||||
# "base models only" check it used to be before that service
|
||||
# trained itself at startup (see docker-compose.yml's healthcheck
|
||||
# for the same reasoning).
|
||||
timeout 900 bash -c 'until curl -sf http://localhost:8000/health > /dev/null; do sleep 2; done'
|
||||
|
||||
- run: pnpm install --frozen-lockfile
|
||||
- run: pnpm --filter api exec prisma migrate deploy
|
||||
|
|
|
|||
|
|
@ -94,17 +94,15 @@ services:
|
|||
# This service trains itself from scratch on every start (no model
|
||||
# ever persisted to disk, see its own README) — `/health` only
|
||||
# returns 200 once that's done, not just once the base spaCy models
|
||||
# are loaded. Measured at ~690s per locale (~1340s for fr+en
|
||||
# combined) against the current ~74-technique corpus, each now
|
||||
# rebalanced toward 20 `utterances` in addition to its `synonyms`
|
||||
# (`intent_service/locale_pipeline.py`'s `_TRAINING_ITERATIONS`,
|
||||
# raised from `10` after a smaller value measurably failed this
|
||||
# repo's own F1 quality gate — see that constant's own comment) —
|
||||
# are loaded. Measured at ~335s per locale (~670s for fr+en combined)
|
||||
# against the current ~74-technique corpus, trained on each
|
||||
# technique's own synonyms in addition to its example phrases
|
||||
# (`intent_service/locale_pipeline.py`'s `_TRAINING_ITERATIONS`) —
|
||||
# `start_period` generous enough that failing checks during that
|
||||
# whole window never count against `retries` (which would otherwise
|
||||
# flip this container to "unhealthy" mid-training, blocking `app`'s
|
||||
# own `depends_on: condition: service_healthy` indefinitely).
|
||||
start_period: 1800s
|
||||
start_period: 900s
|
||||
|
||||
# Deliberately its own image, not built into `app`'s (see
|
||||
# services/tech-step-llm-worker/Dockerfile's own doc comment) — a
|
||||
|
|
|
|||
|
|
@ -43,13 +43,7 @@ Workflow mainteneur pour changer le corpus :
|
|||
rapport de `apps/api/src/scripts/list-pending-training-suggestions.ts`)
|
||||
pour une technique, ou `intent_service/utensil_vocabulary.py` pour un
|
||||
ustensile (pas de rapport équivalent pour ce dernier — pas de mécanisme
|
||||
de correction utilisateur sur les ustensiles aujourd'hui). Vise 20
|
||||
`utterances` par locale (voir `training_data.py`'s own doc comment) —
|
||||
une technique ajoutée/éditée avec moins que ça, exécuter
|
||||
`augment_utterances.py` (racine de ce service) pour la remettre à
|
||||
niveau (`tests/test_training_data_balance.py` fait respecter un
|
||||
plancher de 12 en CI — 20 est visé, pas garanti pour une technique au
|
||||
vocabulaire propre trop pauvre, voir ce script's own doc comment).
|
||||
de correction utilisateur sur les ustensiles aujourd'hui).
|
||||
2. **Redémarrer ce service** (`docker compose restart tech-step-intent-service`,
|
||||
ou simplement redéployer) — le nouveau corpus n'a d'effet qu'une fois
|
||||
réentraîné au démarrage, contrairement à l'ancienne version qui pouvait
|
||||
|
|
@ -89,20 +83,19 @@ côté `apps/api`.
|
|||
|
||||
**Ce service met plusieurs minutes à devenir `healthy`** — contrairement à
|
||||
node-nlp (entraînement quasi instantané), entraîner le `textcat` sur le
|
||||
corpus réel (~74 techniques, chacune rééquilibrée vers 20 `utterances` en
|
||||
plus de ses `synonyms` — voir `training_data.py`/`locale_pipeline.py`)
|
||||
prend de l'ordre de 690 secondes par locale (mesuré localement, sans GPU),
|
||||
donc environ 1340 secondes (~22 minutes) pour `fr`+`en` combinés à chaque
|
||||
démarrage du process. `docker-compose.yml` et
|
||||
corpus réel (~74 techniques, chaque technique entraînée sur ses `synonyms`
|
||||
en plus de ses `utterances` — voir `locale_pipeline.py`) prend de l'ordre
|
||||
de 335 secondes par locale (mesuré localement, sans GPU), donc environ 670
|
||||
secondes (~11 minutes) pour `fr`+`en` combinés à chaque démarrage du
|
||||
process. `docker-compose.yml` et
|
||||
`.github/workflows/ci.yml` ont un `start_period`/timeout d'attente
|
||||
généreux pour ça (`1800s`) — voir leurs propres commentaires. C'est un
|
||||
compromis assumé, pas un défaut de configuration à corriger : moins
|
||||
d'itérations entraîne plus vite mais laisse des verdicts corrects sous
|
||||
`CONFIDENCE_THRESHOLD`, voire fait chuter le F1 agrégé sous le seuil de
|
||||
`test/recipe-matching/tech-step-eval.test.ts` (constaté concrètement en
|
||||
CI — voir le commentaire de `_TRAINING_ITERATIONS`/`_TRAINING_BATCH_SIZE`
|
||||
dans `locale_pipeline.py` pour le détail de cette calibration, et celui de
|
||||
`CONFIDENCE_THRESHOLD`, `apps/api/src/lib/recipe-matching/tech-step-matcher.ts`).
|
||||
généreux pour ça — voir leurs propres commentaires. C'est un compromis
|
||||
assumé, pas un défaut de configuration à corriger : moins d'itérations
|
||||
entraîne plus vite mais laisse des verdicts corrects sous
|
||||
`CONFIDENCE_THRESHOLD` (voir le commentaire de cette constante,
|
||||
`apps/api/src/lib/recipe-matching/tech-step-matcher.ts`, et celui de
|
||||
`_TRAINING_ITERATIONS`/`_TRAINING_BATCH_SIZE` dans `locale_pipeline.py`
|
||||
pour le détail du compromis).
|
||||
|
||||
## Logs
|
||||
|
||||
|
|
@ -173,12 +166,10 @@ vraie instance de ce service tournant (voir `apps/api/.env.test`), conforme
|
|||
|
||||
## Limitations connues
|
||||
|
||||
- **Démarrage lent** (~22 minutes) — voir "Temps de démarrage" ci-dessus.
|
||||
- **Démarrage lent** (~11 minutes) — voir "Temps de démarrage" ci-dessus.
|
||||
Une optimisation possible non explorée : parallélisation de
|
||||
l'entraînement `fr`/`en` (actuellement séquentiel,
|
||||
`PipelineRegistry.initialize`) — diviserait potentiellement ce temps par
|
||||
deux, contrairement à réduire `_TRAINING_ITERATIONS` qui dégrade
|
||||
directement la qualité (voir cette constante's own comment).
|
||||
`PipelineRegistry.initialize`).
|
||||
- **`CONFIDENCE_THRESHOLD` côté `apps/api` est un placeholder** depuis
|
||||
l'élargissement du corpus à ~74 techniques (calibré à la main, pas via
|
||||
une vraie repasse de `calibrate-tech-step-threshold.ts` contre
|
||||
|
|
|
|||
|
|
@ -1,271 +0,0 @@
|
|||
"""Maintainer script — tops up every technique's `utterances` (both locales)
|
||||
to a minimum of 20 each, preserving all existing utterances/synonyms/comments
|
||||
verbatim. Re-run this whenever a technique is added/edited with fewer than
|
||||
20 `utterances` per locale — see `training_data.py`'s own module doc comment
|
||||
for why 20 is the target (a textcat class starved of examples relative to
|
||||
its siblings is a real source of confidently-wrong classifications, not
|
||||
just a theoretical concern — this is what motivated the rebalance in the
|
||||
first place).
|
||||
|
||||
**Generation strategy, in priority order** — this matters, see the
|
||||
regression this script's own history records:
|
||||
|
||||
1. **Synonym substitution** (`_synonym_variants`) — for every existing
|
||||
utterance whose leading phrase exactly matches one of the technique's
|
||||
own `synonyms` (e.g. `melt`'s "faire fondre le beurre" starts with the
|
||||
synonym "faire fondre"), swap in every *other* synonym from the same
|
||||
list ("liquéfier le beurre", "faire chauffer le beurre", ...). This is
|
||||
the primary source precisely because it injects genuinely
|
||||
technique-*distinguishing* vocabulary (the corpus's own hand-picked
|
||||
synonym list) rather than filler shared across every class.
|
||||
2. **Modal-frame wrapping** (`_frame_variants`) — only used to fill
|
||||
whatever's still missing after (1) is exhausted, and deliberately kept
|
||||
to a *small* frame pool (3 per locale, not the dozen tried in an
|
||||
earlier attempt at this script). A first version of this script relied
|
||||
on frame-wrapping as the *primary* mechanism with 12/10 frames per
|
||||
locale: it reached 20 utterances everywhere, but measurably **hurt**
|
||||
`test/recipe-matching/tech-step-eval.test.ts`'s aggregate F1 (0.80 ->
|
||||
0.79, confirmed twice in CI, once even after doubling
|
||||
`_TRAINING_ITERATIONS`) — every one of the 74 classes ended up sharing
|
||||
the same handful of high-frequency connector words ("il", "faut",
|
||||
"de", "à", "veillez"...), which a bag-of-words classifier reads as
|
||||
*reduced* inter-class separability, not neutral padding. Frame-wrapping
|
||||
is grammatically safe but structurally low-value; kept only as a
|
||||
fallback for techniques whose synonym list is too short to reach 20 on
|
||||
its own (e.g. `julienne`, 4 synonyms).
|
||||
|
||||
Declarative/result-state utterances ("le beurre doit être liquide") are
|
||||
never used as a source for either strategy (would be ungrammatical once
|
||||
wrapped/substituted) — `is_fr_infinitive_led`/`is_en_imperative_led` decide
|
||||
which existing utterances are safe sources for (2); (1) has its own,
|
||||
stricter "starts with a known synonym" check that already excludes them.
|
||||
|
||||
Run from `services/tech-step-intent-service/` (this directory):
|
||||
`./.venv/Scripts/python.exe augment_utterances.py` (Windows) or
|
||||
`.venv/bin/python augment_utterances.py` (Linux/macOS) — needs the service's
|
||||
own `uv sync`'d virtualenv, see this service's README. Rewrites
|
||||
`training_data.py` in place by textual splicing (AST only to *locate* each
|
||||
`utterances=[...]` list's line range — never to regenerate the file), so
|
||||
every existing comment, `synonyms` list, and hand-written utterance survives
|
||||
untouched. A technique already at/above 20 for a locale is left untouched —
|
||||
re-running this script is always safe, never re-pads an already-balanced
|
||||
entry (see `top_up`).
|
||||
"""
|
||||
|
||||
import ast
|
||||
import sys
|
||||
|
||||
SRC_PATH = "intent_service/training_data.py"
|
||||
|
||||
# Small fallback frame pool — see this module's own doc comment for why it's
|
||||
# deliberately short (3 per locale, not a dozen) and only ever a fallback
|
||||
# behind synonym substitution.
|
||||
FR_FRAMES = [
|
||||
"il faut {u}",
|
||||
"veillez à {u}",
|
||||
"pensez à {u}",
|
||||
"n'oubliez pas de {u}",
|
||||
"assurez-vous de {u}",
|
||||
]
|
||||
EN_FRAMES = [
|
||||
"make sure to {u}",
|
||||
"remember to {u}",
|
||||
"be sure to {u}",
|
||||
"don't forget to {u}",
|
||||
"take care to {u}",
|
||||
]
|
||||
|
||||
EN_VERB_WHITELIST = {
|
||||
"make", "add", "pour", "mix", "stir", "cut", "place", "cover", "remove", "heat", "let",
|
||||
"keep", "turn", "cook", "bake", "roast", "grill", "fry", "boil", "simmer", "whisk", "fold",
|
||||
"chop", "mince", "peel", "drain", "season", "rest", "plate", "coat", "melt", "sauté", "saute",
|
||||
"braise", "blanch", "marinate", "brown", "glaze", "thicken", "reduce", "dilute", "loosen",
|
||||
"moisten", "sift", "toast", "zest", "scald", "pod", "shell", "hollow", "shock", "emulsify",
|
||||
"decant", "dust", "sweat", "rub", "punch", "confit", "caramelize", "score", "line", "clarify",
|
||||
"stew", "dice", "fillet", "proof", "poach", "pasteurize", "sterilize", "can", "preserve",
|
||||
"tie", "truss", "baste", "spoon", "brush", "whip", "beat", "work", "sear", "flatten", "press",
|
||||
"knead", "run", "cool", "warm", "combine", "blend", "arrange", "present", "sprinkle", "strain",
|
||||
"separate", "bring", "grate", "continue", "deglaze", "scrape", "char", "break", "slice", "set",
|
||||
"adjust", "switch", "sterilize", "secure", "mark", "butter", "crush", "julienne", "reheat",
|
||||
"smother", "build", "scoop", "plunge", "increase", "pass", "collect", "have", "adjust",
|
||||
"dry-toast", "dry-roast", "heat-treat", "pre-bake", "salt", "soak",
|
||||
}
|
||||
EN_ADVERB_SKIP = {
|
||||
"coarsely", "roughly", "finely", "quickly", "lightly", "briefly", "gently", "carefully",
|
||||
"gradually", "very", "thoroughly", "evenly", "generously", "slowly", "thinly", "deep", "blind",
|
||||
"dry",
|
||||
}
|
||||
|
||||
_TARGET = 20
|
||||
|
||||
|
||||
def is_fr_infinitive_led(u: str) -> bool:
|
||||
first = u.split(" ", 1)[0].lower()
|
||||
return first.endswith(("er", "ir", "re")) and len(first) > 2
|
||||
|
||||
|
||||
def is_en_imperative_led(u: str) -> bool:
|
||||
words = u.lower().replace(",", "").split()
|
||||
if not words:
|
||||
return False
|
||||
first = words[0]
|
||||
if first in EN_VERB_WHITELIST:
|
||||
return True
|
||||
if first in EN_ADVERB_SKIP and len(words) > 1:
|
||||
return words[1] in EN_VERB_WHITELIST
|
||||
return False
|
||||
|
||||
|
||||
def _synonym_variants(existing: list[str], synonyms: list[str]) -> list[str]:
|
||||
"""Substitutes every *other* synonym in place of whichever synonym an
|
||||
existing utterance's leading phrase exactly matches — see this module's
|
||||
own doc comment for why this is the primary generation strategy."""
|
||||
if len(synonyms) < 2:
|
||||
return []
|
||||
seen = set(existing)
|
||||
sorted_synonyms = sorted(set(synonyms), key=len, reverse=True)
|
||||
out: list[str] = []
|
||||
for u in existing:
|
||||
lower_u = u.lower()
|
||||
matched = next(
|
||||
(
|
||||
syn
|
||||
for syn in sorted_synonyms
|
||||
if lower_u == syn.lower() or lower_u.startswith(f"{syn.lower()} ")
|
||||
),
|
||||
None,
|
||||
)
|
||||
if matched is None:
|
||||
continue
|
||||
rest = u[len(matched) :]
|
||||
for syn in sorted_synonyms:
|
||||
if syn == matched:
|
||||
continue
|
||||
candidate = f"{syn}{rest}"
|
||||
if candidate in seen:
|
||||
continue
|
||||
seen.add(candidate)
|
||||
out.append(candidate)
|
||||
return out
|
||||
|
||||
|
||||
def _frame_variants(existing: list[str], frames: list[str], is_led) -> list[str]:
|
||||
sources = [u for u in existing if is_led(u)]
|
||||
if not sources:
|
||||
return []
|
||||
seen = set(existing)
|
||||
out: list[str] = []
|
||||
for frame in frames:
|
||||
for u in sources:
|
||||
candidate = frame.format(u=u)
|
||||
if candidate in seen:
|
||||
continue
|
||||
seen.add(candidate)
|
||||
out.append(candidate)
|
||||
return out
|
||||
|
||||
|
||||
def top_up(existing: list[str], synonyms: list[str], locale: str) -> list[str]:
|
||||
if len(existing) >= _TARGET:
|
||||
return []
|
||||
needed = _TARGET - len(existing)
|
||||
pool = _synonym_variants(existing, synonyms)
|
||||
if len(pool) < needed:
|
||||
frames = FR_FRAMES if locale == "fr" else EN_FRAMES
|
||||
is_led = is_fr_infinitive_led if locale == "fr" else is_en_imperative_led
|
||||
# Frame variants must also dedupe against the synonym-substitution
|
||||
# pool already chosen, not just `existing` — otherwise the two
|
||||
# sources could independently produce the same string.
|
||||
already = set(existing) | set(pool)
|
||||
for candidate in _frame_variants(existing, frames, is_led):
|
||||
if candidate in already:
|
||||
continue
|
||||
pool.append(candidate)
|
||||
already.add(candidate)
|
||||
return pool[:needed]
|
||||
|
||||
|
||||
def main() -> None:
|
||||
with open(SRC_PATH, encoding="utf-8") as f:
|
||||
source = f.read()
|
||||
tree = ast.parse(source)
|
||||
lines = source.splitlines(keepends=True)
|
||||
|
||||
module_body = tree.body
|
||||
training_data_list = None
|
||||
for node in module_body:
|
||||
if isinstance(node, ast.AnnAssign) and isinstance(node.target, ast.Name):
|
||||
if node.target.id == "TECH_STEP_TRAINING_DATA":
|
||||
training_data_list = node.value
|
||||
break
|
||||
if training_data_list is None or not isinstance(training_data_list, ast.List):
|
||||
print("Could not locate TECH_STEP_TRAINING_DATA list", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
insertions: list[tuple[int, str, list[str]]] = []
|
||||
total_added = 0
|
||||
shortfalls: list[tuple[str, str, int]] = []
|
||||
|
||||
for entry_call in training_data_list.elts:
|
||||
assert isinstance(entry_call, ast.Call)
|
||||
uid = None
|
||||
for kw in entry_call.keywords:
|
||||
if kw.arg == "uid":
|
||||
assert isinstance(kw.value, ast.Constant)
|
||||
uid = kw.value.value
|
||||
for kw in entry_call.keywords:
|
||||
if kw.arg not in ("fr", "en"):
|
||||
continue
|
||||
locale = kw.arg
|
||||
locale_call = kw.value
|
||||
assert isinstance(locale_call, ast.Call)
|
||||
utterances_list_node = None
|
||||
synonyms_list_node = None
|
||||
for inner_kw in locale_call.keywords:
|
||||
if inner_kw.arg == "utterances":
|
||||
utterances_list_node = inner_kw.value
|
||||
elif inner_kw.arg == "synonyms":
|
||||
synonyms_list_node = inner_kw.value
|
||||
if utterances_list_node is None:
|
||||
continue
|
||||
assert isinstance(utterances_list_node, ast.List)
|
||||
existing = [
|
||||
elt.value for elt in utterances_list_node.elts if isinstance(elt, ast.Constant)
|
||||
]
|
||||
synonyms = (
|
||||
[elt.value for elt in synonyms_list_node.elts if isinstance(elt, ast.Constant)]
|
||||
if isinstance(synonyms_list_node, ast.List)
|
||||
else []
|
||||
)
|
||||
new_ones = top_up(existing, synonyms, locale)
|
||||
final_count = len(existing) + len(new_ones)
|
||||
if final_count < _TARGET:
|
||||
shortfalls.append((uid, locale, final_count))
|
||||
if not new_ones:
|
||||
continue
|
||||
last_elt = utterances_list_node.elts[-1]
|
||||
insert_after_line = last_elt.end_lineno - 1
|
||||
indent = lines[insert_after_line][
|
||||
: len(lines[insert_after_line]) - len(lines[insert_after_line].lstrip())
|
||||
]
|
||||
new_lines = [f'{indent}"{s}",\n' for s in new_ones]
|
||||
insertions.append((insert_after_line, uid, new_lines))
|
||||
total_added += len(new_ones)
|
||||
|
||||
insertions.sort(key=lambda t: t[0], reverse=True)
|
||||
for line_idx, uid, new_lines in insertions:
|
||||
lines[line_idx + 1 : line_idx + 1] = new_lines
|
||||
|
||||
with open(SRC_PATH, "w", encoding="utf-8", newline="\n") as f:
|
||||
f.writelines(lines)
|
||||
|
||||
print(f"Added {total_added} new utterances across {len(insertions)} (technique, locale) pairs.")
|
||||
if shortfalls:
|
||||
print(f"{len(shortfalls)} (uid, locale) pair(s) still below {_TARGET} — not enough synonym")
|
||||
print("variety to reach the target without falling back to more generic frames:")
|
||||
for uid, locale, count in shortfalls:
|
||||
print(f" {uid} ({locale}): {count}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -60,7 +60,7 @@ _TEXTCAT_PIPE_NAME = "textcat"
|
|||
# mais avec une confiance dérisoire — bien en dessous de tout seuil
|
||||
# raisonnable pour `CONFIDENCE_THRESHOLD` (`tech-step-matcher.ts`).
|
||||
#
|
||||
# Quatre passes de calibration successives, toutes mesurées contre le
|
||||
# Trois passes de calibration successives, toutes mesurées contre le
|
||||
# corpus réel (74 techniques) :
|
||||
# 1. `150` itérations (calibré pour le corpus original, ~26 techniques) ne
|
||||
# passe plus à l'échelle une fois élargi : `150` sur 74 classes
|
||||
|
|
@ -87,29 +87,7 @@ _TEXTCAT_PIPE_NAME = "textcat"
|
|||
# `CONFIDENCE_THRESHOLD`'s propre commentaire, `tech-step-matcher.ts`)
|
||||
# — ce qui précède est une mesure manuelle ponctuelle, pas un
|
||||
# remplacement de cette calibration.
|
||||
# 4. Le corpus a ensuite été rééquilibré vers 20 `utterances` par technique
|
||||
# et par locale (voir `training_data.py`'s propre commentaire de tête
|
||||
# pour l'algorithme exact et pourquoi certaines techniques restent
|
||||
# volontairement en dessous de 20). Deux tentatives ont mesurablement
|
||||
# échoué avant celle-ci : `_TRAINING_ITERATIONS` inchangé (25) avec un
|
||||
# générateur reposant principalement sur des tournures modales
|
||||
# génériques a fait chuter le F1 agrégé
|
||||
# (`test/recipe-matching/tech-step-eval.test.ts`) à `0.79`, sous le
|
||||
# seuil `0.8` ; doubler `_TRAINING_ITERATIONS` (`10` -> `20`) sur ce
|
||||
# même corpus n'a pas aidé (`0.791`, toujours sous le seuil) — la cause
|
||||
# n'était pas un manque d'itérations mais un générateur qui diluait la
|
||||
# séparabilité entre classes (voir `training_data.py`). Le générateur a
|
||||
# donc été revu (substitution de synonyme en priorité, tournures
|
||||
# modales seulement en complément limité) et `_TRAINING_ITERATIONS`
|
||||
# remonté à `20` sur ce nouveau corpus — mesuré : ~691s (fr, 1923
|
||||
# exemples) / ~645s (en, 1807 exemples), ~1336s pour fr+en combinés.
|
||||
# Confiance nettement rétablie sur les techniques auparavant en échec
|
||||
# (`sweat` ~0.99, `bainMarie` ~0.98, `julienne` ~0.88). À
|
||||
# confirmer/affiner par une vraie repasse de
|
||||
# `calibrate-tech-step-threshold.ts` comme aux étapes précédentes — ce
|
||||
# qui précède reste une mesure ponctuelle plus la gate F1 de CI, pas un
|
||||
# remplacement de cette calibration.
|
||||
_TRAINING_ITERATIONS = 20
|
||||
_TRAINING_ITERATIONS = 25
|
||||
_TRAINING_BATCH_SIZE = 16
|
||||
# Arrêt anticipé : `_TRAINING_ITERATIONS` reste le plafond (le pire cas ne
|
||||
# change pas), un corpus/locale qui converge plus vite n'a pas à payer les
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -1,34 +0,0 @@
|
|||
"""Garde-fou de non-régression pour l'équilibrage du corpus (voir
|
||||
`training_data.py`'s propre commentaire de tête) : un textcat entraîné sur
|
||||
des classes très inégales en nombre d'exemples est une source réelle de
|
||||
classifications confiantes mais fausses sur une phrase jamais vue (constaté
|
||||
en pratique — voir l'historique Git de ce fichier).
|
||||
|
||||
`_MIN_UTTERANCES_PER_LOCALE` est volontairement `12`, pas `20` : la
|
||||
génération vise `20` (`augment_utterances.py`'s `_TARGET`) mais s'arrête
|
||||
avant si la technique n'a pas assez de vocabulaire distinctif propre
|
||||
(`synonyms`) pour l'atteindre sans retomber massivement sur des tournures
|
||||
génériques partagées par toutes les classes — une première version de ce
|
||||
script forçait `20` partout via ce mécanisme et a mesurablement *dégradé*
|
||||
`test/recipe-matching/tech-step-eval.test.ts` (F1 agrégé) plutôt que de
|
||||
l'améliorer, en réduisant la séparabilité entre classes plus qu'en ajoutant
|
||||
un vrai signal. `12` reste très au-dessus du plancher d'origine (3-7) tout
|
||||
en laissant `augment_utterances.py` s'arrêter honnêtement plutôt que de
|
||||
forcer un compte rond au prix de la qualité."""
|
||||
|
||||
from intent_service.training_data import TECH_STEP_TRAINING_DATA
|
||||
|
||||
_MIN_UTTERANCES_PER_LOCALE = 12
|
||||
|
||||
|
||||
def test_every_technique_has_at_least_the_minimum_utterances_per_locale():
|
||||
short = [
|
||||
(entry.uid, locale, len(getattr(entry, locale).utterances))
|
||||
for entry in TECH_STEP_TRAINING_DATA
|
||||
for locale in ("fr", "en")
|
||||
if len(getattr(entry, locale).utterances) < _MIN_UTTERANCES_PER_LOCALE
|
||||
]
|
||||
assert short == [], (
|
||||
f"{len(short)} (uid, locale) pair(s) below the {_MIN_UTTERANCES_PER_LOCALE}-utterance "
|
||||
f"floor: {short}"
|
||||
)
|
||||
Loading…
Reference in a new issue