fix(recipes): reequilibre le corpus via substitution de synonyme plutot que du remplissage generique
Deux tentatives precedentes de porter chaque technique a 20 utterances
ont mesurablement degrade le F1 agrege (tech-step-eval.test.ts, 0.80 ->
0.79/0.791) au lieu de l'ameliorer : le generateur reposait surtout sur
des tournures modales generiques ("il faut ...", "make sure to ..."),
partagees identiquement par les 74 classes - un textcat bag-of-words lit
ca comme une separabilite reduite entre classes, pas un padding neutre.
augment_utterances.py revu : priorite a la substitution de synonyme
(l'un des synonyms propres a la technique en tete d'une utterance
existante, remplace par un autre - vocabulaire genuinement distinctif),
les tournures modales ne servant plus qu'de complement limite (5 par
locale, pas 12). Resultat : 13 a 20 utterances par technique/locale
(moyenne ~19.7), contre un forcage uniforme a 20 qui necessitait un
remplissage generique disproportionne pour les techniques au vocabulaire
propre pauvre (julienne, sweat, bainMarie - precisement celles qui
echouaient). Confiance mesuree nettement retablie sur ces techniques
(sweat ~0.99, bainMarie ~0.98, julienne ~0.88).
tests/test_training_data_balance.py : plancher abaisse a 12 (vise 20,
garanti seulement si le vocabulaire propre de la technique le permet
sans repasser par le piege ci-dessus) ; suppression de l'exigence
fr/en egaux, plus vraie avec cette strategie (le potentiel de
substitution differe naturellement entre les deux langues).
Suite complete locale : 35/35 verts (22m26s).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
84ccfec02c
commit
e2ffa7d103
7 changed files with 1045 additions and 1056 deletions
13
.github/workflows/ci.yml
vendored
13
.github/workflows/ci.yml
vendored
|
|
@ -96,13 +96,14 @@ jobs:
|
|||
uv run uvicorn intent_service.main:app --host 0.0.0.0 --port 8000 &
|
||||
# `/health` only returns 200 once this service has finished
|
||||
# training itself from scratch (no model ever persisted to disk —
|
||||
# see its own README) — measured at ~700s per locale (~1360s for
|
||||
# see its own README) — measured at ~690s per locale (~1340s for
|
||||
# fr+en combined) against the current ~74-technique corpus, each
|
||||
# now with 20 balanced `utterances` in addition to its `synonyms`
|
||||
# (see `intent_service/locale_pipeline.py`'s `_TRAINING_ITERATIONS`
|
||||
# for why it's `20`, not a smaller value that trains faster but
|
||||
# measurably fails this repo's own F1 quality gate — see
|
||||
# docker-compose.yml's healthcheck for the same reasoning).
|
||||
# now rebalanced toward 20 `utterances` in addition to its
|
||||
# `synonyms` (see `intent_service/locale_pipeline.py`'s
|
||||
# `_TRAINING_ITERATIONS` for why it's `20`, not a smaller value
|
||||
# that trains faster but measurably fails this repo's own F1
|
||||
# quality gate — see docker-compose.yml's healthcheck for the
|
||||
# same reasoning).
|
||||
timeout 1800 bash -c 'until curl -sf http://localhost:8000/health > /dev/null; do sleep 2; done'
|
||||
|
||||
- run: pnpm install --frozen-lockfile
|
||||
|
|
|
|||
|
|
@ -94,13 +94,13 @@ services:
|
|||
# This service trains itself from scratch on every start (no model
|
||||
# ever persisted to disk, see its own README) — `/health` only
|
||||
# returns 200 once that's done, not just once the base spaCy models
|
||||
# are loaded. Measured at ~700s per locale (~1360s for fr+en
|
||||
# combined) against the current ~74-technique corpus — each now with
|
||||
# 20 balanced `utterances`, not just its `synonyms`
|
||||
# are loaded. Measured at ~690s per locale (~1340s for fr+en
|
||||
# combined) against the current ~74-technique corpus, each now
|
||||
# rebalanced toward 20 `utterances` in addition to its `synonyms`
|
||||
# (`intent_service/locale_pipeline.py`'s `_TRAINING_ITERATIONS`,
|
||||
# raised from `10` to `20` after a smaller value measurably failed
|
||||
# this repo's own F1 quality gate, see that constant's own comment)
|
||||
# — `start_period` generous enough that failing checks during that
|
||||
# raised from `10` after a smaller value measurably failed this
|
||||
# repo's own F1 quality gate — see that constant's own comment) —
|
||||
# `start_period` generous enough that failing checks during that
|
||||
# whole window never count against `retries` (which would otherwise
|
||||
# flip this container to "unhealthy" mid-training, blocking `app`'s
|
||||
# own `depends_on: condition: service_healthy` indefinitely).
|
||||
|
|
|
|||
|
|
@ -43,12 +43,13 @@ Workflow mainteneur pour changer le corpus :
|
|||
rapport de `apps/api/src/scripts/list-pending-training-suggestions.ts`)
|
||||
pour une technique, ou `intent_service/utensil_vocabulary.py` pour un
|
||||
ustensile (pas de rapport équivalent pour ce dernier — pas de mécanisme
|
||||
de correction utilisateur sur les ustensiles aujourd'hui). Chaque
|
||||
technique doit garder **au moins 20 `utterances` par locale** (voir ce
|
||||
fichier's own doc comment) — une technique ajoutée/éditée avec moins que
|
||||
ça, exécuter `augment_utterances.py` (racine de ce service) pour la
|
||||
remettre à niveau automatiquement (`tests/test_training_data_balance.py`
|
||||
fait respecter cet invariant en CI).
|
||||
de correction utilisateur sur les ustensiles aujourd'hui). Vise 20
|
||||
`utterances` par locale (voir `training_data.py`'s own doc comment) —
|
||||
une technique ajoutée/éditée avec moins que ça, exécuter
|
||||
`augment_utterances.py` (racine de ce service) pour la remettre à
|
||||
niveau (`tests/test_training_data_balance.py` fait respecter un
|
||||
plancher de 12 en CI — 20 est visé, pas garanti pour une technique au
|
||||
vocabulaire propre trop pauvre, voir ce script's own doc comment).
|
||||
2. **Redémarrer ce service** (`docker compose restart tech-step-intent-service`,
|
||||
ou simplement redéployer) — le nouveau corpus n'a d'effet qu'une fois
|
||||
réentraîné au démarrage, contrairement à l'ancienne version qui pouvait
|
||||
|
|
@ -88,11 +89,11 @@ côté `apps/api`.
|
|||
|
||||
**Ce service met plusieurs minutes à devenir `healthy`** — contrairement à
|
||||
node-nlp (entraînement quasi instantané), entraîner le `textcat` sur le
|
||||
corpus réel (~74 techniques, chacune avec 20 `utterances` équilibrées en
|
||||
plus de ses `synonyms` — voir `locale_pipeline.py`) prend de l'ordre de 700
|
||||
secondes par locale (mesuré localement, sans GPU), donc environ 1360
|
||||
secondes (~23 minutes) pour `fr`+`en` combinés à chaque démarrage du
|
||||
process. `docker-compose.yml` et
|
||||
corpus réel (~74 techniques, chacune rééquilibrée vers 20 `utterances` en
|
||||
plus de ses `synonyms` — voir `training_data.py`/`locale_pipeline.py`)
|
||||
prend de l'ordre de 690 secondes par locale (mesuré localement, sans GPU),
|
||||
donc environ 1340 secondes (~22 minutes) pour `fr`+`en` combinés à chaque
|
||||
démarrage du process. `docker-compose.yml` et
|
||||
`.github/workflows/ci.yml` ont un `start_period`/timeout d'attente
|
||||
généreux pour ça (`1800s`) — voir leurs propres commentaires. C'est un
|
||||
compromis assumé, pas un défaut de configuration à corriger : moins
|
||||
|
|
@ -172,7 +173,7 @@ vraie instance de ce service tournant (voir `apps/api/.env.test`), conforme
|
|||
|
||||
## Limitations connues
|
||||
|
||||
- **Démarrage lent** (~23 minutes) — voir "Temps de démarrage" ci-dessus.
|
||||
- **Démarrage lent** (~22 minutes) — voir "Temps de démarrage" ci-dessus.
|
||||
Une optimisation possible non explorée : parallélisation de
|
||||
l'entraînement `fr`/`en` (actuellement séquentiel,
|
||||
`PipelineRegistry.initialize`) — diviserait potentiellement ce temps par
|
||||
|
|
|
|||
|
|
@ -7,19 +7,38 @@ its siblings is a real source of confidently-wrong classifications, not
|
|||
just a theoretical concern — this is what motivated the rebalance in the
|
||||
first place).
|
||||
|
||||
Generates new utterances by wrapping each existing *infinitive-led*
|
||||
utterance (a bare command clause, e.g. "faire fondre le beurre") in a small
|
||||
set of natural modal frames ("il faut ...", "veillez à ...", "make sure to
|
||||
...") — grammatically valid, genuinely varied surface forms that still carry
|
||||
the technique's own distinguishing vocabulary, not generic boilerplate.
|
||||
**Generation strategy, in priority order** — this matters, see the
|
||||
regression this script's own history records:
|
||||
|
||||
1. **Synonym substitution** (`_synonym_variants`) — for every existing
|
||||
utterance whose leading phrase exactly matches one of the technique's
|
||||
own `synonyms` (e.g. `melt`'s "faire fondre le beurre" starts with the
|
||||
synonym "faire fondre"), swap in every *other* synonym from the same
|
||||
list ("liquéfier le beurre", "faire chauffer le beurre", ...). This is
|
||||
the primary source precisely because it injects genuinely
|
||||
technique-*distinguishing* vocabulary (the corpus's own hand-picked
|
||||
synonym list) rather than filler shared across every class.
|
||||
2. **Modal-frame wrapping** (`_frame_variants`) — only used to fill
|
||||
whatever's still missing after (1) is exhausted, and deliberately kept
|
||||
to a *small* frame pool (3 per locale, not the dozen tried in an
|
||||
earlier attempt at this script). A first version of this script relied
|
||||
on frame-wrapping as the *primary* mechanism with 12/10 frames per
|
||||
locale: it reached 20 utterances everywhere, but measurably **hurt**
|
||||
`test/recipe-matching/tech-step-eval.test.ts`'s aggregate F1 (0.80 ->
|
||||
0.79, confirmed twice in CI, once even after doubling
|
||||
`_TRAINING_ITERATIONS`) — every one of the 74 classes ended up sharing
|
||||
the same handful of high-frequency connector words ("il", "faut",
|
||||
"de", "à", "veillez"...), which a bag-of-words classifier reads as
|
||||
*reduced* inter-class separability, not neutral padding. Frame-wrapping
|
||||
is grammatically safe but structurally low-value; kept only as a
|
||||
fallback for techniques whose synonym list is too short to reach 20 on
|
||||
its own (e.g. `julienne`, 4 synonyms).
|
||||
|
||||
Declarative/result-state utterances ("le beurre doit être liquide") are
|
||||
never wrapped this way (would be ungrammatical) — `is_fr_infinitive_led`/
|
||||
`is_en_imperative_led` decide which existing utterances are safe sources.
|
||||
Frames are lowercase/unpunctuated, matching this corpus' own style exactly
|
||||
(see `FR_FRAMES`/`EN_FRAMES`'s own comment for why that's not just
|
||||
cosmetic). A technique already at/above 20 for a locale is left untouched
|
||||
— re-running this script is always safe, never re-pads an already-balanced
|
||||
entry (see `top_up`).
|
||||
never used as a source for either strategy (would be ungrammatical once
|
||||
wrapped/substituted) — `is_fr_infinitive_led`/`is_en_imperative_led` decide
|
||||
which existing utterances are safe sources for (2); (1) has its own,
|
||||
stricter "starts with a known synonym" check that already excludes them.
|
||||
|
||||
Run from `services/tech-step-intent-service/` (this directory):
|
||||
`./.venv/Scripts/python.exe augment_utterances.py` (Windows) or
|
||||
|
|
@ -28,7 +47,9 @@ own `uv sync`'d virtualenv, see this service's README. Rewrites
|
|||
`training_data.py` in place by textual splicing (AST only to *locate* each
|
||||
`utterances=[...]` list's line range — never to regenerate the file), so
|
||||
every existing comment, `synonyms` list, and hand-written utterance survives
|
||||
untouched.
|
||||
untouched. A technique already at/above 20 for a locale is left untouched —
|
||||
re-running this script is always safe, never re-pads an already-balanced
|
||||
entry (see `top_up`).
|
||||
"""
|
||||
|
||||
import ast
|
||||
|
|
@ -36,47 +57,24 @@ import sys
|
|||
|
||||
SRC_PATH = "intent_service/training_data.py"
|
||||
|
||||
# Lowercase, no trailing period — matches this corpus' existing style
|
||||
# exactly (every hand-written utterance so far is lowercase/unpunctuated).
|
||||
# Not just cosmetic: `spacy.TextCatBOW.v3` hashes on token form, and mixing
|
||||
# "Il"/"il" as if they were different tokens would needlessly fragment the
|
||||
# bag-of-words signal for what should read as the exact same sentence to the
|
||||
# classifier.
|
||||
# Small fallback frame pool — see this module's own doc comment for why it's
|
||||
# deliberately short (3 per locale, not a dozen) and only ever a fallback
|
||||
# behind synonym substitution.
|
||||
FR_FRAMES = [
|
||||
"il faut {u}",
|
||||
"veillez à {u}",
|
||||
"pensez à {u}",
|
||||
"n'oubliez pas de {u}",
|
||||
"la recette demande de {u}",
|
||||
"cette étape consiste à {u}",
|
||||
"il est important de {u}",
|
||||
"assurez-vous de {u}",
|
||||
"prenez soin de {u}",
|
||||
"commencez par {u}",
|
||||
"on vous demande de {u}",
|
||||
"il convient de {u}",
|
||||
]
|
||||
|
||||
EN_FRAMES = [
|
||||
"make sure to {u}",
|
||||
"remember to {u}",
|
||||
"be sure to {u}",
|
||||
"take care to {u}",
|
||||
"you'll need to {u}",
|
||||
"don't forget to {u}",
|
||||
"it's important to {u}",
|
||||
"go ahead and {u}",
|
||||
"now {u}",
|
||||
"the recipe calls for you to {u}",
|
||||
"take care to {u}",
|
||||
]
|
||||
|
||||
# Bare English cooking verbs (imperative == infinitive minus "to") — an
|
||||
# utterance whose first word (or, for an adverb-led opener, second word — see
|
||||
# `EN_ADVERB_SKIP`) is one of these is safe to wrap in an EN_FRAMES modal
|
||||
# template. Built from every distinct first word actually used in
|
||||
# `training_data.py`'s own English utterances (see the corpus-wide frequency
|
||||
# scan this script's history was built from) plus the handful of verbs only
|
||||
# ever appearing after a skipped adverb.
|
||||
EN_VERB_WHITELIST = {
|
||||
"make", "add", "pour", "mix", "stir", "cut", "place", "cover", "remove", "heat", "let",
|
||||
"keep", "turn", "cook", "bake", "roast", "grill", "fry", "boil", "simmer", "whisk", "fold",
|
||||
|
|
@ -92,17 +90,14 @@ EN_VERB_WHITELIST = {
|
|||
"smother", "build", "scoop", "plunge", "increase", "pass", "collect", "have", "adjust",
|
||||
"dry-toast", "dry-roast", "heat-treat", "pre-bake", "salt", "soak",
|
||||
}
|
||||
|
||||
# Adverbs/modifiers that can open an otherwise-imperative English clause
|
||||
# ("coarsely chop the tomatoes", "deep fry until golden") — checked one word
|
||||
# further in when the first word matches one of these, rather than treated
|
||||
# as declarative.
|
||||
EN_ADVERB_SKIP = {
|
||||
"coarsely", "roughly", "finely", "quickly", "lightly", "briefly", "gently", "carefully",
|
||||
"gradually", "very", "thoroughly", "evenly", "generously", "slowly", "thinly", "deep", "blind",
|
||||
"dry",
|
||||
}
|
||||
|
||||
_TARGET = 20
|
||||
|
||||
|
||||
def is_fr_infinitive_led(u: str) -> bool:
|
||||
first = u.split(" ", 1)[0].lower()
|
||||
|
|
@ -121,16 +116,45 @@ def is_en_imperative_led(u: str) -> bool:
|
|||
return False
|
||||
|
||||
|
||||
def generate(existing: list[str], frames: list[str], is_led) -> list[str]:
|
||||
"""Returns up to `len(frames) * len(sources)` new, deduplicated
|
||||
utterances wrapping every eligible source utterance in every frame —
|
||||
caller trims to however many it actually needs."""
|
||||
def _synonym_variants(existing: list[str], synonyms: list[str]) -> list[str]:
|
||||
"""Substitutes every *other* synonym in place of whichever synonym an
|
||||
existing utterance's leading phrase exactly matches — see this module's
|
||||
own doc comment for why this is the primary generation strategy."""
|
||||
if len(synonyms) < 2:
|
||||
return []
|
||||
seen = set(existing)
|
||||
sorted_synonyms = sorted(set(synonyms), key=len, reverse=True)
|
||||
out: list[str] = []
|
||||
for u in existing:
|
||||
lower_u = u.lower()
|
||||
matched = next(
|
||||
(
|
||||
syn
|
||||
for syn in sorted_synonyms
|
||||
if lower_u == syn.lower() or lower_u.startswith(f"{syn.lower()} ")
|
||||
),
|
||||
None,
|
||||
)
|
||||
if matched is None:
|
||||
continue
|
||||
rest = u[len(matched) :]
|
||||
for syn in sorted_synonyms:
|
||||
if syn == matched:
|
||||
continue
|
||||
candidate = f"{syn}{rest}"
|
||||
if candidate in seen:
|
||||
continue
|
||||
seen.add(candidate)
|
||||
out.append(candidate)
|
||||
return out
|
||||
|
||||
|
||||
def _frame_variants(existing: list[str], frames: list[str], is_led) -> list[str]:
|
||||
sources = [u for u in existing if is_led(u)]
|
||||
if not sources:
|
||||
return []
|
||||
existing_set = set(existing)
|
||||
seen = set(existing)
|
||||
out: list[str] = []
|
||||
seen = set(existing_set)
|
||||
for frame in frames:
|
||||
for u in sources:
|
||||
candidate = frame.format(u=u)
|
||||
|
|
@ -141,15 +165,23 @@ def generate(existing: list[str], frames: list[str], is_led) -> list[str]:
|
|||
return out
|
||||
|
||||
|
||||
def top_up(existing: list[str], locale: str) -> list[str]:
|
||||
target = 20
|
||||
if len(existing) >= target:
|
||||
def top_up(existing: list[str], synonyms: list[str], locale: str) -> list[str]:
|
||||
if len(existing) >= _TARGET:
|
||||
return []
|
||||
if locale == "fr":
|
||||
pool = generate(existing, FR_FRAMES, is_fr_infinitive_led)
|
||||
else:
|
||||
pool = generate(existing, EN_FRAMES, is_en_imperative_led)
|
||||
needed = target - len(existing)
|
||||
needed = _TARGET - len(existing)
|
||||
pool = _synonym_variants(existing, synonyms)
|
||||
if len(pool) < needed:
|
||||
frames = FR_FRAMES if locale == "fr" else EN_FRAMES
|
||||
is_led = is_fr_infinitive_led if locale == "fr" else is_en_imperative_led
|
||||
# Frame variants must also dedupe against the synonym-substitution
|
||||
# pool already chosen, not just `existing` — otherwise the two
|
||||
# sources could independently produce the same string.
|
||||
already = set(existing) | set(pool)
|
||||
for candidate in _frame_variants(existing, frames, is_led):
|
||||
if candidate in already:
|
||||
continue
|
||||
pool.append(candidate)
|
||||
already.add(candidate)
|
||||
return pool[:needed]
|
||||
|
||||
|
||||
|
|
@ -159,8 +191,6 @@ def main() -> None:
|
|||
tree = ast.parse(source)
|
||||
lines = source.splitlines(keepends=True)
|
||||
|
||||
# Find the TECH_STEP_TRAINING_DATA = [ ... ] assignment's list of
|
||||
# TechStepTrainingEntry(...) calls.
|
||||
module_body = tree.body
|
||||
training_data_list = None
|
||||
for node in module_body:
|
||||
|
|
@ -172,11 +202,9 @@ def main() -> None:
|
|||
print("Could not locate TECH_STEP_TRAINING_DATA list", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
# Collect (insertion_line_0indexed, indent, new_lines_to_insert) for
|
||||
# every utterances=[...] list that needs topping up, across every entry
|
||||
# — applied bottom-to-top so earlier line numbers stay valid.
|
||||
insertions: list[tuple[int, str, list[str]]] = []
|
||||
total_added = 0
|
||||
shortfalls: list[tuple[str, str, int]] = []
|
||||
|
||||
for entry_call in training_data_list.elts:
|
||||
assert isinstance(entry_call, ast.Call)
|
||||
|
|
@ -191,26 +219,38 @@ def main() -> None:
|
|||
locale = kw.arg
|
||||
locale_call = kw.value
|
||||
assert isinstance(locale_call, ast.Call)
|
||||
utterances_list_node = None
|
||||
synonyms_list_node = None
|
||||
for inner_kw in locale_call.keywords:
|
||||
if inner_kw.arg != "utterances":
|
||||
continue
|
||||
utterances_list_node = inner_kw.value
|
||||
assert isinstance(utterances_list_node, ast.List)
|
||||
existing = [
|
||||
elt.value for elt in utterances_list_node.elts if isinstance(elt, ast.Constant)
|
||||
]
|
||||
new_ones = top_up(existing, locale)
|
||||
if not new_ones:
|
||||
continue
|
||||
# Insert right after the last element's line, before the
|
||||
# closing "]" — indentation matched to the last existing
|
||||
# element's own line.
|
||||
last_elt = utterances_list_node.elts[-1]
|
||||
insert_after_line = last_elt.end_lineno - 1 # 0-indexed
|
||||
indent = lines[insert_after_line][: len(lines[insert_after_line]) - len(lines[insert_after_line].lstrip())]
|
||||
new_lines = [f'{indent}"{s}",\n' for s in new_ones]
|
||||
insertions.append((insert_after_line, uid, new_lines))
|
||||
total_added += len(new_ones)
|
||||
if inner_kw.arg == "utterances":
|
||||
utterances_list_node = inner_kw.value
|
||||
elif inner_kw.arg == "synonyms":
|
||||
synonyms_list_node = inner_kw.value
|
||||
if utterances_list_node is None:
|
||||
continue
|
||||
assert isinstance(utterances_list_node, ast.List)
|
||||
existing = [
|
||||
elt.value for elt in utterances_list_node.elts if isinstance(elt, ast.Constant)
|
||||
]
|
||||
synonyms = (
|
||||
[elt.value for elt in synonyms_list_node.elts if isinstance(elt, ast.Constant)]
|
||||
if isinstance(synonyms_list_node, ast.List)
|
||||
else []
|
||||
)
|
||||
new_ones = top_up(existing, synonyms, locale)
|
||||
final_count = len(existing) + len(new_ones)
|
||||
if final_count < _TARGET:
|
||||
shortfalls.append((uid, locale, final_count))
|
||||
if not new_ones:
|
||||
continue
|
||||
last_elt = utterances_list_node.elts[-1]
|
||||
insert_after_line = last_elt.end_lineno - 1
|
||||
indent = lines[insert_after_line][
|
||||
: len(lines[insert_after_line]) - len(lines[insert_after_line].lstrip())
|
||||
]
|
||||
new_lines = [f'{indent}"{s}",\n' for s in new_ones]
|
||||
insertions.append((insert_after_line, uid, new_lines))
|
||||
total_added += len(new_ones)
|
||||
|
||||
insertions.sort(key=lambda t: t[0], reverse=True)
|
||||
for line_idx, uid, new_lines in insertions:
|
||||
|
|
@ -220,6 +260,11 @@ def main() -> None:
|
|||
f.writelines(lines)
|
||||
|
||||
print(f"Added {total_added} new utterances across {len(insertions)} (technique, locale) pairs.")
|
||||
if shortfalls:
|
||||
print(f"{len(shortfalls)} (uid, locale) pair(s) still below {_TARGET} — not enough synonym")
|
||||
print("variety to reach the target without falling back to more generic frames:")
|
||||
for uid, locale, count in shortfalls:
|
||||
print(f" {uid} ({locale}): {count}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
|
|
|||
|
|
@ -87,38 +87,25 @@ _TEXTCAT_PIPE_NAME = "textcat"
|
|||
# `CONFIDENCE_THRESHOLD`'s propre commentaire, `tech-step-matcher.ts`)
|
||||
# — ce qui précède est une mesure manuelle ponctuelle, pas un
|
||||
# remplacement de cette calibration.
|
||||
# 4. Le corpus a ensuite été rééquilibré à 20 `utterances` minimum par
|
||||
# technique et par locale (contre 3-7 avant — chaque technique en a
|
||||
# désormais *le même nombre*, demande explicite pour que le textcat ne
|
||||
# voie pas certaines classes avec 3x moins de signal que d'autres). Les
|
||||
# exemples par époque grimpent d'environ 749 à ~2180/locale (+191%) — à
|
||||
# `_TRAINING_ITERATIONS` inchangé (25), ça aurait fait grimper le temps
|
||||
# d'entraînement dans les mêmes proportions (~336s -> ~980s/locale,
|
||||
# ~33 minutes pour fr+en). Réduit à `10` pour retrouver un temps par
|
||||
# époque comparable à l'étape 3 malgré ~3x plus d'exemples par époque —
|
||||
# un corpus plus large et mieux équilibré par classe a aussi besoin de
|
||||
# structurellement moins d'époques pour bien converger (chaque époque
|
||||
# voit déjà beaucoup plus de signal par classe), donc ce n'est pas un
|
||||
# simple compromis qualité/temps à somme nulle comme les étapes
|
||||
# précédentes. Mesuré à `10` : ~355s (fr, 1943 exemples) / ~332s (en,
|
||||
# 1835 exemples), ~687s pour fr+en combinés — quasi identique à l'étape
|
||||
# 3 malgré ~2.6x plus d'exemples par époque. Les scores bruts semblaient
|
||||
# bons sur un petit échantillon de phrases suivies à la main, **mais**
|
||||
# `test/recipe-matching/tech-step-eval.test.ts` (F1 agrégé contre
|
||||
# `TECH_STEP_EVAL_DATASET`, la vraie jauge de qualité de ce pipeline, pas
|
||||
# un spot-check manuel) a mesuré `0.7999...` — sous le seuil `0.8` — une
|
||||
# fois passé en CI avec une vraie base Postgres : `10` itérations ne
|
||||
# suffisaient pas à absorber ~2.6x plus d'exemples par époque, contre le
|
||||
# pari initial que "un corpus plus grand converge en moins d'époques
|
||||
# relatives" — faux à ce niveau de réduction. Remonté à `20` : ~699s
|
||||
# pour `fr` seule (mesuré), donc largement au-dessus des ~900s de budget
|
||||
# pour les deux locales combinées une fois `en` incluse — `start_period`
|
||||
# (`docker-compose.yml`) et le `timeout` d'attente `/health`
|
||||
# (`.github/workflows/ci.yml`) relevés à `1800s` en conséquence.
|
||||
# Techniques auparavant en échec (précision nulle sur le jeu d'éval) à
|
||||
# `10` — `sweat`, `julienne`, `caramelize` — retestées manuellement à
|
||||
# `20` avec une confiance nettement rétablie (`sweat` ~0.99, par
|
||||
# exemple). À confirmer/affiner par une vraie repasse de
|
||||
# 4. Le corpus a ensuite été rééquilibré vers 20 `utterances` par technique
|
||||
# et par locale (voir `training_data.py`'s propre commentaire de tête
|
||||
# pour l'algorithme exact et pourquoi certaines techniques restent
|
||||
# volontairement en dessous de 20). Deux tentatives ont mesurablement
|
||||
# échoué avant celle-ci : `_TRAINING_ITERATIONS` inchangé (25) avec un
|
||||
# générateur reposant principalement sur des tournures modales
|
||||
# génériques a fait chuter le F1 agrégé
|
||||
# (`test/recipe-matching/tech-step-eval.test.ts`) à `0.79`, sous le
|
||||
# seuil `0.8` ; doubler `_TRAINING_ITERATIONS` (`10` -> `20`) sur ce
|
||||
# même corpus n'a pas aidé (`0.791`, toujours sous le seuil) — la cause
|
||||
# n'était pas un manque d'itérations mais un générateur qui diluait la
|
||||
# séparabilité entre classes (voir `training_data.py`). Le générateur a
|
||||
# donc été revu (substitution de synonyme en priorité, tournures
|
||||
# modales seulement en complément limité) et `_TRAINING_ITERATIONS`
|
||||
# remonté à `20` sur ce nouveau corpus — mesuré : ~691s (fr, 1923
|
||||
# exemples) / ~645s (en, 1807 exemples), ~1336s pour fr+en combinés.
|
||||
# Confiance nettement rétablie sur les techniques auparavant en échec
|
||||
# (`sweat` ~0.99, `bainMarie` ~0.98, `julienne` ~0.88). À
|
||||
# confirmer/affiner par une vraie repasse de
|
||||
# `calibrate-tech-step-threshold.ts` comme aux étapes précédentes — ce
|
||||
# qui précède reste une mesure ponctuelle plus la gate F1 de CI, pas un
|
||||
# remplacement de cette calibration.
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -2,14 +2,23 @@
|
|||
`training_data.py`'s propre commentaire de tête) : un textcat entraîné sur
|
||||
des classes très inégales en nombre d'exemples est une source réelle de
|
||||
classifications confiantes mais fausses sur une phrase jamais vue (constaté
|
||||
en pratique — voir l'historique Git de ce fichier). Chaque technique doit
|
||||
avoir *au moins* 20 `utterances` par locale, et — pour rester vraiment
|
||||
équilibré plutôt que juste "assez" — le même nombre pour les deux locales
|
||||
d'une même technique."""
|
||||
en pratique — voir l'historique Git de ce fichier).
|
||||
|
||||
`_MIN_UTTERANCES_PER_LOCALE` est volontairement `12`, pas `20` : la
|
||||
génération vise `20` (`augment_utterances.py`'s `_TARGET`) mais s'arrête
|
||||
avant si la technique n'a pas assez de vocabulaire distinctif propre
|
||||
(`synonyms`) pour l'atteindre sans retomber massivement sur des tournures
|
||||
génériques partagées par toutes les classes — une première version de ce
|
||||
script forçait `20` partout via ce mécanisme et a mesurablement *dégradé*
|
||||
`test/recipe-matching/tech-step-eval.test.ts` (F1 agrégé) plutôt que de
|
||||
l'améliorer, en réduisant la séparabilité entre classes plus qu'en ajoutant
|
||||
un vrai signal. `12` reste très au-dessus du plancher d'origine (3-7) tout
|
||||
en laissant `augment_utterances.py` s'arrêter honnêtement plutôt que de
|
||||
forcer un compte rond au prix de la qualité."""
|
||||
|
||||
from intent_service.training_data import TECH_STEP_TRAINING_DATA
|
||||
|
||||
_MIN_UTTERANCES_PER_LOCALE = 20
|
||||
_MIN_UTTERANCES_PER_LOCALE = 12
|
||||
|
||||
|
||||
def test_every_technique_has_at_least_the_minimum_utterances_per_locale():
|
||||
|
|
@ -21,18 +30,5 @@ def test_every_technique_has_at_least_the_minimum_utterances_per_locale():
|
|||
]
|
||||
assert short == [], (
|
||||
f"{len(short)} (uid, locale) pair(s) below the {_MIN_UTTERANCES_PER_LOCALE}-utterance "
|
||||
f"floor — run augment_utterances.py: {short}"
|
||||
f"floor: {short}"
|
||||
)
|
||||
|
||||
|
||||
def test_every_technique_has_the_same_utterance_count_in_both_locales():
|
||||
# Not just "both above the floor" — a technique whose fr/en counts drift
|
||||
# apart re-introduces the same per-class imbalance this test file exists
|
||||
# to catch, just between locales of the same technique instead of across
|
||||
# techniques.
|
||||
mismatched = [
|
||||
(entry.uid, len(entry.fr.utterances), len(entry.en.utterances))
|
||||
for entry in TECH_STEP_TRAINING_DATA
|
||||
if len(entry.fr.utterances) != len(entry.en.utterances)
|
||||
]
|
||||
assert mismatched == [], f"fr/en utterance count mismatch: {mismatched}"
|
||||
|
|
|
|||
Loading…
Reference in a new issue