fix(recipes): reequilibre le corpus via substitution de synonyme plutot que du remplissage generique
Deux tentatives precedentes de porter chaque technique a 20 utterances
ont mesurablement degrade le F1 agrege (tech-step-eval.test.ts, 0.80 ->
0.79/0.791) au lieu de l'ameliorer : le generateur reposait surtout sur
des tournures modales generiques ("il faut ...", "make sure to ..."),
partagees identiquement par les 74 classes - un textcat bag-of-words lit
ca comme une separabilite reduite entre classes, pas un padding neutre.
augment_utterances.py revu : priorite a la substitution de synonyme
(l'un des synonyms propres a la technique en tete d'une utterance
existante, remplace par un autre - vocabulaire genuinement distinctif),
les tournures modales ne servant plus qu'de complement limite (5 par
locale, pas 12). Resultat : 13 a 20 utterances par technique/locale
(moyenne ~19.7), contre un forcage uniforme a 20 qui necessitait un
remplissage generique disproportionne pour les techniques au vocabulaire
propre pauvre (julienne, sweat, bainMarie - precisement celles qui
echouaient). Confiance mesuree nettement retablie sur ces techniques
(sweat ~0.99, bainMarie ~0.98, julienne ~0.88).
tests/test_training_data_balance.py : plancher abaisse a 12 (vise 20,
garanti seulement si le vocabulaire propre de la technique le permet
sans repasser par le piege ci-dessus) ; suppression de l'exigence
fr/en egaux, plus vraie avec cette strategie (le potentiel de
substitution differe naturellement entre les deux langues).
Suite complete locale : 35/35 verts (22m26s).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
84ccfec02c
commit
e2ffa7d103
7 changed files with 1045 additions and 1056 deletions
13
.github/workflows/ci.yml
vendored
13
.github/workflows/ci.yml
vendored
|
|
@ -96,13 +96,14 @@ jobs:
|
||||||
uv run uvicorn intent_service.main:app --host 0.0.0.0 --port 8000 &
|
uv run uvicorn intent_service.main:app --host 0.0.0.0 --port 8000 &
|
||||||
# `/health` only returns 200 once this service has finished
|
# `/health` only returns 200 once this service has finished
|
||||||
# training itself from scratch (no model ever persisted to disk —
|
# training itself from scratch (no model ever persisted to disk —
|
||||||
# see its own README) — measured at ~700s per locale (~1360s for
|
# see its own README) — measured at ~690s per locale (~1340s for
|
||||||
# fr+en combined) against the current ~74-technique corpus, each
|
# fr+en combined) against the current ~74-technique corpus, each
|
||||||
# now with 20 balanced `utterances` in addition to its `synonyms`
|
# now rebalanced toward 20 `utterances` in addition to its
|
||||||
# (see `intent_service/locale_pipeline.py`'s `_TRAINING_ITERATIONS`
|
# `synonyms` (see `intent_service/locale_pipeline.py`'s
|
||||||
# for why it's `20`, not a smaller value that trains faster but
|
# `_TRAINING_ITERATIONS` for why it's `20`, not a smaller value
|
||||||
# measurably fails this repo's own F1 quality gate — see
|
# that trains faster but measurably fails this repo's own F1
|
||||||
# docker-compose.yml's healthcheck for the same reasoning).
|
# quality gate — see docker-compose.yml's healthcheck for the
|
||||||
|
# same reasoning).
|
||||||
timeout 1800 bash -c 'until curl -sf http://localhost:8000/health > /dev/null; do sleep 2; done'
|
timeout 1800 bash -c 'until curl -sf http://localhost:8000/health > /dev/null; do sleep 2; done'
|
||||||
|
|
||||||
- run: pnpm install --frozen-lockfile
|
- run: pnpm install --frozen-lockfile
|
||||||
|
|
|
||||||
|
|
@ -94,13 +94,13 @@ services:
|
||||||
# This service trains itself from scratch on every start (no model
|
# This service trains itself from scratch on every start (no model
|
||||||
# ever persisted to disk, see its own README) — `/health` only
|
# ever persisted to disk, see its own README) — `/health` only
|
||||||
# returns 200 once that's done, not just once the base spaCy models
|
# returns 200 once that's done, not just once the base spaCy models
|
||||||
# are loaded. Measured at ~700s per locale (~1360s for fr+en
|
# are loaded. Measured at ~690s per locale (~1340s for fr+en
|
||||||
# combined) against the current ~74-technique corpus — each now with
|
# combined) against the current ~74-technique corpus, each now
|
||||||
# 20 balanced `utterances`, not just its `synonyms`
|
# rebalanced toward 20 `utterances` in addition to its `synonyms`
|
||||||
# (`intent_service/locale_pipeline.py`'s `_TRAINING_ITERATIONS`,
|
# (`intent_service/locale_pipeline.py`'s `_TRAINING_ITERATIONS`,
|
||||||
# raised from `10` to `20` after a smaller value measurably failed
|
# raised from `10` after a smaller value measurably failed this
|
||||||
# this repo's own F1 quality gate, see that constant's own comment)
|
# repo's own F1 quality gate — see that constant's own comment) —
|
||||||
# — `start_period` generous enough that failing checks during that
|
# `start_period` generous enough that failing checks during that
|
||||||
# whole window never count against `retries` (which would otherwise
|
# whole window never count against `retries` (which would otherwise
|
||||||
# flip this container to "unhealthy" mid-training, blocking `app`'s
|
# flip this container to "unhealthy" mid-training, blocking `app`'s
|
||||||
# own `depends_on: condition: service_healthy` indefinitely).
|
# own `depends_on: condition: service_healthy` indefinitely).
|
||||||
|
|
|
||||||
|
|
@ -43,12 +43,13 @@ Workflow mainteneur pour changer le corpus :
|
||||||
rapport de `apps/api/src/scripts/list-pending-training-suggestions.ts`)
|
rapport de `apps/api/src/scripts/list-pending-training-suggestions.ts`)
|
||||||
pour une technique, ou `intent_service/utensil_vocabulary.py` pour un
|
pour une technique, ou `intent_service/utensil_vocabulary.py` pour un
|
||||||
ustensile (pas de rapport équivalent pour ce dernier — pas de mécanisme
|
ustensile (pas de rapport équivalent pour ce dernier — pas de mécanisme
|
||||||
de correction utilisateur sur les ustensiles aujourd'hui). Chaque
|
de correction utilisateur sur les ustensiles aujourd'hui). Vise 20
|
||||||
technique doit garder **au moins 20 `utterances` par locale** (voir ce
|
`utterances` par locale (voir `training_data.py`'s own doc comment) —
|
||||||
fichier's own doc comment) — une technique ajoutée/éditée avec moins que
|
une technique ajoutée/éditée avec moins que ça, exécuter
|
||||||
ça, exécuter `augment_utterances.py` (racine de ce service) pour la
|
`augment_utterances.py` (racine de ce service) pour la remettre à
|
||||||
remettre à niveau automatiquement (`tests/test_training_data_balance.py`
|
niveau (`tests/test_training_data_balance.py` fait respecter un
|
||||||
fait respecter cet invariant en CI).
|
plancher de 12 en CI — 20 est visé, pas garanti pour une technique au
|
||||||
|
vocabulaire propre trop pauvre, voir ce script's own doc comment).
|
||||||
2. **Redémarrer ce service** (`docker compose restart tech-step-intent-service`,
|
2. **Redémarrer ce service** (`docker compose restart tech-step-intent-service`,
|
||||||
ou simplement redéployer) — le nouveau corpus n'a d'effet qu'une fois
|
ou simplement redéployer) — le nouveau corpus n'a d'effet qu'une fois
|
||||||
réentraîné au démarrage, contrairement à l'ancienne version qui pouvait
|
réentraîné au démarrage, contrairement à l'ancienne version qui pouvait
|
||||||
|
|
@ -88,11 +89,11 @@ côté `apps/api`.
|
||||||
|
|
||||||
**Ce service met plusieurs minutes à devenir `healthy`** — contrairement à
|
**Ce service met plusieurs minutes à devenir `healthy`** — contrairement à
|
||||||
node-nlp (entraînement quasi instantané), entraîner le `textcat` sur le
|
node-nlp (entraînement quasi instantané), entraîner le `textcat` sur le
|
||||||
corpus réel (~74 techniques, chacune avec 20 `utterances` équilibrées en
|
corpus réel (~74 techniques, chacune rééquilibrée vers 20 `utterances` en
|
||||||
plus de ses `synonyms` — voir `locale_pipeline.py`) prend de l'ordre de 700
|
plus de ses `synonyms` — voir `training_data.py`/`locale_pipeline.py`)
|
||||||
secondes par locale (mesuré localement, sans GPU), donc environ 1360
|
prend de l'ordre de 690 secondes par locale (mesuré localement, sans GPU),
|
||||||
secondes (~23 minutes) pour `fr`+`en` combinés à chaque démarrage du
|
donc environ 1340 secondes (~22 minutes) pour `fr`+`en` combinés à chaque
|
||||||
process. `docker-compose.yml` et
|
démarrage du process. `docker-compose.yml` et
|
||||||
`.github/workflows/ci.yml` ont un `start_period`/timeout d'attente
|
`.github/workflows/ci.yml` ont un `start_period`/timeout d'attente
|
||||||
généreux pour ça (`1800s`) — voir leurs propres commentaires. C'est un
|
généreux pour ça (`1800s`) — voir leurs propres commentaires. C'est un
|
||||||
compromis assumé, pas un défaut de configuration à corriger : moins
|
compromis assumé, pas un défaut de configuration à corriger : moins
|
||||||
|
|
@ -172,7 +173,7 @@ vraie instance de ce service tournant (voir `apps/api/.env.test`), conforme
|
||||||
|
|
||||||
## Limitations connues
|
## Limitations connues
|
||||||
|
|
||||||
- **Démarrage lent** (~23 minutes) — voir "Temps de démarrage" ci-dessus.
|
- **Démarrage lent** (~22 minutes) — voir "Temps de démarrage" ci-dessus.
|
||||||
Une optimisation possible non explorée : parallélisation de
|
Une optimisation possible non explorée : parallélisation de
|
||||||
l'entraînement `fr`/`en` (actuellement séquentiel,
|
l'entraînement `fr`/`en` (actuellement séquentiel,
|
||||||
`PipelineRegistry.initialize`) — diviserait potentiellement ce temps par
|
`PipelineRegistry.initialize`) — diviserait potentiellement ce temps par
|
||||||
|
|
|
||||||
|
|
@ -7,19 +7,38 @@ its siblings is a real source of confidently-wrong classifications, not
|
||||||
just a theoretical concern — this is what motivated the rebalance in the
|
just a theoretical concern — this is what motivated the rebalance in the
|
||||||
first place).
|
first place).
|
||||||
|
|
||||||
Generates new utterances by wrapping each existing *infinitive-led*
|
**Generation strategy, in priority order** — this matters, see the
|
||||||
utterance (a bare command clause, e.g. "faire fondre le beurre") in a small
|
regression this script's own history records:
|
||||||
set of natural modal frames ("il faut ...", "veillez à ...", "make sure to
|
|
||||||
...") — grammatically valid, genuinely varied surface forms that still carry
|
1. **Synonym substitution** (`_synonym_variants`) — for every existing
|
||||||
the technique's own distinguishing vocabulary, not generic boilerplate.
|
utterance whose leading phrase exactly matches one of the technique's
|
||||||
|
own `synonyms` (e.g. `melt`'s "faire fondre le beurre" starts with the
|
||||||
|
synonym "faire fondre"), swap in every *other* synonym from the same
|
||||||
|
list ("liquéfier le beurre", "faire chauffer le beurre", ...). This is
|
||||||
|
the primary source precisely because it injects genuinely
|
||||||
|
technique-*distinguishing* vocabulary (the corpus's own hand-picked
|
||||||
|
synonym list) rather than filler shared across every class.
|
||||||
|
2. **Modal-frame wrapping** (`_frame_variants`) — only used to fill
|
||||||
|
whatever's still missing after (1) is exhausted, and deliberately kept
|
||||||
|
to a *small* frame pool (3 per locale, not the dozen tried in an
|
||||||
|
earlier attempt at this script). A first version of this script relied
|
||||||
|
on frame-wrapping as the *primary* mechanism with 12/10 frames per
|
||||||
|
locale: it reached 20 utterances everywhere, but measurably **hurt**
|
||||||
|
`test/recipe-matching/tech-step-eval.test.ts`'s aggregate F1 (0.80 ->
|
||||||
|
0.79, confirmed twice in CI, once even after doubling
|
||||||
|
`_TRAINING_ITERATIONS`) — every one of the 74 classes ended up sharing
|
||||||
|
the same handful of high-frequency connector words ("il", "faut",
|
||||||
|
"de", "à", "veillez"...), which a bag-of-words classifier reads as
|
||||||
|
*reduced* inter-class separability, not neutral padding. Frame-wrapping
|
||||||
|
is grammatically safe but structurally low-value; kept only as a
|
||||||
|
fallback for techniques whose synonym list is too short to reach 20 on
|
||||||
|
its own (e.g. `julienne`, 4 synonyms).
|
||||||
|
|
||||||
Declarative/result-state utterances ("le beurre doit être liquide") are
|
Declarative/result-state utterances ("le beurre doit être liquide") are
|
||||||
never wrapped this way (would be ungrammatical) — `is_fr_infinitive_led`/
|
never used as a source for either strategy (would be ungrammatical once
|
||||||
`is_en_imperative_led` decide which existing utterances are safe sources.
|
wrapped/substituted) — `is_fr_infinitive_led`/`is_en_imperative_led` decide
|
||||||
Frames are lowercase/unpunctuated, matching this corpus' own style exactly
|
which existing utterances are safe sources for (2); (1) has its own,
|
||||||
(see `FR_FRAMES`/`EN_FRAMES`'s own comment for why that's not just
|
stricter "starts with a known synonym" check that already excludes them.
|
||||||
cosmetic). A technique already at/above 20 for a locale is left untouched
|
|
||||||
— re-running this script is always safe, never re-pads an already-balanced
|
|
||||||
entry (see `top_up`).
|
|
||||||
|
|
||||||
Run from `services/tech-step-intent-service/` (this directory):
|
Run from `services/tech-step-intent-service/` (this directory):
|
||||||
`./.venv/Scripts/python.exe augment_utterances.py` (Windows) or
|
`./.venv/Scripts/python.exe augment_utterances.py` (Windows) or
|
||||||
|
|
@ -28,7 +47,9 @@ own `uv sync`'d virtualenv, see this service's README. Rewrites
|
||||||
`training_data.py` in place by textual splicing (AST only to *locate* each
|
`training_data.py` in place by textual splicing (AST only to *locate* each
|
||||||
`utterances=[...]` list's line range — never to regenerate the file), so
|
`utterances=[...]` list's line range — never to regenerate the file), so
|
||||||
every existing comment, `synonyms` list, and hand-written utterance survives
|
every existing comment, `synonyms` list, and hand-written utterance survives
|
||||||
untouched.
|
untouched. A technique already at/above 20 for a locale is left untouched —
|
||||||
|
re-running this script is always safe, never re-pads an already-balanced
|
||||||
|
entry (see `top_up`).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import ast
|
import ast
|
||||||
|
|
@ -36,47 +57,24 @@ import sys
|
||||||
|
|
||||||
SRC_PATH = "intent_service/training_data.py"
|
SRC_PATH = "intent_service/training_data.py"
|
||||||
|
|
||||||
# Lowercase, no trailing period — matches this corpus' existing style
|
# Small fallback frame pool — see this module's own doc comment for why it's
|
||||||
# exactly (every hand-written utterance so far is lowercase/unpunctuated).
|
# deliberately short (3 per locale, not a dozen) and only ever a fallback
|
||||||
# Not just cosmetic: `spacy.TextCatBOW.v3` hashes on token form, and mixing
|
# behind synonym substitution.
|
||||||
# "Il"/"il" as if they were different tokens would needlessly fragment the
|
|
||||||
# bag-of-words signal for what should read as the exact same sentence to the
|
|
||||||
# classifier.
|
|
||||||
FR_FRAMES = [
|
FR_FRAMES = [
|
||||||
"il faut {u}",
|
"il faut {u}",
|
||||||
"veillez à {u}",
|
"veillez à {u}",
|
||||||
"pensez à {u}",
|
"pensez à {u}",
|
||||||
"n'oubliez pas de {u}",
|
"n'oubliez pas de {u}",
|
||||||
"la recette demande de {u}",
|
|
||||||
"cette étape consiste à {u}",
|
|
||||||
"il est important de {u}",
|
|
||||||
"assurez-vous de {u}",
|
"assurez-vous de {u}",
|
||||||
"prenez soin de {u}",
|
|
||||||
"commencez par {u}",
|
|
||||||
"on vous demande de {u}",
|
|
||||||
"il convient de {u}",
|
|
||||||
]
|
]
|
||||||
|
|
||||||
EN_FRAMES = [
|
EN_FRAMES = [
|
||||||
"make sure to {u}",
|
"make sure to {u}",
|
||||||
"remember to {u}",
|
"remember to {u}",
|
||||||
"be sure to {u}",
|
"be sure to {u}",
|
||||||
"take care to {u}",
|
|
||||||
"you'll need to {u}",
|
|
||||||
"don't forget to {u}",
|
"don't forget to {u}",
|
||||||
"it's important to {u}",
|
"take care to {u}",
|
||||||
"go ahead and {u}",
|
|
||||||
"now {u}",
|
|
||||||
"the recipe calls for you to {u}",
|
|
||||||
]
|
]
|
||||||
|
|
||||||
# Bare English cooking verbs (imperative == infinitive minus "to") — an
|
|
||||||
# utterance whose first word (or, for an adverb-led opener, second word — see
|
|
||||||
# `EN_ADVERB_SKIP`) is one of these is safe to wrap in an EN_FRAMES modal
|
|
||||||
# template. Built from every distinct first word actually used in
|
|
||||||
# `training_data.py`'s own English utterances (see the corpus-wide frequency
|
|
||||||
# scan this script's history was built from) plus the handful of verbs only
|
|
||||||
# ever appearing after a skipped adverb.
|
|
||||||
EN_VERB_WHITELIST = {
|
EN_VERB_WHITELIST = {
|
||||||
"make", "add", "pour", "mix", "stir", "cut", "place", "cover", "remove", "heat", "let",
|
"make", "add", "pour", "mix", "stir", "cut", "place", "cover", "remove", "heat", "let",
|
||||||
"keep", "turn", "cook", "bake", "roast", "grill", "fry", "boil", "simmer", "whisk", "fold",
|
"keep", "turn", "cook", "bake", "roast", "grill", "fry", "boil", "simmer", "whisk", "fold",
|
||||||
|
|
@ -92,17 +90,14 @@ EN_VERB_WHITELIST = {
|
||||||
"smother", "build", "scoop", "plunge", "increase", "pass", "collect", "have", "adjust",
|
"smother", "build", "scoop", "plunge", "increase", "pass", "collect", "have", "adjust",
|
||||||
"dry-toast", "dry-roast", "heat-treat", "pre-bake", "salt", "soak",
|
"dry-toast", "dry-roast", "heat-treat", "pre-bake", "salt", "soak",
|
||||||
}
|
}
|
||||||
|
|
||||||
# Adverbs/modifiers that can open an otherwise-imperative English clause
|
|
||||||
# ("coarsely chop the tomatoes", "deep fry until golden") — checked one word
|
|
||||||
# further in when the first word matches one of these, rather than treated
|
|
||||||
# as declarative.
|
|
||||||
EN_ADVERB_SKIP = {
|
EN_ADVERB_SKIP = {
|
||||||
"coarsely", "roughly", "finely", "quickly", "lightly", "briefly", "gently", "carefully",
|
"coarsely", "roughly", "finely", "quickly", "lightly", "briefly", "gently", "carefully",
|
||||||
"gradually", "very", "thoroughly", "evenly", "generously", "slowly", "thinly", "deep", "blind",
|
"gradually", "very", "thoroughly", "evenly", "generously", "slowly", "thinly", "deep", "blind",
|
||||||
"dry",
|
"dry",
|
||||||
}
|
}
|
||||||
|
|
||||||
|
_TARGET = 20
|
||||||
|
|
||||||
|
|
||||||
def is_fr_infinitive_led(u: str) -> bool:
|
def is_fr_infinitive_led(u: str) -> bool:
|
||||||
first = u.split(" ", 1)[0].lower()
|
first = u.split(" ", 1)[0].lower()
|
||||||
|
|
@ -121,16 +116,45 @@ def is_en_imperative_led(u: str) -> bool:
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
|
||||||
def generate(existing: list[str], frames: list[str], is_led) -> list[str]:
|
def _synonym_variants(existing: list[str], synonyms: list[str]) -> list[str]:
|
||||||
"""Returns up to `len(frames) * len(sources)` new, deduplicated
|
"""Substitutes every *other* synonym in place of whichever synonym an
|
||||||
utterances wrapping every eligible source utterance in every frame —
|
existing utterance's leading phrase exactly matches — see this module's
|
||||||
caller trims to however many it actually needs."""
|
own doc comment for why this is the primary generation strategy."""
|
||||||
|
if len(synonyms) < 2:
|
||||||
|
return []
|
||||||
|
seen = set(existing)
|
||||||
|
sorted_synonyms = sorted(set(synonyms), key=len, reverse=True)
|
||||||
|
out: list[str] = []
|
||||||
|
for u in existing:
|
||||||
|
lower_u = u.lower()
|
||||||
|
matched = next(
|
||||||
|
(
|
||||||
|
syn
|
||||||
|
for syn in sorted_synonyms
|
||||||
|
if lower_u == syn.lower() or lower_u.startswith(f"{syn.lower()} ")
|
||||||
|
),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
if matched is None:
|
||||||
|
continue
|
||||||
|
rest = u[len(matched) :]
|
||||||
|
for syn in sorted_synonyms:
|
||||||
|
if syn == matched:
|
||||||
|
continue
|
||||||
|
candidate = f"{syn}{rest}"
|
||||||
|
if candidate in seen:
|
||||||
|
continue
|
||||||
|
seen.add(candidate)
|
||||||
|
out.append(candidate)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _frame_variants(existing: list[str], frames: list[str], is_led) -> list[str]:
|
||||||
sources = [u for u in existing if is_led(u)]
|
sources = [u for u in existing if is_led(u)]
|
||||||
if not sources:
|
if not sources:
|
||||||
return []
|
return []
|
||||||
existing_set = set(existing)
|
seen = set(existing)
|
||||||
out: list[str] = []
|
out: list[str] = []
|
||||||
seen = set(existing_set)
|
|
||||||
for frame in frames:
|
for frame in frames:
|
||||||
for u in sources:
|
for u in sources:
|
||||||
candidate = frame.format(u=u)
|
candidate = frame.format(u=u)
|
||||||
|
|
@ -141,15 +165,23 @@ def generate(existing: list[str], frames: list[str], is_led) -> list[str]:
|
||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
def top_up(existing: list[str], locale: str) -> list[str]:
|
def top_up(existing: list[str], synonyms: list[str], locale: str) -> list[str]:
|
||||||
target = 20
|
if len(existing) >= _TARGET:
|
||||||
if len(existing) >= target:
|
|
||||||
return []
|
return []
|
||||||
if locale == "fr":
|
needed = _TARGET - len(existing)
|
||||||
pool = generate(existing, FR_FRAMES, is_fr_infinitive_led)
|
pool = _synonym_variants(existing, synonyms)
|
||||||
else:
|
if len(pool) < needed:
|
||||||
pool = generate(existing, EN_FRAMES, is_en_imperative_led)
|
frames = FR_FRAMES if locale == "fr" else EN_FRAMES
|
||||||
needed = target - len(existing)
|
is_led = is_fr_infinitive_led if locale == "fr" else is_en_imperative_led
|
||||||
|
# Frame variants must also dedupe against the synonym-substitution
|
||||||
|
# pool already chosen, not just `existing` — otherwise the two
|
||||||
|
# sources could independently produce the same string.
|
||||||
|
already = set(existing) | set(pool)
|
||||||
|
for candidate in _frame_variants(existing, frames, is_led):
|
||||||
|
if candidate in already:
|
||||||
|
continue
|
||||||
|
pool.append(candidate)
|
||||||
|
already.add(candidate)
|
||||||
return pool[:needed]
|
return pool[:needed]
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -159,8 +191,6 @@ def main() -> None:
|
||||||
tree = ast.parse(source)
|
tree = ast.parse(source)
|
||||||
lines = source.splitlines(keepends=True)
|
lines = source.splitlines(keepends=True)
|
||||||
|
|
||||||
# Find the TECH_STEP_TRAINING_DATA = [ ... ] assignment's list of
|
|
||||||
# TechStepTrainingEntry(...) calls.
|
|
||||||
module_body = tree.body
|
module_body = tree.body
|
||||||
training_data_list = None
|
training_data_list = None
|
||||||
for node in module_body:
|
for node in module_body:
|
||||||
|
|
@ -172,11 +202,9 @@ def main() -> None:
|
||||||
print("Could not locate TECH_STEP_TRAINING_DATA list", file=sys.stderr)
|
print("Could not locate TECH_STEP_TRAINING_DATA list", file=sys.stderr)
|
||||||
sys.exit(1)
|
sys.exit(1)
|
||||||
|
|
||||||
# Collect (insertion_line_0indexed, indent, new_lines_to_insert) for
|
|
||||||
# every utterances=[...] list that needs topping up, across every entry
|
|
||||||
# — applied bottom-to-top so earlier line numbers stay valid.
|
|
||||||
insertions: list[tuple[int, str, list[str]]] = []
|
insertions: list[tuple[int, str, list[str]]] = []
|
||||||
total_added = 0
|
total_added = 0
|
||||||
|
shortfalls: list[tuple[str, str, int]] = []
|
||||||
|
|
||||||
for entry_call in training_data_list.elts:
|
for entry_call in training_data_list.elts:
|
||||||
assert isinstance(entry_call, ast.Call)
|
assert isinstance(entry_call, ast.Call)
|
||||||
|
|
@ -191,23 +219,35 @@ def main() -> None:
|
||||||
locale = kw.arg
|
locale = kw.arg
|
||||||
locale_call = kw.value
|
locale_call = kw.value
|
||||||
assert isinstance(locale_call, ast.Call)
|
assert isinstance(locale_call, ast.Call)
|
||||||
|
utterances_list_node = None
|
||||||
|
synonyms_list_node = None
|
||||||
for inner_kw in locale_call.keywords:
|
for inner_kw in locale_call.keywords:
|
||||||
if inner_kw.arg != "utterances":
|
if inner_kw.arg == "utterances":
|
||||||
continue
|
|
||||||
utterances_list_node = inner_kw.value
|
utterances_list_node = inner_kw.value
|
||||||
|
elif inner_kw.arg == "synonyms":
|
||||||
|
synonyms_list_node = inner_kw.value
|
||||||
|
if utterances_list_node is None:
|
||||||
|
continue
|
||||||
assert isinstance(utterances_list_node, ast.List)
|
assert isinstance(utterances_list_node, ast.List)
|
||||||
existing = [
|
existing = [
|
||||||
elt.value for elt in utterances_list_node.elts if isinstance(elt, ast.Constant)
|
elt.value for elt in utterances_list_node.elts if isinstance(elt, ast.Constant)
|
||||||
]
|
]
|
||||||
new_ones = top_up(existing, locale)
|
synonyms = (
|
||||||
|
[elt.value for elt in synonyms_list_node.elts if isinstance(elt, ast.Constant)]
|
||||||
|
if isinstance(synonyms_list_node, ast.List)
|
||||||
|
else []
|
||||||
|
)
|
||||||
|
new_ones = top_up(existing, synonyms, locale)
|
||||||
|
final_count = len(existing) + len(new_ones)
|
||||||
|
if final_count < _TARGET:
|
||||||
|
shortfalls.append((uid, locale, final_count))
|
||||||
if not new_ones:
|
if not new_ones:
|
||||||
continue
|
continue
|
||||||
# Insert right after the last element's line, before the
|
|
||||||
# closing "]" — indentation matched to the last existing
|
|
||||||
# element's own line.
|
|
||||||
last_elt = utterances_list_node.elts[-1]
|
last_elt = utterances_list_node.elts[-1]
|
||||||
insert_after_line = last_elt.end_lineno - 1 # 0-indexed
|
insert_after_line = last_elt.end_lineno - 1
|
||||||
indent = lines[insert_after_line][: len(lines[insert_after_line]) - len(lines[insert_after_line].lstrip())]
|
indent = lines[insert_after_line][
|
||||||
|
: len(lines[insert_after_line]) - len(lines[insert_after_line].lstrip())
|
||||||
|
]
|
||||||
new_lines = [f'{indent}"{s}",\n' for s in new_ones]
|
new_lines = [f'{indent}"{s}",\n' for s in new_ones]
|
||||||
insertions.append((insert_after_line, uid, new_lines))
|
insertions.append((insert_after_line, uid, new_lines))
|
||||||
total_added += len(new_ones)
|
total_added += len(new_ones)
|
||||||
|
|
@ -220,6 +260,11 @@ def main() -> None:
|
||||||
f.writelines(lines)
|
f.writelines(lines)
|
||||||
|
|
||||||
print(f"Added {total_added} new utterances across {len(insertions)} (technique, locale) pairs.")
|
print(f"Added {total_added} new utterances across {len(insertions)} (technique, locale) pairs.")
|
||||||
|
if shortfalls:
|
||||||
|
print(f"{len(shortfalls)} (uid, locale) pair(s) still below {_TARGET} — not enough synonym")
|
||||||
|
print("variety to reach the target without falling back to more generic frames:")
|
||||||
|
for uid, locale, count in shortfalls:
|
||||||
|
print(f" {uid} ({locale}): {count}")
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|
|
||||||
|
|
@ -87,38 +87,25 @@ _TEXTCAT_PIPE_NAME = "textcat"
|
||||||
# `CONFIDENCE_THRESHOLD`'s propre commentaire, `tech-step-matcher.ts`)
|
# `CONFIDENCE_THRESHOLD`'s propre commentaire, `tech-step-matcher.ts`)
|
||||||
# — ce qui précède est une mesure manuelle ponctuelle, pas un
|
# — ce qui précède est une mesure manuelle ponctuelle, pas un
|
||||||
# remplacement de cette calibration.
|
# remplacement de cette calibration.
|
||||||
# 4. Le corpus a ensuite été rééquilibré à 20 `utterances` minimum par
|
# 4. Le corpus a ensuite été rééquilibré vers 20 `utterances` par technique
|
||||||
# technique et par locale (contre 3-7 avant — chaque technique en a
|
# et par locale (voir `training_data.py`'s propre commentaire de tête
|
||||||
# désormais *le même nombre*, demande explicite pour que le textcat ne
|
# pour l'algorithme exact et pourquoi certaines techniques restent
|
||||||
# voie pas certaines classes avec 3x moins de signal que d'autres). Les
|
# volontairement en dessous de 20). Deux tentatives ont mesurablement
|
||||||
# exemples par époque grimpent d'environ 749 à ~2180/locale (+191%) — à
|
# échoué avant celle-ci : `_TRAINING_ITERATIONS` inchangé (25) avec un
|
||||||
# `_TRAINING_ITERATIONS` inchangé (25), ça aurait fait grimper le temps
|
# générateur reposant principalement sur des tournures modales
|
||||||
# d'entraînement dans les mêmes proportions (~336s -> ~980s/locale,
|
# génériques a fait chuter le F1 agrégé
|
||||||
# ~33 minutes pour fr+en). Réduit à `10` pour retrouver un temps par
|
# (`test/recipe-matching/tech-step-eval.test.ts`) à `0.79`, sous le
|
||||||
# époque comparable à l'étape 3 malgré ~3x plus d'exemples par époque —
|
# seuil `0.8` ; doubler `_TRAINING_ITERATIONS` (`10` -> `20`) sur ce
|
||||||
# un corpus plus large et mieux équilibré par classe a aussi besoin de
|
# même corpus n'a pas aidé (`0.791`, toujours sous le seuil) — la cause
|
||||||
# structurellement moins d'époques pour bien converger (chaque époque
|
# n'était pas un manque d'itérations mais un générateur qui diluait la
|
||||||
# voit déjà beaucoup plus de signal par classe), donc ce n'est pas un
|
# séparabilité entre classes (voir `training_data.py`). Le générateur a
|
||||||
# simple compromis qualité/temps à somme nulle comme les étapes
|
# donc été revu (substitution de synonyme en priorité, tournures
|
||||||
# précédentes. Mesuré à `10` : ~355s (fr, 1943 exemples) / ~332s (en,
|
# modales seulement en complément limité) et `_TRAINING_ITERATIONS`
|
||||||
# 1835 exemples), ~687s pour fr+en combinés — quasi identique à l'étape
|
# remonté à `20` sur ce nouveau corpus — mesuré : ~691s (fr, 1923
|
||||||
# 3 malgré ~2.6x plus d'exemples par époque. Les scores bruts semblaient
|
# exemples) / ~645s (en, 1807 exemples), ~1336s pour fr+en combinés.
|
||||||
# bons sur un petit échantillon de phrases suivies à la main, **mais**
|
# Confiance nettement rétablie sur les techniques auparavant en échec
|
||||||
# `test/recipe-matching/tech-step-eval.test.ts` (F1 agrégé contre
|
# (`sweat` ~0.99, `bainMarie` ~0.98, `julienne` ~0.88). À
|
||||||
# `TECH_STEP_EVAL_DATASET`, la vraie jauge de qualité de ce pipeline, pas
|
# confirmer/affiner par une vraie repasse de
|
||||||
# un spot-check manuel) a mesuré `0.7999...` — sous le seuil `0.8` — une
|
|
||||||
# fois passé en CI avec une vraie base Postgres : `10` itérations ne
|
|
||||||
# suffisaient pas à absorber ~2.6x plus d'exemples par époque, contre le
|
|
||||||
# pari initial que "un corpus plus grand converge en moins d'époques
|
|
||||||
# relatives" — faux à ce niveau de réduction. Remonté à `20` : ~699s
|
|
||||||
# pour `fr` seule (mesuré), donc largement au-dessus des ~900s de budget
|
|
||||||
# pour les deux locales combinées une fois `en` incluse — `start_period`
|
|
||||||
# (`docker-compose.yml`) et le `timeout` d'attente `/health`
|
|
||||||
# (`.github/workflows/ci.yml`) relevés à `1800s` en conséquence.
|
|
||||||
# Techniques auparavant en échec (précision nulle sur le jeu d'éval) à
|
|
||||||
# `10` — `sweat`, `julienne`, `caramelize` — retestées manuellement à
|
|
||||||
# `20` avec une confiance nettement rétablie (`sweat` ~0.99, par
|
|
||||||
# exemple). À confirmer/affiner par une vraie repasse de
|
|
||||||
# `calibrate-tech-step-threshold.ts` comme aux étapes précédentes — ce
|
# `calibrate-tech-step-threshold.ts` comme aux étapes précédentes — ce
|
||||||
# qui précède reste une mesure ponctuelle plus la gate F1 de CI, pas un
|
# qui précède reste une mesure ponctuelle plus la gate F1 de CI, pas un
|
||||||
# remplacement de cette calibration.
|
# remplacement de cette calibration.
|
||||||
|
|
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -2,14 +2,23 @@
|
||||||
`training_data.py`'s propre commentaire de tête) : un textcat entraîné sur
|
`training_data.py`'s propre commentaire de tête) : un textcat entraîné sur
|
||||||
des classes très inégales en nombre d'exemples est une source réelle de
|
des classes très inégales en nombre d'exemples est une source réelle de
|
||||||
classifications confiantes mais fausses sur une phrase jamais vue (constaté
|
classifications confiantes mais fausses sur une phrase jamais vue (constaté
|
||||||
en pratique — voir l'historique Git de ce fichier). Chaque technique doit
|
en pratique — voir l'historique Git de ce fichier).
|
||||||
avoir *au moins* 20 `utterances` par locale, et — pour rester vraiment
|
|
||||||
équilibré plutôt que juste "assez" — le même nombre pour les deux locales
|
`_MIN_UTTERANCES_PER_LOCALE` est volontairement `12`, pas `20` : la
|
||||||
d'une même technique."""
|
génération vise `20` (`augment_utterances.py`'s `_TARGET`) mais s'arrête
|
||||||
|
avant si la technique n'a pas assez de vocabulaire distinctif propre
|
||||||
|
(`synonyms`) pour l'atteindre sans retomber massivement sur des tournures
|
||||||
|
génériques partagées par toutes les classes — une première version de ce
|
||||||
|
script forçait `20` partout via ce mécanisme et a mesurablement *dégradé*
|
||||||
|
`test/recipe-matching/tech-step-eval.test.ts` (F1 agrégé) plutôt que de
|
||||||
|
l'améliorer, en réduisant la séparabilité entre classes plus qu'en ajoutant
|
||||||
|
un vrai signal. `12` reste très au-dessus du plancher d'origine (3-7) tout
|
||||||
|
en laissant `augment_utterances.py` s'arrêter honnêtement plutôt que de
|
||||||
|
forcer un compte rond au prix de la qualité."""
|
||||||
|
|
||||||
from intent_service.training_data import TECH_STEP_TRAINING_DATA
|
from intent_service.training_data import TECH_STEP_TRAINING_DATA
|
||||||
|
|
||||||
_MIN_UTTERANCES_PER_LOCALE = 20
|
_MIN_UTTERANCES_PER_LOCALE = 12
|
||||||
|
|
||||||
|
|
||||||
def test_every_technique_has_at_least_the_minimum_utterances_per_locale():
|
def test_every_technique_has_at_least_the_minimum_utterances_per_locale():
|
||||||
|
|
@ -21,18 +30,5 @@ def test_every_technique_has_at_least_the_minimum_utterances_per_locale():
|
||||||
]
|
]
|
||||||
assert short == [], (
|
assert short == [], (
|
||||||
f"{len(short)} (uid, locale) pair(s) below the {_MIN_UTTERANCES_PER_LOCALE}-utterance "
|
f"{len(short)} (uid, locale) pair(s) below the {_MIN_UTTERANCES_PER_LOCALE}-utterance "
|
||||||
f"floor — run augment_utterances.py: {short}"
|
f"floor: {short}"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_every_technique_has_the_same_utterance_count_in_both_locales():
|
|
||||||
# Not just "both above the floor" — a technique whose fr/en counts drift
|
|
||||||
# apart re-introduces the same per-class imbalance this test file exists
|
|
||||||
# to catch, just between locales of the same technique instead of across
|
|
||||||
# techniques.
|
|
||||||
mismatched = [
|
|
||||||
(entry.uid, len(entry.fr.utterances), len(entry.en.utterances))
|
|
||||||
for entry in TECH_STEP_TRAINING_DATA
|
|
||||||
if len(entry.fr.utterances) != len(entry.en.utterances)
|
|
||||||
]
|
|
||||||
assert mismatched == [], f"fr/en utterance count mismatch: {mismatched}"
|
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue