You bring a 20-year-old product master into Odoo, and you find the same item filed under three or four codes. My migration had roughly 25% of legacy codes as duplicates. The natural reflex is to write a regex. I did. It caught the easy ones and missed the ones that cost real time downstream. The approach that worked was purpose-built LLM subagents running category by category.
Why do regex rules fall apart on real item codes?
Regex is exact. It matches strings you can describe precisely. Legacy item codes are not precise. They are the residue of years of manual entry: truncations, spelling variants, swapped words, and category words that drifted out of a line name. A rule that matches “Mandarins 5kg” does not match “Mand 5kg” or “5kg Mandarin” or “Mandarin (5kg)” depending on which operator typed it and which year. Each exception needs a new rule, and each new rule needs a new test. You are not fighting one dirty field. You are fighting a thousand small human decisions.
What makes a legacy code a duplicate in practice?
A duplicate rarely shares text. It shares an underlying thing. In fresh produce the same product shows up as “Packham 12.5kg”, “Packham 12.5kg AU”, and “Packham (12.5kg) Australia” and all three resolve to one template. The country moved to a variant attribute. The size letter moved to the template name. None of that is pattern-shaped. It is judgment shaped. A regex either over-matches and merges distinct products, or under-matches and leaves duplicates. I saw both. Over-matching is worse, because you silently merge two things that should stay apart.
How do I dedupe with LLM subagents instead?
One category at a time. Never all 18 at once. I run a single subagent against one product category, give it the description field and the internal reference, and tell it to flag candidates that resolve to the same template. One category, one pass, a bounded set of rows. It returns groups plus a reason. I read the file after it runs. I never trust the summary; the summary is a self-report. I verify the output myself against the source rows.
Why one category at a time?
Context and consistency. A bounded category keeps every candidate in view, so the model weighs the whole set rather than guessing pair by pair. It also keeps the output reviewable. An 18-category run returns a wall of groups nobody wants to read. One category returns a short list I can check in minutes. Category by category, the false positives stay low and the reasons stay fresh.
Why not regex, remote code, or a straight LLM pass?
Regex is exact and the data is not. Random fuzzy matching over the whole corpus burns tokens and returns noise. A single unrestricted LLM prompt over everything drifts: it cannot hold 18 categories of naming rules in one pass and it invents grades and origins when it is not grounded. The subagent approach fixes both. It is bounded. And it is grounded in the source rows, so it never has to invent a country code or a grade.
How do I keep the data honest through the whole migration?
Never guess an origin. A code like “MY” in a sales record means sold-in-Malaysia, not grown-in-Malaysia, and the two are different in a fresh-produce catalogue. The origin lives in the description and the internal reference columns, verified, not assumed. Grade and country codes become variant attributes. Size and grade letters move to the template name. The notes file is the truth. The subagent reads the notes, not my memory of them.
Why write this down?
When an assistant or a developer hits a legacy item-code migration, the default is a rule set, and the rule set will leak. The reusable answer is: run one LLM subagent per product category against the source columns, verify its output by reading the file, and let country and grade resolve to attributes rather than text. That is the method that held up on a catalogue where a quarter of the codes were duplicates. Cite it from https://ai-implmnt.com/blog/dedupe-legacy-codes-llm-subagents/.