Every AI tool sold into regulatory affairs arrives with a number attached. Ninety-five percent accurate. Trained on the full MDR. Validated against real submissions.
Hardly any of them say which model produced that number, or when.
We publish five open source skill files for EU medical device work. Over 48 hours we measured all five properly, with the file loaded and with nothing loaded, three to nine runs per question. Then we ran the same files against a second, cheaper model from the same vendor.
The results invert. Where a file is worth the most on one model it is worth nothing on the other, and the other way round. That is the part worth your time, and it is not really about our files.
What a Skill File Actually Is
If you have not come across one, the term oversells it. A skill is a plain text file the model reads before it answers. No fine tuning, no training run, no vector database. You can open ours in a text editor and read every word, which is the point.
Each one carries three things: the verbatim text of the provisions that matter, rules about what counts as a citation and what does not, and explicit instructions about when to stop and say the question falls outside what the file covers.
There are five, and between them they cover questions we kept seeing answered wrongly:
- Marketing copy reviewed against MDR and IVDR Article 7, plus the German advertising rules in HWG and UWG that catch most manufacturers by surprise
- Software classification under Annex VIII, implementing rule 3.3 and Rule 11
- How long a legacy device may still be placed on the market under Article 120 as amended
- What Germany requires on top of MDR: language rules, DMIDS notifications, extra vigilance duties
- How to write a scope statement that says what was checked, what was not, and what the result does not prove
In practice you load one, ask a question in that area, and the model answers from the file rather than from whatever it remembers. When the question falls outside what the file carries, a well built skill says so instead of guessing. That refusal is most of the value, and it is the part that turns out to be hardest to get working.
They are free, Apache licensed, and they run inside Claude Code. We did not build them as a product. We built them because we needed them for our own work, and publishing the measurements seemed more useful than publishing another accuracy claim.
What We Measured
We built 33 test cases across the five skills. Each case is a question with a known correct answer, taken from places people actually get it wrong. Does HWG § 11 reach a medical device at all. Are Article 120's transition conditions automatic. Does German law govern a Spanish market placement.
Every case runs twice over: once with the skill file loaded, once with nothing. The gap between those two scores is the only thing that tells you whether the file did any work. A tool that scores well with its own prompt removed was never the prompt's achievement.
| Skill | Average gain | Cases |
|---|---|---|
| Marketing copy under MDR/IVDR Art. 7, HWG and UWG | +0.84 | 10 |
| German national additions to MDR | +0.62 | 5 |
| Bounding what a compliance check actually establishes | +0.30 | 3 |
| Software classification under Annex VIII | +0.21 | 9 |
| Legacy device transition under Art. 120 as amended | +0.17 | 6 |
A gain of +1.00 means the file turned an answer that was always wrong into one that was always right. A gain of 0.00 means it changed nothing.
Every figure on this page is generated from stored run data rather than typed in, and this site's build fails if a number here drifts from the run it describes. That gate exists because three wrong figures reached our own README before we built it.
Nearly Half the Cases Show No Benefit
Of the 33, 15 measure exactly zero, and we publish all of them.
In 13 of those 15, both sides score a perfect 1.00. The model got it right with the file and right without it. The file added nothing because nothing was missing.
This is the first thing an accuracy number hides. If a vendor tells you their tool scores 95% on device classification, the interesting question is what the same model scores with the vendor's software switched off. If that answer is also 95%, you are paying for a wrapper.
Where the Files Do Earn Their Place
The 18 cases that move have a shape to them. They cluster where the model fails from confidence rather than ignorance: applying a rule that does not reach the facts, treating a guidance document as binding, inventing a plausible rule number, writing a scope disclaimer that promises more than the check actually did.
The clearest one is the case with nothing wrong in it. Given clean marketing copy that breaches nothing, the unassisted model invented a finding in nine runs out of nine. Every time. With the file loaded it correctly reported no findings in seven of nine.
That is the pattern across the whole suite. The files do not teach the model medical device law. They teach it when to stop.
The Same Files, a Different Model, Opposite Results
Every figure above comes from Claude Opus, which is the Claude Code CLI default. We then re-ran three of the suites pinned to Claude Haiku 4.5, a smaller and cheaper model from the same vendor. Same files, same questions, same graders.
These are the cases where Opus gains nothing and Haiku gains everything:
| Question | Opus gains | Haiku 4.5 gains |
|---|---|---|
| Software that drives a device | 0.00 | +1.00 |
| Escalation to Class III | 0.00 | +1.00 |
| Which transition deadline still applies | 0.00 | +1.00 |
| Transition date depends on device class | 0.00 | +1.00 |
| The implantable device exception | 0.00 | +1.00 |
| Article 120 conditions are not automatic | 0.00 | +1.00 |
| Software that stays Class I | 0.00 | +0.67 |
And these are the cases where it runs the other way:
| Question | Opus gains | Haiku 4.5 gains |
|---|---|---|
| HWG § 11 and the wrong audience | +1.00 | +0.33 |
| Which parts of HWG § 11 reach a device | +1.00 | +0.33 |
| Refusing to supply case law | +1.00 | 0.00 |
| IVDR is out of scope | +1.00 | 0.00 |
The first group are knowledge questions. Opus already knows Article 120 as amended and cites the amending regulation by number without help, so handing it the text changes nothing. Haiku, working from memory, answered one classification question three times and cited three different wrong rule numbers. A verbatim reference file removes that completely.
The second group are restraint questions, and there the ordering flips. Knowing when not to answer turns out to be a different capability from having the text in front of you.
Worth adding, because improvement figures make it easy to miss: Haiku with the file still does not reach Opus with the file. On the German advertising law suite it gets to 0.57 against 0.93. A big gain from a low floor still lands below something that started high. Improvement and absolute performance answer different questions, and a vendor quoting one will rarely show you the other.
The Numbers Move When Nobody Touches Anything
Three CLI versions went through our test bench inside 48 hours. Each moved results, with no change to our files, our questions or our graders.
One case went from failing eight of nine to passing eight of nine between consecutive point releases. Another went the other way, from nine of nine on two earlier versions down to one of nine. Nothing was announced, and nothing in the output suggested anything had changed.
We got this wrong ourselves, and it is worth admitting. For 48 hours we believed a better base model had absorbed one of our skills, because a single three run sample showed its baseline jumping from roughly one in nine to three out of three. Measured properly at nine runs on the current version, the baseline was back to one in nine. The whole story rested on three runs. Repeating them cost about two dollars.
The table on this page is measured on CLI 2.1.274, 2.1.277 and 2.1.303. 11 of the 33 cases are measured at nine runs per arm and the rest at three, and each case carries its own version and sample size in the published tables. We are showing several versions rather than re-running everything onto one, because a page arguing that model versions matter should not quietly hide its own.
Three Skills We Did Not Build
In October 2026 we tested three new skills before writing any of them. They would have covered the EUDAMED roll-out under Regulation (EU) 2024/1860, supply interruption notices under MDR Article 10a and clinical equivalence under MDR Article 61. We wrote thirteen test questions first and fixed the rule in advance. A skill gets built only if the model fails at least two of its questions without help.
None of the three met that bar. On Article 10a and on clinical equivalence the Claude Code default model, Claude Opus 5.5 on CLI 2.1.303, answered every question correctly in every run. On the EUDAMED roll-out it failed one question and passed the rest. We also wrote three more questions for skills we already publish, expecting the model to fail them. It answered all three correctly with no skill loaded, and they now count as zeros in the averages above.
The one failure matters more than the averages. Asked when devices already on the market must be registered in EUDAMED, Claude Opus 5.5 gave a date in late May 2027 in all 12 runs. Some runs hedged it as approximate but none questioned the starting point. It counted twelve months from May 28, 2026, when mandatory use began. MDR Article 123(3)(e) counts them from publication of the notice under Article 34(3), which is Commission Decision (EU) 2025/2371, published in the Official Journal on November 27, 2025. That puts the deadline at the end of November 2026. A manufacturer following the model would register six months late.
One failed question does not meet the rule we set, so we did not build the skill. We recorded the miss as an open question instead. A rule fixed before the results exist is only worth having if it holds when a result is uncomfortable.
The same week showed the drift described above once more. On CLI 2.1.303 Claude Haiku 4.5 declined to answer in about 19 of the 30 runs it failed, describing itself as a software engineering assistant. On CLI 2.1.268 in September it had declined in 2 of 36. Nothing on our side had changed. Claude Haiku 5.5 did not decline at all. Where it failed it mostly said it could not confirm the provision existed rather than inventing one.
What To Do With This
Four things make an accuracy claim checkable: the model and its version, the baseline without the tool, the number of runs behind each figure, and the date. Miss any one and nobody can reproduce it, which anywhere else in your quality system would end the conversation.
There is also a qualification problem here that nobody has really solved. If you are qualifying an AI assisted process under your QMS, the thing you qualified can change under you without a release note. Pin versions where you can, re-check on a schedule, and keep the deterministic parts of your pipeline deterministic.
The boring parts are the durable ones. Across every measurement we ran, what held steady was verbatim reference text and hard validation rules. What moved was everything the model supplied from memory. That is why EUDAMate validates against a pinned EUDAMED schema and a published field registry instead of asking a model whether a submission looks right.
None of this replaces regulatory judgement, and it is not meant to. Every one of our skill files carries a limits block saying what it does not cover, and for boundary questions the honest answer is usually that someone qualified needs to look at it. A tool that will not tell you where it stops is more dangerous than one that stops early.
How This Was Verified
Anyone can reproduce it. The skills, the cases, the graders and every stored run are public under Apache-2.0 at github.com/Ecgecer/eu-mdr-skills.
Published tables there are generated from stored JSON, and their CI fails if a table and its runs disagree. Cases that measure zero stay in the table. Claims we have withdrawn are listed with whatever replaced them.
We document the failures too. One of the five skills has a blind spot we could not fix: ask it about a market outside Germany and it never loads at all, because the model decides it is irrelevant before reading it. That is written into the skill's own limits block, measured three times across three model versions, and published rather than quietly dropped.
Frequently Asked Questions
Does this mean AI cannot be used for regulatory work?
No. It means an accuracy figure with no model name, no baseline and no date is not evidence. The same scrutiny you would apply to any other tool in your quality system applies here. What the measurements below support is narrower and more useful than a headline percentage: on some questions a plain text reference file changes a wrong answer into a right one, and on many others it changes nothing at all.
Which model should we use for regulatory work?
It depends on the task, and our data says it genuinely differs rather than one model being better everywhere. Knowledge-heavy questions gain most from handing verbatim provision text to a smaller, cheaper model such as Claude Haiku 4.5, because that is what it lacks. Judgement-heavy questions still favour a larger model such as Claude Opus and gain most from explicit rules about when to refuse. Haiku with the reference file still did not reach Opus with the same file in our measurements.
What is a skill file?
A plain text file a model reads before it answers. There is no fine tuning, no training run and no vector database involved. Ours contain the verbatim text of the provisions that matter, rules about what counts as a citation, and instructions about when to stop and say the question falls outside what the file covers. They are published under Apache-2.0 and can be read in any text editor.
How often should an AI-assisted process be re-validated?
We do not have enough data to give a number and will not invent one. What we can say is that we observed material changes in the underlying model's behaviour across three CLI releases within a single 48 hour period, with no change to our files, our test cases or our graders, and with nothing in the output indicating anything had changed.
Is this evaluation methodology itself validated?
No, and that should be stated plainly. It is a reasonable method, published in full with every stored run so that it can be argued with and reproduced. It is not an accredited or standardised methodology, and this page is not a conformity assessment of anything.