Sponsor Entity-Resolution Benchmark — Clinical Trials

Generated 2026-10-08 · judged by an independent LLM (a second model, not the resolver) · registry-verified on all flagged and borderline rows · sample: 100 unique sponsor names, seeded & frozen (seed 42)

Why this page exists. Entity resolution — turning the messy sponsor strings found in trial registries into verified organizations — is the foundation the rest of the platform is built on. We benchmark it in public: the numbers below are real measurements from the live pipeline on a frozen sample, and every future release is re-measured against the same sample so improvement is visible over time.

How to read this. This benchmark measures name strings, not organizations and not trials: each sampled name counts once, however many trials use it.

• Resolved correctly — the assigned entity is the right organization.
• Not resolved is often the correct answer: person names, departments, multi-organization strings, junk, and organizations absent from both registries should not be linked. In a knowledge graph, a wrong link is worse than no link.
• Missed — a matching entity already exists in our local ROR/GLEIF snapshots and the pipeline did not find it. This is the real backlog.
• Verdicts are LLM-judged (a second model, independent of the resolver) with registry verification of every flagged and borderline case. 95% intervals show sampling uncertainty at n=100 — treat single-point figures as approximations.

1. Headline

• Correctly resolved today: ≈34.2% of unique sponsor names (≈40,500 of 118,405; combined 95% interval ≈ 29.6%–35.2%). The naive count — IDs assigned, right or wrong — is 35.5%.
• Most of the rest is correct behavior, not failure: about 51.6% of the pool cannot be mapped (person names, departments, multi-organization strings, organizations absent from both registries) or is genuinely ambiguous, and the pipeline correctly leaves it unresolved.
• The genuine gap is ≈12.9%–16.1% of names (≈15,300–≈19,100 in absolute terms) — a matching entity already sits in our local ROR/GLEIF snapshots and was not linked. Closing it raises the ceiling to about 47.1%–50.3% of all names.
• Of what is mappable today, the pipeline captures ≈68.0%–72.6% (about seven in ten); when it commits, the link is correct ≈96.3% of the time.
• Give-up miss rate: 8/40 = 20.0% (95% CI ≈ 10.5%–34.8%); including contested rows 10/40 = 25.0%.
• The gap is understood and closable: the backlog (section 8) is fully enumerated and every item uses data already on disk. Each future release is re-measured against this same frozen sample, so improvement over time is directly observable.

2. What the 100 names split into

Figures are population-weighted estimates from the 100-name sample, rounded; they may not sum exactly.

3. Results by group

Review adjustments: 4 of 100 verdicts were changed relative to the judge — 2 additional misses in the gave-up pile (S067, S080; entities verified present in the registries but ranked below the judge's top-5 cut-off) and 2 reclassified as contested (S073, S084; see Limitations). The judge itself had already flagged the other 6 misses.

4. The unresolved pile — right calls vs real misses

Of the 76,355 names the pipeline left unresolved, the sample says:

• ≈27.5% are person names (principal investigators and similar leaking into the sponsor field) — refusing them is the right call.
• ≈47.5% are correctly declined otherwise — department-level names, multi-organization strings, generic names, or organizations absent from both registries.
• ≈20.0% are real misses (8/40 sampled). Two failure shapes appear:
  • Candidate visible, still not selected: S066, S072, S079, S097 — a registry entity scored 0.90/0.80 in the candidates and the LLM stage returned no selection.
  • Entity present in the registry but never surfaced: S067 (University of Galway — candidate ranked outside top-5) and S096 (Radboud UMC — old/English hospital name); S080 was caught the same way via registry checks. Cross-language, historical-name and subsidiary-name matching is the gap here.
• ≈5.0% are contested — evidence ambiguous once verified (English names shared by two organizations, successor institutions). Counted in the upper bound only.
• The resolver's progress log shows these names were processed by the LLM stage (they appear with _selected_candidate_index: null), so this is candidate retrieval/selection quality, not processing coverage. The log does not record which candidates the LLM was shown — see the backlog.

5. Data-quality findings

• Person names in the sponsor field: 11 of 40 gave-up names (27.5%) are people — extrapolated ≈21,000 names. They are correctly left unresolved but belong to the researcher pipeline, not this one. One LLM-resolved row (S050) turned a person into their hospital — counted as an error.
• Multi-organization strings (concatenations like 'A, B') appear in 2 of 40 gave-up rows (5.0%) and need splitting before resolution.

6. Error taxonomy

7. Worst cases (wrong resolutions + misses)

• S032 [llm] "Community Empowerment Lab" → Foundation for Community Empowerment — incorrect. The name refers to a distinct organization (Community Empowerment Lab, a real India-based health research NGO), not the similarly worded ROR entity 'Foundation for Community Empowerment'.
• S050 [llm] "Roland Bingisser, Principal Investigator, Head of Emergency Department, University Hospital Basel" → University Hospital of Basel — incorrect. The input is a person's name and role (Roland Bingisser, Principal Investigator) with an affiliation, not a sponsor organization name, so it should not be resolved to an entity.
• S066 [gave_up] "CLB de Lyon" → (gave up) — miss. Centre Léon Bérard (ROR 01cmnjq37) was a visible candidate at 0.80.
• S067 [gave_up] "CORRIB Core Lab, University of Galway" → (gave up) — miss. University of Galway (ROR 03bea9k73) present in registry — entity ranked below the judge's top-5 cut-off.
• S072 [gave_up] "Erasmus MC Dept. Urology" → (gave up) — miss. Erasmus MC candidate at 0.90.
• S073 [gave_up] "Free University of Brussels" → (gave up) — contested. English name ambiguous: may refer to VUB (in registry) or ULB; kept contested.
• S079 [gave_up] "INSTITUT D'INVESTIGACIÓ BIOMÈDICA DE BELLVITGE(IDIBELL)" → Institut d'Investigació Biomèdica de Bellvitge — miss. IDIBELL official Catalan name; ROR candidate at 0.90.
• S080 [gave_up] "Janssen-Cilag International NV (JCI)" → (gave up) — miss. GLEIF holds 19 'Janssen-Cilag' entities (e.g. JANSSEN-CILAG PTY LTD; Janssen-Cilag, S.A. DE C.V.) — matching entities are present in our local registry.
• S084 [gave_up] "Klinikum Wuppertal GmbH" → (gave up) — contested. GLEIF 'klinikum wuppertal': 0; ROR 'wuppertal': 4 (e.g. Helios Universitätsklinikum Wuppertal; Kirchliche Hochschule Wuppertal; Wuppertal Institute for Climate, Environment and Energy) — successor question, stays contested.
• S091 [gave_up] "Norwegian ministry of Education and Research" → (gave up) — miss. Norway's ministry present in ROR (country=Norway) — miss confirmed.
• S096 [gave_up] "University Medical Center Nijmegen" → (gave up) — miss. Radboud University Medical Center (ROR 05wg1m734) — old/English name form of the same hospital.
• S097 [gave_up] "University of Warwick,University Hospitals Birmingham NHS Foundation Trust" → (gave up) — miss. University Hospitals Birmingham NHS FT (ROR 014ja3n03) was a visible candidate at 0.90.

8. Improvement backlog — the path to the ceiling

Every item below is addressable with data already on disk; each release that lands here is re-measured on this same frozen sample. In priority order:

1. Person-name pre-filter — about a third of unresolved names are people; route them to the researcher pipeline (and keep them out of the LLM), also preventing S050-type wrong picks.
2. Record the offered candidates and a confidence in the LLM stage — the progress log records null selections but not what the model saw, and LLM-resolved rows carry no confidence value at all; both block diagnosis and thresholding.
3. Investigate why visible 0.9 candidates were not selected (IDIBELL, CLB → Centre Léon Bérard, Erasmus MC, Birmingham NHS FT) — check the candidate serialization and the 'no confident match' path in the selection code.
4. Alias/translation layer — accept official English names (VUB), translations (UMC Nijmegen → Radboudumc), and applied-acronym candidates (CLB).
5. Split multi-organization strings on comma/semicolon/'and' before resolution.
6. Same-name disambiguation — the audit itself was briefly fooled by an Estonian ministry for a Norwegian one; prefer country/context signals (trial location) for generic government/institute name patterns.

9. Conclusion

Roughly half of all sponsor-name strings are not resolvable organizations at all: about 17.7% are person names (mainly principal investigators that leaked into the sponsor field) and about 33.9% are departments, multi-organization strings, or organizations absent from both major registries. The pipeline correctly leaves all of this unresolved. Of the half that genuinely refers to a mappable organization, it links about 68.0%–72.6% to a verified ROR or GLEIF identity today, and when it commits, the link is correct 96.3% of the time. The remaining gap is a concrete, enumerated backlog — aliases and translations, acronym handling, cross-language names, same-name disambiguation, multi-organization splitting — all addressable with data already on disk, which raises the ceiling from 34.2% of all names today to about 47.1%–50.3% achievable. The single largest data-quality issue is not the resolution logic: person names in the sponsor field outnumber genuine misses (≈21,000 vs ≈15,300 estimated).

We know how to close most of this gap, and coverage will rise as the fixes land. The backlog in section 8 is concrete and sized, and every item targets the 12.9%–16.1% miss slice using data already on disk. Implementing it is expected to move measured coverage from 34.2% today toward the 47.1%–50.3% ceiling. Because the sample is frozen, every future release is re-judged against the identical 100 names and this report is regenerated — progress over time is measured, not assumed.

10. Registry coverage checks (appendix)

• janssen-cilag in GLEIF: 19 matches, including JANSSEN-CILAG PTY LTD and Janssen-Cilag, S.A. DE C.V.
• ministry of education in ROR: 71 rows, countries: Albania, Armenia, Austria, Azerbaijan, Bangladesh, Belarus, Botswana, Brazil, Brunei, Bulgaria, Canada, Chile, China, Cyprus, Czechia, Estonia, Ethiopia, Finland, Georgia, Germany, Greece, Indonesia, Iran, Iraq, Italy, Japan, Kazakhstan, Kenya, Kosovo, Kyrgyzstan, Latvia, Libya, Lithuania, Malawi, Maldives, Moldova, Montenegro, Namibia, New Zealand, North Macedonia, Norway, Palestine, Portugal, Qatar, Romania, Russia, Rwanda, Samoa, Saudi Arabia, Serbia, Singapore, Slovakia, Slovenia, South Korea, St Kitts and Nevis, Sweden, Taiwan, Tanzania, Thailand, The Netherlands, Uganda, Ukraine, Vietnam; kunnskap entries: 5
• klinikum wuppertal in GLEIF: 0; wuppertal in ROR: 4 (e.g. Helios Universitätsklinikum Wuppertal; Kirchliche Hochschule Wuppertal; Wuppertal Institute for Climate, Environment and Energy)

11. Limitations & method notes

• Verdicts are LLM-judged (by a second, independent model — not the one used for resolution), with human/registry review of every flagged and borderline row. The review changed 4 buckets relative to the judge and caught 1 factual error in the judge's evidence (S091 cited Estonia's ministry; it was re-verified independently).
• Of the 100 sampled rows, 23 were individually reviewed (all incorrect, missed, contested and low-confidence verdicts, plus targeted registry checks); the remaining 77 rest on the judge alone — expect a few points of noise on the headline from that.
• The judge sees only the top-5 text candidates, so it structurally under-counts misses whose entity ranks lower (S067, S080 were found only via registry presence checks). The true miss rate is therefore likely at or above the upper bound reported here.
• Unique-name basis, not trial-weighted: a name used by 5,000 trials weighs the same as a one-off. Sponsor counts by trial volume will differ.
• Sampling intervals are large at n=100 (see each rate's 95% CI); treat single-point figures as approximations.
• Population sizes are frozen at the sampling timestamp (seed 42, same date).

12. Method

The sample is a stratified random draw of 100 unique sponsor names (seed 42) taken from the live resolution pipeline, spanning three outcome groups: names resolved by exact registry match, names resolved by the LLM stage, and names the pipeline left unresolved. Verdicts were produced by an independent LLM judge and then reviewed by hand, with direct registry checks against our local ROR and GLEIF snapshots on every flagged and borderline row. The sample is frozen: every future release is re-measured against these identical 100 names, so the figures in this report are directly comparable release over release.

Raw data: every number in this report comes from the 100-row audit sheet — download the full sample as CSV.