OAEI 2026 track
DISO-OAEI distributes ten curated ontologies spanning cyber security, command & control, and smart environments — and evaluates matching systems on two tasks: classic global alignment against a repaired silver standard, and candidate ranking with explicit NIL abstention.
The tasks
Task 1 measures whole-alignment quality the traditional OAEI way; Task 2 isolates local decision quality — including knowing when no match exists.
Submit one full OAEI Alignment RDF per pair. Scored with partiality-corrected precision, recall and F1 — micro- and macro-averaged — against the repaired silver standard R⁺≈, with the unrepaired R≈ as secondary. Self-score locally with the provided tools.
Task description →For each source entity, rank a fixed 50-candidate pool best-first — placing the NIL candidate early to abstain when no equivalent exists. Unsupervised; scored organiser-side with Hits@{1,3,5,10} and MRR, macro-averaged over both pairs.
Task description →Results
The best-performing systems of the 2026 evaluation, from the final CodaBench leaderboards and, for Task 1, the systems run by the organisers (marked). Full standings, with notes on the organiser-run systems: results.
| # | System | Micro-F1, R⁺≈ |
|---|---|---|
| 1 | LogMapLLMFamily used to build the referenceLogMapLLM belongs to the LogMap family of matchers, one of the families whose consensus votes produced the partial reference alignments (before manual review and repair). Bear this in mind when comparing its scores with those of other systems. | 0.870 |
| 2 | LogMapFamily used to build the referenceLogMap belongs to the LogMap family of matchers, one of the families whose consensus votes produced the partial reference alignments (before manual review and repair). Bear this in mind when comparing its scores with those of other systems. | 0.858 |
| 3 | LogMapKGFamily used to build the referenceLogMapKG belongs to the LogMap family of matchers, one of the families whose consensus votes produced the partial reference alignments (before manual review and repair). Bear this in mind when comparing its scores with those of other systems. organiser run | 0.856 |
| 4 | MOSAIC-cli organiser run | 0.569 |
| 5 | MatchaFamily used to build the referenceMatcha belongs to the Matcha family of matchers, one of the families whose consensus votes produced the partial reference alignments (before manual review and repair). Bear this in mind when comparing its scores with those of other systems. | 0.550 |
| # | System | MRR (macro) |
|---|---|---|
| 1 | Matcha | 0.725 |
Data
The 2026 datasets are published on the Hugging Face Hub: OAEI-ML/diso-oaei (the v2026 tag pins this edition). Verify your copies against the SHA-256 checksums below — they were computed from the bytes served by Hugging Face and re-verified against the repository's canonical files.
| File | Contents | SHA-256 |
|---|---|---|
| ontologies.zip | The 10 task ontologies (OWL) | 8ff0a5fa64ff3ee5c05601b7618d471ea12f36018e2f0cf8c511c9d47af2777d |
| repaired_silver_refs.zip | Task 1 headline references R⁺≈ (6 pairs) | e885b1a9f2a5e39c6529406860db12a4f8ae6e4961f790d2ed919070a9f0a4c8 |
| unrepaired_silver_refs.zip | Task 1 secondary references R≈ (6 pairs) | bd50b4b6472ae790b4c4e949172de8325bead20951a337eb010bb53f948b7c40 |
| pools/uco-stix/pools.jsonl | Task 2 candidate pools — uco-stix (217 queries) | dc375cff4d92e30caf7f54f2cb58036541c7d0b8eb82e00a8cb3c1cb9570d096 |
| pools/stix-d3fend/pools.jsonl | Task 2 candidate pools — stix-d3fend (73 queries) | 5c50fae542a4df49c5cf12e66a24c40b489bfb279909a8f30d08ec170896df6b |
Prefer the CLI? hf download OAEI-ML/diso-oaei --repo-type dataset
(see the quickstart). Individual ontologies and
per-pair reference files are listed on the
ontologies and Task 1 pages.
Reference alignment creation methodology:
diso-mappings.
Timeline
The network
Drawn from the DISO collection — publicly available ontologies for defence, intelligence, national security, and operationally adjacent domains.
Part of the OAEI
DISO-OAEI runs under the Ontology Alignment Evaluation Initiative — the long-running campaign that benchmarks ontology-matching systems.
The OAEI-ML 2026 evaluation tracks are DISO-OAEI (this track) and OAEI Bio-ML. A proud member of the OAEI.
Citation
A Zenodo DOI for the track dataset is pending and will be published here once minted. Until then, cite the underlying collection:
Dilworth, J., Cotovio, P., Herron, D., Pesquita, C., Jimenez-Ruiz, E., Cripps, P., & Dewdney, N. (2026). DISO: Defence, Intelligence and Security Ontologies (1.1.0) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.20059506
People
Jon Dilworth, Pedro Cotovio, Ernesto Jimenez-Ruiz, and Catia Pesquita.
Dave Herron, Paul Cripps, and Nigel Dewdney.
Supported by Turing Innovations Limited and The Alan Turing Institute's Defence and Security Programme via the project GUARD.
Questions or corrections: open an issue on the track repository, contact the organisers, or email contact@oaei-ml.org.