Can a Tool Directory Tell You Your AI Act Risk Tier?

The EU AI Act sorts AI systems into risk tiers, and the tier decides what you have to do — documentation, human oversight, conformity assessment, or nothing at all. So a reasonable question to ask of any AI tool directory, including ours, is: which tier is this one?

We tried to answer it for all 1,370 entries in our directory. We could answer it for none of them, and the reason is not that our data is thin. It is that the question is about you.

The Act asks about deployment; a catalogue records function

Annex III of the Act — the list that defines most high-risk systems — is written as a list of uses: systems that screen job applicants, evaluate creditworthiness, decide who gets into a school, or prioritise patients in an emergency department.

A directory records what a tool is: a language model, an image generator, a code assistant. Those two things are orthogonal. The same general-purpose model is minimal-risk when it drafts marketing copy and high-risk when it screens job applicants, and nothing about the model changes between those two sentences. What changes is who deployed it and what for.

The tier isn't a property of the tool, but of your deployment. No catalogue holds that information because it lives with the user, not the software.

We tried the obvious thing anyway

Stating the argument is cheap. We wanted to know what happens if you ignore it and try to classify mechanically, because that is what an automated compliance tool would do.

Our entries carry a use_cases field, populated for 758 of the 1,370 — 55.3% — with real sentences describing what the vendor says the tool is for. That is a genuine deployment signal, and we checked before assuming otherwise: the first version of this analysis assumed no such field existed, which would have made the entire finding wrong.

So we scanned those 758 for Annex III vocabulary: recruitment, creditworthiness, biometrics, triage, exams, law enforcement, and the rest. That produced 42 hits.

Then we read all 42, and 26 of them — 61.9% — were wrong. Not marginally wrong. Wrong in three distinct and instructive ways.

Three ways a keyword classifier lies to you

Substring. Eight hits matched the word "exam" inside the word "example". A tool whose use cases mention "extracting code examples from tutorials" was flagged as a high-risk educational assessment system. This is the oldest bug in text matching and it is still the most common: we hit the same trap three separate times while researching this wave, including once after having just fixed it.

Domain shift. Thirteen hits were words that mean something specific in the Act and something ordinary in software. "Triage" in Annex III means emergency medical prioritisation. In a tool directory it overwhelmingly means email triage, ticket triage, alert triage. "Migration" in the Act concerns asylum and border control; in software it is a database schema change. The word is identical and the meaning is not adjacent.

Negation. Five hits were the exact opposite of what they appeared to be. A graphic design tool advertises that it lets you make social media assets "without hiring a designer". A keyword classifier sees "hiring" and files it as a high-risk recruitment system. The tool's entire selling point is that recruitment does not happen.

That third category is the one worth carrying away, because it is not a bug you fix by being more careful with regular expressions. Marketing copy is full of claims about what a tool lets you avoid, and avoidance reads identically to activity if you are matching words.

What survived, and why it still is not an answer

After all three filters, 16 entries did describe something in Annex III territory: chest X-ray triage in emergency departments, hiring managers screening applications, fairness testing across underwriting and credit scoring. Those are high-risk deployments, and the field text says so plainly.

And we still cannot assign them a tier, for a reason that survives any improvement to our data. A tool that can be used to triage X-rays is high-risk in a hospital and is not high-risk sitting in a research sandbox. The Act attaches obligations to the deployer, in a context, for a purpose. A vendor's description of what their tool is capable of is evidence about capability, not about your deployment.

So the count of entries we could tier from the catalogue alone is 0, and it would still be 0 if every one of the 1,370 entries had a perfectly written use-case field.

The same limit applies to every directory

It would be convenient to present this as a shortcoming of our catalogue that a better-funded one has solved. It is not. A directory offering a tier column has either invented a typical deployment that belongs to nobody, or repeated whatever the vendor self-declared. Neither is a legal position.

A useful test, if you meet a tool that claims to classify your AI systems automatically: ask what it does with a general-purpose language model. If it returns a single tier, it has guessed, because that model belongs to several tiers at once depending on who points it at what. General-purpose entries make up the bulk of our own directory, which is why the honest answer here is a refusal rather than a number.

What to do instead

The practical consequence is short. Your tier comes from your own use case, written down: what the system decides, about whom, with what consequence, and how much a human can override it. That is a document you write, not a lookup you perform.

A directory is still useful — for finding candidates, comparing what they do, and reading what vendors claim. Our AI directory is built for exactly that, and the use_cases text quoted above is visible on the entries themselves. Just do not mistake it for a compliance answer.

Two related pieces may help while you are in this area. Our guide on what a tool listing cannot tell you about AI risk measures the same directory against the OWASP LLM Top 10 and finds a complementary limit — there the schema is thin, where here the schema is answering a different question entirely. And if the systems you are assessing handle personal data, our password strength tester and hash generator are small practical tools for the security hygiene that sits underneath any of this.

This guide reports a measurement of our own catalogue and explains why a category of question cannot be answered from one. It is educational and is not legal advice; an AI Act classification for a real deployment needs qualified advice on that deployment.