We catalogue 1,366 AI tools. Someone picking one from that directory is, reasonably, trying to make a decision — and part of that decision is whether the thing is safe to point at their work.
We set the ten OWASP LLM risk categories against what our own directory actually records, one field at a time. Our data bears on four of the ten. On the other six it is silent — and so, as far as we can tell, is every other directory of this kind, including the ones with far more entries than ours.
That silence is the most useful thing a catalogue can tell you about itself. It marks the boundary between what a listing settles and what it cannot.
The four it can speak to
LLM03, supply chain. The strongest of the four. We record whether a tool is open source, its repository link where one exists, and its parent company. Those are real supply-chain facts: who controls the artefact, whether you can inspect it, and where it comes from.
LLM06, excessive agency. Partially. Our category field distinguishes tools that act from tools that answer, which is the difference between a chatbot and something with credentials. It is a coarse signal. It records nothing about what permissions a given agent asks for, which is the part that decides the risk.
LLM10, unbounded consumption. Half of it. We record pricing model, whether there is a free tier, and whether an API exists — which bears on whether usage is metered and what runaway use would cost. The technical half, whether the service throttles or degrades under load, is not recorded anywhere.
LLM02, sensitive information disclosure. Weakly, and only by inference. An API's existence tells you data can leave; it tells you nothing about what leaves, where it is processed, or how long it is kept.
The six it cannot
Prompt injection (LLM01), data and model poisoning (LLM04), improper output handling (LLM05), system prompt leakage (LLM07), vector and embedding weaknesses (LLM08) and misinformation (LLM09).
None of these have a field because they are not properties of a tool, but of a specific deployment.
Whether a system is vulnerable to prompt injection depends on what untrusted text it is asked to process and what it is allowed to do afterwards. A product can be safe in one integration and dangerous in another. While a directory entry describes the product, it cannot describe your integration. A field that claimed to would be lying.
Misinformation is the sharpest example. Whether a tool's output is reliable enough depends on the question you are asking it and on what you do with the answer. No boolean settles that, and a "5-star accuracy" column would be worse than nothing — it would be an invitation to stop checking.
What we measured and what we argued
Measured: which fields exist in the directory's schema, and how many of the 1,366 entries populate each. That is a fact about a database and is reproducible by anyone with the repository.
Argued: which field bears on which OWASP category. Every line of that mapping is a judgement of ours, and every line is written out in the measurement script so it can be disagreed with individually. Someone could reasonably argue that open-source status also bears on LLM04, since an inspectable training pipeline is a poisoning defence — we did not count it, because our field records the licence and not the pipeline. Disagree with that specific line and the count moves to five. This approach puts the reasoning on the page, instead of hiding it inside a summary total.
What we have deliberately not produced is a score. No tool in the directory is rated safe or unsafe here, and no ranking is implied. A coverage map says what questions the data can answer; it says nothing about the answers.
Could a directory do better?
Partly. We have to be specific about which parts, because "we cannot know" is too comfortable an answer.
Three of the six blanks could be closed with fields nobody currently keeps. Whether a vendor publishes a model card, whether it states a training-data source, and whether it documents a data-retention period are all facts about the product, checkable from public documentation, and all bear on poisoning and disclosure. They are absent from our directory because nobody has collected them, not because they are unknowable — and collecting them across 1,366 entries is a real project rather than a schema change.
The other three are not closable at any size. Prompt injection, improper output handling and misinformation depend on the integration, and no field about the product can settle a question about your deployment. A directory that added columns for them would be manufacturing false confidence, which is worse than a blank.
The gap divides in half. One half is simply work not yet done. The other is a category error, the kind of problem someone will eventually try to fix by adding a new database column.
What to do with a listing, then
Use it for the four it is good for. Our AI directory is searchable by exactly those attributes, which is a deliberate choice about what a catalogue can honestly index. Supply chain especially: whether something is open source, who owns it, and whether the code can be read are exactly the questions a directory can settle and a vendor's marketing page often will not.
For the other six, a listing is the beginning of the work rather than the end of it. The important questions are about your system. What does this tool see? What can it do afterwards? What happens when it receives hostile input? These can only be answered from within your specific deployment.
If you want the vendor-side version of this, our guide on whether an AI tool is safe for company data covers what certifications do and do not prove, which is the same shape of question asked of compliance badges instead of catalogue fields. For the selection process around it, choosing an AI tool covers the criteria that a listing can settle.
Use a directory for the questions it can settle. The danger comes from expecting it to answer the ones it cannot. Ours holds a field bearing on four of the ten OWASP LLM categories — supply chain most usefully — and nothing at all on the other six. Three of those six are work nobody has done yet and could be closed. The remaining three are properties of your deployment rather than of any product, so no catalogue will ever close them, and a column claiming to would be selling confidence it does not have.
- Produced by
measure-owasp-coverage.cjs, committed with this guide. It reads the liveai_directory_entriesschema and entry populations from the application, then sets them against the ten category identifiers. The script fails rather than reports if the mapping names a field that no longer exists, so the guide cannot quietly go stale as the schema changes. - Measured: the schema and the populations. Argued: which field bears on which category — every line of that mapping is in the script, individually reviewable, and the count moves if you disagree with one.
- ⚠️ COPYRIGHT. OWASP category identifiers and short names only. None of the OWASP Top 10 for LLM Applications text is reproduced here; the descriptions are ours.
- ⚠️ NO RISK SCORE IS PRODUCED AND NONE SHOULD BE INFERRED. Nothing here says any catalogued tool is safe or unsafe. The measurement is about our data, not about the tools.
RECATOOLS publishes the AI directory examined here. This guide reports the limits of our own catalogue and applies to comparable directories for the same structural reasons.
View author profile → · Editorial policy
