An SBOM for your AI: taking inventory of every model, dataset and prompt in a repo

A security reviewer sent me a questionnaire last quarter with a question I could not answer from memory: list every third-party model, dataset, and embedding source your service depends on, with its license. I knew roughly. I could have named the two big models off the top of my head. But "roughly" is not an answer you put in a vendor security review, and it is definitely not an answer that satisfies an auditor asking about AI Act obligations. The real answer was scattered across a dozen files, several of them written by people who had since left.

The uncomfortable part is that this is normal. We can produce a software bill of materials for our packages in seconds - pip freeze, a lockfile, an SBOM tool. But ask a team which AI components a system actually uses, and the honest answer is usually a shrug and a promise to go look. So I built the thing I wished I had had that afternoon.

The AI you depend on is invisible to your existing tooling

A conventional dependency is easy to see. It sits in a manifest, pinned to a version, and a hundred tools know how to read it. The AI parts of your system are nothing like that. They arrive as string literals buried in function calls:

• A model id passed to an openai or anthropic or litellm client - "gpt-4o-mini", "claude-3-5-sonnet" - chosen once and never surfaced anywhere a scanner would look.

• A Hugging Face repo id inside from_pretrained(...) or load_dataset(...), which is simultaneously a model (or dataset) and a license you never declared.

• A base URL pointing at a self-hosted endpoint or an Ollama daemon, which quietly changes where your inference actually runs.

• A prompts/ directory full of templates, plus a few thousand-character system-prompt constants pasted straight into the code - assets that shape behaviour as much as any model does.

• A vector store - Pinecone, Chroma, Qdrant, Weaviate, pgvector - that holds the data your retrieval layer leans on.

None of these show up in requirements.txt. None of them are versioned the way your packages are. And collectively they are your AI supply chain - the exact thing a security review, a data-processing agreement, or the EU AI Act's documentation expectations want you to be able to enumerate. The information exists; it is just never in one place.

modelmuster: one command, a full AI inventory

modelmuster is a small CLI that scans a project directory and produces a structured AI manifest - an inventory of every model, dataset, prompt asset, provider, and vector store it can find, each with a source location and a best-effort license flag. It is completely offline and deterministic: no network, no LLM, no API key. It reads your source, it does not run it. The same repo in always gives the same manifest out.

pip install modelmuster
modelmuster ./path/to/project

 modelmuster · scanned 214 files · 11 components

 component                          ┃ type        ┃ where               ┃ flag
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━
 gpt-4o-mini                        │ model       │ api/chat.py:24      │ -
 claude-3-5-sonnet-20241022         │ model       │ api/summarize.py:31 │ -
 openai (api.openai.com)            │ provider    │ api/chat.py:12      │ -
 meta-llama/Meta-Llama-3-8B         │ model       │ ml/finetune.py:18   │ restricted
 tatsu-lab/alpaca                   │ dataset     │ ml/finetune.py:19   │ non-commercial
 sentence-transformers/all-MiniLM   │ model       │ rag/embed.py:9      │ -
 prompts/system_summary.jinja       │ prompt      │ prompts/            │ -
 SYSTEM_PROMPT (1.8k chars)         │ prompt      │ api/agent.py:7      │ -
 pinecone                           │ vectorstore │ rag/store.py:14     │ -
 ollama (localhost:11434)           │ provider    │ ml/local.py:5       │ -
 mistral-7b (llama.cpp gguf)        │ model       │ ml/local.py:22      │ unknown

That table is the human-readable summary. The default machine output is a structured JSON manifest, one entry per component:

{
  "generated_by": "modelmuster",
  "scanned_files": 214,
  "components": [
    {
      "name": "meta-llama/Meta-Llama-3-8B",
      "type": "model",
      "provider": "huggingface",
      "source": "ml/finetune.py:18",
      "detector": "from_pretrained",
      "license_flag": "restricted"
    },
    {
      "name": "tatsu-lab/alpaca",
      "type": "dataset",
      "provider": "huggingface",
      "source": "ml/finetune.py:19",
      "detector": "load_dataset",
      "license_flag": "non-commercial"
    }
  ]
}

The principle underneath all of it: an inventory you can hand to someone else. A finding that only lives in a terminal is a curiosity; a finding with a name, a type, an exact file:line, and a JSON shape is something you can attach to a review, diff between releases, or feed to another tool.

What it inventories, and how it finds it

modelmuster walks the project's source files and combines three deterministic techniques: Python's ast module for structured call analysis, targeted regular expressions for the shapes ast can't reach, and a plain file scan for on-disk assets. Nothing here is guesswork by a model - it is pattern recognition you could read and audit yourself.

Models and providers. Using ast, it looks at calls into the SDKs teams actually use - openai, anthropic, litellm, google GenAI, cohere - and pulls the literal model= argument out of the call. It records base URLs and custom endpoints, spots Ollama references, and recognises the Hugging Face entry points: from_pretrained, pipeline, hf_hub_download, and load_dataset (which covers both models and datasets). Local model files - a .gguf, a checkpoint path handed to llama.cpp - are picked up from the file scan.

Prompt assets. Prompts are components too. modelmuster inventories prompt template files - a prompts/ directory, *.prompt files, *.jinja templates - and flags large system-prompt string constants embedded directly in code, because a 1,800-character instruction block is a behavioural dependency whether or not anyone treats it as one.

Vector stores. References to Pinecone, Chroma, Qdrant, Weaviate, and pgvector are detected and listed, so the retrieval side of a RAG system shows up in the manifest alongside the models.

For every component it records four things: the name, the type (model / dataset / prompt / provider / vectorstore), the source location as file:line, and a license/risk flag where one can be derived.

The license flag, and exactly how far it goes

For Hugging Face components, modelmuster consults a small curated license map - a snapshot of well-known repo ids and their license class - and attaches a flag: non-commercial, research-only, copyleft, or unknown. That is the single most useful column for a reviewer skimming the manifest, because it turns "here are 40 components" into "here are the three you need to look at."

And here is the part I want to be completely straight about: that flag is a triage aid, not legal advice. The map is curated and best-effort. It lags reality - licenses on the Hub change, repos get relicensed, new models ship every week. A component the map does not recognise is flagged unknown rather than silently blessed. The flag exists to route real cases to a human, not to make the call for them. If you take one sentence from this post about the license column, take that one.

It emits; it does not gate

This is the design decision that makes modelmuster different from most tools in this space: it always exits 0. It is a producer of an inventory, not a gate on your pipeline. It does not have opinions about whether your build should pass. It scans, it emits, it gets out of the way.

That matters because an inventory tool that can fail a build gets disabled the first time it fails a build at an inconvenient moment. modelmuster is meant to run on every commit, in CI, contributing an artifact - the manifest - rather than a verdict. You wire the verdict, if you want one, downstream: diff today's manifest against last release's, or pipe it into a policy check that you control. The tool's job ends at an accurate, current inventory.

The CycloneDX export

A JSON manifest that only modelmuster understands is a dead end for anyone with an existing SBOM workflow. So --format cyclonedx re-emits the same inventory as a CycloneDX-style JSON document using its ML component schema, which is the emerging industry standard for describing machine-learning components in a bill of materials.

modelmuster ./project --format cyclonedx > ai-manifest.cdx.json

The point is not that CycloneDX is magic; it is that your AI inventory should live in the same format, and flow through the same tooling, as the rest of your supply-chain metadata - not in a bespoke shape that needs a bespoke reader.

What modelmuster is not

The AI-supply-chain space is filling up with narrow tools, and the honest thing is to draw the boundaries clearly rather than imply modelmuster does everything:

• It is not a license gate. A sibling tool, licenseward, takes a single ML asset and fails the build (exit 1) when its license forbids commercial shipping. modelmuster does the opposite job on purpose: it enumerates everything and never blocks. One decides; the other describes.

• It is not an SBOM diff. Telling you what changed between two inventories - a model swapped, a dataset added - is a separate job that a tool like sbomdrift handles. modelmuster produces the snapshot that a diff tool would compare; it does not compare.

• It is not a hallucinated-dependency checker. Catching an import of a package that does not exist is what something like phantomdep is for. modelmuster only inventories the AI components that are genuinely referenced.

modelmuster's one job is generating the full AI inventory across a whole project. Keeping it to that one job is why it stays trustworthy.

Honest limitations

Every technique modelmuster uses has an edge, and I would rather write the edges down than let you discover them:

• Static detection misses dynamic ids. A model id assembled from a variable, an environment lookup, or an f-string cannot be resolved by reading the source - modelmuster can note that a call exists but not always name the model behind it. Runtime-configured components are the known blind spot.

• The license map is curated, best-effort, and not legal advice. It is a snapshot that lags the Hub. unknown means "not in the map," not "safe." Always confirm a component's real license before acting on the flag.

• Regex and ast patterns can miss unusual shapes. A provider wrapped in your own abstraction, an SDK the detectors don't yet know, or a prompt loaded in a way I didn't anticipate can slip through. The inventory is thorough, not omniscient - treat a clean scan as "nothing detected," not "nothing there."

None of these are hidden. They are in the README, and they are the reason the tool positions itself as visibility, not a guarantee.

Try it

pip install modelmuster

modelmuster ./project                          # summary table
modelmuster ./project --json                   # structured JSON manifest
modelmuster ./project --format cyclonedx       # CycloneDX ML components

It is MIT-licensed and lives at github.com/jay-tank/modelmuster. The next time a security review asks you to list every model, dataset, and prompt your system depends on, the answer should be one command - not an afternoon of grepping and a shrug.

The license flags are a best-effort triage aid from a curated snapshot, not legal advice - always confirm a component's current license before acting on a finding.