LLM / AI-application security

Codebases now embed LLM calls, agent frameworks, MCP servers and model loading, and that surface has its own vulnerability classes that generic SAST does not cover. Nephilex ships two scanners for it, with every finding mapped to an id in the OWASP Top 10 for LLM Applications 2025 and to a CWE:

Scanner	Reads	Needs
llm	Python and JS/TS files that touch an LLM or agent stack (OpenAI, Anthropic, LangChain, LlamaIndex, LiteLLM, Ollama, Google GenAI, CrewAI, AutoGen, MCP SDKs for Python and TypeScript, OpenAI Agents, transformers)	The data-flow rules need the [ast] extra (pip install "nephilex[ast]"); the line rules always run
agent-config	MCP and agent configuration files (.mcp.json, .cursor/mcp.json, .vscode/mcp.json, claude_desktop_config.json, .claude/settings.json, Windsurf mcp_config.json, Cline, Roo Code, Gemini CLI)	Nothing extra

Every finding carries owasp and metadata.owasp_llm (for example LLM05:2025-Improper Output Handling) and the tags ai-security plus llm, agent and/or mcp, so the JSON and SARIF output can be filtered or routed by category.

nephilex scan ./my-ai-app --scanners llm,agent-config

Rules

Rule	OWASP LLM	CWE	What it flags
llm.insecure-output-handling	LLM05	78, 95, 89, 73/22, 918, 79, 1336, 502	A model response reaching a shell, eval/exec, SQL, a file write/delete path, an outbound URL, unescaped HTML, a template compiler or an unsafe deserializer
llm.tool-handler-exec	LLM06	78, 95	An agent tool whose handler passes its (model-chosen) argument to a shell or eval
llm.agent-shell-tool	LLM06	78, 250	ShellTool, PythonREPLTool, PythonAstREPLTool, load_tools(["terminal"]) given to an agent
llm.agent-file-write-tool	LLM06	732, 73	FileManagementToolkit (or WriteFileTool and friends) exposing write/delete/move
llm.agent-requests-tool	LLM06	918	RequestsToolkit / Requests*Tool enabled with allow_dangerous_requests=True
llm.agent-sql-write-tool	LLM06	89, 250	SQLDatabaseToolkit / create_sql_agent with no read-only scoping in the file
llm.agent-code-execution	LLM06	94, 250	AutoGen use_docker: False or LocalCommandLineCodeExecutor; CrewAI code_execution_mode="unsafe"
llm.indirect-prompt-injection	LLM01	1427, 74	External content (HTTP responses, files read at run time, retrieved documents, email, Google API data, stored free text, tool results) in the prompt of a model whose tools or output can act on it
llm.tool-argument-to-sink	LLM06	73/22, 918, 89, 200	A model-chosen tool argument (or a hand-dispatched tool-call argument) used as a write path, an outbound URL, SQL text or an email recipient
llm.untrusted-message-role	LLM01	1427, 74	The caller supplies message roles or the whole message list
llm.untrusted-tool-selection	LLM06	862, 1427	Request input decides which side-effecting tools an agent gets
llm.secret-sent-to-model	LLM02	200, 522	A credential read from the environment placed in prompt text
llm.untrusted-input-in-system-prompt	LLM01	77, 74	Request input, file contents, retrieved documents, HTTP responses or stored user-written text flowing into a system prompt (Python and JS/TS)
llm.secret-in-system-prompt	LLM07	798, 200	A credential in system-prompt text
llm.internal-url-in-system-prompt	LLM07	200	A private IP, localhost or *.internal-style host in system-prompt text
llm.prompt-injection	LLM01	74	Request input concatenated into a prompt string (line rule)
llm.secret-in-prompt	LLM02	522, 200	A credential in prompt text (line rule)
llm.unrestricted-tool-registration	LLM06	732	allow_dangerous_*=True / dangerous_tools=True opt-in flags (line rule)
llm.trust-remote-code	LLM03	494, 829	trust_remote_code=True
llm.torch-load-pickle	LLM03	502	torch.load without weights_only=True
llm.model-download-http	LLM03	319, 494	Model weights fetched over plaintext HTTP
agent-config.mcp-hardcoded-secret	LLM02	798	A literal credential in an MCP server's env, headers or args
agent-config.mcp-unpinned-package	LLM03	1357, 829	A server started with npx/bunx/pnpm dlx/uvx/pipx run/docker run on an unpinned package or image
agent-config.mcp-plaintext-http	LLM03	319	A remote MCP server over http:// (loopback excluded)
agent-config.mcp-broad-filesystem-root	LLM06	732, 552	A filesystem server (or Claude Code additionalDirectories) granted /, a drive root or a home directory
agent-config.agent-auto-approve	LLM06	862	Tools that run without confirmation: Cline autoApprove, Roo Code alwaysAllow, Gemini CLI trust: true, Claude Code enableAllProjectMcpServers, a bare Bash allow rule, bypassPermissions

llm.insecure-output-handling (LLM05 Improper Output Handling)

Model output is attacker-influenceable: anyone who can put text in front of the model (a chat message, a web page, a retrieved document) can steer what it returns. A prompt injection upstream becomes command execution, SQL injection, SSRF or XSS at the sink.

llm = ChatOpenAI()
sql = llm.invoke(f"Write SQL for: {question}").content
conn.execute(sql)  # flagged: CWE-89

Fix. Have the model choose parameters, not code: bind them into fixed, parameterized queries, map choices onto fixed argv lists, pin URL hosts in code, escape before HTML. If text-to-SQL is the product, run it on a read-only, row-scoped role.

Sources are recognized only when the receiver really is an LLM object, resolved through the file's imports (client = OpenAI(), llm = ChatOpenAI(), chain = prompt | llm, index.as_query_engine(), litellm.completion, ollama.chat, genai.Client().models.generate_content, Vercel AI SDK generateText), so model.predict(X) on a scikit-learn model or a Twilio client.messages.create is not a source. Provider metadata (.id, .usage, .finish_reason) is not model text.

What clears a value, and only for the sink it protects:

• numeric or boolean coercion (int(), float(), len(), Number());
• shlex.quote for shells, html.escape / bleach.clean / DOMPurify.sanitize for HTML, os.path.basename / secure_filename for paths;
• a URL whose scheme and host are literal (f"https://api.example.com/{slug}");
• Pydantic validation that pins the field (Literal, an Enum, int, a pattern=). A plain str field stays tainted: the schema did not constrain it. A schema defined outside the file cannot be inspected and is trusted;
• an allowlist check (if cmd not in ALLOWED: raise, if host in HOSTS:, re.fullmatch, is_relative_to, ALLOWED.includes(x)). A blocklist (if "rm" in cmd) does not clear anything, and checking one property (block.type == "tool_use") does not clear the rest of the object.

json.loads does not clear anything: the parsed dict is still model text.

In JS/TS the sinks also include an Express res.send / res.write / res.end of a string that contains markup (Express serves a string body as text/html) and db.prepare(sql) on a database-like receiver. Both languages follow a call into a function defined once in the same file with the caller's argument taint, so a tool-call argument passed to a local run_tool(...) helper is traced into the helper's body.

llm.tool-handler-exec (LLM06 Excessive Agency)

Tool-call arguments are model output. A tool (LangChain @tool, MCP @mcp.tool(), OpenAI Agents @function_tool, Tool(func=...), StructuredTool.from_function, a dispatch dict of handlers) that shells out or evals its argument gives the agent unbounded capability.

@mcp.tool()
def calculate(expression: str) -> str:
    return str(eval(expression))  # flagged: CWE-95

Fix. Give the tool a narrow, typed interface (Literal or Enum parameters, which this rule treats as safe), never pass arguments through a shell or eval, and require confirmation for side effects.

Agent tools: agent-shell-tool, agent-file-write-tool, agent-requests-tool, agent-sql-write-tool, agent-code-execution (LLM06)

Each finding names the tool, the agent it is wired into (or says no wiring was found in the file), and what makes it excessive.

tools = FileManagementToolkit().get_tools()  # flagged: all write/delete tools, no root_dir
agent = create_react_agent(llm, tools, prompt)

runner = autogen.UserProxyAgent(
    "runner", code_execution_config={"use_docker": False}
)  # flagged: host execution

Not flagged: FileManagementToolkit(selected_tools=["read_file"]), RequestsToolkit without allow_dangerous_requests (LangChain refuses to build it), AutoGen with use_docker: True or code_execution_config=False, CrewAI's default safe (Docker) mode, and a SQL connection with visible read-only scoping (mode=ro, postgresql_readonly). A human-approval guard in the same file (HumanApprovalCallbackHandler, LangGraph interrupt_before / interrupt_on, HumanInTheLoopMiddleware) suppresses the tool rules.

Fix. Replace broad tools with narrow ones that each do one operation, confine file tools with root_dir and selected_tools, run generated code in a sandbox (Docker executor, E2B, Modal), connect SQL agents with a read-only role, and keep destructive tools behind human approval. agent-sql-write-tool is reported with LOW confidence because privileges cannot be seen in code.

llm.untrusted-input-in-system-prompt (LLM01 Prompt Injection)

docs = store.similarity_search(q)
context = "\n".join(d.page_content for d in docs)
messages = [{"role": "system", "content": "Answer using: " + context}]  # flagged

System prompts are {"role": "system" | "developer"} messages, LangChain SystemMessage and ("system", ...) tuples, Anthropic system=, OpenAI instructions=, system_prompt= / system_message= / system_instruction= arguments, and variables named like SYSTEM_PROMPT. Sources are request parameters (including FastAPI/Flask route parameters), file reads, vector-store retrieval, document loaders, HTTP responses, and free-text fields of a row loaded through an ORM (Store.query.get_or_404(id).policy_text: names such as *_text, description, bio, notes, policy, body). In JS/TS the same rule covers {role: "system" | "developer"} objects and the system / instructions options of OpenAI, Anthropic and Vercel AI SDK calls, with req.body, req.query, req.params and await req.json() as sources:

const { system, message } = req.body;
await anthropic.messages.create({ model, max_tokens: 600, system: system || DEFAULT_SYSTEM,  // flagged
  messages: [{ role: "user", content: String(message) }] });

Fix. Keep the system prompt static; pass untrusted data in a delimited user-role message and constrain what the model may do with it.

llm.indirect-prompt-injection (LLM01 Prompt Injection)

Text a model reads can instruct it. When an application puts content written by third parties into a prompt, and that model can act (call a tool that runs commands, writes files, sends data or fetches URLs, or produce output that reaches a dangerous sink), whoever writes the content controls the action.

def handle_ticket(ticket_id: int) -> str:
    ticket = requests.get(f"{HELPDESK}/tickets/{ticket_id}", timeout=10).json()
    messages = [
        {"role": "system", "content": "Resolve the ticket and refund when owed."},
        {"role": "user", "content": f"Subject: {ticket['subject']}\n\n{ticket['description']}"},
    ]
    resp = client.chat.completions.create(model="gpt-4o", messages=messages, tools=TOOLS)  # flagged
    for tc in resp.choices[0].message.tool_calls or []:
        args = json.loads(tc.function.arguments)
        issue_refund(args["order_id"], args["amount"])  # issue_refund POSTs args to /refunds

The finding is reported at the model call and says which content entered the prompt (and on which line), which tools can act on it and what their model-chosen arguments reach, and which sinks the output reaches. For the full file this snippet comes from (benchmarks/agent/corpus/ipi-support-ticket-refund-tool/vulnerable):

An HTTP response (line 29) is placed in the prompt of client.chat.completions.create() without separation from the instructions, and the model's output reaches requests.post() request body (line 24). Whoever can write that content can steer what the model does.

External content is: HTTP responses (requests, httpx, fetch, axios), files read from a caller-chosen path, uploaded files, vector-store retrieval and document loaders (including query engines and retrieval chains that retrieve into their own prompt), IMAP mail and parsed email messages, Google API reads (service.users().messages().get(...).execute()), ORM free-text fields, the return value of a tool that returns any of these, and built-in search, browsing, mail and file tools (Tavily, DuckDuckGo, Wikipedia, RequestsGetTool, Gmail, FileManagementToolkit). Request input from the caller is not external content for this rule: that is direct prompt injection by the user, covered by the tool and output rules.

What the model can reach:

• tools: for every tool the model or agent is given (tools=, bind_tools, create_react_agent, AgentExecutor, ReActAgent.from_tools, Vercel AI SDK tools: {...}), the sinks its model-chosen arguments reach inside the tool body, after guards: shell, eval, file write path, unpinned URL, SQL text, HTTP request body, email recipient, Google API writes. Built-in shell, REPL, file-write, requests, SQL and Gmail-send tools count as their capability;
• output: model output reaching any sink of insecure-output-handling, a request body, an email recipient or a data-changing API call, including through hand-written tool-dispatch loops and calls into local helper functions.

The severity is HIGH unless the only reach is HTML (MEDIUM). Delimiting the content (tags, a user-role message) does not clear the finding; it does not stop a model from following instructions in the content.

Not flagged: the same content summarized by a model with no tools and output that is only returned; tools whose arguments are Literal/Enum/zod enum, allowlisted (if host not in HOSTS: return, row = TABLE.get(key); if row is None: return) or confined (resolve() plus is_relative_to); a structured output schema that pins the fields that reach the action.

Fix. Give the model that reads untrusted content no side-effecting tools, or require human approval for each side effect; constrain tool arguments with allowlists or Literal/enum schemas; validate output before it reaches a sink; or split the work so a model with no tools reads the content and only a validated, constrained value reaches the privileged step.

llm.tool-argument-to-sink (LLM06 Excessive Agency)

The tool-chaining counterpart of tool-handler-exec: a model-chosen argument of a tool (LangChain @tool, MCP @mcp.tool() and TypeScript server.tool / server.registerTool, Vercel AI SDK and OpenAI Agents tool({execute}), LangChain.js tool()), or a tool-call argument that code dispatches by hand, used as a file path to write, an outbound URL, SQL text or an email recipient.

@mcp.tool()
def save_note(path: str, content: str) -> str:
    target = NOTES_ROOT / path
    target.write_text(content, encoding="utf-8")  # flagged: CWE-73, CWE-22

server.tool("fetch_url", "Fetch a web page.", { url: z.string().url() }, async ({ url }) => {
  const res = await fetch(url);  // flagged: CWE-918
  return { content: [{ type: "text", text: await res.text() }] };
});

A recipient is an email To/Cc/Bcc header (msg["To"] = to, also inside a tuple assignment), smtp.sendmail(..., to_addrs), Django send_mail recipients, nodemailer sendMail({to}), Resend and SendGrid to.

Fix. Resolve paths and require them to stay inside a base directory (Path.resolve().is_relative_to(BASE), or path.resolve plus startsWith(BASE + path.sep)); pin URL hosts or check hostname against an allowlist and refuse redirects; bind SQL parameters; map recipients from a fixed set.

llm.untrusted-message-role (LLM01 Prompt Injection)

@app.post("/chat")
def chat(req: ChatRequest):
    messages = [{"role": "system", "content": SYSTEM_PROMPT}] + req.messages
    resp = client.chat.completions.create(model="gpt-4o-mini", messages=messages)  # flagged

Roles are the trust boundary of a chat prompt. Flagged when request data is used as message objects (a client-supplied transcript) or as a role value ({"role": req.role, ...}, { role, content } destructured from req.body). Request text inside a message whose role is a literal is the correct pattern and is not flagged, including a server-side history the user turn is appended to.

Fix. Accept only the new user text, keep the transcript on the server, and set every role in code.

llm.untrusted-tool-selection (LLM06 Excessive Agency)

@app.post("/ask")
def ask(req: AskRequest, user: User = Depends(get_current_user)):
    tools = [get_issue, set_status] if req.allow_writes else [get_issue]
    agent = create_react_agent(ChatOpenAI(model="gpt-4o-mini"), tools)  # flagged

Flagged when request input (including typed flags such as allow_writes: bool) chooses the tool list given to an agent or model (a conditional, a subscript, a comprehension or an if block that appends), and that list includes a tool with a side effect. The same code with the flag computed from the authenticated user (user.role in {...}) is not flagged.

Fix. Decide tools on the server from the authenticated user's permissions.

llm.secret-sent-to-model (LLM02 Sensitive Information Disclosure)

const prompt = [
  `Authenticate with this secret key: ${process.env.STRIPE_SECRET_KEY}`,  // flagged at the model call
  `Question: ${question}`,
].join("\n");

An environment variable whose name says it holds a credential (*SECRET*, *TOKEN*, *PASSWORD*, *API_KEY*, *_KEY) flowing into prompt text of a model call. Using the key to construct a client (new OpenAI({ apiKey }), Github(token)) is not flagged, and neither is the data that client returns.

Fix. Keep the credential in code or a tool that calls the service; give the model a placeholder.

llm.secret-in-system-prompt and llm.internal-url-in-system-prompt (LLM07 System Prompt Leakage)

SYSTEM_PROMPT = (
    "You are support. Admin password: Sup3rS3cret99. Billing: http://billing.internal/api"
)

System prompts are routinely extracted, so anything in them is effectively public. Secrets are matched with the secrets scanner's signatures plus provider key formats and password: <value> prose; the snippet is always redacted.

Fix. Keep credentials and internal endpoints in server-side code or tools, never in text the model can repeat; rotate any exposed credential.

Line rules (prompt-injection, secret-in-prompt, unrestricted-tool-registration, trust-remote-code, torch-load-pickle, model-download-http)

Well-known signatures that need no data flow. trust_remote_code=True, torch.load without weights_only=True and plaintext model downloads are LLM03 Supply Chain. Where a data-flow rule reports the same line, the line rule is dropped. Without tree-sitter, a regex pass stands in for insecure-output-handling and the agent-shell-tool line rule for the agent analysis.

agent-config.mcp-hardcoded-secret (LLM02 Sensitive Information Disclosure)

{"mcpServers": {"brave": {"command": "node", "args": ["index.js"],
  "env": {"BRAVE_API_KEY": "BSAq8Zx3LmN7pQ2rT5vW9yK1"}}}}

Literal values in env, headers and --flag value arguments are checked with the secrets scanner's signatures and, for secret-named keys, an entropy test; ${VAR}, ${env:VAR}, ${input:id} and placeholders are not secrets. The value is redacted in every snippet. When the secrets scanner also matches the line, the engine reports it once.

Fix. Rotate, then reference the value: ${VAR} in .mcp.json, ${env:VAR} in Cursor and Windsurf, ${input:id} with password: true in VS Code, or an envFile kept out of version control.

agent-config.mcp-unpinned-package (LLM03 Supply Chain)

{"mcpServers": {"github": {"command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"]}}}

An unpinned spec is resolved at every start, so a compromised or typosquatted release runs with the developer's privileges and feeds tool descriptions to the model. npx -y additionally skips the install prompt. Exact versions (pkg@1.2.3, uvx pkg==1.2.3, uvx --from 'pkg==1.2.3' cmd), local paths, git specs and image digests are not flagged; @latest, ranges and bare names are.

Fix. Pin exact versions (or an image digest), or install the server as a locked dependency.

agent-config.mcp-plaintext-http (LLM03 Supply Chain)

A remote server over http:// exposes auth headers and lets anyone on the path rewrite tool descriptions and results, which is prompt injection by another route. Loopback addresses are not flagged. Fix. Use https://.

agent-config.mcp-broad-filesystem-root (LLM06 Excessive Agency)

{"mcpServers": {"fs": {"command": "npx",
  "args": ["-y", "@modelcontextprotocol/server-filesystem@2025.8.21", "/"]}}}

Also checks docker --mount / -v sources for the filesystem image and Claude Code permissions.additionalDirectories. Root and drive roots are HIGH, home directories MEDIUM. Fix. Grant only the project or a dedicated workspace.

agent-config.agent-auto-approve (LLM06 Excessive Agency)

{"permissions": {"allow": ["Bash"]}}

Confirmation prompts are the last check between a prompt injection and a side effect. Flagged: non-empty Cline autoApprove and Roo Code alwaysAllow, Gemini CLI trust: true, Claude Code enableAllProjectMcpServers: true, a bare Bash/Bash(*)/PowerShell allow rule, and defaultMode: "bypassPermissions" (LOW: Claude Code v2.1.257 and later ignore it in project files). Scoped rules such as Bash(npm run test *) are not flagged. Fix. Remove auto-approval or limit it to read-only tools.

Scope and honesty

The data-flow tier is intra-file. Within a file it is per function, follows calls into functions defined once in the file (up to three levels deep), and runs loop bodies twice so a tool result appended to the message list reaches the next model call (nested JS functions see their enclosing scope). A value that crosses a module boundary, or a prompt built in one file and sent from another, is not traced, and neither are keyword-argument splats into a helper (send_email(**block.input)) or comprehension variables. Pydantic models defined in other files are trusted rather than guessed at, which favors precision over recall. The configuration rules read only the files listed above, by their documented key names. Treat findings as high-value leads and a clean result as "no known LLM or agent anti-patterns found", not a proof of safety.

The labeled corpus in benchmarks/accuracy.py has a vulnerable and a safe case for each data-flow and configuration rule, and CI fails if any is misclassified.

Benchmark: labeled agent corpus with a held-out split

benchmarks/agent/ holds 38 scenarios (76 cases: 38 vulnerable, 38 safe) of LLM-application and agent code: 25 Python, 8 TypeScript and 5 JavaScript scenarios covering indirect prompt injection (12), tool chaining (8), trust boundaries (8), output handling (3), excessive agency (5), system prompt leakage (1) and sensitive disclosure (1). Each scenario pairs a vulnerable case with a safe case that does the same thing correctly, labeled with the OWASP LLM 2025 ids a correct report may carry. The corpus was written by a separate author who did not read the scanner code or run the scanner.

Scenarios were split before any rule was written: a scenario is held out when the SHA-256 of its id is divisible by 3 (24 dev, 14 held-out scenarios). The corpus, the split and a baseline score were committed first; rule work looked only at dev cases; the held-out split was scored once, after the rules were committed. A vulnerable case counts as caught only when an llm or agent-config finding at LOW or above carries one of its expected OWASP ids; any such finding on a safe case is a false positive.

python benchmarks/agent/run.py                  # both splits
python benchmarks/agent/run.py --only dev --details dev

Split	Cases (vuln / safe)	Before: recall	Before: precision	After: recall	After: precision
dev	24 / 24	29.2% (7/24)	87.5% (1 FP)	100.0% (24/24)	96.0% (1 FP)
held-out	14 / 14	35.7% (5/14)	83.3% (1 FP)	64.3% (9/14)	69.2% (4 FP)

The held-out result is much worse than dev, and it is the number to believe: the dev score measures how well the rules fit the cases they were written against. On held-out indirect prompt injection, 2 of 6 vulnerable cases were caught. The held-out misses, read after the score was recorded:

• a Slack SDK read (slack.conversations.history) is not a known external source, so the injection was missed (the model-chosen URL fetch was reported as LLM05 SSRF, which the label does not accept);
• tool-call arguments passed as send_email(**block.input) and a tool call made inside a list comprehension were not followed;
• tools chosen by request input among JSON tool schemas, whose implementations are lambdas in a dict, were not resolved to their side effects;
• marked.parse(answer) matched the zod-style .parse validation heuristic, so model output rendered as HTML was treated as validated;
• false positives: a host allowlist checked through a variable (parsed = urlparse(url); if parsed.hostname not in HOSTS: return) did not clear url; a FileManagementToolkit confined with root_dir still counted as a file-write capability; and the two bugs below.

Two of the held-out false positives came from mistakes in the new rules and were fixed afterwards: a client built with a credential (new WebClient(process.env.SLACK_BOT_TOKEN)) carried the secret into its API responses, and untrusted-tool-selection fired on a tool object that only reads uploaded documents (one held-out TP had been caught for that wrong reason). Because held-out cases were read to find them, the re-score is not a held-out measurement: dev 100.0% / 96.0% (unchanged), held-out 57.1% recall (8/14) / 80.0% precision (2 FP). Results are in benchmarks/agent/results/ (baseline.json, after.json, post-review.json).

The remaining dev false positive is a support bot whose refund tool only files a refund proposal for human review: the code is structurally the same as issuing the refund, and nothing in it says a person approves the request.