Static analysis inspects an agent skill's code and instructions without executing anything, which makes it fast, cheap, and a reasonable first pass for pre-deployment triage. It is also, on its own, an unreliable verdict on whether a skill is actually malicious. Both things are true at once, and the data on existing tools makes the gap concrete.

What the evaluation actually found

When we benchmarked five widely used static and rule-based scanning approaches against a balanced set of 100 malicious and 100 benign agent skills, the best individual tool reached an F1 score around 0.84, meaning even the strongest performer still misjudged roughly one in six verdicts. The weaker tools in the same test fared worse: one scored an F1 of just 0.02, correctly catching a single malicious skill out of a hundred while flagging zero false positives, which sounds precise until you realize it means the tool essentially never flags anything at all.

The noise problem was just as significant as the miss rate. Ninety one of the 100 completely benign skills in the test were flagged by two or more of the five tools evaluated. That is not a rounding error. That is a scanning approach that treats the majority of ordinary developer automation as suspicious.

Broad rule-triggering strategies can capture many malicious samples, but they also over-flag benign agent skills unless alert counts are carefully calibrated, a tradeoff that shows up consistently across every static tool we tested.

Where static analysis genuinely helps

None of this means static analysis is worthless. It's the only layer that can evaluate a skill before it ever executes, which matters enormously for a pre-merge gate on a pull request. A high-severity static match, an unmistakable credential-harvesting pattern or an obfuscated execution chain, is a legitimate reason to block a merge until a human reviews it. Static analysis is also the cheapest layer computationally, which is exactly why it belongs early in a pipeline, not as the only checkpoint in it.

What it structurally cannot do

Static analysis reads what's on disk. It cannot observe what a skill does when a remote MCP server responds a certain way, when a specific runtime condition triggers a hidden branch, or when a natural-language instruction is manipulative enough to steer an agent without ever containing a recognizable malicious code pattern. Obfuscation defeats keyword and signature matching by design. And because a skill can pass static review today and still change behavior tomorrow if a remote dependency updates, a single static scan is a snapshot, not a guarantee.

FAQ

Should security teams skip static analysis given its limitations? No. It's still the fastest, cheapest way to catch the obvious cases and to gate a pull request before a human ever needs to look at it. The limitation is treating a clean static scan as proof of safety rather than as one input among several.

Why do static scanners produce so many false positives on benign skills? Because the same patterns that indicate malicious behavior, network calls, file access, encoding, also appear constantly in ordinary, legitimate developer automation. Distinguishing intent requires context static analysis alone doesn't have.

What actually closes the gap? Runtime observation through sandboxed dynamic analysis, combined with a reasoning layer that can weigh evidence in context, catches the behavior and the intent that static rules structurally can't see.

Related reading: "Inside the Sandbox: What Dynamic Analysis Reveals About Agent Skills," "One Verdict Isn't Enough: The Case for Hybrid Agent Skill Scanning"