Make better AppSec decisions in a world of AI and agents—with Chris Romeo.
ISSUE 84 · OCTOBER 14, 2026 · SIX-MINUTE READ
A clean AI review still needs evidence.
AI can help you review more code, and GitHub’s commit-count stats show that September 2026 had more than five times the commits of September 2025. Your release decision still needs a clear account of what was checked, what holds, and what remains unknown.
YOUR 30-SECOND APPSEC BRIEFING
The decision: What evidence do you need before accepting an AI security review?
What AI changes: A convincing explanation can arrive faster than anyone can validate its assumptions.
The action: Pick one security claim, trace the relevant code, and test it with one test that should pass and one that should fail.
CHRIS’S TAKE
Ask what the review actually checked.
Imagine a pull request comes back with an AI review that says authorization looks good. The explanation is tidy. The code compiles. The normal tests pass. You’re ready to merge.
Then someone asks a simple question: can a user from one customer account read another customer’s invoice? The review never traced that path. Nobody tried it. The reassuring summary has answered a broader question than the evidence supports.
LLM responses can vary between runs, even with identical prompts. This could create a false sense of security if we aren’t careful. Prompt engineering is a powerful discipline, and you can write prompts to guide the agent toward potential problems, but even solid prompt engineering doesn't guarantee identical output.
AppSec teams need to focus on coverage. AI can read code, suggest attack paths, and help build checks. That is useful work. We still have to decide whether those checks cover the property we care about.
In our latest conversation with Brook Schoenfield, we discuss how easy it is to accept an AI answer that feels complete. Brook also describes using AI to trace whether a weakness is reachable in code. That gives us a practical direction: ask for something we can inspect.
For that invoice example, I want the request path, the caller’s identity, the account identifier, and the authorization decision. I want to see what happens when the caller changes the invoice ID. If another route reaches the same data, include it in the review too.
Running the model again may expose another path or challenge an assumption. Agreement between runs can still repeat the same blind spot. Keep the useful analysis, then check the code and the behavior.
A test is evidence with limits. One blocked request does not prove every route is protected. Record the scope: which endpoint, which identities, which data, and which version of the code. That makes the next review easier to assess.
The team should be able to explain the control after the agent session ends. Keep the reasoning with the change. Give the unresolved questions an owner. If the check exposes a weakness, fix it and rerun the check.
TRY THIS WITH YOUR TEAM · 20 MINUTES
Turn one reassuring claim into a check.
Choose a recent AI-assisted pull request with an authorization claim. Use a local or isolated test environment with synthetic data and two test accounts. Have a developer explain the expected access rule before opening the AI review.
Ask the agent: (this is an example, so you’ll have to modify “invoices” to match whatever piece of data you land on to test)
Review this change for the rule that a user may read only invoices belonging to their own account. Trace the request to the authorization check and data access. Cite files and functions, list assumptions and alternate paths, and propose one allowed case and one denied case. Do not modify files or access live services. If you cannot establish the rule, say what evidence is missing.
Trace it: Open the cited code. Does the account restriction reach the actual data query? Check alternate entry points the model identifies.
Exercise it: With the developer, run the project’s supported test path for an invoice owned by the caller and an invoice owned by the other test account. Confirm the denied case returns no protected data. Record results and environment.
Keep the evidence: Attach the code references and test results to the pull request. Assign any gap to an owner; add a regression test when a fix is needed.
Leave with one sentence: “We checked ___ on commit ___ using ___; we still need to verify ___.”
Know someone adopting AI code review? Forward this issue and try the exercise together.
Worthwhile Security Reads
Three references for turning an AI security claim into an inspectable control. These are practical background reads to keep nearby.
SECURITY FUNDAMENTALS · SALTZER AND SCHROEDER
The classic paper explains fail-safe defaults and complete mediation: deny access unless it is permitted, and check authority whenever an object is accessed. This is often cited, but I added it here in case you’ve never read it. Saltzer and Schroeder’s work still holds up in the age of AI Security.
CHRIS’S TAKE
Follow the request all the way.
Apply those principles to the invoice example. A check on the first screen needs to hold at data access, too. Ask what happens through a second endpoint or a background task.
Put it to work: Trace one protected object through every route your team knows can reach it.
AI MODEL WELFARE · ANTHROPIC
Anthropic’s forthcoming policy, effective November 12, prohibits sustained, needless abuse toward its models. Violations can lead to restricted or terminated access. To me, Anthropic’s conversation style and model-welfare policies encourage anthropomorphism. I’d prefer these systems to behave more like machines.
CHRIS’S TAKE
Define the boundary before enforcing it.
My AppSec question: how do you distinguish abuse from adversarial testing? Teams need clear rules they can apply when testing model behavior—and a way to challenge enforcement mistakes.
Put it to work: Review your AI provider’s testing rules before your next red-team exercise.
WORKLOAD IDENTITY · SPIFFE
SPIFFE defines standards for workload identity, including short-lived identity documents that services can use to authenticate to other workloads. I see SPIFFE as a promising foundation for agent identity. Giving agents distinct identities can help us distinguish their actions from a developer’s.
CHRIS’S TAKE
Knowing the caller is the start.
A verified workload identity lets you identify the caller. Your service still needs a policy for what that caller can do. Keep those decisions visible when an agent starts calling tools.
Put it to work: Write down the identity and allowed operation for one agent-to-service call.
From the Podcasts
Checking AI output and enforcing agent access: the newest full episodes available as of October 7.
Application Security Podcast
Brook S.E. Schoenfield joins Chris and Robert to discuss AI threat modeling, the risk of losing understanding of AI-written code, and using AI to trace reachable weaknesses. Listen for how your team can keep improving security without waiting for a perfect model.
 | Quick clip The Illusion of Omniscient PerfectionBrook explains why a friendly, confident AI answer can feel more complete than it is. A quick reminder to inspect the evidence. |
AI Security Table
The Security Table is now the AI Security Table. Chris, Izar, and Matt discuss agent identity, SPIFFE, least privilege, where access control belongs, and what nondeterministic review means for security engineers. Bring the conversation back to controls your team can enforce and results it can check.
 | Quick clip Sign the Letter or Answer to the MachinesA lighter moment from the table: will the machines remember who signed against them? The crew gives the joke a name—the Brockman effect. |
Where to Find Chris
I’ll be at OWASP Global AppSec USA in San Francisco, November 5–6. I’m moderating a keynote debate and recording a live episode of The Application Security Podcast. If you’re there, come say hello—and join us for the debate and live recording.
ONE QUESTION FOR YOU
What evidence would make you trust an AI security review?
Hit reply with one check your team uses—or one claim you’re still struggling to verify.
— Chris
Securing AI. Using AI for security. AppSec judgment that connects the two.
