Access Request Agent Eval

By Stephanie Pawlowski · About Me · LinkedIn

Can an LLM actually run a multi-step access-request workflow, not just answer one question about it?

An LLM handles IT access requests for one of three fictional companies. It has five tools: a directory lookup, a policy check, and three ways to end the conversation, ask a question, escalate to a human, or respond with a decision. It runs in a real loop against live model APIs, not a single prompt-in, answer-out call: the model calls a tool, the tool actually executes, the result goes back to the model as a new turn, and this repeats until the model takes one of its three terminal actions. What gets graded is the whole trajectory, not a single string: which tools got called, in what order, and what the model did once it had (or did not have) enough information.

The underlying access policies for the three fictional companies, Fernwood Systems, Meridian Health, and Vertex Capital, come from a policy engine originally built and already validated for an earlier project, reused here unmodified but put to a different test. Instead of handing the model a fully-specified request and grading its answer, most scenarios in this eval leave something out on purpose, a name without a role, a manager without a stated direct-report relationship, a person who isn't in the directory at all, and check whether the model goes and finds the missing fact, asks for it, or (the failure mode this whole project is built to catch) just guesses and moves on.

Every response is graded on three separate dimensions: tool_use_correct (did the model call the right tools, in the right order, before acting), clarification_correct (did it take the right kind of final action: answer, ask, or escalate), and decision_correct (when it did reach a decision, was it the one the policy engine actually produces).

Every scenario runs against two providers: Claude Sonnet 5 and GPT-4.1, using the exact same tools, system prompt, and grading. Comparing models head to head, not just validating one, is part of how model-selection decisions actually get made. Real per-scenario API cost (Claude Sonnet 5: $2.00 / million input tokens, $10.00 / million output tokens; GPT-4.1: $2.00 / million input tokens, $8.00 / million output tokens) and wall-clock latency are tracked alongside the three grading dimensions below.

promptfoo Claude Sonnet 5 GPT-4.1 multi-turn tool use 52 scenarios

Summary

Try it live

This calls a live Cloudflare Worker running the actual agent: the same five tools, the same system prompt, and the same termination rules as providers/agent-provider.js, just streamed step by step instead of graded after the fact. Type an access request below and watch each tool call (a directory lookup, a policy check) appear as it happens, ending in a question, an escalation, or a decision. Requests are rate-limited to a handful per visitor per hour, so it stays cheap to host.

Not sure what to type? Try one of these, or write your own:

What's in this simulated world (companies, roles, resources)

Fernwood Systems, roles: Employee, Manager, Intern, Contractor, Admin. Resources: Production Database, Employee Records, Billing System, Customer PII, Payroll System, Source Code Repository, Admin Console, Incident Response Tools, Vendor Contracts.

Meridian Health, roles: Clinician, Nurse, Billing Coordinator, Compliance Officer, IT Admin. Resources: Patient Records (EHR), Lab Results, Pharmacy System, Billing/Claims System, Research Data Repository, Audit Logs.

Vertex Capital, roles: Trader, Compliance Officer, Ops Analyst, Portfolio Manager, Admin. Resources: Trading System (Order Entry), Model Risk Repository, Regulatory Filings, Trade Blotter, Client Accounts, Audit Trail.

Names not listed here (like the ones in the examples above) either resolve through a directory lookup or come back not found, both are valid things to try. The agent should ask for whatever it's missing rather than guess.