An evaluation framework and serverless CRM that measures agentic task performance across different model families on Amazon Bedrock.
Also known as aws-samples/sample-crm-agent-eval
- Stars
- 1
- Tools
- —
- Downloads, 30 days
- —
- Latest version
- —
- Last commit
- Nov 12, 2026
- License
- MIT-0
What is sample-crm-agent-eval MCP?
sample-crm-agent-eval is an evaluation framework and serverless CRM that measures agentic task performance across different model families on Amazon Bedrock.
Summary generated from the project's README
About sample-crm-agent-eval MCP server
Researchers and developers can evaluate how different model families perform on agentic tasks using this framework and serverless CRM. Built on AWS, the system uses Amazon Bedrock to run identical CRM tasks against Anthropic and OpenAI models, measuring specific metrics like correctness, write safety, tool use, and token counts. It functions as a measurement instrument rather than a production tool, providing a controlled environment to compare model behaviors.
The architecture utilizes Amazon Bedrock AgentCore Gateway to expose CRM tools and an assistant via two distinct MCP patterns. One pattern provides access to individual CRM tools through an identity interceptor and action Lambdas, while the second exposes the entire assistant as a single chat tool. Users interact with the system via an external MCP client or a React SPA, with all model calls authenticated through IAM credentials rather than vendor-specific API keys.
What you can do with it
- 1Compare model performanceRun the same set of CRM tasks across Anthropic and OpenAI models to compare their accuracy and efficiency.
- 2Test tool use accuracyEvaluate how effectively different models utilize specific CRM tools and follow tool-calling protocols.
- 3Verify agentic safetyMeasure the write safety of agents to ensure they do not perform unauthorized or high-risk data mutations.
- 4Benchmark token consumptionAnalyze the token counts used by different model families when completing complex agentic workflows.
Written from the project's README. Every use case is checked against a quote from it.
Configuration
CRM_DEMO_USER_PASSWORDCRM_EXPECTED_ACCOUNT_IDCRM_OPERATOR_PRINCIPAL_ARNSEVAL_SUBSTRATE
| Name | Description | |
|---|---|---|
| CRM_DEMO_USER_PASSWORD | — | |
| CRM_EXPECTED_ACCOUNT_ID | — | |
| CRM_OPERATOR_PRINCIPAL_ARNS | — | |
| EVAL_SUBSTRATE | — |
Automated checks
Ran Nov 15, 2026Full report →Not verified yet. A server needs a score of 80, no serious finding, and enough data for us to check. How checks work · What to fix for more points
- Source repository linked and reachableaws-samples/sample-crm-agent-eval is public and reachablegithub.com/aws-samples/sample-crm-agent-eval ↗Pass
- Package names this repository as its sourceNot checked
- Build provenance (Sigstore attestation)Not checked
- Open source license (OSI)MIT-0spdx.org/licenses/MIT-0.html ↗Pass
- Commit or release in the last yearLast commit or release 2026-11-12github.com/aws-samples/sample-crm-agent-eval ↗Pass
- OpenSSF ScorecardNot checked
- Package name not close to a popular packageNot checked
- No critical advisory in the package or its dependenciesNot checked
- No hidden instructions to the model0 tool descriptions and the README read; no override phrases, credential paths or hidden charactersPass
- Destructive tools are labelledNot checked
- Remote endpoint answers the MCP handshakeNot checked
README
View on GitHubAgentic CRM, MCP Patterns, and Eval Framework
A serverless CRM on AWS, a multi-agent assistant built on Strands Agents, and two MCP patterns on Amazon Bedrock AgentCore Gateway: one Gateway exposes the CRM tools, one per action Lambda behind an identity interceptor, and a second exposes the assistant itself as a single chat tool. Together they form a measurement instrument: the same agentic CRM tasks run against Anthropic and OpenAI model families on Amazon Bedrock, and the harness records correctness, write safety, tool use and token counts for each. It measures. It does not declare a winner.
This is a demo and a research instrument, not a production CRM.
Contents
- Architecture
- Prerequisites
- Install
- Configure
- Deploy
- Use
- Teardown
- Limitations
- Development
- Repository layout
- Documentation
- License
Architecture
- One enforcement boundary. All three entry paths reach the same CRM action Lambdas, and authorization is enforced in those handlers only. Each handler decodes the caller's identity from exactly one channel: Cognito authorizer claims, the identity the Gateway interceptor injects, or the user claims the agent forwards. A request with no identity is refused.
- One tool definition.
src/crm/toolspec/definitions.pygenerates the handler input validators, the Gateway's MCP tool definitions and the agents' tool registry. A gate fails the build if any copy diverges. - Agents.
src/agents/supervisor.pyhandles each chat turn with direct read tools and four subagents: research, mutation, analytics and composer. High-risk changes go through a proposal that a user approves or rejects. No tool lets an agent commit one directly.docs/agent-architecture.mddescribes the deployment, the agent workflow and one chat turn. - Relay.
src/relay/is a small Node.js Lambda. It verifies the Cognito token, calls the AgentCore Runtime with SigV4 and streams its server-sent events back unchanged. - Models. Every model call goes through Amazon Bedrock, signed with IAM credentials. The project reads and accepts no vendor or Bedrock API key.
- Tenants. Data is partitioned by tenant, and every index partition key includes the tenant.
- Hardening. Every Lambda and the API stage run active X-Ray tracing. Every CloudFront behaviour
carries the module's security-headers policy (HSTS,
X-Content-Type-Options,X-Frame-Options,Referrer-Policy,X-XSS-Protection) with an enforcedContent-Security-Policy. The CSP allows scripts only from the site itself plus the inline theme script, by hash, and allows connections only to the site and Cognito. Every versioned bucket underinfra/expires noncurrent versions (the state-backend buckets thatinfra/bootstrap_state_backend.shcreates have no lifecycle rule), and the CloudTrail buckets have ACLs disabled. The smoke checks send their HTTP requests through one opener that refuses any URL scheme buthttps; the JWKS fetch inscripts/smoke/cognito_claims.pygoes through PyJWT instead.docs/security-baseline.mdlists the controls the plan-time gate enforces;docs/development.mdcovers the static-analysis configuration.
| Tenant | Contents |
|---|---|
demo |
the seeded data you use in the UI |
canary |
never seeded; checks assert that it stays empty, which shows that eval sweeps stay inside their own tenant |
eval-<run_id> and eval-<run_id>-a<arm>r<repeat> |
created for one eval sweep, one per (arm, repeat) so no unit sees another's writes, and deleted when the sweep ends |
Prerequisites
| Tool | Version |
|---|---|
| Python | exactly the version in .python-version (3.13.18); a newer host python3 makes pip install -e . fail |
| Node.js | 26.13.0 for frontend/ (frontend/.nvmrc); 24 or later for src/relay/ (deployed on the nodejs24.x Lambda runtime) |
| Terraform | 1.17.0 or later (licensed under BUSL-1.1); the AWS provider is pinned to 6.71.0 |
| Docker | for DynamoDB Local (amazon/dynamodb-local:3.3.2, pinned by digest in scripts/run_local_dynamodb.sh) |
| ShellCheck | 0.11.0 (pinned in CI); make verify runs it |
| gitleaks | 8.30.1 (pinned in CI); optional locally, where check_secret_scan skips without it |
AWS CLI v2, jq, git, zip, unzip, npm |
for packaging and deployment |
To deploy, you also need:
- An AWS account with access to the Bedrock models listed under Configure.
- An existing VPC with at least two private subnets in different Availability Zones, their route tables, and one internet-routed public subnet for the NAT gateway.
- One or more private subnets for the AgentCore Runtime, in AgentCore-supported Availability Zones.
- An API Gateway CloudWatch Logs role, set once per account and region. The REST API stage writes
access logs, and API Gateway accepts that only when the account's
cloudwatchRoleArnsetting names an IAM role thatapigateway.amazonaws.comcan assume, with theAmazonAPIGatewayPushToCloudWatchLogsmanaged policy. The stack itself does not create it. Check withaws apigateway get-account --profile default --region <region> --no-cli-pager. - A region in
infra/regions.json. You choose the region; it must be one the map lists (scripts/region_map.py regionslists them). The bootstrap root, the pre-deploy network check and the Runtime smoke check all read their AgentCore-supported zones from the map. The first two refuse a region it does not list, and the smoke check fails on one. See Deploying to another region.
In an account that has none of these, scripts/bootstrap_account.sh creates the VPC, its subnets
and route tables, and the API Gateway logging role and setting. See
Deploying into a new account.
Install
python3.13 -m venv .venv # the interpreter whose version matches .python-version
# On a version mismatch this prints "wrong interpreter" and installs no Python packages.
{ [ "$(.venv/bin/python --version)" = "Python $(cat .python-version)" ] || { echo "wrong interpreter" >&2; false; }; } &&
.venv/bin/pip install --no-input --no-deps -r requirements.txt &&
.venv/bin/pip install --no-input --no-deps -e .
(cd src/relay && npm ci)
(cd frontend && npm ci)
scripts/run_local_dynamodb.sh start # DynamoDB Local on http://localhost:8000
requirements.txt is a fully pinned lock file, so install it with --no-deps.
Try it offline
These commands need no AWS credentials and cost nothing:
terraform -chdir=infra init -backend=false # providers for the plan-time tests
make verify # lint, types, unit, stub, integration and all offline gates
.venv/bin/python3 -m src.eval.harness # a stub eval sweep into results/stub/; no model call
Configure
AWS profile and region. scripts/deploy.sh, scripts/teardown.sh and
infra/bootstrap_state_backend.sh resolve the profile and region in this order: the --profile
and --region flags, then AWS_PROFILE and AWS_REGION, then default and us-east-1. Each
script passes the resolved pair to every aws and terraform call. The deploy and teardown scripts
refuse to run when AWS_ACCESS_KEY_ID or AWS_SESSION_TOKEN is set. Use a named profile.
Application code resolves the same way and then falls back to config/app.yaml (copy it from
config/app.yaml.example). The commands in this README write --region <region>: use any region
in infra/regions.json, for example us-east-1, and the same one in every command.
State backend (once per account). This script creates an encrypted S3 state bucket, an access
log bucket, a KMS key and a DynamoDB lock table, and writes the gitignored infra/backend.hcl:
CRM_EXPECTED_ACCOUNT_ID=<account-id> \
CRM_OPERATOR_PRINCIPAL_ARNS=arn:aws:iam::<account-id>:role/<operator-role> \
infra/bootstrap_state_backend.sh --profile default --region <region>
The script refuses to run without CRM_EXPECTED_ACCOUNT_ID (or --expected-account-id), and when the
profile resolves to a different account. It also requires the operator list, from
CRM_OPERATOR_PRINCIPAL_ARNS (comma-separated) or a repeatable --operator-principal-arn. Use the
same ARNs as crm_operator_principal_arns below. Every run applies key, bucket and lock-table policies
that deny the backend to every other principal, and it refuses unless the caller is in the list. The access-log bucket uses SSE-S3, because S3 server access
logging does not deliver to an SSE-KMS bucket; the state bucket and lock table use the KMS key.
Terraform variables. Copy the example and replace every placeholder:
cp infra/terraform.tfvars.example infra/terraform.tfvars
After scripts/bootstrap_account.sh, you don't fill in the six network keys yourself.
scripts/render_vpc_tfvars.py writes vpc_id, vpc_private_subnet_ids,
vpc_private_route_table_ids, vpc_nat_public_subnet_id, vpc_runtime_subnet_ids and
vpc_runtime_route_table_ids from the bootstrap root's outputs, and leaves every other line as it
was.
| Variable | Required | Meaning |
|---|---|---|
vpc_id |
yes | the existing VPC to deploy into |
vpc_private_subnet_ids |
yes | at least two private subnets, in different Availability Zones, for Lambda and endpoint interfaces |
vpc_private_route_table_ids |
yes | those subnets' route tables, which get a NAT route and gateway endpoints |
vpc_nat_gateway_enabled |
yes | create a NAT gateway; with false the relay cannot fetch Cognito's signing keys and rejects every request |
vpc_nat_public_subnet_id |
when vpc_nat_gateway_enabled is true |
an internet-routed subnet for the NAT gateway, not in the private list; the plan fails when NAT is enabled and this is unset |
api_origin_ingress_security_group_ids |
yes | the CloudFront VPC origin security group (see Deploy) |
vpc_runtime_subnet_ids |
yes | private subnets for the AgentCore Runtime; the Runtime runs in VPC mode and the plan fails when this list is empty |
vpc_runtime_route_table_ids |
yes, for every Runtime subnet route table not already in vpc_private_route_table_ids (defaults to []) |
the Runtime subnets' route tables; each gets the S3 and DynamoDB gateway-endpoint associations and no NAT route. The Runtime pulls its ECR image layers from S3, and its security group allows egress only to the S3/DynamoDB prefix lists and the interface endpoints, so a Runtime subnet whose route table is in neither list has no S3 path and the image pull fails |
crm_operator_principal_arns |
yes | the IAM role or user ARNs of the operators who deploy, seed and test the stack, for example ["arn:aws:iam::<account-id>:role/<operator-role>"] (the role ARN, not an assumed-role session ARN). Only these principals and the project's Lambda roles can use the DynamoDB tables and their KMS key, and the other stores listed under Limitations; a plan run by anyone else fails. See "DynamoDB principal boundary" and "Store principal boundary" in docs/runbook.md for changing operators and recovery |
vpc_cidr_block |
no | the VPC's CIDR; defaults to 10.20.0.0/20 and must match the VPC |
service, environment, owner, cost_center, data_classification |
no | resource tags |
infra/variables.tf documents every variable. The deploy script refuses to apply while any
REPLACE_ME placeholder remains, including in api_origin_ingress_security_group_ids and the
Runtime and NAT keys.
Bedrock model access. Enable these inference profiles in the region you deploy to:
global.anthropic.claude-opus-5-5, global.anthropic.claude-sonnet-5-5,
global.anthropic.claude-haiku-5-5, global.openai.gpt-6-astra, global.openai.gpt-6.1-sol,
global.openai.gpt-6.1-terra and global.openai.gpt-6-luna. Each model's capabilities (effort scale, prompt-cache minimum, inference
profile) are in config/model_capabilities.yaml; they were measured in us-east-1. The global.
profiles are cross-region, so Bedrock can serve these models from other regions. The Runtime's
Bedrock grant is derived from that file, so the Runtime may invoke exactly the models it lists. Each family's model and effort
per tier are in config/models.yaml:
| Tier | Anthropic | OpenAI |
|---|---|---|
| supervisor | Claude Sonnet 5.5, effort medium |
GPT-6.1 Sol, effort medium |
| subagent | Claude Sonnet 5.5, effort low |
GPT-6.1 Terra, effort low |
| tool | Claude Haiku 5.5, effort low |
GPT-6 Luna, effort low |
These are the defaults. OpenAI is the in-product default family (default_family in
config/models.yaml): a user with no saved settings gets the OpenAI column at every tier, in the
tuning controls, in GET /api/capabilities and in each chat turn whose tuning names no family, from
the web app or the MCP chat Gateway. A saved setting takes precedence over the default. Eval sweeps
are not affected: every sweep runs both families, whatever default_family says.
Claude Opus 5.5 and GPT-6 Astra stay in config/model_capabilities.yaml
and remain selectable in the tuning controls, but neither is a default. Claude Sonnet 5.5 is the
default at two tiers: its assignable_tiers entry in that file lists the tiers config/models.yaml
may assign it to.
Tuning. A saved setting and a chat turn from the web app can set each tier's family, model and effort, and nothing else: there is no temperature parameter. An eval arm sets its family and the supervisor's effort. The tuning wire format is schema 3.
- API: a tuning that still carries
temperatureis refused with HTTP 400validation_failed. - Streaming relay: such a tuning is refused with an SSE
errorframe whoseerror_kindisinvalid_request. - MCP: a chat turn takes no tuning and uses the caller's saved settings. A call carrying any
argument besides
messageandconversation_id,temperatureincluded, returns aninvalid_requesterror result. - Effort fallback: a tier whose effort is left unset uses the effort
config/models.yamlgives that tier, when the tier uses its configured model, and otherwise the model's own default. - Stored-row migration: saved settings are stored as version-2 rows; an older row is still read, with any recorded temperature ignored, and is rewritten at the next save. A saved setting that names a model no longer configured falls back to the defaults and says so.
Data retention. A live sweep with the OpenAI models needs the account's Bedrock data-retention
mode to allow them. Check it read-only with
aws bedrock get-account-data-retention --region <region>. The setting applies to the whole
account, so the account owner decides whether to change it. This project never changes it.
Deploy
Deploying creates billable resources: a NAT gateway, an AgentCore Runtime and Gateway, CloudFront, Lambda, API Gateway, DynamoDB, KMS keys, logs and Bedrock usage.
scripts/package_lambdas.sh # Python Lambda zips; clears infra/build/lambda except relay.zip
scripts/package_relay.sh # relay zip; the two can run in either order
CRM_EXPECTED_ACCOUNT_ID=<account-id> scripts/deploy.sh --profile default --region <region>
CRM_EXPECTED_ACCOUNT_ID is required. scripts/deploy.sh refuses before rendering, initialising
or applying anything when it is unset, is not 12 digits, differs from the account named in
infra/backend.hcl, or differs from the account the profile resolves to.
scripts/deploy.sh runs the pre-deploy checks, applies Terraform, publishes the SPA and its
config.json, and writes results/tf-outputs.json. Its other flags are --template PATH,
--no-auto-approve (needs a terminal) and --skip-frontend. It applies without prompting by
default.
First deployment. This takes two applies, because AWS creates the CloudFront VPC origin security group only when the VPC origin is first created:
-
Set
api_origin_ingress_security_group_idsto a security group that already exists in the VPC. The groups this stack creates do not exist yet, so create an empty placeholder group. A group with no members admits nothing to the load balancer:aws ec2 create-security-group --profile default --region <region> --no-cli-pager --vpc-id <vpc-id> \ --group-name crm-agent-eval-bootstrap --description "Placeholder for the first apply" \ --query GroupId --output textRun
scripts/deploy.sh./api/*fails until step 2 is done. -
Read the id of the group AWS created, set it in
infra/terraform.tfvarsand runscripts/deploy.shagain. After that apply you can delete the placeholder group:aws ec2 describe-security-groups --profile default --region <region> --no-cli-pager \ --filters Name=vpc-id,Values=<vpc-id> Name=group-name,Values=CloudFront-VPCOrigins-Service-SG \ --query 'SecurityGroups[].GroupId' --output text
After deploying, an unauthenticated request to https://<cloudfront_domain_name>/api/...
should return 401 or 403. A 502 or 504 means the API origin is unreachable. To run the full
deployed-resource checks:
make verify-deploy VERIFY_DEPLOY_ARGS="--i-understand-this-is-billable"
docs/runbook.md covers each step, its common failures and the per-layer checks.
Deploying into a new account
docs/runbook.md ("Deploying into a new account") is the full procedure. In order, it uses these
scripts:
| Script | What it does |
|---|---|
scripts/session_profile.sh |
write copies the current temporary session into a private directory holding one profile. exec runs a command with every other credential source removed, so no other profile can resolve. status shows the minutes left; delete removes it |
infra/bootstrap_state_backend.sh |
the state backend, as under Configure |
scripts/bootstrap_account.sh |
applies infra/bootstrap_account/: a VPC with one public, two private and two Runtime subnets, route-less private and Runtime route tables, the emptied default security group, and the API Gateway CloudWatch Logs role and setting. Then it runs scripts/render_vpc_tfvars.py and the adopted-VPC preflight. --destroy refuses while the main stack's state lists resources |
scripts/render_vpc_tfvars.py |
writes the six network keys of infra/terraform.tfvars from the bootstrap outputs; --check reports drift without writing |
scripts/deploy.sh |
the two applies under First deployment |
scripts/provision_gateway_principal.py |
writes the reps row of the MCP Gateway's service principal svc-mcp-gateway, so MCP calls can read the demo accounts |
scripts/migrate_supervisor_effort.py |
on a stack with saved user settings, moves every saved supervisor effort of low to medium, leaving every other value as saved. Dry run by default; --apply writes. See "Migrate saved supervisor effort" in docs/runbook.md |
scripts/bootstrap_account.sh, infra/bootstrap_state_backend.sh and scripts/deploy.sh all require
CRM_EXPECTED_ACCOUNT_ID, and each refuses when the profile resolves to another account.
Deploying to another region
infra/regions.json is the one list of supported regions and their AgentCore VPC zones. Read it with
.venv/bin/python3 scripts/region_map.py regions, or az-ids --region R for one region's zones. Pass
the region as --region R to every script. The bootstrap state backend puts the region in its bucket
names and in infra/backend.hcl. A second region therefore gets its own backend, and the single
gitignored infra/backend.hcl names one region at a time.
To add a US region:
- Read the "Supported Availability Zones" table on the AgentCore VPC page, and check on the regions page that Runtime and Gateway are available there.
- Add an entry to
infra/regions.json:agentcore_az_ids(at least 2), andbootstrapids taken from them. Updatesource.checked. - Run
.venv/bin/python3 scripts/region_map.py check, thenmake verify.
Only US commercial regions (us-east-N, us-west-N) are accepted. GovCloud is another partition and
is refused. If you add an ACM certificate or a CloudFront-scope WAF, it must be created in us-east-1
whatever the stack's region. Neither exists today.
sample-crm-agent-eval MCP questions
Is this a production CRM?
No, this is intended as a demo and a research instrument rather than a production-ready CRM.
What does the evaluation harness measure?
The harness records correctness, write safety, tool use, and token counts for different models.
Does the system require vendor or Bedrock API keys?
No, the project reads and accepts no vendor or Bedrock API keys; all model calls go through Amazon Bedrock using IAM credentials.