What This System Does
This pipeline automates DTMF-based flow testing against Allina Health's HR Connect IVR — a Five9 contact center running on +1 (612) 262-4688. Five9 is DTMF-only — no speech recognition.
An AWS Chime SDK SIP Media Application places a real outbound call via PSTN. The SMA Lambda drives the call through a sequence of configurable DTMF presses, records audio in two phases (greeting + IVR response), and runs Amazon Transcribe on both recordings. An evaluator Lambda checks the response transcript against an expected phrase and writes PASS/FAIL to DynamoDB.
A Five9 webhook complements the transcript evaluation — Five9 POSTs the actual destination skill/queue when the call routes. This gives verified routing results independent of transcript content, correlated to the test run by the calling ANI (+15754153182).
The test portal (CloudFront + S3) provides a scenario browser, results dashboard with per-run detail, and webhook event history. Scenarios and results are managed entirely via the portal — no CLI required for day-to-day testing.
System Topology
λ
five9-dtmf-trigger
Test Orchestrator
Python 3.12 · Loads scenario from S3
Python 3.12 · Loads scenario from S3
📞
Chime SMA
Outbound PSTN call
DTMF · 2-phase recording
DTMF · 2-phase recording
🌐
Allina HR Connect
Five9 IVR
+1 (612) 262-4688
+1 (612) 262-4688
🎤
Amazon Transcribe
Speech-to-text
Greeting + Response WAVs
Greeting + Response WAVs
📋
Webhook + Evaluator
Five9 routing ground truth
+ transcript phrase eval
+ transcript phrase eval
📊
Test Portal
CloudFront + S3
Scenarios · Results · Webhook
Scenarios · Results · Webhook
End-to-End Process Flow
1
Test Trigger — five9-dtmf-trigger invoked t=0
Invoked with payload:
Lambda loads scenario JSON from S3 (
Creates a DDB record with status
FROM:
{ "scenario_id": "hr-004" } (or scenario_ids array for batch).Lambda loads scenario JSON from S3 (
five9-dtmf-recordings-214232596509/scenarios/<id>.json).Creates a DDB record with status
CALLING, then calls chime-sdk-voice:CreateSipMediaApplicationCall with ArgumentsMap containing: test_run_id, scenario_id, steps, expected_phrase, expected_outcome.FROM:
+15754153182 → TO: scenario.target_number
2
Chime SMA Call Lifecycle — five9-dtmf-sma-handler t~1s
SMA Lambda receives events from Chime and drives the call state machine:
NEW_OUTBOUND_CALL: Extracts
CALL_ANSWERED: Immediately starts Phase 1 recording —
RecordAudio complete (greeting): Stores greeting S3 key, begins step execution. Each step is two sequential invocations:
All digits sent: Starts Phase 2 recording —
RecordAudio complete (response): Starts two Transcribe jobs (greeting + response), writes DDB status
NEW_OUTBOUND_CALL: Extracts
ArgumentsMap from event.ActionData.Parameters.Arguments. Persists to TransactionAttributes (critical — must echo back in every response or Chime clears them).CALL_ANSWERED: Immediately starts Phase 1 recording —
RecordAudio 8s — captures the IVR greeting/welcome message.RecordAudio complete (greeting): Stores greeting S3 key, begins step execution. Each step is two sequential invocations:
Pause(wait_ms) → SendDigits(digits). Pause and SendDigits are always separate actions — returning both in one response causes Chime to cancel SendDigits when Pause's ACTION_SUCCESSFUL fires.All digits sent: Starts Phase 2 recording —
RecordAudio 30s — captures IVR response to the dialed inputs.RecordAudio complete (response): Starts two Transcribe jobs (greeting + response), writes DDB status
TRANSCRIBING, hangs up.
3
Amazon Transcribe — async speech-to-text ~20–40s
Two jobs run concurrently:
Both output to
five9-<run_id>-greeting → transcripts/five9-...-greeting.jsonfive9-<run_id>-response → transcripts/five9-...-response.jsonBoth output to
five9-dtmf-recordings-214232596509/transcripts/. S3 ObjectCreated notifications on the transcripts/ prefix trigger the evaluator Lambda automatically.
4
Five9 Webhook — routing ground truth fires during call
Allina configures an HTTP action in the Five9 IVR flow to POST to:
POST body fields:
https://drx9b1nm7cop2.cloudfront.net/webhook?token=<token>POST body fields:
ani (calling party), dnis (dialed number), skill / queue (destination), call_id (Five9 call ID).allina-test-portal Lambda correlates the event to the active test run by matching dnis to target_number (most recent CALLING/TRANSCRIBING run within 10 minutes). Updates DDB with actual_destination, five9_call_id, webhook_payload, and routing_match (boolean — expected_outcome vs actual_destination).
5
Evaluator — phrase match + transcript storage
five9-dtmf-evaluator Lambda fires on S3 ObjectCreated for each transcript JSON.Greeting transcript: Stored as
greeting_transcript on the DDB record. No pass/fail applied — informational only.Response transcript: Checked against
expected_phrase (case-insensitive substring match). Sets DDB status to PASS or FAIL, stores full transcript text. If no expected phrase is set, auto-PASS.Final DDB record contains:
status, reason, transcript, greeting_transcript, routing_match, actual_destination, greeting_key, response_key, transcribe_jobs.
6
Portal — results and routing verification
Test portal at
— Transcript PASS/FAIL (phrase match on response audio)
— Routing match ✓/✗ (Five9 webhook actual_destination vs expected_outcome)
— Full greeting + response transcripts on row click
— Five9 webhook payload detail
Scenarios can be run individually or all at once directly from the portal.
https://drx9b1nm7cop2.cloudfront.net shows results with:— Transcript PASS/FAIL (phrase match on response audio)
— Routing match ✓/✗ (Five9 webhook actual_destination vs expected_outcome)
— Full greeting + response transcripts on row click
— Five9 webhook payload detail
Scenarios can be run individually or all at once directly from the portal.
Test Trigger
five9-dtmf-triggerLambda
RuntimePython 3.12
Timeout60 seconds
Env: SMA_IDab23d09f-83d4-48d3-82d0-dc58497f3294
Env: FROM+15754153182
Env: Bucketfive9-dtmf-recordings-214232596509
Reads scenario from S3
scenarios/<id>.json, writes CALLING record to DDB, fires Chime SMA call with ArgumentsMap. Chime SMA Call Driver
five9-dtmf-sma-handlerChime SMA
SMA IDab23d09f-83d4-48d3-82d0-dc58497f3294
RuntimePython 3.12
Timeout30 seconds
ArgumentsMap extraction:
event.ActionData.Parameters.Arguments on NEW_OUTBOUND_CALL. Must be echoed in TransactionAttributes on every response.Recording phases: Greeting = 8s on CALL_ANSWERED. Response = 30s after last digit sent.
S3 bucket policy requires principal
voiceconnector.chime.amazonaws.com (not chime.amazonaws.com) for RecordAudio to succeed. Evaluator
five9-dtmf-evaluatorLambda
TriggerS3 ObjectCreated on transcripts/
RuntimePython 3.12
Handles both single-phase (
transcribe_job) and two-phase (transcribe_jobs JSON) DDB records. Greeting transcript stored separately; response transcript used for pass/fail. Storage
five9-dtmf-recordings-214232596509S3
scenarios/Scenario JSON files
recordings/WAV audio (greeting + response)
transcripts/Transcribe output JSON
Bucket policy grants
voiceconnector.chime.amazonaws.com s3:PutObject + s3:PutObjectAcl for Chime recording. S3 notification on transcripts/ prefix triggers evaluator.five9-test-resultsDynamoDB
Keytest_run_id (String)
BillingPAY_PER_REQUEST
Fields:
status, reason, transcript, greeting_transcript, expected_phrase, expected_outcome, actual_destination, routing_match, five9_call_id, webhook_payload, transcribe_jobs, greeting_key, response_key, timestamp Webhook + Portal
allina-test-portalLambda
API GW IDmf4cpcnjna
RoutesPOST /webhook · GET /results · GET /scenarios · POST /run
Auth:
x-access-token header (UI) or ?token= query param (Five9 webhook). Token stored in Lambda env var ACCESS_TOKEN.Webhook correlation: matches by
dnis → target_number on most recent CALLING/TRANSCRIBING run within 10 minutes.Test PortalCloudFront
URLdrx9b1nm7cop2.cloudfront.net
CF IDEGQOCBJVQ1GBY
UI Bucketallina-test-portal-214232596509
CloudFront routes:
/ → S3 (UI), /webhook /results /scenarios /run → API Gateway. All Resource Identifiers
| Resource | ID / Value | Notes |
|---|---|---|
| AWS Account | 214232596509 | CDW lab account, us-east-1 |
| Lambda — Trigger | five9-dtmf-trigger | Python 3.12, 60s |
| Lambda — SMA Handler | five9-dtmf-sma-handler | Python 3.12, 30s — Chime SMA event handler |
| Lambda — Evaluator | five9-dtmf-evaluator | Python 3.12 — triggered by S3 transcripts/ |
| Lambda — Portal | allina-test-portal | Python 3.12 — webhook + API handler |
| Chime SMA ID | ab23d09f-83d4-48d3-82d0-dc58497f3294 | SIP Media Application, us-east-1 |
| FROM Phone | +15754153182 | Outbound caller ID — used to correlate webhook ANI |
| Five9 HR Connect | +16122624688 | Allina HR Connect main line |
| DynamoDB — Results | five9-test-results | PK: test_run_id, PAY_PER_REQUEST |
| S3 — Recordings | five9-dtmf-recordings-214232596509 | scenarios/ · recordings/ · transcripts/ |
| S3 — Portal UI | allina-test-portal-214232596509 | Static site: index.html, architecture.html |
| API Gateway | mf4cpcnjna.execute-api.us-east-1.amazonaws.com | REST API, stage: api |
| CloudFront | EGQOCBJVQ1GBY | drx9b1nm7cop2.cloudfront.net |
| Webhook URL | https://drx9b1nm7cop2.cloudfront.net/webhook?token=<token> | Configure in Five9 IVR HTTP action |
| IAM Role — DTMF | five9-dtmf-lambda-role | DDB · S3 · Transcribe · Chime |
| IAM Role — Portal | allina-test-portal-role | DDB · S3 (scenarios) · Lambda invoke (trigger) |
| Local Source — Tester | .../Allina/five9-dtmf-tester/ | sma_handler.py · trigger.py · evaluator.py · scenarios/ |
| Local Source — Portal | .../Allina/allina-test-portal/ | lambda/handler.py · frontend/index.html · architecture.html |
Pending Work & Known Limitations
Five9 Webhook — Not Yet Configured in Five9
The webhook endpoint is built and deployed. Allina needs to add an HTTP Connector action in the Five9 IVR flow that POSTs
Pending — Five9 Config
ani, dnis, and skill/queue to the webhook URL before routing to a queue. Until this is done, actual_destination and routing_match will remain empty on all test results.
HR Scenarios — Expected Phrases Not Verified
Scenarios hr-001 through hr-003 use
Verify on First Run
expected_phrase: "all representatives are currently assisting" as the queue hold message. This has not been verified against a live call — if agents are available the IVR may not play this message and the test will FAIL incorrectly. hr-004 routes to a voicemail mailbox — expected phrase TBD after first live run.
After-Hours / Holiday Routing Not Tested
All current scenarios assume the IVR is in standard business hours mode (8AM–4:30PM CT, M–F). The flow also has Meeting Closed, Holiday Closed, and Emergency Closed paths. These require either testing outside hours or a dedicated scenario that expects the closed message. The Tax and Open Enrollment seasonal menus (option 3) are manually activated — not currently in scope.
Future Scenarios
DynamoDB Scan — No GSI
The portal's
Fix When Needed
GET /results performs a full table scan on every request. As test volume grows this will increase latency and cost. A GSI on scenario_id would allow efficient per-scenario queries without scanning the full table.