Skip to main content
A scorecard turns every call into a graded record: yes/no criteria with points, summing to 100. After each call, an LLM reads the transcript and answers each criterion with pass, fail, or N/A. No partial credit. The scorecard gates the whole analysis, not just the score. Conversions, callbacks, and sentiment all ride the same grading pass. No scorecard means no score, no conversion rows, no sentiment — the call still records and transcribes, but nothing gets extracted.

​
Setup

Two places to put one, same shape:
  • Assistant level — config.grading.scorecard on create or update assistant. Grades every non-campaign call: web, inbound, direct API dials. Editable any time.
  • Campaign level — config.scorecard on create campaign. Grades that campaign’s calls. Strictly either/or: a campaign call uses the campaign’s scorecard only. A campaign without one is not graded at all, even when its assistant has one. Locks at launch, so get it right in draft.
The API enforces the shape at write time. Points are positive integers that sum to exactly 100. Every rule needs a unique ruleId, a title, and an instruction. scoringType is binary — anything else is a 400.

​
How grading runs

  • After the call ends, one LLM pass reads the transcript and answers each rule.
  • The transcript includes every tool call and its result — “the agent actually booked” is checkable.
  • The score is recomputed server-side from the per-rule answers. The model doesn’t get to invent a total.
  • N/A rules drop out of the score. The score is round(100 × earned ÷ applicable points), so 40 earned of 60 applicable is 67, not 40.
  • If no rule applied, the score is null. Check for null before you compare. In JavaScript, null < 60 is true.
  • Connected calls only. If the caller never spoke, it’s a silent pickup and skips grading.
  • If the grader’s output is malformed twice in a row, the call is marked analysis_failed and gets no score rather than a wrong one.

​
Writing criteria

One rule: a criterion must be answerable from the transcript alone. The grader cannot see your CRM, the caller’s intent, or whether the price quoted was correct. And the grader is instructed that unclear, ambiguous, or merely implied evidence scores zero. Vague criteria don’t score fuzzily. They score 0. Put the N/A condition in the instruction. The grader decides whether a rule applies from its instruction, like booked above: “Mark N/A if the wrong person answered.” Missing evidence or a skipped step is never N/A. It’s a fail. Observable beats vibes. Write criteria as acts: “read the reference number back,” “confirmed the right person,” “stayed in scope.” “Was empathetic” mostly has no citable moment, so it scores 0. One boundary: instructions are data, not commands. You can’t use an instruction to ask for partial credit, and nothing said on the call can override the grading protocol.

​
Points

Points are the only importance signal — put the weight on the outcome. In the example above, booked is 35 of 100. Five to eight rules is plenty: grading is all-or-nothing per call, and integer points summing to 100 get harder to split meaningfully with every rule you add.

​
Evidence

Set evidenceRequired: true and every answer (failures included) must carry a timestamp and a quote from the transcript. Every point becomes auditable. Cheap insurance. An N/A always carries a reason, and usually a quote, whatever the flag says.

​
Where scores land

  • Session detail has everything: the score and the full per-criterion breakdown, next to the transcript it came from. Each criterion carries its status (pass, fail, or not_applicable), plus a reason and evidence when the grader gave them. An N/A shows as N/A, never as a fail: passed is null. The grade is self-contained — rule titles and points are written into it, so you never need the rubric to read a result.
  • Campaign surfaces roll them up. Stats carries avgScore and the bands: great ≥ 80, ok 60–79, low < 60, plus analysisFailed. A call with a null score sits outside avgScore and the bands. The calls list shows each call’s score and sentiment.
  • The call.graded webhook fires with { callId, score } as each grade lands, and score can be null. Use the webhook to alert yourself on low scores — and call back the customers whose calls didn’t go well.