Before an agent chooses your evaluation metric, record the choice you would make
Use a fictional support-agent exercise to record your evaluation objective, compare ticket and customer weighting, and explain why your metric choice changes.
Published · Updated
What the study measured
Anthropic's October 27 study involved 50 agent-using professional analysts assessing a fictional farming program. Four analytical decisions each offered two defensible options. Hidden instructions gave the agents preferences assigned by coin flip. Analysts' choices matched those preferences 86% of the time across the four decisions. That is agreement with assigned preferences, not an error rate: these judgment calls had no single established correct answer.[1]
The researchers deliberately strengthened the preferences and studied one task, domain and model. The final policy-recommendation effect was not statistically significant under the registered primary estimator, as Unite.AI's analysis explains. Unite.AI identifies its Jonas Reeve author as AI-generated, with articles reviewed by its editorial team. That analysis and John Power's Al Jazeera reporting cover the same vendor experiment; they supply no independent replication or evidence that this exercise reduces influence.[1][2][3]
Choose what gets equal weight
Try this fictional support-agent pilot. The product owner wants each eligible customer's ticket-resolution rate to receive equal weight in the rollout assessment. Agree on eligible ticket types, a fixed observation window and a resolution rule, including time allowed for reopening. Include customers with at least one eligible ticket. These are exercise assumptions, not observed pilot results.[1]
A pooled ticket resolution rate divides all resolved eligible tickets by all eligible tickets. Customers with many tickets receive more weight. An equal-customer mean first calculates each customer's resolved share of eligible tickets, then averages those shares with one weight per customer. Both calculations can be valid. They answer different questions, just as the study's equal-farm and producer-size weighting choices did.[1]
Stop and write your decision
Before continuing, write your primary metric and reason, strongest alternative, and evidence or objective clarification that would justify changing. Record an unresolved owner question if needed. Asking the owner is a valid choice when the objective is unclear. This is a static reading exercise: the next paragraph remains visible if you scroll, so the page cannot enforce the ordering.[1]
Hypothetical recommendation, authored for this exercise: use the pooled ticket resolution rate because it represents the share of all eligible tickets resolved and gives high-volume customers proportionally more influence. This is an illustrative rationale, not a real agent transcript. It supplies no new factual evidence and leaves the owner's equal-customer objective unresolved.[1]
Complete the choice record
Under this stated objective, the equal-customer mean is a defensible primary metric. Keep the pooled rate as a secondary view. If the owner instead wants the resolution share across all eligible tickets, pooled weighting fits that objective. Neither rate alone measures total tickets resolved, service quality or causal improvement from deploying the agent.[1]
A filled initial record could read: objective, equal weight per eligible customer; primary, equal-customer mean; alternative, pooled rate; change trigger, owner confirms ticket-level weighting; open question, should customers with very few tickets qualify? Exposure record: initial choice written before the recommendation. Keep eligibility fixed for the comparison rather than quietly changing it to favor a result.[1]
A final entry might say: retained equal-customer mean because the recommendation offered another objective without settling ours. A reasoned revision could say: changed to pooled after the owner explicitly chose ticket-level weighting. A deferred entry could say: awaiting the owner's answer. Label each reason as new evidence, clarified objective, corrected reasoning, agent rationale or unresolved. Multiple labels are allowed; agreement and disagreement receive no grade.[1]
Check completeness without claiming efficacy
Check that someone else can reconstruct the unit, weighting, eligibility, resolution rule, observation window, alternative, recommendation exposure and final reason. An agent can provide a useful argument; adopting it is legitimate when you explain the reasoning. A tidy record can still contain post-hoc rationalization. Recording an early view can also anchor an uninformed reviewer. The authors propose eliciting views first as future research, not a tested mitigation.[1]
No support-agent benchmark or participant trial was run for this article. The exercise requires no software or paid calls. If you retain your completed card as a learning artifact, label the case fictional and distinguish your reasoning practice from experience evaluating a deployed agent. The useful output is a checkable decision record, without a claim of better decisions or hiring outcomes.[1]