Your decision model returns confidence. Which number should your threshold use?
Separate selected-option probability, normalized confidence and ordered ratings before setting a decision-model threshold. Work through a fictional example.
Published · Updated
Read the quantity before choosing the cutoff
Baseten's October 29 joey-1 launch makes this a practical integration question. Its announcement describes a label and calibrated probability returned in one forward pass, with the same API across joey-1, elephant-1 and mammoth-1. MarkTechPost explains the decision-model format; The Next Web notes that the tests behind the vendor's strongest-under-3B claim had not been published. Those reports do not independently establish accuracy on your tickets.[1][3][4]
The October 21 Base Labs technical report covers mammoth-1 and elephant-1. For a choice question, it defines confidence as c = (pmax - 1/k) / (1 - 1/k), where pmax is the largest option probability and k is the option count. This formula needs k greater than one. It maps a uniform distribution to zero and all probability on one option to one.[2]
Keep a response-semantics card beside the routing rule. For choice, record the selected option, its probability and the separate confidence. For yes/no, identify the probability of true. For an ordered rating, retain the level definitions and the returned distribution. A rating's expected level describes its position on that defined scale; it does not directly state the probability that a classification is correct.[2]
Predict the result for two fictional responses
Suppose a fictional ticket classifier returns two options with probabilities 0.8 and 0.2. A second fictional response has five options with probabilities 0.8, 0.05, 0.05, 0.05 and 0.05. These are invented distributions for a static exercise, not observed model outputs. Before reading the calculation, decide whether a selected-probability cutoff of 0.7 accepts both responses.[2]
Both pass that probability rule: the selected probability is 0.8 in each. Their normalized confidence differs. With two options, c = (0.8 - 0.5) / 0.5 = 0.6. With five options, c = (0.8 - 0.2) / 0.8 = 0.75. Applying 0.7 to confidence accepts only the five-option response. That is a different, stricter decision in this example; it does not demonstrate unsafe over-acceptance.[2]
Option count belongs in the response contract because it affects this normalization. Renaming, splitting or merging categories also changes the question the model answers. Equal top probabilities alone cannot establish equal empirical correctness across those tasks. Record the taxonomy version alongside the field and cutoff, rather than carrying a threshold into a revised category set without checking it.[2]
Check the probability on held-out decisions
The technical report says calibration is trained and evaluated on probability p alone. Its training objective includes a Brier calibration term. That objective does not establish calibration on your task, and the larger-model report is not an independent joey-1 evaluation. The joey announcement describes a shared family API and training recipe; keep those compatibility claims separate from evidence about your own labels.[1][2][3]
For a proposed local check, freeze the model revision, question wording, category definitions and labeling rules. Reserve examples that did not inform tuning or threshold selection. Group selected probabilities into declared intervals. For each interval, retain the number of examples and the number labeled correct, so a seemingly accurate interval with few examples remains visibly uncertain. This evaluation plan has not been executed here.[2]
Choose candidate cutoffs on a development set according to the consequence of a wrong route. On a separate holdout, report accepted decisions, their mistakes and coverage: the share of all examples accepted. Record what happens to the rejected cases. Baseten describes escalation to a larger model, while reporting also discusses human review. Neither destination guarantees that an uncertain ticket will be resolved correctly.[1][3]
The reusable result is a small contract: question type, selected field, option set, cutoff meaning and held-out evidence. If you deliberately choose confidence instead of probability, evaluate that exact rule and name it accurately. No source supplies a universal cutoff or proves target-domain reliability. Correct field selection answers the semantic question; labeled decisions are still needed to evaluate the application.[2][4]