Thresholds

Myles is calibrated: across many decisions where it gives an option 0.9, that option is right about 90% of the time. The exception is committed yes/no answers, reported at 0.801 or 0.199, which are right about 70% of the time (see Question types). A calibrated probability is something you can act on directly. This page covers how: one threshold, two thresholds with a person in between, or choosing the action with the lowest expected cost.

Answers are probabilities and can be wrong. You decide how they are used and are responsible for the decisions made with them. For decisions that significantly affect people or are regulated, have a person review the answer first. See Outputs and limitations.

What a probability means here

Every answer carries a probability for every option. The probabilities were calibrated on held-out data the model never trained on, separately for each question type, so 0.9 means roughly the same thing whether the question is about routing, a policy or a safety check. Calibration describes many decisions together, not one: among answers given 0.9, about one in ten will be wrong.

Yes/no answers are committed: probabilities for uncertain answers are reported as at least 0.80 towards the chosen side (see noul). Set yes/no thresholds above 0.801 and below 0.199. A threshold between them, such as 0.7, would automate uncertain answers that a person should see.

One threshold

The simplest policy is to act automatically above a threshold and do nothing (or the safe default) below it. A higher threshold automates fewer cases with fewer errors.

if answers["spam"]["noul"] >= 0.95:
    move_to_spam(message)

Two thresholds and a person in between

When both outcomes lead to an action, use two thresholds. Confident answers are automated in either direction and the band in between goes to a person, with the probabilities attached so they can see why it came to them.

refund = answers["refund"]["noul"]
if refund >= 0.9:
    approve()
elif refund <= 0.1:
    decline()
else:
    send_to_a_person(case, answers)   # pass the probabilities along

Widen the band to send more to people and make fewer automatic mistakes. Narrow it to automate more. Watch the share that lands in the band: it is the work your team will see.

Choosing by cost

When mistakes cost different amounts, write down what each action costs for each true outcome and pick the action with the lowest expected cost. The probabilities are the weights. For a refund request:

COSTS = {  # action: (cost if allowed, cost if not allowed)
    "approve": (0, 40),
    "decline": (25, 0),
    "review": (3, 3),
}
 
def best_action(p_allowed):
    expected = {action: p_allowed * yes + (1 - p_allowed) * no
                for action, (yes, no) in COSTS.items()}
    return min(expected, key=expected.get)
 
best_action(answers["refund"]["noul"])   # 0.97 -> "approve", 0.85 -> "review", 0.02 -> "decline"

The same works for a choice question: one row per action, one column per label, and the expected cost of an action is the sum over labels of its cost times that label's probability. The costs are yours to set, and changing them moves the thresholds without touching the model.

Certified error rates

For a certified error rate, choose the error you can accept first, then find the threshold that keeps decisions inside it on one held-out set and check it on another. Set one threshold for each kind of input, because a routing decision and a safety check behave differently.

For the Myles-400M build behind our 91.8% result, on unseen test data: at a 5% target it keeps 73% of decisions with 2.7% error. At 2%, it keeps 40% with 1.4%. At 1%, it keeps 16% with 0.5%. Source: Myles-400M on unseen test data, September 2026. The method and full results are on the Models page.

What to log

To explain a decision later, keep a record of each one on your side:

• the X-Request-Id response header, which finds the call in our records without exposing its content;
• the model from the response;
• the probabilities you acted on;
• the thresholds or cost table in force at the time, with a version;
• the action taken, and who took it if a person did.

The hosted endpoint does not store the states, questions or answers of live requests (see Data), so your own log is the record of what was decided and why.

Check on your own data first

A threshold set on our data is a starting point, not a guarantee on yours. Before automating a decision:

1. Collect a few hundred real cases with the answer a careful person would give.
2. Run them through the API and keep the probabilities.
3. Choose thresholds or costs on half of them, then check the error rate and the share automated on the other half.
4. Repeat after any change to the question, the options or the model.

We help pilot partners set and certify thresholds against their own decisions, so talk to us if you would like help.