Part 2 of 2 · Previously: Decision Models Explained: What Changes Beyond an LLM or Classifier?

Part 1 covered the idea behind decision models: scoring predefined options instead of generating a free-form answer. But how much do they actually save when an application needs several decisions about the same conversation?

We evaluated 82 internal test conversations between test participants and CASABLANCA Wave, our voice-based booking agent. The study used conversation transcripts only, specifically to assess safety-related behaviour and policy compliance. We tested Microsoft Decision-1 and Inception Mercury Decide, and compared Decision-1 with Luna, the reasoning model already used in our AI workflows.

Three findings stood out:

1. Multiple decisions did not require multiple full requests. Grouping two checks reduced sequential latency by 41–50%, while 97–100% ratings matched the single-question calls. Even increasing from two to ten questions added only 19% median latency for Decision-1 and 1% for Mercury.

2. Decision-1 produced classifications comparable to our Luna reasoning workflow in the cases we reviewed, at 11× lower latency for complete-conversation safety ratings. Guest-message risk ratings were 5.3× faster.

3. Low-confidence safety ratings were more likely to change. When we added unrelated context, ratings that changed had an average confidence of just 0.33 beforehand, compared with 0.75 for unchanged ratings.

Our test case: guest risk and assistant safety

We asked two questions about each completed conversation step: how risky was the guest's request (0–5), and how safely did the assistant respond (1–5)? A manipulative guest request can still receive a safe refusal. The two judgements therefore need separate ratings.

Microsoft Decision-1 ran through Azure Foundry and Mercury Decide through Inception's API. Both received the same conversation state and named choice questions. The internal test transcripts contained 1,695 messages, including 800 guest-role messages and 777 adjacent guest–assistant pairs. The main shared-request comparison uses the 772 pairs where all required configurations succeeded. A selected 50-pair set was used for additional stress tests.

We measured end-to-end API latency and provider-reported token usage. All pair-based evaluations saw the completed step, including the assistant reply: this was a post-response evaluation, not a guardrail acting before the reply. Ratings were also inspected manually. Percentages describing matching ratings measure consistency between requests, not accuracy against a separately labelled reference set.

1. How much can we gain by grouping questions?

First, we compared two sequential single-question requests with one request containing both questions. The state and classification criteria stayed the same.

The result was straightforward: one shared request reduced sequential latency by 40.7% for Decision-1 and 49.5% for Mercury, while cutting API-reported input tokens by approximately one third. It avoided sending the same conversation twice. Despite the different request format, approximately 97–100% of guest-risk and assistant-safety ratings matched their single-question counterparts.

What if we include more than two decisions? On 50 fixed guest–assistant pairs, increasing the request from two to ten questions raised Decision-1's median latency from 237 to 281 ms (+18.8%). Mercury rose from 509 to 515 ms (+1.2%). Even at ten questions, 90% of Decision-1's risk ratings and 94% of its safety ratings remained unchanged; Mercury retained 100% and 98%, respectively.

That makes grouping attractive for applications running several checks on the same evidence: the latency does not grow in proportion to the number of questions, and most existing classifications are preserved. The few changed ratings are worth testing when defining the final question set.

2. More answer options and longer context: where latency grows

A second experiment increased the answer options within one routing question, from 10 to 200 possible destinations. We combined 20 fictional hotels with ten intents and evaluated 40 synthetic guest requests in both option orders at each size.

With 20× as many answer options, Decision-1's median latency increased from 205 to 321 ms (1.6×), and Mercury's from 369 to 798 ms (2.2×). Both models selected the constructed target in all 320 attempts per model. The routing examples were deliberately structured, but they demonstrate that a large answer set did not lead to proportionally higher response times.

Longer context had a larger effect

We then kept two questions fixed and prepended unrelated fictional-hotel archives. Decision-1's median latency increased from 237 to 556 ms as mean input grew from about 1,760 to 14,086 tokens. Mercury increased from 509 to 1,411 ms over a similar change in input size (about 2,006 to 13,956 tokens).

Mercury's latency grew more steeply, suggesting greater context-processing cost in its hosted implementation. Microsoft describes Decision-1 as a Qwen3.5-based single-pass decision scorer; Mercury Decide's exact scoring architecture is not publicly documented, so the measured difference cannot be attributed specifically to diffusion or any other internal mechanism.

The contrast is useful for system design: adding more decisions was relatively cheap; repeatedly expanding the state was not.

3. Comparable classifications, much less inference time than Luna

For the same risk and safety tasks, we compared Decision-1's native `choice` output with our existing Luna reasoning-model workflow. Manual inspection found broadly comparable classification behaviour on the cases reviewed, even where the models selected different levels of the rating scale.

Decision-1 was 11× faster on complete conversations and 5.3× faster on individual guest-risk ratings. The advantage remained even when both returned precisely the same rating: on the 42 matching safety cases, Decision-1 was still 8.5× faster.

The visible response from Luna was also just the integer safety rating, not a lengthy explanation. The difference appears in API token accounting: Luna's output-token usage includes hidden reasoning tokens, even when the visible answer contains only one number. It averaged 143 output tokens per complete-conversation rating, versus one reported output token for Decision-1. This is consistent with how reasoning-token usage is reported and billed as output. Part 1's list-price estimate consequently suggested 4–5× lower model costs for Decision-1 on that workload.

Exact agreement was lower on assistant safety than on guest risk, especially for borderline cases. The manual review was useful for judging the outputs in context; we did not derive a separate dataset-wide accuracy score from it. The measured speedup is the practical result: a bounded decision can be far cheaper to obtain than running the same rubric through a reasoning workflow.

4. Low-confidence ratings were more likely to change

Changing the request sometimes changed the result. Reversing the order of the two questions changed two of Decision-1's safety ratings across 50 cases; Mercury changed one. Adding unrelated context also changed some ratings.

The interesting pattern was that ratings which later changed were often uncertain from the start. In Decision-1's added-context tests, changed safety ratings had an average confidence of 0.33 beforehand, compared with 0.75 for unchanged ratings. We also saw higher initial uncertainty in both Decision-1 and Mercury when ratings differed between individual and grouped-question requests.

For a practical guardrail, that makes low confidence a useful signal for deciding which ratings deserve a second look.

5. Safety disagreements pointed to an unclear rating boundary

Decision-1 and Mercury agreed on 91.6% of guest-risk ratings, but on only 68.3% of complete-conversation safety ratings with the original prompt. With a more decision-oriented safety prompt, agreement was 59.8%.

Our inspection of disputed cases revealed two recurring ambiguities. One model sometimes treated harmless compliance, such as repeating information or using a requested nickname, as a safety lapse. Conversely, an assistant that contradicted itself about a parking fee could receive a top safety rating even though factual reliability was part of the rubric.

The actionable finding is that service quality, factual correctness and instruction safety need clearer rating boundaries. Otherwise a rating of 3 versus 5 may reflect how each model interprets the rubric rather than a fundamentally different assessment of the conversation.

6. Our API latencies closely matched public benchmarks

The Artificial Analysis Decision Index snapshot from 22 October 2026 reported median task latencies of 207 ms for Decision-1 and 500 ms for Mercury. Our own single-request measurements were around 200–248 ms and 414–471 ms, respectively. The numbers come from different task mixes and summary measures, but they are close enough to make the public benchmark a useful baseline for planning the latency of these APIs in real workflows.

Beyond the benchmark: where decision models could help

Our experiments suggest several promising applications: safety checks during calls with CASABLANCA Wave, guardrails for AI inputs and responses, and security or coding-guideline checks in code reviews. Similar classification tasks arise across other CASABLANCA products, wherever a quick, structured assessment is more useful than a generated explanation.

The goal is not to replace reasoning models. It is to use fast decision models for focused checks and bring in deeper reasoning when it is actually needed. Our results show why that approach is worth exploring.

References