Introducing DFIP: Faster inference for decision models

We've just published DFIP: Decision-Focused Inference Path for System One Models on arXiv, by Ismail Pelaseyed and Alan Zabihi. The paper studies how to reduce the time decision models take to return an answer. The code and benchmark results are open source.

A sustained 2.47× speedup could cut GPU compute costs by roughly 60%, turning $100 of compute into about $40. That estimate assumes the same workload and GPU price, with the saved time translating into less billed compute. The benchmark measures warm model latency; it doesn't establish that saving for a deployed service.

Faster decisions for agents

An agent reading a web page may need to check it for prompt injection before using it. A proposed tool call may need a risk check before it runs. Each check adds time before the agent can continue.

Decision models, also called System One models, return a choice, probability, or score for questions like these. An application supplies the possible answers and uses the result to route work or decide whether to escalate it. Security-One is our decision model for security triage.

Even a tiny answer can involve repeated CPU instructions to launch GPU work, plus calculations for output tokens the application never uses. DFIP tests where reducing that work makes decisions faster.

What DFIP changes

DFIP applies established inference techniques at the points each model supports:

• It reuses a captured GPU execution plan through CUDA graphs, reducing the overhead of launching the same operations again.
• For Liquid, it computes scores for the answer tokens the model needs, reducing work on unused output tokens.
• For GLiNER, it batches requests with the same question structure and similar input lengths, reducing wasted work from padding shorter inputs.

These are separate paths across different models. The PPLX-Decider headline comes from GPU execution replay. In the Liquid experiments, replay supplied most of the gain over uncompiled inference; computing fewer output scores alone made little difference to total request time.

The latency benchmarks

We compared DFIP with native inference, including native GPU extensions or the model's released compiler settings. The first three rows measure short yes/no requests with the model already loaded and warmed up. They exclude network transport and cold starts.

PPLX-Decider ran on an NVIDIA B200; the other models used an L40S. The rows compare each model with its own baseline, so they aren't a speed ranking between models. GLiNER measures completion time for a group of requests, rather than one yes/no request.

The PPLX-Decider result spans five completed container runs, with a 95% confidence interval of 1.88×–3.26×. Clef completed four of five planned runs, with an interval of 1.33×–1.71×; the fifth failed during the native baseline check. Liquid completed three runs and tied its compiled baseline. GLiNER's result comes from one worker and needs further replication. The paper reports the full comparisons.

Checking the answers, too

We audited 3,024 decisions per model for PPLX-Decider and Clef. Both preserved all 1,112 returned categorical choices. PPLX-Decider also had no observed changes in exposed probabilities, scores, or confidence in that audit.

Clef's scores and confidence did change. In a separate serving test, 21 matched response pairs crossed a confidence threshold of 0.9. If your application escalates below that threshold, the same winning answer can still lead to a different action. Faster inference needs checks against the fields your code uses.

Where the gains stop

Changing external traffic reversed the original replay gains for Decider and Clef. Capturing new execution plans and waiting in queues added latency; tests also recorded dropped requests and a CUDA error on the replay path. GLiNER finished its resident workload faster, but had worse p95 latency under arriving requests than compiled native batching. Requests near the slow end took longer.

DFIP is a research preview. The results show useful latency reductions for specific workloads, with serving limits still to solve. If you're building decisions into an agent, measure the whole request path with your own input lengths, traffic, and confidence thresholds.

Read the paper on arXiv, or use the code, raw results, and reproduction instructions on GitHub to inspect the benchmarks. The packaged client uses native inference by default; the acceleration experiments have separate research entry points.