Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions content/user-guide/inference/decision-models/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -270,3 +270,8 @@ Interleave a System One model with a generative one — typed guards, tool fan-o
- [Typed decisions for agentic pipelines](../../../tutorials/agents/system-one-agents/_index): three pipelines and a measured A/B against a one-shot generative baseline.
- [Agents](../../agents/_index): the loop these decisions most often sit inside.
- [Tasks](../../tasks/_index): the unit each call runs in.
{{< variant union >}}
{{< markdown >}}
- [Decisions on the Inference Gateway](../inference-gateway/decisions): call `POST /v1/decisions` through the gateway, with TypeSafe models, hosted LLMs, or your own vLLM and SGLang apps.
{{< /markdown >}}
{{< /variant >}}
3 changes: 3 additions & 0 deletions content/user-guide/inference/inference-gateway/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,4 +76,7 @@ Reach the gateway from any local tool, with or without exposing it publicly.
**[Guardrails](./guardrails/_index)**
Check every prompt and every answer with your own code.

**[Decisions](./decisions)**
Ask a model typed questions about a state with `POST /v1/decisions`.

{{< subpage-cards >}}
68 changes: 68 additions & 0 deletions content/user-guide/inference/inference-gateway/decisions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
---
title: Decisions
description: Send a state and named questions to the Inference Gateway's POST /v1/decisions endpoint and get a typed answer to each.
icon: sliders
weight: 7
variants: -flyte +union
---

# Decisions

`POST /v1/decisions` asks a model several typed questions about one piece of state in a single call. Each named question gets its own answer: a probability, a chosen option, or a score. Use it for the narrow judgments described in [Decision models](../decision-models/_index), such as routing, moderation, and guards, through the same endpoint and virtual keys as the rest of your model traffic.

## Call the endpoint

```bash
curl https://<your-gateway-endpoint>/v1/decisions \
-H "Authorization: Bearer sk-un-..." \
-H "Content-Type: application/json" \
-d '{
"model": "typesafe/<model>",
"state": "Customer: I was charged twice for one order.",
"questions": {
"is_billing": {
"kind": "noul",
"instructions": "Is this a billing issue?"
},
"urgency": {
"kind": "choice",
"instructions": "How urgent is it?",
"criteria": {
"low": "can wait",
"high": "reply today"
}
}
}
}'
```

If the gateway is behind Union sign-in, run `flyte proxy app` first and call `http://127.0.0.1:8600/v1/decisions` instead. See [Use from your laptop](./use-from-laptop).

Replace `<your-gateway-endpoint>` with your gateway's endpoint from **Settings → Inference Gateway**, `sk-un-...` with a virtual key, and `<model>` with a model your `typesafe` provider allows.

When the gateway has a deployed `typesafe` provider, or a vLLM or SGLang provider with **Answers decisions** on, the gateway's overview in **Settings → Inference Gateway** shows this command with your endpoint and a model filled in.

## The request

| Field | Meaning |
|---|---|
| `model` | The `provider/model` that answers, for example `typesafe/<model>` or `qwen/qwen3-8b`. The provider must answer decisions; see [Which providers answer decisions](#which-providers-answer-decisions). |
| `state` | The text the questions are about. |
| `questions` | The questions, keyed by name. |

Each question has a `kind` and `instructions`, the question itself:

| `kind` | Answer | `criteria` |
|---|---|---|
| `noul` | A probability. | Not needed. |
| `choice` | One of the options. | A mapping of each option to a description of when it applies. |

## Which providers answer decisions

| Provider | Answers decisions |
|---|---|
| `typesafe` | Yes. TypeSafe's decision models answer decisions only, not chat or completions. |
| Hosted LLM providers | Chat models that support tool calling. The gateway asks the model through one forced tool call. |
| vLLM and SGLang apps | Only when the provider was added with **Answers decisions** on. The gateway asks the model through one forced tool call on chat completions, so the model must support tool calling. With the setting off, the gateway refuses decision requests for that provider. See [Add a self-hosted model](./providers#add-a-self-hosted-model). |

A gateway on an older image may not serve `/v1/decisions`. If the gateway's overview shows an update available, redeploy the gateway.
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ You can put a model you serve on Union behind the gateway, next to hosted provid
| **Model id** | The model the app serves (its `--served-model-name`). Filled in from the app when it records one. |
| **Name** | The provider name, which callers use as the model prefix. |
| **Cost** | USD per 1 million input tokens and per 1 million output tokens. |
| **Answers decisions** | Lets callers send this app `POST /v1/decisions`. The model must support tool calling. |
| **Answers decisions** | Lets callers send this app `POST /v1/decisions`. The model must support tool calling. See [Decisions](./decisions). |
| **Weight** | The load-balancing weight among providers. Defaults to `1`. |

A provider named `qwen` whose app serves `qwen3-8b` is called as `qwen/qwen3-8b`.
Expand Down
Loading