Structured Logprobs
GitHub
A new Python library called structured-logprobs adds confidence scores to OpenAI's structured JSON outputs. Now you can see exactly how sure the model was about each field it filled in, not just trust the output blindly.
Based on reporting by GitHub — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Structured Outputs solved one annoying problem with LLMs: getting them to actually follow your JSON schema instead of hallucinating a field or inventing an enum value that doesn't exist. But it never told you how confident the model was about the values it chose. A new open-source library, structured-logprobs, tries to close that gap by attaching token-level log probabilities to each field in an OpenAI chat completion response.
The mechanics are straightforward. You send a request to something like gpt-4o-2024-08-06 with logprobs enabled and a JSON schema attached via response_format. Then you run the completion through one of two functions: add_logprobs, which appends a separate log_probs dictionary mapping each key to its probability, or add_logprobs_inline, which stitches the probabilities directly into the JSON content next to the values they describe. In the library's own example, a model asked for the capital of France, two nice colors, and a die roll returns something like -5.5e-07 for "capital_of_France" but -0.48 for "die_shows" — a clear signal that the model is nearly certain about Paris and considerably less sure about which digit it just made up.
That gap matters. A near-zero log probability means the token was essentially the only plausible choice; a value like -0.48 means the model was hedging between options even though the schema forced it to commit to one. For anything resembling classification, extraction, or automated decision-making, that distinction is the difference between trusting a field outright and flagging it for a human to double-check.
Under the hood, the library relies on a character-to-token mapping function to line up JSON string positions with the underlying token stream, which is fiddlier than it sounds given how tokenizers chop up punctuation and multi-character values. It's a narrow tool, built for a specific pain point in production LLM pipelines rather than a general-purpose reliability framework. But for teams already leaning on Structured Outputs to pull fields out of documents, forms, or user text, it turns a black-box JSON blob into something with at least a rough confidence gradient attached.
My take — AI-written commentary, not fact-checked reporting
This is exactly the kind of unglamorous tooling that actually moves LLM applications toward production, and it deserves more attention than the next benchmark-chasing model release. Structured Outputs without confidence scores was always a trap — it made hallucinated values look just as tidy as correct ones. I'd like to see this pattern become a standard API feature rather than a community bolt-on, because right now it only works for OpenAI's models, and the industry needs consistent uncertainty signals across providers, not one-off libraries plugging gaps closed vendors leave open.
Read more about this at: GitHub