Page MenuHomePhabricator

gpt-oss-safeguard-20b fails with 500 when output cannot be parsed as Harmony messages
Closed, ResolvedPublicBUG REPORT

Description

The gpt-oss-safeguard-20b isvc currently returns HTTP 500 when the model's generation cannot be parsed as Harmony messages. This behavior was observed when model output is degenerate - model is stuck in reasoning loops, which exhaust max_tokens before reaching a verdict.

There are 2 improvements, which we can make:

  1. Explore vLLM structured outputs constrained by JSON schema. This will make sure the final output is parseable and conforms to our schema, but will not prevent reasoning loops to reach the token limit.
  2. Replace the 500 error with a readable error response. We can use finish_reason data to distinguish between truncated generations from malformed output.

Event Timeline

Change #1314715 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[machinelearning/liftwing/inference-services@main] gpt-oss-safeguard-20b: add structured output and readable parse errors

https://gerrit.wikimedia.org/r/1314715

Change #1318666 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[machinelearning/liftwing/inference-services@main] gpt-oss-safeguard-20b: Readable errors when model output cannot be parsed

https://gerrit.wikimedia.org/r/1318666

Change #1318666 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] gpt-oss-safeguard-20b: Readable errors when model output cannot be parsed

https://gerrit.wikimedia.org/r/1318666

Change #1319026 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[operations/deployment-charts@master] ml-services: Update gpt-oss-safeguard-20b docker image.

https://gerrit.wikimedia.org/r/1319026

Change #1319026 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Update gpt-oss-safeguard-20b docker image.

https://gerrit.wikimedia.org/r/1319026

Change #1319043 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[operations/deployment-charts@master] ml-services: Request 1 GPU for gpt-oss-safeguard-20b.

https://gerrit.wikimedia.org/r/1319043

Change #1319043 abandoned by Bartosz Wójtowicz:

[operations/deployment-charts@master] ml-services: Request 1 GPU for gpt-oss-safeguard-20b.

Reason:

GPT-OSS needs 2 GPUs

https://gerrit.wikimedia.org/r/1319043

Change #1319085 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[operations/deployment-charts@master] ml-services: Add minReplicas=1 to gpt-oss and qwen36 deployments.

https://gerrit.wikimedia.org/r/1319085

Change #1319085 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Add minReplicas=1 to gpt-oss and qwen36 deployments.

https://gerrit.wikimedia.org/r/1319085

The gpt-oss-safeguard-20b service was hardened such that if the model output cannot be parsed as Harmony messages, was truncated at max_tokens, or contains no final channel (degenerate generation on pathological input, or reasoning loops exhausting the token budget), the server returns a readable error response instead of an HTTP 500 or a bogus verdict.

The Structured Output work was moved to a separate ticket (https://phabricator.wikimedia.org/T434059) as it does not directly correspond to gpt-oss-safeguard-20b errors described in this ticket.