The gpt-oss-safeguard-20b isvc currently returns HTTP 500 when the model's generation cannot be parsed as Harmony messages. This behavior was observed when model output is degenerate - model is stuck in reasoning loops, which exhaust max_tokens before reaching a verdict.
There are 2 improvements, which we can make:
- Explore vLLM structured outputs constrained by JSON schema. This will make sure the final output is parseable and conforms to our schema, but will not prevent reasoning loops to reach the token limit.
- Replace the 500 error with a readable error response. We can use finish_reason data to distinguish between truncated generations from malformed output.