test service_tests::test_python3_callback_z6821_q41607 ... FAILED

failures:

---- service_tests::test_python3_callback_z6821_q41607 stdout ----

thread 'service_tests::test_python3_callback_z6821_q41607' (1248470) panicked at evaluator-layer/tests/test_service.rs:953:9:
assertion `left == right` failed
  left: 500
 right: 200
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Diagnosis: test_python3_callback_z6821_q41607

Root cause: the RustPython interpreter process aborts itself — it is not a host-side resource limit. The captured wasm backtrace is:

rustpython_common::refcount::refcount_overflow → std::process::abort → wasm `unreachable` trap

When the executor aborts, it writes no result line, so make_mapped_result("") fails and endpoints.rs returns a bare HTTP 500 (test expects 200).

What I ruled out

- Fuel exhaustion (the guess in the T417164 TODO), memory-page limit, and wall-clock timeout — the run dies in ~6 s with no fuel/memory error, and the trap is an explicit abort, not a   
fuel/OOM trap. The wasm stack is already generous (16 MB, executor.rs:199), so it's also not a native stack overflow.

What actually happens (traced with stage markers in the live executor)

The callback round-trip gets remarkably far before dying:
1. callback response received (177 KB, nesting depth 142) → json.loads OK
2. serialization.deserialize OK (its list-spine walk is iterative; the 144-element Z881 lists collapse to flat Python lists — the deserialized structure is only depth 12)
3. user function returns the value → serialization.serialize OK
4. abort during the final result-encoding/output stage, operating on the re-serialized ~15,600-node structure.

It is not simple recursion depth

My first backtrace pointed at the generator-based json.dumps encoder (_make_iterencode's yield from chain = the Coro::send_none frames). But when I replaced json.dumps with a
verified-equivalent iterative encoder, the abort persisted — it just relocated to a dict lookup (refcount_overflow inside Dict::get_hint, shallow stack). So the trigger is
reference-count pressure inside RustPython on this specific large, mixed structure, not call-stack depth.

It is RustPython-specific and data-shape-specific

- CPython processes the identical payload through all four stages with no trouble → this is a RustPython bug/limitation, not a logic error in the executor.
- It's data-dependent: the full result (15,613 nodes) aborts; the smaller "tweaked" result (7,489 nodes) succeeds — which is exactly why the tweaked file was used as the workaround.     
- I could not reproduce it with synthetic single-call inputs (nested lists to depth 500, nested objects to depth 140, flat lists of 20,000 elements all run fine). Only the real callback 
round-trip with this particular structure triggers it, so it's tied to the specific shape of the deserialized→re-serialized orchestrator result, not raw size or depth alone.

How to fix — options

1. [Not needed here]
2. Real fix (recommended to track under T430014): upgrade RustPython and rebuild rustpython_uncompiled.wasm, then re-validate against the full file. A refcount_overflow abort on a       
~15K-node structure that CPython handles trivially is almost certainly fixed in a newer RustPython. This is the only path that lets the test pass on the real full response.
3. Production robustness (worth doing independently of this test): today an executor self-abort surfaces as a bare HTTP 500 with no diagnostics. Worth (a) logging the wasm trap/backtrace
at error with the request_id, and (b) returning a structured error that distinguishes "interpreter aborted" from other failures — this is the real substance behind TODO (T417164), which 
currently only mentions fuel. Single-use executors mean the aborted instance is already dropped, so there's no contamination concern.

My suggestion: file the RustPython upgrade as the actual fix under T430014 (#2), and treat #3 as a separate observability improvement.

Stop the presses — I'm glad you asked before we built. The source tells us something decisive, and it kills the version-bump idea. Here's what I found.

  What refcount_overflow actually is
  
  RustPython packs its reference count into a single usize with this bit layout (crates/common/src/refcount.rs):

  [1 bit: destructed] [1 bit: reserved] [1 bit: leaked] [N bits: weak_count] [M bits: strong_count]
  64-bit: N=30, M=31
  32-bit: N=14, M=15  
  const STRONG: usize = (1 << STRONG_WIDTH) - 1;   // the overflow trigger

  inc() does fetch_add(1) then aborts if the strong count hits STRONG.

  The critical fact: your executor is compiled to wasm32-wasip1, so usize is 32 bits → the strong refcount is only 15 bits wide → it overflows the instant any single object reaches 32,768 simultaneous references. On a 64-bit host this
  same code has a 31-bit count (~2.1 billion) and would never trip — which is exactly why CPython and any 64-bit Python handle the identical data fine, but the wasm32 RustPython aborts.

  That also quantifies the threshold cleanly: the tweaked result (~7,500 nodes) stays under 32,767 references-to-some-shared-object; the full result (~15,600 nodes, which re-expands during serialization into far more dicts all sharing
  interned keys like "Z1K1" and shared type-dicts) pushes some heavily-shared object past 32,767. It's a hard numeric ceiling, not a transient stack/recursion thing — consistent with the iterative encoder not helping and the abort
  landing in a dict lookup.
  
  Does main fix it? No.

  I diffed refcount.rs at 0.5.0 vs main. The only differences are cosmetic:
  - a cfg_select! refactor of the abort statement, and
  - an added #[must_use] attribute.

  The bit-layout constants — FLAG_BITS=3, the 15-bit strong width on 32-bit — are byte-for-byte identical in main. So building from main HEAD would reproduce the exact same abort. The hypothesis behind suggestion #2 was wrong, and I'd
  have wasted ~20 minutes and a few GB proving it. Let's not run the build.

  One more relevant detail: RustPython has a leaked (immortal/interned) bit, but the hot-path inc() increments leaked objects anyway (only safe_inc() checks leaked()). CPython solved precisely this class of problem with true immortal
  objects (PEP 683) that are never inc/dec'd. RustPython hasn't — so even interned singletons can overflow on wasm32. That's a genuine upstream gap, but it's not implemented in main, so there's nothing to pull.

  So where does that leave the real options

  - Version bump (was #2): ruled out. Confirmed by source.
  - Patch/fork RustPython to either (a) widen the strong count by stealing bits (there's a spare "reserved" bit; rebalancing weak↔strong is also possible), or (b) make inc/dec truly skip leaked objects. Both are real fixes but mean
  maintaining a RustPython fork — a much bigger commitment than a tag bump, and definitely a team decision.
  - Reduce the materialized object/reference count in the executor's Python serialization — e.g., the executor's own convert_list_to_zlist shares one list_type dict across all nodes of a list, concentrating references; but the dominant
  pressure is interned key strings, which you can't easily avoid. Realistically this only buys headroom, not a guarantee.
  - Treat it as a known wasm32 limitation (recommended): keep the test on the tweaked file, document why (the 15-bit wasm32 strong-count ceiling), and make the production path fail gracefully instead of a bare HTTP 500 — which is
  exactly suggestion #3, and now clearly the highest-value next step, because this will happen in production on large-enough results regardless of RustPython version.

The pipeline in the executor, for this callback, is: receive the orchestrator response → json.loads → deserialize (into Python objects) → user function returns it → serialize (rebuild   
  ZObject form) → JSON-encode to a string → write to stdout. My instrumentation showed json.loads, deserialize, and serialize all completing, and the abort landing in the final 
  JSON-encode-to-string stage (it crashed inside json.dumps; when I swapped in an iterative encoder it crashed there instead). So it's the last step — encoding the fully-built large result  into the JSON string destined for stdout.
  
  But here's the important nuance about why, because it's not really the encoding algorithm that's at fault (that's why the iterative encoder didn't help):

  - The refcount overflow is a whole-structure-in-memory problem, not a per-operation one. By the encode stage, the entire serialized result is materialized at once — thousands of dict nodes, all sharing a small number of interned      
  objects (most importantly key strings like "Z1K1", present in essentially every node).
  - On wasm32, RustPython's strong refcount is 15 bits → ceiling 32,767. When the materialized structure holds more than ~32,767 simultaneous references to one shared interned object, the next inc() on it aborts. The encode stage is    
  just where that threshold happens to get crossed, because that's when the whole thing is live and the encoder is walking it, touching those shared keys.

  Certainty level: the stage (serialize/encode of the large result) and the mechanism (15-bit wasm32 refcount ceiling) are confirmed — the latter straight from the RustPython source. The specific object that overflows (I'm saying       
  "interned key string like Z1K1") is inferred, not proven — I traced it to the stage and the ceiling, but didn't add instrumentation to name the exact object. I could, if it matters, but it wouldn't change the conclusions.