User Details
User Details
- User Since
- Nov 7 2023, 9:56 PM (134 w, 4 d)
- Availability
- Available
- LDAP User
- Unknown
- MediaWiki User
- GChoi-WMF [ Global Accounts ]
Fri, Jun 5
Fri, Jun 5
ecarg added a comment to T427027: Abstract Wikipedia labeling does not fallback to `mul` when no language-specific label exists.
Will beging by identifying the places (in our WL code) where lang fallback is currently handled ad hoc. I can push up draft on a utility class we can route everything through
ecarg updated the task description for T428072: [rust] Publish wf-observability as a shared crate for Wikimedia Rust services.
Thu, Jun 4
Thu, Jun 4
Nvm/update: I was able to check with the Rust pods running in Prod eqiad right now and can confirm that:
- otel-collector is running and Rust evaluator logs are clean with no export errors.
- traces are appearing in Jaeger confirming OTel is working end-to-end
ecarg added a comment to T426794: OTel export failures on Rust evaluator in staging.
I don't think we can't fully verify OTel end-to-end in staging without the collector. That said, this does confirm:
- that Rust Eval is attempting to export
- that OTel config is wired up correctly
- OTEL_EXPORTER_OTLP_ENDPOINT is set and OTEL_SERVICE_NAME is set
- createCustomSpans: true is set
ecarg added a comment to T426794: OTel export failures on Rust evaluator in staging.
@Jdforrester-WMF
The errors are specific to the Rust Evals. I just checked again and the Node Eval logs (Staging) do not have OTel export errors.
Wed, Jun 3
Wed, Jun 3
ecarg closed T420426: separate out observability for rust services, a subtask of T421847: [Hypothesis] WE2.3.14 Begin Porting Orchestrator to Rust, as Resolved.
I see the status was changed to 'Open' again, was this intentional? Personally, comfortable resolving this; feel free to move around if other thoughts!
ecarg added a comment to T420426: separate out observability for rust services.
Are we comfortable resolving this task since observability is in its own workspace at this point anyway?
ecarg added a comment to T420426: separate out observability for rust services.
followup task to separate into own crate is done in https://phabricator.wikimedia.org/T428072
ecarg changed the status of T428072: [rust] Publish wf-observability as a shared crate for Wikimedia Rust services from Open to In Progress.
ecarg triaged T428072: [rust] Publish wf-observability as a shared crate for Wikimedia Rust services as Medium priority.
ecarg added a comment to T428072: [rust] Publish wf-observability as a shared crate for Wikimedia Rust services.
@Jdforrester-WMF decided to make this independent of that! I don't have strong opinion though!
ecarg updated the task description for T428072: [rust] Publish wf-observability as a shared crate for Wikimedia Rust services.
Tue, Jun 2
Tue, Jun 2
ecarg changed the status of T424310: Set links to pages on Abstract Wikipedia that don't yet exist as red/"new" from Open to In Progress.
ecarg changed the status of T424310: Set links to pages on Abstract Wikipedia that don't yet exist as red/"new", a subtask of T421835: [Hypothesis] WE2.3.7 Supporting Core Contributors, from Open to In Progress.
Mon, Jun 1
Mon, Jun 1
Fri, May 29
Fri, May 29
Thu, May 28
Thu, May 28
ecarg changed the status of T426330: Cache realize/fullyRealize/asZObject state on WFBase, a subtask of T413840: Optimize composition language V2, from Open to In Progress.
ecarg changed the status of T426330: Cache realize/fullyRealize/asZObject state on WFBase from Open to In Progress.
ecarg added a comment to T421996: Create an automation against the logs.
Thank you @herron
That makes sense, being able to see state over time is valuable. We have a normalized_message field in our index as well as message. I'm looking into building a Data Table visualization that aggregates on that field, grouped by service and severity level. Do you (or anyone reading this!) know if there's a way to have it send a periodic Slack summary of those aggregated messages, as a low-friction way to stay informed without active monitoring?
Wed, May 27
Wed, May 27
ecarg closed T423651: Show label alongside QID in Abstract Wikipedia page titles, a subtask of T421835: [Hypothesis] WE2.3.7 Supporting Core Contributors, as Resolved.
Tue, May 26
Tue, May 26
ecarg changed the status of T418886: Drop Z831 pre-defined Function and built-in Implementation, as we no longer want to support this model from Open to In Progress.
may need to drop definition of ; or switch uses of it in a way in the test code; drop in orch and then schemata
Fri, May 22
Fri, May 22
Thu, May 21
Thu, May 21
ecarg updated the task description for T427018: Decide whether non-WFObject subclasses should be validated in maybeValidateInternal().
ecarg moved T426335: Audit and tighten WFFunction.maybeValidateInternal from Ready to In Code review on the Abstract Wikipedia team (26Q4 (Apr–Jun)) board.
ecarg triaged T427019: Find a principled solution for skippable validators in maybeValidateInternal() as Low priority.
ecarg triaged T427018: Decide whether non-WFObject subclasses should be validated in maybeValidateInternal() as Low priority.
ecarg added a comment to T426335: Audit and tighten WFFunction.maybeValidateInternal.
The function only runs validation when the argument is a WFObject; other WFBase subclasses are silently skipped. Unclear whether WFList, WFFunctionCall, etc. should also flow through.
- Only skipping WFObject makes sense to me bc the others are more primitive-y. But it seems like WFList does have a type, which means it could be silent-skipped.
Wed, May 20
Wed, May 20
ecarg added a comment to T424310: Set links to pages on Abstract Wikipedia that don't yet exist as red/"new".
(currently pending discussion/confirmation with Product+Design before proceeding)
ecarg updated the task description for T426914: Replace numeric ZID heuristic in maybeValidateInternal with registry lookup.
ecarg triaged T426914: Replace numeric ZID heuristic in maybeValidateInternal with registry lookup as Medium priority.
ecarg changed the status of T426914: Replace numeric ZID heuristic in maybeValidateInternal with registry lookup from Open to In Progress.
ecarg changed the status of T426914: Replace numeric ZID heuristic in maybeValidateInternal with registry lookup, a subtask of T426335: Audit and tighten WFFunction.maybeValidateInternal, from Open to In Progress.
ecarg added a comment to T426335: Audit and tighten WFFunction.maybeValidateInternal.
I will tackle this part of this task first and defer the other two as they require team decision + design
ecarg added a comment to T426335: Audit and tighten WFFunction.maybeValidateInternal.
For:
The "is this a built-in validator we can skip" check is a heuristic on numeric ZID ranges (< 400, or 500 <= ZID < 10000). Brittle; should consult the actual built-in registry.
It seems like:
- WFFunctionCall.from — done by recently merged PR
- WFFunction.callWith — argument loop already parallelized (implementation loop is intentionally serial)
- WFList.realize/fullyRealize — already parallelized
- WFCompositionImpl.evaluate — nothing to parallelize
ecarg updated the task description for T426334: Parallelise remaining independent awaits with Promise.all.
reclaiming as mentioned
ecarg closed T424265: [AW] edit page title should show article title and id, a subtask of T423651: Show label alongside QID in Abstract Wikipedia page titles, as Resolved.
Completed, confirmed in Prod:
(removing qid in parens can be followup task)
Completed and confirmed in Prod:
ecarg closed T424095: [AW] view history title should show article title and id, a subtask of T423651: Show label alongside QID in Abstract Wikipedia page titles, as Resolved.
Tue, May 19
Tue, May 19
ecarg added a comment to T426794: OTel export failures on Rust evaluator in staging.
Update: The opentelemetry-collector namespace has no pods in staging:
ecarg renamed T426792: Route staging orchestrator to staging Rust evaluators and verify x-request-id propagation from requestId not propagated to log context in Rust evaluator to Route staging orchestrator to staging Rust evaluators and verify x-request-id propagation.
ecarg added a comment to T426792: Route staging orchestrator to staging Rust evaluators and verify x-request-id propagation.
Update: generateFunctionsMetrics: true is already set in staging, so x-request-id should be forwarded correctly once Orch is wired to route to the Rust evaluators. The evaluatorUri values in the staging Helm config currently point to localhost:6512/6513 (NodeJS Evals). A follow-up is needed to update these to point to the Rust evaluator services. At that point this ticket can be closed.
ecarg triaged T426792: Route staging orchestrator to staging Rust evaluators and verify x-request-id propagation as Medium priority.
ecarg added a comment to T426792: Route staging orchestrator to staging Rust evaluators and verify x-request-id propagation.
ex:
kubectl logs function-evaluator-evaluator-rust-javascript-585d445b6-rvsk8
Defaulted container "function-evaluator-evaluator-rust-javascript" out of: function-evaluator-evaluator-rust-javascript, function-evaluator-evaluator-rust-javascript-tls-proxy
{"@timestamp":"2026-05-19T14:51:29.402416505+00:00","http.method":null,"http.status_code":null,"log.level":"INFO","message":"rust evaluator logger initialized","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T14:51:29.402569550+00:00","http.method":null,"http.status_code":null,"log.level":"INFO","message":"rust evaluator service starting","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T14:51:31.713495083+00:00","http.method":null,"http.status_code":null,"log.level":"ERROR","message":"\"Failed during the export process\"","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T21:28:36.908639960+00:00","http.method":null,"http.status_code":null,"log.level":"INFO","message":"received request for evaluation request_id=unknown-request-id","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
state machine log:
state machine log: calling execute in executor..., original time: 2026-05-19T21:28:36.910Z
state machine log:
state machine log:
state machine log: ...finished calling execute in executor, original time: 2026-05-19T21:28:36.911Z
state machine log:
{"@timestamp":"2026-05-19T21:28:41.711551961+00:00","http.method":null,"http.status_code":null,"log.level":"ERROR","message":"\"Failed during the export process\"","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
ecarg@deploy1003:~$ kubectl logs function-evaluator-evaluator-rust-python-75fcdc788-q882q
Defaulted container "function-evaluator-evaluator-rust-python" out of: function-evaluator-evaluator-rust-python, function-evaluator-evaluator-rust-python-tls-proxy
{"@timestamp":"2026-05-19T14:51:43.407903255+00:00","http.method":null,"http.status_code":null,"log.level":"INFO","message":"rust evaluator logger initialized","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T14:51:43.408053786+00:00","http.method":null,"http.status_code":null,"log.level":"INFO","message":"rust evaluator service starting","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T14:51:43.595171131+00:00","http.method":null,"http.status_code":null,"log.level":"ERROR","message":"\"Failed during the export process\"","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T14:51:44.286470712+00:00","http.method":null,"http.status_code":null,"log.level":"ERROR","message":"","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T14:51:48.595993929+00:00","http.method":null,"http.status_code":null,"log.level":"ERROR","message":"\"Failed during the export process\"","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T14:54:01.238355760+00:00","http.method":null,"http.status_code":null,"log.level":"INFO","message":"received request for evaluation request_id=unknown-request-id","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
state machine log: calling execute in executor..., original time: 2026-05-19T14:54:01.241032
state machine log: ...finished calling execute in executor, original time: 2026-05-19T14:54:01.243123
{"@timestamp":"2026-05-19T14:54:03.596314815+00:00","http.method":null,"http.status_code":null,"log.level":"ERROR","message":"\"Failed during the export process\"","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T21:37:56.136814572+00:00","http.method":null,"http.status_code":null,"log.level":"INFO","message":"received request for evaluation request_id=unknown-request-id","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
state machine log: calling execute in executor..., original time: 2026-05-19T21:37:56.139517
state machine log: ...finished calling execute in executor, original time: 2026-05-19T21:37:56.141278
{"@timestamp":"2026-05-19T21:37:58.595575687+00:00","http.method":null,"http.status_code":null,"log.level":"ERROR","message":"\"Failed during the export process\"","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}ecarg added a comment to T426794: OTel export failures on Rust evaluator in staging.
ex:
kubectl logs function-evaluator-evaluator-rust-javascript-585d445b6-rvsk8
Defaulted container "function-evaluator-evaluator-rust-javascript" out of: function-evaluator-evaluator-rust-javascript, function-evaluator-evaluator-rust-javascript-tls-proxy
{"@timestamp":"2026-05-19T14:51:29.402416505+00:00","http.method":null,"http.status_code":null,"log.level":"INFO","message":"rust evaluator logger initialized","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T14:51:29.402569550+00:00","http.method":null,"http.status_code":null,"log.level":"INFO","message":"rust evaluator service starting","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T14:51:31.713495083+00:00","http.method":null,"http.status_code":null,"log.level":"ERROR","message":"\"Failed during the export process\"","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T21:28:36.908639960+00:00","http.method":null,"http.status_code":null,"log.level":"INFO","message":"received request for evaluation request_id=unknown-request-id","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
state machine log:
state machine log: calling execute in executor..., original time: 2026-05-19T21:28:36.910Z
state machine log:
state machine log:
state machine log: ...finished calling execute in executor, original time: 2026-05-19T21:28:36.911Z
state machine log:
{"@timestamp":"2026-05-19T21:28:41.711551961+00:00","http.method":null,"http.status_code":null,"log.level":"ERROR","message":"\"Failed during the export process\"","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
ecarg@deploy1003:~$ kubectl logs function-evaluator-evaluator-rust-python-75fcdc788-q882q
Defaulted container "function-evaluator-evaluator-rust-python" out of: function-evaluator-evaluator-rust-python, function-evaluator-evaluator-rust-python-tls-proxy
{"@timestamp":"2026-05-19T14:51:43.407903255+00:00","http.method":null,"http.status_code":null,"log.level":"INFO","message":"rust evaluator logger initialized","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T14:51:43.408053786+00:00","http.method":null,"http.status_code":null,"log.level":"INFO","message":"rust evaluator service starting","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T14:51:43.595171131+00:00","http.method":null,"http.status_code":null,"log.level":"ERROR","message":"\"Failed during the export process\"","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T14:51:44.286470712+00:00","http.method":null,"http.status_code":null,"log.level":"ERROR","message":"","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T14:51:48.595993929+00:00","http.method":null,"http.status_code":null,"log.level":"ERROR","message":"\"Failed during the export process\"","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T14:54:01.238355760+00:00","http.method":null,"http.status_code":null,"log.level":"INFO","message":"received request for evaluation request_id=unknown-request-id","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
state machine log: calling execute in executor..., original time: 2026-05-19T14:54:01.241032
state machine log: ...finished calling execute in executor, original time: 2026-05-19T14:54:01.243123
{"@timestamp":"2026-05-19T14:54:03.596314815+00:00","http.method":null,"http.status_code":null,"log.level":"ERROR","message":"\"Failed during the export process\"","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
{"@timestamp":"2026-05-19T21:37:56.136814572+00:00","http.method":null,"http.status_code":null,"log.level":"INFO","message":"received request for evaluation request_id=unknown-request-id","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}
state machine log: calling execute in executor..., original time: 2026-05-19T21:37:56.139517
state machine log: ...finished calling execute in executor, original time: 2026-05-19T21:37:56.141278
{"@timestamp":"2026-05-19T21:37:58.595575687+00:00","http.method":null,"http.status_code":null,"log.level":"ERROR","message":"\"Failed during the export process\"","service.name":"function-evaluator","span.id":null,"trace.id":null,"x-request-id":null}Thu, May 14
Thu, May 14
ecarg added a comment to T423651: Show label alongside QID in Abstract Wikipedia page titles.
Daphne's copyable qid looks good 👍
I have started on the patch for the title fetching
Execution proceeded despite no canonical connected implementation on Z825 — implementation appears to have been injected inline via Z8K4 in the request itself. Also, Z825K3 is passed as an unevaluated Z7 (call to Z20808) rather than a literal Z20420, which may have cause of the TypeError. Deferring for now.
Wed, May 13
Wed, May 13
ecarg changed the status of T426273: Cannot read properties of undefined (reading 'Z1K1') from Open to In Progress.
ecarg moved T426273: Cannot read properties of undefined (reading 'Z1K1') from Incoming to Ready on the Abstract Wikipedia team (26Q4 (Apr–Jun)) board.
Mon, May 11
Mon, May 11
ecarg changed the status of T423369: [Rust] Add OpenTelemetry trace export to /observability from Open to In Progress.
ecarg changed the status of T423369: [Rust] Add OpenTelemetry trace export to /observability, a subtask of T421849: [Hypothesis] WE2.3.16 Productionize Rust Evaluator and Deprecate Node Evaluator, from Open to In Progress.
According to recent news, this will be handled via Sloth alerting by SRE; will close this.
Fri, May 8
Fri, May 8
ecarg added a comment to T421996: Create an automation against the logs.
I believe we have used prometheus-es-exporter to alert on log levels before, here is an example. We then disabled this because they got too noisy and now only alert on rules defined here via Grafana.
May 7 2026
May 7 2026
ecarg added a comment to T424563: fix missing error type in orchestrate catch block.
I will update (and close as necessary) by the end of this week as I monitor the k8s pods because I was ooo the past week and didn't get to check before yesterday's deploy
Apr 29 2026
Apr 29 2026
ecarg moved T424563: fix missing error type in orchestrate catch block from Incoming to Ready to deploy on the Abstract Wikipedia team (26Q4 (Apr–Jun)) board.
ecarg moved T424563: fix missing error type in orchestrate catch block from To Triage to 26Q4 (Apr–Jun) on the Abstract Wikipedia team board.
Apr 27 2026
Apr 27 2026
ecarg moved T424265: [AW] edit page title should show article title and id from Ready to In Code review on the Abstract Wikipedia team (26Q4 (Apr–Jun)) board.
ecarg added a comment to T424563: fix missing error type in orchestrate catch block.
Update:
error.unknown_error does exist in error.js of schemata; I think that the error here is referring to the caught exception object from the catch ( error ) clause, not the imported error constant from error.js.
ecarg changed the status of T424563: fix missing error type in orchestrate catch block from Open to In Progress.
Apr 24 2026
Apr 24 2026
ecarg moved T423627: create new k8s service for (testing) Rust Evaluator from Ready to In Code review on the Abstract Wikipedia team (26Q4 (Apr–Jun)) board.
Apr 23 2026
Apr 23 2026
ecarg changed the status of T424265: [AW] edit page title should show article title and id, a subtask of T423651: Show label alongside QID in Abstract Wikipedia page titles, from Open to In Progress.
ecarg changed the status of T424265: [AW] edit page title should show article title and id from Open to In Progress.
ecarg moved T424265: [AW] edit page title should show article title and id from Incoming to Ready on the Abstract Wikipedia team (26Q4 (Apr–Jun)) board.