Page MenuHomePhabricator
Create Task
Maniphest T395064

Log `NOT_STORED` responses from memcached `add` to monitor concurrent-population races
Open, MediumPublic
Actions

Description

Originally filed as "Improve the memcached concurrency treatment" and narrowed during investigation. The known cause of theMemcached.add returning NOT_STORED on back-to-back-with-same-key calls (extra setKeyValuePairSync invocations in the fetch path) was eliminated by T412326 (closed Resolved).

Remaining scope: add logging when theMemcached.add returns NOT_STORED, so we can monitor whether residual back-to-back same-key races still occur in production.

Acceptance criteria:

  • Log entries are emitted when theMemcached.add returns NOT_STORED (at a level appropriate for periodic review, not per-request noise).
  • Logs include enough context (cache key, call site) to identify which code paths are still racing.
  • After ~2–4 weeks of logging, post a brief comment summarising what production data shows so we can decide whether any further concurrency work is justified.

See DMartin-WMF's 2025-12-09 comment for the diagnostic detail.

Related Objects

Event Timeline

cmassaro created this task.May 22 2025, 5:35 PM
cmassaro added a parent task: T390549: Implement Object and Wikidata entity caching mechanisms in the function-orchestrator to drive up user experienced performance when making function calls.
Jdforrester-WMF triaged this task as High priority.May 28 2025, 3:37 PM
Jdforrester-WMF edited projects, added: OKR-Work, Abstract Wikipedia team (25Q4 (Apr–Jun)); removed: Abstract Wikipedia team.
Jdforrester-WMF moved this task from Incoming to Ready on the Abstract Wikipedia team (25Q4 (Apr–Jun)) board.Jun 11 2025, 3:50 PM
cmassaro added a project: Abstract Wikipedia Fix-It tasks.Jun 21 2025, 8:53 PM
cmassaro removed a parent task: T390549: Implement Object and Wikidata entity caching mechanisms in the function-orchestrator to drive up user experienced performance when making function calls.
DSantamaria edited projects, added: Abstract Wikipedia team; removed: Abstract Wikipedia team (25Q4 (Apr–Jun)), OKR-Work.Jul 9 2025, 1:34 PM
DSantamaria moved this task from To Triage to Engineering Backlog on the Abstract Wikipedia team board.
DMartin-WMF subscribed.EditedDec 9 2025, 5:47 PM

In local development/testing I've observed the following which is probably relevant to this ticket:

The unit test retrieval from wikilambda_fetch populates memcached results in two calls to setKeyValuePairSync, and, during the 2nd time through that function, the call to theMemcached.add returns an error (which unfortunately is not very informative; it just says NOT_STORED).

This also appears to be the case with these tests:

  • retrieving multiple items from memcached
  • ReferenceResolver re-retrieves missing values in memcached from wikilambda_fetch and populates

EDIT: the extra calls to setKeyValuePairSync that we know about will be eliminated by T412326. what remains here is to arrange for logging of these NOT_STORED returns, so we can monitor how often this situation (2 back-to-back calls to setKeyValuePairSync with the same key and the 2nd one calls gets before the first one has stored its value) continues to occur.

Jdforrester-WMF mentioned this in T391475: Consider whether we need an additional caching service if memcached isn't enough.May 13 2026, 8:11 PM
Jdforrester-WMF mentioned this in T400657: Consider not awaiting the secondary memcached population Promise.May 13 2026, 8:15 PM
Jdforrester-WMF renamed this task from Improve the memcached concurrency treatment to Log `NOT_STORED` responses from memcached `add` to monitor concurrent-population races.May 13 2026, 8:29 PM
Jdforrester-WMF lowered the priority of this task from High to Medium.
Jdforrester-WMF updated the task description. (Show Details)
Jdforrester-WMF subscribed.

Re-prioritising to Medium and re-scoping during an Engineering Backlog triage. The original framing has narrowed significantly since filing: DMartin's investigation comment (2025-12-09) found that the back-to-back setKeyValuePairSync calls producing NOT_STORED from theMemcached.add were caused by extra fetch-path calls, and T412326 (now closed Resolved) eliminated those. What remains is observability — logging the NOT_STORED returns so we can monitor whether residual races still occur in production. Rewriting the title and description to make that narrow deliverable visible: the previous framing was intimidating relative to the actual remaining scope, and a clearer card is easier for someone to pick up.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits