Updates from code code review and comments here:

1. Variant tagging uses unmanaged-data fields rather than custom keys:
   • resolved module group goes in in action_context
   • assigned variant in goes in action_subtype
2. Tagging happens in send(), not submitInteraction(), per @Sfaci's note above about the deprecation.

In T428679#12209147, @Michael wrote:

I'm still confused and have not yet understood everything here. It appears there is some duplication between what pdo/PersonalDashboard does and what mpo/TestKitchen does.

For experiment-time variant validation, devs additionally use TK's existing ?mpo=experiment:variant. pdo for pre-experiment dev validation when no TK config exists; mpo for post-experiment validation once the experiment is live.

I don't fully understand that. Literally ?mpo=foo:bar works perfectly fine out of the box and needs nothing on the test-kitchen side to be configured or started or live. What does pdo do that mpo does not do? If it is mainly suppressing product health metrics instrumentation, then I wonder if that should not rather be an upstreamed functionality in WikimediaEvents/TestKitchen.

I expect ExperimentTestKitchenManager and StaticExperimentManager to go away soon, in favor of just using WikimediaEvents/TestKitchen in CI and locally directly, with a FeatureManager wrapper for convenience. These classes are holdovers from a time when there was no proper experiment tooling. GEHomepageDefaultVariant is left-over complexity that needs to be cleaned up if we ever find the time. Part of the reasoning is that we want to test the integration with TestKitchen in our tests.

Part of what we need to do here is untangle the the application code -> database error side of this: wikilink shouldn't be trying to run userlist updates on former partners. the underlying links_urlpattern_collections foreignkey failure is still live in the code, and it'll happen again for the next partner we delete if we don't fix it.

SHOW ENGINE INNODB STATUS surfaced this as the latest foreign-key error on prod, timestamped 2026-08-11 23:36:31:

So we're still gathering link events, but we'll need to fix several issues that come together to make the userlist update job sad before we backfill and re-enable the cron. I'll write up a task list here.

hotfix to pause userlist update:
https://github.com/WikipediaLibrary/externallinks/commit/de5186f19220af1ef3221db0449abf84a0d2d3bc

verified that the update user list script is busted and has been for some time; it's been silently incrementing db ids and then discarding them until it passed the 32 bit integer limit. We have problems visible going back to April 18, 2025, which is the beginning of the discoverable record of this problem (eg. it could have started before then). That was when we fixed a whole other set of issues that were keeping the cron jobs from completing for different reasons.

merged

Verified that the api credentials are on the box where they are supposed to be and the job is firing every hour:

crons-1  | 2026-08-11T07:05:01.950708000Z [INFO] Executing users_update_lists.py
crons-1  | 2026-08-11T08:05:02.321202000Z [INFO] Executing users_update_lists.py
crons-1  | 2026-08-11T09:05:01.593929000Z [INFO] Executing users_update_lists.py
crons-1  | 2026-08-11T10:05:02.165584000Z [INFO] Executing users_update_lists.py
crons-1  | 2026-08-11T11:05:02.114458000Z [INFO] Executing users_update_lists.py
crons-1  | 2026-08-11T12:05:01.997846000Z [INFO] Executing users_update_lists.py

haven't verified what's actually happening inside those runs though

Experiments.php sketch

unstuck, now it needs a rebase!

In T433096#12189334, @MusikAnimal wrote:

The // NoneType TypeError no longer appears on the default worker after deploy

I'm still seeing such errors, but they might be a different issue:

worker-default-1  | [2026-08-05 03:43:54,335: ERROR/ForkPoolWorker-4] Task api.tasks.process_article[75c29b79-137d-4b89-8af3-4e5707ff2caa] raised unexpected: TypeError('sequence item 2: expected str instance, NoneType found')
worker-default-1  | Traceback (most recent call last):
worker-default-1  |   File "/app/api/utils_pickles.py", line 141, in pickle_load
worker-default-1  |     obj = pickle.load(gz)
worker-default-1  |   File "/usr/local/lib/python3.9/gzip.py", line 300, in read
worker-default-1  |     return self._buffer.read(size)
worker-default-1  |   File "/usr/local/lib/python3.9/_compression.py", line 68, in readinto
worker-default-1  |     data = self.read(len(byte_view))
worker-default-1  |   File "/usr/local/lib/python3.9/gzip.py", line 506, in read
worker-default-1  |     raise EOFError("Compressed file ended before the "
worker-default-1  | EOFError: Compressed file ended before the end-of-stream marker was reached
worker-default-1  | 
worker-default-1  | During handling of the above exception, another exception occurred:
worker-default-1  | 
worker-default-1  | Traceback (most recent call last):
worker-default-1  |   File "/app/api/tasks.py", line 28, in process_article_task
worker-default-1  |     with WPHandler(page_title, page_id=page_id, revision_id=revision_id, language=language, is_user_request=is_user_request) as wp:
worker-default-1  |   File "/app/api/handler.py", line 339, in __enter__
worker-default-1  |     self.wikiwho = pickle_load(self._load_path)
worker-default-1  |   File "/app/api/utils_pickles.py", line 141, in pickle_load
worker-default-1  |     obj = pickle.load(gz)
worker-default-1  |   File "/usr/local/lib/python3.9/site-packages/billiard/pool.py", line 229, in soft_timeout_sighandler
worker-default-1  |     raise SoftTimeLimitExceeded()
worker-default-1  | billiard.exceptions.SoftTimeLimitExceeded: SoftTimeLimitExceeded()
worker-default-1  | 
worker-default-1  | During handling of the above exception, another exception occurred:
worker-default-1  | 
worker-default-1  | Traceback (most recent call last):
worker-default-1  |   File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 385, in trace_task
worker-default-1  |     R = retval = fun(*args, **kwargs)
worker-default-1  |   File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 648, in __protected_call__
worker-default-1  |     return self.run(*args, **kwargs)
worker-default-1  |   File "/app/api/tasks.py", line 78, in process_article
worker-default-1  |     process_article_task(language, page_title, cache_key_timeout=default_task_soft_time_limit)
worker-default-1  |   File "/app/api/tasks.py", line 47, in process_article_task
worker-default-1  |     cache.delete(cache_key)
worker-default-1  |   File "/usr/local/lib/python3.9/site-packages/django/core/cache/backends/memcached.py", line 91, in delete
worker-default-1  |     key = self.make_key(key, version=version)
worker-default-1  |   File "/usr/local/lib/python3.9/site-packages/django/core/cache/backends/memcached.py", line 71, in make_key
worker-default-1  |     return force_str(super(BaseMemcachedCache, self).make_key(key, version))
worker-default-1  |   File "/usr/local/lib/python3.9/site-packages/django/core/cache/backends/base.py", line 102, in make_key
worker-default-1  |     new_key = self.key_func(key, self.key_prefix, version)
worker-default-1  |   File "/app/wikiwho_api/settings_base.py", line 274, in make_key
worker-default-1  |     return ':'.join([key_prefix, str(version), key])

@jsn.sherman Are you able to confirm this is a separate issue? if so, we can close this task I think.

Stalling on T433063; waiting on its patches to get merged

How the scoring works

@Chlod is a solid technical contributor who has demonstrated good judgement in the years we have known each other. Exactly the kind of human you want to have on hand for backports and troubleshooting. Let's do this!

Okay the new T433063 drops the mobile/desktop axis from rendering entirely, so nothing keys on the skin or wgMFMode for layout anymore. Detail (compact vs full) is now viewport-driven through a useViewport composable in the dashboard app; islands, integrations included, receive detail and never sniff for themselves, so the behaviour is consistent by construction and width-driven rather than skin- or wgMFMode-based. Util::isMobile() survives only as a server-side analytics label, out of the render path.

I had to make some choices along the way to get through the implementation:

normalized message error count as of backport was 10,273. It looks like it has stopped, but we'll keep this open for a bit.

thanks @kostajh for the quick patch and @MusikAnimal for the quick backport; let's keep an eye on the error rate now that the patch is out on wmf.12

In T432147#12148394, @MusikAnimal wrote:

Thank you so much for this, Jason! I learned a lot.

note that I initially set the env to "production" during vm setup, but I should have set it to "main" for this project. It didn't do any harm, but deploys weren't going to get through that way. I updated the instance metadata and the patches are auto deploying as expected.

I thought you were referring to a Django env var akin to Symfony's APP_ENV, in which case something like production made sense. I should have asked! Oh well, no harm no foul.

If you want, we can create a production branch so that it matches other Mod Tools repos. That might prevent accidental deploys and such.

note that I initially set the env to "production" during vm setup, but I should have set it to "main" for this project. It didn't do any harm, but deploys weren't going to get through that way. I updated the instance metadata and the patches are auto deploying as expected.

okay, looking better with the patch. before:

I'm seeing some jobs succeed and a lot fail with the smell of binary incompatibility. Initially, I was worried that I did this, but I don't think so:
AttributeError("'Wikiwho' object has no attribute 'spam_hashes_set'")

Okay, we're back up and running; expect things to be shaky as the box warms up and tries to handle requests. @Ragesoss can you check your dashboard?

The web proxies are connected to the new host, but I have all of the services down on it, so I find that not surprising. Let's recheck when we're back up.

chown finished on another pickle store; that's pickles-02 and pickles-03 done. We're just waiting on pickles now. (note that I kicked them off in parallel since it's 3 separate volumes)

Note that the django database moved and we validated (on accident) that pickles were resolving, but we're still waiting on chown to finish fixing up permission across 2/3 stores. Web proxies are moved over, services are configured, but we have them all shutdown pending the chown. I'll check in periodically and update here when things are back up and running.