So, the way this currently works is that mw-misc (home of noc.wikimedia.org) is updated during the testservers deployment stage, just like mw-debug. Meaning, hitting noc during that stage should observe the new changes immediately.

@Krinkle - Does that satisfy your use case, or are you proposing to have a debug equivalent that's updated during the testservers stage, while noc is updated with the production stage (similar to production wikis)? Thanks!

@Scott_French If I understand correctly, this means that:

We can test changes mid-deploy on noc by enabling WikimediaDebug. This correctly bypasses Varnish/ATS cache, but is be ignored by the ATS director, because noc.wikimedia.org has no mw-debug equivalent, and so routes to k8s-mw-misc as it normally would, which is fine because that servergroup receives Scap changes at the same time as mw-debug.

More often than not, when working on NOC, it's ad-hoc debugging or iterating on a change that isn't ready for deployment. Given low impact of errors there, we don't need a lot of deployment verification there once a patch has been code reviewed and merged. Deploying and fixing/reverting as-needed is easy enough.

The ad-hoc case has moved away from mw-debug to mw-experimental. I suppose the question then becomes whether mw-experimental (and perhaps mw-debug for consistency/simplicity) could accept the noc.wikimedia.org vhost (source).

It seems fragile/confusing to have it "work" kind of by accident where XWD:mw-debug applies to caching but is then actually ignored. It also means Logstash messages don't end up under the mwdebug dashboard, which is where one would normally be working when using WikimediaDebug or mw-experimental.

Thanks, @Krinkle - That helps clarify. For noc.wikimedia.org, presence of the x-wikimedia-debug header affects caching behavior and response header cleanup, but has no role in backend selection at ATS. Instead, requests always route to mw-misc, which also happens to be updated in the testservers stage, so the "feedback loop" for testing merged code is sort of equivalent.

In any case, I think it makes sense to reevaluate this, particularly because of that last part, which you also note - ad-hoc testing of WIP code.

It also means Logstash messages don't end up under the mwdebug dashboard, which is where one would normally be working when using WikimediaDebug or mw-experimental.

I am curious about this aspect. My vague recollection is that the code in the noc docroot doesn't initialize / emit "mediawiki-ish" (for lack of a better term) logging. Was that the case at some point in the past?