Great!

My fault, sorry about that!

This is allowed, but you need to use deploy (read-write) credentials. Try it after

Done! Please wait up to 30 minutes for that to propagate to all servers, then you should be all set. I'll resolve this but feel free to reopen it if you have any trouble using your new access.

Chatted with @Scott_French about this today. The cause is Sophroid's http probe in service.yaml, routed to a port that doesn't handle HTTP, only GRPC.

This is done! Wait up to 30 minutes for it to propagate to all hosts, and you'll be all set. Resolving the task, feel free to reopen it if you have any trouble.

Hi @APDube-WMF! I see you provided an SSH key on the task, but if Superset access is all you need, we won't actually need it. I'll set you up without SSH access for now, but if you also need access to anything listed at level 2 or higher, like the stat servers, let me know, and we can always add your SSH key later.

Never mind! Imagine my surprise to find that key already there. :) This was done in https://gerrit.wikimedia.org/r/1298282, thanks @ssingh!

Verified out of band, updating.

Hi @mahmoud.abdelsattar.wmde! I see you already have restricted access (using the SSH key you included), so we should be able to additionally grant you analytics-privatedata-users. (For anyone following along, that effectively means level 2 access, even though only level 1 is strictly required for Superset.)

All done! This will take up to 30 minutes to roll out everywhere, then make sure to follow the instructions in T428262#11990183.

SSH public key (must be a separate key from Wikimedia cloud SSH access): N/A (already in modules/admin/data/data.yaml)

I'm always happy to get involved if you need me, but my colleague @CDanis from the SLO working group is your best contact for the followup here. :)

Moving to Pending until the app-side work is done on the orchestrator and evaluator, then we can wrap this up. In order:

Word from the Abstract Wikipedia folks is that you should go ahead and reimage the mc-wf hosts without any prework -- just, one at a time please.

I've reserved port 4974 for this, the next available service port after 4970 (evaluator) and 4971 (orchestrator main port).

From the logs:

Let's leave it open until we move it from "draft" to "approved." :)

Done.

Built and published:

I chatted with @MoritzMuehlenhoff about this today (thanks Moritz).

Relevant log lines from this one:

We actually added a test for that case already (in T387549, for this incident):

I think listing it in that incident was a mistake, actually -- there weren't any releases in state failed in that event, so this feature wouldn't have affected things at all. (I think the incident author wanted a one-line "roll back mediawiki without having to touch the charts repo" command, and thought from the task title that's what this task is. I'm not sure if I agree that feature would be a good idea, but it's not the same thing being discussed here.)

Transient failure, followed by a successful run, resolving.

Transient failure, followed by a successful run, resolving.

Posting as serviceops triage: Adding Releng for awareness (who don't own the schedule-deployment tool but do manage other deployment calendar automation).

Thanks for digging into it!

Suppose version A is running, and we're deploying version B.

I agree that we should take the explode call out of the hot path, but reworking the existing env variables is probably more than we want to tackle -- we could orchestrate the config change for ourselves, but it'd also be a visible change for non-WMF users of MediaWiki with memcache. Not impossible but a lot of work, especially if we don't end up keeping MemcachedWrapper in the long run.

On the prerequisites:

• Double-checked, and mw-mcrouter has all the routes except /local/wf. That's fine, because...
• Per @Jdforrester-WMF, we don't need to keep the /local/wf default. Nothing in mw-* namespaces, including mw-wikifunctions, uses it. (The orchestrator, running in the wikifunctions namespace does, but that's out of scope here.)

So, the story is that

• https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1269038 configures the shared mcrouter daemonset (mw-mcrouter), which is accessible at $_SERVER['MCROUTER_SERVER'] (i.e. 10.64.72.12:4442 in eqiad, 10.192.72.12:4442 in codfw), but
• it isn't having the desired effect, because https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1266290 sets $eqiadDCWFMC and $codfwDCWFMC to talk to 127.0.0.1:11213 (in-pod mcrouter) rather than using the shared mcrouter, and
• that won't work because nothing is listening on that port: outside of mw-wikifunctions, most mediawiki pods don't have an in-pod mcrouter. So when you try to connect to 127.0.0.1:11213, nobody is home.

Weirdly, Envoy failed to start after the change, with this in the logs:

We were actually just talking about this in the SLOs group last week (adding @Vgutierrez and @CDanis).

In the serviceops meeting today we decided to go ahead with this.

(Service Ops triage here: moving this to Needs Info for discussion at our team meeting.)

One more axis to consider: Best-practices-wise, for alerting on Kubernetes platforms, there's a distinction between control plane and data plane.

Resolving; the remaining hosts will go straight to 1.35.9 in T419637 instead.

Unsurprisingly /var/log/kern.log on kubestage1006 (hosting the above example pod) is full of lines like:

The actual profiles are at modules/profile/files/kubernetes/node/wikifunctions-evaluator and .../wikifunctions-orchestrator (which is the same except s/evaluator/orchestrator/g). The staging and prod containers have the same apparmor annotation[1] so should have the same signals policy, and anyway it looks correct at a glance.

Service Ops triage here: Agreed there's nothing for us to do, thanks @ayounsi - untagging us.

This makes tracking response difficult, and in some situations recently we have had to move to #wikimedia-sre in order to communicate properly. If we wish to pursue this as an official pattern, it needs to be documented and recorded.

Here you go: https://gitlab.wikimedia.org/repos/sre/serviceops-kitchensink/-/merge_requests/28