Perfect, thank you!

Thanks @APizzata-WMF! Sorry to take so long to reply; I'm about to be out of office for a couple of weeks, so leaving some thoughts here before I go, but I'll be happy to talk more when I get back.

Thanks @mpopov and @JVanderhoop-WMF! Appreciate the correction in particular, that sounds really promising.

Do you mind if we keep this open until the doc updates are done? Just so we don't lose track of 'em.

Thanks @gengh and sorry for the delayed response here. Some questions about the documentation, but not blockers on moving forward with the storage setup as we've built it out. Once we're agreed on these comments, we can consider this resolved from my POV -- no need to wait for another round of review unless you have questions about anything below. (I'm optimistically checking our "reviewed and agreed" box on that basis.)

And cc @KOfori for awareness -- I see @Eevans is OOO until the 28th, this will probably need attention before he comes back.

Good enough for me!

In T428191#12332563, @MLechvien-WMF wrote:

@RLazarus do you think you can take care of mc-wf[2001-2002].codfw.wmnet ?

Also filed T438270 for the scap side.

@MoritzMuehlenhoff grabbed this overnight, so the build is fixed (thank you!). Dropping priority and leaving open for the monitoring/alerting question.

(Reporting from a serviceops team meeting.)

Thanks, the imageGCHighThresholdPercent kicking in at the same time as imagefs.available evictionHard is the piece I was missing. Unused images would explain this more neatly than my "Did the MW pull use a lot more than the final 2.4 GiB of imagefs, but only transiently?" theory, with the MW pull still being an obvious trigger to push it over the line.

I opened T437539 for the part I think we might look into in ServiceOps. The only thing left to do here, if you like, is to discuss increasing the resource quota for wikifunctions. My instinct is not to bother until the next time you scale up, unless you're running into trouble in situations other than this one. If you're happy with that, we can just resolve this.

You should be able to fork the repo, push the changes to your fork, and open a merge request on the original repo. Instructions here: https://www.mediawiki.org/wiki/GitLab/Workflows/Making_a_merge_request

It was sluggish for me too, I just wasn't sure exactly what's normal for the service.

Resolving thing 3:

Couple of things.

Okay sounds good! For next steps, two things:

From the above, I've spun off the TODOs for ServiceOps into T436204 and T436205. Unless I hear otherwise, I'll resolve this task as having answered the question, but let me know if we can add any more information here.

(Serviceops triage here -- rescuing this from the abyss where it was drifting without our tag.)

Never mind -- thanks @Scott_French for this logstash pointer and subsequent discussion: https://logstash.wikimedia.org/app/discover#/doc/0fade920-6712-11eb-8327-370b46f9e7a5/ecs-default-1-1.11.0-8-2026.35?id=88jYMqABfN-RlYin0FQh

(ServiceOps triage here.) Neat, I don't think we've seen this before! Thanks for the report.

Sorry for the slow response here. Structurally, looks good to me! Some questions:

The script didn't crash, and rerunning it was correct.

(Drive-by comment here, happened to see @Asukite's message in passing.)

Also, I seemingly don’t have enough permissions to run the “learn more about the release” commands:

As discussed offline, over to you!

Thanks both!

We identified and worked around the underlying problem that caused us to roll back earlier -- changeprop-jobqueue in codfw is now running only on machines in rows A through D, whose network connections aren't subject to whatever is causing T433881. It got a little late in the afternoon to keep going, but we can try rolling forward again tomorrow (thanks James for posting it).

Deployed https://gerrit.wikimedia.org/r/1309293 today, which closes this up.

Wikifunctions is now using kube-dns-internalonly.

Service port removed at https://wikitech.wikimedia.org/w/index.php?title=Kubernetes/Service_ports&diff=prev&oldid=2434700.

All the Kubernetes work here is done, and I went ahead and removed us from the deployment calendar tomorrow. Let me know if you need anything else from Service Ops here.

Status from this morning: done through step 7 in T416623#12075480, i.e. LVS work is complete. admin_ng is dirty from unrelated work so I'm optimistically waiting to hear back before proceeding.

admin_ng has some unapplied diffs in wikikube and aux clusters that look related to this, e.g.:

(ServiceOps triage here.) We did something similar in T422804 to provide access to recommendation-api-ng and T427902 for liftwing-api-server. We just need a little more information about what service you want to reach.

The usual:

Status: This is done up through backing the LVS setup out to lvs_setup (i.e. through https://wikitech.wikimedia.org/wiki/LVS#Remove_network_probes_/_monitoring, and step 4½ in T416623#12075480.)

Yeah, officially, this isn't really Puppet window material. It has way too many moving parts -- the Puppet window is really just for quick one-off patches, and in general we shouldn't expect Puppet window deployers to handle changes like this.