Note October 27 is also scheduled for the MediaWiki datacenter switchback -- please let's not have both events going at the same time. :) The switchback is scheduled to wrap up by 15:00 UTC, but it's possible we'll still be doing cleanup work for a while after, depending on how things go.

In T257118#6523783, @Krinkle wrote:

cache-text, cache-upload, etcd, ircd, and memc are most certainly also still in use and we need at least one of each.

For memc we can most likely size down, I don't think we added this many intentionally but rather added new ones during rebuilds/upgrades and maybe left old ones?

cc @RLazarus Do you know if any of these were used for tests, and whether we could e.g. get a way with just one or two of the deployment-memc* instances? If so, where do we need to check for configs and update things etc. is the latest one the "right" one to keep?

This should be done! Thanks one last time for your patience.

October 27 confirmed, and I just filed T264364. Meet you over there. :)

Apologies, I've had a hectic few days. :) It turns out the engprod offsite is the week of Oct 26, so there already won't be a MW release train that week -- releng is understandably reluctant to skip the release two weeks in a row, which is a good argument against Oct 21. We don't need anyone from engprod online for the switchover itself, though, so we could run it during their offsite.

Thanks @KFrancis! Passing this along to @ArielGlenn as the current SRE Clinic Duty person.

October 21 looks good, tentatively. Let me confirm with folks.

I see you already put this on the agenda for the next SRE meeting on Monday, thanks. :) I expect it'll be uncontroversial, but if you don't need it urgently, we'll bring it up at the meeting and get it merged that afternoon.

Thank you @Trizek-WMF! Really appreciate all your work on this, and your team's.

Yeah, this does look like it was related to the eventgate-main deployments at 13:20 and 13:33 -- traffic may not have shifted gracefully from the old instances to the new ones, resulting in a transient spike of errors.

Any list administrator can disable archiving new messages, at https://lists.wikimedia.org/mailman/admin/wikidata-bugs/archive -- just change "Archive messages?" to "no" and submit.

@conny-kawohl_WMDE Thanks!

And @bete one request (at the same time, to speed things along): Could you please ask your manager to comment on this Phab ticket, giving approval?

Hello Bereket! I can handle the LDAP changes for you.

Added to the agenda for the next SRE meeting, 2020-09-21.

I purged /api/rest_v1/page/mobile-html/ and /api/rest_v1/page/mobile-html-offline-resources/ in ATS, then re-ran Joe's command from T262437#6448216 for both. This should now be fully expunged from cache, although it may persist in your browser cache for up to a day or until you refresh.

@Trizek-WMF Question from @debt earlier, will you be posting to wikitech-l also, or only wikitech-ambassadors? No wrong answer AFAIK, just confirming the plan.

Done: https://wikitech.wikimedia.org/wiki/Switch_Datacenter#Schedule_for_2020_switch

Yeah, sorry that's later than I expected -- we're meeting today to confirm the timing details and I'll post the update immediately after, so a little over two hours from now.

@Papaul Over to you, thanks!

@Papaul Sorry for the delay, we only just got mc2037 up and serving. Opened T261168 for the decom, starting the service owner steps now.

I know it's 2020, but Jessie for uniformity please. Upgrading those hosts is a work in progress. If we weren't about to do the switchover, I'd say install something newer and see what happens, but let's do one thing at a time.

Yes please -- let's use the spare machine, add the memory from mc2028, and name the new host mc2037 (that is, let's use @elukey's option #1, but not reuse the name "mc2028", at @Volans's request).

I bet we can do 14:00 UTC. I'm finishing up the timeline with my SRE colleagues this week, I'll confirm and get back to you. After that we'll post on Wikitech, it'll be on the same page.

No, it'll be roughly a month -- there's a variety of maintenance we'd like to do in eqiad while we're serving from codfw.

The simple version of this is done. We might eventually want to do something more elaborate -- the advantage would be that httpbb could be run without explicitly passing a test file, and by default it could deduce (via Puppetry) the appropriate set of tests to run. But this is good enough for now.

The only user-impacting section of the process will be a read-only period for all wikis while we move MediaWiki itself -- that should last about 3-5 minutes, somewhere between 13:30 and 15:30 UTC on 2020-09-01.

Yep, we're all set. Please revert the RAM quota at your convenience, and if there are any Ceph resources to reclaim, feel free to do that too -- we won't be needing it for the foreseeable. Thanks!

Thanks for the update; I was going to check back in here soon. We've actually rearranged this build a bit, and in the end we won't need the expanded disk after all, so we'll be settling back into a regular m1.xlarge instance.

Thanks @Trizek-WMF! It took a moment to get everything else lined up, but we're moving forward with September 1.

Sorry for the delay, but this is still in progress -- I've checked in with Legal and they're still working on it. Thanks for your patience, still.

It looks like we'll try to do this: ideally we'll aim to do the switchover from eqiad to codfw in either mid-to-late August or early September, and the switch back to eqiad about a month later. (I can promise we won't do anything in July; we wouldn't ask you to work on that kind of short notice, and we won't have our act together yet anyway.)

Side note: This question is also interesting from a DC switchover perspective (T243316) since that will also effectively be a Redis flush. In previous switchovers we only explicitly handled replication for sessions data, and now that's out of Redis. If there's anything else in there that we can't afford to drop and recreate, now would be a great time to know that.

Summarizing here a conversation @elukey and I had in #wikimedia-serviceops:

Thanks for checking -- not sure yet, but as we're planning out Q1 on our side too, I'm starting to take everyone's temperature about it. I'll let you know as soon as I have some idea whether it will happen, and I'll make sure to clear any potential dates with you.

In T251349#6166850, @MBinder_WMF wrote:

Should I instead point to the private key I generated (per https://wikitech.wikimedia.org/wiki/Production_access#Generating_your_SSH_key)?

@soworu Ping on my question above. :)

Hi CPT, adding you back -- is it possible this is due to this section of security_response_header_filter.js in Restbase?

All set!

Trying to route this -- @ssingh, should this be assigned to you?

@MBinder_WMF On the offchance you'd already tried logging into phab1001, and it didn't work, but you hadn't gotten around to saying anything yet -- that was because of my mistake, just fixed it, so try again. :)

Thanks! I've removed mnoor but not mshaver from the wmf LDAP group, let me know if anything suddenly stops working.

Hi @MNoorWMF, just checking on this -- can you please confirm that the mshaver account is working for you? If it is, I'll remove access from mnoor so that we don't leave both lying around.

Hi @MBinder_WMF! I've created your shell account and added it to the group that @Dzahn created for you. I just ran puppet on bast4002 (your nearest bastion) and both Phab hosts, so you should be able to test your access now by sshing to phab1001.eqiad.wmnet, after following the Setting_up_your_SSH_config steps in the page that Daniel linked.

After chatting with @colewhite (thanks!) I've moved Rodolfo from the wmf group over to the nda group, which should still provide all the access necessary here. Let me know if you have any trouble.

This uses your LDAP password -- the same one you use to log into Wikitech. The username should be either your Wikitech username (Rodolfo Valentim) or your uid/"shell username" (rodolfovalentim) -- the login prompt should tell you which.

@dcipoletti Thanks, looks good! All I need you to do now is:

• sign L3 (there should be an option to sign it right on that page) and
• let me know that you've read https://wikitech.wikimedia.org/wiki/Analytics/Data_access#User_responsibilities (by just saying so here).

Thanks Katie!

Looks like this comes from docker_pkg/cli.py line 76, which is the last line here:

I see @Nuria's authorization on the parent task, so going ahead with this.

@dcipoletti Can you please ask your manager to comment on this task, approving your access?

Agree we ought to do this, and I think it's something Envoy can do nearly out of the box. If there's an element Envoy doesn't support, I agree that it's the over-limit FIFO queue. (Out of all the reverse proxies we could use, I'm focusing on Envoy because we're actively deploying it in other parts of the infrastructure, and if it's feasible I think we'd like to use it here too for consistency.)

Hi Daniel, welcome to the Foundation! I've added you to the wmf LDAP group, feel free to reopen if you need anything else.

Hi Carly, I've added you to the wmf LDAP group -- feel free to reopen if you need anything else.

Thanks @bd808! It doesn't sound like there's presently anything for the SRE clinic-duty person to do here, so I'm optimistically removing SRE-Access-Requests, but feel free to add it back if you need anything from me (or subsequent clinicians).

Hi Segun, thanks for the clear and complete request! I've granted you "restricted user" access for almost all those domains, which I think is the right level of access for your needs. If you need "full user," let me know and I'll upgrade you (or file another task, if you find out later).