In T266717#7171743, @Legoktm wrote:

I guess the concern is that starting the scripts right away just adds extra pressure and load while things are starting up right?

My guess is, your SSH key always had a passphrase associated, but on your old laptop, the Mac OS keychain was storing it for you (that's the "UseKeychain yes") in your .ssh/config. I don't know much of anything about AirDrop, but my hunch is that keychain entry didn't get copied over to your new machine, which is why you're being prompted for it now.

It may or may not be a regression -- we were getting a fair amount of this traffic during the incident, not just isolated requests. One hypothesis is that we can handle a single heavy history query from the occasional power user here and there, but even a smallish distributed flood of them still constitutes a DOS vector.

Two moving parts here in the exim config, comparing lists1001's config to other hosts where root mail does work.

Done -- just re-enabled puppet, so they'll get picked up over the next 30m.

In T276029#6869338, @Joe wrote:

@RLazarus in https://phabricator.wikimedia.org/T248093#6076630 you mentioned committing a script for automating cert renewal, and I see it indeed. Renewing the certs should amount to just running the script, correct?

See T266717 for some related discussion.

This will stop alerting when T262211 is resolved and relforge1003,1004 are in service. The alias is just a pointer to the elasticsearch::relforge role, so nothing further needs to be done once the role is applied.

Nice find! Thanks for tracking this down.

Yep!

No, I haven't had a chance to look at k8splay at all yet.

In T266717#6722705, @Joe wrote:

I think we have a better way to avoid this. Basically we want to stop running scripts once we get into the readonly phase. So we could modify the wrapper to start only if both the following conditions are met:

1. the {"name": "WMFMasterDatacenter", "scope": "common"} variable is equal to the local datacenter
2. the 'ReadOnly' object with scope equal to the current datacenter is set to false

So scripts will stop running once we enter the readonly phase, well before we actually switch datacenter.

Doh, thanks @AntiCompositeNumber. :)

So, since the root cause here is a known issue, I'm duping this ticket over to the SVG rendering issue, but thanks @RoySmith for filing -- the subtasks are still open and will hopefully yield distinct improvements here.

Ah, cheers @AntiCompositeNumber, I thought it sounded familiar. So it sounds like this will get better with T216815, which I think I've heard mutterings about doing Soon™.

Thanks to @CDanis for digging into this with me. There are a couple of different things going on.

In T270209#6702440, @RLazarus wrote:

only on the 2880px- URL -- the smaller ones work fine.

I can't repro the CORS issue exactly, but I am getting a 503 from Varnish for https://upload.wikimedia.org/wikipedia/commons/thumb/b/ba/Circuit_de_la_Sarthe_track_map.svg/2880px-Circuit_de_la_Sarthe_track_map.svg.png ... but only on the 2880px- URL -- the smaller ones work fine. I suspect that's why @Aklapper couldn't immediately repro: my browser didn't try to fetch the 2880px edition until I resized the window big enough.

Thanks @Varnent for offering to look at this, as our primary contact with VIP. It turns out two other VIP-hosted domains, techblog.wikimedia.org and wikimediaendowment.org, also don't set an HSTS header.

Emailed Comms about it, will route this appropriately when I hear back.

@MPhamWMF You're all set! After we discussed a bit more, consensus is that you don't need to sign L3. It may take up to 30 minutes for the change to roll out everywhere.

In T270438#6700941, @elukey wrote:

@RLazarus reporting a question from IRC: do we need to have L3 or something similar to be signed?

You're all set! Nikolay, you should have received an automated email with the admin password. Yekaterina won't have received that email automatically; please share the admin password with her privately, or coordinate between yourselves to reset it to something new.

Merged! It'll be rolled out everywhere in 30 minutes, feel free to reopen if you need anything else.

Great! Pardon a brief delay in getting you set up with the additional access @Ottomata mentioned -- you're the guinea pig for a new arrangement, so I'd like to get some feedback from European-time-zone colleagues on how to implement it, and we should be able to get it committed tomorrow.

Why, I'd swear that wasn't there yesterday...

Hi @MPhamWMF, welcome to the Foundation! Your wikitech username was indeed the right one (thanks!) and delightfully I used it to pull up your shell uid "mttp", just to complete the set. ;)

This is done:

Great! Feel free to reopen, or file a new ticket under Wikimedia-Mailing-lists, if you need anything else.

Perfect! That's done, then -- both list owners should have just received an automated email containing the new admin password. Please confirm you can use it to log in at https://lists.wikimedia.org/mailman/admin/wikimedia-ma, and then I'll close this ticket -- or, let me know if you need anything else. :)

Hi @Anass_Sedrati, happy to work on this. :)

Looks that way! I'll take care of this today, thanks for flagging.

... Or @Dzahn could just fix it in https://gerrit.wikimedia.org/r/649999. :D Thanks!

@thcipriani Based on the context in T250241 I'm guessing this already has your blessing (creating a releasers-mwcli group containing Jeena and Brennen), but do you mind approving here for the record? Then I'll get it rolling and merge https://gerrit.wikimedia.org/r/649958 first thing in the morning.

@thcipriani @aezell Thanks!

@toan You should be all set! The access group changes might take up to 30 min to roll out everywhere. Let me know if you have any trouble.

@KFrancis Thanks!

Already in the nda group:

@KFrancis Thank you!

Oh wow, I filed this and went to bed, love to wake up and see it fully handled. :) Thanks all!

In T259979#6692703, @RLazarus wrote:

Quick correction -- this is now live on all appservers, but the old URL is still cached by the traffic layer for 24 hours or so. We could purge the cache if we were in a great hurry, but all things being equal, safer to let it expire -- I'll check back tomorrow and make sure this updated as expected.

Quick correction -- this is now live on all appservers, but the old URL is still cached by the traffic layer for 24 hours or so. We could purge the cache if we were in a great hurry, but all things being equal, safer to let it expire -- I'll check back tomorrow and make sure this updated as expected.

Deployed and tested:

(P.S. Today is technology's department-wide Fun Day, so further progress from Tyler and me might not come until tomorrow; sorry for the inconvenience. Give a yell if this is urgent, and we can get it taken care of.)

@STran Thanks!

Happened to see this go by -- I've dropped a single comment on the review, requesting a test be updated to match, and then I'll merge ASAP. Apologies for letting it stall for so long!

NDA discussion is happening over at T269777, then this will be ready to go.

@toan In addition to the agreement you signed with WMDE, you'll need to sign another with WMF -- @KFrancis will get you set up. Once that's taken care of, I can go ahead on both this task and T269678.

I've taken over SRE clinic duty from @jbond for the week -- @KFrancis I've emailed you his address. Once that's all set, I can go ahead with granting access. Thanks!

@MattCleinman Thanks for the update! I've added you to the wmf group.

@Tchanders From the "WIP" in the title, I'm guessing this isn't ready for SRE to work on yet, so I'm assigning it back to you. Feel free to pass it to me when it's ready for action. If you have any questions or you'd like some help getting it filled out, just say the word!

Yep, the alert has cleared. Thanks!

Oh, that's a good idea! We'd set maintenance_host to the empty string before the switchover, so that no new jobs would start anywhere, then set it to the new hostname afterward.