Mon, Jun 21
Tue, Jun 8
My guess is, your SSH key always had a passphrase associated, but on your old laptop, the Mac OS keychain was storing it for you (that's the "UseKeychain yes") in your .ssh/config. I don't know much of anything about AirDrop, but my hunch is that keychain entry didn't get copied over to your new machine, which is why you're being prompted for it now.
Fri, Jun 4
It may or may not be a regression -- we were getting a fair amount of this traffic during the incident, not just isolated requests. One hypothesis is that we can handle a single heavy history query from the occasional power user here and there, but even a smallish distributed flood of them still constitutes a DOS vector.
Wed, May 26
May 20 2021
Apr 30 2021
Two moving parts here in the exim config, comparing lists1001's config to other hosts where root mail does work.
Apr 14 2021
Done -- just re-enabled puppet, so they'll get picked up over the next 30m.
Mar 1 2021
Feb 19 2021
See T266717 for some related discussion.
Feb 11 2021
Feb 5 2021
This will stop alerting when T262211 is resolved and relforge1003,1004 are in service. The alias is just a pointer to the elasticsearch::relforge role, so nothing further needs to be done once the role is applied.
Jan 21 2021
Nice find! Thanks for tracking this down.
Jan 11 2021
Jan 8 2021
No, I haven't had a chance to look at k8splay at all yet.
Jan 5 2021
Dec 18 2020
Doh, thanks @AntiCompositeNumber. :)
So, since the root cause here is a known issue, I'm duping this ticket over to the SVG rendering issue, but thanks @RoySmith for filing -- the subtasks are still open and will hopefully yield distinct improvements here.
Thanks to @CDanis for digging into this with me. There are a couple of different things going on.
I can't repro the CORS issue exactly, but I am getting a 503 from Varnish for https://upload.wikimedia.org/wikipedia/commons/thumb/b/ba/Circuit_de_la_Sarthe_track_map.svg/2880px-Circuit_de_la_Sarthe_track_map.svg.png ... but only on the 2880px- URL -- the smaller ones work fine. I suspect that's why @Aklapper couldn't immediately repro: my browser didn't try to fetch the 2880px edition until I resized the window big enough.
Thanks @Varnent for offering to look at this, as our primary contact with VIP. It turns out two other VIP-hosted domains, techblog.wikimedia.org and wikimediaendowment.org, also don't set an HSTS header.
Emailed Comms about it, will route this appropriately when I hear back.
@MPhamWMF You're all set! After we discussed a bit more, consensus is that you don't need to sign L3. It may take up to 30 minutes for the change to roll out everywhere.
You're all set! Nikolay, you should have received an automated email with the admin password. Yekaterina won't have received that email automatically; please share the admin password with her privately, or coordinate between yourselves to reset it to something new.
Merged! It'll be rolled out everywhere in 30 minutes, feel free to reopen if you need anything else.
Dec 17 2020
Great! Pardon a brief delay in getting you set up with the additional access @Ottomata mentioned -- you're the guinea pig for a new arrangement, so I'd like to get some feedback from European-time-zone colleagues on how to implement it, and we should be able to get it committed tomorrow.
Why, I'd swear that wasn't there yesterday...
Hi @MPhamWMF, welcome to the Foundation! Your wikitech username was indeed the right one (thanks!) and delightfully I used it to pull up your shell uid "mttp", just to complete the set. ;)
This is done:
Great! Feel free to reopen, or file a new ticket under Wikimedia-Mailing-lists, if you need anything else.
Perfect! That's done, then -- both list owners should have just received an automated email containing the new admin password. Please confirm you can use it to log in at https://lists.wikimedia.org/mailman/admin/wikimedia-ma, and then I'll close this ticket -- or, let me know if you need anything else. :)
Hi @Anass_Sedrati, happy to work on this. :)
Looks that way! I'll take care of this today, thanks for flagging.
Dec 16 2020
@thcipriani Based on the context in T250241 I'm guessing this already has your blessing (creating a releasers-mwcli group containing Jeena and Brennen), but do you mind approving here for the record? Then I'll get it rolling and merge https://gerrit.wikimedia.org/r/649958 first thing in the morning.
@toan You should be all set! The access group changes might take up to 30 min to roll out everywhere. Let me know if you have any trouble.
Already in the nda group:
@KFrancis Thank you!
Oh wow, I filed this and went to bed, love to wake up and see it fully handled. :) Thanks all!
Dec 15 2020
Quick correction -- this is now live on all appservers, but the old URL is still cached by the traffic layer for 24 hours or so. We could purge the cache if we were in a great hurry, but all things being equal, safer to let it expire -- I'll check back tomorrow and make sure this updated as expected.
Deployed and tested:
(P.S. Today is technology's department-wide Fun Day, so further progress from Tyler and me might not come until tomorrow; sorry for the inconvenience. Give a yell if this is urgent, and we can get it taken care of.)
Happened to see this go by -- I've dropped a single comment on the review, requesting a test be updated to match, and then I'll merge ASAP. Apologies for letting it stall for so long!
Dec 14 2020
NDA discussion is happening over at T269777, then this will be ready to go.
@toan In addition to the agreement you signed with WMDE, you'll need to sign another with WMF -- @KFrancis will get you set up. Once that's taken care of, I can go ahead on both this task and T269678.
@MattCleinman Thanks for the update! I've added you to the wmf group.
@Tchanders From the "WIP" in the title, I'm guessing this isn't ready for SRE to work on yet, so I'm assigning it back to you. Feel free to pass it to me when it's ready for action. If you have any questions or you'd like some help getting it filled out, just say the word!
Yep, the alert has cleared. Thanks!
Dec 8 2020
Dec 3 2020
Oh, that's a good idea! We'd set maintenance_host to the empty string before the switchover, so that no new jobs would start anywhere, then set it to the new hostname afterward.