Page MenuHomePhabricator

RLazarus (Reuven Lazarus)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Saturday

  • Clear sailing ahead.

User Details

User Since
Oct 15 2019, 4:02 PM (88 w, 2 d)
Availability
Available
IRC Nick
rlazarus
LDAP User
RLazarus
MediaWiki User
Unknown

Recent Activity

Yesterday

RLazarus added a comment to T266717: Disable maintenance scripts via conftool.

I guess the concern is that starting the scripts right away just adds extra pressure and load while things are starting up right?

Wed, Jun 23, 5:49 PM · Patch-For-Review, Datacenter-Switchover, conftool, serviceops, SRE

Mon, Jun 21

RLazarus reassigned T266717: Disable maintenance scripts via conftool from RLazarus to Legoktm.
Mon, Jun 21, 7:00 PM · Patch-For-Review, Datacenter-Switchover, conftool, serviceops, SRE

Tue, Jun 8

RLazarus added a comment to T284588: Need to ssh with my new laptop.

My guess is, your SSH key always had a passphrase associated, but on your old laptop, the Mac OS keychain was storing it for you (that's the "UseKeychain yes") in your .ssh/config. I don't know much of anything about AirDrop, but my hunch is that keychain entry didn't get copied over to your new machine, which is why you're being prompted for it now.

Tue, Jun 8, 6:19 PM · SRE-Access-Requests, SRE

Fri, Jun 4

RLazarus added a comment to T284274: action=history allows for limits as high as 5000, which is probably too high.

It may or may not be a regression -- we were getting a fair amount of this traffic during the incident, not just isolated requests. One hypothesis is that we can handle a single heavy history query from the occasional power user here and there, but even a smallish distributed flood of them still constitutes a DOS vector.

Fri, Jun 4, 6:09 PM · Platform Team Workboards (Clinic Duty Team), Vuln-DoS, MediaWiki-Page-history, Security, Security-Team

Wed, May 26

RLazarus created T283762: Icinga alerts mention the wrong data center.
Wed, May 26, 8:49 PM · SRE

May 20 2021

RLazarus created T283267: Update [[wt:Template:Irc]] to link to Libera instead of Freenode.
May 20 2021, 5:40 PM · User-bd808, wikimedia-irc-libera

Apr 30 2021

RLazarus added a comment to T280744: Mail to root@lists1001.wikimedia.org from noreply@lists1001.wikimedia.org doesn't work.

Two moving parts here in the exim config, comparing lists1001's config to other hosts where root mail does work.

Apr 30 2021, 10:30 PM · Mail, SRE

Apr 14 2021

Dzahn awarded T276029: Renew certs for mcrouter on all mw appservers a Grey Medal token.
Apr 14 2021, 6:33 PM · SRE, serviceops
RLazarus closed T276029: Renew certs for mcrouter on all mw appservers as Resolved.

Done -- just re-enabled puppet, so they'll get picked up over the next 30m.

Apr 14 2021, 2:13 PM · SRE, serviceops
RLazarus claimed T276029: Renew certs for mcrouter on all mw appservers.
Apr 14 2021, 1:48 PM · SRE, serviceops

Mar 1 2021

RLazarus added a comment to T276029: Renew certs for mcrouter on all mw appservers.

@RLazarus in https://phabricator.wikimedia.org/T248093#6076630 you mentioned committing a script for automating cert renewal, and I see it indeed. Renewing the certs should amount to just running the script, correct?

Mar 1 2021, 5:56 PM · SRE, serviceops

Feb 19 2021

RLazarus added a comment to T267607: upgrade mwmaint servers to buster .

See T266717 for some related discussion.

Feb 19 2021, 3:22 PM · SRE, serviceops

Feb 11 2021

RLazarus renamed T274472: Investigate and repool db1134 from db1134 placeholder task to Investigate and repool db1134.
Feb 11 2021, 3:21 AM · Patch-For-Review, ops-eqiad, DBA, SRE

Feb 5 2021

RLazarus assigned T273353: cumin alias relforge matched 0 hosts to RKemper.

This will stop alerting when T262211 is resolved and relforge1003,1004 are in service. The alias is just a pointer to the elasticsearch::relforge role, so nothing further needs to be done once the role is applied.

Feb 5 2021, 4:40 PM · SRE

Jan 21 2021

RLazarus added a comment to T270517: Investigate opcache hit rate on Buster appserver.

Nice find! Thanks for tracking this down.

Jan 21 2021, 3:22 PM · observability, Icinga, SRE, serviceops

Jan 11 2021

RLazarus closed T243318: FY2020-2021 Q1 codfw -> eqiad switchback, a subtask of T243314: FY2020-2021 Q1 DC switchover and switchback, as Resolved.
Jan 11 2021, 3:55 PM · Patch-For-Review, Goal, SRE
RLazarus closed T243318: FY2020-2021 Q1 codfw -> eqiad switchback as Resolved.
Jan 11 2021, 3:55 PM · SRE
RLazarus closed T243316: FY2020-2021 Q1 eqiad -> codfw switchover, a subtask of T243314: FY2020-2021 Q1 DC switchover and switchback, as Resolved.
Jan 11 2021, 3:55 PM · Patch-For-Review, Goal, SRE
RLazarus closed T243316: FY2020-2021 Q1 eqiad -> codfw switchover as Resolved.
Jan 11 2021, 3:55 PM · SRE
RLazarus closed T243314: FY2020-2021 Q1 DC switchover and switchback as Resolved.

Yep!

Jan 11 2021, 3:54 PM · Patch-For-Review, Goal, SRE

Jan 8 2021

RLazarus added a comment to T271554: k8splay project has broken puppet because of incorrect FQDNs.

No, I haven't had a chance to look at k8splay at all yet.

Jan 8 2021, 7:13 PM · Cloud-VPS

Jan 5 2021

RLazarus added a comment to T266717: Disable maintenance scripts via conftool.

I think we have a better way to avoid this. Basically we want to stop running scripts once we get into the readonly phase. So we could modify the wrapper to start only if both the following conditions are met:

  1. the {"name": "WMFMasterDatacenter", "scope": "common"} variable is equal to the local datacenter
  2. the 'ReadOnly' object with scope equal to the current datacenter is set to false

So scripts will stop running once we enter the readonly phase, well before we actually switch datacenter.

Jan 5 2021, 5:38 PM · Patch-For-Review, Datacenter-Switchover, conftool, serviceops, SRE

Dec 18 2020

RLazarus added a comment to T270209: Image fails to load with CORS violation.

Doh, thanks @AntiCompositeNumber. :)

Dec 18 2020, 11:35 PM · Thumbor, serviceops, SRE
RLazarus merged T200866: rsvg-convert times out while generating large thumbnails with heavy use of Gaussian blur into T270209: Image fails to load with CORS violation.
Dec 18 2020, 9:46 PM · Thumbor, serviceops, SRE
RLazarus merged task T200866: rsvg-convert times out while generating large thumbnails with heavy use of Gaussian blur into T270209: Image fails to load with CORS violation.
Dec 18 2020, 9:45 PM · Upstream, Wikimedia-SVG-rendering, Thumbor
RLazarus added a comment to T270209: Image fails to load with CORS violation.

So, since the root cause here is a known issue, I'm duping this ticket over to the SVG rendering issue, but thanks @RoySmith for filing -- the subtasks are still open and will hopefully yield distinct improvements here.

Dec 18 2020, 9:45 PM · Thumbor, serviceops, SRE
RLazarus added a comment to T270209: Image fails to load with CORS violation.

Ah, cheers @AntiCompositeNumber, I thought it sounded familiar. So it sounds like this will get better with T216815, which I think I've heard mutterings about doing Soon™.

Dec 18 2020, 9:34 PM · Thumbor, serviceops, SRE
RLazarus added a subtask for T270209: Image fails to load with CORS violation: T270528: Fall back to smaller image sizes?.
Dec 18 2020, 9:29 PM · Thumbor, serviceops, SRE
RLazarus added a parent task for T270528: Fall back to smaller image sizes?: T270209: Image fails to load with CORS violation.
Dec 18 2020, 9:29 PM · MediaViewer
RLazarus created T270528: Fall back to smaller image sizes?.
Dec 18 2020, 9:29 PM · MediaViewer
RLazarus added a subtask for T270209: Image fails to load with CORS violation: T270526: Set CORS headers on error pages?.
Dec 18 2020, 9:13 PM · Thumbor, serviceops, SRE
RLazarus added a parent task for T270526: Set CORS headers on error pages?: T270209: Image fails to load with CORS violation.
Dec 18 2020, 9:13 PM · SRE, Traffic
RLazarus triaged T270526: Set CORS headers on error pages? as Medium priority.
Dec 18 2020, 9:13 PM · SRE, Traffic
RLazarus edited projects for T270209: Image fails to load with CORS violation, added: serviceops, Thumbor; removed Traffic.

Thanks to @CDanis for digging into this with me. There are a couple of different things going on.

Dec 18 2020, 8:59 PM · Thumbor, serviceops, SRE
RLazarus added a comment to T270209: Image fails to load with CORS violation.

only on the 2880px- URL -- the smaller ones work fine.

Dec 18 2020, 8:08 PM · Thumbor, serviceops, SRE
RLazarus added a project to T270209: Image fails to load with CORS violation: Traffic.

I can't repro the CORS issue exactly, but I am getting a 503 from Varnish for https://upload.wikimedia.org/wikipedia/commons/thumb/b/ba/Circuit_de_la_Sarthe_track_map.svg/2880px-Circuit_de_la_Sarthe_track_map.svg.png ... but only on the 2880px- URL -- the smaller ones work fine. I suspect that's why @Aklapper couldn't immediately repro: my browser didn't try to fetch the 2880px edition until I resized the window big enough.

Dec 18 2020, 8:07 PM · Thumbor, serviceops, SRE
RLazarus removed a project from T260083: Reshard commonswiki_file elasticsearch index: Patch-For-Review.
Dec 18 2020, 6:42 PM · SRE, Discovery-Search (Current work)
RLazarus added a subtask for T245757: Upgrade MediaWiki clusters to Debian Buster (debian 10): T270517: Investigate opcache hit rate on Buster appserver.
Dec 18 2020, 6:38 PM · Release-Engineering-Team (Radar), SRE, serviceops
RLazarus added a parent task for T270517: Investigate opcache hit rate on Buster appserver: T245757: Upgrade MediaWiki clusters to Debian Buster (debian 10).
Dec 18 2020, 6:38 PM · observability, Icinga, SRE, serviceops
RLazarus triaged T270517: Investigate opcache hit rate on Buster appserver as Medium priority.
Dec 18 2020, 6:37 PM · observability, Icinga, SRE, serviceops
RLazarus added a member for SRE: RLazarus.
Dec 18 2020, 5:17 PM
RLazarus renamed T270034: Send HSTS header on all Wordpress VIP-hosted domains from Send HSTS header on diff.wikimedia.org to Send HSTS header on all VIP-hosted domains.
Dec 18 2020, 5:11 PM · Technical blog, SRE, Traffic, HTTPS, Diff-blog
RLazarus reassigned T270034: Send HSTS header on all Wordpress VIP-hosted domains from RLazarus to Varnent.

Thanks @Varnent for offering to look at this, as our primary contact with VIP. It turns out two other VIP-hosted domains, techblog.wikimedia.org and wikimediaendowment.org, also don't set an HSTS header.

Dec 18 2020, 5:11 PM · Technical blog, SRE, Traffic, HTTPS, Diff-blog
RLazarus claimed T270034: Send HSTS header on all Wordpress VIP-hosted domains.

Emailed Comms about it, will route this appropriately when I hear back.

Dec 18 2020, 4:53 PM · Technical blog, SRE, Traffic, HTTPS, Diff-blog
RLazarus closed T270438: LDAP access to the wmf group for Mike Pham as Resolved.

@MPhamWMF You're all set! After we discussed a bit more, consensus is that you don't need to sign L3. It may take up to 30 minutes for the change to roll out everywhere.

Dec 18 2020, 3:21 PM · LDAP-Access-Requests, SRE
RLazarus added a comment to T270438: LDAP access to the wmf group for Mike Pham.

@RLazarus reporting a question from IRC: do we need to have L3 or something similar to be signed?

Dec 18 2020, 2:44 PM · LDAP-Access-Requests, SRE
RLazarus closed T270392: Create a mailing list for North-West Russia Wiki-Historians UG as Resolved.

You're all set! Nikolay, you should have received an automated email with the admin password. Yekaterina won't have received that email automatically; please share the admin password with her privately, or coordinate between yourselves to reset it to something new.

Dec 18 2020, 12:26 AM · SRE, Wikimedia-Mailing-lists
RLazarus claimed T270392: Create a mailing list for North-West Russia Wiki-Historians UG.
Dec 18 2020, 12:10 AM · SRE, Wikimedia-Mailing-lists
RLazarus added a parent task for T270350: Requesting access to releases1002/2002 for jhuneidi, brennen: T250241: Set up distribution of MediaWiki-Docker CLI.
Dec 18 2020, 12:04 AM · Patch-For-Review, SRE-Access-Requests, SRE
RLazarus added a subtask for T250241: Set up distribution of MediaWiki-Docker CLI: T270350: Requesting access to releases1002/2002 for jhuneidi, brennen.
Dec 18 2020, 12:04 AM · Patch-For-Review, mwcli, Release-Engineering-Team (Doing), User-brennen, MediaWiki-Docker
RLazarus updated the task description for T270350: Requesting access to releases1002/2002 for jhuneidi, brennen.
Dec 18 2020, 12:04 AM · Patch-For-Review, SRE-Access-Requests, SRE
RLazarus closed T270350: Requesting access to releases1002/2002 for jhuneidi, brennen as Resolved.

Merged! It'll be rolled out everywhere in 30 minutes, feel free to reopen if you need anything else.

Dec 18 2020, 12:03 AM · Patch-For-Review, SRE-Access-Requests, SRE
RLazarus created P13599 (An Untitled Masterwork).
Dec 18 2020, 12:01 AM

Dec 17 2020

RLazarus added a comment to T270438: LDAP access to the wmf group for Mike Pham.

Great! Pardon a brief delay in getting you set up with the additional access @Ottomata mentioned -- you're the guinea pig for a new arrangement, so I'd like to get some feedback from European-time-zone colleagues on how to implement it, and we should be able to get it committed tomorrow.

Dec 17 2020, 11:36 PM · LDAP-Access-Requests, SRE
RLazarus reopened T270438: LDAP access to the wmf group for Mike Pham as "Open".

Why, I'd swear that wasn't there yesterday...

Dec 17 2020, 10:54 PM · LDAP-Access-Requests, SRE
RLazarus closed T270438: LDAP access to the wmf group for Mike Pham as Resolved.

Hi @MPhamWMF, welcome to the Foundation! Your wikitech username was indeed the right one (thanks!) and delightfully I used it to pull up your shell uid "mttp", just to complete the set. ;)

Dec 17 2020, 10:35 PM · LDAP-Access-Requests, SRE
RLazarus updated the task description for T264590: Add lilients_WMDE to the ldap/wmde and ldap/nda group.
Dec 17 2020, 10:29 PM · LDAP-Access-Requests, SRE
RLazarus closed T264590: Add lilients_WMDE to the ldap/wmde and ldap/nda group as Resolved.

This is done:

Dec 17 2020, 10:29 PM · LDAP-Access-Requests, SRE
RLazarus created P13598 (An Untitled Masterwork).
Dec 17 2020, 10:12 PM
RLazarus claimed T270438: LDAP access to the wmf group for Mike Pham.
Dec 17 2020, 10:06 PM · LDAP-Access-Requests, SRE
RLazarus closed T270434: Update admin password for Wikimedia-MA mailing list as Resolved.

Great! Feel free to reopen, or file a new ticket under Wikimedia-Mailing-lists, if you need anything else.

Dec 17 2020, 9:56 PM · SRE-Access-Requests, Wikimedia-Mailing-lists, SRE
RLazarus renamed T270434: Update admin password for Wikimedia-MA mailing list from New mailing list requested for Wikimedia MA User Group to Update admin password for Wikimedia-MA mailing list.
Dec 17 2020, 9:50 PM · SRE-Access-Requests, Wikimedia-Mailing-lists, SRE
RLazarus added a comment to T270434: Update admin password for Wikimedia-MA mailing list.

Perfect! That's done, then -- both list owners should have just received an automated email containing the new admin password. Please confirm you can use it to log in at https://lists.wikimedia.org/mailman/admin/wikimedia-ma, and then I'll close this ticket -- or, let me know if you need anything else. :)

Dec 17 2020, 9:49 PM · SRE-Access-Requests, Wikimedia-Mailing-lists, SRE
RLazarus added a comment to T270434: Update admin password for Wikimedia-MA mailing list.

Hi @Anass_Sedrati, happy to work on this. :)

Dec 17 2020, 9:15 PM · SRE-Access-Requests, Wikimedia-Mailing-lists, SRE
RLazarus claimed T264590: Add lilients_WMDE to the ldap/wmde and ldap/nda group.

Looks that way! I'll take care of this today, thanks for flagging.

Dec 17 2020, 4:02 PM · LDAP-Access-Requests, SRE
RLazarus reassigned T270354: Add notes URL for Puppet failure alerts from RLazarus to Dzahn.

... Or @Dzahn could just fix it in https://gerrit.wikimedia.org/r/649999. :D Thanks!

Dec 17 2020, 3:45 PM · Puppet, SRE
RLazarus created T270354: Add notes URL for Puppet failure alerts.
Dec 17 2020, 12:20 AM · Puppet, SRE

Dec 16 2020

RLazarus triaged T270350: Requesting access to releases1002/2002 for jhuneidi, brennen as Medium priority.
Dec 16 2020, 11:59 PM · Patch-For-Review, SRE-Access-Requests, SRE
RLazarus added a comment to T270350: Requesting access to releases1002/2002 for jhuneidi, brennen.

@thcipriani Based on the context in T250241 I'm guessing this already has your blessing (creating a releasers-mwcli group containing Jeena and Brennen), but do you mind approving here for the record? Then I'll get it rolling and merge https://gerrit.wikimedia.org/r/649958 first thing in the morning.

Dec 16 2020, 11:57 PM · Patch-For-Review, SRE-Access-Requests, SRE
RLazarus closed T270125: Requesting access to deployment group for STran as Resolved.

@thcipriani @aezell Thanks!

Dec 16 2020, 11:49 PM · SRE, SRE-Access-Requests
RLazarus updated the task description for T270125: Requesting access to deployment group for STran.
Dec 16 2020, 11:49 PM · SRE, SRE-Access-Requests
RLazarus updated the task description for T270125: Requesting access to deployment group for STran.
Dec 16 2020, 6:30 PM · SRE, SRE-Access-Requests
RLazarus closed T269678: Requesting access to Analytics Data for toan as Resolved.

@toan You should be all set! The access group changes might take up to 30 min to roll out everywhere. Let me know if you have any trouble.

Dec 16 2020, 5:41 PM · SRE, SRE-Access-Requests
RLazarus closed T269843: NDA for Superset Request from WMDE Employee - Mohammed Sadat as Resolved.

@KFrancis Thanks!

Dec 16 2020, 5:40 PM · SRE, LDAP-Access-Requests
RLazarus added a comment to T269678: Requesting access to Analytics Data for toan.

Already in the nda group:

Dec 16 2020, 5:14 PM · SRE, SRE-Access-Requests
RLazarus updated the task description for T269678: Requesting access to Analytics Data for toan.
Dec 16 2020, 5:03 PM · SRE, SRE-Access-Requests
RLazarus closed T269777: Requesting access to releasers-wikibase for toan as Resolved.

@KFrancis Thank you!

Dec 16 2020, 5:03 PM · SRE, SRE-Access-Requests
RLazarus awarded T270220: ifup@eno1.service failed on some buster hosts a Love token.
Dec 16 2020, 3:55 PM · SRE, serviceops
RLazarus added a comment to T270220: ifup@eno1.service failed on some buster hosts.

Oh wow, I filed this and went to bed, love to wake up and see it fully handled. :) Thanks all!

Dec 16 2020, 3:54 PM · SRE, serviceops

Dec 15 2020

RLazarus triaged T270220: ifup@eno1.service failed on some buster hosts as High priority.
Dec 15 2020, 11:24 PM · SRE, serviceops
RLazarus created T270220: ifup@eno1.service failed on some buster hosts.
Dec 15 2020, 11:23 PM · SRE, serviceops
RLazarus added a comment to T259979: Redirect wikimedia.org/research to research.wikimedia.org instead of some external closed survey.

Quick correction -- this is now live on all appservers, but the old URL is still cached by the traffic layer for 24 hours or so. We could purge the cache if we were in a great hurry, but all things being equal, safer to let it expire -- I'll check back tomorrow and make sure this updated as expected.

Dec 15 2020, 5:32 PM · Research, Wikimedia-Apache-configuration, SRE
RLazarus added a comment to T259979: Redirect wikimedia.org/research to research.wikimedia.org instead of some external closed survey.

Quick correction -- this is now live on all appservers, but the old URL is still cached by the traffic layer for 24 hours or so. We could purge the cache if we were in a great hurry, but all things being equal, safer to let it expire -- I'll check back tomorrow and make sure this updated as expected.

Dec 15 2020, 4:58 PM · Research, Wikimedia-Apache-configuration, SRE
RLazarus closed T259979: Redirect wikimedia.org/research to research.wikimedia.org instead of some external closed survey as Resolved.

Deployed and tested:

Dec 15 2020, 4:24 PM · Research, Wikimedia-Apache-configuration, SRE
RLazarus added a comment to T270125: Requesting access to deployment group for STran.

(P.S. Today is technology's department-wide Fun Day, so further progress from Tyler and me might not come until tomorrow; sorry for the inconvenience. Give a yell if this is urgent, and we can get it taken care of.)

Dec 15 2020, 3:53 PM · SRE, SRE-Access-Requests
RLazarus updated the task description for T270125: Requesting access to deployment group for STran.
Dec 15 2020, 3:50 PM · SRE, SRE-Access-Requests
RLazarus moved T270125: Requesting access to deployment group for STran from Awaiting User Input to Manager/NDA Approval/Confirmation on the SRE-Access-Requests board.

@STran Thanks!

Dec 15 2020, 3:49 PM · SRE, SRE-Access-Requests
RLazarus claimed T259979: Redirect wikimedia.org/research to research.wikimedia.org instead of some external closed survey.
Dec 15 2020, 3:35 PM · Research, Wikimedia-Apache-configuration, SRE
RLazarus added a comment to T259979: Redirect wikimedia.org/research to research.wikimedia.org instead of some external closed survey.

Happened to see this go by -- I've dropped a single comment on the review, requesting a test be updated to match, and then I'll merge ASAP. Apologies for letting it stall for so long!

Dec 15 2020, 3:35 PM · Research, Wikimedia-Apache-configuration, SRE
Dzahn awarded T269693: mediawiki_job_wikidata-updateQueryServiceLag failing a Barnstar token.
Dec 15 2020, 2:45 AM · Discovery-Search (Current work), SRE, Wikidata, Wikidata-Query-Service

Dec 14 2020

RLazarus added a comment to T269678: Requesting access to Analytics Data for toan.

NDA discussion is happening over at T269777, then this will be ready to go.

Dec 14 2020, 11:11 PM · SRE, SRE-Access-Requests
RLazarus added a comment to T269777: Requesting access to releasers-wikibase for toan.

@toan In addition to the agreement you signed with WMDE, you'll need to sign another with WMF -- @KFrancis will get you set up. Once that's taken care of, I can go ahead on both this task and T269678.

Dec 14 2020, 11:10 PM · SRE, SRE-Access-Requests
RLazarus added a comment to T269843: NDA for Superset Request from WMDE Employee - Mohammed Sadat.

I've taken over SRE clinic duty from @jbond for the week -- @KFrancis I've emailed you his address. Once that's all set, I can go ahead with granting access. Thanks!

Dec 14 2020, 11:08 PM · SRE, LDAP-Access-Requests
RLazarus closed T269696: LDAP access to wmf group for Matt Cleinman as Resolved.

@MattCleinman Thanks for the update! I've added you to the wmf group.

Dec 14 2020, 10:17 PM · SRE, LDAP-Access-Requests
RLazarus assigned T270125: Requesting access to deployment group for STran to Tchanders.

@Tchanders From the "WIP" in the title, I'm guessing this isn't ready for SRE to work on yet, so I'm assigning it back to you. Feel free to pass it to me when it's ready for action. If you have any questions or you'd like some help getting it filled out, just say the word!

Dec 14 2020, 9:40 PM · SRE-Access-Requests, SRE
RLazarus closed T269693: mediawiki_job_wikidata-updateQueryServiceLag failing as Resolved.

Yep, the alert has cleared. Thanks!

Dec 14 2020, 7:15 PM · Discovery-Search (Current work), SRE, Wikidata-Query-Service, Wikidata

Dec 8 2020

RLazarus created T269693: mediawiki_job_wikidata-updateQueryServiceLag failing.
Dec 8 2020, 5:01 PM · Discovery-Search (Current work), SRE, Wikidata-Query-Service, Wikidata
RLazarus added a member for acl*sre-team: Legoktm.
Dec 8 2020, 2:52 PM

Dec 3 2020

RLazarus added a comment to T266717: Disable maintenance scripts via conftool.

Oh, that's a good idea! We'd set maintenance_host to the empty string before the switchover, so that no new jobs would start anywhere, then set it to the new hostname afterward.

Dec 3 2020, 11:00 PM · Patch-For-Review, Datacenter-Switchover, conftool, serviceops, SRE
RLazarus updated the task description for T269179: Updates to warmup script (2020-2021).
Dec 3 2020, 3:48 PM · Patch-For-Review, Datacenter-Switchover, SRE, serviceops