Dzahn (Daniel Zahn)
Operations Engineer

Projects (17)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Sep 30 2014, 4:39 PM (176 w, 6 d)
Availability
Available
IRC Nick
mutante
LDAP User
Dzahn
MediaWiki User
Unknown

Recent Activity

Fri, Feb 16

Dzahn added a comment to T160357: Allow those with CheckUser right to access AbuseLog private information on WMF projects.

The situation before was to run daily. Since there were issues with it i temp. reduced it to weekly. Then "Reedy fixed it(tm)" happened and i simply reverted to how it was before just now. So it's back to daily.

Fri, Feb 16, 6:29 PM · WMF-Legal, Patch-For-Review, Wikimedia-Site-requests, Stewards-and-global-tools, Security-Team, AbuseFilter
Dzahn added a comment to T37707: Complete unification of all accounts to SUL.

fwiw: 55 people just got a notification and clicked on it just to be informed that this task isn't Epic anymore but now just "other completed". worth it?

Fri, Feb 16, 6:12 PM · User-notice, Notice, Epic, Roadmap, SUL-Finalization, Tracking

Thu, Feb 15

Dzahn added a comment to T182832: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state.

I fixed the server status page. it's available again now.

What was the issue? I can see a big hole in https://grafana.wikimedia.org/dashboard/db/phabricator?orgId=1&panelId=7&fullscreen&from=now-30d&to=now :(

Thu, Feb 15, 9:29 PM · Patch-For-Review, User-Elukey, Release-Engineering-Team (Kanban), Operations, Phabricator
Dzahn added a comment to T187035: Ops Onboarding for Valentín Gutiérrez.

@Vgutierrez re: Icinga command permissions. should be all done. the ultimate test is if you try a "schedule downtime" or "disable/enable notifications" or "send acknowledgement" for something from the Icinga web ui. caveat is that auth_ldap will let you login with and without capitalization but to get the permissions above you need to match the "cn" from LDAP, so capitalized. With the other version you would still be logged in but not have the permissions.

Thu, Feb 15, 5:15 PM · Patch-For-Review, Ops-Access-Requests, Traffic, Operations
Dzahn awarded Blog Post: Phabricator Updates for February 2018 a Party Time token.
Thu, Feb 15, 1:36 PM · Release, Phabricator-Bot-Requests
Dzahn added a comment to T187434: Include apache_exporter in puppet module apache.

But we also want to replace the apache module with the httpd module (which doesn't have monitoring.pp anymore because we didn't want the diamond collector). So it should be added there as well.

Thu, Feb 15, 1:34 PM · User-fgiunchedi, Goal, Operations
Dzahn added a comment to T187035: Ops Onboarding for Valentín Gutiérrez.

I just added myself to root and security aliases.

Thu, Feb 15, 1:31 PM · Patch-For-Review, Ops-Access-Requests, Traffic, Operations
Dzahn awarded T177974: Drop #wikimedia-codereview channel a Like token.
Thu, Feb 15, 5:31 AM · Patch-For-Review, wikimedia-irc-freenode
Dzahn added a comment to T182832: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state.

I fixed the server status page. it's available again now.

Thu, Feb 15, 2:46 AM · Patch-For-Review, User-Elukey, Release-Engineering-Team (Kanban), Operations, Phabricator

Tue, Feb 13

Dzahn added a comment to T185862: icinga ACK shows as CRIT when delivered via SMS.

before (there is "Subject: " in each message and the same content is repeated twice. the actual status as in "PROBLEM" or "ACK" is not shown at all. which means sending an ACK shows as another CRIT)

Tue, Feb 13, 10:30 PM · Icinga, Patch-For-Review, Operations, monitoring
Dzahn closed T187055: ldap/ops membership for vgutierrez as Resolved.

was already done in T187035#3966316

Tue, Feb 13, 10:23 PM · LDAP-Access-Requests, Operations
Dzahn closed T187055: ldap/ops membership for vgutierrez, a subtask of T187035: Ops Onboarding for Valentín Gutiérrez, as Resolved.
Tue, Feb 13, 10:23 PM · Patch-For-Review, Ops-Access-Requests, Traffic, Operations
Dzahn updated the task description for T187035: Ops Onboarding for Valentín Gutiérrez.
Tue, Feb 13, 10:09 PM · Patch-For-Review, Ops-Access-Requests, Traffic, Operations
Dzahn added a comment to T187035: Ops Onboarding for Valentín Gutiérrez.

@RobH could you do one more Racktables user? thanks!

Tue, Feb 13, 10:06 PM · Patch-For-Review, Ops-Access-Requests, Traffic, Operations
Dzahn added a comment to T187035: Ops Onboarding for Valentín Gutiérrez.

please take a look at

Tue, Feb 13, 10:04 PM · Patch-For-Review, Ops-Access-Requests, Traffic, Operations
Dzahn added a comment to T187035: Ops Onboarding for Valentín Gutiérrez.

subscribed you to both ops mailing lists (others like wikitech-l are optional and self-service)

Tue, Feb 13, 10:01 PM · Patch-For-Review, Ops-Access-Requests, Traffic, Operations
Dzahn added a comment to T187035: Ops Onboarding for Valentín Gutiérrez.

added to "WMF-NDA" in Phabricator

Tue, Feb 13, 9:43 PM · Patch-For-Review, Ops-Access-Requests, Traffic, Operations
Dzahn added a member for WMF-NDA: Vgutierrez.
Tue, Feb 13, 9:42 PM
Dzahn added a comment to T182832: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state.

I didn't even mean to imply you have to rush it, just that it's not worth to upload a puppet change for like one more (weekly) restart that we can do manually.

Tue, Feb 13, 7:22 PM · Patch-For-Review, User-Elukey, Release-Engineering-Team (Kanban), Operations, Phabricator
Dzahn added a comment to T182832: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state.

I suppose we don't bother with the cron to restart Apache anymore and first wait for the version upgrade.

Tue, Feb 13, 6:40 PM · Patch-For-Review, User-Elukey, Release-Engineering-Team (Kanban), Operations, Phabricator

Mon, Feb 12

Dzahn added a comment to T151070: Experiment with php 7.1 on phabricator.

T182832#3965043 now about upgrading to 7.1. experimenting early wouldn't have been bad after all.

Mon, Feb 12, 10:48 PM · Phabricator
Dzahn added a comment to T160714: Upload php7.1 to apt.wm.org.

This issue is coming back on T182832

Mon, Feb 12, 10:46 PM · Phabricator, Operations
Dzahn updated subscribers of T182832: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state.

There is T151070 (Experiment with php 7.1 on phabricator) and T160714 (Upload php7.1 to apt.wm.org) but it was declined. See details from @Muehlenhoff on T160714#3109333.

Mon, Feb 12, 10:45 PM · Patch-For-Review, User-Elukey, Release-Engineering-Team (Kanban), Operations, Phabricator
Dzahn added a comment to T187078: Re-consider ` >/dev/null 2>&1` as output of many cron'd MW maintenance scripts.

17:30 < Hauskatze> I wonder if for the next round of https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/mediawiki/manifests/maintenance/purge_abusefilter.pp;cacc0b5224994e40df6c23bc7b8781305061ed57$4 we could actually store the log ?
17:31 < Hauskatze> re-running after fixing it, not running since 2016

Mon, Feb 12, 10:37 PM · Patch-For-Review, Wikimedia-General-or-Unknown, Operations
Dzahn added a comment to T182832: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state.
  1. Since #1 is a wontfix in php 5.6, should we look into upgrading to php 7.1? That would seem like a very wise thing to do for many reasons.
Mon, Feb 12, 8:59 PM · Patch-For-Review, User-Elukey, Release-Engineering-Team (Kanban), Operations, Phabricator
Dzahn added a comment to T187035: Ops Onboarding for Valentín Gutiérrez.

added to WMF-NDA-Requests group on Phabricator

Mon, Feb 12, 5:44 PM · Patch-For-Review, Ops-Access-Requests, Traffic, Operations
Dzahn added a member for WMF-NDA-Requests: Vgutierrez.
Mon, Feb 12, 5:43 PM

Fri, Feb 9

Dzahn closed T186020: Expand meitnerium's root partition to 100G as Resolved.
Fri, Feb 9, 8:16 PM · Operations, Analytics-Kanban, User-Elukey
Dzahn added a project to T186918: prometheus: ganglia-gen and outdated Ganglia:cluster resource name: Prometheus-metrics-monitoring.
Fri, Feb 9, 8:14 PM · Prometheus-metrics-monitoring, Patch-For-Review, Operations
Dzahn updated subscribers of T186918: prometheus: ganglia-gen and outdated Ganglia:cluster resource name.
Fri, Feb 9, 8:12 PM · Prometheus-metrics-monitoring, Patch-For-Review, Operations
Dzahn created T186918: prometheus: ganglia-gen and outdated Ganglia:cluster resource name.
Fri, Feb 9, 8:12 PM · Prometheus-metrics-monitoring, Patch-For-Review, Operations
Dzahn added a comment to T184116: Setup gitiles for use with gerrit as a repo viewer.

I agree we should _not_ use a second domain, yea. Also the short version with just "g" or something similar seems nice.

Fri, Feb 9, 3:58 AM · Patch-For-Review, Gerrit
Dzahn claimed T185862: icinga ACK shows as CRIT when delivered via SMS.
Fri, Feb 9, 12:05 AM · Icinga, Patch-For-Review, Operations, monitoring

Thu, Feb 8

Dzahn added a comment to T186826: Delete email list “Collaboration Standup”.

@jmatazzoni If it's a group (list) of people but the word "lists" isn't actually part of the address. (something@wikimedia.org as opposed to something@lists.wikimedia.org) then that means it's handled by OIT. You can reach them via email at techsupport@wikimedia.org for any questions. If it is a real "lists.wikimedia.org" list then you are in the right place in Phabricator. In this case it was the former though.

Thu, Feb 8, 11:01 PM · Wikimedia-Mailing-lists
hashar awarded T186288: replace all Ubuntu (trusty) hosts in production with Debian a Love token.
Thu, Feb 8, 10:01 PM · cloud-services-team, Operations
Dzahn updated the task description for T186288: replace all Ubuntu (trusty) hosts in production with Debian.
Thu, Feb 8, 10:00 PM · cloud-services-team, Operations
Dzahn reopened T186020: Expand meitnerium's root partition to 100G as "Open".
Thu, Feb 8, 9:42 PM · Operations, Analytics-Kanban, User-Elukey
Dzahn reassigned T186020: Expand meitnerium's root partition to 100G from Dzahn to elukey.

Icinga for meitnerium looks fine. no disk space warnings.

Thu, Feb 8, 9:42 PM · Operations, Analytics-Kanban, User-Elukey
Dzahn closed T186020: Expand meitnerium's root partition to 100G as Resolved.
Thu, Feb 8, 9:33 PM · Operations, Analytics-Kanban, User-Elukey
Dzahn awarded T186020: Expand meitnerium's root partition to 100G a Hungry Hippo token.
Thu, Feb 8, 9:32 PM · Operations, Analytics-Kanban, User-Elukey
Dzahn updated the task description for T186288: replace all Ubuntu (trusty) hosts in production with Debian.
Thu, Feb 8, 9:26 PM · cloud-services-team, Operations
Dzahn updated the task description for T186288: replace all Ubuntu (trusty) hosts in production with Debian.
Thu, Feb 8, 9:20 PM · cloud-services-team, Operations
Dzahn added a parent task for T172487: decom iridium: T152129: reinstall iridium (phabricator) as phab1001 with jessie.
Thu, Feb 8, 8:43 PM · ops-eqiad, hardware-requests, Operations
Dzahn added a subtask for T152129: reinstall iridium (phabricator) as phab1001 with jessie: T172487: decom iridium.
Thu, Feb 8, 8:43 PM · RelEng-Archive-FY201718-Q1, Phabricator, Operations
Dzahn updated the task description for T186288: replace all Ubuntu (trusty) hosts in production with Debian.
Thu, Feb 8, 8:40 PM · cloud-services-team, Operations

Wed, Feb 7

Dzahn closed T185493: Onboard bstorm to WMF as Resolved.

Looks like we are all done. If there are any issues or things missing, please just reopen it.

Wed, Feb 7, 12:19 AM · Patch-For-Review, cloud-services-team (Kanban), Operations
Dzahn updated the task description for T185493: Onboard bstorm to WMF.
Wed, Feb 7, 12:18 AM · Patch-For-Review, cloud-services-team (Kanban), Operations
Dzahn added a comment to T185493: Onboard bstorm to WMF.

Confirmed i saw the new signature now.

Wed, Feb 7, 12:15 AM · Patch-For-Review, cloud-services-team (Kanban), Operations

Tue, Feb 6

Dzahn added a comment to T185493: Onboard bstorm to WMF.

I can't see your signature yet. I tried my default keyserver (hkps://hkps.pool.sks-keyservers.net) and pgp.mit.edu. Depending on which keyserver you used it might just take a while until they have synced.

Tue, Feb 6, 10:15 PM · Patch-For-Review, cloud-services-team (Kanban), Operations
Dzahn removed a subtask for T127083: Deletion and Blocking: Unknown Object (Task).
Tue, Feb 6, 9:10 PM
Dzahn removed a subtask for T122127: Translation of namespaces for Gilaki : Unknown Object (Task).
Tue, Feb 6, 9:08 PM · MW-1.27-release-notes, MW-1.27-release (WMF-deploy-2016-01-12_(1.27.0-wmf.10)), MediaWiki-Internationalization
Dzahn added a comment to T122035: Need to access revision histories of wikipedia pages.

The subtask was automatically created by Herald (for shell access requests) but it was closed as Invalid now.

Tue, Feb 6, 9:07 PM · MediaWiki-Revision-deletion, Cloud-Services
Dzahn removed a subtask for T122035: Need to access revision histories of wikipedia pages: Unknown Object (Task).
Tue, Feb 6, 9:06 PM · MediaWiki-Revision-deletion, Cloud-Services
Dzahn removed a subtask for T118677: Nastaleeq font for Western Punjabi: Unknown Object (Task).
Tue, Feb 6, 9:05 PM · UniversalLanguageSelector
Dzahn added a comment to T184116: Setup gitiles for use with gerrit as a repo viewer.

I think the answer is "no separate domain"?

Tue, Feb 6, 8:55 PM · Patch-For-Review, Gerrit
Dzahn added a comment to T184116: Setup gitiles for use with gerrit as a repo viewer.

cool, this ticket seems resolved

Tue, Feb 6, 8:55 PM · Patch-For-Review, Gerrit
Dzahn added a comment to T185493: Onboard bstorm to WMF.

get key:

Tue, Feb 6, 8:49 PM · Patch-For-Review, cloud-services-team (Kanban), Operations
Dzahn added a comment to T185282: Create subdomain for Design and Wikimedia User Interface Style Guide .

@Prtksxna I asked around a bit and yea there is already precedence for this. Others are doing it that way. So as long as i can let puppet clone from Gerrit and you can handle how it gets synced into Gerrit with people setting up Gerrit projects, that would work, yea.

Tue, Feb 6, 8:39 PM · Domains, Operations, WMF-Design, Design, WikimediaUI Style Guide, Traffic
Dzahn added a comment to T185493: Onboard bstorm to WMF.

@chasemp Adding the key to pwstore requires that it has at least 2 signatures on it. Since you already confirmed the key during hangout, could you add one of those? Like sign the key and then upload the signed version to keyserver? Then it should show signatures with gpg --list-sigs 2051251AF5172F75 I could then do the second one.

Tue, Feb 6, 8:33 PM · Patch-For-Review, cloud-services-team (Kanban), Operations
Dzahn closed T179871: Subdomain and hosting for research landing page as Resolved.

Ok, since it's already on the Done colum and all subtasks are checked off, seems pretty resolved to me. Of course reopen if anyone feels differently.

Tue, Feb 6, 8:09 PM · Epic, Research-outreach, Research, Research-management
Dzahn closed T179871: Subdomain and hosting for research landing page, a subtask of T107389: Wikimedia Research discoverability: designing a landing page for the team, as Resolved.
Tue, Feb 6, 8:09 PM · Research-landing-page, Research, Epic, Research-management
Dzahn added a comment to T179871: Subdomain and hosting for research landing page.

Can also be closed as resolved now, right?

Tue, Feb 6, 7:29 PM · Epic, Research-outreach, Research, Research-management
Dzahn closed T183916: Create subdomain for Research landing page as Resolved.
Tue, Feb 6, 6:45 PM · Research, Operations, Domains, Traffic
Dzahn closed T183916: Create subdomain for Research landing page, a subtask of T179871: Subdomain and hosting for research landing page, as Resolved.
Tue, Feb 6, 6:45 PM · Epic, Research-outreach, Research, Research-management
Dzahn added a comment to T183916: Create subdomain for Research landing page.

I re-enabled the git cloning. At first there were some conflicts when puppet tried to git pull. Deleting the entire /srv/org/wikimedia/research directory and running puppet fixed that and files were then current.

Tue, Feb 6, 5:38 PM · Research, Operations, Domains, Traffic
Dzahn added a comment to T186620: Phabricator down due to "Failed to `proc_open()`: proc_open() expects parameter 2 to be array".

https://wikitech.wikimedia.org/wiki/Incident_documentation/20180206-Phabricator

Tue, Feb 6, 5:24 PM · User-Elukey, Release-Engineering-Team, Operations, Phabricator
Dzahn created P6663 mw1262-hhvm-20180206.
Tue, Feb 6, 4:32 PM

Mon, Feb 5

Dzahn added a comment to T186415: Create trusted group in gerrit.

Yea we should disable that feature or people will start using it.

Mon, Feb 5, 7:58 PM · Developer-Relations, Gerrit
Dzahn updated the task description for T185493: Onboard bstorm to WMF.
Mon, Feb 5, 6:37 PM · Patch-For-Review, cloud-services-team (Kanban), Operations
Dzahn added a member for acl*operations-team: Bstorm.
Mon, Feb 5, 6:36 PM
Dzahn added a comment to T185591: Requesting access to ops group in admin for bstorm.
[bast1001:~] $ id bstorm
uid=18713(bstorm) gid=500(wikidev) groups=500(wikidev),50(staff),700(ops),600(all-users)
Mon, Feb 5, 6:28 PM · Patch-For-Review, Operations, Ops-Access-Requests
Dzahn changed the status of T185591: Requesting access to ops group in admin for bstorm, a subtask of T185493: Onboard bstorm to WMF, from Stalled to Open.
Mon, Feb 5, 5:56 PM · Patch-For-Review, cloud-services-team (Kanban), Operations
Dzahn changed the status of T185591: Requesting access to ops group in admin for bstorm from Stalled to Open.
Mon, Feb 5, 5:56 PM · Patch-For-Review, Operations, Ops-Access-Requests
Dzahn added a comment to T186415: Create trusted group in gerrit.

Do we really want private changes in gerrit?

Mon, Feb 5, 4:58 PM · Developer-Relations, Gerrit

Fri, Feb 2

Dzahn updated subscribers of T185988: Requesting access to analytics-users / webrequest for Esteban.
Fri, Feb 2, 5:19 PM · Ops-Access-Requests, Operations
Dzahn added a comment to T186123: rack/setup/install db2093 (WAS: rack/setup/install tendril2001).

After talking with jcrespo on IRC:

Fri, Feb 2, 2:20 PM · Patch-For-Review, ops-codfw, DBA, Operations
Dzahn added a comment to T186020: Expand meitnerium's root partition to 100G.

gnt-instance shutdown <fqdn>

gnt-instance startup <fqdn>
Fri, Feb 2, 1:55 PM · Operations, Analytics-Kanban, User-Elukey
Dzahn added a comment to T172487: decom iridium.

Yes, i can't do them though because i don't have the access to disable switch ports.

Fri, Feb 2, 1:53 PM · ops-eqiad, hardware-requests, Operations
Dzahn updated the task description for T186288: replace all Ubuntu (trusty) hosts in production with Debian.
Fri, Feb 2, 1:23 AM · cloud-services-team, Operations
Dzahn added a project to T186288: replace all Ubuntu (trusty) hosts in production with Debian: cloud-services-team.
Fri, Feb 2, 1:20 AM · cloud-services-team, Operations
Dzahn renamed T186288: replace all Ubuntu (trusty) hosts in production with Debian from upgrade all Ubuntu (trusty) hosts in production to replace all Ubuntu (trusty) hosts in production with Debian.
Fri, Feb 2, 1:18 AM · cloud-services-team, Operations
Dzahn updated the task description for T186288: replace all Ubuntu (trusty) hosts in production with Debian.
Fri, Feb 2, 1:17 AM · cloud-services-team, Operations
Dzahn created T186288: replace all Ubuntu (trusty) hosts in production with Debian.
Fri, Feb 2, 1:13 AM · cloud-services-team, Operations
Dzahn added a comment to T186020: Expand meitnerium's root partition to 100G.

@elukey so yea, now we'd have to restart the instance from ganeti, as the comment above says rebooting from within the instance won't do it. You said above archiva downtime needs some heads-up, so didn't do that yet, but the new hardware is there if you wanna continue

Fri, Feb 2, 12:39 AM · Operations, Analytics-Kanban, User-Elukey
Dzahn added a comment to T186020: Expand meitnerium's root partition to 100G.
..
Fri Feb  2 00:23:13 2018  - INFO: - device disk/1: 99.30% done, 21s remaining (estimated)
Fri Feb  2 00:23:34 2018  - INFO: - device disk/1: 100.00% done, 1s remaining (estimated)
Fri Feb  2 00:23:35 2018  - INFO: - device disk/1: 100.00% done, 0s remaining (estimated)
Fri Feb  2 00:23:36 2018  - INFO: Instance meitnerium.wikimedia.org's disks are in sync
Modified instance meitnerium.wikimedia.org
 - disk/1 -> add:size=102400,mode=rw
Please don't forget that most parameters take effect only at the next (re)start of the instance initiated by ganeti; restarting from within the instance will not be enough.
[ganeti1004:~] $
Fri, Feb 2, 12:34 AM · Operations, Analytics-Kanban, User-Elukey
Dzahn triaged T185151: replace msw1-esams as Normal priority.
Fri, Feb 2, 12:15 AM · Operations, netops, ops-esams
Dzahn triaged T185350: Vet reliability of the response_size field for data analysis purposes as Normal priority.

@Tbayer purely from a ticket triaging perspective: since the ticket title is "vet reliability of the response_size field.." and you said "we can conclude with reasonable confidence that erroneous logging of the response_size field is not the cause", can we call it resolved?

Fri, Feb 2, 12:06 AM · Operations, Traffic, Analytics-Data-Quality

Thu, Feb 1

Dzahn triaged T185494: Degraded RAID on restbase-dev1006 as Normal priority.
Thu, Feb 1, 11:49 PM · User-Eevans, ops-eqiad, Operations
Dzahn triaged T185582: Wikipedia no longer accessible to those using some braille devices as Normal priority.
Thu, Feb 1, 11:49 PM · Operations, Traffic, Browser-Support-Internet-Explorer, Accessibility
Dzahn added a comment to T185582: Wikipedia no longer accessible to those using some braille devices.

@Cameron11598 Could you forward Brandon's answer above to the original reporter? Let us know if there is any feedback from him. As i see it now i think that reply is the best we can offer in this case and there isn't much else on this ticket that is really actionable on our end.

Thu, Feb 1, 11:49 PM · Operations, Traffic, Browser-Support-Internet-Explorer, Accessibility
Dzahn triaged T185814: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 as Low priority.
Thu, Feb 1, 11:46 PM · Operations, Puppet
Dzahn added a comment to T185814: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5.

@Paladox can you add a specific host name where this happens?

Thu, Feb 1, 11:45 PM · Operations, Puppet
Dzahn added a project to T185969: Beta English Wikipedia: History of the page 'Bird' generates a 500 or 503 error: Beta-Cluster-Infrastructure.
Thu, Feb 1, 11:44 PM · Beta-Cluster-Infrastructure, Operations, Wikimedia-General-or-Unknown
Dzahn triaged T185969: Beta English Wikipedia: History of the page 'Bird' generates a 500 or 503 error as Normal priority.

Can't reproduce it right now. It works for me.

Thu, Feb 1, 11:44 PM · Beta-Cluster-Infrastructure, Operations, Wikimedia-General-or-Unknown
Dzahn triaged T186020: Expand meitnerium's root partition to 100G as High priority.
Thu, Feb 1, 11:35 PM · Operations, Analytics-Kanban, User-Elukey
Dzahn added a comment to T186020: Expand meitnerium's root partition to 100G.

Yea, that makes sense. I also think it's the easiest way to create a new disk in ganeti and then mount it.

Thu, Feb 1, 11:34 PM · Operations, Analytics-Kanban, User-Elukey
Dzahn assigned T186049: db1051 database host BBU issues to Cmjohnson.
Thu, Feb 1, 11:33 PM · Patch-For-Review, ops-eqiad, Operations, DBA
Dzahn triaged T186049: db1051 database host BBU issues as Normal priority.
Thu, Feb 1, 11:33 PM · Patch-For-Review, ops-eqiad, Operations, DBA
Dzahn triaged T186153: outdated DjVu file page thumbnail in cache as Normal priority.
Thu, Feb 1, 11:32 PM · media-storage, Wikimedia-Site-requests, Operations
Dzahn triaged T186069: Icinga: page in case all MediaWiki are throwing 5xx as High priority.
Thu, Feb 1, 11:30 PM · Patch-For-Review, Wikimedia-Incident, Icinga, Operations, monitoring