Page MenuHomePhabricator

Dzahn (Daniel Zahn)
DisabledAdministrator

Projects (17)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Thursday

  • Clear sailing ahead.

User Details

User Since
Sep 30 2014, 4:39 PM (245 w, 6 d)
Roles
Administrator, Disabled
Availability
Available
IRC Nick
mutante
LDAP User
Dzahn
MediaWiki User
Unknown

Recent Activity

Fri, May 31

Dzahn added a comment to T219850: contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%):.

The long term fix should still be T178663

Fri, May 31, 12:08 AM · Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure, Operations

Thu, May 30

Dzahn added a comment to T224706: Debian mirror in sync with upstream .

The Debian mirror sync uses ftpsync, unlike the Ubuntu mirror sync. So you won't find a puppetized cron and rsync like you do for Ubuntu.

Thu, May 30, 11:37 PM · Operations
Dzahn created T224691: labmon / prometheus - query error - monitoring artifacts - Icinga UNKNOWN.
Thu, May 30, 7:01 PM · observability, Cloud-Services
Dzahn added a comment to T223393: switch wikitech to PHP 7.2.

Once you've done that, I'd say let's be bold and just change the proxy/rewrite rules from

Thu, May 30, 3:27 PM · wikitech.wikimedia.org, Patch-For-Review, PHP 7.2 support, serviceops, Operations
Dzahn updated the task description for T224549: Track remaining jessie systems in production.
Thu, May 30, 12:12 AM · Operations
Dzahn merged task T224575: Migrate ununpentium/RT to Stretch/Buster into T180641: reinstall RT server with private IP and stretch.
Thu, May 30, 12:12 AM · Operations
Dzahn merged T224575: Migrate ununpentium/RT to Stretch/Buster into T180641: reinstall RT server with private IP and stretch.
Thu, May 30, 12:12 AM · Operations

Wed, May 29

Dzahn added a comment to T224575: Migrate ununpentium/RT to Stretch/Buster.

basically a duplicate of T180641

Wed, May 29, 11:35 PM · Operations
Dzahn claimed T224575: Migrate ununpentium/RT to Stretch/Buster.
Wed, May 29, 11:35 PM · Operations
Dzahn updated the task description for T224549: Track remaining jessie systems in production.
Wed, May 29, 11:34 PM · Operations
Dzahn closed T224323: ganeti VM request - miscweb2001 - equivalent of krypton, a subtask of T224247: upgrade and rename krypton & create its codfw equivalent, as Resolved.
Wed, May 29, 11:34 PM · Patch-For-Review, serviceops, Operations
Dzahn closed T224323: ganeti VM request - miscweb2001 - equivalent of krypton as Resolved.

https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=miscweb2001

Wed, May 29, 11:34 PM · vm-requests, serviceops, Operations
Dzahn updated the task description for T190568: Reimage both phab1001 and phab2001 to stretch.
Wed, May 29, 11:31 PM · Release-Engineering-Team-TODO, Patch-For-Review, serviceops, Phabricator, Operations
Dzahn added a comment to T202367: Productionize dbproxy101[2-7].eqiad.wmnet and dbproxy200[1-4].

CCing @Dzahn as he expressed interest for this to happen in the past, and finally the hardware is here (no actionable needed from you at the moment).

Wed, May 29, 9:32 PM · Patch-For-Review, DBA
Dzahn added a comment to T149845: Something is wrong with installer root disk stuff.

I just ran into this when reinstalling phab2001 from jessie to stretch.

Wed, May 29, 8:22 PM · Operations
Dzahn updated subscribers of T222308: Close the engineering mailing list.

Not sure what counts as consensus for something like this

Wed, May 29, 8:02 PM · Operations, Wikimedia-Mailing-lists

Tue, May 28

Dzahn added a comment to T222788: Request to be added to the ldap/wmde group.

The user name "darthmon" cannot be found anywhere in the admin module. Please add accounts there when adding them to LDAP groups.

Tue, May 28, 10:56 PM · Patch-For-Review, WMF-Legal, LDAP-Access-Requests, Operations, WMF-NDA-Requests
Dzahn assigned T224507: Remove user Greta WMDE from wmde LDAP group to ayounsi.
Tue, May 28, 10:53 PM · LDAP-Access-Requests, Operations
Dzahn closed T224205: don't page all of SRE for phabricator 'phd' service not running as Resolved.
Tue, May 28, 10:50 PM · Patch-For-Review, Phabricator, observability
Dzahn added a comment to T197624: Improve visibility of incoming operations tasks.
transitioning the ~1400 existing tasks currently in "backlog" on the workboard to "acknowledged" without loads of manual work and triggering notifications?
Tue, May 28, 9:58 PM · User-herron, Operations
Dzahn added a comment to T187790: Phabricator: Clean up deadlocked apache processes.

removed again since we are not seeing the leaks anymore since our recent upgrade to stretch and phab1003

Tue, May 28, 8:58 PM · Patch-For-Review, Wikimedia-Incident, User-Elukey, Release-Engineering-Team (Kanban), Operations, Phabricator
Dzahn added a comment to T122144: Move most (all?) exim personal aliases to OIT.

I have removed a bunch of aliases where people responded they were not aware of having them or that they don't need them anymore.

Tue, May 28, 8:40 PM · Mail, Operations
Dzahn added a comment to T222651: Tracking failures in your Matomo Analytics.

There are new failures:

Tue, May 28, 8:31 PM · Analytics
Dzahn updated the task description for T223492: rack/setup/install dbproxy200[1-4].
Tue, May 28, 8:23 PM · Patch-For-Review, ops-codfw, Operations, DBA
Dzahn added a comment to T197624: Improve visibility of incoming operations tasks.
moving public open tasks from Backlog to Acknowledged
Tue, May 28, 8:04 PM · User-herron, Operations
Dzahn added a project to T224454: Create an alert for high memcached bw usage: observability.
Tue, May 28, 7:57 PM · observability, Performance-Team (Radar), User-Elukey, serviceops, Operations
Dzahn reopened T214494: Alerts for mjolnir daemons as "Open".
Tue, May 28, 7:55 PM · Icinga, Discovery-Search (Current work)
Dzahn added a comment to T214494: Alerts for mjolnir daemons.

Current Status: CRITICAL
(for 0d 6h 0m 16s)
Status Information: 121 gt 2

Tue, May 28, 7:54 PM · Icinga, Discovery-Search (Current work)
Dzahn added a comment to T221533: Decommission old coredb machines (<=db2042).

fyi db2035 is shown again as having unhealthy disks (https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=db2035&service=Device+not+healthy+-SMART-)

Tue, May 28, 7:45 PM · DBA
Dzahn added projects to T224517: netbox / netmon1002: netbox report related service units failed: Operations, netbox, observability.
Tue, May 28, 7:42 PM · observability, netbox, Operations
Dzahn created T224517: netbox / netmon1002: netbox report related service units failed.
Tue, May 28, 7:41 PM · observability, netbox, Operations

Sat, May 25

Dzahn added a project to T224313: Requesting access to icinga for tonycepo: observability.
Sat, May 25, 4:42 AM · observability, Operations, SRE-Access-Requests

Fri, May 24

Dzahn added a comment to T224247: upgrade and rename krypton & create its codfw equivalent.

miscweb1001/2001?

Fri, May 24, 10:48 PM · Patch-For-Review, serviceops, Operations
Dzahn created T224323: ganeti VM request - miscweb2001 - equivalent of krypton.
Fri, May 24, 10:38 PM · vm-requests, serviceops, Operations
Dzahn changed the status of T99531: [Task] move wikiba.se webhosting to wikimedia cluster from Open to Stalled.
Fri, May 24, 10:32 PM · Patch-For-Review, User-Addshore, serviceops, wikidata-tech-focus, Traffic, wikiba.se website, Operations, Wikidata-Sprint-2016-11-08, Wikidata
Dzahn changed the status of T99531: [Task] move wikiba.se webhosting to wikimedia cluster, a subtask of T108946: [Epic] Improve the development infrastructure , from Open to Stalled.
Fri, May 24, 10:32 PM · Epic, Wikidata
Dzahn updated the task description for T224247: upgrade and rename krypton & create its codfw equivalent.
Fri, May 24, 10:17 PM · Patch-For-Review, serviceops, Operations
Dzahn closed T224194: switch webserver_misc_apps to PHP 7.2 (7.1) as Declined.

using PHP 7.2 was declined in T224247#5209664

Fri, May 24, 10:16 PM · PHP 7.2 support, serviceops, Operations
Dzahn closed T224194: switch webserver_misc_apps to PHP 7.2 (7.1), a subtask of T210008: upgrade krypton (webserver_misc_apps) to stretch, as Declined.
Fri, May 24, 10:16 PM · serviceops, Operations
Dzahn renamed T224194: switch webserver_misc_apps to PHP 7.2 (7.1) from switch webserver_misc_apps to PHP 7.2 to switch webserver_misc_apps to PHP 7.2 (7.1).
Fri, May 24, 5:28 PM · PHP 7.2 support, serviceops, Operations
Dzahn added a comment to T223496: Requesting access to machines [stat1004, stat1005 (now stat1007), and stat1006] and groups for iflorez.

@Dzahn I guess @georgina would be more appropriate, the expiration contact should be related to the contractors point of contact, not the group's owner.

Fri, May 24, 5:21 PM · Operations, SRE-Access-Requests
Dzahn added a comment to T224254: User alias redirecting to another user alias.

Thanks for removing the number of aliases on our side in general. Note i had just opened a ticket on Zendesk for OIT as well to add all the remaining aliases to legal@ so we can remove all of them on our side.

Fri, May 24, 5:10 PM · Mail, Operations

Thu, May 23

Dzahn updated the task description for T224247: upgrade and rename krypton & create its codfw equivalent.
Thu, May 23, 9:11 PM · Patch-For-Review, serviceops, Operations
Dzahn triaged T224247: upgrade and rename krypton & create its codfw equivalent as Normal priority.
Thu, May 23, 8:50 PM · Patch-For-Review, serviceops, Operations
Dzahn claimed T224247: upgrade and rename krypton & create its codfw equivalent.
Thu, May 23, 8:49 PM · Patch-For-Review, serviceops, Operations
Dzahn created T224247: upgrade and rename krypton & create its codfw equivalent.
Thu, May 23, 8:49 PM · Patch-For-Review, serviceops, Operations
Dzahn added a comment to T190568: Reimage both phab1001 and phab2001 to stretch.

I will bring this up in my next subteam discussion meeting which should be in a week. Until then i will hold on to phab1001. Maybe we wait a few days to keep it as is.. and then install stretch.

Thu, May 23, 4:19 PM · Release-Engineering-Team-TODO, Patch-For-Review, serviceops, Phabricator, Operations
Dzahn added a comment to T125357: /maniphest/report/project/ : Maximum execution time of 10 seconds exceeded.

We increased the max_execution_time to 30s in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/477595/ for php-fpm.

So once we switch to stretch we can switch on php-fpm and then it should work.

Thu, May 23, 4:17 PM · Performance, Phabricator
Dzahn added a comment to T190568: Reimage both phab1001 and phab2001 to stretch.

I am thinking now we could make the process easier and just keep phab1003 as the prod server and just discuss whether we want to keep phab1001 as a permanent stand-by in the same DC or give it back to the spares pool.

Thu, May 23, 4:16 PM · Release-Engineering-Team-TODO, Patch-For-Review, serviceops, Phabricator, Operations
Dzahn added a comment to T190568: Reimage both phab1001 and phab2001 to stretch.

Phabricator has been switched to phab1003 as the prod server now and that meant:

Thu, May 23, 4:15 PM · Release-Engineering-Team-TODO, Patch-For-Review, serviceops, Phabricator, Operations
Dzahn updated the task description for T190568: Reimage both phab1001 and phab2001 to stretch.
Thu, May 23, 4:14 PM · Release-Engineering-Team-TODO, Patch-For-Review, serviceops, Phabricator, Operations
Dzahn awarded T182832: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state a Love token.
Thu, May 23, 4:13 PM · Patch-For-Review, serviceops, User-MModell, Wikimedia-Incident, Release-Engineering-Team (Kanban), Operations, Phabricator
Aklapper awarded T151070: Move Phabricator from PHP 7.0 to PHP 7.2 a Like token.
Thu, May 23, 11:04 AM · Patch-For-Review, PHP 7.2 support, Phabricator
mmodell awarded T151070: Move Phabricator from PHP 7.0 to PHP 7.2 a Orange Medal token.
Thu, May 23, 9:41 AM · Patch-For-Review, PHP 7.2 support, Phabricator
Dzahn created T224205: don't page all of SRE for phabricator 'phd' service not running.
Thu, May 23, 5:33 AM · Patch-For-Review, Phabricator, observability
Dzahn updated the task description for T221389: setup/install WMF7426 as phab1003.eqiad.wmnet.
Thu, May 23, 4:59 AM · Patch-For-Review, serviceops, Operations
Dzahn updated the task description for T221389: setup/install WMF7426 as phab1003.eqiad.wmnet.
Thu, May 23, 4:55 AM · Patch-For-Review, serviceops, Operations
Dzahn updated the task description for T190568: Reimage both phab1001 and phab2001 to stretch.
Thu, May 23, 4:51 AM · Release-Engineering-Team-TODO, Patch-For-Review, serviceops, Phabricator, Operations
Dzahn changed the status of T190568: Reimage both phab1001 and phab2001 to stretch from Stalled to Open.
Thu, May 23, 4:50 AM · Release-Engineering-Team-TODO, Patch-For-Review, serviceops, Phabricator, Operations
Dzahn changed the status of T190568: Reimage both phab1001 and phab2001 to stretch, a subtask of T125357: /maniphest/report/project/ : Maximum execution time of 10 seconds exceeded, from Stalled to Open.
Thu, May 23, 4:50 AM · Performance, Phabricator
Dzahn added a comment to T151070: Move Phabricator from PHP 7.0 to PHP 7.2.

phab is on phab1003 now which runs PHP 7.2

Thu, May 23, 4:49 AM · Patch-For-Review, PHP 7.2 support, Phabricator
Dzahn closed T151070: Move Phabricator from PHP 7.0 to PHP 7.2 as Resolved.
Thu, May 23, 4:49 AM · Patch-For-Review, PHP 7.2 support, Phabricator
Dzahn closed T221259: eqord - ulsfo Telia link down - IC-313592 as Resolved.

23:21 <+icinga-wm> RECOVERY - Router interfaces on cr3-ulsfo is OK: OK: host 198.35.26.192, interfaces up: 68, down: 0, dormant: 0, excluded: 0, unused: 0

Thu, May 23, 4:49 AM · Operations, netops
Dzahn added a comment to T221259: eqord - ulsfo Telia link down - IC-313592.
  • Maintenance window:

Start Date and Time: 2019-May-23 03:00 UTC
End Date and Time: 2019-May-23 07:00 UTC

Thu, May 23, 3:10 AM · Operations, netops
Dzahn reopened T221259: eqord - ulsfo Telia link down - IC-313592 as "Open".

and..it is DOWN again

Thu, May 23, 3:05 AM · Operations, netops
Dzahn added a comment to T221389: setup/install WMF7426 as phab1003.eqiad.wmnet.

02:35 mutante: phabricator - going read-write again

02:24 twentyafterfour: manually started aphlict on phab1003
02:06 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=phab1003-vcs.eqiad.wmnet
02:04 mutante: puppetmaster1001 - sudo -i conftool-merge
01:52 twentyafterfour: phabricator is now served by phab1003 though still in read-only mode for a bit longer
01:52 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=phab1003-vcs.eqiad.wmnet
01:49 mutante: puppetmaster1001 - conftool-merge
01:37 mutante: depooled phab1001-vcs from git-ssh via conftool
01:36 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=phab1001-vcs.eqiad.wmnet
01:33 mutante: run puppet on mx1001/mx2001 - switch mail route for phab to phab1003
01:30 mutante: switched from phab1001 to phab1003 - applied on cp1008 varnish canary first
01:28 twentyafterfour: stopping phd on phab1001
01:18 mutante: phabricator going readonly momentarily
01:09 twentyafterfour: extended phab downtime in icinga, actual downtime hasn't started yet, prep work taking longer than expected
00:45 mutante: phab1003 - rsyncing /srv/repos from phab1001

</pre>

Thu, May 23, 2:39 AM · Patch-For-Review, serviceops, Operations
Dzahn closed T221389: setup/install WMF7426 as phab1003.eqiad.wmnet as Resolved.
Thu, May 23, 2:37 AM · Patch-For-Review, serviceops, Operations
Dzahn closed T221389: setup/install WMF7426 as phab1003.eqiad.wmnet, a subtask of T215335: requesting WMF7426 as phabricator system in eqiad, as Resolved.
Thu, May 23, 2:37 AM · serviceops, Operations, hardware-requests

Wed, May 22

Dzahn created T224194: switch webserver_misc_apps to PHP 7.2 (7.1).
Wed, May 22, 10:44 PM · PHP 7.2 support, serviceops, Operations
Dzahn added a comment to T210008: upgrade krypton (webserver_misc_apps) to stretch.

grafana is not on this host anymore meanwhile. unlinking subtask , not blocking this anymore

Wed, May 22, 10:29 PM · serviceops, Operations
Dzahn renamed T210034: build grafana package for stretch (upgrade grafana stretch package to 6.x?) from build grafana package for stretch to build grafana package for stretch (upgrade grafana stretch package to 6.x?).
Wed, May 22, 10:28 PM · Operations
Dzahn updated subscribers of T210034: build grafana package for stretch (upgrade grafana stretch package to 6.x?).
Wed, May 22, 10:27 PM · Operations
Dzahn removed a parent task for T210034: build grafana package for stretch (upgrade grafana stretch package to 6.x?): T210008: upgrade krypton (webserver_misc_apps) to stretch.
Wed, May 22, 10:23 PM · Operations
Dzahn removed a subtask for T210008: upgrade krypton (webserver_misc_apps) to stretch: T210034: build grafana package for stretch (upgrade grafana stretch package to 6.x?).
Wed, May 22, 10:23 PM · serviceops, Operations
Dzahn updated the task description for T210008: upgrade krypton (webserver_misc_apps) to stretch.
Wed, May 22, 10:21 PM · serviceops, Operations
Dzahn updated the task description for T210008: upgrade krypton (webserver_misc_apps) to stretch.
Wed, May 22, 10:12 PM · serviceops, Operations
Dzahn added a comment to T210008: upgrade krypton (webserver_misc_apps) to stretch.

test

Wed, May 22, 10:12 PM · serviceops, Operations
Dzahn updated the task description for T210008: upgrade krypton (webserver_misc_apps) to stretch.
Wed, May 22, 10:08 PM · serviceops, Operations
Dzahn updated the task description for T210008: upgrade krypton (webserver_misc_apps) to stretch.
Wed, May 22, 10:07 PM · serviceops, Operations
Dzahn updated the task description for T210008: upgrade krypton (webserver_misc_apps) to stretch.
Wed, May 22, 10:06 PM · serviceops, Operations
Dzahn updated the task description for T210008: upgrade krypton (webserver_misc_apps) to stretch.
Wed, May 22, 10:03 PM · serviceops, Operations
Dzahn added a comment to T210034: build grafana package for stretch (upgrade grafana stretch package to 6.x?).

Just checked on this again and i notice that meanwhile somebody has done this.

Wed, May 22, 9:03 PM · Operations
Dzahn added a comment to T212883: Add nap.wikisource to Wikistats.

@Xqt What happened? Was it reverted?

Wed, May 22, 8:39 PM · VPS-project-Wikistats
Dzahn changed the status of T222391: Gerrit Hardware Upgrade, a subtask of T221026: Gerrit thread use GC thrashing, from Open to Stalled.
Wed, May 22, 8:38 PM · VPS-project-codesearch, Patch-For-Review, Release-Engineering-Team, Gerrit
Dzahn changed the status of T222391: Gerrit Hardware Upgrade from Open to Stalled.
Wed, May 22, 8:38 PM · Release-Engineering-Team-TODO, serviceops, Operations, Gerrit
Dzahn updated the task description for T223921: GSuite Test Domain Verification.
Wed, May 22, 5:43 PM · Operations, DNS, Traffic
Dzahn added a comment to T131541: Tools bastions are often unreliable.

@aborrero I am referring to the "cgred" module. I am now suggesting to delete it at https://gerrit.wikimedia.org/r/c/operations/puppet/+/511791

Wed, May 22, 4:52 PM · Toolforge, Patch-For-Review

Tue, May 21

Dzahn added a comment to T131541: Tools bastions are often unreliable.

I am interested if this is still being used nowadays. If it is we will need a systemd initscript for it and we'd want to convert it to use systemd::service, also to make it possible to upgrade to stretch. If not i would not bother or suggest to remove code.

Tue, May 21, 11:15 PM · Toolforge, Patch-For-Review
Dzahn added a comment to T223835: Configure wikimedia.org to enable *:wikimedia.org Matrix user IDs.

The comments on T215042#4977385 sounded like this wasn't going to be done, for the temporary evaluation that it is. ?

Tue, May 21, 10:31 PM · Patch-For-Review, Traffic, DNS, Operations, Wikimedia-Apache-configuration, Matrix
Dzahn updated the task description for T192830: Requesting access to production for SWAT deploy for Urbanecm.
Tue, May 21, 10:26 PM · User-zeljkofilipin, Release-Engineering-Team (Kanban), User-greg, User-Urbanecm, Operations, SRE-Access-Requests
Dzahn created T224065: cloudvirt1028 - no PS redundancy.
Tue, May 21, 10:00 PM · cloud-services-team (Kanban), Operations, ops-eqiad
Dzahn added a comment to T223496: Requesting access to machines [stat1004, stat1005 (now stat1007), and stat1006] and groups for iflorez.

Updating patch to include expiry_date May 31, 2020. Who should be expiry_contact? Nuria?

Tue, May 21, 9:38 PM · Operations, SRE-Access-Requests
Dzahn awarded T192830: Requesting access to production for SWAT deploy for Urbanecm a Like token.
Tue, May 21, 8:52 PM · User-zeljkofilipin, Release-Engineering-Team (Kanban), User-greg, User-Urbanecm, Operations, SRE-Access-Requests
Dzahn reassigned T223496: Requesting access to machines [stat1004, stat1005 (now stat1007), and stat1006] and groups for iflorez from Dzahn to Nuria.
Tue, May 21, 7:47 PM · Operations, SRE-Access-Requests
Dzahn added a comment to T223496: Requesting access to machines [stat1004, stat1005 (now stat1007), and stat1006] and groups for iflorez.

Thanks @Iflorez for attention to detail. So the full story is that's all one LDAP user but there are different fields:

Tue, May 21, 7:45 PM · Operations, SRE-Access-Requests

Mon, May 20

Dzahn closed T130532: Offer Korean Locales "ko_KR.euckr" and "ko_KR.utf8" on Tool Labs as Resolved.
Mon, May 20, 11:58 PM · Wikimedia-Hackathon-2019, Patch-For-Review, Toolforge
Dzahn assigned T192830: Requesting access to production for SWAT deploy for Urbanecm to greg.

@greg So this is approved by you?

Mon, May 20, 10:21 PM · User-zeljkofilipin, Release-Engineering-Team (Kanban), User-greg, User-Urbanecm, Operations, SRE-Access-Requests
Dzahn assigned T223262: Request: add awight to contint-docker to Tobi_WMDE_SW.
Mon, May 20, 10:19 PM · Release-Engineering-Team-TODO, Operations, SRE-Access-Requests, Continuous-Integration-Infrastructure
Dzahn added a comment to T130532: Offer Korean Locales "ko_KR.euckr" and "ko_KR.utf8" on Tool Labs.

Thanks @bd808 . Merged your change. Ran puppet, i saw it add the new locale. Looks like i just didn't use the "extended" version of the list?

Mon, May 20, 8:41 PM · Wikimedia-Hackathon-2019, Patch-For-Review, Toolforge
Dzahn added a comment to T179126: Clarify that spaces are not allowed in Phabricator usernames when creating an account.

Indeed, there is a one byte difference in the string now so strings don't match.

Mon, May 20, 8:06 PM · Patch-For-Review, Phabricator