Dzahn (Daniel Zahn)Administrator
Operations Engineer

Projects (16)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Sep 30 2014, 4:39 PM (211 w, 6 d)
Roles
Administrator
Availability
Available
IRC Nick
mutante
LDAP User
Dzahn
MediaWiki User
Unknown

Recent Activity

Today

Dzahn added a comment to T207175: add icinga1001 to send_nsca and pfw rules in FRACK.

@cwdent has the send_nsca part also been done?

Mon, Oct 22, 7:00 PM · netops, Operations, fundraising-tech-ops
Dzahn added a comment to T192129: Redirect living-style-guide.wmflabs.org to design.wikimedia.org/style-guide.

Yep, was resolved in T200304 , kind of a duplicate.

Mon, Oct 22, 6:58 PM · cloud-services-team (Kanban), VPS-Projects

Yesterday

hashar awarded T206579: Releases Jenkins Icinga check failing after restricting access a Y So Serious token.
Sun, Oct 21, 8:56 PM · Patch-For-Review, monitoring, Operations, Continuous-Integration-Infrastructure, Release-Engineering-Team

Fri, Oct 19

Dzahn added a comment to T195926: Create wiki for Wikimedia South Africa.

It's the right thing to add operations because we need to do DNS and Apache. But after that is done you can remove it. ACK.

Fri, Oct 19, 11:53 PM · User-Urbanecm, Wiki-Setup (Create)
Dzahn added a comment to T206777: Create Wikipedia Shan.

shn has been created in DNS:

Fri, Oct 19, 7:21 PM · Patch-For-Review, User-Urbanecm, Wiki-Setup (Create)
Dzahn moved T204477: Create punjabi.wikimedia.org for Punjabi Wikimedians User Group from Create to Stalled / Needs Input on the Wiki-Setup board.
Fri, Oct 19, 6:02 PM · Patch-For-Review, Wiki-Setup (Create), User-Urbanecm
Dzahn assigned T204477: Create punjabi.wikimedia.org for Punjabi Wikimedians User Group to satdeep_gill.
Fri, Oct 19, 6:02 PM · Patch-For-Review, Wiki-Setup (Create), User-Urbanecm
Dzahn added a comment to T204477: Create punjabi.wikimedia.org for Punjabi Wikimedians User Group.

Still needs user input there. Moving to new column "Stalled / Needs Input" on the Wiki-Setup workboard i created for this.

Fri, Oct 19, 6:02 PM · Patch-For-Review, Wiki-Setup (Create), User-Urbanecm
Dzahn changed the status of T204477: Create punjabi.wikimedia.org for Punjabi Wikimedians User Group from Stalled to Open.

While we're on Datacenter-Switchover-2018 we ain't allowed to create new wikis.

Fri, Oct 19, 6:00 PM · Patch-For-Review, Wiki-Setup (Create), User-Urbanecm
Dzahn moved T205546: Create Wiktionary Cantonese from Create to Stalled / Needs Input on the Wiki-Setup board.
Fri, Oct 19, 5:59 PM · Wiki-Setup (Create), Patch-For-Review, User-Urbanecm
Dzahn moved T167375: Creation of a "Campaign" Wiki from Create to Stalled / Needs Input on the Wiki-Setup board.
Fri, Oct 19, 5:57 PM · Wiki-Setup, User-Urbanecm, Fundraising-Backlog
Dzahn added a comment to T206777: Create Wikipedia Shan.

^ there might be an issue with this, it seems "gdnsd reload-zones" isn't the correct syntax anymore.. hold on for a bit..

Fri, Oct 19, 5:55 PM · Patch-For-Review, User-Urbanecm, Wiki-Setup (Create)
Niharika awarded T200742: partial German translation shows up in English text on wikimediafoundation.org a Burninate token.
Fri, Oct 19, 5:55 PM · wikimediafoundation.org
Dzahn moved T184961: Close chairwiki from Close to Stalled / Needs Input on the Wiki-Setup board.
Fri, Oct 19, 5:42 PM · Wiki-Setup, Patch-For-Review, Release-Engineering-Team (Kanban)
Dzahn added a comment to T184961: Close chairwiki.

@Reedy @Jalexander Could you give access to @Raystorm ?

Fri, Oct 19, 5:41 PM · Wiki-Setup, Patch-For-Review, Release-Engineering-Team (Kanban)
Dzahn renamed T201188: Close wikimania2018wiki (July 2019) from Close wikimania2018wiki to Close wikimania2018wiki (July 2019).
Fri, Oct 19, 5:39 PM · Wiki-Setup, User-MarcoAurelio
Dzahn moved T201188: Close wikimania2018wiki (July 2019) from Close to Stalled / Needs Input on the Wiki-Setup board.
Fri, Oct 19, 5:38 PM · Wiki-Setup, User-MarcoAurelio
Dzahn awarded T205584: Close internal.wikimedia.org a Like token.
Fri, Oct 19, 5:37 PM · Patch-For-Review, User-Urbanecm, Wiki-Setup (Close)
Dzahn moved T207052: Create fishbowl wiki for Vietnam Wikimedians User Group from Untriaged to Stalled / Needs Input on the Wiki-Setup board.
Fri, Oct 19, 5:33 PM · Wiki-Setup, Patch-For-Review, User-Urbanecm
Dzahn moved T207052: Create fishbowl wiki for Vietnam Wikimedians User Group from Create to Untriaged on the Wiki-Setup board.
Fri, Oct 19, 5:33 PM · Wiki-Setup, Patch-For-Review, User-Urbanecm
Dzahn moved T195926: Create wiki for Wikimedia South Africa from Untriaged to Create on the Wiki-Setup board.
Fri, Oct 19, 5:28 PM · User-Urbanecm, Wiki-Setup (Create)
Dzahn edited projects for T195926: Create wiki for Wikimedia South Africa, added: Wiki-Setup; removed Patch-For-Review.
Fri, Oct 19, 5:28 PM · User-Urbanecm, Wiki-Setup (Create)
Dzahn triaged T195926: Create wiki for Wikimedia South Africa as Normal priority.
Fri, Oct 19, 5:27 PM · User-Urbanecm, Wiki-Setup (Create)
Dzahn added a comment to T195926: Create wiki for Wikimedia South Africa.

https://za.wikimedia.org
https://za.m.wikimedia.org

Fri, Oct 19, 5:25 PM · User-Urbanecm, Wiki-Setup (Create)
Dzahn added a comment to T207052: Create fishbowl wiki for Vietnam Wikimedians User Group.

Ok, i will remove myself from this ticket for now but that doesn't mean i have an opinion on this one way or another. Once there is some consensus and you need technical help again, please simply re-add me. Cheers

Fri, Oct 19, 4:22 PM · Wiki-Setup, Patch-For-Review, User-Urbanecm
Dzahn updated subscribers of T207487: Puppet broken on deployment-deploy*.

looks like @elukey made this as the fix:

Fri, Oct 19, 4:18 PM · Patch-For-Review, Puppet, Beta-Cluster-Infrastructure
Dzahn closed T206579: Releases Jenkins Icinga check failing after restricting access as Resolved.

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=releases1001&service=HTTP+releases-jenkins.wikimedia.org
https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=releases2001&service=HTTP+releases-jenkins.wikimedia.org

Fri, Oct 19, 4:12 PM · Patch-For-Review, monitoring, Operations, Continuous-Integration-Infrastructure, Release-Engineering-Team
Dzahn added a comment to T206579: Releases Jenkins Icinga check failing after restricting access.

Needs to be /login but not /login/

Fri, Oct 19, 3:55 PM · Patch-For-Review, monitoring, Operations, Continuous-Integration-Infrastructure, Release-Engineering-Team
Dzahn added a comment to T206579: Releases Jenkins Icinga check failing after restricting access.

https://releases-jenkins.wikimedia.org/login/ -> HTTP ERROR 404

Fri, Oct 19, 3:54 PM · Patch-For-Review, monitoring, Operations, Continuous-Integration-Infrastructure, Release-Engineering-Team
Dzahn added a comment to T206579: Releases Jenkins Icinga check failing after restricting access.

adding /login/ to the URL changes it from "403 Forbidden" to "404 Not Found"

Fri, Oct 19, 3:54 PM · Patch-For-Review, monitoring, Operations, Continuous-Integration-Infrastructure, Release-Engineering-Team
Dzahn added a comment to T207052: Create fishbowl wiki for Vietnam Wikimedians User Group.

Of course i am not involved in this issue so i can't comment much but let me say as an outsider i am a bit confused because you are saying "I don't really object the creation" followed by "i stand by the decision of stalling". I haven't heard of user groups with a board before, only chapters and this is what Urbanecm said above as well. Though you seem to be saying that existence of a board is necessary to create the group.

Fri, Oct 19, 2:52 PM · Wiki-Setup, Patch-For-Review, User-Urbanecm

Thu, Oct 18

Dzahn renamed T207428: cr2-esams - BGP WARNING - AS15426/IPv4 from BGP WARNING - AS15426/IPv4 to cr2-esams - BGP WARNING - AS15426/IPv4.
Thu, Oct 18, 10:37 PM · netops, Operations
Dzahn created T207428: cr2-esams - BGP WARNING - AS15426/IPv4.
Thu, Oct 18, 10:37 PM · netops, Operations
Dzahn added projects to T206579: Releases Jenkins Icinga check failing after restricting access: Operations, monitoring.
Thu, Oct 18, 10:32 PM · Patch-For-Review, monitoring, Operations, Continuous-Integration-Infrastructure, Release-Engineering-Team
Dzahn added a comment to T206579: Releases Jenkins Icinga check failing after restricting access.

This is a warning level alert on the hosts releases1001 and releases2001.

Thu, Oct 18, 10:31 PM · Patch-For-Review, monitoring, Operations, Continuous-Integration-Infrastructure, Release-Engineering-Team
Jonesey95 awarded T200742: partial German translation shows up in English text on wikimediafoundation.org a Dislike token.
Thu, Oct 18, 5:16 PM · wikimediafoundation.org
Dzahn added a comment to T162070: Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases.

icinga will stop using the mysql module once on stretch. Once T202782 is resolved and einsteinium isn't the prod Icinga server anymore this part will be done.

Thu, Oct 18, 3:12 PM · Patch-For-Review, Operations, DBA
Dzahn triaged T207330: Requesting access to production servers (mwlog*, mmaint* ?) for kharlan as Normal priority.
Thu, Oct 18, 2:48 PM · Patch-For-Review, Operations, SRE-Access-Requests
Dzahn added a comment to T207052: Create fishbowl wiki for Vietnam Wikimedians User Group.

Ok, no problem. It can stay in "stalled" status, that's ok.

Thu, Oct 18, 2:23 PM · Wiki-Setup, Patch-For-Review, User-Urbanecm
Dzahn added a comment to T207387: Puppet failures on trusty due to libmonitoring-plugin-perl.

Implemented as suggested on https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/466951/ and https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/467011/

Thu, Oct 18, 1:55 PM · cloud-services-team
Dzahn updated subscribers of T207387: Puppet failures on trusty due to libmonitoring-plugin-perl.
Thu, Oct 18, 1:43 PM · cloud-services-team
Dzahn added a comment to T207330: Requesting access to production servers (mwlog*, mmaint* ?) for kharlan.

Ok. The need for "sudo as www-data" wasn't obvious to me from the request or the linked wikitech page. I see it when looking at actual mwrepl source though.

Thu, Oct 18, 12:31 PM · Patch-For-Review, Operations, SRE-Access-Requests
Dzahn added a comment to T207052: Create fishbowl wiki for Vietnam Wikimedians User Group.

Objections were raised.

Thu, Oct 18, 12:24 PM · Wiki-Setup, Patch-For-Review, User-Urbanecm
Krenair awarded T200742: partial German translation shows up in English text on wikimediafoundation.org a Burninate token.
Thu, Oct 18, 12:17 PM · wikimediafoundation.org
Dzahn closed T207239: Make aklapper a co-admin of the listadmins@ mailing list as Resolved.

Done. I logged in at the admin interface using the master password from pwstore and added aklapper@ as an additional admin.

Thu, Oct 18, 12:17 PM · Operations, Wikimedia-Mailing-lists
Dzahn added a comment to T207330: Requesting access to production servers (mwlog*, mmaint* ?) for kharlan.

I saw we also have "maintenance-log-readers". It allows for access to mwmaint* hosts and reading logs (includes running journcalctl, dmesg, anything as syslog user).

Thu, Oct 18, 12:11 PM · Patch-For-Review, Operations, SRE-Access-Requests
Dzahn added a comment to T200742: partial German translation shows up in English text on wikimediafoundation.org.

That's not a bug - it is a design feature. Once the translations are completed - you will see them integrated across the design. :)

Thu, Oct 18, 12:05 PM · wikimediafoundation.org
Dzahn renamed T207330: Requesting access to production servers (mwlog*, mmaint* ?) for kharlan from Requesting access to production servers for kharlan to Requesting access to production servers (mwlog*, mmaint* ?) for kharlan.
Thu, Oct 18, 11:56 AM · Patch-For-Review, Operations, SRE-Access-Requests
Dzahn added a comment to T207330: Requesting access to production servers (mwlog*, mmaint* ?) for kharlan.

Since the request is specifically for viewing logs i recommend using the group "mw-log-readers" which was made specifically for this purpose.

Thu, Oct 18, 11:54 AM · Patch-For-Review, Operations, SRE-Access-Requests
Aklapper awarded T200742: partial German translation shows up in English text on wikimediafoundation.org a Heartbreak token.
Thu, Oct 18, 4:13 AM · wikimediafoundation.org

Wed, Oct 17

Dzahn awarded T196968: Re-organize the apache configuration for MediaWiki in puppet a Barnstar token.
Wed, Oct 17, 6:03 PM · User-Joe, Patch-For-Review, Wikimedia-Apache-configuration, Operations
Dzahn added a comment to T207051: Create Legalpad for WLM permission form.

I assume you can rename it without needing an admin? If not let me know.

Wed, Oct 17, 6:00 PM · Wiki-Loves-Monuments (2018), Legalpad

Tue, Oct 16

Dzahn triaged T207243: Images are not shown correctly in Planet as Normal priority.
Tue, Oct 16, 11:45 PM · Patch-For-Review, Wikimedia-Planet
Dzahn added a comment to T207051: Create Legalpad for WLM permission form.

Cool, sounds good:) I guess we can resolve it here then.

Tue, Oct 16, 8:04 PM · Wiki-Loves-Monuments (2018), Legalpad
Dzahn added a comment to T207051: Create Legalpad for WLM permission form.

Edited the custom policy of the new pad, L36 to allow "members of legal" and @JeanFred

Tue, Oct 16, 7:08 PM · Wiki-Loves-Monuments (2018), Legalpad
Dzahn added a comment to T207051: Create Legalpad for WLM permission form.

I think we should first let Stephen try if it works to edit the pad above now. Let's find out why the custom policy works or doesn't work, it says Legal Team should be able to edit. Let's not just fix it by using admin rights?

Tue, Oct 16, 7:03 PM · Wiki-Loves-Monuments (2018), Legalpad
Dzahn added a comment to T207051: Create Legalpad for WLM permission form.

I created L36 - https://phabricator.wikimedia.org/legalpad/view/36/

Tue, Oct 16, 7:01 PM · Wiki-Loves-Monuments (2018), Legalpad
Dzahn created P7684 (An Untitled Masterwork).
Tue, Oct 16, 6:26 PM
Dzahn updated the task description for T207009: Onboarding Cas Rusnov.
Tue, Oct 16, 5:00 PM · Patch-For-Review, Operations
Dzahn added a subtask for T202782: upgrade icinga server to stretch and replace einsteinium: T207175: add icinga1001 to send_nsca and pfw rules in FRACK.
Tue, Oct 16, 2:48 PM · Patch-For-Review, monitoring, Operations
Dzahn added a parent task for T207175: add icinga1001 to send_nsca and pfw rules in FRACK: T202782: upgrade icinga server to stretch and replace einsteinium.
Tue, Oct 16, 2:48 PM · netops, Operations, fundraising-tech-ops
Dzahn created T207175: add icinga1001 to send_nsca and pfw rules in FRACK.
Tue, Oct 16, 2:48 PM · netops, Operations, fundraising-tech-ops

Mon, Oct 15

Dzahn updated the task description for T207009: Onboarding Cas Rusnov.
Mon, Oct 15, 7:13 PM · Patch-For-Review, Operations
Dzahn updated the task description for T207009: Onboarding Cas Rusnov.
Mon, Oct 15, 7:07 PM · Patch-For-Review, Operations
Dzahn added a comment to T207009: Onboarding Cas Rusnov.

added to wmf and ops LDAP groups.

Mon, Oct 15, 6:57 PM · Patch-For-Review, Operations
Dzahn updated the task description for T207009: Onboarding Cas Rusnov.
Mon, Oct 15, 6:57 PM · Patch-For-Review, Operations
Dzahn added a comment to T207009: Onboarding Cas Rusnov.

subscribed to ops and ops-private

Mon, Oct 15, 6:47 PM · Patch-For-Review, Operations
Dzahn updated the task description for T207009: Onboarding Cas Rusnov.
Mon, Oct 15, 6:47 PM · Patch-For-Review, Operations
Dzahn added a comment to T207069: Mobile DNS entry for vewikimedia is missing.
host ve.m.wikimedia.org
ve.m.wikimedia.org has address 208.80.154.224
ve.m.wikimedia.org has IPv6 address 2620:0:861:ed1a::1
Mon, Oct 15, 6:06 PM · Patch-For-Review, DNS, Traffic, User-Urbanecm, Operations
Dzahn added a comment to T207069: Mobile DNS entry for vewikimedia is missing.

is MobileFrontend enabled?

Mon, Oct 15, 6:00 PM · Patch-For-Review, DNS, Traffic, User-Urbanecm, Operations
Dzahn claimed T207069: Mobile DNS entry for vewikimedia is missing.
Mon, Oct 15, 6:00 PM · Patch-For-Review, DNS, Traffic, User-Urbanecm, Operations
Dzahn closed T206217: Allow Analytics team members to restart Turnilo and Superset as Resolved.

This should resolve the ticket. Please reopen if something doesn't work.

Mon, Oct 15, 5:54 PM · Patch-For-Review, SRE-Access-Requests, Analytics, Operations
Dzahn added a comment to T206217: Allow Analytics team members to restart Turnilo and Superset.

approved in SRE meeting

Mon, Oct 15, 5:03 PM · Patch-For-Review, SRE-Access-Requests, Analytics, Operations

Sat, Oct 13

Dzahn added a comment to T206131: add monitoring to alert on hosts without RAID.

Yes, i was thinking the same. I did not expect to add this as an Icinga check. I expected to add a script that is run by cron and sends out email to tell us.

Sat, Oct 13, 12:48 PM · monitoring, Operations
Dzahn reopened T201133: db1069 (x1 master) memory errors as "Open".

Service
Memory correctable errors -EDAC-
On Host
db1069
(db1069)

Sat, Oct 13, 3:13 AM · ops-eqiad, Operations, DBA
Dzahn reopened T201133: db1069 (x1 master) memory errors, a subtask of T189107: DB meta task for next DC failover issues, as Open.
Sat, Oct 13, 3:13 AM · Patch-For-Review, Epic, Operations, DBA
Dzahn reopened T164955: Degraded RAID on heze as "Open".

Current Status: CRITICAL (for 0d 8h 6m 11s)
Status Information: cluster=misc device=megaraid,8 instance=heze:9100 job=node site=codfw

Sat, Oct 13, 3:11 AM · Operations, ops-codfw
Dzahn added a comment to T202782: upgrade icinga server to stretch and replace einsteinium.

confirmed on icinga1001 that the fix for check_ssl is actually:

Sat, Oct 13, 2:37 AM · Patch-For-Review, monitoring, Operations
Dzahn added a comment to T202782: upgrade icinga server to stretch and replace einsteinium.

new approach to make check_ssl work:

Sat, Oct 13, 2:26 AM · Patch-For-Review, monitoring, Operations
Dzahn added a comment to T202782: upgrade icinga server to stretch and replace einsteinium.

@Cwhite regarding the Juniper alerts that have "Can't locate Nagios/Plugin/Getopt.pm", Google sent me to our own phab where Fundraising tech fixed it in T195522

Sat, Oct 13, 2:23 AM · Patch-For-Review, monitoring, Operations
Dzahn triaged T203645: rspec-puppet fails with Could not find the daemon directory (tested [/etc/sv,/var/lib/service]) as Normal priority.
Sat, Oct 13, 1:43 AM · Patch-For-Review, Continuous-Integration-Config, Operations
Dzahn triaged T206131: add monitoring to alert on hosts without RAID as Normal priority.
Sat, Oct 13, 1:42 AM · monitoring, Operations

Fri, Oct 12

Dzahn added a comment to T179568: all mailing lists should have descriptions.

@Aklapper thanks. i didn't know the "for affiliate campaign" part of it. That makes it work related.

Fri, Oct 12, 8:32 PM · Operations, Wikimedia-Mailing-lists
Dzahn added a comment to T187994: netfilter software at WMF: iptables vs nftables.

bpfilter made it into 4.18 kernel and there are claims that it would "eventually replace both iptables and nftables"

Fri, Oct 12, 7:02 PM · Operations
Dzahn triaged T179568: all mailing lists should have descriptions as Low priority.
Fri, Oct 12, 6:34 PM · Operations, Wikimedia-Mailing-lists
Dzahn added a comment to T179568: all mailing lists should have descriptions.
Fri, Oct 12, 6:34 PM · Operations, Wikimedia-Mailing-lists
Dzahn added a comment to T179568: all mailing lists should have descriptions.

Automatic mail to primary list admins of all lists without description sent with:

Fri, Oct 12, 6:32 PM · Operations, Wikimedia-Mailing-lists
Restricted Application added a project to T179568: all mailing lists should have descriptions: Operations.

affected lists as of today:

Fri, Oct 12, 5:58 PM · Operations, Wikimedia-Mailing-lists
Dzahn closed T141566: Change digest function of wikimedia-l@ so it send emails only once a day as Declined.
Fri, Oct 12, 5:48 PM · Operations, Wikimedia-Mailing-lists
Dzahn triaged T203964: Create a spicerack cookbook to empty a ganeti node from VMs as Normal priority.
Fri, Oct 12, 5:46 PM · Operations-Software-Development, User-jijiki, User-Joe, Operations
Dzahn updated the task description for T206870: decommission betelgeuse.
Fri, Oct 12, 5:43 PM · decommission, fundraising-tech-ops, Operations, ops-codfw
Dzahn added a project to T206870: decommission betelgeuse: decommission.
Fri, Oct 12, 5:43 PM · decommission, fundraising-tech-ops, Operations, ops-codfw
Dzahn added a comment to T206870: decommission betelgeuse.

https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Active_-%3E_Decommissioned

Fri, Oct 12, 5:42 PM · decommission, fundraising-tech-ops, Operations, ops-codfw
Dzahn triaged T206870: decommission betelgeuse as Normal priority.
Fri, Oct 12, 5:41 PM · decommission, fundraising-tech-ops, Operations, ops-codfw
Dzahn closed T206688: SOA serial numbers returned by authoritative nameservers differ as Resolved.

Thanks again. As always it's greatly appreciated that you add all the technical details and background.

Fri, Oct 12, 5:41 PM · Domains, Traffic, Operations
Dzahn added a comment to T206688: SOA serial numbers returned by authoritative nameservers differ .

I have replied by email and referred to this ticket. I also pointed out we don't agree that this check is even useful (but that it was fixed nevertheless).

Fri, Oct 12, 5:40 PM · Domains, Traffic, Operations
Dzahn triaged T206166: puppet compiler set to eqiad as primary dc while prod is codfw as Normal priority.
Fri, Oct 12, 5:27 PM · Operations
Dzahn claimed T206166: puppet compiler set to eqiad as primary dc while prod is codfw.
Fri, Oct 12, 5:27 PM · Operations
Dzahn triaged T206336: SRE quarterly goal: Ability to serve a fraction of the production traffic from PHP7 as High priority.
Fri, Oct 12, 5:26 PM · Operations
Dzahn triaged T203943: Convert automation scripts to spicerack cookbooks as Normal priority.
Fri, Oct 12, 5:26 PM · Operations-Software-Development, User-jijiki, User-Joe, Operations