Page MenuHomePhabricator

Dzahn (Daniel Zahn)
Operations EngineerAdministrator

Projects (17)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Sep 30 2014, 4:39 PM (270 w, 6 d)
Roles
Administrator
Availability
Available
IRC Nick
mutante
LDAP User
Dzahn
MediaWiki User
Unknown

Recent Activity

Sat, Dec 7

bd808 awarded T240047: Request creation of devtools VPS project a Like token.
Sat, Dec 7, 8:24 PM · Cloud-VPS (Project-requests)

Fri, Dec 6

Dzahn added a comment to T239139: rack/setup/install frdb1003.frack.eqiad.wmnet.

https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=frdb1003

Fri, Dec 6, 10:16 PM · Operations, fundraising-tech-ops, ops-eqiad
Dzahn closed T236190: mw1252 - Memory correctable errors -EDAC- as Resolved.

I don't know why but the alert in Icinga has cleared since 14 days.

Fri, Dec 6, 10:15 PM · serviceops, Operations, DC-Ops
Dzahn awarded T185319: IRC RecentChanges feed: code stewardship request a Barnstar token.
Fri, Dec 6, 10:04 PM · Tools, Operations, Analytics, Wikimedia-IRC-RC-Server, Code-Stewardship-Reviews
Dzahn added a comment to T202356: Create Security Patches board/workflow.

IMO, the largest gains for this proposal would be seen from also incorporating a private front-end (likely another gerrit) so as to provide consistency...

Fri, Dec 6, 9:50 PM · Security-Team
Dzahn added a comment to T238956: switch prod Phabricator from phab1003 to phab1001.

@Volans Thanks for reporting! The SPF record has been updated now in DNS. This should go away now.

Fri, Dec 6, 9:33 PM · Patch-For-Review, serviceops, Release-Engineering-Team
Dzahn added a project to T218308: Add gerrit.wikimedia.org to the Phabricator CSP: Traffic.
Fri, Dec 6, 8:41 PM · ContentSecurityPolicy, Traffic, Security-Team, Operations, Phabricator, Gerrit
Dzahn updated subscribers of T232883: Make PHD run on the backup phabricator server (phab2001, currently).

@20after4 per recent talks we had.. is it still our goal?

Fri, Dec 6, 8:37 PM · Release-Engineering-Team-TODO (201912), serviceops, Release-Engineering-Team (Development services), Phabricator, Documentation, User-MModell
Dzahn closed T236309: define requirements for new devtools cloud vps project to replace "git", "gerrit" and "phabricator" as Resolved.
Fri, Dec 6, 8:22 PM · Phabricator, Gerrit
Dzahn closed T236309: define requirements for new devtools cloud vps project to replace "git", "gerrit" and "phabricator", a subtask of T240047: Request creation of devtools VPS project, as Resolved.
Fri, Dec 6, 8:22 PM · Cloud-VPS (Project-requests)
Dzahn added a comment to T240047: Request creation of devtools VPS project.

also see support from releng at T236309#5693892

Fri, Dec 6, 8:21 PM · Cloud-VPS (Project-requests)
Dzahn added a comment to T236309: define requirements for new devtools cloud vps project to replace "git", "gerrit" and "phabricator".

For the new project, please file a ticket here: https://phabricator.wikimedia.org/project/profile/2875/

Fri, Dec 6, 8:20 PM · Phabricator, Gerrit
Dzahn added a parent task for T229703: Close "gerrit" Cloud VPS project: T240047: Request creation of devtools VPS project.
Fri, Dec 6, 8:19 PM · cloud-services-team (Kanban), Cloud-VPS (Project-requests), VPS-Projects
Dzahn added a subtask for T240047: Request creation of devtools VPS project: T229703: Close "gerrit" Cloud VPS project.
Fri, Dec 6, 8:19 PM · Cloud-VPS (Project-requests)
Dzahn added a parent task for T236571: "phabricator" Cloud VPS project jessie deprecation: T240047: Request creation of devtools VPS project.
Fri, Dec 6, 8:19 PM · Cloud-VPS (Debian Jessie Deprecation)
Dzahn added a subtask for T240047: Request creation of devtools VPS project: T236571: "phabricator" Cloud VPS project jessie deprecation.
Fri, Dec 6, 8:19 PM · Cloud-VPS (Project-requests)
Dzahn added a subtask for T240047: Request creation of devtools VPS project: T236569: "git" Cloud VPS project jessie deprecation.
Fri, Dec 6, 8:19 PM · Cloud-VPS (Project-requests)
Dzahn added a parent task for T236569: "git" Cloud VPS project jessie deprecation: T240047: Request creation of devtools VPS project.
Fri, Dec 6, 8:19 PM · Cloud-VPS (Debian Jessie Deprecation)
Dzahn added a subtask for T240047: Request creation of devtools VPS project: T236309: define requirements for new devtools cloud vps project to replace "git", "gerrit" and "phabricator".
Fri, Dec 6, 8:18 PM · Cloud-VPS (Project-requests)
Dzahn added a parent task for T236309: define requirements for new devtools cloud vps project to replace "git", "gerrit" and "phabricator": T240047: Request creation of devtools VPS project.
Fri, Dec 6, 8:18 PM · Phabricator, Gerrit
Dzahn added a comment to T236571: "phabricator" Cloud VPS project jessie deprecation.

We requested a new unified project to replace this and 2 other projects at once. gerrit, git and phabricator.

Fri, Dec 6, 8:17 PM · Cloud-VPS (Debian Jessie Deprecation)
Dzahn added a comment to T236569: "git" Cloud VPS project jessie deprecation.

We requested a new unified project to replace this and 2 other projects at once. gerrit, git and phabricator.

Fri, Dec 6, 8:17 PM · Cloud-VPS (Debian Jessie Deprecation)
Dzahn updated subscribers of T229703: Close "gerrit" Cloud VPS project.

We requested a new unified project to replace this and 2 other projects at once. gerrit, git and phabricator.

Fri, Dec 6, 8:15 PM · cloud-services-team (Kanban), Cloud-VPS (Project-requests), VPS-Projects
Dzahn created T240047: Request creation of devtools VPS project.
Fri, Dec 6, 8:13 PM · Cloud-VPS (Project-requests)
Dzahn added a comment to T236571: "phabricator" Cloud VPS project jessie deprecation.

These are actually on stretch. But they have been upgraded in place, that's why they pop-up on the jessie list.

Fri, Dec 6, 7:13 PM · Cloud-VPS (Debian Jessie Deprecation)
Dzahn awarded T240011: Do not direct /maniphest to security-team-needs-triage as default global query a Barnstar token.
Fri, Dec 6, 6:05 PM · Security-Team, Phabricator
Dzahn added a comment to T108199: Evaluate Phabricator for its ability to export and import data.

@Weshouman I am not sure if that's what this is about but i want to point out there are Phabricator dumps at https://dumps.wikimedia.org/other/misc/ nowadays and the last comment on this ticket was a couple years ago.

Fri, Dec 6, 4:44 AM · Phabricator

Thu, Dec 5

Dzahn updated the task description for T238956: switch prod Phabricator from phab1003 to phab1001.
Thu, Dec 5, 10:26 PM · Patch-For-Review, serviceops, Release-Engineering-Team
Dzahn removed a project from T238956: switch prod Phabricator from phab1003 to phab1001: Patch-For-Review.
Thu, Dec 5, 10:21 PM · Patch-For-Review, serviceops, Release-Engineering-Team
Dzahn closed T238956: switch prod Phabricator from phab1003 to phab1001 as Resolved.

We switched from phab1003 to phab1001.

Thu, Dec 5, 10:15 PM · Patch-For-Review, serviceops, Release-Engineering-Team
Dzahn updated the task description for T238956: switch prod Phabricator from phab1003 to phab1001.
Thu, Dec 5, 10:13 PM · Patch-For-Review, serviceops, Release-Engineering-Team
Dzahn added a comment to T238956: switch prod Phabricator from phab1003 to phab1001.

I suspect that this move inadvertently caused T239786: Viewing MediaWiki repository in diffusion results in an Unhandled Exception ("CommandException") as I only started seeing it today.

Thu, Dec 5, 10:12 PM · Patch-For-Review, serviceops, Release-Engineering-Team
Dzahn closed T239786: Viewing MediaWiki repository in diffusion results in an Unhandled Exception ("CommandException") as Resolved.
Thu, Dec 5, 10:09 PM · Diffusion, Phabricator
Dzahn added a comment to T239786: Viewing MediaWiki repository in diffusion results in an Unhandled Exception ("CommandException").

This has been fixed. Sorry for the inconvenience.

Thu, Dec 5, 10:09 PM · Diffusion, Phabricator
Dzahn placed T134271: Replace ircd-ratbox with something newer/maintained up for grabs.
Thu, Dec 5, 8:26 PM · Patch-For-Review, Wikimedia-IRC-RC-Server, Operations
Dzahn added a comment to T134271: Replace ircd-ratbox with something newer/maintained.

Following the "cookie licking on Phab" discussion i'm unassigning this from me because i am not going to work on it soon and T185319 says that Analytics took over stewardship for RC feeds.

Thu, Dec 5, 8:25 PM · Patch-For-Review, Wikimedia-IRC-RC-Server, Operations
Dzahn added a comment to T185319: IRC RecentChanges feed: code stewardship request.

Is this really replacing the IRCd from T134271 ?

Thu, Dec 5, 8:25 PM · Tools, Operations, Analytics, Wikimedia-IRC-RC-Server, Code-Stewardship-Reviews
Dzahn added a comment to T236833: wt2html: Out of memory crashers.

If you want to change the memory_limit for just Parsoid servers again that is now:

Thu, Dec 5, 8:05 PM · Wikimedia-production-error, serviceops, Operations, Parsoid-PHP
Dzahn placed T236833: wt2html: Out of memory crashers up for grabs.
Thu, Dec 5, 7:59 PM · Wikimedia-production-error, serviceops, Operations, Parsoid-PHP
Dzahn added a comment to T236833: wt2html: Out of memory crashers.

This was assigned to me to (make it possible to) raise the memory limit. That has happened now.

Thu, Dec 5, 7:59 PM · Wikimedia-production-error, serviceops, Operations, Parsoid-PHP
Dzahn added a comment to T228575: Decrease issues with assignee field set for years without progress (aka cookie licking).

Don't they have to be nagged to .. do the review.. though?

Yes, but the person to do the review is not the assignee (patch author).

Thu, Dec 5, 7:39 PM · Developer-Advocacy (Oct-Dec 2019), Phabricator
Dzahn updated the task description for T239054: Reimage all mediawiki servers .
Thu, Dec 5, 6:44 PM · Operations, serviceops
Dzahn added a comment to T239758: mw2259 down and mgmt does not exist?.

Thank you very much @Papaul.

Thu, Dec 5, 6:43 PM · DC-Ops, ops-codfw, serviceops, Operations
Dzahn placed T239151: Gerrit VM to test data migration up for grabs.

I'm unassigning from me for now because i'll be on a vacation and don't want this to be blocked on me for no reason. If anyone else can push this forward meanwhile one way or the other, please do. Otherwise i'll take it back later.

Thu, Dec 5, 6:31 PM · Gerrit, vm-requests, Operations
Dzahn added a comment to T239151: Gerrit VM to test data migration.

Just a thought.. If the 2 machines for contint are granted (T239880) first and are taken from available spare pool.. then possibly one of them could work as temp Gerrit test machine and then become contint?

Thu, Dec 5, 6:29 PM · Gerrit, vm-requests, Operations
Dzahn awarded T239880: Replacement hardware for buster/stretch upgrade of contint1001 and contint2001 a Like token.
Thu, Dec 5, 6:27 PM · Continuous-Integration-Infrastructure (phase-out-jessie), DC-Ops, hardware-requests, Operations
Dzahn updated subscribers of T239880: Replacement hardware for buster/stretch upgrade of contint1001 and contint2001.
Thu, Dec 5, 6:27 PM · Continuous-Integration-Infrastructure (phase-out-jessie), DC-Ops, hardware-requests, Operations
Dzahn added a comment to T239880: Replacement hardware for buster/stretch upgrade of contint1001 and contint2001.

I'd be the one to take these from "role::spare"-state to serving the puppet roles. We'll see what breaks on buster and whether we can go to buster or just stretch for now.

Thu, Dec 5, 6:26 PM · Continuous-Integration-Infrastructure (phase-out-jessie), DC-Ops, hardware-requests, Operations
Dzahn added a comment to T101017: Early security release access for Lcawte (ShoutWiki).

If you (security-team) want to go through WMF volunteer NDA process, please contact Rachel in Legal, she'll follow-up.

Thu, Dec 5, 6:19 PM · Security-Team, ShoutWiki, WMF-Legal, WMF-NDA-Requests
Dzahn changed the status of T236216: rack/setup/install ganeti300[123] from Stalled to Open.
Thu, Dec 5, 6:14 PM · Operations, ops-esams
Dzahn updated the task description for T238956: switch prod Phabricator from phab1003 to phab1001.
Thu, Dec 5, 1:48 AM · Patch-For-Review, serviceops, Release-Engineering-Team

Wed, Dec 4

Dzahn added a comment to T228575: Decrease issues with assignee field set for years without progress (aka cookie licking).

One could argue that it's even less of a problem to unassign people from effectively stalled tasks if these people don't even pay attention. As I tried to argue above: What would be the harm?

Wed, Dec 4, 11:00 PM · Developer-Advocacy (Oct-Dec 2019), Phabricator
Dzahn added a comment to T228575: Decrease issues with assignee field set for years without progress (aka cookie licking).

Don't they have to be nagged to .. do the review.. though?

Wed, Dec 4, 10:57 PM · Developer-Advocacy (Oct-Dec 2019), Phabricator
Dzahn added a comment to T239151: Gerrit VM to test data migration.

The old puppetdb hosts (puppetdb1001) should be ready to go away, @jbond merged the patches to stop broadcasting to it last week. It also has 16G RAM, so those would be freed up when that's done.

Wed, Dec 4, 10:55 PM · Gerrit, vm-requests, Operations
Dzahn added a comment to T239151: Gerrit VM to test data migration.

The way I understand the message: the virtualization servers in group row_A lack free memory to allocate a VM. But maybe another group would have memory available?
You should be able to list the group and free memory with:

sudo gnt-node list -o name,group,mfree

You can try another group, else bring down memory ? :)

Wed, Dec 4, 10:51 PM · Gerrit, vm-requests, Operations
Dzahn added a comment to T238956: switch prod Phabricator from phab1003 to phab1001.

Unfortunately we have to switch back to the server before, change a BIOS setting in the current server, reimage it and then switch back a third and hopefully final time.

Wed, Dec 4, 7:14 PM · Patch-For-Review, serviceops, Release-Engineering-Team
Dzahn updated the task description for T238956: switch prod Phabricator from phab1003 to phab1001.
Wed, Dec 4, 7:13 PM · Patch-For-Review, serviceops, Release-Engineering-Team
Dzahn added a comment to T239300: Add Maryum to Puppet.

i see your user exists now for example on bast1002:

Wed, Dec 4, 6:44 PM · Patch-For-Review, Operations, Discovery-Search (Current work)
Aklapper awarded T238956: switch prod Phabricator from phab1003 to phab1001 a Love token.
Wed, Dec 4, 12:20 PM · Patch-For-Review, serviceops, Release-Engineering-Team
Dzahn added a comment to T210411: Applayer services without TLS.

https://ticket.wikimedia.org (OTRS) has been switched to use https://ticket.discovery.wmnet (envoy on mendelevium).

Wed, Dec 4, 5:16 AM · Patch-For-Review, serviceops, Operations, Traffic
Dzahn updated the task description for T210411: Applayer services without TLS.
Wed, Dec 4, 5:15 AM · Patch-For-Review, serviceops, Operations, Traffic
Dzahn updated the task description for T238956: switch prod Phabricator from phab1003 to phab1001.
Wed, Dec 4, 5:04 AM · Patch-For-Review, serviceops, Release-Engineering-Team
Dzahn added a comment to T238956: switch prod Phabricator from phab1003 to phab1001.

@Dzahn: Can we switch it over to AHCI in the bios? Do we need DC-Ops for that?

Wed, Dec 4, 4:55 AM · Patch-For-Review, serviceops, Release-Engineering-Team

Tue, Dec 3

Dzahn claimed T238956: switch prod Phabricator from phab1003 to phab1001.

This is happening today, starting in about 15 minutes.

Tue, Dec 3, 11:45 PM · Patch-For-Review, serviceops, Release-Engineering-Team
Dzahn updated the task description for T238956: switch prod Phabricator from phab1003 to phab1001.
Tue, Dec 3, 11:44 PM · Patch-For-Review, serviceops, Release-Engineering-Team
Dzahn renamed T239758: mw2259 down and mgmt does not exist? from mw2259 and mgmt down to mw2259 down and mgmt does not exist?.
Tue, Dec 3, 10:17 PM · DC-Ops, ops-codfw, serviceops, Operations
Dzahn updated the task description for T239054: Reimage all mediawiki servers .
Tue, Dec 3, 10:02 PM · Operations, serviceops
Dzahn added a comment to T239758: mw2259 down and mgmt does not exist?.

Also: I noticed in Icinga there is no "mw2259.mgmt" (while for example mw2247.mgmt exists). It's simply not there:

Tue, Dec 3, 9:59 PM · DC-Ops, ops-codfw, serviceops, Operations
Dzahn updated the task description for T239758: mw2259 down and mgmt does not exist?.
Tue, Dec 3, 9:58 PM · DC-Ops, ops-codfw, serviceops, Operations
Dzahn triaged T239758: mw2259 down and mgmt does not exist? as Medium priority.
Tue, Dec 3, 9:36 PM · DC-Ops, ops-codfw, serviceops, Operations
Dzahn updated subscribers of T239758: mw2259 down and mgmt does not exist?.

@Papaul Could you take a look at this onsite?

Tue, Dec 3, 9:36 PM · DC-Ops, ops-codfw, serviceops, Operations
Dzahn added projects to T239758: mw2259 down and mgmt does not exist?: Operations, serviceops, ops-codfw, DC-Ops.
Tue, Dec 3, 9:35 PM · DC-Ops, ops-codfw, serviceops, Operations
Dzahn created T239758: mw2259 down and mgmt does not exist?.
Tue, Dec 3, 9:34 PM · DC-Ops, ops-codfw, serviceops, Operations
Dzahn added a comment to T237707: doc1001 permission problems.

Resolved?

Tue, Dec 3, 8:16 PM · Release-Engineering-Team (CI & Testing services), Continuous-Integration-Infrastructure
Dzahn added a comment to T237707: doc1001 permission problems.

Both patches are merged now. Git status of /srv/docroot is clean.

Tue, Dec 3, 8:16 PM · Release-Engineering-Team (CI & Testing services), Continuous-Integration-Infrastructure
Dzahn added a comment to T239300: Add Maryum to Puppet.

I didn't add myself to the users because I wasn't sure what my uid was

Tue, Dec 3, 8:06 PM · Patch-For-Review, Operations, Discovery-Search (Current work)
Dzahn added a subtask for T239300: Add Maryum to Puppet: T239654: Requesting access to production shell for Maryum Styles.
Tue, Dec 3, 8:05 PM · Patch-For-Review, Operations, Discovery-Search (Current work)
Dzahn added a parent task for T239654: Requesting access to production shell for Maryum Styles: T239300: Add Maryum to Puppet.
Tue, Dec 3, 8:05 PM · Discovery-Search (Current work), Operations, SRE-Access-Requests
Dzahn updated subscribers of T239654: Requesting access to production shell for Maryum Styles.

+@Nuria per T239300#5710194

Tue, Dec 3, 8:04 PM · Discovery-Search (Current work), Operations, SRE-Access-Requests
Dzahn added a comment to T239300: Add Maryum to Puppet.

This ticket seems to be a duplicate of T239654. Adding her there as well. Linking Gerrit change i amended to.

Tue, Dec 3, 8:04 PM · Patch-For-Review, Operations, Discovery-Search (Current work)
Dzahn added a comment to T236180: Deploy search platform airflow service.

fwiw, somebody on mysql support channel told me about "explicit_defaults_for_timestamp" that "scope is system and session.You can set it per session".

Tue, Dec 3, 7:08 PM · Patch-For-Review, Discovery-Search
ema awarded T239629: https://releases-jenkins.wikimedia.org yields a 502 unreachable a Love token.
Tue, Dec 3, 8:18 AM · Traffic, Release-Engineering-Team-TODO, Continuous-Integration-Infrastructure, Operations
Dzahn updated the task description for T239054: Reimage all mediawiki servers .
Tue, Dec 3, 2:39 AM · Operations, serviceops
Dzahn added a comment to T236180: Deploy search platform airflow service.

explicit_defaults_for_timestamp = 1 will become the standard in future mysql versions.

Tue, Dec 3, 2:35 AM · Patch-For-Review, Discovery-Search
Dzahn removed a project from T239629: https://releases-jenkins.wikimedia.org yields a 502 unreachable: Patch-For-Review.
Tue, Dec 3, 2:16 AM · Traffic, Release-Engineering-Team-TODO, Continuous-Integration-Infrastructure, Operations
Dzahn closed T239629: https://releases-jenkins.wikimedia.org yields a 502 unreachable as Resolved.
Tue, Dec 3, 2:16 AM · Traffic, Release-Engineering-Team-TODO, Continuous-Integration-Infrastructure, Operations
Dzahn added a comment to T239629: https://releases-jenkins.wikimedia.org yields a 502 unreachable.

@hashar It's fixed for me now. It was missing the releases-jenkins.wikimedia.org as a SAN on the TLS certificate.

Tue, Dec 3, 2:15 AM · Traffic, Release-Engineering-Team-TODO, Continuous-Integration-Infrastructure, Operations
Dzahn claimed T239629: https://releases-jenkins.wikimedia.org yields a 502 unreachable.
Tue, Dec 3, 1:58 AM · Traffic, Release-Engineering-Team-TODO, Continuous-Integration-Infrastructure, Operations
Dzahn added a comment to T239151: Gerrit VM to test data migration.

Tried to create it but unfortunately:

Tue, Dec 3, 1:47 AM · Gerrit, vm-requests, Operations

Mon, Dec 2

Dzahn added a comment to T210411: Applayer services without TLS.

https://noc.wikimedia.org has been switched to use https://mwmaint.discovery.wmnet (envoy on mwmaint1002).

Mon, Dec 2, 10:57 PM · Patch-For-Review, serviceops, Operations, Traffic
Dzahn updated the task description for T210411: Applayer services without TLS.
Mon, Dec 2, 10:57 PM · Patch-For-Review, serviceops, Operations, Traffic
Dzahn updated the task description for T239054: Reimage all mediawiki servers .
Mon, Dec 2, 9:39 PM · Operations, serviceops
Dzahn added a comment to T239093: Add Daimona to #mediawiki_security.

Approved in SRE meeting.

Mon, Dec 2, 5:41 PM · Operations

Fri, Nov 29

Dzahn awarded T209019: Accessing qrwp.org without a language-code prefix should redirect to another site (not fail) a Love token.
Fri, Nov 29, 3:25 AM · QRpedia-General

Thu, Nov 28

Dzahn updated the task description for T239054: Reimage all mediawiki servers .
Thu, Nov 28, 2:33 AM · Operations, serviceops
Dzahn added a comment to T239151: Gerrit VM to test data migration.

Lowered the memory request as that seems to be out-of-line with the usage of most Ganeti VMs, hopefully 16G would work for a Ganeti VM?

Thu, Nov 28, 2:13 AM · Gerrit, vm-requests, Operations
Dzahn awarded T212824: notebook/stat server(s) running out of memory a Barnstar token.
Thu, Nov 28, 2:02 AM · Analytics-Kanban, Product-Analytics, User-Elukey, Operations, Analytics
Dzahn updated the task description for T239054: Reimage all mediawiki servers .
Thu, Nov 28, 1:18 AM · Operations, serviceops

Wed, Nov 27

Dzahn updated the task description for T239054: Reimage all mediawiki servers .
Wed, Nov 27, 11:13 PM · Operations, serviceops
Dzahn committed rLPRI8f26eaf5114b: add fake SSL key for ticket.discovery.wmnet (authored by Dzahn).
add fake SSL key for ticket.discovery.wmnet
Wed, Nov 27, 10:33 PM