Page MenuHomePhabricator

Dzahn (Daniel Zahn)
Operations EngineerAdministrator

Projects (24)

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Sep 30 2014, 4:39 PM (315 w, 6 d)
Roles
Administrator
Availability
Available
IRC Nick
mutante
LDAP User
Dzahn
MediaWiki User
Unknown

Recent Activity

Today

Dzahn added a comment to T265490: rate limited etherpad.

@hashar Please let me know if you still see the issue or not. I am hopeful it might be fixed and can't reproduce it right now but also it seems we only had spikes during meetings when many people at once used it, like today's quarterly review meeting.

Mon, Oct 19, 11:31 PM · Patch-For-Review, Operations, Wikimedia-Etherpad
Dzahn added a comment to T265490: rate limited etherpad.

^ Uhm.. but this went up. Trying to manually reproduce I could not... also before the change. Need more users at once.

Mon, Oct 19, 11:05 PM · Patch-For-Review, Operations, Wikimedia-Etherpad
Dzahn added a comment to T265922: Access to peek2001.codfw.wmnet.

Yes, SRE has root access to it. There is no other admin role applied though.

Mon, Oct 19, 9:54 PM · Patch-For-Review, Security-Team, SRE-Access-Requests, Operations
Dzahn claimed T265922: Access to peek2001.codfw.wmnet.
Mon, Oct 19, 9:53 PM · Patch-For-Review, Security-Team, SRE-Access-Requests, Operations
Dzahn updated subscribers of T265936: improve mw maintenance server switch over and discovery names.
Mon, Oct 19, 6:40 PM · serviceops, Operations
Dzahn created T265936: improve mw maintenance server switch over and discovery names.
Mon, Oct 19, 6:40 PM · serviceops, Operations

Sat, Oct 17

Dzahn added a comment to T265653: (Need By: TBD) setup/install deploy1002.

@RobH Please do not assign service puppet roles on new hosts. That will almost never work. Just add new hosts in site with the "insetup" role and hand over. Service implementation will be to apply the role along with other changes in Hiera etc.

Sat, Oct 17, 3:25 PM · ops-eqiad, DC-Ops, Operations

Fri, Oct 16

Dzahn added a comment to T159750: E-mail for people in different OIT LDAP object unit .

Tags with names of teams that don't use Phab usually create the wrong expectation that the team would look into tasks in Phab.

Fri, Oct 16, 5:19 PM · WMF-Office-IT, Operations, Mail
Dzahn changed the status of T161004: Remove disabled users from internal mailing lists, a subtask of T142815: Enhance account handling (meta bug), from Open to Stalled.
Fri, Oct 16, 12:58 AM · Operations
Dzahn changed the status of T161004: Remove disabled users from internal mailing lists, a subtask of T160158: Make disabled accounts visible in the corp mirror LDAP replica, from Open to Stalled.
Fri, Oct 16, 12:58 AM · WMF-Office-IT, LDAP, Operations
Dzahn changed the status of T161004: Remove disabled users from internal mailing lists from Open to Stalled.
Fri, Oct 16, 12:58 AM · WMF-Office-IT, LDAP, Operations
Dzahn closed T100860: Edit optoutresearch@ mailing list recipients as Resolved.
Fri, Oct 16, 12:57 AM · Design-Research, Operations
Dzahn added a comment to T100860: Edit optoutresearch@ mailing list recipients.

Since we never got a reply here I recently asked ITS (formerly OIT) to move optoutresearch@ over to them. Harry confirmed it was created on their side and then I removed it on the SRE side. So if there are edits needed please contact them now.

Fri, Oct 16, 12:56 AM · Design-Research, Operations
Dzahn assigned T241944: Register wikipersonas.org and redirect URL to Dendelele.
Fri, Oct 16, 12:53 AM · Design-Research, Operations, Domains, Traffic
Dzahn added a comment to T159750: E-mail for people in different OIT LDAP object unit .

@Aklapper I see that "WMF-Office-IT" was archived but there does not seem to be a replacement for "ITS". There are tickets though that should still be tagged.. what should we do with those?

Fri, Oct 16, 12:50 AM · WMF-Office-IT, Operations, Mail
Dzahn added a project to T159750: E-mail for people in different OIT LDAP object unit : WMF-Office-IT.
Fri, Oct 16, 12:49 AM · WMF-Office-IT, Operations, Mail
Dzahn added a comment to T257906: Move testreduce away from scandium to a separate Buster Ganeti VM.

Alright, sounds good. Thank you @ssastry

Fri, Oct 16, 12:15 AM · Patch-For-Review, Parsoid-Tests, Parsoid, serviceops, Operations

Thu, Oct 15

Dzahn added a comment to T257906: Move testreduce away from scandium to a separate Buster Ganeti VM.

I am not sure what exactly needs to be done here as the next step. I have created a VM and both rt client/server and vd client/server are on it. Also added the ability to stop both clients.

Thu, Oct 15, 11:56 PM · Patch-For-Review, Parsoid-Tests, Parsoid, serviceops, Operations
Dzahn reassigned T252932: Forwarding or alias for fundraising@ from JGulingan to MBeat33.

re-assigning based on the last comment

Thu, Oct 15, 11:50 PM · fr-donorservices, Operations, Mail
Dzahn added a comment to T122144: Move most (all?) exim personal aliases to WMF ITS.

We will remove remaining personal aliases for non-staff once 2020 is over. The users will be contacted that they have time until end of year to switch to an alternative. This was decided in cooperation with ITS and the board.

Thu, Oct 15, 11:48 PM · Epic, Mail, Operations
Dzahn closed T265659: Please close unused mailing list fkc-l as Resolved.

"No such list fkc-l"

Thu, Oct 15, 11:44 PM · Wikimedia-Mailing-lists, Operations
Dzahn reassigned T265653: (Need By: TBD) setup/install deploy1002 from Dzahn to RobH.
Thu, Oct 15, 11:17 PM · ops-eqiad, DC-Ops, Operations
Dzahn added a comment to T265653: (Need By: TBD) setup/install deploy1002.

@RobH Please leave it on stretch for now. It relies on Mediawiki classes that are not ready for buster just yet.

Thu, Oct 15, 11:16 PM · ops-eqiad, DC-Ops, Operations
Dzahn added a comment to T265568: daily-image-l stopped sending on 2020-10-11.

ah, no it's antispam indeed:

Thu, Oct 15, 6:26 PM · Tools, Wikimedia-Mailing-lists, Commons, Operations
Dzahn added a comment to T265568: daily-image-l stopped sending on 2020-10-11.

Maybe the mails are not going out from @tools.wmflabs.org anymore but now from .wmcloud.org and that means it does not match the list of subscribers anymore. Since recently the names of instances changed in some places.

Thu, Oct 15, 6:10 PM · Tools, Wikimedia-Mailing-lists, Commons, Operations
Dzahn awarded T265172: Requesting adding to ores-admin for Ladsgroup a Like token.
Thu, Oct 15, 12:56 AM · Machine Learning Platform, Operations, SRE-Access-Requests
Dzahn added a comment to T265486: Deploy design-strategy Gerrit repo to design.wikimedia.org/design-strategy/.

Do you want the URL changed or is that fine as it is?

Thu, Oct 15, 12:21 AM · Operations, Product-Design-Strategy
Dzahn added a comment to T265486: Deploy design-strategy Gerrit repo to design.wikimedia.org/design-strategy/.

@Volker_E It's already here just that it's at https://design.wikimedia.org/strategy/

Thu, Oct 15, 12:20 AM · Operations, Product-Design-Strategy

Wed, Oct 14

Dzahn added a comment to T265558: decommission wtp2001 through wtp2020.

wtp2001 - wtp2005 depooled

Wed, Oct 14, 11:58 PM · Patch-For-Review, decommission-hardware
Dzahn claimed T265558: decommission wtp2001 through wtp2020.
Wed, Oct 14, 11:57 PM · Patch-For-Review, decommission-hardware
Dzahn added a comment to T265486: Deploy design-strategy Gerrit repo to design.wikimedia.org/design-strategy/.

There is an existing setup for this. puppet already clones from design/strategy to /srv/org/wikimedia/design-strategy. All you have to do is merge stuff in that repo. It's just like design/landing-page or design/blog. It is NOT like design/style-guide because that uses scap and the deployment server but as far as I remember the only reason to do that was that you wanted to use git-lfs, right?

Wed, Oct 14, 11:40 PM · Operations, Product-Design-Strategy
Dzahn added a comment to T247441: move 20 new codfw parsoid servers (parse2*) into production.

There was a request to decom the wtp2* servers. There is a new ticket for that at T265558.

Wed, Oct 14, 11:30 PM · serviceops, Operations
Dzahn created T265558: decommission wtp2001 through wtp2020.
Wed, Oct 14, 11:23 PM · Patch-For-Review, decommission-hardware

Tue, Oct 13

Dzahn awarded T260111: All sorts of random drifts in wikis in s3 a Party Time token.
Tue, Oct 13, 9:25 PM · Epic, DBA
Dzahn added a comment to T264698: Degraded RAID on mw2279.
21:08 <+icinga-wm> PROBLEM - Ensure local MW versions match expected deployment on mw2279 is CRITICAL: CRITICAL: 956 mismatched wikiversions 
                   https://wikitech.wikimedia.org/wiki/Application_servers
21:09 < mutante> interesting. is that being used as a test host right now?
Tue, Oct 13, 9:14 PM · serviceops, Operations, ops-codfw
Dzahn added a comment to T261403: Move dedcode/mwaddlink from github to gerrit.

if you help me with requesting a gerrit-repo that would be great (no experience with that yet). also

Tue, Oct 13, 7:51 PM · Patch-For-Review, serviceops, Release-Engineering-Team, Growth-Structured-Tasks, Growth-Team

Mon, Oct 12

Dzahn placed T264797: Update 2030.wikimedia.org redirect to new URI up for grabs.
Mon, Oct 12, 1:50 PM · Patch-For-Review, Wikimedia-Apache-configuration, Operations

Sat, Oct 10

Dzahn added a comment to T264859: Create Inari Sámi Wikipedia.

How can it check if it's in the wikistats? Is there an API?

Sat, Oct 10, 7:23 PM · MW-1.36-notes (1.36.0-wmf.13; 2020-10-12), User-Urbanecm, Wiki-Setup (Create), Wikimedia-Language-setup
Dzahn awarded T264910: Wikistats: Several wikis are absent from the wiki-table export a Barnstar token.
Sat, Oct 10, 8:15 AM · VPS-project-Wikistats
Dzahn closed T264910: Wikistats: Several wikis are absent from the wiki-table export as Resolved.

Also MW version 1.36 is now recognized as good version and colored green.

Sat, Oct 10, 1:43 AM · VPS-project-Wikistats
Dzahn added a comment to T264910: Wikistats: Several wikis are absent from the wiki-table export.

Thank you very much for the fixes. Yes, wikivoyage was missing and has now been added.

Sat, Oct 10, 1:40 AM · VPS-project-Wikistats
Dzahn added a watcher for VPS-project-Wikistats: Dzahn.
Sat, Oct 10, 12:29 AM
Dzahn added a comment to T264991: Upgrade the MediaWiki appservers to debian buster, icu63.
  • prometheus-nutcracker-exporter
Sat, Oct 10, 12:17 AM · Operations, serviceops

Fri, Oct 9

Dzahn added a parent task for T137291: Transition all use of EasyTimeline to the Graph extension and decommission it from Wikimedia's servers: T264991: Upgrade the MediaWiki appservers to debian buster, icu63.
Fri, Oct 9, 11:53 PM · Technical-Debt, Wikimedia-Extension-setup, Multimedia, Epic, MediaWiki-extensions-Graph, EasyTimeline
Dzahn added a parent task for T253377: WMF deployed EasyTimeline extension depends on Ploticus package which is not available in Debian Buster (but available again in Debian Bullseye): T264991: Upgrade the MediaWiki appservers to debian buster, icu63.
Fri, Oct 9, 11:53 PM · Operations, EasyTimeline, Packaging
Dzahn added subtasks for T264991: Upgrade the MediaWiki appservers to debian buster, icu63: T253377: WMF deployed EasyTimeline extension depends on Ploticus package which is not available in Debian Buster (but available again in Debian Bullseye), T137291: Transition all use of EasyTimeline to the Graph extension and decommission it from Wikimedia's servers.
Fri, Oct 9, 11:53 PM · Operations, serviceops
Dzahn added a comment to T264991: Upgrade the MediaWiki appservers to debian buster, icu63.

Fix all of our puppet code for MediaWiki for incompatibilities with buster

Fri, Oct 9, 11:31 PM · Operations, serviceops
Dzahn added a comment to T250515: Please provide our special component/php72 in buster-wikimedia.

T264991#6532447

Fri, Oct 9, 11:26 PM · Patch-For-Review, Packaging, serviceops
Dzahn added a comment to T264991: Upgrade the MediaWiki appservers to debian buster, icu63.

As well as T250515 for the PHP packages.

Fri, Oct 9, 11:26 PM · Operations, serviceops
Dzahn added a comment to T245757: upgrade MediaWiki appservers to Debian 10 (buster).

A new ticket for this has been created at T264991.

Fri, Oct 9, 11:25 PM · Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO, serviceops
Dzahn added a comment to T264991: Upgrade the MediaWiki appservers to debian buster, icu63.

There was already T245757 with dependency tickets.

Fri, Oct 9, 11:24 PM · Operations, serviceops
Dzahn added a comment to T260271: (Need By: TBD) rack/setup/install maps20[05-10].codfw.wmnet.

Is there a ticket for moving these into production?

Fri, Oct 9, 11:11 PM · Operations, Maps, ops-codfw, DC-Ops
Dzahn added a comment to T260271: (Need By: TBD) rack/setup/install maps20[05-10].codfw.wmnet.

maps2010 is reported as down since about 3 days

Fri, Oct 9, 11:04 PM · Operations, Maps, ops-codfw, DC-Ops
Dzahn added a comment to T265175: Remove Chase Pettet from security@ alias in Google.

Hi @sbassett security@ isn't in exim anymore nowadays. It's in Google, you'll have to ask OIT via Zendesk please:

Fri, Oct 9, 7:49 PM · Security-Team

Thu, Oct 8

Dzahn added a comment to T264504: Don't get a mail to confirm my email address (mx2001 is blacklisted by abusix blacklist).

in addition to having different message contents are also sent outward via the main mx host interface instead of the wiki-mail-site.wikimedia.org bulk mail interface.

Thu, Oct 8, 9:37 PM · Operations, Security, Mail
Dzahn added a comment to T209709: Feature: enable prometheus-nginx-exporter for nginx metrics.

+1. nginx is in use on install servers, puppetmaster, puppetdb, mirrors, docker registry. But if there is a goal to remove it maybe we should do another ticket for that specifically, with checkboxes.

Thu, Oct 8, 8:27 PM · observability
Dzahn added a comment to T265071: Renable SSH access for Lex Nasser, analytics intern.

Using the same key should be fine. But we will need a new "expiry_date" please. And should we use expiry_contact: nruiz@ like before?

Thu, Oct 8, 7:35 PM · Operations, SRE-Access-Requests, Analytics
Dzahn added a comment to T264894: Deployment of latest Design Style Guide Gerrit clone doesn't seem to succeed.

The deploy-style-guide.sh script git pulls from https://gerrit.wikimedia.org/r/p/design/style-guide.git

Thu, Oct 8, 2:01 AM · Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), Release-Engineering-Team (Development services), Gerrit, Wikimedia-GitHub, Wikimedia Design Style Guide, Operations
Dzahn added projects to T264894: Deployment of latest Design Style Guide Gerrit clone doesn't seem to succeed: Wikimedia-GitHub, Gerrit.
Thu, Oct 8, 1:51 AM · Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), Release-Engineering-Team (Development services), Gerrit, Wikimedia-GitHub, Wikimedia Design Style Guide, Operations
Dzahn added a comment to T264894: Deployment of latest Design Style Guide Gerrit clone doesn't seem to succeed.

I checked the git status on both backends, miscweb1002 and miscweb2002 and they are both at commit e3fda830a1573a398eda486341f2f6c2cc25bffe " “Design principles”: Improve “Consistency” image and make it mobile friendly (#390)" from June 30th 2020.

Thu, Oct 8, 1:50 AM · Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), Release-Engineering-Team (Development services), Gerrit, Wikimedia-GitHub, Wikimedia Design Style Guide, Operations

Wed, Oct 7

Dzahn added a project to T264918: Access to the Logstash for John Bolorinos: SRE-Access-Requests.
Wed, Oct 7, 8:05 PM · LDAP-Access-Requests, Operations
Dzahn added a watcher for Wiki-Setup: Dzahn.
Wed, Oct 7, 7:50 PM
Dzahn added a member for Wiki-Setup: Dzahn.
Wed, Oct 7, 7:50 PM
Dzahn added a comment to T264859: Create Inari Sámi Wikipedia.

https://www.wikidata.org/wiki/Q100154036

Wed, Oct 7, 7:41 PM · MW-1.36-notes (1.36.0-wmf.13; 2020-10-12), User-Urbanecm, Wiki-Setup (Create), Wikimedia-Language-setup
Dzahn added a comment to T264859: Create Inari Sámi Wikipedia.

added to DNS:

Wed, Oct 7, 7:38 PM · MW-1.36-notes (1.36.0-wmf.13; 2020-10-12), User-Urbanecm, Wiki-Setup (Create), Wikimedia-Language-setup
Dzahn removed a project from T242602: Sort out plan for install* servers in edge sites: Patch-For-Review.
Wed, Oct 7, 5:19 PM · Operations
Dzahn closed T242602: Sort out plan for install* servers in edge sites as Resolved.

I'm declaring this resolved. Everything is done according to the plan. Let me know if you think otherwise.

Wed, Oct 7, 5:19 PM · Operations
Dzahn added a comment to T242602: Sort out plan for install* servers in edge sites.

quoting from the original task description and replying inline:

Wed, Oct 7, 12:50 AM · Operations
Dzahn removed a project from T252526: serve tftpboot environment from the install servers and create one in each edge POP: Patch-For-Review.
Wed, Oct 7, 12:43 AM · Operations
Dzahn updated the task description for T252526: serve tftpboot environment from the install servers and create one in each edge POP.
Wed, Oct 7, 12:42 AM · Operations
Dzahn closed T252526: serve tftpboot environment from the install servers and create one in each edge POP, a subtask of T242602: Sort out plan for install* servers in edge sites, as Resolved.
Wed, Oct 7, 12:41 AM · Operations
Dzahn closed T252526: serve tftpboot environment from the install servers and create one in each edge POP as Resolved.
Wed, Oct 7, 12:41 AM · Operations
Dzahn added a comment to T252526: serve tftpboot environment from the install servers and create one in each edge POP.

☑️ eqsin tested - reinstalled testvm5001 and confirmed in syslog that install5001 is serving both DHCP and TFTP now; not bast5001 (tftp) or install2003 (dhcp) anymore; and OS install is working

Wed, Oct 7, 12:39 AM · Operations

Tue, Oct 6

Dzahn committed rOHPUf4e2ba2a8a51: switch DHCP servers in POPs to new local install hosts (authored by Dzahn).
switch DHCP servers in POPs to new local install hosts
Tue, Oct 6, 11:29 PM
Dzahn awarded T210993: Deprecate Diamond collectors in Cloud VPS a Orange Medal token.
Tue, Oct 6, 9:04 PM · Patch-For-Review, cloud-services-team (Kanban), observability, Operations

Mon, Oct 5

Dzahn updated subscribers of T210993: Deprecate Diamond collectors in Cloud VPS.

asked Mukunda about it and he says it is probably obsolete. asked -traffic because i do see @ema still logged in there not too long ago

Mon, Oct 5, 10:50 PM · Patch-For-Review, cloud-services-team (Kanban), observability, Operations
Dzahn added a comment to T220531: Get the clouddb-services systems into Shinken and possibly icinga.

shinken instance and puppet module have been deleted. this leaves just the icinga part and / or is blocked by T250206

Mon, Oct 5, 10:41 PM · cloud-services-team (Kanban), Data-Services
Dzahn added a comment to T85624: Better abstractions for puppet & icinga/nagios/shinken.

The shinken module has been deleted in https://gerrit.wikimedia.org/r/c/operations/puppet/+/629464

Mon, Oct 5, 10:39 PM · Operations, observability

Sat, Oct 3

Dzahn awarded T256930: Dereckson's blog is broken a Like token.
Sat, Oct 3, 3:23 PM · User-Dereckson
Dzahn created P12911 bashisms - z.sh.
Sat, Oct 3, 12:53 AM
Dzahn created P12910 bashisms.
Sat, Oct 3, 12:47 AM
Dzahn added a comment to T200984: Stop introducing new code expanded from erb templates.

also see: T148494 and T245266 for open tasks about adding lint checks for shell scripts.

Sat, Oct 3, 12:11 AM · Operations, Puppet
Dzahn added a comment to T200984: Stop introducing new code expanded from erb templates.

This seems somewhat but not exactly a duplicate of T254480. The other ticket would be resolved once we have removed all cases of it from the repo.

Sat, Oct 3, 12:07 AM · Operations, Puppet

Fri, Oct 2

Dzahn added a comment to T227485: Decommission analytics10[28-31,33-41].

The hosts here are showing up in a weird state. When running the DNS cookbook you get warnings that these hosts exist but are not "in devices". Though running the cookbook above should have cleaned that up.

Fri, Oct 2, 9:43 PM · ops-eqiad, decommission-hardware, Operations

Thu, Oct 1

Dzahn created P12890 (An Untitled Masterwork).
Thu, Oct 1, 10:15 PM
Dzahn edited P12865 building etherpad package.
Thu, Oct 1, 1:19 AM
Dzahn created P12865 building etherpad package.
Thu, Oct 1, 1:16 AM

Wed, Sep 30

Dzahn added a comment to T264163: Add link to codesearch in header.

Wed, Sep 30, 7:46 PM · Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), Release-Engineering-Team (Development services), Gerrit
Dzahn added a comment to T252526: serve tftpboot environment from the install servers and create one in each edge POP.

netops could you please deploy https://gerrit.wikimedia.org/r/c/operations/homer/public/+/631261 ?

Wed, Sep 30, 7:24 PM · Operations

Tue, Sep 29

Dzahn added a comment to T159412: Convert all of our site.pp/roles to the role/profile paradigm.

This topic branch shows previous work I had done for this but never linked to this ticket: 2 more were just merged today:

Tue, Sep 29, 9:17 PM · Patch-For-Review, Cloud-Services, Technical-Debt, Puppet, Operations
Dzahn added a comment to T238305: Servers freezing across the caching cluster.

Just a few minutes ago: db2125 crashed - mgmt iface also not available T260670

Tue, Sep 29, 9:03 PM · Operations, Traffic
Dzahn reassigned T262921: Add Bereket teshome to the ldap/wmde and ldap/nda group from ArielGlenn to bete.

@bete You have been added to both groups as requested. Things should work now. Let us know if any issues.

Tue, Sep 29, 8:22 PM · LDAP-Access-Requests, Operations
Dzahn updated the task description for T262921: Add Bereket teshome to the ldap/wmde and ldap/nda group.
Tue, Sep 29, 8:21 PM · LDAP-Access-Requests, Operations
Dzahn added a comment to T262921: Add Bereket teshome to the ldap/wmde and ldap/nda group.

@ArielGlenn LDAP-only users also need to be added to puppet admins module nowadays.

Tue, Sep 29, 6:55 PM · LDAP-Access-Requests, Operations

Mon, Sep 28

Dzahn edited P12821 install server flows.
Mon, Sep 28, 11:59 PM
Dzahn created P12821 install server flows.
Mon, Sep 28, 11:45 PM

Sat, Sep 26

Dzahn awarded T262525: Create arbcom-ru@wikimedia.org a Like token.
Sat, Sep 26, 1:47 AM · Operations, Wikimedia-Mailing-lists

Fri, Sep 25

Dzahn added a comment to T263830: contint.wikimedia.org: add TLS termination.

This was basically all done:

Fri, Sep 25, 6:43 PM · Continuous-Integration-Infrastructure, Operations, Traffic

Thu, Sep 24

Dzahn added a comment to T263073: Large, steady increase in unprocessed cloudelastic job.cirrusSearchElasticaWrite messages.

The maintenance periodic timer has been removed on mwmaint hosts.

Thu, Sep 24, 9:41 PM · Patch-For-Review, Discovery-Search (Current work)
Dzahn created T263790: cloudinfra hosts switching between 2 puppet changes / changes on every puppet run.
Thu, Sep 24, 8:11 PM · cloud-services-team (Kanban)