Page MenuHomePhabricator
Create Task
Maniphest T434142

Figure out what to do with an-worker1189
Closed, ResolvedPublic
Actions

Description

Per this Slack conversation with @Scott_French , an-worker1189, which is currently associated with Puppet role analytics_cluster::hadoop::worker is not in Puppet, but seems to be creating TCP connections to confd ports. I'm not familiar with the hadoop workers, but @RKemper might remember that this host had RAID issues in the past.

Creating this ticket to:

  • Check the current status of an-worker1189. Is it currently part of the Hadoop cluster? Answer: yes
  • Decide as a team what do to with this host (retire, reimage, repurpose?)

Related Objects

Event Timeline

bking created this task.Wed, Aug 5, 7:33 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptWed, Aug 5, 7:33 PM
bking changed the task status from Open to In Progress.Wed, Aug 5, 7:41 PM
bking claimed this task.
bking triaged this task as Medium priority.
bking edited projects, added: Data-Platform-SRE (2026-07-03 - 2026-07-31); removed: Data-Platform-SRE.
bking updated the task description. (Show Details)
bking added a comment.Wed, Aug 5, 7:44 PM

I checked an-worker1189's status using this Wikitech guide , the Hadoop web UI shows it is an active namenode (see attached)

Screenshot 2026-08-05 at 19.42.53.png (2,362×1,304 px, 377 KB)
.

So it's likely we'll just need to re-enroll this host into Puppet. I'll give that a shot now.

Stashbot added a comment.Wed, Aug 5, 7:51 PM

Mentioned in SAL (#wikimedia-operations) [2026-08-05T19:51:17Z] <inflatador> [bking@puppetserver1001] ~$ sudo puppetserver ca sign --certname an-worker1189.eqiad.wmnet T434142

bking closed this task as Resolved.Wed, Aug 5, 7:56 PM
bking moved this task from Backlog - project to Done on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.

Yes, it looks like Puppet was disabled for a server reboot and then never got re-enabled:

The last Puppet run was at Thu Jul 16 19:04:21 UTC 2026 (28840 minutes ago). Puppet is disabled. Reboot.

I manually re-enrolled the host in Puppet. So to answer the original question, an-worker1189 is an active production server that fell out of Puppet. That's been fixed, so I'm closing out this ticket.

If you are interested in the work to get this host onto Bookworm or later, see T406746 .

bking updated the task description. (Show Details)Wed, Aug 5, 8:01 PM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.Fri, Aug 7, 7:52 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits