Page MenuHomePhabricator

toolsbeta-puppetmaster3 is unreachable
Closed, ResolvedPublic

Description

In contrast to T109853, the console output for this host is empty. If this is due to T109752, the instance is probably gone for good?

Event Timeline

scfc raised the priority of this task from to Needs Triage.
scfc updated the task description. (Show Details)
scfc added a project: Toolforge.
scfc subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
scfc claimed this task.

I didn't have enough patience; after a lengthy wait after the reboot, the instance finally came up. /var/log/syslog says:

[…]
Aug 24 13:26:34 toolsbeta-puppetmaster3 crontab[8163]: (root) LIST (root)
Aug 24 13:26:34 toolsbeta-puppetmaster3 crontab[8165]: (root) LIST (puppet)
Aug 24 13:26:46 toolsbeta-puppetmaster3 puppet-agent[7843]: Finished catalog run in 17.20 seconds
Aug 24 13:27:01 toolsbeta-puppetmaster3 CRON[8580]: (root) CMD (/usr/local/sbin/certcleaner.py > /dev/null 2>&1)
Aug 24 13:28:01 toolsbeta-puppetmaster3 CRON[8598]: (root) CMD (/usr/local/sbin/certcleaner.py > /dev/null 2>&1)
Aug 24 13:28:21 toolsbeta-puppetmaster3 nslcd[1050]: [c7d38b] <passwd(all)> passwd entry uid=80686,ou=people,dc=wikimedia,dc=org denied by validnames option:
Aug 24 13:29:01 toolsbeta-puppetmaster3 CRON[8620]: (root) CMD (/usr/local/sbin/certcleaner.py > /dev/null 2>&1)
Aug 24 13:30:01 toolsbeta-puppetmaster3 CRON[8640]: (root) CMD (/usr/local/bin/git-sync-upstream >>/var/log/git-sync-upstream.log 2>&1)
Aug 24 13:30:01 toolsbeta-puppetmaster3 CRON[8641]: (root) CMD (/usr/local/sbin/certcleaner.py > /dev/null 2>&1)
Aug 24 13:30:23 toolsbeta-puppetmaster3 nslcd[1050]: [397850] <passwd(all)> passwd entry uid=80686,ou=people,dc=wikimedia,dc=org denied by validnames option:
Aug 27 17:20:33 toolsbeta-puppetmaster3 kernel: imklog 5.8.6, log source = /proc/kmsg started.
Aug 27 17:20:33 toolsbeta-puppetmaster3 rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="812" x-info="http://www.rsyslog.com"] start
Aug 27 17:20:33 toolsbeta-puppetmaster3 rsyslogd: rsyslogd's groupid changed to 103
Aug 27 17:20:33 toolsbeta-puppetmaster3 rsyslogd: rsyslogd's userid changed to 101
Aug 27 17:20:33 toolsbeta-puppetmaster3 rsyslogd-2039: Could not open output pipe '/dev/xconsole' [try http://www.rsyslog.com/e/2039 ]
[…]

so apparently August 24th had some bad karma.