can't log in to phab-01.eqiad.wmflabs
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• mmodell
	Feb 3 2016, 4:07 PM

Description

phab-01 and 02 are currently inaccessible via ssh. @yuvipanda did some troubleshooting yesterday and was unable to get either one to allow ssh logins. Rebooting the instances hasn't helped and I have no idea how to troubleshoot it any further.

I'm temporarily using phab-scap.eqiad.wmflabs, however, a lot of configuration and test data is accumulated in phab-01, it'd be a shame to lose it.

As a last resort, assuming it's not a simple fix to get ssh working, is there any way to mount the disk image from phab-01 into another running vm so that I can extract the data?

Related Objects
Search...

Status	Assigned	Task
Invalid	None	T127617 Upgrade phab-01.wmflabs.org
Resolved	Paladox	T137270 https://phab-01.wmflabs.org returns a core exception
Resolved	• mmodell	T125666 can't log in to phab-01.eqiad.wmflabs

Event Timeline

• mmodell created this task.Feb 3 2016, 4:07 PM

• mmodell raised the priority of this task from to Needs Triage.

• mmodell updated the task description. (Show Details)

• mmodell added projects: Cloud-VPS, Phabricator.

• mmodell added subscribers: • mmodell, yuvipanda.

Restricted Application added a project: Cloud-Services. · View Herald TranscriptFeb 3 2016, 4:07 PM

Restricted Application added subscribers: StudiesWorld, Luke081515, scfc, Aklapper. · View Herald Transcript

Luke081515 triaged this task as High priority.Feb 4 2016, 6:17 PM

Luke081515 moved this task from To Triage to Misc on the Phabricator board.

Negative24 subscribed.Feb 5 2016, 4:01 AM

I logged into phab-02 yesterday (actually I still am in tmux). /tmp is a 1 MiB partition that was full when I logged in with temp files from a mkinitramfs generation (probably from a kernel upgrade) which I cleared out. Now, /tmp is again full with mkinitramfs files.

And from auth.log: User twentyafterfour from bastion-01.bastion.eqiad.wmflabs not allowed because not listed in AllowUsers.

I got this mail:

Puppet is failing to run on the "phab-03" instance in the Wikimedia Labs
project "phabricator"

when running puppet there:

phab-03 is a Puppet client of deploy.phabricator.eqiad.wmflabs (puppetclient)
The last Puppet run was at Fri Feb 5 21:32:35 UTC 2016 (3245 minutes ago).

@yuvipanda how were you able to debug and what did you see? i also cant login as root on that instance.

Over two days I've gotten four email. Two about phab-03 and two about harbormaster1.

I'm trying to make sense of the Grafana disk space available graphs. I don't know specifically how the root's partition is fluctuating but it would be helpful to have all the other partitions available too. (maybe we could get a Grafana admin to add the * to view all drives)

• mmodell mentioned this in T126323: broken labs instances (ssh or perms), do we care?.Feb 9 2016, 6:52 PM

• mmodell closed this task as Resolved.Feb 18 2016, 4:11 AM

• mmodell claimed this task.