Page MenuHomePhabricator

wikistream.wmflabs.org down - unable to ssh to ws-web
Closed, ResolvedPublic

Description

The wikistream.wmflabs.org service is currently down, and I wanted to login to ws-web to see what the problem might be. I created the wikistream project a few years ago, and in the past I've been able to ssh into bastion..wmflabs.org and then ssh to ws-web. Today when I do that I am able to login to bastion.wmflabs.org successfully but I am unable to login to ws-web.

Permission denied (publickey).

Please advise.

Event Timeline

Are you using a different SSH key to what you were using previously?

Add your current pubkey to https://wikitech.wikimedia.org/wiki/Special:Preferences#mw-prefsection-openstack ?

I can see that the ssh-key for my laptop's id_rsa.pub and the id_rsa.pub in my homedir on bastion.wmflabs.org are present in that list.

I don't know if T103148 is related or not. Maybe my homedir on ws-web was somehow modified?

Looking at auth.log on ws-web, I saw a bunch of:

Sep  2 10:35:20 ws-web sshd[16317]: Connection from 10.68.17.232 port 52570 on 10.68.22.255 port 22
Sep  2 10:35:20 ws-web sshd[16317]: Invalid user edsu from 10.68.17.232
Sep  2 10:35:20 ws-web sshd[16317]: input_userauth_request: invalid user edsu [preauth]
Sep  2 10:35:20 ws-web sshd[16317]: Connection closed by 10.68.17.232 [preauth]

I see that /home/edsu does exist, probably from the time Yuvi removed NFS and copied it over, and user Edsu has projectadmin in project wikistream.

root@ws-web:/home# ls -l
total 8
drwx------ 4  2171 500 4096 Feb 10  2017 edsu

Then I tried to look up id edsu which returned nothing, which seemed odd. I rebooted the instance, and it's now up and claims

root@ws-web:~# id edsu
uid=2171(edsu) gid=500(wikidev) groups=50062(project-bastion),50340(project-wikistream),50380(project-tools),52296(tools.anon),52297(tools.linkypedia),52306(tools.congressedits),500(wikidev)

So I'm pretty sure it is fine now, and some auth related service got restarted and fixed itself, not sure what exactly.

@edsu check now?

I believe that auth-related service would likely be nslcd/nscd (I forget which).

edsu claimed this task.

Yes, I'm able to log in now. Thanks so much!