Page MenuHomePhabricator
Create Task
Maniphest T124133

NFS overload is causing instances to freeze
Closed, ResolvedPublic
Actions

Assigned To
None
Authored By
Andrew
Jan 20 2016, 2:14 AM
Tags
Referenced Files
F3251879: Screen Shot 2016-01-20 at 5.36.37 PM.png
Jan 21 2016, 12:11 AM
F3251874: Screen Shot 2016-01-20 at 5.36.49 PM.png
Jan 21 2016, 12:11 AM
F3251876: Screen Shot 2016-01-20 at 5.36.43 PM.png
Jan 21 2016, 12:11 AM
F3251872: Screen Shot 2016-01-20 at 5.36.55 PM.png
Jan 21 2016, 12:11 AM
F3251931: Screen Shot 2016-01-20 at 5.42.32 PM.png
Jan 21 2016, 12:11 AM
F3250796: Screen Shot 2016-01-20 at 3.44.43 PM.png
Jan 20 2016, 10:18 PM
F3250981: Screen Shot 2016-01-20 at 4.00.09 PM.png
Jan 20 2016, 10:18 PM
F3250823: Screen Shot 2016-01-20 at 3.58.04 PM.png
Jan 20 2016, 10:18 PM
View All 12 Files
Subscribers
Aklapper
• AlexMonk-WMF
Andrew
• chasemp
hashar
• MZMcBride
Paladox
View All 12 Subscribers

Related Objects
Search...

Event Timeline

Andrew created this task.Jan 20 2016, 2:14 AM
Andrew raised the priority of this task from to Needs Triage.
Andrew updated the task description. (Show Details)
Andrew added projects: Cloud-Services, Cloud-VPS.
Andrew subscribed.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptJan 20 2016, 2:14 AM
Andrew added subtasks: T123835: tools-webgrid-lighttpd-1204 locked up, T124038: Webservice stuck, won't stop, can't restart.Jan 20 2016, 2:15 AM
Magnus reopened subtask T124038: Webservice stuck, won't stop, can't restart as Open.Jan 20 2016, 10:56 AM
scfc subscribed.Jan 20 2016, 1:37 PM

@Andrew: If an instance locks up, do you want to take care of that yourself for debugging purposes, or is rebooting and mentioning the instance name here/in the blocking bugs's titles (to check which virtual host that relates to) enough?

scfc renamed this task from Labs instances sometimes freeze (Tracking bug) to Labs instances sometimes freeze.Jan 20 2016, 2:29 PM
scfc added a project: Tracking-Neverending.
scfc set Security to None.
scfc added a subtask: T124162: tools-webgrid-lighttpd-1209 frozen.
scfc closed subtask T124038: Webservice stuck, won't stop, can't restart as Resolved.Jan 20 2016, 2:33 PM
scfc closed subtask T124162: tools-webgrid-lighttpd-1209 frozen as Resolved.Jan 20 2016, 3:43 PM
• MZMcBride subscribed.Jan 20 2016, 5:24 PM
• chasemp triaged this task as High priority.Jan 20 2016, 7:46 PM
• chasemp closed subtask T123835: tools-webgrid-lighttpd-1204 locked up as Resolved.
• chasemp added a subtask: T122719: tools-webgrid-lighttpd-1201 webservices and ssh unaccessible.Jan 20 2016, 7:49 PM
• chasemp closed subtask T122719: tools-webgrid-lighttpd-1201 webservices and ssh unaccessible as Resolved.
• chasemp subscribed.EditedJan 20 2016, 8:41 PM

I did some looking regarding the lockup of tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs since I was able to catch it and I indicated in the subtask but want to surface a few things here.

Linux tools-webgrid-lighttpd-1209 3.2.0-75-virtual #110-Ubuntu SMP Tue Dec 16 19:24:01 UTC 2014 x86_64 x86_64 x86_64 GNU/Linu

Login hung with:

ssh tools-webgrid-lighttpd-1209.eqiad.wmflabs
The authenticity of host 'tools-webgrid-lighttpd-1209.eqiad.wmflabs (<no hostip for proxy command>)' can't be established.
RSA key fingerprint is 74:91:d3:4a:5b:87:34:ed:7a:cf:0f:d6:4d:c6:56:cc.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'tools-webgrid-lighttpd-1209.eqiad.wmflabs' (RSA) to the list of known hosts.

It was responsive to salt at the time and so I was able to poke around. I could ls in root's home dir for instance:

salt 'tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs' cmd.run 'ls'

I was able to issue top via salt and can see the CPU is not overly burdened yet the load is really high

https://phabricator.wikimedia.org/P2488

P2488 tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs top 1
1root@labcontrol1001:~# salt 'tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs' cmd.run 'top -b -n 1'
2tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs:
3 top - 13:36:26 up 21 days, 10:19, 0 users, load average: 102.97, 102.23, 101.3
4 Tasks: 490 total, 1 running, 489 sleeping, 0 stopped, 0 zombie
5 Cpu(s): 34.4%us, 3.0%sy, 0.0%ni, 61.7%id, 0.1%wa, 0.0%hi, 0.4%si, 0.5%st
6 Mem: 8178496k total, 5849252k used, 2329244k free, 248580k buffers
7 Swap: 25034744k total, 0k used, 25034744k free, 3512616k cached
8
9 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10 347 root 20 0 12176 1440 840 R 4 0.0 0:00.04 top
11 1003 root 20 0 942m 3888 1656 S 2 0.0 4:24.48 nscd
12 1039 nslcd 20 0 437m 6952 1868 S 2 0.1 4:19.64 nslcd
13 1 root 20 0 20064 2224 1284 S 0 0.0 0:07.84 init
14 2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
15 3 root 20 0 0 0 0 S 0 0.0 0:50.51 ksoftirqd/0
16 4 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/0:0
17 5 root 20 0 0 0 0 S 0 0.0 0:00.08 kworker/u:0
18 6 root RT 0 0 0 0 S 0 0.0 0:04.11 migration/0
19 7 root RT 0 0 0 0 S 0 0.0 0:11.51 watchdog/0
20 8 root RT 0 0 0 0 S 0 0.0 0:04.01 migration/1
21 10 root 20 0 0 0 0 S 0 0.0 4:03.32 ksoftirqd/1
22 11 root 20 0 0 0 0 S 0 0.0 3:11.57 kworker/0:1
23 12 root RT 0 0 0 0 S 0 0.0 0:11.46 watchdog/1
24 13 root RT 0 0 0 0 S 0 0.0 0:04.94 migration/2
25 15 root 20 0 0 0 0 S 0 0.0 0:49.39 ksoftirqd/2
26 16 root RT 0 0 0 0 S 0 0.0 0:10.07 watchdog/2
27 17 root RT 0 0 0 0 S 0 0.0 0:03.62 migration/3
28 18 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/3:0
29 19 root 20 0 0 0 0 S 0 0.0 0:40.26 ksoftirqd/3
30 20 root RT 0 0 0 0 S 0 0.0 0:09.34 watchdog/3
31 21 root 0 -20 0 0 0 S 0 0.0 0:00.00 cpuset
32 22 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper
33 23 root 20 0 0 0 0 S 0 0.0 0:00.00 kdevtmpfs
34 24 root 0 -20 0 0 0 S 0 0.0 0:00.00 netns
35 25 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/u:1
36 26 root 20 0 0 0 0 S 0 0.0 0:05.61 sync_supers
37 27 root 20 0 0 0 0 S 0 0.0 0:00.30 bdi-default
38 28 root 0 -20 0 0 0 S 0 0.0 0:00.00 kintegrityd
39 29 root 0 -20 0 0 0 S 0 0.0 0:00.00 kblockd
40 30 root 0 -20 0 0 0 S 0 0.0 0:00.00 ata_sff
41 31 root 20 0 0 0 0 S 0 0.0 0:00.00 khubd
42 32 root 0 -20 0 0 0 S 0 0.0 0:00.00 md
43 33 root 20 0 0 0 0 S 0 0.0 1:25.58 kworker/2:1
44 35 root 20 0 0 0 0 S 0 0.0 0:01.62 khungtaskd
45 36 root 20 0 0 0 0 S 0 0.0 0:03.19 kswapd0
46 37 root 25 5 0 0 0 S 0 0.0 0:00.00 ksmd
47 38 root 39 19 0 0 0 S 0 0.0 0:00.00 khugepaged
48 39 root 20 0 0 0 0 S 0 0.0 0:00.00 fsnotify_mark
49 40 root 20 0 0 0 0 S 0 0.0 0:00.00 ecryptfs-kthrea
50 41 root 0 -20 0 0 0 S 0 0.0 0:00.00 crypto
51 49 root 0 -20 0 0 0 S 0 0.0 0:00.00 kthrotld
52 50 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_0
53 51 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_1
54 72 root 0 -20 0 0 0 S 0 0.0 0:00.00 devfreq_wq
55 214 root 20 0 0 0 0 S 0 0.0 48:15.07 kworker/1:1
56 215 root 20 0 0 0 0 S 0 0.0 2:28.44 kworker/3:1
57 221 root 0 -20 0 0 0 S 0 0.0 0:00.00 kdmflush
58 228 root 0 -20 0 0 0 S 0 0.0 0:00.00 kdmflush
59 236 root 20 0 0 0 0 S 0 0.0 0:41.15 jbd2/vda1-8
60 237 root 0 -20 0 0 0 S 0 0.0 0:00.00 ext4-dio-unwrit
61 320 root 20 0 17228 636 456 S 0 0.0 0:00.06 upstart-udev-br
62 331 root 20 0 15208 1264 740 S 0 0.0 0:00.05 udevd
63 338 root 20 0 19196 1008 724 S 0 0.0 0:03.72 rpcbind
64 345 root 20 0 588m 39m 1368 S 0 0.5 0:00.00 salt-minion
65 346 root 20 0 4396 608 508 S 0 0.0 0:00.00 sh
66 367 statd 20 0 27336 1980 1288 S 0 0.0 0:00.00 rpc.statd
67 458 root 20 0 0 0 0 S 0 0.0 1:54.53 vballoon
68 464 root 20 0 15204 852 324 S 0 0.0 0:00.00 udevd
69 465 root 20 0 15184 832 324 S 0 0.0 0:00.00 udevd
70 474 root 0 -20 0 0 0 S 0 0.0 0:00.00 kpsmoused
71 487 root 20 0 0 0 0 S 0 0.0 0:05.63 jbd2/dm-1-8
72 488 root 0 -20 0 0 0 S 0 0.0 0:00.00 ext4-dio-unwrit
73 568 root 20 0 15184 384 196 S 0 0.0 0:00.00 upstart-socket-
74 608 root 20 0 15032 1084 544 S 0 0.0 0:00.07 dhclient3
75 661 root 0 -20 0 0 0 S 0 0.0 0:00.00 rpciod
76 678 root 0 -20 0 0 0 S 0 0.0 0:00.00 nfsiod
77 682 root 20 0 0 0 0 S 0 0.0 0:00.00 nfsv4.0-svc
78 684 root 20 0 0 0 0 S 0 0.0 0:35.27 flush-253:0
79 732 ntp 20 0 39360 2832 1972 S 0 0.0 3:14.95 ntpd
80 748 diamond 20 0 570m 25m 5300 S 0 0.3 30:42.31 diamond
81 802 syslog 20 0 241m 5152 1068 S 0 0.1 2:21.22 rsyslogd
82 804 messageb 20 0 19580 684 452 S 0 0.0 0:00.02 dbus-daemon
83 866 root 20 0 33404 880 748 S 0 0.0 0:00.00 getty
84 872 root 20 0 33404 876 748 S 0 0.0 0:00.00 getty
85 874 root 20 0 588m 47m 9528 S 0 0.6 15:11.31 salt-minion
86 878 root 20 0 33404 880 748 S 0 0.0 0:00.00 getty
87 879 root 20 0 33404 880 748 S 0 0.0 0:00.00 getty
88 882 root 20 0 33404 876 748 S 0 0.0 0:00.00 getty
89 886 root 20 0 52568 2956 2336 S 0 0.0 0:01.21 sshd
90 887 daemon 20 0 10572 352 220 S 0 0.0 0:00.05 atd
91 888 root 20 0 12776 932 724 S 0 0.0 0:07.11 cron
92 909 root 20 0 15976 700 524 S 0 0.0 4:07.51 irqbalance
93 933 sgeadmin 20 0 22548 1988 1544 S 0 0.0 0:00.00 sge_shepherd
94 935 tools.po 20 0 52300 2936 1576 D 0 0.0 5:48.38 lighttpd
95 936 Debian-e 20 0 45392 1148 732 S 0 0.0 0:01.98 exim4
96 937 sgeadmin 20 0 22548 1996 1544 S 0 0.0 0:00.00 sge_shepherd
97 939 tools.sd 20 0 52152 2920 1684 D 0 0.0 1:37.51 lighttpd
98 940 sgeadmin 20 0 22548 1988 1544 S 0 0.0 0:00.00 sge_shepherd
99 942 tools.wi 20 0 52332 3156 1660 D 0 0.0 1:33.50 lighttpd
100 959 root 20 0 47044 1528 944 S 0 0.0 1:45.36 lldpd
101 960 _lldpd 20 0 42804 676 204 S 0 0.0 0:39.55 lldpd
102 966 root 20 0 6992 384 216 S 0 0.0 0:00.00 mcelog
103 975 tools.po 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
104 981 nagios 20 0 23340 1108 616 S 0 0.0 4:30.94 nrpe
105 995 tools.po 20 0 307m 13m 8880 S 0 0.2 0:00.38 php-cgi
106 1001 tools.wi 20 0 307m 13m 8868 S 0 0.2 0:00.03 php-cgi
107 1004 tools.wi 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
108 1005 tools.wi 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
109 1007 tools.po 20 0 17092 1388 1132 D 0 0.0 137:34.48 gam_server
110 1008 tools.wi 20 0 307m 13m 8868 S 0 0.2 0:00.03 php-cgi
111 1017 tools.wi 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
112 1024 tools.wi 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
113 1035 tools.wi 20 0 16972 1280 1108 D 0 0.0 11:42.46 gam_server
114 1050 tools.sd 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
115 1052 tools.sd 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
116 1053 tools.sd 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
117 1054 tools.sd 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
118 1056 tools.sd 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
119 1057 tools.sd 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
120 1058 tools.sd 20 0 4392 604 508 S 0 0.0 0:00.00 server.sh
121 1061 tools.sd 20 0 443m 25m 5368 S 0 0.3 57:06.40 python
122 1063 tools.sd 20 0 16972 1288 1112 D 0 0.0 2:06.84 gam_server
123 1066 root 20 0 8768 524 408 S 0 0.0 0:00.53 inetd
124 1105 sgeadmin 20 0 62768 8076 2208 D 0 0.1 515:38.85 sge_execd
125 1276 www-data 20 0 44332 1048 292 S 0 0.0 1:22.48 lighttpd
126 1304 root 20 0 34320 816 704 S 0 0.0 0:00.00 getty
127 1305 root 20 0 7332 716 604 S 0 0.0 0:00.00 getty
128 3104 tools.ca 20 0 314m 12m 4376 S 0 0.2 0:13.14 php-cgi
129 3136 tools.po 20 0 308m 8184 2668 S 0 0.1 0:01.56 php-cgi
130 3302 tools.wi 20 0 339m 9928 3380 S 0 0.1 0:02.84 php-cgi
131 5686 tools.po 20 0 308m 8304 2668 S 0 0.1 0:01.56 php-cgi
132 5766 tools.wi 20 0 339m 9.8m 3324 S 0 0.1 0:02.54 php-cgi
133 6414 tools.wi 20 0 338m 8856 3284 S 0 0.1 0:00.10 php-cgi
134 6703 sgeadmin 20 0 22548 2000 1552 S 0 0.0 0:00.00 sge_shepherd
135 6705 51058 20 0 53044 3796 1692 D 0 0.0 1:44.01 lighttpd
136 6729 51058 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
137 6731 51058 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
138 6732 51058 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
139 6733 51058 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
140 6735 51058 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
141 6736 51058 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
142 6738 51058 20 0 17316 1624 1144 D 0 0.0 719:19.58 gam_server
143 6956 sgeadmin 20 0 22548 1996 1548 S 0 0.0 0:00.00 sge_shepherd
144 6958 tools.ca 20 0 52532 3268 1588 D 0 0.0 9:51.10 lighttpd
145 6988 tools.ca 20 0 307m 13m 8880 S 0 0.2 0:00.33 php-cgi
146 6992 tools.ca 20 0 307m 13m 8880 S 0 0.2 0:01.05 php-cgi
147 6997 tools.ca 20 0 16972 1284 1108 D 0 0.0 5:16.09 gam_server
148 7011 sgeadmin 20 0 22548 2008 1552 S 0 0.0 0:00.00 sge_shepherd
149 7013 52421 20 0 52188 2804 1556 D 0 0.0 1:36.84 lighttpd
150 7033 52421 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
151 7035 52421 20 0 312m 9436 2936 S 0 0.1 0:00.26 php-cgi
152 7036 52421 20 0 312m 9464 2924 S 0 0.1 0:00.24 php-cgi
153 7037 52421 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
154 7039 52421 20 0 312m 9552 2964 S 0 0.1 0:08.39 php-cgi
155 7040 52421 20 0 312m 9540 2952 S 0 0.1 0:08.17 php-cgi
156 7042 52421 20 0 16972 1284 1112 D 0 0.0 7:39.02 gam_server
157 7056 sgeadmin 20 0 22548 2000 1552 S 0 0.0 0:00.00 sge_shepherd
158 7058 tools.al 20 0 52072 2796 1628 D 0 0.0 1:36.00 lighttpd
159 7082 tools.al 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
160 7084 tools.al 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
161 7085 tools.al 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
162 7086 tools.al 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
163 7088 tools.al 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
164 7089 tools.al 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
165 7091 tools.al 20 0 16972 1288 1112 D 0 0.0 9:44.89 gam_server
166 7092 sgeadmin 20 0 22548 1992 1548 S 0 0.0 0:00.00 sge_shepherd
167 7094 tools.ti 20 0 52168 2844 1636 S 0 0.0 1:35.84 lighttpd
168 7118 tools.ti 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
169 7120 tools.ti 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
170 7121 tools.ti 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
171 7122 tools.ti 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
172 7124 tools.ti 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
173 7125 tools.ti 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
174 7127 tools.ti 20 0 16972 1288 1112 D 0 0.0 34:08.19 gam_server
175 7189 sgeadmin 20 0 22548 2000 1548 S 0 0.0 0:00.00 sge_shepherd
176 7191 tools.ex 20 0 52076 2500 1468 D 0 0.0 1:35.43 lighttpd
177 7211 tools.ex 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
178 7213 tools.ex 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
179 7214 tools.ex 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
180 7215 tools.ex 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
181 7217 tools.ex 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
182 7218 tools.ex 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
183 7220 tools.ex 20 0 16972 1284 1112 D 0 0.0 44:15.11 gam_server
184 7379 sgeadmin 20 0 22548 1992 1548 S 0 0.0 0:00.00 sge_shepherd
185 7381 tools.po 20 0 52460 3172 1552 D 0 0.0 1:35.47 lighttpd
186 7437 sgeadmin 20 0 22548 2000 1548 S 0 0.0 0:00.00 sge_shepherd
187 7439 tools.di 20 0 52076 2492 1468 D 0 0.0 0:22.88 lighttpd
188 7440 sgeadmin 20 0 22548 2000 1552 S 0 0.0 0:00.00 sge_shepherd
189 7442 tools.ia 20 0 52192 2912 1688 D 0 0.0 0:23.43 lighttpd
190 7443 sgeadmin 20 0 22548 1996 1552 S 0 0.0 0:00.00 sge_shepherd
191 7449 tools.mp 20 0 52180 2800 1560 D 0 0.0 0:23.28 lighttpd
192 7464 tools.po 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
193 7466 tools.po 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
194 7467 tools.po 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
195 7468 tools.po 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
196 7470 tools.po 20 0 325m 23m 3836 S 0 0.3 0:11.78 php-cgi
197 7471 tools.po 20 0 325m 23m 3840 S 0 0.3 0:12.67 php-cgi
198 7473 tools.po 20 0 16972 1288 1112 D 0 0.0 29:49.62 gam_server
199 7551 tools.mp 20 0 307m 13m 8868 S 0 0.2 0:00.04 php-cgi
200 7556 tools.ia 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
201 7557 tools.mp 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
202 7558 tools.mp 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
203 7560 tools.ia 20 0 325m 27m 4920 S 0 0.3 0:02.84 php-cgi
204 7561 tools.ia 20 0 319m 20m 4856 S 0 0.3 0:01.34 php-cgi
205 7562 tools.mp 20 0 307m 13m 8868 S 0 0.2 0:00.03 php-cgi
206 7564 tools.mp 20 0 315m 14m 5344 S 0 0.2 0:12.80 php-cgi
207 7565 tools.mp 20 0 315m 14m 5316 S 0 0.2 0:13.37 php-cgi
208 7566 tools.ia 20 0 307m 13m 8868 S 0 0.2 0:00.04 php-cgi
209 7568 tools.ia 20 0 381m 83m 4928 S 0 1.0 1:13.47 php-cgi
210 7569 tools.ia 20 0 376m 77m 4948 S 0 1.0 1:09.92 php-cgi
211 7571 tools.mp 20 0 16972 1292 1112 D 0 0.0 5:07.47 gam_server
212 7573 tools.ia 20 0 16972 1284 1112 D 0 0.0 1:15.15 gam_server
213 7587 tools.di 20 0 307m 13m 8868 S 0 0.2 0:00.03 php-cgi
214 7589 tools.di 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
215 7590 tools.di 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
216 7591 tools.di 20 0 307m 13m 8868 S 0 0.2 0:00.03 php-cgi
217 7593 tools.di 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
218 7594 tools.di 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
219 7596 tools.di 20 0 16972 1288 1112 D 0 0.0 1:14.38 gam_server
220 7774 sgeadmin 20 0 22548 1992 1548 S 0 0.0 0:00.00 sge_shepherd
221 7776 tools.pl 20 0 52580 2980 1568 D 0 0.0 0:09.27 lighttpd
222 7844 tools.pl 20 0 307m 13m 8868 S 0 0.2 0:00.05 php-cgi
223 7858 tools.pl 20 0 307m 13m 8880 S 0 0.2 0:00.04 php-cgi
224 8615 sgeadmin 20 0 22548 2000 1548 S 0 0.0 0:00.00 sge_shepherd
225 8617 tools.hr 20 0 52572 3256 1588 D 0 0.0 1:39.15 lighttpd
226 8640 tools.hr 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
227 8643 tools.hr 20 0 309m 9928 2868 S 0 0.1 0:25.69 php-cgi
228 8644 tools.hr 20 0 313m 13m 2848 S 0 0.2 0:47.89 php-cgi
229 8649 tools.hr 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
230 8653 tools.hr 20 0 310m 10m 2904 S 0 0.1 2:50.51 php-cgi
231 8654 tools.hr 20 0 309m 9608 2892 S 0 0.1 2:53.32 php-cgi
232 8661 tools.hr 20 0 16972 1280 1108 D 0 0.0 55:25.57 gam_server
233 9482 sgeadmin 20 0 22548 2004 1548 S 0 0.0 0:00.00 sge_shepherd
234 9484 51141 20 0 52216 2732 1556 D 0 0.0 1:34.91 lighttpd
235 9504 51141 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
236 9506 51141 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
237 9507 51141 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
238 9508 51141 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
239 9510 51141 20 0 309m 9.8m 3144 S 0 0.1 0:03.74 php-cgi
240 9511 51141 20 0 308m 9372 2696 S 0 0.1 0:03.67 php-cgi
241 9513 51141 20 0 16972 1284 1108 D 0 0.0 78:19.16 gam_server
242 9576 tools.sl 20 0 308m 7764 2500 S 0 0.1 0:00.63 php-cgi
243 9579 tools.sl 20 0 308m 7776 2500 S 0 0.1 0:00.59 php-cgi
244 9614 sgeadmin 20 0 22548 2008 1548 S 0 0.0 0:00.00 sge_shepherd
245 9616 52196 20 0 52208 2852 1568 D 0 0.0 1:32.69 lighttpd
246 9636 52196 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
247 9638 52196 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
248 9639 52196 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
249 9640 52196 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
250 9642 52196 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
251 9643 52196 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
252 9645 52196 20 0 16972 1280 1108 D 0 0.0 4:52.14 gam_server
253 9741 sgeadmin 20 0 22548 2008 1552 S 0 0.0 0:00.00 sge_shepherd
254 9743 tools.fa 20 0 52076 2720 1564 D 0 0.0 1:33.61 lighttpd
255 9763 tools.fa 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
256 9765 tools.fa 20 0 307m 6704 1652 S 0 0.1 0:00.00 php-cgi
257 9766 tools.fa 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
258 9767 tools.fa 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
259 9769 tools.fa 20 0 307m 6752 1668 S 0 0.1 0:00.23 php-cgi
260 9770 tools.fa 20 0 307m 6752 1668 S 0 0.1 0:00.22 php-cgi
261 9772 tools.fa 20 0 16972 1288 1112 D 0 0.0 7:09.21 gam_server
262 9786 sgeadmin 20 0 22548 1992 1548 S 0 0.0 0:00.00 sge_shepherd
263 9788 tools.ig 20 0 52196 2772 1524 D 0 0.0 2:03.84 lighttpd
264 9808 tools.ig 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
265 9810 tools.ig 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
266 9811 tools.ig 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
267 9812 tools.ig 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
268 9814 tools.ig 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
269 9815 tools.ig 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
270 9817 tools.ig 20 0 16972 1284 1112 D 0 0.0 48:21.34 gam_server
271 9820 sgeadmin 20 0 22548 1992 1548 S 0 0.0 0:00.00 sge_shepherd
272 9822 tools.wi 20 0 52320 3036 1560 D 0 0.0 1:31.40 lighttpd
273 9842 tools.wi 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
274 9844 tools.wi 20 0 321m 23m 5172 S 0 0.3 0:20.60 php-cgi
275 9845 tools.wi 20 0 321m 23m 5244 S 0 0.3 0:13.71 php-cgi
276 9846 tools.wi 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
277 9848 tools.wi 20 0 319m 22m 5260 S 0 0.3 1:35.27 php-cgi
278 9849 tools.wi 20 0 318m 21m 5244 S 0 0.3 2:13.41 php-cgi
279 9851 tools.wi 20 0 16972 1280 1108 D 0 0.0 11:18.52 gam_server
280 9852 sgeadmin 20 0 22548 1996 1552 S 0 0.0 0:00.00 sge_shepherd
281 9854 tools.fa 20 0 52076 2504 1468 D 0 0.0 1:32.80 lighttpd
282 9874 tools.fa 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
283 9876 tools.fa 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
284 9877 tools.fa 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
285 9878 tools.fa 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
286 9880 tools.fa 20 0 309m 9496 4000 S 0 0.1 0:00.47 php-cgi
287 9881 tools.fa 20 0 335m 9592 4128 S 0 0.1 0:00.41 php-cgi
288 9883 tools.fa 20 0 16972 1284 1112 D 0 0.0 7:39.37 gam_server
289 9897 sgeadmin 20 0 22548 1996 1548 S 0 0.0 0:00.00 sge_shepherd
290 9899 tools.am 20 0 52196 2920 1584 D 0 0.0 1:33.41 lighttpd
291 9919 tools.am 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
292 9921 tools.am 20 0 308m 8244 2836 S 0 0.1 0:00.01 php-cgi
293 9922 tools.am 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
294 9923 tools.am 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
295 9925 tools.am 20 0 308m 8308 2872 S 0 0.1 0:01.50 php-cgi
296 9926 tools.am 20 0 308m 8316 2880 S 0 0.1 0:01.39 php-cgi
297 9928 tools.am 20 0 16972 1284 1112 D 0 0.0 45:06.96 gam_server
298 9940 sgeadmin 20 0 22548 1992 1548 S 0 0.0 0:00.00 sge_shepherd
299 9942 tools.an 20 0 52076 2732 1560 D 0 0.0 1:35.78 lighttpd
300 9962 tools.an 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
301 9964 tools.an 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
302 9965 tools.an 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
303 9966 tools.an 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
304 9968 tools.an 20 0 308m 7496 2184 S 0 0.1 0:00.66 php-cgi
305 9969 tools.an 20 0 308m 7568 2184 S 0 0.1 0:00.64 php-cgi
306 9971 tools.an 20 0 16972 1284 1108 D 0 0.0 5:24.35 gam_server
307 9974 sgeadmin 20 0 22548 1996 1548 S 0 0.0 0:00.00 sge_shepherd
308 9976 tools.wi 20 0 105m 5620 2616 D 0 0.1 13:07.44 lighttpd
309 10000 tools.wi 20 0 307m 13m 8880 S 0 0.2 0:00.17 php-cgi
310 10004 tools.wi 20 0 307m 13m 8880 S 0 0.2 0:01.60 php-cgi
311 10009 tools.wi 20 0 17084 1356 1128 D 0 0.0 38:40.14 gam_server
312 10021 sgeadmin 20 0 22548 2000 1552 S 0 0.0 0:00.00 sge_shepherd
313 10023 tools.si 20 0 52076 2724 1556 S 0 0.0 1:31.24 lighttpd
314 10041 tools.si 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
315 10043 tools.si 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
316 10044 tools.si 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
317 10045 tools.si 20 0 307m 13m 8868 S 0 0.2 0:00.06 php-cgi
318 10047 tools.si 20 0 308m 7312 1920 S 0 0.1 0:00.09 php-cgi
319 10048 tools.si 20 0 308m 7312 1920 S 0 0.1 0:00.08 php-cgi
320 10050 tools.si 20 0 16972 1284 1112 D 0 0.0 7:15.61 gam_server
321 10707 root 20 0 0 0 0 S 0 0.0 0:01.72 flush-252:1
322 10717 tools.ip 20 0 312m 8372 2820 S 0 0.1 0:02.47 php-cgi
323 11665 sgeadmin 20 0 22548 2004 1552 S 0 0.0 0:00.00 sge_shepherd
324 11667 tools.sl 20 0 52076 2728 1560 D 0 0.0 0:58.88 lighttpd
325 11668 sgeadmin 20 0 22548 1992 1548 S 0 0.0 0:00.00 sge_shepherd
326 11670 tools.ip 20 0 52412 3084 1600 D 0 0.0 0:54.08 lighttpd
327 11671 sgeadmin 20 0 22548 1996 1548 S 0 0.0 0:00.00 sge_shepherd
328 11673 tools.gr 20 0 52076 2504 1468 D 0 0.0 0:41.06 lighttpd
329 11692 sgeadmin 20 0 22548 1996 1548 S 0 0.0 0:00.00 sge_shepherd
330 11695 tools.cg 20 0 54380 3052 1684 S 0 0.0 2:49.81 lighttpd
331 11732 tools.ip 20 0 307m 13m 8868 S 0 0.2 0:00.04 php-cgi
332 11738 tools.gr 20 0 307m 13m 8868 S 0 0.2 0:00.03 php-cgi
333 11763 tools.gr 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
334 11766 tools.gr 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
335 11777 tools.sl 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
336 11783 tools.ip 20 0 307m 13m 8880 S 0 0.2 0:00.03 php-cgi
337 11787 tools.gr 20 0 307m 13m 8868 S 0 0.2 0:00.03 php-cgi
338 11791 tools.sl 20 0 308m 7780 2504 S 0 0.1 0:00.52 php-cgi
339 11793 tools.sl 20 0 308m 7776 2504 S 0 0.1 0:00.55 php-cgi
340 11799 tools.gr 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
341 11802 tools.gr 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
342 11807 tools.cg 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
343 11808 tools.sl 20 0 307m 13m 8880 S 0 0.2 0:00.07 php-cgi
344 11810 tools.ip 20 0 16972 1288 1112 D 0 0.0 3:54.09 gam_server
345 11812 tools.cg 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
346 11814 tools.cg 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
347 11815 tools.gr 20 0 16972 1284 1112 D 0 0.0 1:46.12 gam_server
348 11823 tools.cg 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
349 11825 tools.sl 20 0 16972 1288 1112 D 0 0.0 2:04.72 gam_server
350 11827 tools.cg 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
351 11828 tools.cg 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
352 11833 tools.cg 20 0 464m 24m 4032 S 0 0.3 150:23.33 cgstat.fcgi
353 11837 tools.cg 20 0 16972 1280 1108 D 0 0.0 3:08.01 gam_server
354 12193 tools.po 20 0 308m 8172 2668 S 0 0.1 0:01.32 php-cgi
355 12366 tools.po 20 0 308m 8348 2780 S 0 0.1 0:01.21 php-cgi
356 12576 sgeadmin 20 0 22548 1996 1548 S 0 0.0 0:00.00 sge_shepherd
357 12578 51177 20 0 52196 2960 1676 D 0 0.0 0:00.94 lighttpd
358 12602 51177 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
359 12604 51177 20 0 310m 9728 3012 S 0 0.1 0:14.63 php-cgi
360 12605 51177 20 0 310m 9492 3008 S 0 0.1 0:13.47 php-cgi
361 12606 51177 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
362 12608 51177 20 0 310m 9696 3012 S 0 0.1 1:56.29 php-cgi
363 12609 51177 20 0 309m 9372 3012 S 0 0.1 2:35.91 php-cgi
364 12611 51177 20 0 16972 1288 1112 D 0 0.0 0:02.22 gam_server
365 12653 tools.ca 20 0 315m 13m 4376 S 0 0.2 0:19.64 php-cgi
366 12678 tools.ca 20 0 313m 11m 4376 S 0 0.1 0:19.28 php-cgi
367 17951 tools.ip 20 0 312m 8472 2816 S 0 0.1 0:11.19 php-cgi
368 19724 51559 20 0 308m 9.9m 4120 S 0 0.1 0:07.98 php-cgi
369 19776 tools.pl 20 0 16972 1344 1136 D 0 0.0 0:48.26 gam_server
370 19804 tools.pl 20 0 309m 9.9m 3996 D 0 0.1 0:08.49 php-cgi
371 20311 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/1:2
372 20806 51559 20 0 313m 14m 4208 S 0 0.2 0:22.41 php-cgi
373 21249 sgeadmin 20 0 22548 1988 1544 S 0 0.0 0:00.00 sge_shepherd
374 21251 51559 20 0 53980 4716 1584 D 0 0.1 1:22.45 lighttpd
375 21277 51559 20 0 307m 13m 8868 S 0 0.2 0:00.03 php-cgi
376 21282 51559 20 0 307m 13m 8880 S 0 0.2 0:00.09 php-cgi
377 21287 51559 20 0 16972 1284 1112 D 0 0.0 8:32.86 gam_server
378 21398 tools.pl 20 0 309m 9.8m 3976 S 0 0.1 0:00.63 php-cgi
379 21412 tools.pl 20 0 309m 9.8m 3976 S 0 0.1 0:00.54 php-cgi
380 21467 tools.je 20 0 16972 1288 1112 S 0 0.0 0:00.00 gam_server
381 21867 sgeadmin 20 0 22548 2000 1548 S 0 0.0 0:00.00 sge_shepherd
382 21869 tools.je 20 0 52072 2552 1520 D 0 0.0 0:00.67 lighttpd
383 21894 tools.je 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
384 21896 tools.je 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
385 21897 tools.je 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
386 21898 tools.je 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
387 21900 tools.je 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
388 21901 tools.je 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
389 21947 tools.wi 20 0 339m 9876 3508 S 0 0.1 0:00.37 php-cgi
390 22097 tools.pl 20 0 309m 9.8m 3976 S 0 0.1 0:00.65 php-cgi
391 22309 sgeadmin 20 0 22548 2112 1608 S 0 0.0 0:00.00 sge_shepherd
392 22338 tools.ft 20 0 0 0 0 D 0 0.0 0:00.00 php-cgi
393 22344 tools.ft 20 0 16972 1288 1112 S 0 0.0 0:00.00 gam_server
394 22554 tools.ft 20 0 0 0 0 D 0 0.0 0:00.00 lighttpd
395 22578 tools.ft 20 0 25572 1188 668 D 0 0.0 0:00.00 sge_shepherd
396 23117 tools.ip 20 0 312m 8364 2816 S 0 0.1 0:01.85 php-cgi
397 23178 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
398 23179 sshd 20 0 62672 1336 632 S 0 0.0 0:00.00 sshd
399 23301 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
400 23302 root 20 0 4396 608 508 S 0 0.0 0:00.00 sh
401 23307 root 20 0 38908 1000 852 S 0 0.0 0:00.00 find
402 23308 root 20 0 31124 824 680 D 0 0.0 0:00.32 fuser
403 23529 root 20 0 35860 1184 852 S 0 0.0 0:00.00 cron
404 23531 root 20 0 4396 612 508 S 0 0.0 0:00.00 sh
405 23533 root 20 0 4296 604 496 S 0 0.0 0:00.00 run-parts
406 23699 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
407 23700 sshd 20 0 62672 1340 628 S 0 0.0 0:00.00 sshd
408 23930 tools.ip 20 0 312m 8352 2812 S 0 0.1 0:01.39 php-cgi
409 24287 root 0 -20 20044 7764 3392 S 0 0.1 0:15.49 atop
410 24358 root 20 0 4396 612 508 S 0 0.0 0:00.00 exim4-base
411 24363 root 20 0 31396 608 516 S 0 0.0 0:00.00 xargs
412 24364 root 20 0 10336 612 504 D 0 0.0 0:00.00 start-stop-daem
413 24392 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
414 24393 root 20 0 4396 612 508 S 0 0.0 0:00.00 sh
415 24398 root 20 0 38908 1000 852 S 0 0.0 0:00.00 find
416 24399 root 20 0 31124 824 680 D 0 0.0 0:00.38 fuser
417 25014 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
418 25015 sshd 20 0 62672 1332 628 S 0 0.0 0:00.00 sshd
419 25137 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
420 25138 root 20 0 4396 604 504 S 0 0.0 0:00.00 sh
421 25143 root 20 0 38908 1000 852 S 0 0.0 0:00.00 find
422 25144 root 20 0 31124 824 680 D 0 0.0 0:00.35 fuser
423 25447 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
424 25448 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
425 25570 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
426 25571 root 20 0 4396 608 504 S 0 0.0 0:00.00 sh
427 25576 root 20 0 38908 996 852 S 0 0.0 0:00.00 find
428 25577 root 20 0 31124 824 680 D 0 0.0 0:00.46 fuser
429 26198 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
430 26199 sshd 20 0 62672 1336 632 S 0 0.0 0:00.00 sshd
431 26345 root 20 0 35860 1184 852 S 0 0.0 0:00.00 cron
432 26347 root 20 0 4396 608 508 S 0 0.0 0:00.00 sh
433 26354 root 20 0 38908 996 852 S 0 0.0 0:00.00 find
434 26355 root 20 0 31124 824 680 D 0 0.0 0:00.29 fuser
435 26570 51559 20 0 310m 11m 4292 S 0 0.1 0:20.13 php-cgi
436 26722 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
437 26723 sshd 20 0 62672 1336 632 S 0 0.0 0:00.01 sshd
438 26869 root 20 0 35860 1184 852 S 0 0.0 0:00.00 cron
439 26872 root 20 0 4396 612 508 S 0 0.0 0:00.00 sh
440 26878 root 20 0 38908 1000 852 S 0 0.0 0:00.00 find
441 26879 root 20 0 31124 828 680 D 0 0.0 0:00.41 fuser
442 27491 tools.ca 20 0 314m 12m 4380 S 0 0.2 0:20.21 php-cgi
443 27554 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
444 27555 sshd 20 0 62672 1336 628 S 0 0.0 0:00.01 sshd
445 27677 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
446 27678 root 20 0 4396 608 508 S 0 0.0 0:00.00 sh
447 27683 root 20 0 38908 996 852 S 0 0.0 0:00.00 find
448 27684 root 20 0 31124 824 680 D 0 0.0 0:00.41 fuser
449 27987 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.02 sshd
450 27988 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
451 28097 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
452 28098 root 20 0 4396 612 508 S 0 0.0 0:00.00 sh
453 28103 root 20 0 38908 996 852 S 0 0.0 0:00.00 find
454 28104 root 20 0 31124 824 680 D 0 0.0 0:00.44 fuser
455 28210 51559 20 0 308m 9800 4188 S 0 0.1 0:08.18 php-cgi
456 28719 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
457 28720 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
458 28842 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
459 28843 root 20 0 4396 612 504 S 0 0.0 0:00.00 sh
460 28848 root 20 0 38908 996 852 S 0 0.0 0:00.00 find
461 28849 root 20 0 31124 824 680 D 0 0.0 0:00.34 fuser
462 29152 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.02 sshd
463 29153 sshd 20 0 62672 1336 628 S 0 0.0 0:00.01 sshd
464 29275 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
465 29276 root 20 0 4396 608 508 S 0 0.0 0:00.00 sh
466 29281 root 20 0 38908 996 852 S 0 0.0 0:00.00 find
467 29282 root 20 0 31124 824 680 D 0 0.0 0:00.31 fuser
468 29897 scfc 20 0 97832 4104 3084 D 0 0.1 0:00.01 sshd
469 29898 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
470 30020 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
471 30021 root 20 0 4396 608 508 S 0 0.0 0:00.00 sh
472 30026 root 20 0 38908 996 852 S 0 0.0 0:00.00 find
473 30027 root 20 0 31124 820 680 D 0 0.0 0:00.35 fuser
474 30330 scfc 20 0 97832 4092 3072 D 0 0.1 0:00.00 sshd
475 30331 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
476 30453 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
477 30454 root 20 0 4396 612 508 S 0 0.0 0:00.00 sh
478 30459 root 20 0 38908 996 852 S 0 0.0 0:00.00 find
479 30460 root 20 0 31124 820 680 D 0 0.0 0:00.37 fuser
480 31008 tools.ph 20 0 97832 4100 3084 D 0 0.1 0:00.00 sshd
481 31009 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
482 31197 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
483 31198 root 20 0 4396 612 504 S 0 0.0 0:00.00 sh
484 31203 root 20 0 38908 996 852 S 0 0.0 0:00.00 find
485 31204 root 20 0 31124 824 680 D 0 0.0 0:00.30 fuser
486 31252 root 20 0 0 0 0 S 0 0.0 0:47.78 kworker/2:0
487 31508 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
488 31509 sshd 20 0 62672 1340 628 S 0 0.0 0:00.00 sshd
489 31618 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
490 31619 root 20 0 4396 608 508 S 0 0.0 0:00.00 sh
491 31624 root 20 0 38908 996 852 S 0 0.0 0:00.00 find
492 31625 root 20 0 31124 828 680 D 0 0.0 0:00.34 fuser
493 32361 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
494 32362 root 20 0 4396 608 508 S 0 0.0 0:00.00 sh
495 32367 root 20 0 38908 996 852 S 0 0.0 0:00.00 find
496 32368 root 20 0 31124 824 680 D 0 0.0 0:00.30 fuser
497 32685 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
498 32686 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
499 32713 root 20 0 98.9m 4284 3232 D 0 0.1 0:00.02 sshd

load average: 102.97, 102.23, 101.3

root@labcontrol1001:~# salt 'tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs' cmd.run 'uptime'
tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs:
 13:37:12 up 21 days, 10:20,  0 users,  load average: 102.99, 102.34, 101.41
root@labcontrol1001:~# vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0 492212 3853144 2482140 25370184    0    0     0    38    0    0 16  2 82  0  0

dmesg

P2489 tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs dmesg
1root@labcontrol1001:~# salt 'tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs' cmd.run 'dmesg'
2tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs:
3 [ 0.000000] Initializing cgroup subsys cpuset
4 [ 0.000000] Initializing cgroup subsys cpu
5 [ 0.000000] Linux version 3.2.0-75-virtual (buildd@kissel) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #110-Ubuntu SMP Tue Dec 16 19:24:01 UTC 2014 (Ubuntu 3.2.0-75.110-virtual 3.2.64)
6 [ 0.000000] Command line: root=UUID=a548aa63-f471-4b8e-aa4f-7e6ab09ec721 ro quiet splash console=ttyS0
7 [ 0.000000] KERNEL supported cpus:
8 [ 0.000000] Intel GenuineIntel
9 [ 0.000000] AMD AuthenticAMD
10 [ 0.000000] Centaur CentaurHauls
11 [ 0.000000] BIOS-provided physical RAM map:
12 [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
13 [ 0.000000] BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
14 [ 0.000000] BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
15 [ 0.000000] BIOS-e820: 0000000000100000 - 00000000bfffe000 (usable)
16 [ 0.000000] BIOS-e820: 00000000bfffe000 - 00000000c0000000 (reserved)
17 [ 0.000000] BIOS-e820: 00000000feffc000 - 00000000ff000000 (reserved)
18 [ 0.000000] BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved)
19 [ 0.000000] BIOS-e820: 0000000100000000 - 0000000240000000 (usable)
20 [ 0.000000] NX (Execute Disable) protection: active
21 [ 0.000000] SMBIOS 2.4 present.
22 [ 0.000000] DMI: OpenStack Foundation OpenStack Nova, BIOS Bochs 01/01/2011
23 [ 0.000000] e820 update range: 0000000000000000 - 0000000000010000 (usable) ==> (reserved)
24 [ 0.000000] e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
25 [ 0.000000] No AGP bridge found
26 [ 0.000000] last_pfn = 0x240000 max_arch_pfn = 0x400000000
27 [ 0.000000] MTRR default type: write-back
28 [ 0.000000] MTRR fixed ranges enabled:
29 [ 0.000000] 00000-9FFFF write-back
30 [ 0.000000] A0000-BFFFF uncachable
31 [ 0.000000] C0000-FFFFF write-protect
32 [ 0.000000] MTRR variable ranges enabled:
33 [ 0.000000] 0 base 00C0000000 mask FFC0000000 uncachable
34 [ 0.000000] 1 disabled
35 [ 0.000000] 2 disabled
36 [ 0.000000] 3 disabled
37 [ 0.000000] 4 disabled
38 [ 0.000000] 5 disabled
39 [ 0.000000] 6 disabled
40 [ 0.000000] 7 disabled
41 [ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
42 [ 0.000000] last_pfn = 0xbfffe max_arch_pfn = 0x400000000
43 [ 0.000000] found SMP MP-table at [ffff8800000f0b10] f0b10
44 [ 0.000000] initial memory mapped : 0 - 20000000
45 [ 0.000000] Base memory trampoline at [ffff88000009a000] 9a000 size 20480
46 [ 0.000000] Using GB pages for direct mapping
47 [ 0.000000] init_memory_mapping: 0000000000000000-00000000bfffe000
48 [ 0.000000] 0000000000 - 0080000000 page 1G
49 [ 0.000000] 0080000000 - 00bfe00000 page 2M
50 [ 0.000000] 00bfe00000 - 00bfffe000 page 4k
51 [ 0.000000] kernel direct mapping tables up to bfffe000 @ 1fffd000-20000000
52 [ 0.000000] init_memory_mapping: 0000000100000000-0000000240000000
53 [ 0.000000] 0100000000 - 0240000000 page 1G
54 [ 0.000000] kernel direct mapping tables up to 240000000 @ bfffd000-bfffe000
55 [ 0.000000] RAMDISK: 37b09000 - 37ff0000
56 [ 0.000000] ACPI: RSDP 00000000000f08c0 00014 (v00 BOCHS )
57 [ 0.000000] ACPI: RSDT 00000000bffffcc4 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001)
58 [ 0.000000] ACPI: FACP 00000000bffff1c0 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001)
59 [ 0.000000] ACPI: DSDT 00000000bfffe040 01180 (v01 BOCHS BXPCDSDT 00000001 BXPC 00000001)
60 [ 0.000000] ACPI: FACS 00000000bfffe000 00040
61 [ 0.000000] ACPI: SSDT 00000000bffff234 00A00 (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001)
62 [ 0.000000] ACPI: APIC 00000000bffffc34 00090 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001)
63 [ 0.000000] ACPI: RSDT 00000000bffffcc4 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001)
64 [ 0.000000] ACPI: Local APIC address 0xfee00000
65 [ 0.000000] No NUMA configuration found
66 [ 0.000000] Faking a node at 0000000000000000-0000000240000000
67 [ 0.000000] Initmem setup node 0 0000000000000000-0000000240000000
68 [ 0.000000] NODE_DATA [000000023fffb000 - 000000023fffffff]
69 [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
70 [ 0.000000] kvm-clock: cpu 0, msr 0:1cf7681, boot clock
71 [ 0.000000] [ffffea0000000000-ffffea0008ffffff] PMD -> [ffff880237600000-ffff88023f5fffff] on node 0
72 [ 0.000000] Zone PFN ranges:
73 [ 0.000000] DMA 0x00000010 -> 0x00001000
74 [ 0.000000] DMA32 0x00001000 -> 0x00100000
75 [ 0.000000] Normal 0x00100000 -> 0x00240000
76 [ 0.000000] Movable zone start PFN for each node
77 [ 0.000000] early_node_map[3] active PFN ranges
78 [ 0.000000] 0: 0x00000010 -> 0x0000009f
79 [ 0.000000] 0: 0x00000100 -> 0x000bfffe
80 [ 0.000000] 0: 0x00100000 -> 0x00240000
81 [ 0.000000] On node 0 totalpages: 2097037
82 [ 0.000000] DMA zone: 64 pages used for memmap
83 [ 0.000000] DMA zone: 5 pages reserved
84 [ 0.000000] DMA zone: 3914 pages, LIFO batch:0
85 [ 0.000000] DMA32 zone: 16320 pages used for memmap
86 [ 0.000000] DMA32 zone: 766014 pages, LIFO batch:31
87 [ 0.000000] Normal zone: 20480 pages used for memmap
88 [ 0.000000] Normal zone: 1290240 pages, LIFO batch:31
89 [ 0.000000] ACPI: PM-Timer IO Port: 0xb008
90 [ 0.000000] ACPI: Local APIC address 0xfee00000
91 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
92 [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
93 [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
94 [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
95 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
96 [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
97 [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
98 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
99 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
100 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
101 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
102 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
103 [ 0.000000] ACPI: IRQ0 used by override.
104 [ 0.000000] ACPI: IRQ2 used by override.
105 [ 0.000000] ACPI: IRQ5 used by override.
106 [ 0.000000] ACPI: IRQ9 used by override.
107 [ 0.000000] ACPI: IRQ10 used by override.
108 [ 0.000000] ACPI: IRQ11 used by override.
109 [ 0.000000] Using ACPI (MADT) for SMP configuration information
110 [ 0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
111 [ 0.000000] nr_irqs_gsi: 40
112 [ 0.000000] PM: Registered nosave memory: 000000000009f000 - 00000000000a0000
113 [ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000
114 [ 0.000000] PM: Registered nosave memory: 00000000000f0000 - 0000000000100000
115 [ 0.000000] PM: Registered nosave memory: 00000000bfffe000 - 00000000c0000000
116 [ 0.000000] PM: Registered nosave memory: 00000000c0000000 - 00000000feffc000
117 [ 0.000000] PM: Registered nosave memory: 00000000feffc000 - 00000000ff000000
118 [ 0.000000] PM: Registered nosave memory: 00000000ff000000 - 00000000fffc0000
119 [ 0.000000] PM: Registered nosave memory: 00000000fffc0000 - 0000000100000000
120 [ 0.000000] Allocating PCI resources starting at c0000000 (gap: c0000000:3effc000)
121 [ 0.000000] Booting paravirtualized kernel on KVM
122 [ 0.000000] setup_percpu: NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:4 nr_node_ids:1
123 [ 0.000000] PERCPU: Embedded 27 pages/cpu @ffff88023fc00000 s78784 r8192 d23616 u524288
124 [ 0.000000] pcpu-alloc: s78784 r8192 d23616 u524288 alloc=1*2097152
125 [ 0.000000] pcpu-alloc: [0] 0 1 2 3
126 [ 0.000000] kvm-clock: cpu 0, msr 2:3fc12681, primary cpu clock
127 [ 0.000000] KVM setup async PF for cpu 0
128 [ 0.000000] kvm-stealtime: cpu 0, msr 23fc0cd40
129 [ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 2060168
130 [ 0.000000] Policy zone: Normal
131 [ 0.000000] Kernel command line: root=UUID=a548aa63-f471-4b8e-aa4f-7e6ab09ec721 ro quiet splash console=ttyS0
132 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
133 [ 0.000000] xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
134 [ 0.000000] Checking aperture...
135 [ 0.000000] No AGP bridge found
136 [ 0.000000] Calgary: detecting Calgary via BIOS EBDA area
137 [ 0.000000] Calgary: Unable to locate Rio Grande table in EBDA - bailing!
138 [ 0.000000] Memory: 8169768k/9437184k available (6574k kernel code, 1049036k absent, 218380k reserved, 6622k data, 924k init)
139 [ 0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
140 [ 0.000000] Hierarchical RCU implementation.
141 [ 0.000000] RCU dyntick-idle grace-period acceleration is enabled.
142 [ 0.000000] NR_IRQS:4352 nr_irqs:712 16
143 [ 0.000000] Console: colour VGA+ 80x25
144 [ 0.000000] console [ttyS0] enabled
145 [ 0.000000] allocated 67108864 bytes of page_cgroup
146 [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
147 [ 0.000000] Detected 2693.248 MHz processor.
148 [ 0.004000] Calibrating delay loop (skipped) preset value.. 5386.49 BogoMIPS (lpj=10772992)
149 [ 0.004000] pid_max: default: 32768 minimum: 301
150 [ 0.004000] Security Framework initialized
151 [ 0.004000] AppArmor: AppArmor initialized
152 [ 0.004000] Yama: becoming mindful.
153 [ 0.004000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
154 [ 0.004752] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
155 [ 0.006354] Mount-cache hash table entries: 256
156 [ 0.006455] Initializing cgroup subsys cpuacct
157 [ 0.006459] Initializing cgroup subsys memory
158 [ 0.006466] Initializing cgroup subsys devices
159 [ 0.006468] Initializing cgroup subsys freezer
160 [ 0.006469] Initializing cgroup subsys blkio
161 [ 0.006474] Initializing cgroup subsys perf_event
162 [ 0.007691] mce: CPU supports 10 MCE banks
163 [ 0.014557] ACPI: Core revision 20110623
164 [ 0.015614] ftrace: allocating 26600 entries in 105 pages
165 [ 0.016251] Enabling x2apic
166 [ 0.016261] Enabled x2apic
167 [ 0.016536] Switched APIC routing to physical x2apic.
168 [ 0.017618] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
169 [ 0.017620] CPU0: Intel Xeon E312xx (Sandy Bridge) stepping 01
170 [ 0.020001] Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only.
171 [ 0.020001] NMI watchdog disabled (cpu0): hardware events not enabled
172 [ 0.020001] Booting Node 0, Processors #1
173 [ 0.020001] smpboot cpu 1: start_ip = 9a000
174 [ 0.032017] kvm-clock: cpu 1, msr 2:3fc92681, secondary cpu clock
175 [ 0.032040] KVM setup async PF for cpu 1
176 [ 0.032040] kvm-stealtime: cpu 1, msr 23fc8cd40
177 [ 0.032055] NMI watchdog disabled (cpu1): hardware events not enabled
178 [ 0.032238] #2
179 [ 0.032239] smpboot cpu 2: start_ip = 9a000
180 [ 0.048039] KVM setup async PF for cpu 2
181 [ 0.048049] NMI watchdog disabled (cpu2): hardware events not enabled
182 [ 0.048039] kvm-stealtime: cpu 2, msr 23fd0cd40
183 [ 0.048039] kvm-clock: cpu 2, msr 2:3fd12681, secondary cpu clock
184 [ 0.048185] #3 Ok.
185 [ 0.048186] smpboot cpu 3: start_ip = 9a000
186 [ 0.064013] kvm-clock: cpu 3, msr 2:3fd92681, secondary cpu clock
187 [ 0.064031] KVM setup async PF for cpu 3
188 [ 0.064043] NMI watchdog disabled (cpu3): hardware events not enabled
189 [ 0.064031] kvm-stealtime: cpu 3, msr 23fd8cd40
190 [ 0.064106] Brought up 4 CPUs
191 [ 0.064108] Total of 4 processors activated (21545.98 BogoMIPS).
192 [ 0.064626] devtmpfs: initialized
193 [ 0.064776] EVM: security.selinux
194 [ 0.064777] EVM: security.SMACK64
195 [ 0.064778] EVM: security.capability
196 [ 0.064858] print_constraints: dummy:
197 [ 0.064860] RTC time: 3:17:04, date: 12/30/15
198 [ 0.068008] NET: Registered protocol family 16
199 [ 0.068140] ACPI: bus type pci registered
200 [ 0.068250] Trying to unpack rootfs image as initramfs...
201 [ 0.068190] PCI: Using configuration type 1 for base access
202 [ 0.080340] bio: create slab <bio-0> at 0
203 [ 0.080487] ACPI: Added _OSI(Module Device)
204 [ 0.080487] ACPI: Added _OSI(Processor Device)
205 [ 0.080487] ACPI: Added _OSI(3.0 _SCP Extensions)
206 [ 0.080487] ACPI: Added _OSI(Processor Aggregator Device)
207 [ 0.080607] ACPI: EC: Look up EC in DSDT
208 [ 0.081934] ACPI: Interpreter enabled
209 [ 0.081937] ACPI: (supports S0 S3 S4 S5)
210 [ 0.081949] ACPI: Using IOAPIC for interrupt routing
211 [ 0.084243] ACPI: No dock devices found.
212 [ 0.084245] HEST: Table not found.
213 [ 0.084248] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
214 [ 0.084303] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
215 [ 0.084384] pci_root PNP0A03:00: host bridge window [io 0x0000-0x0cf7]
216 [ 0.084386] pci_root PNP0A03:00: host bridge window [io 0x0d00-0xadff]
217 [ 0.084387] pci_root PNP0A03:00: host bridge window [io 0xae0f-0xaeff]
218 [ 0.084389] pci_root PNP0A03:00: host bridge window [io 0xaf20-0xafdf]
219 [ 0.084391] pci_root PNP0A03:00: host bridge window [io 0xafe4-0xffff]
220 [ 0.084392] pci_root PNP0A03:00: host bridge window [mem 0x000a0000-0x000bffff]
221 [ 0.084394] pci_root PNP0A03:00: host bridge window [mem 0xc0000000-0xfebfffff]
222 [ 0.084440] pci 0000:00:00.0: [8086:1237] type 0 class 0x000600
223 [ 0.084731] pci 0000:00:01.0: [8086:7000] type 0 class 0x000601
224 [ 0.085159] pci 0000:00:01.1: [8086:7010] type 0 class 0x000101
225 [ 0.090040] pci 0000:00:01.1: reg 20: [io 0xc0a0-0xc0af]
226 [ 0.091515] pci 0000:00:01.2: [8086:7020] type 0 class 0x000c03
227 [ 0.095076] pci 0000:00:01.2: reg 20: [io 0xc040-0xc05f]
228 [ 0.096443] pci 0000:00:01.3: [8086:7113] type 0 class 0x000680
229 [ 0.096805] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI
230 [ 0.096816] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB
231 [ 0.096959] pci 0000:00:02.0: [1013:00b8] type 0 class 0x000300
232 [ 0.100728] pci 0000:00:02.0: reg 10: [mem 0xfc000000-0xfdffffff pref]
233 [ 0.102137] pci 0000:00:02.0: reg 14: [mem 0xfebd0000-0xfebd0fff]
234 [ 0.109420] pci 0000:00:02.0: reg 30: [mem 0xfebc0000-0xfebcffff pref]
235 [ 0.109552] pci 0000:00:03.0: [1af4:1000] type 0 class 0x000200
236 [ 0.111468] pci 0000:00:03.0: reg 10: [io 0xc060-0xc07f]
237 [ 0.112631] pci 0000:00:03.0: reg 14: [mem 0xfebd1000-0xfebd1fff]
238 [ 0.118768] pci 0000:00:03.0: reg 30: [mem 0xfeb80000-0xfebbffff pref]
239 [ 0.119022] pci 0000:00:04.0: [1af4:1001] type 0 class 0x000100
240 [ 0.120656] pci 0000:00:04.0: reg 10: [io 0xc000-0xc03f]
241 [ 0.123278] pci 0000:00:04.0: reg 14: [mem 0xfebd2000-0xfebd2fff]
242 [ 0.130233] pci 0000:00:05.0: [1af4:1002] type 0 class 0x0000ff
243 [ 0.130940] pci 0000:00:05.0: reg 10: [io 0xc080-0xc09f]
244 [ 0.135853] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
245 [ 0.136157] pci0000:00: Unable to request _OSC control (_OSC support mask: 0x1e)
246 [ 0.137798] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
247 [ 0.137879] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
248 [ 0.137946] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
249 [ 0.138010] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
250 [ 0.138053] ACPI: PCI Interrupt Link [LNKS] (IRQs *9)
251 [ 0.138167] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
252 [ 0.138170] vgaarb: loaded
253 [ 0.138171] vgaarb: bridge control possible 0000:00:02.0
254 [ 0.138259] i2c-core: driver [aat2870] using legacy suspend method
255 [ 0.138261] i2c-core: driver [aat2870] using legacy resume method
256 [ 0.138319] SCSI subsystem initialized
257 [ 0.138459] libata version 3.00 loaded.
258 [ 0.138459] usbcore: registered new interface driver usbfs
259 [ 0.138459] usbcore: registered new interface driver hub
260 [ 0.138459] usbcore: registered new device driver usb
261 [ 0.138459] PCI: Using ACPI for IRQ routing
262 [ 0.138459] PCI: pci_cache_line_size set to 64 bytes
263 [ 0.138459] reserve RAM buffer: 000000000009fc00 - 000000000009ffff
264 [ 0.138459] reserve RAM buffer: 00000000bfffe000 - 00000000bfffffff
265 [ 0.138459] NetLabel: Initializing
266 [ 0.138459] NetLabel: domain hash size = 128
267 [ 0.138459] NetLabel: protocols = UNLABELED CIPSOv4
268 [ 0.138459] NetLabel: unlabeled traffic allowed by default
269 [ 0.138459] Switching to clocksource kvm-clock
270 [ 0.147661] AppArmor: AppArmor Filesystem Enabled
271 [ 0.147686] pnp: PnP ACPI init
272 [ 0.147701] ACPI: bus type pnp registered
273 [ 0.147807] pnp 00:00: [bus 00-ff]
274 [ 0.147810] pnp 00:00: [io 0x0cf8-0x0cff]
275 [ 0.147812] pnp 00:00: [io 0x0000-0x0cf7 window]
276 [ 0.147814] pnp 00:00: [io 0x0d00-0xadff window]
277 [ 0.147816] pnp 00:00: [io 0xae0f-0xaeff window]
278 [ 0.147818] pnp 00:00: [io 0xaf20-0xafdf window]
279 [ 0.147820] pnp 00:00: [io 0xafe4-0xffff window]
280 [ 0.147822] pnp 00:00: [mem 0x000a0000-0x000bffff window]
281 [ 0.147824] pnp 00:00: [mem 0xc0000000-0xfebfffff window]
282 [ 0.147879] pnp 00:00: Plug and Play ACPI device, IDs PNP0a03 (active)
283 [ 0.147896] pnp 00:01: [io 0x0070-0x0071]
284 [ 0.147950] pnp 00:01: [irq 8]
285 [ 0.147952] pnp 00:01: [io 0x0072-0x0077]
286 [ 0.147971] pnp 00:01: Plug and Play ACPI device, IDs PNP0b00 (active)
287 [ 0.147985] pnp 00:02: [io 0x0060]
288 [ 0.147986] pnp 00:02: [io 0x0064]
289 [ 0.148027] pnp 00:02: [irq 1]
290 [ 0.148048] pnp 00:02: Plug and Play ACPI device, IDs PNP0303 (active)
291 [ 0.148088] pnp 00:03: [irq 12]
292 [ 0.148106] pnp 00:03: Plug and Play ACPI device, IDs PNP0f13 (active)
293 [ 0.148125] pnp 00:04: [io 0x03f2-0x03f5]
294 [ 0.148126] pnp 00:04: [io 0x03f7]
295 [ 0.148152] pnp 00:04: [irq 6]
296 [ 0.148154] pnp 00:04: [dma 2]
297 [ 0.148170] pnp 00:04: Plug and Play ACPI device, IDs PNP0700 (active)
298 [ 0.148243] pnp 00:05: [io 0x03f8-0x03ff]
299 [ 0.148268] pnp 00:05: [irq 4]
300 [ 0.148288] pnp 00:05: Plug and Play ACPI device, IDs PNP0501 (active)
301 [ 0.148320] pnp 00:06: [io 0x02f8-0x02ff]
302 [ 0.148346] pnp 00:06: [irq 3]
303 [ 0.148365] pnp 00:06: Plug and Play ACPI device, IDs PNP0501 (active)
304 [ 0.148535] pnp 00:07: [io 0xaf00-0xaf1f]
305 [ 0.148558] pnp 00:07: Plug and Play ACPI device, IDs PNP0a06 (active)
306 [ 0.148562] pnp: PnP ACPI: found 8 devices
307 [ 0.148563] ACPI: ACPI bus type pnp unregistered
308 [ 0.154735] Freeing initrd memory: 5020k freed
309 [ 0.156475] PCI: max bus depth: 0 pci_try_num: 1
310 [ 0.156483] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7]
311 [ 0.156485] pci_bus 0000:00: resource 5 [io 0x0d00-0xadff]
312 [ 0.156487] pci_bus 0000:00: resource 6 [io 0xae0f-0xaeff]
313 [ 0.156488] pci_bus 0000:00: resource 7 [io 0xaf20-0xafdf]
314 [ 0.156490] pci_bus 0000:00: resource 8 [io 0xafe4-0xffff]
315 [ 0.156491] pci_bus 0000:00: resource 9 [mem 0x000a0000-0x000bffff]
316 [ 0.156493] pci_bus 0000:00: resource 10 [mem 0xc0000000-0xfebfffff]
317 [ 0.156522] NET: Registered protocol family 2
318 [ 0.158694] IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)
319 [ 0.160752] TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
320 [ 0.163141] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
321 [ 0.163705] TCP: Hash tables configured (established 524288 bind 65536)
322 [ 0.163707] TCP reno registered
323 [ 0.163718] UDP hash table entries: 4096 (order: 5, 131072 bytes)
324 [ 0.163743] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes)
325 [ 0.163820] NET: Registered protocol family 1
326 [ 0.163829] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
327 [ 0.163862] pci 0000:00:01.0: PIIX3: Enabling Passive Release
328 [ 0.163885] pci 0000:00:01.0: Activating ISA DMA hang workarounds
329 [ 0.164064] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
330 [ 0.164094] pci 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11
331 [ 0.164700] pci 0000:00:01.2: PCI INT D disabled
332 [ 0.164734] pci 0000:00:02.0: Boot video device
333 [ 0.164768] PCI: CLS 0 bytes, default 64
334 [ 0.164771] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
335 [ 0.164773] Placing 64MB software IO TLB between ffff8800bbffe000 - ffff8800bfffe000
336 [ 0.164774] software IO TLB at phys 0xbbffe000 - 0xbfffe000
337 [ 0.165378] audit: initializing netlink socket (disabled)
338 [ 0.165392] type=2000 audit(1451445425.164:1): initialized
339 [ 0.186266] HugeTLB registered 2 MB page size, pre-allocated 0 pages
340 [ 0.188600] VFS: Disk quotas dquot_6.5.2
341 [ 0.188644] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
342 [ 0.189119] fuse init (API version 7.17)
343 [ 0.189187] msgmni has been set to 15966
344 [ 0.190429] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
345 [ 0.190473] io scheduler noop registered
346 [ 0.190475] io scheduler deadline registered (default)
347 [ 0.190503] io scheduler cfq registered
348 [ 0.190584] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
349 [ 0.190597] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
350 [ 0.190701] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
351 [ 0.190707] ACPI: Power Button [PWRF]
352 [ 0.191626] ERST: Table is not found!
353 [ 0.191628] GHES: HEST is not enabled!
354 [ 0.191822] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10
355 [ 0.191855] virtio-pci 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 10 (level, high) -> IRQ 10
356 [ 0.192442] virtio-pci 0000:00:03.0: setting latency timer to 64
357 [ 0.192598] virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11
358 [ 0.193251] virtio-pci 0000:00:04.0: setting latency timer to 64
359 [ 0.193931] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
360 [ 0.193935] virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10
361 [ 0.194542] virtio-pci 0000:00:05.0: setting latency timer to 64
362 [ 0.194650] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
363 [ 0.224162] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
364 [ 0.288823] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
365 [ 0.481936] 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
366 [ 0.510491] 00:06: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
367 [ 0.512194] Linux agpgart interface v0.103
368 [ 0.513224] brd: module loaded
369 [ 0.513769] loop: module loaded
370 [ 0.513994] virtio-pci 0000:00:04.0: irq 40 for MSI/MSI-X
371 [ 0.514015] virtio-pci 0000:00:04.0: irq 41 for MSI/MSI-X
372 [ 0.523443] vda: vda1 vda2 vda3
373 [ 0.524714] ata_piix 0000:00:01.1: version 2.13
374 [ 0.525653] ata_piix 0000:00:01.1: setting latency timer to 64
375 [ 0.526604] scsi0 : ata_piix
376 [ 0.526784] scsi1 : ata_piix
377 [ 0.526817] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc0a0 irq 14
378 [ 0.526819] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc0a8 irq 15
379 [ 0.527095] Fixed MDIO Bus: probed
380 [ 0.527116] tun: Universal TUN/TAP device driver, 1.6
381 [ 0.527117] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
382 [ 0.527636] virtio-pci 0000:00:03.0: irq 42 for MSI/MSI-X
383 [ 0.527658] virtio-pci 0000:00:03.0: irq 43 for MSI/MSI-X
384 [ 0.527678] virtio-pci 0000:00:03.0: irq 44 for MSI/MSI-X
385 [ 0.530559] PPP generic driver version 2.4.2
386 [ 0.530738] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
387 [ 0.530754] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
388 [ 0.530762] uhci_hcd: USB Universal Host Controller Interface driver
389 [ 0.530796] uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11
390 [ 0.531908] uhci_hcd 0000:00:01.2: setting latency timer to 64
391 [ 0.531917] uhci_hcd 0000:00:01.2: UHCI Host Controller
392 [ 0.532022] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1
393 [ 0.532132] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c040
394 [ 0.532308] hub 1-0:1.0: USB hub found
395 [ 0.532311] hub 1-0:1.0: 2 ports detected
396 [ 0.532406] usbcore: registered new interface driver libusual
397 [ 0.532432] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
398 [ 0.533093] serio: i8042 KBD port at 0x60,0x64 irq 1
399 [ 0.533098] serio: i8042 AUX port at 0x60,0x64 irq 12
400 [ 0.533242] mousedev: PS/2 mouse device common for all mice
401 [ 0.533513] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1
402 [ 0.533595] rtc_cmos 00:01: RTC can wake from S4
403 [ 0.533948] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0
404 [ 0.534089] rtc0: alarms up to one day, 114 bytes nvram
405 [ 0.534149] device-mapper: uevent: version 1.0.3
406 [ 0.534272] device-mapper: ioctl: 4.22.0-ioctl (2011-10-19) initialised: dm-devel@redhat.com
407 [ 0.534279] cpuidle: using governor ladder
408 [ 0.534280] cpuidle: using governor menu
409 [ 0.534281] EFI Variables Facility v0.08 2004-May-17
410 [ 0.534438] TCP cubic registered
411 [ 0.534510] NET: Registered protocol family 10
412 [ 0.534913] NET: Registered protocol family 17
413 [ 0.534917] Registering the dns_resolver key type
414 [ 0.535155] PM: Hibernation image not present or could not be loaded.
415 [ 0.535166] registered taskstats version 1
416 [ 0.541159] Magic number: 11:327:261
417 [ 0.541341] rtc_cmos 00:01: setting system clock to 2015-12-30 03:17:05 UTC (1451445425)
418 [ 0.541390] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
419 [ 0.541392] EDD information not available.
420 [ 0.681539] Freeing unused kernel memory: 924k freed
421 [ 0.681782] Write protecting the kernel read-only data: 12288k
422 [ 0.685301] Freeing unused kernel memory: 1600k freed
423 [ 0.688812] Freeing unused kernel memory: 1184k freed
424 [ 0.705094] udevd[96]: starting version 175
425 [ 0.756510] FDC 0 is a S82078B
426 [ 0.833462] EXT4-fs (vda1): INFO: recovery required on readonly filesystem
427 [ 0.833465] EXT4-fs (vda1): write access will be enabled during recovery
428 [ 0.844052] usb 1-1: new full-speed USB device number 2 using uhci_hcd
429 [ 1.164095] Refined TSC clocksource calibration: 2693.240 MHz.
430 [ 1.942300] EXT4-fs (vda1): orphan cleanup on readonly fs
431 [ 1.951970] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 929480
432 [ 1.952277] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 929479
433 [ 1.952297] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 400782
434 [ 1.958008] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 917889
435 [ 1.967787] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 396189
436 [ 1.972165] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 396188
437 [ 1.972183] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 396190
438 [ 1.972195] EXT4-fs (vda1): ext4_orphan_cleanup: deleting unreferenced inode 396191
439 [ 1.972214] EXT4-fs (vda1): 8 orphan inodes deleted
440 [ 1.972216] EXT4-fs (vda1): recovery complete
441 [ 1.974820] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null)
442 [ 2.895874] EXT4-fs (vda1): re-mounted. Opts: (null)
443 [ 2.919695] Adding 499708k swap on /dev/vda2. Priority:-1 extents:1 across:499708k
444 [ 2.937313] Adding 24535036k swap on /dev/mapper/vd-big. Priority:-2 extents:1 across:24535036k
445 [ 3.136816] udevd[331]: starting version 175
446 [ 3.189244] init: portmap-wait (statd) main process (325) killed by TERM signal
447 [ 3.242560] init: statd-mounting main process (307) killed by TERM signal
448 [ 3.492225] type=1400 audit(1451445428.449:2): apparmor="STATUS" operation="profile_load" name="/sbin/dhclient" pid=434 comm="apparmor_parser"
449 [ 3.492563] type=1400 audit(1451445428.449:3): apparmor="STATUS" operation="profile_load" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=434 comm="apparmor_parser"
450 [ 3.492761] type=1400 audit(1451445428.449:4): apparmor="STATUS" operation="profile_load" name="/usr/lib/connman/scripts/dhclient-script" pid=434 comm="apparmor_parser"
451 [ 3.548350] type=1400 audit(1451445428.505:5): apparmor="STATUS" operation="profile_load" name="/usr/sbin/ntpd" pid=476 comm="apparmor_parser"
452 [ 3.685897] EXT4-fs (dm-1): recovery complete
453 [ 3.694116] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
454 [ 3.958484] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2
455 [ 4.006775] RPC: Registered named UNIX socket transport module.
456 [ 4.006779] RPC: Registered udp transport module.
457 [ 4.006780] RPC: Registered tcp transport module.
458 [ 4.006782] RPC: Registered tcp NFSv4.1 backchannel transport module.
459 [ 4.033055] FS-Cache: Loaded
460 [ 4.064119] FS-Cache: Netfs 'nfs' registered for caching
461 [ 7.204811] init: failsafe main process (760) killed by TERM signal
462 [ 7.210176] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
463 [ 7.240525] init: idmapd main process (790) terminated with status 1
464 [ 7.240548] init: idmapd main process ended, respawning
465 [ 7.331149] type=1400 audit(1451445432.285:6): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/ntpd" pid=837 comm="apparmor_parser"
466 [ 7.331287] type=1400 audit(1451445432.285:7): apparmor="STATUS" operation="profile_replace" name="/sbin/dhclient" pid=836 comm="apparmor_parser"
467 [ 7.331622] type=1400 audit(1451445432.285:8): apparmor="STATUS" operation="profile_replace" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=836 comm="apparmor_parser"
468 [ 7.331808] type=1400 audit(1451445432.285:9): apparmor="STATUS" operation="profile_replace" name="/usr/lib/connman/scripts/dhclient-script" pid=836 comm="apparmor_parser"
469 [ 7.353274] type=1400 audit(1451445432.309:10): apparmor="STATUS" operation="profile_load" name="/usr/sbin/tcpdump" pid=839 comm="apparmor_parser"
470 [ 7.880362] Bridge firewalling registered
471 [ 7.894156] 8021q: 802.1Q VLAN Support v1.8
472 [ 7.894177] 8021q: adding VLAN 0 to HW filter on device eth0
473 [ 10.888350] init: plymouth-upstart-bridge main process (811) killed by TERM signal
474 [ 14.576018] eth0: no IPv6 routers present
475 [ 2936.758668] php-cgi[4128]: segfault at 1000721 ip 00000000006ab997 sp 00007fff08af43a0 error 4 in php5-cgi[400000+74b000]
476 [53468.813845] php-cgi[14671]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
477 [139546.579818] php-cgi[22309]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
478 [174120.061485] hrtimer: interrupt took 13793356 ns
479 [195479.304553] php-cgi[29920]: segfault at 1000721 ip 00000000006ab997 sp 00007fff156437e0 error 4 in php5-cgi[400000+74b000]
480 [219963.651751] php-cgi[23093]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
481 [238130.849451] php-cgi[15386]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
482 [239775.415534] php-cgi[20460]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
483 [290128.864174] nfs: server labstore.svc.eqiad.wmnet not responding, still trying
484 [290128.867965] nfs: server labstore.svc.eqiad.wmnet OK
485 [300327.497694] php-cgi[9676]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
486 [347941.232681] php-cgi[21541]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
487 [350678.288449] php-cgi[24367]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
488 [420041.024272] php-cgi[31316]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
489 [432082.638819] php-cgi[14769]: segfault at 1000621 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
490 [432193.667944] php-cgi[22949]: segfault at 1000621 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
491 [444910.961794] php-cgi[12353]: segfault at 1000721 ip 00000000006ab997 sp 00007fff79363c40 error 4 in php5-cgi[400000+74b000]
492 [448131.642751] php-cgi[7679]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
493 [476250.508910] php-cgi[27147]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
494 [512354.037344] php-cgi[15095]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
495 [524779.346697] php-cgi[1551]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
496 [525429.749972] php-cgi[3971]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
497 [525495.779639] php-cgi[4257]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
498 [548444.180537] php-cgi[4644]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
499 [575872.931961] php-cgi[16528]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
500 [575959.483094] php-cgi[17344]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
501 [620666.166895] php-cgi[17290]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
502 [622293.978948] php-cgi[17276]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
503 [622403.307397] php-cgi[22679]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
504 [626034.716682] php-cgi[27751]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
505 [626140.964389] php-cgi[28068]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
506 [629346.579807] php-cgi[591]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
507 [631779.924371] php-cgi[3675]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
508 [644286.697451] php-cgi[18242]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d65f0 error 4 in libc-2.15.so[2b21cb637000+1b4000]
509 [644356.636084] php-cgi[26349]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d65f0 error 4 in libc-2.15.so[2b21cb637000+1b4000]
510 [656703.847339] php-cgi[5353]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
511 [666684.799412] php-cgi[25758]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d65f0 error 4 in libc-2.15.so[2b21cb637000+1b4000]
512 [666762.726962] php-cgi[28910]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d65f0 error 4 in libc-2.15.so[2b21cb637000+1b4000]
513 [667844.907121] php-cgi[28935]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
514 [669031.626629] php-cgi[23097]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
515 [669150.903987] php-cgi[31502]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
516 [670069.448186] php-cgi[23131]: segfault at 1000701 ip 00002b91093d9c0c sp 00007fff793626c0 error 4 in libc-2.15.so[2b910939f000+1b4000]
517 [670380.762753] php-cgi[31924]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
518 [672105.005058] php-cgi[2111]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
519 [672200.060559] php-cgi[22847]: segfault at 1000701 ip 00002b91093d9c0c sp 00007fff793626c0 error 4 in libc-2.15.so[2b910939f000+1b4000]
520 [675483.264637] php-cgi[4935]: segfault at 1000701 ip 00002b91093d9c0c sp 00007fff793626c0 error 4 in libc-2.15.so[2b910939f000+1b4000]
521 [692726.890311] php-cgi[4840]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
522 [757583.815531] php-cgi[4891]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
523 [757699.938674] php-cgi[16039]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
524 [800938.066527] php-cgi[13373]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
525 [837061.079669] php-cgi[28426]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
526 [851341.052673] php-cgi[19831]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
527 [873202.073199] php-cgi[18723]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
528 [909253.116276] php-cgi[10417]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
529 [912257.040402] php-cgi[12888]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
530 [912378.280526] php-cgi[10053]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
531 [1002188.171786] php-cgi[14065]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
532 [1011568.225294] php-cgi[29979]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
533 [1017467.376666] php-cgi[10938]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
534 [1066499.152850] php-cgi[15723]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
535 [1088549.355099] php-cgi[1211]: segfault at 1000701 ip 00002b91093d9c0c sp 00007fff793626c0 error 4 in libc-2.15.so[2b910939f000+1b4000]
536 [1088848.476838] php-cgi[18829]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
537 [1089158.709384] php-cgi[21949]: segfault at 1000701 ip 00002b91093d9c0c sp 00007fff793626c0 error 4 in libc-2.15.so[2b910939f000+1b4000]
538 [1089681.831204] php-cgi[24278]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
539 [1092392.417687] php-cgi[24622]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
540 [1092502.359006] php-cgi[21384]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
541 [1100448.513301] php-cgi[5745]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
542 [1114363.744175] php-cgi[27746]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
543 [1114459.239278] php-cgi[25111]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
544 [1117618.231736] php-cgi[30055]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
545 [1117698.442539] php-cgi[30082]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
546 [1122059.639156] php-cgi[5791]: segfault at 1000721 ip 00000000006ab997 sp 00007fff3c70ab60 error 4 in php5-cgi[400000+74b000]
547 [1125793.771330] php-cgi[24772]: segfault at 80001000321 ip 00000000006ab997 sp 00007fff79363c40 error 4 in php5-cgi[400000+74b000]
548 [1138597.494314] php-cgi[26707]: segfault at 80001000321 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
549 [1138712.628288] php-cgi[32623]: segfault at 80001000321 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
550 [1146112.797948] php-cgi[21280]: segfault at 1000621 ip 00000000006ab997 sp 00007fff3114f6f0 error 4 in php5-cgi[400000+74b000]
551 [1160937.060290] php-cgi[27984]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d65f0 error 4 in libc-2.15.so[2b21cb637000+1b4000]
552 [1161030.745013] php-cgi[30514]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d65f0 error 4 in libc-2.15.so[2b21cb637000+1b4000]
553 [1161905.045022] php-cgi[3235]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
554 [1164021.261110] php-cgi[3207]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
555 [1164177.432751] php-cgi[6237]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
556 [1168027.700249] php-cgi[648]: segfault at 1000701 ip 00002b91093d9c0c sp 00007fff793626c0 error 4 in libc-2.15.so[2b910939f000+1b4000]
557 [1168129.520325] php-cgi[3437]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
558 [1172285.544942] php-cgi[14427]: segfault at 1000701 ip 00002b91093d9c0c sp 00007fff793626c0 error 4 in libc-2.15.so[2b910939f000+1b4000]
559 [1175876.329221] php-cgi[17446]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
560 [1189273.420110] php-cgi[14675]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d65f0 error 4 in libc-2.15.so[2b21cb637000+1b4000]
561 [1189402.662682] php-cgi[9231]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d65f0 error 4 in libc-2.15.so[2b21cb637000+1b4000]
562 [1192378.562387] php-cgi[15143]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
563 [1192478.693317] php-cgi[20531]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
564 [1196153.034435] php-cgi[20578]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
565 [1196233.162343] php-cgi[26021]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
566 [1198044.797320] php-cgi[26136]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
567 [1198716.037040] php-cgi[8698]: segfault at 7fff3c70f001 ip 000000000072fe0c sp 00007fff3c7092e0 error 6 in php5-cgi[400000+74b000]
568 [1215911.943177] php-cgi[20189]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d65f0 error 4 in libc-2.15.so[2b21cb637000+1b4000]
569 [1234172.495639] php-cgi[18046]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
570 [1237981.896215] php-cgi[23442]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
571 [1238085.846152] php-cgi[23905]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
572 [1240522.616202] php-cgi[23883]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
573 [1240638.225184] php-cgi[16297]: segfault at 1000701 ip 00002b91093d9c0c sp 00007fff793626c0 error 4 in libc-2.15.so[2b910939f000+1b4000]
574 [1243792.870099] php-cgi[28992]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
575 [1243863.594307] php-cgi[31533]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
576 [1266674.783669] php-cgi[28756]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
577 [1274221.166275] php-cgi[24743]: segfault at 80001000321 ip 00000000006ab997 sp 00007fff79363c40 error 4 in php5-cgi[400000+74b000]
578 [1286276.049129] php-cgi[31116]: segfault at 80001000321 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
579 [1286359.862446] php-cgi[31143]: segfault at 80001000321 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
580 [1289205.136403] php-cgi[31160]: segfault at 80001000321 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
581 [1306371.761429] php-cgi[25621]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
582 [1308389.577910] type=1400 audit(1452753815.288:11): apparmor="STATUS" operation="profile_replace" name="/sbin/dhclient" pid=1585 comm="apparmor_parser"
583 [1308389.578057] type=1400 audit(1452753815.288:12): apparmor="STATUS" operation="profile_replace" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=1585 comm="apparmor_parser"
584 [1308389.578158] type=1400 audit(1452753815.288:13): apparmor="STATUS" operation="profile_replace" name="/usr/lib/connman/scripts/dhclient-script" pid=1585 comm="apparmor_parser"
585 [1309936.047987] php-cgi[28405]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
586 [1310064.560624] php-cgi[28439]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
587 [1310997.635236] php-cgi[17764]: segfault at 7fff3c70f001 ip 000000000072fe0c sp 00007fff3c7092e0 error 6 in php5-cgi[400000+74b000]
588 [1313070.616375] php-cgi[2233]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
589 [1313163.060517] php-cgi[2158]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
590 [1315019.847360] php-cgi[4591]: segfault at 1000721 ip 00000000006ab997 sp 00007fff3c70ab60 error 4 in php5-cgi[400000+74b000]
591 [1334781.741422] php-cgi[4406]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
592 [1378571.482006] php-cgi[30398]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
593 [1389424.688007] php-cgi[16783]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
594 [1392942.241115] php-cgi[24505]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
595 [1393062.786484] php-cgi[29787]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
596 [1414639.606954] php-cgi[27522]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
597 [1418790.980520] php-cgi[22379]: segfault at 1000701 ip 00002b91093d9c0c sp 00007fff793626c0 error 4 in libc-2.15.so[2b910939f000+1b4000]
598 [1447258.634919] php-cgi[27403]: segfault at 1000721 ip 00000000006ab997 sp 00007fff3c70ab60 error 4 in php5-cgi[400000+74b000]
599 [1486813.267343] php-cgi[4750]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
600 [1514204.356575] php-cgi[23818]: segfault at 1000721 ip 00000000006ab997 sp 00007fff3c70ab60 error 4 in php5-cgi[400000+74b000]
601 [1516742.957357] php-cgi[12943]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
602 [1516825.636815] php-cgi[18353]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
603 [1519598.960057] php-cgi[20749]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
604 [1522908.736949] php-cgi[23432]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
605 [1533448.379892] php-cgi[15525]: segfault at 7fff3c70f001 ip 000000000072fe0c sp 00007fff3c7092e0 error 6 in php5-cgi[400000+74b000]
606 [1537152.140213] php-cgi[12680]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
607 [1562012.327561] php-cgi[6509]: segfault at 7fff3c70f001 ip 000000000072fe0c sp 00007fff3c7092e0 error 6 in php5-cgi[400000+74b000]
608 [1565967.049632] php-cgi[22292]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
609 [1595022.260547] php-cgi[3080]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
610 [1623111.495594] php-cgi[10714]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
611 [1647261.956719] php-cgi[12795]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
612 [1647262.065589] php-cgi[17958]: segfault at 1000701 ip 00002b91093d9c0c sp 00007fff793626c0 error 4 in libc-2.15.so[2b910939f000+1b4000]
613 [1647362.914089] php-cgi[17972]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
614 [1651183.355985] php-cgi[20606]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
615 [1656708.349900] php-cgi[25790]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
616 [1687230.932884] php-cgi[9063]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
617 [1705292.668192] php-cgi[434]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
618 [1719869.063518] php-cgi[22073]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
619 [1719969.527141] php-cgi[27239]: segfault at 80001000321 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
620 [1722996.475524] php-cgi[27266]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
621 [1723104.503873] php-cgi[11306]: segfault at 1000721 ip 00000000006ab997 sp 00007fff79363c40 error 4 in php5-cgi[400000+74b000]
622 [1723331.816459] php-cgi[24515]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
623 [1745046.887606] php-cgi[26645]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
624 [1745119.560187] php-cgi[32056]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
625 [1748118.784763] php-cgi[2542]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
626 [1748190.890766] php-cgi[32073]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
627 [1750418.531413] php-cgi[4974]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
628 [1759410.723578] php-cgi[13369]: segfault at 1000701 ip 00002b91093d9c0c sp 00007fff793626c0 error 4 in libc-2.15.so[2b910939f000+1b4000]
629 [1771364.560906] php-cgi[4221]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
630 [1771467.407169] php-cgi[6904]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
631 [1793495.817149] php-cgi[7199]: segfault at 1000701 ip 00002b91093d9c0c sp 00007fff79360b40 error 4 in libc-2.15.so[2b910939f000+1b4000]
632 [1793599.748935] php-cgi[6966]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d65f0 error 4 in libc-2.15.so[2b21cb637000+1b4000]
633 [1795475.309218] php-cgi[9824]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
634 [1796064.724207] php-cgi[14367]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
635 [1798754.486805] php-cgi[10544]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
636 [1798825.411214] php-cgi[19884]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
637 [1808304.162852] php-cgi[22936]: segfault at 1000721 ip 00000000006ab997 sp 00007fff85536ca0 error 4 in php5-cgi[400000+74b000]
638 [1814770.619007] php-cgi[8847]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d65f0 error 4 in libc-2.15.so[2b21cb637000+1b4000]
639 [1814842.372966] php-cgi[3024]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d65f0 error 4 in libc-2.15.so[2b21cb637000+1b4000]
640 [1817183.390569] php-cgi[9103]: segfault at 1000621 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
641 [1817265.527446] php-cgi[9128]: segfault at 1000621 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]

[1817265.527446] php-cgi[9128]: segfault at 1000621 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]

ps awwlx --sort=vsz

P2490 ps awwlx --sort=vsz
1root@labcontrol1001:~# salt 'tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs' cmd.run 'ps awwlx --sort=vsz'
2tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs:
3 F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND
4 1 0 2 0 20 0 0 0 kthrea S ? 0:00 [kthreadd]
5 1 0 3 2 20 0 0 0 run_ks S ? 0:50 [ksoftirqd/0]
6 1 0 4 2 20 0 0 0 worker S ? 0:00 [kworker/0:0]
7 5 0 5 2 20 0 0 0 worker S ? 0:00 [kworker/u:0]
8 1 0 6 2 -100 - 0 0 cpu_st S ? 0:04 [migration/0]
9 5 0 7 2 -100 - 0 0 watchd S ? 0:11 [watchdog/0]
10 1 0 8 2 -100 - 0 0 cpu_st S ? 0:04 [migration/1]
11 1 0 10 2 20 0 0 0 run_ks S ? 4:03 [ksoftirqd/1]
12 5 0 11 2 20 0 0 0 worker S ? 3:11 [kworker/0:1]
13 5 0 12 2 -100 - 0 0 watchd S ? 0:11 [watchdog/1]
14 1 0 13 2 -100 - 0 0 cpu_st S ? 0:04 [migration/2]
15 1 0 15 2 20 0 0 0 run_ks S ? 0:49 [ksoftirqd/2]
16 5 0 16 2 -100 - 0 0 watchd S ? 0:10 [watchdog/2]
17 1 0 17 2 -100 - 0 0 cpu_st S ? 0:03 [migration/3]
18 1 0 18 2 20 0 0 0 worker S ? 0:00 [kworker/3:0]
19 1 0 19 2 20 0 0 0 run_ks S ? 0:40 [ksoftirqd/3]
20 5 0 20 2 -100 - 0 0 watchd S ? 0:09 [watchdog/3]
21 1 0 21 2 0 -20 0 0 rescue S< ? 0:00 [cpuset]
22 1 0 22 2 0 -20 0 0 rescue S< ? 0:00 [khelper]
23 5 0 23 2 20 0 0 0 devtmp S ? 0:00 [kdevtmpfs]
24 1 0 24 2 0 -20 0 0 rescue S< ? 0:00 [netns]
25 1 0 25 2 20 0 0 0 worker S ? 0:00 [kworker/u:1]
26 1 0 26 2 20 0 0 0 bdi_sy S ? 0:05 [sync_supers]
27 1 0 27 2 20 0 0 0 bdi_fo S ? 0:00 [bdi-default]
28 1 0 28 2 0 -20 0 0 rescue S< ? 0:00 [kintegrityd]
29 1 0 29 2 0 -20 0 0 rescue S< ? 0:00 [kblockd]
30 1 0 30 2 0 -20 0 0 rescue S< ? 0:00 [ata_sff]
31 1 0 31 2 20 0 0 0 hub_th S ? 0:00 [khubd]
32 1 0 32 2 0 -20 0 0 rescue S< ? 0:00 [md]
33 5 0 33 2 20 0 0 0 worker S ? 1:25 [kworker/2:1]
34 1 0 35 2 20 0 0 0 watchd S ? 0:01 [khungtaskd]
35 1 0 36 2 20 0 0 0 kswapd S ? 0:03 [kswapd0]
36 1 0 37 2 25 5 0 0 ksm_sc SN ? 0:00 [ksmd]
37 1 0 38 2 39 19 0 0 khugep SN ? 0:00 [khugepaged]
38 1 0 39 2 20 0 0 0 fsnoti S ? 0:00 [fsnotify_mark]
39 1 0 40 2 20 0 0 0 ecrypt S ? 0:00 [ecryptfs-kthrea]
40 1 0 41 2 0 -20 0 0 rescue S< ? 0:00 [crypto]
41 1 0 49 2 0 -20 0 0 rescue S< ? 0:00 [kthrotld]
42 1 0 50 2 20 0 0 0 scsi_e S ? 0:00 [scsi_eh_0]
43 1 0 51 2 20 0 0 0 scsi_e S ? 0:00 [scsi_eh_1]
44 1 0 72 2 0 -20 0 0 rescue S< ? 0:00 [devfreq_wq]
45 5 0 214 2 20 0 0 0 worker S ? 48:15 [kworker/1:1]
46 5 0 215 2 20 0 0 0 worker S ? 2:28 [kworker/3:1]
47 1 0 221 2 0 -20 0 0 rescue S< ? 0:00 [kdmflush]
48 1 0 228 2 0 -20 0 0 rescue S< ? 0:00 [kdmflush]
49 1 0 236 2 20 0 0 0 kjourn S ? 0:41 [jbd2/vda1-8]
50 1 0 237 2 0 -20 0 0 rescue S< ? 0:00 [ext4-dio-unwrit]
51 1 0 458 2 20 0 0 0 balloo S ? 1:54 [vballoon]
52 1 0 474 2 0 -20 0 0 rescue S< ? 0:00 [kpsmoused]
53 1 0 487 2 20 0 0 0 kjourn S ? 0:05 [jbd2/dm-1-8]
54 1 0 488 2 0 -20 0 0 rescue S< ? 0:00 [ext4-dio-unwrit]
55 1 0 661 2 0 -20 0 0 rescue S< ? 0:00 [rpciod]
56 1 0 678 2 0 -20 0 0 rescue S< ? 0:00 [nfsiod]
57 1 0 682 2 20 0 0 0 svc_re S ? 0:00 [nfsv4.0-svc]
58 1 0 684 2 20 0 0 0 bdi_wr S ? 0:35 [flush-253:0]
59 1 0 10707 2 20 0 0 0 bdi_wr S ? 0:01 [flush-252:1]
60 1 0 20311 2 20 0 0 0 worker S ? 0:00 [kworker/1:2]
61 1 51044 22338 1 20 0 0 0 rpc_wa D ? 0:00 [php-cgi]
62 1 51044 22554 1 20 0 0 0 rpc_wa D ? 0:00 [lighttpd]
63 5 0 31252 2 20 0 0 0 worker S ? 0:47 [kworker/2:0]
64 0 0 23533 23531 20 0 4296 604 poll_s S ? 0:00 run-parts --report /etc/cron.daily
65 0 51103 1058 939 20 0 4392 604 wait S ? 0:00 /bin/sh /data/project/sdbot/server.sh
66 4 0 407 406 20 0 4396 608 wait Ss ? 0:00 /bin/sh -c [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete
67 0 0 417 416 20 0 4396 608 wait S ? 0:00 /bin/sh -c ps awwlx --sort=vsz
68 4 0 23302 23301 20 0 4396 608 wait Ss ? 0:00 /bin/sh -c [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete
69 4 0 23531 23529 20 0 4396 612 wait Ss ? 0:00 /bin/sh -c test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
70 4 0 24358 23533 20 0 4396 612 wait S ? 0:00 /bin/sh /etc/cron.daily/exim4-base
71 4 0 24393 24392 20 0 4396 612 wait Ss ? 0:00 /bin/sh -c [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete
72 4 0 25138 25137 20 0 4396 604 wait Ss ? 0:00 /bin/sh -c [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete
73 4 0 25571 25570 20 0 4396 608 wait Ss ? 0:00 /bin/sh -c [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete
74 4 0 26347 26345 20 0 4396 608 wait Ss ? 0:00 /bin/sh -c [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete
75 4 0 26872 26869 20 0 4396 612 wait Ss ? 0:00 /bin/sh -c [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete
76 4 0 27678 27677 20 0 4396 608 wait Ss ? 0:00 /bin/sh -c [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete
77 4 0 28098 28097 20 0 4396 612 wait Ss ? 0:00 /bin/sh -c [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete
78 4 0 28843 28842 20 0 4396 612 wait Ss ? 0:00 /bin/sh -c [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete
79 4 0 29276 29275 20 0 4396 608 wait Ss ? 0:00 /bin/sh -c [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete
80 4 0 30021 30020 20 0 4396 608 wait Ss ? 0:00 /bin/sh -c [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete
81 4 0 30454 30453 20 0 4396 612 wait Ss ? 0:00 /bin/sh -c [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete
82 4 0 31198 31197 20 0 4396 612 wait Ss ? 0:00 /bin/sh -c [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete
83 4 0 31619 31618 20 0 4396 608 wait Ss ? 0:00 /bin/sh -c [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete
84 4 0 32362 32361 20 0 4396 608 wait Ss ? 0:00 /bin/sh -c [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete
85 4 0 418 417 20 0 6836 780 - R ? 0:00 ps awwlx --sort=vsz
86 1 0 966 1 20 0 6988 384 poll_s Ss ? 0:00 /usr/sbin/mcelog --daemon
87 4 0 1305 1 20 0 7332 716 n_tty_ Ss+ ttyS0 0:00 /sbin/getty -L 115200 ttyS0 xterm
88 5 0 1066 1 20 0 8768 524 poll_s Ss ? 0:00 /usr/sbin/inetd
89 4 0 24364 24363 20 0 10336 612 rpc_wa D ? 0:00 start-stop-daemon --start --exec /usr/sbin/exim_tidydb --chuid Debian-exim:Debian-exim -- /var/spool/exim4 wait-remote_smtp
90 1 1 887 1 20 0 10572 352 hrtime Ss ? 0:00 atd
91 1 0 888 1 20 0 12776 932 hrtime Ss ? 0:07 cron
92 1 0 608 1 20 0 15032 1084 poll_s Ss ? 0:00 dhclient3 -e IF_METRIC=100 -pf /var/run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases -1 eth0
93 5 0 465 331 20 0 15184 832 ep_pol S ? 0:00 /sbin/udevd --daemon
94 1 0 568 1 20 0 15184 384 poll_s S ? 0:00 upstart-socket-bridge --daemon
95 5 0 464 331 20 0 15204 852 ep_pol S ? 0:00 /sbin/udevd --daemon
96 5 0 331 1 20 0 15208 1264 ep_pol Ss ? 0:00 /sbin/udevd --daemon
97 5 0 909 1 20 0 15976 700 hrtime Ss ? 4:07 /usr/sbin/irqbalance
98 0 51063 1035 1 20 0 16972 1280 rpc_wa D ? 11:42 /usr/lib/gamin/gam_server
99 0 51103 1063 1 20 0 16972 1288 rpc_wa D ? 2:06 /usr/lib/gamin/gam_server
100 0 51184 6997 1 20 0 16972 1284 rpc_wa D ? 5:16 /usr/lib/gamin/gam_server
101 0 52421 7042 1 20 0 16972 1284 rpc_wa D ? 7:39 /usr/lib/gamin/gam_server
102 0 52299 7091 1 20 0 16972 1288 rpc_wa D ? 9:44 /usr/lib/gamin/gam_server
103 0 52017 7127 1 20 0 16972 1288 rpc_wa D ? 34:08 /usr/lib/gamin/gam_server
104 0 52437 7220 1 20 0 16972 1284 rpc_wa D ? 44:15 /usr/lib/gamin/gam_server
105 0 52324 7473 1 20 0 16972 1288 rpc_wa D ? 29:49 /usr/lib/gamin/gam_server
106 0 51391 7571 1 20 0 16972 1292 rpc_wa D ? 5:07 /usr/lib/gamin/gam_server
107 0 51471 7573 1 20 0 16972 1284 rpc_wa D ? 1:15 /usr/lib/gamin/gam_server
108 0 52347 7596 1 20 0 16972 1288 rpc_wa D ? 1:14 /usr/lib/gamin/gam_server
109 0 51580 8661 1 20 0 16972 1280 rpc_wa D ? 55:25 /usr/lib/gamin/gam_server
110 0 51141 9513 1 20 0 16972 1284 rpc_wa D ? 78:19 /usr/lib/gamin/gam_server
111 0 52196 9645 1 20 0 16972 1280 rpc_wa D ? 4:52 /usr/lib/gamin/gam_server
112 0 52465 9772 1 20 0 16972 1288 rpc_wa D ? 7:09 /usr/lib/gamin/gam_server
113 0 52249 9817 1 20 0 16972 1284 rpc_wa D ? 48:21 /usr/lib/gamin/gam_server
114 0 51792 9851 1 20 0 16972 1280 rpc_wa D ? 11:18 /usr/lib/gamin/gam_server
115 0 52309 9883 1 20 0 16972 1284 rpc_wa D ? 7:39 /usr/lib/gamin/gam_server
116 0 51396 9928 1 20 0 16972 1284 rpc_wa D ? 45:06 /usr/lib/gamin/gam_server
117 0 51962 9971 1 20 0 16972 1284 rpc_wa D ? 5:24 /usr/lib/gamin/gam_server
118 0 51122 10050 1 20 0 16972 1284 rpc_wa D ? 7:15 /usr/lib/gamin/gam_server
119 0 51419 11810 1 20 0 16972 1288 rpc_wa D ? 3:54 /usr/lib/gamin/gam_server
120 0 52508 11815 1 20 0 16972 1284 rpc_wa D ? 1:46 /usr/lib/gamin/gam_server
121 0 52406 11825 1 20 0 16972 1288 rpc_wa D ? 2:04 /usr/lib/gamin/gam_server
122 0 52450 11837 1 20 0 16972 1280 rpc_wa D ? 3:08 /usr/lib/gamin/gam_server
123 0 51177 12611 1 20 0 16972 1288 rpc_wa D ? 0:02 /usr/lib/gamin/gam_server
124 0 51591 19776 1 20 0 16972 1344 rpc_wa D ? 0:48 /usr/lib/gamin/gam_server
125 0 51559 21287 1 20 0 16972 1284 rpc_wa D ? 8:32 /usr/lib/gamin/gam_server
126 0 51605 21467 1 20 0 16972 1288 poll_s S ? 0:00 /usr/lib/gamin/gam_server
127 0 51044 22344 1 20 0 16972 1288 poll_s S ? 0:00 /usr/lib/gamin/gam_server
128 0 51512 10009 1 20 0 17084 1356 rpc_wa D ? 38:40 /usr/lib/gamin/gam_server
129 0 51585 1007 1 20 0 17092 1388 rpc_wa D ? 137:34 /usr/lib/gamin/gam_server
130 1 0 320 1 20 0 17228 636 poll_s S ? 0:00 upstart-udev-bridge --daemon
131 0 51058 6738 1 20 0 17316 1624 rpc_wa D ? 719:19 /usr/lib/gamin/gam_server
132 5 0 338 1 20 0 19196 1008 poll_s Ss ? 0:03 rpcbind -w
133 5 102 804 1 20 0 19580 684 poll_s Ss ? 0:00 dbus-daemon --system --fork --activation=upstart
134 4 0 24287 1 0 -20 20044 7764 pause S<L ? 0:15 /usr/bin/atop -a -w /var/log/atop.log 600
135 4 0 1 0 20 0 20064 2224 poll_s Ss ? 0:07 /sbin/init
136 4 600 933 1105 20 0 22548 1988 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
137 4 600 937 1105 20 0 22548 1996 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
138 4 600 940 1105 20 0 22548 1988 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
139 4 600 6703 1105 20 0 22548 2000 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
140 4 600 6956 1105 20 0 22548 1996 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
141 4 600 7011 1105 20 0 22548 2008 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
142 4 600 7056 1105 20 0 22548 2000 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
143 4 600 7092 1105 20 0 22548 1992 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
144 4 600 7189 1105 20 0 22548 2000 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
145 4 600 7379 1105 20 0 22548 1992 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
146 4 600 7437 1105 20 0 22548 2000 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
147 4 600 7440 1105 20 0 22548 2000 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
148 4 600 7443 1105 20 0 22548 1996 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
149 4 600 7774 1105 20 0 22548 1992 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
150 4 600 8615 1105 20 0 22548 2000 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
151 4 600 9482 1105 20 0 22548 2004 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
152 4 600 9614 1105 20 0 22548 2008 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
153 4 600 9741 1105 20 0 22548 2008 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
154 4 600 9786 1105 20 0 22548 1992 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
155 4 600 9820 1105 20 0 22548 1992 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
156 4 600 9852 1105 20 0 22548 1996 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
157 4 600 9897 1105 20 0 22548 1996 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
158 4 600 9940 1105 20 0 22548 1992 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
159 4 600 9974 1105 20 0 22548 1996 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
160 4 600 10021 1105 20 0 22548 2000 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
161 4 600 11665 1105 20 0 22548 2004 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
162 4 600 11668 1105 20 0 22548 1992 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
163 4 600 11671 1105 20 0 22548 1996 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
164 4 600 11692 1105 20 0 22548 1996 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
165 4 600 12576 1105 20 0 22548 1996 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
166 4 600 21249 1105 20 0 22548 1988 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
167 4 600 21867 1105 20 0 22548 2000 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
168 4 600 22309 1105 20 0 22548 2112 wait S ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
169 5 110 981 1 20 0 23340 1108 poll_s Ss ? 4:30 /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d
170 5 51044 22578 22309 20 0 25572 1188 rpc_wa Ds ? 0:00 /usr/lib/gridengine/sge_shepherd -bg
171 5 105 367 1 20 0 27336 1980 poll_s Ss ? 0:00 rpc.statd -L
172 4 0 413 412 20 0 31124 828 rpc_wa D ? 0:00 fuser -s ./sess_j3ab2crkijve95l1e8kkv6bej5
173 4 0 23308 23307 20 0 31124 824 rpc_wa D ? 0:00 fuser -s ./sess_j3ab2crkijve95l1e8kkv6bej5
174 4 0 24399 24398 20 0 31124 824 rpc_wa D ? 0:00 fuser -s ./sess_j3ab2crkijve95l1e8kkv6bej5
175 4 0 25144 25143 20 0 31124 824 rpc_wa D ? 0:00 fuser -s ./sess_j3ab2crkijve95l1e8kkv6bej5
176 4 0 25577 25576 20 0 31124 824 rpc_wa D ? 0:00 fuser -s ./sess_j3ab2crkijve95l1e8kkv6bej5
177 4 0 26355 26354 20 0 31124 824 rpc_wa D ? 0:00 fuser -s ./sess_j3ab2crkijve95l1e8kkv6bej5
178 4 0 26879 26878 20 0 31124 828 rpc_wa D ? 0:00 fuser -s ./sess_j3ab2crkijve95l1e8kkv6bej5
179 4 0 27684 27683 20 0 31124 824 rpc_wa D ? 0:00 fuser -s ./sess_j3ab2crkijve95l1e8kkv6bej5
180 4 0 28104 28103 20 0 31124 824 rpc_wa D ? 0:00 fuser -s ./sess_j3ab2crkijve95l1e8kkv6bej5
181 4 0 28849 28848 20 0 31124 824 rpc_wa D ? 0:00 fuser -s ./sess_j3ab2crkijve95l1e8kkv6bej5
182 4 0 29282 29281 20 0 31124 824 rpc_wa D ? 0:00 fuser -s ./sess_j3ab2crkijve95l1e8kkv6bej5
183 4 0 30027 30026 20 0 31124 820 rpc_wa D ? 0:00 fuser -s ./sess_j3ab2crkijve95l1e8kkv6bej5
184 4 0 30460 30459 20 0 31124 820 rpc_wa D ? 0:00 fuser -s ./sess_j3ab2crkijve95l1e8kkv6bej5
185 4 0 31204 31203 20 0 31124 824 rpc_wa D ? 0:00 fuser -s ./sess_j3ab2crkijve95l1e8kkv6bej5
186 4 0 31625 31624 20 0 31124 828 rpc_wa D ? 0:00 fuser -s ./sess_j3ab2crkijve95l1e8kkv6bej5
187 4 0 32368 32367 20 0 31124 824 rpc_wa D ? 0:00 fuser -s ./sess_j3ab2crkijve95l1e8kkv6bej5
188 0 0 24363 24358 20 0 31396 608 wait S ? 0:00 xargs -0r -n 1 start-stop-daemon --start --exec /usr/sbin/exim_tidydb --chuid Debian-exim:Debian-exim -- /var/spool/exim4
189 4 0 866 1 20 0 33404 880 n_tty_ Ss+ tty4 0:00 /sbin/getty -8 38400 tty4
190 4 0 872 1 20 0 33404 876 n_tty_ Ss+ tty5 0:00 /sbin/getty -8 38400 tty5
191 4 0 878 1 20 0 33404 880 n_tty_ Ss+ tty2 0:00 /sbin/getty -8 38400 tty2
192 4 0 879 1 20 0 33404 880 n_tty_ Ss+ tty3 0:00 /sbin/getty -8 38400 tty3
193 4 0 882 1 20 0 33404 876 n_tty_ Ss+ tty6 0:00 /sbin/getty -8 38400 tty6
194 4 0 1304 1 20 0 34320 816 n_tty_ Ss+ tty1 0:00 /sbin/getty -8 38400 tty1
195 1 0 406 888 20 0 35860 1188 wait S ? 0:00 CRON
196 1 0 23301 888 20 0 35860 1188 wait S ? 0:00 CRON
197 1 0 23529 888 20 0 35860 1184 wait S ? 0:00 CRON
198 1 0 24392 888 20 0 35860 1188 wait S ? 0:00 CRON
199 1 0 25137 888 20 0 35860 1188 wait S ? 0:00 CRON
200 1 0 25570 888 20 0 35860 1188 wait S ? 0:00 CRON
201 1 0 26345 888 20 0 35860 1184 wait S ? 0:00 CRON
202 1 0 26869 888 20 0 35860 1184 wait S ? 0:00 CRON
203 1 0 27677 888 20 0 35860 1188 wait S ? 0:00 CRON
204 1 0 28097 888 20 0 35860 1188 wait S ? 0:00 CRON
205 1 0 28842 888 20 0 35860 1188 wait S ? 0:00 CRON
206 1 0 29275 888 20 0 35860 1188 wait S ? 0:00 CRON
207 1 0 30020 888 20 0 35860 1188 wait S ? 0:00 CRON
208 1 0 30453 888 20 0 35860 1188 wait S ? 0:00 CRON
209 1 0 31197 888 20 0 35860 1188 wait S ? 0:00 CRON
210 1 0 31618 888 20 0 35860 1188 wait S ? 0:00 CRON
211 1 0 32361 888 20 0 35860 1188 wait S ? 0:00 CRON
212 0 0 412 407 20 0 38908 1000 wait S ? 0:00 find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +24 ! -execdir fuser -s {} ; -delete
213 0 0 23307 23302 20 0 38908 1000 wait S ? 0:00 find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +24 ! -execdir fuser -s {} ; -delete
214 0 0 24398 24393 20 0 38908 1000 wait S ? 0:00 find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +24 ! -execdir fuser -s {} ; -delete
215 0 0 25143 25138 20 0 38908 1000 wait S ? 0:00 find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +24 ! -execdir fuser -s {} ; -delete
216 0 0 25576 25571 20 0 38908 996 wait S ? 0:00 find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +24 ! -execdir fuser -s {} ; -delete
217 0 0 26354 26347 20 0 38908 996 wait S ? 0:00 find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +24 ! -execdir fuser -s {} ; -delete
218 0 0 26878 26872 20 0 38908 1000 wait S ? 0:00 find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +24 ! -execdir fuser -s {} ; -delete
219 0 0 27683 27678 20 0 38908 996 wait S ? 0:00 find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +24 ! -execdir fuser -s {} ; -delete
220 0 0 28103 28098 20 0 38908 996 wait S ? 0:00 find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +24 ! -execdir fuser -s {} ; -delete
221 0 0 28848 28843 20 0 38908 996 wait S ? 0:00 find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +24 ! -execdir fuser -s {} ; -delete
222 0 0 29281 29276 20 0 38908 996 wait S ? 0:00 find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +24 ! -execdir fuser -s {} ; -delete
223 0 0 30026 30021 20 0 38908 996 wait S ? 0:00 find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +24 ! -execdir fuser -s {} ; -delete
224 0 0 30459 30454 20 0 38908 996 wait S ? 0:00 find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +24 ! -execdir fuser -s {} ; -delete
225 0 0 31203 31198 20 0 38908 996 wait S ? 0:00 find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +24 ! -execdir fuser -s {} ; -delete
226 0 0 31624 31619 20 0 38908 996 wait S ? 0:00 find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +24 ! -execdir fuser -s {} ; -delete
227 0 0 32367 32362 20 0 38908 996 wait S ? 0:00 find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +24 ! -execdir fuser -s {} ; -delete
228 5 111 732 1 20 0 39360 2832 poll_s Ss ? 3:14 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 111:114
229 5 104 960 959 20 0 42804 676 poll_s S ? 0:39 /usr/sbin/lldpd
230 1 33 1276 1 20 0 44332 1048 ep_pol S ? 1:22 /usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf
231 5 107 936 1 20 0 45392 1148 poll_s Ss ? 0:01 /usr/sbin/exim4 -oP /var/run/exim4/eximqr.pid -q10m
232 5 0 959 1 20 0 47044 1528 wait_f Ss ? 1:45 /usr/sbin/lldpd
233 4 52299 7058 7056 20 0 52072 2796 rpc_wa Ds ? 1:36 /usr/sbin/lighttpd -f /var/run/lighttpd/alex.conf -D
234 4 51605 21869 21867 20 0 52072 2552 rpc_wa Ds ? 0:00 /usr/sbin/lighttpd -f /var/run/lighttpd/jembot.conf -D
235 4 52437 7191 7189 20 0 52076 2500 rpc_wa Ds ? 1:35 /usr/sbin/lighttpd -f /var/run/lighttpd/extreg-wos.conf -D
236 4 52347 7439 7437 20 0 52076 2492 rpc_wa Ds ? 0:22 /usr/sbin/lighttpd -f /var/run/lighttpd/dioceses.conf -D
237 4 52465 9743 9741 20 0 52076 2720 rpc_wa Ds ? 1:33 /usr/sbin/lighttpd -f /var/run/lighttpd/family.conf -D
238 4 52309 9854 9852 20 0 52076 2504 rpc_wa Ds ? 1:32 /usr/sbin/lighttpd -f /var/run/lighttpd/fawikiauto.conf -D
239 4 51962 9942 9940 20 0 52076 2732 rpc_wa Ds ? 1:35 /usr/sbin/lighttpd -f /var/run/lighttpd/anno.conf -D
240 4 51122 10023 10021 20 0 52076 2724 ep_pol Ss ? 1:31 /usr/sbin/lighttpd -f /var/run/lighttpd/simplewikt.conf -D
241 4 52406 11667 11665 20 0 52076 2728 rpc_wa Ds ? 0:58 /usr/sbin/lighttpd -f /var/run/lighttpd/slumpartikel.conf -D
242 4 52508 11673 11671 20 0 52076 2504 rpc_wa Ds ? 0:41 /usr/sbin/lighttpd -f /var/run/lighttpd/grillitus.conf -D
243 4 51103 939 937 20 0 52152 2920 rpc_wa Ds ? 1:37 /usr/sbin/lighttpd -f /var/run/lighttpd/sdbot.conf -D
244 4 52017 7094 7092 20 0 52168 2844 ep_pol Ss ? 1:35 /usr/sbin/lighttpd -f /var/run/lighttpd/tilde.conf -D
245 4 51391 7449 7443 20 0 52180 2800 rpc_wa Ds ? 0:23 /usr/sbin/lighttpd -f /var/run/lighttpd/mp.conf -D
246 4 52421 7013 7011 20 0 52188 2804 rpc_wa Ds ? 1:36 /usr/sbin/lighttpd -f /var/run/lighttpd/commons-delinquent.conf -D
247 4 51471 7442 7440 20 0 52192 2912 rpc_wa Ds ? 0:23 /usr/sbin/lighttpd -f /var/run/lighttpd/ia-upload.conf -D
248 4 52249 9788 9786 20 0 52196 2772 rpc_wa Ds ? 2:03 /usr/sbin/lighttpd -f /var/run/lighttpd/igloo.conf -D
249 4 51396 9899 9897 20 0 52196 2920 rpc_wa Ds ? 1:33 /usr/sbin/lighttpd -f /var/run/lighttpd/amdb.conf -D
250 4 51177 12578 12576 20 0 52196 2960 rpc_wa Ds ? 0:00 /usr/sbin/lighttpd -f /var/run/lighttpd/dewikinews-rss.conf -D
251 4 52196 9616 9614 20 0 52208 2852 rpc_wa Ds ? 1:32 /usr/sbin/lighttpd -f /var/run/lighttpd/inactiveadmins.conf -D
252 4 51141 9484 9482 20 0 52216 2732 rpc_wa Ds ? 1:34 /usr/sbin/lighttpd -f /var/run/lighttpd/purodha-testing.conf -D
253 4 51585 935 933 20 0 52300 2936 rpc_wa Ds ? 5:48 /usr/sbin/lighttpd -f /var/run/lighttpd/potd-feed.conf -D
254 4 51792 9822 9820 20 0 52320 3036 rpc_wa Ds ? 1:31 /usr/sbin/lighttpd -f /var/run/lighttpd/wikitrip.conf -D
255 4 51063 942 940 20 0 52332 3156 rpc_wa Ds ? 1:33 /usr/sbin/lighttpd -f /var/run/lighttpd/wikilint.conf -D
256 4 51419 11670 11668 20 0 52412 3084 rpc_wa Ds ? 0:54 /usr/sbin/lighttpd -f /var/run/lighttpd/ipp.conf -D
257 4 52324 7381 7379 20 0 52460 3172 rpc_wa Ds ? 1:35 /usr/sbin/lighttpd -f /var/run/lighttpd/powow.conf -D
258 4 51184 6958 6956 20 0 52532 3268 rpc_wa Ds ? 9:51 /usr/sbin/lighttpd -f /var/run/lighttpd/catfood.conf -D
259 4 0 886 1 20 0 52568 2956 poll_s Ss ? 0:01 /usr/sbin/sshd -D
260 4 51580 8617 8615 20 0 52572 3256 rpc_wa Ds ? 1:39 /usr/sbin/lighttpd -f /var/run/lighttpd/hroest.conf -D
261 4 51591 7776 7774 20 0 52580 2980 rpc_wa Ds ? 0:09 /usr/sbin/lighttpd -f /var/run/lighttpd/pltools.conf -D
262 4 51058 6705 6703 20 0 53044 3796 rpc_wa Ds ? 1:44 /usr/sbin/lighttpd -f /var/run/lighttpd/betacommand-dev.conf -D
263 4 51559 21251 21249 20 0 53980 4716 rpc_wa Ds ? 1:22 /usr/sbin/lighttpd -f /var/run/lighttpd/quentinv57-tools.conf -D
264 4 52450 11695 11692 20 0 54380 3052 ep_pol Ss ? 2:49 /usr/sbin/lighttpd -f /var/run/lighttpd/cgstat.conf -D
265 5 106 23179 23178 20 0 62672 1336 unix_s S ? 0:00 sshd: scfc [net]
266 5 106 23700 23699 20 0 62672 1340 unix_s S ? 0:00 sshd: scfc [net]
267 5 106 25015 25014 20 0 62672 1332 unix_s S ? 0:00 sshd: scfc [net]
268 5 106 25448 25447 20 0 62672 1336 unix_s S ? 0:00 sshd: scfc [net]
269 5 106 26199 26198 20 0 62672 1336 unix_s S ? 0:00 sshd: scfc [net]
270 5 106 26723 26722 20 0 62672 1336 unix_s S ? 0:00 sshd: scfc [net]
271 5 106 27555 27554 20 0 62672 1336 unix_s S ? 0:00 sshd: scfc [net]
272 5 106 27988 27987 20 0 62672 1336 unix_s S ? 0:00 sshd: scfc [net]
273 5 106 28720 28719 20 0 62672 1336 unix_s S ? 0:00 sshd: scfc [net]
274 5 106 29153 29152 20 0 62672 1336 unix_s S ? 0:00 sshd: scfc [net]
275 5 106 29898 29897 20 0 62672 1336 unix_s S ? 0:00 sshd: scfc [net]
276 5 106 30331 30330 20 0 62672 1336 unix_s S ? 0:00 sshd: scfc [net]
277 5 106 31009 31008 20 0 62672 1336 unix_s S ? 0:00 sshd: tools.phetools [net]
278 5 106 31509 31508 20 0 62672 1340 unix_s S ? 0:00 sshd: scfc [net]
279 5 106 32686 32685 20 0 62672 1336 unix_s S ? 0:00 sshd: scfc [net]
280 5 600 1105 1 20 0 62768 8076 rpc_wa Dl ? 515:38 /usr/lib/gridengine/sge_execd
281 4 2267 23178 886 20 0 97832 4096 rpc_wa Ds ? 0:00 sshd: scfc [priv]
282 4 2267 23699 886 20 0 97832 4100 rpc_wa Ds ? 0:00 sshd: scfc [priv]
283 4 2267 25014 886 20 0 97832 4100 rpc_wa Ds ? 0:00 sshd: scfc [priv]
284 4 2267 25447 886 20 0 97832 4100 rpc_wa Ds ? 0:00 sshd: scfc [priv]
285 4 2267 26198 886 20 0 97832 4096 rpc_wa Ds ? 0:00 sshd: scfc [priv]
286 4 2267 26722 886 20 0 97832 4096 rpc_wa Ds ? 0:00 sshd: scfc [priv]
287 4 2267 27554 886 20 0 97832 4096 rpc_wa Ds ? 0:00 sshd: scfc [priv]
288 4 2267 27987 886 20 0 97832 4096 rpc_wa Ds ? 0:00 sshd: scfc [priv]
289 4 2267 28719 886 20 0 97832 4096 rpc_wa Ds ? 0:00 sshd: scfc [priv]
290 4 2267 29152 886 20 0 97832 4100 rpc_wa Ds ? 0:00 sshd: scfc [priv]
291 4 2267 29897 886 20 0 97832 4104 rpc_wa Ds ? 0:00 sshd: scfc [priv]
292 4 2267 30330 886 20 0 97832 4092 rpc_wa Ds ? 0:00 sshd: scfc [priv]
293 4 52004 31008 886 20 0 97832 4100 rpc_wa Ds ? 0:00 sshd: tools.phetools [priv]
294 4 2267 31508 886 20 0 97832 4100 rpc_wa Ds ? 0:00 sshd: scfc [priv]
295 4 2267 32685 886 20 0 97832 4100 rpc_wa Ds ? 0:00 sshd: scfc [priv]
296 4 0 32713 886 20 0 101284 4284 rpc_wa Ds ? 0:00 sshd: rush [priv]
297 4 51512 9976 9974 20 0 108080 5620 rpc_wa Ds ? 13:07 /usr/sbin/lighttpd -f /var/run/lighttpd/wikihistory.conf -D
298 5 101 802 1 20 0 247752 5152 poll_s Sl ? 2:21 rsyslogd -c5
299 0 51585 975 935 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
300 0 51585 995 935 20 0 314696 13876 wait Ss ? 0:00 /usr/bin/php-cgi
301 0 51063 1001 942 20 0 314696 13856 wait Ss ? 0:00 /usr/bin/php-cgi
302 1 51063 1004 1001 20 0 314696 5320 wait_f S ? 0:00 /usr/bin/php-cgi
303 1 51063 1005 1001 20 0 314696 5320 wait_f S ? 0:00 /usr/bin/php-cgi
304 0 51063 1008 942 20 0 314696 13856 wait Ss ? 0:00 /usr/bin/php-cgi
305 1 51063 1017 1008 20 0 314696 5320 wait_f S ? 0:00 /usr/bin/php-cgi
306 1 51063 1024 1008 20 0 314696 5320 wait_f S ? 0:00 /usr/bin/php-cgi
307 0 51103 1050 939 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
308 1 51103 1052 1050 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
309 1 51103 1053 1050 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
310 0 51103 1054 939 20 0 314696 13856 wait Ss ? 0:00 /usr/bin/php-cgi
311 1 51103 1056 1054 20 0 314696 5320 wait_f S ? 0:00 /usr/bin/php-cgi
312 1 51103 1057 1054 20 0 314696 5320 wait_f S ? 0:00 /usr/bin/php-cgi
313 0 51058 6729 6705 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
314 1 51058 6731 6729 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
315 1 51058 6732 6729 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
316 0 51058 6733 6705 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
317 1 51058 6735 6733 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
318 1 51058 6736 6733 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
319 0 51184 6988 6958 20 0 314696 13876 wait Ss ? 0:00 /usr/bin/php-cgi
320 0 51184 6992 6958 20 0 314696 13872 wait Ss ? 0:01 /usr/bin/php-cgi
321 0 52421 7033 7013 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
322 0 52421 7037 7013 20 0 314696 13848 wait Ss ? 0:00 /usr/bin/php-cgi
323 0 52299 7082 7058 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
324 1 52299 7084 7082 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
325 1 52299 7085 7082 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
326 0 52299 7086 7058 20 0 314696 13856 wait Ss ? 0:00 /usr/bin/php-cgi
327 1 52299 7088 7086 20 0 314696 5320 wait_f S ? 0:00 /usr/bin/php-cgi
328 1 52299 7089 7086 20 0 314696 5320 wait_f S ? 0:00 /usr/bin/php-cgi
329 0 52017 7118 7094 20 0 314696 13856 wait Ss ? 0:00 /usr/bin/php-cgi
330 1 52017 7120 7118 20 0 314696 5320 wait_f S ? 0:00 /usr/bin/php-cgi
331 1 52017 7121 7118 20 0 314696 5320 wait_f S ? 0:00 /usr/bin/php-cgi
332 0 52017 7122 7094 20 0 314696 13856 wait Ss ? 0:00 /usr/bin/php-cgi
333 1 52017 7124 7122 20 0 314696 5320 wait_f S ? 0:00 /usr/bin/php-cgi
334 1 52017 7125 7122 20 0 314696 5320 wait_f S ? 0:00 /usr/bin/php-cgi
335 0 52437 7211 7191 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
336 1 52437 7213 7211 20 0 314696 5320 wait_f S ? 0:00 /usr/bin/php-cgi
337 1 52437 7214 7211 20 0 314696 5320 wait_f S ? 0:00 /usr/bin/php-cgi
338 0 52437 7215 7191 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
339 1 52437 7217 7215 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
340 1 52437 7218 7215 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
341 0 52324 7464 7381 20 0 314696 13856 wait Ss ? 0:00 /usr/bin/php-cgi
342 1 52324 7466 7464 20 0 314696 5320 wait_f S ? 0:00 /usr/bin/php-cgi
343 1 52324 7467 7464 20 0 314696 5320 wait_f S ? 0:00 /usr/bin/php-cgi
344 0 52324 7468 7381 20 0 314696 13856 wait Ss ? 0:00 /usr/bin/php-cgi
345 0 51391 7551 7449 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
346 0 51471 7556 7442 20 0 314696 13856 wait Ss ? 0:00 /usr/bin/php-cgi
347 1 51391 7557 7551 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
348 1 51391 7558 7551 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
349 0 51391 7562 7449 20 0 314696 13856 wait Ss ? 0:00 /usr/bin/php-cgi
350 0 51471 7566 7442 20 0 314696 13848 wait Ss ? 0:00 /usr/bin/php-cgi
351 0 52347 7587 7439 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
352 1 52347 7589 7587 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
353 1 52347 7590 7587 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
354 0 52347 7591 7439 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
355 1 52347 7593 7591 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
356 1 52347 7594 7591 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
357 0 51591 7844 7776 20 0 314696 13856 wait Ss ? 0:00 /usr/bin/php-cgi
358 0 51591 7858 7776 20 0 314696 13872 wait Ss ? 0:00 /usr/bin/php-cgi
359 0 51580 8640 8617 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
360 0 51580 8649 8617 20 0 314696 13848 wait Ss ? 0:00 /usr/bin/php-cgi
361 0 51141 9504 9484 20 0 314696 13856 wait Ss ? 0:00 /usr/bin/php-cgi
362 1 51141 9506 9504 20 0 314696 5320 wait_f S ? 0:00 /usr/bin/php-cgi
363 1 51141 9507 9504 20 0 314696 5320 wait_f S ? 0:00 /usr/bin/php-cgi
364 0 51141 9508 9484 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
365 0 52196 9636 9616 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
366 1 52196 9638 9636 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
367 1 52196 9639 9636 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
368 0 52196 9640 9616 20 0 314696 13856 wait Ss ? 0:00 /usr/bin/php-cgi
369 1 52196 9642 9640 20 0 314696 5320 wait_f S ? 0:00 /usr/bin/php-cgi
370 1 52196 9643 9640 20 0 314696 5320 wait_f S ? 0:00 /usr/bin/php-cgi
371 0 52465 9763 9743 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
372 1 52465 9765 9763 20 0 314696 6704 wait_f S ? 0:00 /usr/bin/php-cgi
373 1 52465 9766 9763 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
374 0 52465 9767 9743 20 0 314696 13856 wait Ss ? 0:00 /usr/bin/php-cgi
375 0 52249 9808 9788 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
376 1 52249 9810 9808 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
377 1 52249 9811 9808 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
378 0 52249 9812 9788 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
379 1 52249 9814 9812 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
380 1 52249 9815 9812 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
381 0 51792 9842 9822 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
382 0 51792 9846 9822 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
383 0 52309 9874 9854 20 0 314696 13848 wait Ss ? 0:00 /usr/bin/php-cgi
384 1 52309 9876 9874 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
385 1 52309 9877 9874 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
386 0 52309 9878 9854 20 0 314696 13856 wait Ss ? 0:00 /usr/bin/php-cgi
387 0 51396 9919 9899 20 0 314696 13856 wait Ss ? 0:00 /usr/bin/php-cgi
388 1 51396 9922 9919 20 0 314696 5320 wait_f S ? 0:00 /usr/bin/php-cgi
389 0 51396 9923 9899 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
390 0 51962 9962 9942 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
391 1 51962 9964 9962 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
392 1 51962 9965 9962 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
393 0 51962 9966 9942 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
394 0 51512 10000 9976 20 0 314696 13876 wait Ss ? 0:00 /usr/bin/php-cgi
395 0 51512 10004 9976 20 0 314696 13876 wait Ss ? 0:01 /usr/bin/php-cgi
396 0 51122 10041 10023 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
397 1 51122 10043 10041 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
398 1 51122 10044 10041 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
399 0 51122 10045 10023 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
400 0 51419 11732 11670 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
401 0 52508 11738 11673 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
402 1 52508 11763 11738 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
403 1 52508 11766 11738 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
404 0 52406 11777 11667 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
405 0 51419 11783 11670 20 0 314696 13872 wait Ss ? 0:00 /usr/bin/php-cgi
406 0 52508 11787 11673 20 0 314696 13856 wait Ss ? 0:00 /usr/bin/php-cgi
407 1 52508 11799 11787 20 0 314696 5320 wait_f S ? 0:00 /usr/bin/php-cgi
408 1 52508 11802 11787 20 0 314696 5320 wait_f S ? 0:00 /usr/bin/php-cgi
409 0 52450 11807 11695 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
410 0 52406 11808 11667 20 0 314696 13872 wait Ss ? 0:00 /usr/bin/php-cgi
411 1 52450 11812 11807 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
412 1 52450 11814 11807 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
413 0 52450 11823 11695 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
414 1 52450 11827 11823 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
415 1 52450 11828 11823 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
416 0 51177 12602 12578 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
417 0 51177 12606 12578 20 0 314696 13848 wait Ss ? 0:00 /usr/bin/php-cgi
418 0 51559 21277 21251 20 0 314696 13856 wait Ss ? 0:00 /usr/bin/php-cgi
419 0 51559 21282 21251 20 0 314696 13872 wait Ss ? 0:00 /usr/bin/php-cgi
420 0 51605 21894 21869 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
421 1 51605 21896 21894 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
422 1 51605 21897 21894 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
423 0 51605 21898 21869 20 0 314696 13852 wait Ss ? 0:00 /usr/bin/php-cgi
424 1 51605 21900 21898 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
425 1 51605 21901 21898 20 0 314696 5316 wait_f S ? 0:00 /usr/bin/php-cgi
426 1 52465 9769 9767 20 0 315208 6752 wait_f S ? 0:00 /usr/bin/php-cgi
427 1 52465 9770 9767 20 0 315208 6752 wait_f S ? 0:00 /usr/bin/php-cgi
428 1 51122 10047 10045 20 0 315468 7312 wait_f S ? 0:00 /usr/bin/php-cgi
429 1 51122 10048 10045 20 0 315468 7312 wait_f S ? 0:00 /usr/bin/php-cgi
430 1 51585 5686 975 20 0 315472 8304 wait_f S ? 0:01 /usr/bin/php-cgi
431 1 51962 9968 9966 20 0 315576 7496 wait_f S ? 0:00 /usr/bin/php-cgi
432 1 51962 9969 9966 20 0 315576 7568 wait_f S ? 0:00 /usr/bin/php-cgi
433 1 51585 3136 975 20 0 315584 8184 wait_f S ? 0:01 /usr/bin/php-cgi
434 1 51585 12193 995 20 0 315584 8172 wait_f S ? 0:01 /usr/bin/php-cgi
435 1 51585 12366 995 20 0 315584 8348 wait_f S ? 0:01 /usr/bin/php-cgi
436 1 51396 9921 9919 20 0 315816 8244 wait_f S ? 0:00 /usr/bin/php-cgi
437 1 51559 28210 21277 20 0 315888 9800 wait_f S ? 0:08 /usr/bin/php-cgi
438 1 51141 9511 9508 20 0 315992 9372 wait_f S ? 0:03 /usr/bin/php-cgi
439 1 51559 19724 21282 20 0 316048 10128 wait_f S ? 0:07 /usr/bin/php-cgi
440 1 52406 9576 11808 20 0 316300 7764 wait_f S ? 0:00 /usr/bin/php-cgi
441 1 52406 9579 11808 20 0 316300 7776 wait_f S ? 0:00 /usr/bin/php-cgi
442 1 52406 11791 11777 20 0 316300 7780 wait_f S ? 0:00 /usr/bin/php-cgi
443 1 52406 11793 11777 20 0 316300 7776 wait_f S ? 0:00 /usr/bin/php-cgi
444 1 51396 9925 9923 20 0 316408 8308 wait_f S ? 0:01 /usr/bin/php-cgi
445 1 51396 9926 9923 20 0 316408 8316 wait_f S ? 0:01 /usr/bin/php-cgi
446 1 52309 9880 9878 20 0 316592 9496 wait_f S ? 0:00 /usr/bin/php-cgi
447 1 51177 12609 12606 20 0 316672 9372 wait_f S ? 2:35 /usr/bin/php-cgi
448 1 51591 19804 7858 20 0 316836 10088 rpc_wa D ? 0:08 /usr/bin/php-cgi
449 1 51591 21398 7844 20 0 316836 10052 wait_f S ? 0:00 /usr/bin/php-cgi
450 1 51591 21412 7844 20 0 316836 10024 wait_f S ? 0:00 /usr/bin/php-cgi
451 1 51591 22097 7858 20 0 316836 10020 wait_f S ? 0:00 /usr/bin/php-cgi
452 1 51141 9510 9508 20 0 317236 10032 wait_f S ? 0:03 /usr/bin/php-cgi
453 1 51580 8643 8640 20 0 317436 9928 wait_f S ? 0:25 /usr/bin/php-cgi
454 1 51580 8654 8649 20 0 317436 9608 wait_f S ? 2:53 /usr/bin/php-cgi
455 1 51177 12605 12602 20 0 317724 9492 wait_f S ? 0:13 /usr/bin/php-cgi
456 1 51177 12604 12602 20 0 317980 9728 wait_f S ? 0:14 /usr/bin/php-cgi
457 1 51177 12608 12606 20 0 317980 9696 wait_f S ? 1:56 /usr/bin/php-cgi
458 1 51580 8653 8649 20 0 318204 10536 wait_f S ? 2:50 /usr/bin/php-cgi
459 1 51559 26570 21282 20 0 318352 11708 wait_f S ? 0:20 /usr/bin/php-cgi
460 1 51419 10717 11732 20 0 319680 8372 sock_w S ? 0:02 /usr/bin/php-cgi
461 1 51419 17951 11732 20 0 319704 8472 wait_f S ? 0:11 /usr/bin/php-cgi
462 1 51419 23117 11783 20 0 319704 8364 wait_f S ? 0:01 /usr/bin/php-cgi
463 1 51419 23930 11783 20 0 319704 8352 sock_w S ? 0:01 /usr/bin/php-cgi
464 1 52421 7035 7033 20 0 320360 9436 wait_f S ? 0:00 /usr/bin/php-cgi
465 1 52421 7039 7037 20 0 320360 9552 wait_f S ? 0:08 /usr/bin/php-cgi
466 1 52421 7040 7037 20 0 320360 9540 wait_f S ? 0:08 /usr/bin/php-cgi
467 1 52421 7036 7033 20 0 320376 9464 wait_f S ? 0:00 /usr/bin/php-cgi
468 1 51559 20806 21277 20 0 320776 14944 wait_f S ? 0:22 /usr/bin/php-cgi
469 1 51184 12678 6992 20 0 320972 11980 wait_f S ? 0:19 /usr/bin/php-cgi
470 1 51580 8644 8640 20 0 321532 14104 wait_f S ? 0:47 /usr/bin/php-cgi
471 1 51184 3104 6988 20 0 322380 13032 wait_f S ? 0:13 /usr/bin/php-cgi
472 1 51184 27491 6988 20 0 322428 13016 wait_f S ? 0:20 /usr/bin/php-cgi
473 1 51391 7565 7562 20 0 322680 14632 wait_f S ? 0:13 /usr/bin/php-cgi
474 1 51184 12653 6992 20 0 322908 13628 wait_f S ? 0:19 /usr/bin/php-cgi
475 1 51391 7564 7562 20 0 322944 15012 wait_f S ? 0:12 /usr/bin/php-cgi
476 1 51792 9849 9846 20 0 326256 21508 wait_f S ? 2:13 /usr/bin/php-cgi
477 1 51471 7561 7556 20 0 326660 21256 wait_f S ? 0:01 /usr/bin/php-cgi
478 1 51792 9848 9846 20 0 327536 22788 wait_f S ? 1:35 /usr/bin/php-cgi
479 1 51792 9844 9842 20 0 329056 24068 wait_f S ? 0:20 /usr/bin/php-cgi
480 1 51792 9845 9842 20 0 329056 24136 wait_f S ? 0:13 /usr/bin/php-cgi
481 1 52324 7470 7468 20 0 333052 23832 wait_f S ? 0:11 /usr/bin/php-cgi
482 1 52324 7471 7468 20 0 333068 23832 wait_f S ? 0:12 /usr/bin/php-cgi
483 1 51471 7560 7556 20 0 333648 28136 wait_f S ? 0:02 /usr/bin/php-cgi
484 1 52309 9881 9878 20 0 343428 9592 wait_f S ? 0:00 /usr/bin/php-cgi
485 1 51512 6414 10000 20 0 346520 8856 wait_f S ? 0:00 /usr/bin/php-cgi
486 1 51512 3302 10004 20 0 347368 9928 wait_f S ? 0:02 /usr/bin/php-cgi
487 1 51512 5766 10004 20 0 347472 10032 wait_f S ? 0:02 /usr/bin/php-cgi
488 1 51512 21947 10000 20 0 347496 9876 wait_f S ? 0:00 /usr/bin/php-cgi
489 1 51471 7569 7566 20 0 385204 79768 wait_f S ? 1:09 /usr/bin/php-cgi
490 1 51471 7568 7566 20 0 390832 85288 wait_f S ? 1:13 /usr/bin/php-cgi
491 5 103 1039 1 20 0 448304 6952 hrtime Ssl ? 4:19 /usr/sbin/nslcd
492 0 51103 1061 1058 20 0 454280 26132 poll_s Sl ? 57:06 python -m src.server
493 0 52450 11833 11695 20 0 475248 24960 poll_s Sl ? 150:23 /usr/bin/python /data/project/cgstat/public_html/cgstat.fcgi
494 4 109 748 1 20 0 584232 25600 poll_s Ssl ? 30:42 /usr/bin/python /usr/bin/diamond --foreground --skip-fork --skip-pidfile
495 1 0 416 1 20 0 602800 40508 pipe_w S ? 0:00 /usr/bin/python /usr/bin/salt-minion
496 0 0 874 1 20 0 602800 48664 poll_s Ssl ? 15:11 /usr/bin/python /usr/bin/salt-minion
497 1 0 1003 1 20 0 964936 3896 ep_pol Ssl ? 4:24 /usr/sbin/nscd

top 2

P2491 tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs top 2
1root@labcontrol1001:~# salt 'tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs' cmd.run 'top -n 1 -b'
2tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs:
3 top - 13:48:01 up 21 days, 10:30, 0 users, load average: 105.00, 104.50, 103.0
4 Tasks: 499 total, 1 running, 498 sleeping, 0 stopped, 0 zombie
5 Cpu(s): 34.4%us, 3.0%sy, 0.0%ni, 61.7%id, 0.1%wa, 0.0%hi, 0.4%si, 0.5%st
6 Mem: 8178496k total, 5939988k used, 2238508k free, 248664k buffers
7 Swap: 25034744k total, 0k used, 25034744k free, 3513192k cached
8
9 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10 965 root 20 0 12176 1460 852 R 4 0.0 0:00.03 top
11 1 root 20 0 20064 2224 1284 S 0 0.0 0:07.85 init
12 2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
13 3 root 20 0 0 0 0 S 0 0.0 0:50.51 ksoftirqd/0
14 4 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/0:0
15 5 root 20 0 0 0 0 S 0 0.0 0:00.08 kworker/u:0
16 6 root RT 0 0 0 0 S 0 0.0 0:04.11 migration/0
17 7 root RT 0 0 0 0 S 0 0.0 0:11.51 watchdog/0
18 8 root RT 0 0 0 0 S 0 0.0 0:04.01 migration/1
19 10 root 20 0 0 0 0 S 0 0.0 4:03.32 ksoftirqd/1
20 11 root 20 0 0 0 0 S 0 0.0 3:11.59 kworker/0:1
21 12 root RT 0 0 0 0 S 0 0.0 0:11.46 watchdog/1
22 13 root RT 0 0 0 0 S 0 0.0 0:04.94 migration/2
23 15 root 20 0 0 0 0 S 0 0.0 0:49.39 ksoftirqd/2
24 16 root RT 0 0 0 0 S 0 0.0 0:10.07 watchdog/2
25 17 root RT 0 0 0 0 S 0 0.0 0:03.62 migration/3
26 18 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/3:0
27 19 root 20 0 0 0 0 S 0 0.0 0:40.27 ksoftirqd/3
28 20 root RT 0 0 0 0 S 0 0.0 0:09.34 watchdog/3
29 21 root 0 -20 0 0 0 S 0 0.0 0:00.00 cpuset
30 22 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper
31 23 root 20 0 0 0 0 S 0 0.0 0:00.00 kdevtmpfs
32 24 root 0 -20 0 0 0 S 0 0.0 0:00.00 netns
33 25 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/u:1
34 26 root 20 0 0 0 0 S 0 0.0 0:05.61 sync_supers
35 27 root 20 0 0 0 0 S 0 0.0 0:00.30 bdi-default
36 28 root 0 -20 0 0 0 S 0 0.0 0:00.00 kintegrityd
37 29 root 0 -20 0 0 0 S 0 0.0 0:00.00 kblockd
38 30 root 0 -20 0 0 0 S 0 0.0 0:00.00 ata_sff
39 31 root 20 0 0 0 0 S 0 0.0 0:00.00 khubd
40 32 root 0 -20 0 0 0 S 0 0.0 0:00.00 md
41 33 root 20 0 0 0 0 S 0 0.0 1:25.59 kworker/2:1
42 35 root 20 0 0 0 0 S 0 0.0 0:01.62 khungtaskd
43 36 root 20 0 0 0 0 S 0 0.0 0:03.19 kswapd0
44 37 root 25 5 0 0 0 S 0 0.0 0:00.00 ksmd
45 38 root 39 19 0 0 0 S 0 0.0 0:00.00 khugepaged
46 39 root 20 0 0 0 0 S 0 0.0 0:00.00 fsnotify_mark
47 40 root 20 0 0 0 0 S 0 0.0 0:00.00 ecryptfs-kthrea
48 41 root 0 -20 0 0 0 S 0 0.0 0:00.00 crypto
49 49 root 0 -20 0 0 0 S 0 0.0 0:00.00 kthrotld
50 50 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_0
51 51 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_1
52 72 root 0 -20 0 0 0 S 0 0.0 0:00.00 devfreq_wq
53 214 root 20 0 0 0 0 S 0 0.0 48:15.09 kworker/1:1
54 215 root 20 0 0 0 0 S 0 0.0 2:28.45 kworker/3:1
55 221 root 0 -20 0 0 0 S 0 0.0 0:00.00 kdmflush
56 228 root 0 -20 0 0 0 S 0 0.0 0:00.00 kdmflush
57 236 root 20 0 0 0 0 S 0 0.0 0:41.19 jbd2/vda1-8
58 237 root 0 -20 0 0 0 S 0 0.0 0:00.00 ext4-dio-unwrit
59 320 root 20 0 17228 636 456 S 0 0.0 0:00.06 upstart-udev-br
60 331 root 20 0 15208 1264 740 S 0 0.0 0:00.05 udevd
61 338 root 20 0 19196 1008 724 S 0 0.0 0:03.73 rpcbind
62 367 statd 20 0 27336 1980 1288 S 0 0.0 0:00.00 rpc.statd
63 406 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
64 407 root 20 0 4396 608 508 S 0 0.0 0:00.00 sh
65 412 root 20 0 38908 1000 852 S 0 0.0 0:00.00 find
66 413 root 20 0 31124 828 680 D 0 0.0 0:00.30 fuser
67 454 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
68 455 root 20 0 4396 608 504 S 0 0.0 0:00.00 sh
69 456 root 20 0 36264 1460 1248 S 0 0.0 0:00.00 puppet-run
70 458 root 20 0 0 0 0 S 0 0.0 1:54.55 vballoon
71 464 root 20 0 15204 852 324 S 0 0.0 0:00.00 udevd
72 465 root 20 0 15184 832 324 S 0 0.0 0:00.00 udevd
73 474 root 0 -20 0 0 0 S 0 0.0 0:00.00 kpsmoused
74 487 root 20 0 0 0 0 S 0 0.0 0:05.64 jbd2/dm-1-8
75 488 root 0 -20 0 0 0 S 0 0.0 0:00.00 ext4-dio-unwrit
76 503 root 20 0 35380 660 544 S 0 0.0 0:00.00 timeout
77 504 root 20 0 175m 88m 4104 D 0 1.1 0:06.96 puppet
78 568 root 20 0 15184 384 196 S 0 0.0 0:00.00 upstart-socket-
79 608 root 20 0 15032 1084 544 S 0 0.0 0:00.07 dhclient3
80 661 root 0 -20 0 0 0 S 0 0.0 0:00.00 rpciod
81 678 root 0 -20 0 0 0 S 0 0.0 0:00.00 nfsiod
82 682 root 20 0 0 0 0 S 0 0.0 0:00.00 nfsv4.0-svc
83 684 root 20 0 0 0 0 S 0 0.0 0:35.29 flush-253:0
84 732 ntp 20 0 39360 2832 1972 S 0 0.0 3:15.01 ntpd
85 748 diamond 20 0 570m 25m 5300 S 0 0.3 30:43.00 diamond
86 802 syslog 20 0 241m 5152 1068 S 0 0.1 2:21.28 rsyslogd
87 804 messageb 20 0 19580 684 452 S 0 0.0 0:00.02 dbus-daemon
88 866 root 20 0 33404 880 748 S 0 0.0 0:00.00 getty
89 872 root 20 0 33404 876 748 S 0 0.0 0:00.00 getty
90 874 root 20 0 588m 47m 9528 S 0 0.6 15:11.73 salt-minion
91 878 root 20 0 33404 880 748 S 0 0.0 0:00.00 getty
92 879 root 20 0 33404 880 748 S 0 0.0 0:00.00 getty
93 882 root 20 0 33404 876 748 S 0 0.0 0:00.00 getty
94 886 root 20 0 52568 2956 2336 S 0 0.0 0:01.21 sshd
95 887 daemon 20 0 10572 352 220 S 0 0.0 0:00.05 atd
96 888 root 20 0 12776 932 724 S 0 0.0 0:07.11 cron
97 909 root 20 0 15976 700 524 S 0 0.0 4:07.61 irqbalance
98 933 sgeadmin 20 0 22548 1988 1544 S 0 0.0 0:00.00 sge_shepherd
99 935 tools.po 20 0 52300 2936 1576 D 0 0.0 5:48.38 lighttpd
100 936 Debian-e 20 0 45392 1148 732 S 0 0.0 0:01.98 exim4
101 937 sgeadmin 20 0 22548 1996 1544 S 0 0.0 0:00.00 sge_shepherd
102 939 tools.sd 20 0 52152 2920 1684 D 0 0.0 1:37.51 lighttpd
103 940 sgeadmin 20 0 22548 1988 1544 S 0 0.0 0:00.00 sge_shepherd
104 942 tools.wi 20 0 52332 3156 1660 D 0 0.0 1:33.50 lighttpd
105 959 root 20 0 47044 1528 944 S 0 0.0 1:45.40 lldpd
106 960 _lldpd 20 0 42804 676 204 S 0 0.0 0:39.56 lldpd
107 963 root 20 0 588m 39m 1372 S 0 0.5 0:00.00 salt-minion
108 964 root 20 0 4396 612 508 S 0 0.0 0:00.00 sh
109 966 root 20 0 6992 384 216 S 0 0.0 0:00.00 mcelog
110 975 tools.po 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
111 981 nagios 20 0 23340 1108 616 S 0 0.0 4:31.03 nrpe
112 995 tools.po 20 0 307m 13m 8880 S 0 0.2 0:00.38 php-cgi
113 1001 tools.wi 20 0 307m 13m 8868 S 0 0.2 0:00.03 php-cgi
114 1003 root 20 0 942m 3904 1656 S 0 0.0 4:24.56 nscd
115 1004 tools.wi 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
116 1005 tools.wi 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
117 1007 tools.po 20 0 17092 1388 1132 D 0 0.0 137:34.48 gam_server
118 1008 tools.wi 20 0 307m 13m 8868 S 0 0.2 0:00.03 php-cgi
119 1017 tools.wi 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
120 1024 tools.wi 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
121 1035 tools.wi 20 0 16972 1280 1108 D 0 0.0 11:42.46 gam_server
122 1039 nslcd 20 0 437m 6952 1868 S 0 0.1 4:19.75 nslcd
123 1050 tools.sd 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
124 1052 tools.sd 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
125 1053 tools.sd 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
126 1054 tools.sd 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
127 1056 tools.sd 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
128 1057 tools.sd 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
129 1058 tools.sd 20 0 4392 604 508 S 0 0.0 0:00.00 server.sh
130 1061 tools.sd 20 0 443m 25m 5368 S 0 0.3 57:06.48 python
131 1063 tools.sd 20 0 16972 1288 1112 D 0 0.0 2:06.84 gam_server
132 1066 root 20 0 8768 524 408 S 0 0.0 0:00.53 inetd
133 1105 sgeadmin 20 0 62768 8076 2208 D 0 0.1 515:39.00 sge_execd
134 1276 www-data 20 0 44332 1048 292 S 0 0.0 1:22.49 lighttpd
135 1304 root 20 0 34320 816 704 S 0 0.0 0:00.00 getty
136 1305 root 20 0 7332 716 604 S 0 0.0 0:00.00 getty
137 3104 tools.ca 20 0 314m 12m 4376 S 0 0.2 0:13.14 php-cgi
138 3136 tools.po 20 0 308m 8184 2668 S 0 0.1 0:01.56 php-cgi
139 3302 tools.wi 20 0 339m 9928 3380 S 0 0.1 0:02.84 php-cgi
140 5686 tools.po 20 0 308m 8304 2668 S 0 0.1 0:01.56 php-cgi
141 5766 tools.wi 20 0 339m 9.8m 3324 S 0 0.1 0:02.54 php-cgi
142 6414 tools.wi 20 0 338m 8856 3284 S 0 0.1 0:00.10 php-cgi
143 6703 sgeadmin 20 0 22548 2000 1552 S 0 0.0 0:00.00 sge_shepherd
144 6705 51058 20 0 53044 3796 1692 D 0 0.0 1:44.01 lighttpd
145 6729 51058 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
146 6731 51058 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
147 6732 51058 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
148 6733 51058 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
149 6735 51058 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
150 6736 51058 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
151 6738 51058 20 0 17316 1624 1144 D 0 0.0 719:19.58 gam_server
152 6956 sgeadmin 20 0 22548 1996 1548 S 0 0.0 0:00.00 sge_shepherd
153 6958 tools.ca 20 0 52532 3268 1588 D 0 0.0 9:51.10 lighttpd
154 6988 tools.ca 20 0 307m 13m 8880 S 0 0.2 0:00.33 php-cgi
155 6992 tools.ca 20 0 307m 13m 8880 S 0 0.2 0:01.05 php-cgi
156 6997 tools.ca 20 0 16972 1284 1108 D 0 0.0 5:16.09 gam_server
157 7011 sgeadmin 20 0 22548 2008 1552 S 0 0.0 0:00.00 sge_shepherd
158 7013 52421 20 0 52188 2804 1556 D 0 0.0 1:36.84 lighttpd
159 7033 52421 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
160 7035 52421 20 0 312m 9436 2936 S 0 0.1 0:00.26 php-cgi
161 7036 52421 20 0 312m 9464 2924 S 0 0.1 0:00.24 php-cgi
162 7037 52421 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
163 7039 52421 20 0 312m 9552 2964 S 0 0.1 0:08.39 php-cgi
164 7040 52421 20 0 312m 9540 2952 S 0 0.1 0:08.17 php-cgi
165 7042 52421 20 0 16972 1284 1112 D 0 0.0 7:39.02 gam_server
166 7056 sgeadmin 20 0 22548 2000 1552 S 0 0.0 0:00.00 sge_shepherd
167 7058 tools.al 20 0 52072 2796 1628 D 0 0.0 1:36.00 lighttpd
168 7082 tools.al 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
169 7084 tools.al 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
170 7085 tools.al 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
171 7086 tools.al 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
172 7088 tools.al 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
173 7089 tools.al 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
174 7091 tools.al 20 0 16972 1288 1112 D 0 0.0 9:44.89 gam_server
175 7092 sgeadmin 20 0 22548 1992 1548 S 0 0.0 0:00.00 sge_shepherd
176 7094 tools.ti 20 0 52168 2844 1636 S 0 0.0 1:35.86 lighttpd
177 7118 tools.ti 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
178 7120 tools.ti 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
179 7121 tools.ti 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
180 7122 tools.ti 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
181 7124 tools.ti 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
182 7125 tools.ti 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
183 7127 tools.ti 20 0 16972 1288 1112 D 0 0.0 34:08.19 gam_server
184 7189 sgeadmin 20 0 22548 2000 1548 S 0 0.0 0:00.00 sge_shepherd
185 7191 tools.ex 20 0 52076 2500 1468 D 0 0.0 1:35.43 lighttpd
186 7211 tools.ex 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
187 7213 tools.ex 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
188 7214 tools.ex 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
189 7215 tools.ex 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
190 7217 tools.ex 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
191 7218 tools.ex 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
192 7220 tools.ex 20 0 16972 1284 1112 D 0 0.0 44:15.11 gam_server
193 7379 sgeadmin 20 0 22548 1992 1548 S 0 0.0 0:00.00 sge_shepherd
194 7381 tools.po 20 0 52460 3172 1552 D 0 0.0 1:35.47 lighttpd
195 7437 sgeadmin 20 0 22548 2000 1548 S 0 0.0 0:00.00 sge_shepherd
196 7439 tools.di 20 0 52076 2492 1468 D 0 0.0 0:22.88 lighttpd
197 7440 sgeadmin 20 0 22548 2000 1552 S 0 0.0 0:00.00 sge_shepherd
198 7442 tools.ia 20 0 52192 2912 1688 D 0 0.0 0:23.43 lighttpd
199 7443 sgeadmin 20 0 22548 1996 1552 S 0 0.0 0:00.00 sge_shepherd
200 7449 tools.mp 20 0 52180 2800 1560 D 0 0.0 0:23.28 lighttpd
201 7464 tools.po 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
202 7466 tools.po 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
203 7467 tools.po 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
204 7468 tools.po 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
205 7470 tools.po 20 0 325m 23m 3836 S 0 0.3 0:11.78 php-cgi
206 7471 tools.po 20 0 325m 23m 3840 S 0 0.3 0:12.67 php-cgi
207 7473 tools.po 20 0 16972 1288 1112 D 0 0.0 29:49.62 gam_server
208 7551 tools.mp 20 0 307m 13m 8868 S 0 0.2 0:00.04 php-cgi
209 7556 tools.ia 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
210 7557 tools.mp 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
211 7558 tools.mp 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
212 7560 tools.ia 20 0 325m 27m 4920 S 0 0.3 0:02.84 php-cgi
213 7561 tools.ia 20 0 319m 20m 4856 S 0 0.3 0:01.34 php-cgi
214 7562 tools.mp 20 0 307m 13m 8868 S 0 0.2 0:00.03 php-cgi
215 7564 tools.mp 20 0 315m 14m 5344 S 0 0.2 0:12.80 php-cgi
216 7565 tools.mp 20 0 315m 14m 5316 S 0 0.2 0:13.37 php-cgi
217 7566 tools.ia 20 0 307m 13m 8868 S 0 0.2 0:00.04 php-cgi
218 7568 tools.ia 20 0 381m 83m 4928 S 0 1.0 1:13.47 php-cgi
219 7569 tools.ia 20 0 376m 77m 4948 S 0 1.0 1:09.92 php-cgi
220 7571 tools.mp 20 0 16972 1292 1112 D 0 0.0 5:07.47 gam_server
221 7573 tools.ia 20 0 16972 1284 1112 D 0 0.0 1:15.15 gam_server
222 7587 tools.di 20 0 307m 13m 8868 S 0 0.2 0:00.03 php-cgi
223 7589 tools.di 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
224 7590 tools.di 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
225 7591 tools.di 20 0 307m 13m 8868 S 0 0.2 0:00.03 php-cgi
226 7593 tools.di 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
227 7594 tools.di 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
228 7596 tools.di 20 0 16972 1288 1112 D 0 0.0 1:14.38 gam_server
229 7774 sgeadmin 20 0 22548 1992 1548 S 0 0.0 0:00.00 sge_shepherd
230 7776 tools.pl 20 0 52580 2980 1568 D 0 0.0 0:09.27 lighttpd
231 7844 tools.pl 20 0 307m 13m 8868 S 0 0.2 0:00.05 php-cgi
232 7858 tools.pl 20 0 307m 13m 8880 S 0 0.2 0:00.04 php-cgi
233 8615 sgeadmin 20 0 22548 2000 1548 S 0 0.0 0:00.00 sge_shepherd
234 8617 tools.hr 20 0 52572 3256 1588 D 0 0.0 1:39.15 lighttpd
235 8640 tools.hr 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
236 8643 tools.hr 20 0 309m 9928 2868 S 0 0.1 0:25.69 php-cgi
237 8644 tools.hr 20 0 313m 13m 2848 S 0 0.2 0:47.89 php-cgi
238 8649 tools.hr 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
239 8653 tools.hr 20 0 310m 10m 2904 S 0 0.1 2:50.51 php-cgi
240 8654 tools.hr 20 0 309m 9608 2892 S 0 0.1 2:53.32 php-cgi
241 8661 tools.hr 20 0 16972 1280 1108 D 0 0.0 55:25.57 gam_server
242 9482 sgeadmin 20 0 22548 2004 1548 S 0 0.0 0:00.00 sge_shepherd
243 9484 51141 20 0 52216 2732 1556 D 0 0.0 1:34.91 lighttpd
244 9504 51141 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
245 9506 51141 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
246 9507 51141 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
247 9508 51141 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
248 9510 51141 20 0 309m 9.8m 3144 S 0 0.1 0:03.74 php-cgi
249 9511 51141 20 0 308m 9372 2696 S 0 0.1 0:03.67 php-cgi
250 9513 51141 20 0 16972 1284 1108 D 0 0.0 78:19.16 gam_server
251 9576 tools.sl 20 0 308m 7764 2500 S 0 0.1 0:00.63 php-cgi
252 9579 tools.sl 20 0 308m 7776 2500 S 0 0.1 0:00.59 php-cgi
253 9614 sgeadmin 20 0 22548 2008 1548 S 0 0.0 0:00.00 sge_shepherd
254 9616 52196 20 0 52208 2852 1568 D 0 0.0 1:32.69 lighttpd
255 9636 52196 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
256 9638 52196 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
257 9639 52196 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
258 9640 52196 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
259 9642 52196 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
260 9643 52196 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
261 9645 52196 20 0 16972 1280 1108 D 0 0.0 4:52.14 gam_server
262 9741 sgeadmin 20 0 22548 2008 1552 S 0 0.0 0:00.00 sge_shepherd
263 9743 tools.fa 20 0 52076 2720 1564 D 0 0.0 1:33.61 lighttpd
264 9763 tools.fa 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
265 9765 tools.fa 20 0 307m 6704 1652 S 0 0.1 0:00.00 php-cgi
266 9766 tools.fa 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
267 9767 tools.fa 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
268 9769 tools.fa 20 0 307m 6752 1668 S 0 0.1 0:00.23 php-cgi
269 9770 tools.fa 20 0 307m 6752 1668 S 0 0.1 0:00.22 php-cgi
270 9772 tools.fa 20 0 16972 1288 1112 D 0 0.0 7:09.21 gam_server
271 9786 sgeadmin 20 0 22548 1992 1548 S 0 0.0 0:00.00 sge_shepherd
272 9788 tools.ig 20 0 52196 2772 1524 D 0 0.0 2:03.84 lighttpd
273 9808 tools.ig 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
274 9810 tools.ig 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
275 9811 tools.ig 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
276 9812 tools.ig 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
277 9814 tools.ig 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
278 9815 tools.ig 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
279 9817 tools.ig 20 0 16972 1284 1112 D 0 0.0 48:21.34 gam_server
280 9820 sgeadmin 20 0 22548 1992 1548 S 0 0.0 0:00.00 sge_shepherd
281 9822 tools.wi 20 0 52320 3036 1560 D 0 0.0 1:31.40 lighttpd
282 9842 tools.wi 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
283 9844 tools.wi 20 0 321m 23m 5172 S 0 0.3 0:20.60 php-cgi
284 9845 tools.wi 20 0 321m 23m 5244 S 0 0.3 0:13.71 php-cgi
285 9846 tools.wi 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
286 9848 tools.wi 20 0 319m 22m 5260 S 0 0.3 1:35.27 php-cgi
287 9849 tools.wi 20 0 318m 21m 5244 S 0 0.3 2:13.41 php-cgi
288 9851 tools.wi 20 0 16972 1280 1108 D 0 0.0 11:18.52 gam_server
289 9852 sgeadmin 20 0 22548 1996 1552 S 0 0.0 0:00.00 sge_shepherd
290 9854 tools.fa 20 0 52076 2504 1468 D 0 0.0 1:32.80 lighttpd
291 9874 tools.fa 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
292 9876 tools.fa 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
293 9877 tools.fa 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
294 9878 tools.fa 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
295 9880 tools.fa 20 0 309m 9496 4000 S 0 0.1 0:00.47 php-cgi
296 9881 tools.fa 20 0 335m 9592 4128 S 0 0.1 0:00.41 php-cgi
297 9883 tools.fa 20 0 16972 1284 1112 D 0 0.0 7:39.37 gam_server
298 9897 sgeadmin 20 0 22548 1996 1548 S 0 0.0 0:00.00 sge_shepherd
299 9899 tools.am 20 0 52196 2920 1584 D 0 0.0 1:33.41 lighttpd
300 9919 tools.am 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
301 9921 tools.am 20 0 308m 8244 2836 S 0 0.1 0:00.01 php-cgi
302 9922 tools.am 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
303 9923 tools.am 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
304 9925 tools.am 20 0 308m 8308 2872 S 0 0.1 0:01.50 php-cgi
305 9926 tools.am 20 0 308m 8316 2880 S 0 0.1 0:01.39 php-cgi
306 9928 tools.am 20 0 16972 1284 1112 D 0 0.0 45:06.96 gam_server
307 9940 sgeadmin 20 0 22548 1992 1548 S 0 0.0 0:00.00 sge_shepherd
308 9942 tools.an 20 0 52076 2732 1560 D 0 0.0 1:35.78 lighttpd
309 9962 tools.an 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
310 9964 tools.an 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
311 9965 tools.an 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
312 9966 tools.an 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
313 9968 tools.an 20 0 308m 7496 2184 S 0 0.1 0:00.66 php-cgi
314 9969 tools.an 20 0 308m 7568 2184 S 0 0.1 0:00.64 php-cgi
315 9971 tools.an 20 0 16972 1284 1108 D 0 0.0 5:24.35 gam_server
316 9974 sgeadmin 20 0 22548 1996 1548 S 0 0.0 0:00.00 sge_shepherd
317 9976 tools.wi 20 0 105m 5620 2616 D 0 0.1 13:07.44 lighttpd
318 10000 tools.wi 20 0 307m 13m 8880 S 0 0.2 0:00.17 php-cgi
319 10004 tools.wi 20 0 307m 13m 8880 S 0 0.2 0:01.60 php-cgi
320 10009 tools.wi 20 0 17084 1356 1128 D 0 0.0 38:40.14 gam_server
321 10021 sgeadmin 20 0 22548 2000 1552 S 0 0.0 0:00.00 sge_shepherd
322 10023 tools.si 20 0 52076 2724 1556 S 0 0.0 1:31.27 lighttpd
323 10041 tools.si 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
324 10043 tools.si 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
325 10044 tools.si 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
326 10045 tools.si 20 0 307m 13m 8868 S 0 0.2 0:00.06 php-cgi
327 10047 tools.si 20 0 308m 7312 1920 S 0 0.1 0:00.09 php-cgi
328 10048 tools.si 20 0 308m 7312 1920 S 0 0.1 0:00.08 php-cgi
329 10050 tools.si 20 0 16972 1284 1112 D 0 0.0 7:15.61 gam_server
330 10707 root 20 0 0 0 0 S 0 0.0 0:01.73 flush-252:1
331 10717 tools.ip 20 0 312m 8372 2820 S 0 0.1 0:02.47 php-cgi
332 11665 sgeadmin 20 0 22548 2004 1552 S 0 0.0 0:00.00 sge_shepherd
333 11667 tools.sl 20 0 52076 2728 1560 D 0 0.0 0:58.88 lighttpd
334 11668 sgeadmin 20 0 22548 1992 1548 S 0 0.0 0:00.00 sge_shepherd
335 11670 tools.ip 20 0 52412 3084 1600 D 0 0.0 0:54.08 lighttpd
336 11671 sgeadmin 20 0 22548 1996 1548 S 0 0.0 0:00.00 sge_shepherd
337 11673 tools.gr 20 0 52076 2504 1468 D 0 0.0 0:41.06 lighttpd
338 11692 sgeadmin 20 0 22548 1996 1548 S 0 0.0 0:00.00 sge_shepherd
339 11695 tools.cg 20 0 54380 3052 1684 S 0 0.0 2:49.85 lighttpd
340 11732 tools.ip 20 0 307m 13m 8868 S 0 0.2 0:00.04 php-cgi
341 11738 tools.gr 20 0 307m 13m 8868 S 0 0.2 0:00.03 php-cgi
342 11763 tools.gr 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
343 11766 tools.gr 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
344 11777 tools.sl 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
345 11783 tools.ip 20 0 307m 13m 8880 S 0 0.2 0:00.03 php-cgi
346 11787 tools.gr 20 0 307m 13m 8868 S 0 0.2 0:00.03 php-cgi
347 11791 tools.sl 20 0 308m 7780 2504 S 0 0.1 0:00.52 php-cgi
348 11793 tools.sl 20 0 308m 7776 2504 S 0 0.1 0:00.55 php-cgi
349 11799 tools.gr 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
350 11802 tools.gr 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
351 11807 tools.cg 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
352 11808 tools.sl 20 0 307m 13m 8880 S 0 0.2 0:00.07 php-cgi
353 11810 tools.ip 20 0 16972 1288 1112 D 0 0.0 3:54.09 gam_server
354 11812 tools.cg 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
355 11814 tools.cg 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
356 11815 tools.gr 20 0 16972 1284 1112 D 0 0.0 1:46.12 gam_server
357 11823 tools.cg 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
358 11825 tools.sl 20 0 16972 1288 1112 D 0 0.0 2:04.72 gam_server
359 11827 tools.cg 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
360 11828 tools.cg 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
361 11833 tools.cg 20 0 464m 24m 4032 S 0 0.3 150:23.41 cgstat.fcgi
362 11837 tools.cg 20 0 16972 1280 1108 D 0 0.0 3:08.01 gam_server
363 12193 tools.po 20 0 308m 8172 2668 S 0 0.1 0:01.32 php-cgi
364 12366 tools.po 20 0 308m 8348 2780 S 0 0.1 0:01.21 php-cgi
365 12576 sgeadmin 20 0 22548 1996 1548 S 0 0.0 0:00.00 sge_shepherd
366 12578 51177 20 0 52196 2960 1676 D 0 0.0 0:00.94 lighttpd
367 12602 51177 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
368 12604 51177 20 0 310m 9728 3012 S 0 0.1 0:14.63 php-cgi
369 12605 51177 20 0 310m 9492 3008 S 0 0.1 0:13.47 php-cgi
370 12606 51177 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
371 12608 51177 20 0 310m 9696 3012 S 0 0.1 1:56.29 php-cgi
372 12609 51177 20 0 309m 9372 3012 S 0 0.1 2:35.91 php-cgi
373 12611 51177 20 0 16972 1288 1112 D 0 0.0 0:02.22 gam_server
374 12653 tools.ca 20 0 315m 13m 4376 S 0 0.2 0:19.64 php-cgi
375 12678 tools.ca 20 0 313m 11m 4376 S 0 0.1 0:19.28 php-cgi
376 17951 tools.ip 20 0 312m 8472 2816 S 0 0.1 0:11.19 php-cgi
377 19724 51559 20 0 308m 9.9m 4120 S 0 0.1 0:07.98 php-cgi
378 19776 tools.pl 20 0 16972 1344 1136 D 0 0.0 0:48.26 gam_server
379 19804 tools.pl 20 0 309m 9.9m 3996 D 0 0.1 0:08.49 php-cgi
380 20311 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/1:2
381 20806 51559 20 0 313m 14m 4208 S 0 0.2 0:22.41 php-cgi
382 21249 sgeadmin 20 0 22548 1988 1544 S 0 0.0 0:00.00 sge_shepherd
383 21251 51559 20 0 53980 4716 1584 D 0 0.1 1:22.45 lighttpd
384 21277 51559 20 0 307m 13m 8868 S 0 0.2 0:00.03 php-cgi
385 21282 51559 20 0 307m 13m 8880 S 0 0.2 0:00.09 php-cgi
386 21287 51559 20 0 16972 1284 1112 D 0 0.0 8:32.86 gam_server
387 21398 tools.pl 20 0 309m 9.8m 3976 S 0 0.1 0:00.63 php-cgi
388 21412 tools.pl 20 0 309m 9.8m 3976 S 0 0.1 0:00.54 php-cgi
389 21467 tools.je 20 0 16972 1288 1112 S 0 0.0 0:00.00 gam_server
390 21867 sgeadmin 20 0 22548 2000 1548 S 0 0.0 0:00.00 sge_shepherd
391 21869 tools.je 20 0 52072 2552 1520 D 0 0.0 0:00.67 lighttpd
392 21894 tools.je 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
393 21896 tools.je 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
394 21897 tools.je 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
395 21898 tools.je 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
396 21900 tools.je 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
397 21901 tools.je 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
398 21947 tools.wi 20 0 339m 9876 3508 S 0 0.1 0:00.37 php-cgi
399 22097 tools.pl 20 0 309m 9.8m 3976 S 0 0.1 0:00.65 php-cgi
400 22309 sgeadmin 20 0 22548 2112 1608 S 0 0.0 0:00.00 sge_shepherd
401 22338 tools.ft 20 0 0 0 0 D 0 0.0 0:00.00 php-cgi
402 22344 tools.ft 20 0 16972 1288 1112 S 0 0.0 0:00.00 gam_server
403 22554 tools.ft 20 0 0 0 0 D 0 0.0 0:00.00 lighttpd
404 22578 tools.ft 20 0 25572 1188 668 D 0 0.0 0:00.00 sge_shepherd
405 23117 tools.ip 20 0 312m 8364 2816 S 0 0.1 0:01.85 php-cgi
406 23178 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
407 23179 sshd 20 0 62672 1336 632 S 0 0.0 0:00.00 sshd
408 23301 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
409 23302 root 20 0 4396 608 508 S 0 0.0 0:00.00 sh
410 23307 root 20 0 38908 1000 852 S 0 0.0 0:00.00 find
411 23308 root 20 0 31124 824 680 D 0 0.0 0:00.32 fuser
412 23529 root 20 0 35860 1184 852 S 0 0.0 0:00.00 cron
413 23531 root 20 0 4396 612 508 S 0 0.0 0:00.00 sh
414 23533 root 20 0 4296 604 496 S 0 0.0 0:00.00 run-parts
415 23699 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
416 23700 sshd 20 0 62672 1340 628 S 0 0.0 0:00.00 sshd
417 23930 tools.ip 20 0 312m 8352 2812 S 0 0.1 0:01.39 php-cgi
418 24287 root 0 -20 20044 7764 3392 S 0 0.1 0:15.59 atop
419 24358 root 20 0 4396 612 508 S 0 0.0 0:00.00 exim4-base
420 24363 root 20 0 31396 608 516 S 0 0.0 0:00.00 xargs
421 24364 root 20 0 10336 612 504 D 0 0.0 0:00.00 start-stop-daem
422 24392 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
423 24393 root 20 0 4396 612 508 S 0 0.0 0:00.00 sh
424 24398 root 20 0 38908 1000 852 S 0 0.0 0:00.00 find
425 24399 root 20 0 31124 824 680 D 0 0.0 0:00.38 fuser
426 25014 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
427 25015 sshd 20 0 62672 1332 628 S 0 0.0 0:00.00 sshd
428 25137 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
429 25138 root 20 0 4396 604 504 S 0 0.0 0:00.00 sh
430 25143 root 20 0 38908 1000 852 S 0 0.0 0:00.00 find
431 25144 root 20 0 31124 824 680 D 0 0.0 0:00.35 fuser
432 25447 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
433 25448 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
434 25570 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
435 25571 root 20 0 4396 608 504 S 0 0.0 0:00.00 sh
436 25576 root 20 0 38908 996 852 S 0 0.0 0:00.00 find
437 25577 root 20 0 31124 824 680 D 0 0.0 0:00.46 fuser
438 26198 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
439 26199 sshd 20 0 62672 1336 632 S 0 0.0 0:00.00 sshd
440 26345 root 20 0 35860 1184 852 S 0 0.0 0:00.00 cron
441 26347 root 20 0 4396 608 508 S 0 0.0 0:00.00 sh
442 26354 root 20 0 38908 996 852 S 0 0.0 0:00.00 find
443 26355 root 20 0 31124 824 680 D 0 0.0 0:00.29 fuser
444 26570 51559 20 0 310m 11m 4292 S 0 0.1 0:20.13 php-cgi
445 26722 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
446 26723 sshd 20 0 62672 1336 632 S 0 0.0 0:00.01 sshd
447 26869 root 20 0 35860 1184 852 S 0 0.0 0:00.00 cron
448 26872 root 20 0 4396 612 508 S 0 0.0 0:00.00 sh
449 26878 root 20 0 38908 1000 852 S 0 0.0 0:00.00 find
450 26879 root 20 0 31124 828 680 D 0 0.0 0:00.41 fuser
451 27491 tools.ca 20 0 314m 12m 4380 S 0 0.2 0:20.21 php-cgi
452 27554 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
453 27555 sshd 20 0 62672 1336 628 S 0 0.0 0:00.01 sshd
454 27677 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
455 27678 root 20 0 4396 608 508 S 0 0.0 0:00.00 sh
456 27683 root 20 0 38908 996 852 S 0 0.0 0:00.00 find
457 27684 root 20 0 31124 824 680 D 0 0.0 0:00.41 fuser
458 27987 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.02 sshd
459 27988 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
460 28097 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
461 28098 root 20 0 4396 612 508 S 0 0.0 0:00.00 sh
462 28103 root 20 0 38908 996 852 S 0 0.0 0:00.00 find
463 28104 root 20 0 31124 824 680 D 0 0.0 0:00.44 fuser
464 28210 51559 20 0 308m 9800 4188 S 0 0.1 0:08.18 php-cgi
465 28719 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
466 28720 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
467 28842 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
468 28843 root 20 0 4396 612 504 S 0 0.0 0:00.00 sh
469 28848 root 20 0 38908 996 852 S 0 0.0 0:00.00 find
470 28849 root 20 0 31124 824 680 D 0 0.0 0:00.34 fuser
471 29152 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.02 sshd
472 29153 sshd 20 0 62672 1336 628 S 0 0.0 0:00.01 sshd
473 29275 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
474 29276 root 20 0 4396 608 508 S 0 0.0 0:00.00 sh
475 29281 root 20 0 38908 996 852 S 0 0.0 0:00.00 find
476 29282 root 20 0 31124 824 680 D 0 0.0 0:00.31 fuser
477 29897 scfc 20 0 97832 4104 3084 D 0 0.1 0:00.01 sshd
478 29898 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
479 30020 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
480 30021 root 20 0 4396 608 508 S 0 0.0 0:00.00 sh
481 30026 root 20 0 38908 996 852 S 0 0.0 0:00.00 find
482 30027 root 20 0 31124 820 680 D 0 0.0 0:00.35 fuser
483 30330 scfc 20 0 97832 4092 3072 D 0 0.1 0:00.00 sshd
484 30331 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
485 30453 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
486 30454 root 20 0 4396 612 508 S 0 0.0 0:00.00 sh
487 30459 root 20 0 38908 996 852 S 0 0.0 0:00.00 find
488 30460 root 20 0 31124 820 680 D 0 0.0 0:00.37 fuser
489 31008 tools.ph 20 0 97832 4100 3084 D 0 0.1 0:00.00 sshd
490 31009 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
491 31197 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
492 31198 root 20 0 4396 612 504 S 0 0.0 0:00.00 sh
493 31203 root 20 0 38908 996 852 S 0 0.0 0:00.00 find
494 31204 root 20 0 31124 824 680 D 0 0.0 0:00.30 fuser
495 31252 root 20 0 0 0 0 S 0 0.0 0:47.78 kworker/2:0
496 31508 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
497 31509 sshd 20 0 62672 1340 628 S 0 0.0 0:00.00 sshd
498 31618 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
499 31619 root 20 0 4396 608 508 S 0 0.0 0:00.00 sh
500 31624 root 20 0 38908 996 852 S 0 0.0 0:00.00 find
501 31625 root 20 0 31124 828 680 D 0 0.0 0:00.34 fuser
502 32361 root 20 0 35860 1188 856 S 0 0.0 0:00.00 cron
503 32362 root 20 0 4396 608 508 S 0 0.0 0:00.00 sh
504 32367 root 20 0 38908 996 852 S 0 0.0 0:00.00 find
505 32368 root 20 0 31124 824 680 D 0 0.0 0:00.30 fuser
506 32685 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
507 32686 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
508 32713 root 20 0 98.9m 4284 3232 D 0 0.1 0:00.02 sshd

I was mostly interested in the really high load values compared to the other seemingly normal usage. So I was looking for procs in D state as I had a suspicion it was IO related.

salt 'tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs' cmd.run "top -b -n 1" >> /tmp/top_b_1

P2492 salt 'tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs' cmd.run "top -b -n 1" >> /tmp/top_b_1
1root@labcontrol1001:~# cat /tmp/top_b_1 | awk '{ print $8, $0}' | sort | egrep ^D
2D 10009 tools.wi 20 0 17084 1356 1128 D 0 0.0 38:40.14 gam_server
3D 10050 tools.si 20 0 16972 1284 1112 D 0 0.0 7:15.61 gam_server
4D 1007 tools.po 20 0 17092 1388 1132 D 0 0.0 137:34.48 gam_server
5D 1035 tools.wi 20 0 16972 1280 1108 D 0 0.0 11:42.46 gam_server
6D 1063 tools.sd 20 0 16972 1288 1112 D 0 0.0 2:06.84 gam_server
7D 1105 sgeadmin 20 0 62768 8076 2208 D 0 0.1 515:39.21 sge_execd
8D 11667 tools.sl 20 0 52076 2728 1560 D 0 0.0 0:58.88 lighttpd
9D 11670 tools.ip 20 0 52412 3084 1600 D 0 0.0 0:54.08 lighttpd
10D 11673 tools.gr 20 0 52076 2504 1468 D 0 0.0 0:41.06 lighttpd
11D 11810 tools.ip 20 0 16972 1288 1112 D 0 0.0 3:54.09 gam_server
12D 11815 tools.gr 20 0 16972 1284 1112 D 0 0.0 1:46.12 gam_server
13D 11825 tools.sl 20 0 16972 1288 1112 D 0 0.0 2:04.72 gam_server
14D 1182 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
15D 11837 tools.cg 20 0 16972 1280 1108 D 0 0.0 3:08.01 gam_server
16D 12578 51177 20 0 52196 2960 1676 D 0 0.0 0:00.94 lighttpd
17D 12611 51177 20 0 16972 1288 1112 D 0 0.0 0:02.22 gam_server
18D 19776 tools.pl 20 0 16972 1344 1136 D 0 0.0 0:48.26 gam_server
19D 19804 tools.pl 20 0 309m 9.9m 3996 D 0 0.1 0:08.49 php-cgi
20D 21251 51559 20 0 53980 4716 1584 D 0 0.1 1:22.45 lighttpd
21D 21287 51559 20 0 16972 1284 1112 D 0 0.0 8:32.86 gam_server
22D 21869 tools.je 20 0 52072 2552 1520 D 0 0.0 0:00.67 lighttpd
23D 22338 tools.ft 20 0 0 0 0 D 0 0.0 0:00.00 php-cgi
24D 22554 tools.ft 20 0 0 0 0 D 0 0.0 0:00.00 lighttpd
25D 22578 tools.ft 20 0 25572 1188 668 D 0 0.0 0:00.00 sge_shepherd
26D 23178 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
27D 23308 root 20 0 31124 824 680 D 0 0.0 0:00.32 fuser
28D 23699 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
29D 24364 root 20 0 10336 612 504 D 0 0.0 0:00.00 start-stop-daem
30D 24399 root 20 0 31124 824 680 D 0 0.0 0:00.38 fuser
31D 25014 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
32D 25144 root 20 0 31124 824 680 D 0 0.0 0:00.35 fuser
33D 25447 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
34D 25577 root 20 0 31124 824 680 D 0 0.0 0:00.46 fuser
35D 26198 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
36D 26355 root 20 0 31124 824 680 D 0 0.0 0:00.29 fuser
37D 26722 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
38D 26879 root 20 0 31124 828 680 D 0 0.0 0:00.41 fuser
39D 27554 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
40D 27684 root 20 0 31124 824 680 D 0 0.0 0:00.41 fuser
41D 27987 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.02 sshd
42D 28104 root 20 0 31124 824 680 D 0 0.0 0:00.44 fuser
43D 28719 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
44D 28849 root 20 0 31124 824 680 D 0 0.0 0:00.34 fuser
45D 29152 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.02 sshd
46D 29282 root 20 0 31124 824 680 D 0 0.0 0:00.31 fuser
47D 29897 scfc 20 0 97832 4104 3084 D 0 0.1 0:00.01 sshd
48D 30027 root 20 0 31124 820 680 D 0 0.0 0:00.35 fuser
49D 30330 scfc 20 0 97832 4092 3072 D 0 0.1 0:00.00 sshd
50D 30460 root 20 0 31124 820 680 D 0 0.0 0:00.37 fuser
51D 31008 tools.ph 20 0 97832 4100 3084 D 0 0.1 0:00.00 sshd
52D 31204 root 20 0 31124 824 680 D 0 0.0 0:00.30 fuser
53D 31508 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
54D 31625 root 20 0 31124 828 680 D 0 0.0 0:00.34 fuser
55D 32368 root 20 0 31124 824 680 D 0 0.0 0:00.30 fuser
56D 32685 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
57D 32713 root 20 0 98.9m 4284 3232 D 0 0.1 0:00.02 sshd
58D 413 root 20 0 31124 828 680 D 0 0.0 0:00.30 fuser
59D 504 root 20 0 175m 88m 4104 D 0 1.1 0:06.96 puppet
60D 6705 51058 20 0 53044 3796 1692 D 0 0.0 1:44.01 lighttpd
61D 6738 51058 20 0 17316 1624 1144 D 0 0.0 719:19.58 gam_server
62D 6958 tools.ca 20 0 52532 3268 1588 D 0 0.0 9:51.10 lighttpd
63D 6997 tools.ca 20 0 16972 1284 1108 D 0 0.0 5:16.09 gam_server
64D 7013 52421 20 0 52188 2804 1556 D 0 0.0 1:36.84 lighttpd
65D 7042 52421 20 0 16972 1284 1112 D 0 0.0 7:39.02 gam_server
66D 7058 tools.al 20 0 52072 2796 1628 D 0 0.0 1:36.00 lighttpd
67D 7091 tools.al 20 0 16972 1288 1112 D 0 0.0 9:44.89 gam_server
68D 7127 tools.ti 20 0 16972 1288 1112 D 0 0.0 34:08.19 gam_server
69D 7191 tools.ex 20 0 52076 2500 1468 D 0 0.0 1:35.43 lighttpd
70D 7220 tools.ex 20 0 16972 1284 1112 D 0 0.0 44:15.11 gam_server
71D 7381 tools.po 20 0 52460 3172 1552 D 0 0.0 1:35.47 lighttpd
72D 7439 tools.di 20 0 52076 2492 1468 D 0 0.0 0:22.88 lighttpd
73D 7442 tools.ia 20 0 52192 2912 1688 D 0 0.0 0:23.43 lighttpd
74D 7449 tools.mp 20 0 52180 2800 1560 D 0 0.0 0:23.28 lighttpd
75D 7473 tools.po 20 0 16972 1288 1112 D 0 0.0 29:49.62 gam_server
76D 7571 tools.mp 20 0 16972 1292 1112 D 0 0.0 5:07.47 gam_server
77D 7573 tools.ia 20 0 16972 1284 1112 D 0 0.0 1:15.15 gam_server
78D 7596 tools.di 20 0 16972 1288 1112 D 0 0.0 1:14.38 gam_server
79D 7776 tools.pl 20 0 52580 2980 1568 D 0 0.0 0:09.27 lighttpd
80D 8617 tools.hr 20 0 52572 3256 1588 D 0 0.0 1:39.15 lighttpd
81D 8661 tools.hr 20 0 16972 1280 1108 D 0 0.0 55:25.57 gam_server
82D 935 tools.po 20 0 52300 2936 1576 D 0 0.0 5:48.38 lighttpd
83D 939 tools.sd 20 0 52152 2920 1684 D 0 0.0 1:37.51 lighttpd
84D 942 tools.wi 20 0 52332 3156 1660 D 0 0.0 1:33.50 lighttpd
85D 9484 51141 20 0 52216 2732 1556 D 0 0.0 1:34.91 lighttpd
86D 9513 51141 20 0 16972 1284 1108 D 0 0.0 78:19.16 gam_server
87D 9616 52196 20 0 52208 2852 1568 D 0 0.0 1:32.69 lighttpd
88D 9645 52196 20 0 16972 1280 1108 D 0 0.0 4:52.14 gam_server
89D 9743 tools.fa 20 0 52076 2720 1564 D 0 0.0 1:33.61 lighttpd
90D 9772 tools.fa 20 0 16972 1288 1112 D 0 0.0 7:09.21 gam_server
91D 9788 tools.ig 20 0 52196 2772 1524 D 0 0.0 2:03.84 lighttpd
92D 9817 tools.ig 20 0 16972 1284 1112 D 0 0.0 48:21.34 gam_server
93D 9822 tools.wi 20 0 52320 3036 1560 D 0 0.0 1:31.40 lighttpd
94D 9851 tools.wi 20 0 16972 1280 1108 D 0 0.0 11:18.52 gam_server
95D 9854 tools.fa 20 0 52076 2504 1468 D 0 0.0 1:32.80 lighttpd
96D 9883 tools.fa 20 0 16972 1284 1112 D 0 0.0 7:39.37 gam_server
97D 9899 tools.am 20 0 52196 2920 1584 D 0 0.0 1:33.41 lighttpd
98D 9928 tools.am 20 0 16972 1284 1112 D 0 0.0 45:06.96 gam_server
99D 9942 tools.an 20 0 52076 2732 1560 D 0 0.0 1:35.78 lighttpd
100D 9971 tools.an 20 0 16972 1284 1108 D 0 0.0 5:24.35 gam_server
101D 9976 tools.wi 20 0 105m 5620 2616 D 0 0.1 13:07.44 lighttpd

for comparison I issue top -b -n 1 | awk '{ print $8, $0}' | sort | egrep ^D on a healthy webgir node I see either no or a small handful of short lived processes in D state.

My thinking is processes are getting stuck in perpetual sleep unable to complete their IO ops and are backpooling load on the host. But why?

auth.log

P2493 salt 'tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs' cmd.run "tail -n 100 /var/log/auth.log"
1tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs:
2 Jan 20 13:49:24 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user root
3 Jan 20 13:49:24 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=list /bin/cat /var/lib/puppet/state/last_run_summary.yaml
4 Jan 20 13:49:24 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=/bin/cat /var/lib/puppet/state/last_run_summary.yaml
5 Jan 20 13:49:24 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user puppet by (uid=109)
6 Jan 20 13:49:24 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user puppet
7 Jan 20 13:50:24 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=root ; COMMAND=/bin/ls /var/spool/cron/crontabs/
8 Jan 20 13:50:24 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user root by (uid=109)
9 Jan 20 13:50:24 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user root
10 Jan 20 13:50:24 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=list /bin/cat /var/lib/puppet/state/last_run_summary.yaml
11 Jan 20 13:50:25 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=/bin/cat /var/lib/puppet/state/last_run_summary.yaml
12 Jan 20 13:50:25 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user puppet by (uid=109)
13 Jan 20 13:50:25 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user puppet
14 Jan 20 13:51:24 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=root ; COMMAND=/bin/ls /var/spool/cron/crontabs/
15 Jan 20 13:51:24 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user root by (uid=109)
16 Jan 20 13:51:24 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user root
17 Jan 20 13:51:25 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=list /bin/cat /var/lib/puppet/state/last_run_summary.yaml
18 Jan 20 13:51:25 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=/bin/cat /var/lib/puppet/state/last_run_summary.yaml
19 Jan 20 13:51:25 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user puppet by (uid=109)
20 Jan 20 13:51:25 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user puppet
21 Jan 20 13:52:25 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=root ; COMMAND=/bin/ls /var/spool/cron/crontabs/
22 Jan 20 13:52:25 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user root by (uid=109)
23 Jan 20 13:52:25 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user root
24 Jan 20 13:52:25 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=list /bin/cat /var/lib/puppet/state/last_run_summary.yaml
25 Jan 20 13:52:25 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=/bin/cat /var/lib/puppet/state/last_run_summary.yaml
26 Jan 20 13:52:25 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user puppet by (uid=109)
27 Jan 20 13:52:25 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user puppet
28 Jan 20 13:53:25 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=root ; COMMAND=/bin/ls /var/spool/cron/crontabs/
29 Jan 20 13:53:25 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user root by (uid=109)
30 Jan 20 13:53:25 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user root
31 Jan 20 13:53:25 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=list /bin/cat /var/lib/puppet/state/last_run_summary.yaml
32 Jan 20 13:53:25 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=/bin/cat /var/lib/puppet/state/last_run_summary.yaml
33 Jan 20 13:53:25 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user puppet by (uid=109)
34 Jan 20 13:53:25 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user puppet
35 Jan 20 13:54:25 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=root ; COMMAND=/bin/ls /var/spool/cron/crontabs/
36 Jan 20 13:54:25 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user root by (uid=109)
37 Jan 20 13:54:25 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user root
38 Jan 20 13:54:25 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=list /bin/cat /var/lib/puppet/state/last_run_summary.yaml
39 Jan 20 13:54:26 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=/bin/cat /var/lib/puppet/state/last_run_summary.yaml
40 Jan 20 13:54:26 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user puppet by (uid=109)
41 Jan 20 13:54:26 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user puppet
42 Jan 20 13:55:01 tools-webgrid-lighttpd-1209 CRON[1110]: pam_unix(cron:session): session opened for user root by (uid=0)
43 Jan 20 13:55:01 tools-webgrid-lighttpd-1209 CRON[1110]: pam_unix(cron:session): session closed for user root
44 Jan 20 13:55:25 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=root ; COMMAND=/bin/ls /var/spool/cron/crontabs/
45 Jan 20 13:55:25 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user root by (uid=109)
46 Jan 20 13:55:26 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user root
47 Jan 20 13:55:26 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=list /bin/cat /var/lib/puppet/state/last_run_summary.yaml
48 Jan 20 13:55:26 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=/bin/cat /var/lib/puppet/state/last_run_summary.yaml
49 Jan 20 13:55:26 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user puppet by (uid=109)
50 Jan 20 13:55:26 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user puppet
51 Jan 20 13:56:26 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=root ; COMMAND=/bin/ls /var/spool/cron/crontabs/
52 Jan 20 13:56:26 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user root by (uid=109)
53 Jan 20 13:56:26 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user root
54 Jan 20 13:56:26 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=list /bin/cat /var/lib/puppet/state/last_run_summary.yaml
55 Jan 20 13:56:26 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=/bin/cat /var/lib/puppet/state/last_run_summary.yaml
56 Jan 20 13:56:26 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user puppet by (uid=109)
57 Jan 20 13:56:26 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user puppet
58 Jan 20 13:57:26 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=root ; COMMAND=/bin/ls /var/spool/cron/crontabs/
59 Jan 20 13:57:26 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user root by (uid=109)
60 Jan 20 13:57:26 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user root
61 Jan 20 13:57:26 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=list /bin/cat /var/lib/puppet/state/last_run_summary.yaml
62 Jan 20 13:57:27 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=/bin/cat /var/lib/puppet/state/last_run_summary.yaml
63 Jan 20 13:57:27 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user puppet by (uid=109)
64 Jan 20 13:57:27 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user puppet
65 Jan 20 13:58:26 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=root ; COMMAND=/bin/ls /var/spool/cron/crontabs/
66 Jan 20 13:58:26 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user root by (uid=109)
67 Jan 20 13:58:26 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user root
68 Jan 20 13:58:27 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=list /bin/cat /var/lib/puppet/state/last_run_summary.yaml
69 Jan 20 13:58:27 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=/bin/cat /var/lib/puppet/state/last_run_summary.yaml
70 Jan 20 13:58:27 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user puppet by (uid=109)
71 Jan 20 13:58:27 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user puppet
72 Jan 20 13:59:27 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=root ; COMMAND=/bin/ls /var/spool/cron/crontabs/
73 Jan 20 13:59:27 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user root by (uid=109)
74 Jan 20 13:59:27 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user root
75 Jan 20 13:59:27 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=list /bin/cat /var/lib/puppet/state/last_run_summary.yaml
76 Jan 20 13:59:27 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=/bin/cat /var/lib/puppet/state/last_run_summary.yaml
77 Jan 20 13:59:27 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user puppet by (uid=109)
78 Jan 20 13:59:27 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user puppet
79 Jan 20 14:00:12 tools-webgrid-lighttpd-1209 sshd[1182]: Set /proc/self/oom_score_adj to 0
80 Jan 20 14:00:12 tools-webgrid-lighttpd-1209 sshd[1182]: Connection from 10.68.17.228 port 58664 on 10.68.17.152 port 22
81 Jan 20 14:00:27 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=root ; COMMAND=/bin/ls /var/spool/cron/crontabs/
82 Jan 20 14:00:27 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user root by (uid=109)
83 Jan 20 14:00:27 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user root
84 Jan 20 14:00:27 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=list /bin/cat /var/lib/puppet/state/last_run_summary.yaml
85 Jan 20 14:00:28 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=/bin/cat /var/lib/puppet/state/last_run_summary.yaml
86 Jan 20 14:00:28 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user puppet by (uid=109)
87 Jan 20 14:00:28 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user puppet
88 Jan 20 14:01:27 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=root ; COMMAND=/bin/ls /var/spool/cron/crontabs/
89 Jan 20 14:01:27 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user root by (uid=109)
90 Jan 20 14:01:27 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user root
91 Jan 20 14:01:28 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=list /bin/cat /var/lib/puppet/state/last_run_summary.yaml
92 Jan 20 14:01:28 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=/bin/cat /var/lib/puppet/state/last_run_summary.yaml
93 Jan 20 14:01:28 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user puppet by (uid=109)
94 Jan 20 14:01:28 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user puppet
95 Jan 20 14:02:28 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=root ; COMMAND=/bin/ls /var/spool/cron/crontabs/
96 Jan 20 14:02:28 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user root by (uid=109)
97 Jan 20 14:02:28 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user root
98 Jan 20 14:02:28 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=list /bin/cat /var/lib/puppet/state/last_run_summary.yaml
99 Jan 20 14:02:28 tools-webgrid-lighttpd-1209 sudo: diamond : TTY=pts/2 ; PWD=/ ; USER=puppet ; COMMAND=/bin/cat /var/lib/puppet/state/last_run_summary.yaml
100 Jan 20 14:02:28 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session opened for user puppet by (uid=109)
101 Jan 20 14:02:28 tools-webgrid-lighttpd-1209 sudo: pam_unix(sudo:session): session closed for user puppet
102root@labcontrol1001:~# salt 'tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs' cmd.run "grep scfc /var/log/auth.log"
103tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs:
104 Jan 17 07:00:10 tools-webgrid-lighttpd-1209 sshd[30620]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
105 Jan 17 07:00:10 tools-webgrid-lighttpd-1209 sshd[30620]: Accepted hostbased for scfc from 10.68.17.228 port 42871 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
106 Jan 17 07:00:10 tools-webgrid-lighttpd-1209 sshd[30620]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
107 Jan 17 07:00:10 tools-webgrid-lighttpd-1209 sshd[30635]: Starting session: command for scfc from 10.68.17.228 port 42871
108 Jan 17 07:00:10 tools-webgrid-lighttpd-1209 sshd[30620]: pam_unix(sshd:session): session closed for user scfc
109 Jan 17 07:30:09 tools-webgrid-lighttpd-1209 sshd[791]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
110 Jan 17 07:30:09 tools-webgrid-lighttpd-1209 sshd[791]: Accepted hostbased for scfc from 10.68.17.228 port 42214 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
111 Jan 17 07:30:09 tools-webgrid-lighttpd-1209 sshd[791]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
112 Jan 17 07:30:09 tools-webgrid-lighttpd-1209 sshd[808]: Starting session: command for scfc from 10.68.17.228 port 42214
113 Jan 17 07:30:09 tools-webgrid-lighttpd-1209 sshd[791]: pam_unix(sshd:session): session closed for user scfc
114 Jan 17 08:00:09 tools-webgrid-lighttpd-1209 sshd[3485]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
115 Jan 17 08:00:10 tools-webgrid-lighttpd-1209 sshd[3485]: Accepted hostbased for scfc from 10.68.17.228 port 41532 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
116 Jan 17 08:00:10 tools-webgrid-lighttpd-1209 sshd[3485]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
117 Jan 17 08:00:10 tools-webgrid-lighttpd-1209 sshd[3500]: Starting session: command for scfc from 10.68.17.228 port 41532
118 Jan 17 08:00:10 tools-webgrid-lighttpd-1209 sshd[3485]: pam_unix(sshd:session): session closed for user scfc
119 Jan 17 08:30:11 tools-webgrid-lighttpd-1209 sshd[6186]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
120 Jan 17 08:30:11 tools-webgrid-lighttpd-1209 sshd[6186]: Accepted hostbased for scfc from 10.68.17.228 port 40429 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
121 Jan 17 08:30:11 tools-webgrid-lighttpd-1209 sshd[6186]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
122 Jan 17 08:30:11 tools-webgrid-lighttpd-1209 sshd[6201]: Starting session: command for scfc from 10.68.17.228 port 40429
123 Jan 17 08:30:11 tools-webgrid-lighttpd-1209 sshd[6186]: pam_unix(sshd:session): session closed for user scfc
124 Jan 17 09:00:09 tools-webgrid-lighttpd-1209 sshd[9005]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
125 Jan 17 09:00:09 tools-webgrid-lighttpd-1209 sshd[9005]: Accepted hostbased for scfc from 10.68.17.228 port 39776 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
126 Jan 17 09:00:09 tools-webgrid-lighttpd-1209 sshd[9005]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
127 Jan 17 09:00:09 tools-webgrid-lighttpd-1209 sshd[9020]: Starting session: command for scfc from 10.68.17.228 port 39776
128 Jan 17 09:00:09 tools-webgrid-lighttpd-1209 sshd[9005]: pam_unix(sshd:session): session closed for user scfc
129 Jan 17 09:30:11 tools-webgrid-lighttpd-1209 sshd[11801]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
130 Jan 17 09:30:12 tools-webgrid-lighttpd-1209 sshd[11801]: Accepted hostbased for scfc from 10.68.17.228 port 39078 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
131 Jan 17 09:30:12 tools-webgrid-lighttpd-1209 sshd[11801]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
132 Jan 17 09:30:12 tools-webgrid-lighttpd-1209 sshd[11830]: Starting session: command for scfc from 10.68.17.228 port 39078
133 Jan 17 09:30:12 tools-webgrid-lighttpd-1209 sshd[11801]: pam_unix(sshd:session): session closed for user scfc
134 Jan 17 10:00:10 tools-webgrid-lighttpd-1209 sshd[14445]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
135 Jan 17 10:00:10 tools-webgrid-lighttpd-1209 sshd[14445]: Accepted hostbased for scfc from 10.68.17.228 port 38435 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
136 Jan 17 10:00:10 tools-webgrid-lighttpd-1209 sshd[14445]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
137 Jan 17 10:00:10 tools-webgrid-lighttpd-1209 sshd[14460]: Starting session: command for scfc from 10.68.17.228 port 38435
138 Jan 17 10:00:10 tools-webgrid-lighttpd-1209 sshd[14445]: pam_unix(sshd:session): session closed for user scfc
139 Jan 17 10:30:12 tools-webgrid-lighttpd-1209 sshd[17071]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
140 Jan 17 10:30:12 tools-webgrid-lighttpd-1209 sshd[17071]: Accepted hostbased for scfc from 10.68.17.228 port 38091 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
141 Jan 17 10:30:12 tools-webgrid-lighttpd-1209 sshd[17071]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
142 Jan 17 10:30:12 tools-webgrid-lighttpd-1209 sshd[17086]: Starting session: command for scfc from 10.68.17.228 port 38091
143 Jan 17 10:30:12 tools-webgrid-lighttpd-1209 sshd[17071]: pam_unix(sshd:session): session closed for user scfc
144 Jan 17 11:00:12 tools-webgrid-lighttpd-1209 sshd[19697]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
145 Jan 17 11:00:12 tools-webgrid-lighttpd-1209 sshd[19697]: Accepted hostbased for scfc from 10.68.17.228 port 36095 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
146 Jan 17 11:00:12 tools-webgrid-lighttpd-1209 sshd[19697]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
147 Jan 17 11:00:12 tools-webgrid-lighttpd-1209 sshd[19712]: Starting session: command for scfc from 10.68.17.228 port 36095
148 Jan 17 11:00:12 tools-webgrid-lighttpd-1209 sshd[19697]: pam_unix(sshd:session): session closed for user scfc
149 Jan 17 11:30:10 tools-webgrid-lighttpd-1209 sshd[22320]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
150 Jan 17 11:30:10 tools-webgrid-lighttpd-1209 sshd[22320]: Accepted hostbased for scfc from 10.68.17.228 port 33924 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
151 Jan 17 11:30:10 tools-webgrid-lighttpd-1209 sshd[22320]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
152 Jan 17 11:30:10 tools-webgrid-lighttpd-1209 sshd[22335]: Starting session: command for scfc from 10.68.17.228 port 33924
153 Jan 17 11:30:10 tools-webgrid-lighttpd-1209 sshd[22320]: pam_unix(sshd:session): session closed for user scfc
154 Jan 17 12:00:11 tools-webgrid-lighttpd-1209 sshd[24925]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
155 Jan 17 12:00:11 tools-webgrid-lighttpd-1209 sshd[24925]: Accepted hostbased for scfc from 10.68.17.228 port 39089 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
156 Jan 17 12:00:11 tools-webgrid-lighttpd-1209 sshd[24925]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
157 Jan 17 12:00:11 tools-webgrid-lighttpd-1209 sshd[24940]: Starting session: command for scfc from 10.68.17.228 port 39089
158 Jan 17 12:00:11 tools-webgrid-lighttpd-1209 sshd[24925]: pam_unix(sshd:session): session closed for user scfc
159 Jan 17 12:30:11 tools-webgrid-lighttpd-1209 sshd[27561]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
160 Jan 17 12:30:11 tools-webgrid-lighttpd-1209 sshd[27561]: Accepted hostbased for scfc from 10.68.17.228 port 56683 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
161 Jan 17 12:30:11 tools-webgrid-lighttpd-1209 sshd[27561]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
162 Jan 17 12:30:11 tools-webgrid-lighttpd-1209 sshd[27576]: Starting session: command for scfc from 10.68.17.228 port 56683
163 Jan 17 12:30:11 tools-webgrid-lighttpd-1209 sshd[27561]: pam_unix(sshd:session): session closed for user scfc
164 Jan 17 13:00:12 tools-webgrid-lighttpd-1209 sshd[30188]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
165 Jan 17 13:00:12 tools-webgrid-lighttpd-1209 sshd[30188]: Accepted hostbased for scfc from 10.68.17.228 port 55574 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
166 Jan 17 13:00:12 tools-webgrid-lighttpd-1209 sshd[30188]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
167 Jan 17 13:00:12 tools-webgrid-lighttpd-1209 sshd[30203]: Starting session: command for scfc from 10.68.17.228 port 55574
168 Jan 17 13:00:12 tools-webgrid-lighttpd-1209 sshd[30188]: pam_unix(sshd:session): session closed for user scfc
169 Jan 17 13:30:13 tools-webgrid-lighttpd-1209 sshd[386]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
170 Jan 17 13:30:13 tools-webgrid-lighttpd-1209 sshd[386]: Accepted hostbased for scfc from 10.68.17.228 port 33464 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
171 Jan 17 13:30:13 tools-webgrid-lighttpd-1209 sshd[386]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
172 Jan 17 13:30:13 tools-webgrid-lighttpd-1209 sshd[401]: Starting session: command for scfc from 10.68.17.228 port 33464
173 Jan 17 13:30:13 tools-webgrid-lighttpd-1209 sshd[386]: pam_unix(sshd:session): session closed for user scfc
174 Jan 17 14:00:12 tools-webgrid-lighttpd-1209 sshd[3107]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
175 Jan 17 14:00:12 tools-webgrid-lighttpd-1209 sshd[3107]: Accepted hostbased for scfc from 10.68.17.228 port 48518 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
176 Jan 17 14:00:12 tools-webgrid-lighttpd-1209 sshd[3107]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
177 Jan 17 14:00:12 tools-webgrid-lighttpd-1209 sshd[3122]: Starting session: command for scfc from 10.68.17.228 port 48518
178 Jan 17 14:00:12 tools-webgrid-lighttpd-1209 sshd[3107]: pam_unix(sshd:session): session closed for user scfc
179 Jan 17 14:09:20 tools-webgrid-lighttpd-1209 sshd[3261]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
180 Jan 17 14:09:20 tools-webgrid-lighttpd-1209 sshd[3261]: Accepted hostbased for scfc from 10.68.17.228 port 40261 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
181 Jan 17 14:09:20 tools-webgrid-lighttpd-1209 sshd[3261]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
182 Jan 17 14:09:20 tools-webgrid-lighttpd-1209 sshd[3276]: Starting session: command for scfc from 10.68.17.228 port 40261
183 Jan 17 14:09:20 tools-webgrid-lighttpd-1209 sshd[3261]: pam_unix(sshd:session): session closed for user scfc
184 Jan 17 14:30:15 tools-webgrid-lighttpd-1209 sshd[5754]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
185 Jan 17 14:30:15 tools-webgrid-lighttpd-1209 sshd[5754]: Accepted hostbased for scfc from 10.68.17.228 port 34070 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
186 Jan 17 14:30:15 tools-webgrid-lighttpd-1209 sshd[5754]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
187 Jan 17 14:30:15 tools-webgrid-lighttpd-1209 sshd[5769]: Starting session: command for scfc from 10.68.17.228 port 34070
188 Jan 17 14:30:15 tools-webgrid-lighttpd-1209 sshd[5754]: pam_unix(sshd:session): session closed for user scfc
189 Jan 17 15:00:10 tools-webgrid-lighttpd-1209 sshd[8440]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
190 Jan 17 15:00:10 tools-webgrid-lighttpd-1209 sshd[8440]: Accepted hostbased for scfc from 10.68.17.228 port 49644 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
191 Jan 17 15:00:10 tools-webgrid-lighttpd-1209 sshd[8440]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
192 Jan 17 15:00:10 tools-webgrid-lighttpd-1209 sshd[8455]: Starting session: command for scfc from 10.68.17.228 port 49644
193 Jan 17 15:00:10 tools-webgrid-lighttpd-1209 sshd[8440]: pam_unix(sshd:session): session closed for user scfc
194 Jan 17 15:30:11 tools-webgrid-lighttpd-1209 sshd[11238]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
195 Jan 17 15:30:11 tools-webgrid-lighttpd-1209 sshd[11238]: Accepted hostbased for scfc from 10.68.17.228 port 44763 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
196 Jan 17 15:30:11 tools-webgrid-lighttpd-1209 sshd[11238]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
197 Jan 17 15:30:11 tools-webgrid-lighttpd-1209 sshd[11253]: Starting session: command for scfc from 10.68.17.228 port 44763
198 Jan 17 15:30:11 tools-webgrid-lighttpd-1209 sshd[11238]: pam_unix(sshd:session): session closed for user scfc
199 Jan 17 16:00:16 tools-webgrid-lighttpd-1209 sshd[13903]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
200 Jan 17 16:00:16 tools-webgrid-lighttpd-1209 sshd[13903]: Accepted hostbased for scfc from 10.68.17.228 port 59315 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
201 Jan 17 16:00:16 tools-webgrid-lighttpd-1209 sshd[13903]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
202 Jan 17 16:00:17 tools-webgrid-lighttpd-1209 sshd[13918]: Starting session: command for scfc from 10.68.17.228 port 59315
203 Jan 17 16:00:17 tools-webgrid-lighttpd-1209 sshd[13903]: pam_unix(sshd:session): session closed for user scfc
204 Jan 17 16:12:22 tools-webgrid-lighttpd-1209 sshd[16050]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
205 Jan 17 16:12:22 tools-webgrid-lighttpd-1209 sshd[16050]: Accepted hostbased for scfc from 10.68.17.228 port 52468 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
206 Jan 17 16:12:22 tools-webgrid-lighttpd-1209 sshd[16050]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
207 Jan 17 16:12:22 tools-webgrid-lighttpd-1209 sshd[16073]: Starting session: command for scfc from 10.68.17.228 port 52468
208 Jan 17 16:12:22 tools-webgrid-lighttpd-1209 sshd[16050]: pam_unix(sshd:session): session closed for user scfc
209 Jan 17 16:30:12 tools-webgrid-lighttpd-1209 sshd[16564]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
210 Jan 17 16:30:12 tools-webgrid-lighttpd-1209 sshd[16564]: Accepted hostbased for scfc from 10.68.17.228 port 38518 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
211 Jan 17 16:30:12 tools-webgrid-lighttpd-1209 sshd[16564]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
212 Jan 17 16:30:12 tools-webgrid-lighttpd-1209 sshd[16579]: Starting session: command for scfc from 10.68.17.228 port 38518
213 Jan 17 16:30:12 tools-webgrid-lighttpd-1209 sshd[16564]: pam_unix(sshd:session): session closed for user scfc
214 Jan 17 17:00:10 tools-webgrid-lighttpd-1209 sshd[19201]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
215 Jan 17 17:00:10 tools-webgrid-lighttpd-1209 sshd[19201]: Accepted hostbased for scfc from 10.68.17.228 port 48427 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
216 Jan 17 17:00:10 tools-webgrid-lighttpd-1209 sshd[19201]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
217 Jan 17 17:00:10 tools-webgrid-lighttpd-1209 sshd[19216]: Starting session: command for scfc from 10.68.17.228 port 48427
218 Jan 17 17:00:10 tools-webgrid-lighttpd-1209 sshd[19201]: pam_unix(sshd:session): session closed for user scfc
219 Jan 17 17:30:12 tools-webgrid-lighttpd-1209 sshd[21834]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
220 Jan 17 17:30:12 tools-webgrid-lighttpd-1209 sshd[21834]: Accepted hostbased for scfc from 10.68.17.228 port 33226 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
221 Jan 17 17:30:12 tools-webgrid-lighttpd-1209 sshd[21834]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
222 Jan 17 17:30:12 tools-webgrid-lighttpd-1209 sshd[21849]: Starting session: command for scfc from 10.68.17.228 port 33226
223 Jan 17 17:30:12 tools-webgrid-lighttpd-1209 sshd[21834]: pam_unix(sshd:session): session closed for user scfc
224 Jan 17 18:00:11 tools-webgrid-lighttpd-1209 sshd[24467]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
225 Jan 17 18:00:11 tools-webgrid-lighttpd-1209 sshd[24467]: Accepted hostbased for scfc from 10.68.17.228 port 59790 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
226 Jan 17 18:00:11 tools-webgrid-lighttpd-1209 sshd[24467]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
227 Jan 17 18:00:11 tools-webgrid-lighttpd-1209 sshd[24482]: Starting session: command for scfc from 10.68.17.228 port 59790
228 Jan 17 18:00:11 tools-webgrid-lighttpd-1209 sshd[24467]: pam_unix(sshd:session): session closed for user scfc
229 Jan 17 18:30:11 tools-webgrid-lighttpd-1209 sshd[27106]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
230 Jan 17 18:30:11 tools-webgrid-lighttpd-1209 sshd[27106]: Accepted hostbased for scfc from 10.68.17.228 port 58624 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
231 Jan 17 18:30:11 tools-webgrid-lighttpd-1209 sshd[27106]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
232 Jan 17 18:30:11 tools-webgrid-lighttpd-1209 sshd[27121]: Starting session: command for scfc from 10.68.17.228 port 58624
233 Jan 17 18:30:11 tools-webgrid-lighttpd-1209 sshd[27106]: pam_unix(sshd:session): session closed for user scfc
234 Jan 17 19:00:12 tools-webgrid-lighttpd-1209 sshd[29732]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
235 Jan 17 19:00:12 tools-webgrid-lighttpd-1209 sshd[29732]: Accepted hostbased for scfc from 10.68.17.228 port 56829 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
236 Jan 17 19:00:12 tools-webgrid-lighttpd-1209 sshd[29732]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
237 Jan 17 19:00:12 tools-webgrid-lighttpd-1209 sshd[29747]: Starting session: command for scfc from 10.68.17.228 port 56829
238 Jan 17 19:00:12 tools-webgrid-lighttpd-1209 sshd[29732]: pam_unix(sshd:session): session closed for user scfc
239 Jan 17 19:30:11 tools-webgrid-lighttpd-1209 sshd[32361]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
240 Jan 17 19:30:11 tools-webgrid-lighttpd-1209 sshd[32361]: Accepted hostbased for scfc from 10.68.17.228 port 36138 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
241 Jan 17 19:30:11 tools-webgrid-lighttpd-1209 sshd[32361]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
242 Jan 17 19:30:11 tools-webgrid-lighttpd-1209 sshd[32376]: Starting session: command for scfc from 10.68.17.228 port 36138
243 Jan 17 19:30:11 tools-webgrid-lighttpd-1209 sshd[32361]: pam_unix(sshd:session): session closed for user scfc
244 Jan 17 20:00:11 tools-webgrid-lighttpd-1209 sshd[2620]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
245 Jan 17 20:00:11 tools-webgrid-lighttpd-1209 sshd[2620]: Accepted hostbased for scfc from 10.68.17.228 port 50466 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
246 Jan 17 20:00:11 tools-webgrid-lighttpd-1209 sshd[2620]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
247 Jan 17 20:00:11 tools-webgrid-lighttpd-1209 sshd[2635]: Starting session: command for scfc from 10.68.17.228 port 50466
248 Jan 17 20:00:12 tools-webgrid-lighttpd-1209 sshd[2620]: pam_unix(sshd:session): session closed for user scfc
249 Jan 17 20:30:12 tools-webgrid-lighttpd-1209 sshd[5278]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
250 Jan 17 20:30:12 tools-webgrid-lighttpd-1209 sshd[5278]: Accepted hostbased for scfc from 10.68.17.228 port 41457 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
251 Jan 17 20:30:12 tools-webgrid-lighttpd-1209 sshd[5278]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
252 Jan 17 20:30:12 tools-webgrid-lighttpd-1209 sshd[5294]: Starting session: command for scfc from 10.68.17.228 port 41457
253 Jan 17 20:30:12 tools-webgrid-lighttpd-1209 sshd[5278]: pam_unix(sshd:session): session closed for user scfc
254 Jan 17 21:00:16 tools-webgrid-lighttpd-1209 sshd[7982]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
255 Jan 17 21:00:16 tools-webgrid-lighttpd-1209 sshd[7982]: Accepted hostbased for scfc from 10.68.17.228 port 45640 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
256 Jan 17 21:00:16 tools-webgrid-lighttpd-1209 sshd[7982]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
257 Jan 17 21:00:16 tools-webgrid-lighttpd-1209 sshd[7997]: Starting session: command for scfc from 10.68.17.228 port 45640
258 Jan 17 21:00:16 tools-webgrid-lighttpd-1209 sshd[7982]: pam_unix(sshd:session): session closed for user scfc
259 Jan 17 21:30:13 tools-webgrid-lighttpd-1209 sshd[10772]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
260 Jan 17 21:30:13 tools-webgrid-lighttpd-1209 sshd[10772]: Accepted hostbased for scfc from 10.68.17.228 port 52523 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
261 Jan 17 21:30:13 tools-webgrid-lighttpd-1209 sshd[10772]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
262 Jan 17 21:30:13 tools-webgrid-lighttpd-1209 sshd[10787]: Starting session: command for scfc from 10.68.17.228 port 52523
263 Jan 17 21:30:13 tools-webgrid-lighttpd-1209 sshd[10772]: pam_unix(sshd:session): session closed for user scfc
264 Jan 17 22:00:12 tools-webgrid-lighttpd-1209 sshd[13442]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
265 Jan 17 22:00:12 tools-webgrid-lighttpd-1209 sshd[13442]: Accepted hostbased for scfc from 10.68.17.228 port 51592 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
266 Jan 17 22:00:12 tools-webgrid-lighttpd-1209 sshd[13442]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
267 Jan 17 22:00:12 tools-webgrid-lighttpd-1209 sshd[13457]: Starting session: command for scfc from 10.68.17.228 port 51592
268 Jan 17 22:00:12 tools-webgrid-lighttpd-1209 sshd[13442]: pam_unix(sshd:session): session closed for user scfc
269 Jan 17 22:30:10 tools-webgrid-lighttpd-1209 sshd[16058]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
270 Jan 17 22:30:10 tools-webgrid-lighttpd-1209 sshd[16058]: Accepted hostbased for scfc from 10.68.17.228 port 51017 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
271 Jan 17 22:30:10 tools-webgrid-lighttpd-1209 sshd[16058]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
272 Jan 17 22:30:11 tools-webgrid-lighttpd-1209 sshd[16073]: Starting session: command for scfc from 10.68.17.228 port 51017
273 Jan 17 22:30:11 tools-webgrid-lighttpd-1209 sshd[16058]: pam_unix(sshd:session): session closed for user scfc
274 Jan 17 23:00:10 tools-webgrid-lighttpd-1209 sshd[18683]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
275 Jan 17 23:00:10 tools-webgrid-lighttpd-1209 sshd[18683]: Accepted hostbased for scfc from 10.68.17.228 port 54692 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
276 Jan 17 23:00:10 tools-webgrid-lighttpd-1209 sshd[18683]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
277 Jan 17 23:00:10 tools-webgrid-lighttpd-1209 sshd[18698]: Starting session: command for scfc from 10.68.17.228 port 54692
278 Jan 17 23:00:10 tools-webgrid-lighttpd-1209 sshd[18683]: pam_unix(sshd:session): session closed for user scfc
279 Jan 17 23:30:10 tools-webgrid-lighttpd-1209 sshd[21306]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
280 Jan 17 23:30:10 tools-webgrid-lighttpd-1209 sshd[21306]: Accepted hostbased for scfc from 10.68.17.228 port 33380 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
281 Jan 17 23:30:10 tools-webgrid-lighttpd-1209 sshd[21306]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
282 Jan 17 23:30:10 tools-webgrid-lighttpd-1209 sshd[21321]: Starting session: command for scfc from 10.68.17.228 port 33380
283 Jan 17 23:30:10 tools-webgrid-lighttpd-1209 sshd[21306]: pam_unix(sshd:session): session closed for user scfc
284 Jan 18 00:00:11 tools-webgrid-lighttpd-1209 sshd[23960]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
285 Jan 18 00:00:11 tools-webgrid-lighttpd-1209 sshd[23960]: Accepted hostbased for scfc from 10.68.17.228 port 47038 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
286 Jan 18 00:00:11 tools-webgrid-lighttpd-1209 sshd[23960]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
287 Jan 18 00:00:11 tools-webgrid-lighttpd-1209 sshd[23975]: Starting session: command for scfc from 10.68.17.228 port 47038
288 Jan 18 00:00:11 tools-webgrid-lighttpd-1209 sshd[23960]: pam_unix(sshd:session): session closed for user scfc
289 Jan 18 00:30:12 tools-webgrid-lighttpd-1209 sshd[26573]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
290 Jan 18 00:30:12 tools-webgrid-lighttpd-1209 sshd[26573]: Accepted hostbased for scfc from 10.68.17.228 port 40769 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
291 Jan 18 00:30:12 tools-webgrid-lighttpd-1209 sshd[26573]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
292 Jan 18 00:30:12 tools-webgrid-lighttpd-1209 sshd[26588]: Starting session: command for scfc from 10.68.17.228 port 40769
293 Jan 18 00:30:12 tools-webgrid-lighttpd-1209 sshd[26573]: pam_unix(sshd:session): session closed for user scfc
294 Jan 18 01:00:10 tools-webgrid-lighttpd-1209 sshd[29194]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
295 Jan 18 01:00:10 tools-webgrid-lighttpd-1209 sshd[29194]: Accepted hostbased for scfc from 10.68.17.228 port 38105 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
296 Jan 18 01:00:10 tools-webgrid-lighttpd-1209 sshd[29194]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
297 Jan 18 01:00:10 tools-webgrid-lighttpd-1209 sshd[29209]: Starting session: command for scfc from 10.68.17.228 port 38105
298 Jan 18 01:00:10 tools-webgrid-lighttpd-1209 sshd[29194]: pam_unix(sshd:session): session closed for user scfc
299 Jan 18 01:30:11 tools-webgrid-lighttpd-1209 sshd[31829]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
300 Jan 18 01:30:11 tools-webgrid-lighttpd-1209 sshd[31829]: Accepted hostbased for scfc from 10.68.17.228 port 55524 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
301 Jan 18 01:30:11 tools-webgrid-lighttpd-1209 sshd[31829]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
302 Jan 18 01:30:12 tools-webgrid-lighttpd-1209 sshd[31844]: Starting session: command for scfc from 10.68.17.228 port 55524
303 Jan 18 01:30:12 tools-webgrid-lighttpd-1209 sshd[31829]: pam_unix(sshd:session): session closed for user scfc
304 Jan 18 02:00:12 tools-webgrid-lighttpd-1209 sshd[2078]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
305 Jan 18 02:00:13 tools-webgrid-lighttpd-1209 sshd[2078]: Accepted hostbased for scfc from 10.68.17.228 port 54055 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
306 Jan 18 02:00:13 tools-webgrid-lighttpd-1209 sshd[2078]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
307 Jan 18 02:00:13 tools-webgrid-lighttpd-1209 sshd[2093]: Starting session: command for scfc from 10.68.17.228 port 54055
308 Jan 18 02:00:13 tools-webgrid-lighttpd-1209 sshd[2078]: pam_unix(sshd:session): session closed for user scfc
309 Jan 18 02:30:10 tools-webgrid-lighttpd-1209 sshd[4700]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
310 Jan 18 02:30:10 tools-webgrid-lighttpd-1209 sshd[4700]: Accepted hostbased for scfc from 10.68.17.228 port 48134 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
311 Jan 18 02:30:10 tools-webgrid-lighttpd-1209 sshd[4700]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
312 Jan 18 02:30:10 tools-webgrid-lighttpd-1209 sshd[4719]: Starting session: command for scfc from 10.68.17.228 port 48134
313 Jan 18 02:30:10 tools-webgrid-lighttpd-1209 sshd[4700]: pam_unix(sshd:session): session closed for user scfc
314 Jan 18 03:00:10 tools-webgrid-lighttpd-1209 sshd[7373]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
315 Jan 18 03:00:11 tools-webgrid-lighttpd-1209 sshd[7373]: Accepted hostbased for scfc from 10.68.17.228 port 53048 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
316 Jan 18 03:00:11 tools-webgrid-lighttpd-1209 sshd[7373]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
317 Jan 18 03:00:11 tools-webgrid-lighttpd-1209 sshd[7390]: Starting session: command for scfc from 10.68.17.228 port 53048
318 Jan 18 03:00:11 tools-webgrid-lighttpd-1209 sshd[7373]: pam_unix(sshd:session): session closed for user scfc
319 Jan 18 03:30:10 tools-webgrid-lighttpd-1209 sshd[10206]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
320 Jan 18 03:30:10 tools-webgrid-lighttpd-1209 sshd[10206]: Accepted hostbased for scfc from 10.68.17.228 port 57728 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
321 Jan 18 03:30:10 tools-webgrid-lighttpd-1209 sshd[10206]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
322 Jan 18 03:30:10 tools-webgrid-lighttpd-1209 sshd[10221]: Starting session: command for scfc from 10.68.17.228 port 57728
323 Jan 18 03:30:10 tools-webgrid-lighttpd-1209 sshd[10206]: pam_unix(sshd:session): session closed for user scfc
324 Jan 18 04:00:09 tools-webgrid-lighttpd-1209 sshd[12855]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
325 Jan 18 04:00:09 tools-webgrid-lighttpd-1209 sshd[12855]: Accepted hostbased for scfc from 10.68.17.228 port 34111 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
326 Jan 18 04:00:09 tools-webgrid-lighttpd-1209 sshd[12855]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
327 Jan 18 04:00:09 tools-webgrid-lighttpd-1209 sshd[12870]: Starting session: command for scfc from 10.68.17.228 port 34111
328 Jan 18 04:00:09 tools-webgrid-lighttpd-1209 sshd[12855]: pam_unix(sshd:session): session closed for user scfc
329 Jan 18 04:30:11 tools-webgrid-lighttpd-1209 sshd[15477]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
330 Jan 18 04:30:11 tools-webgrid-lighttpd-1209 sshd[15477]: Accepted hostbased for scfc from 10.68.17.228 port 38764 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
331 Jan 18 04:30:11 tools-webgrid-lighttpd-1209 sshd[15477]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
332 Jan 18 04:30:11 tools-webgrid-lighttpd-1209 sshd[15492]: Starting session: command for scfc from 10.68.17.228 port 38764
333 Jan 18 04:30:11 tools-webgrid-lighttpd-1209 sshd[15477]: pam_unix(sshd:session): session closed for user scfc
334 Jan 18 05:00:08 tools-webgrid-lighttpd-1209 sshd[18096]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
335 Jan 18 05:00:08 tools-webgrid-lighttpd-1209 sshd[18096]: Accepted hostbased for scfc from 10.68.17.228 port 43565 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
336 Jan 18 05:00:08 tools-webgrid-lighttpd-1209 sshd[18096]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
337 Jan 18 05:00:08 tools-webgrid-lighttpd-1209 sshd[18111]: Starting session: command for scfc from 10.68.17.228 port 43565
338 Jan 18 05:00:08 tools-webgrid-lighttpd-1209 sshd[18096]: pam_unix(sshd:session): session closed for user scfc
339 Jan 18 05:30:09 tools-webgrid-lighttpd-1209 sshd[20729]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
340 Jan 18 05:30:09 tools-webgrid-lighttpd-1209 sshd[20729]: Accepted hostbased for scfc from 10.68.17.228 port 48326 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
341 Jan 18 05:30:09 tools-webgrid-lighttpd-1209 sshd[20729]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
342 Jan 18 05:30:09 tools-webgrid-lighttpd-1209 sshd[20744]: Starting session: command for scfc from 10.68.17.228 port 48326
343 Jan 18 05:30:09 tools-webgrid-lighttpd-1209 sshd[20729]: pam_unix(sshd:session): session closed for user scfc
344 Jan 18 06:00:12 tools-webgrid-lighttpd-1209 sshd[23339]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
345 Jan 18 06:00:12 tools-webgrid-lighttpd-1209 sshd[23339]: Accepted hostbased for scfc from 10.68.17.228 port 52994 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
346 Jan 18 06:00:12 tools-webgrid-lighttpd-1209 sshd[23339]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
347 Jan 18 06:00:12 tools-webgrid-lighttpd-1209 sshd[23354]: Starting session: command for scfc from 10.68.17.228 port 52994
348 Jan 18 06:00:12 tools-webgrid-lighttpd-1209 sshd[23339]: pam_unix(sshd:session): session closed for user scfc
349 Jan 18 06:30:11 tools-webgrid-lighttpd-1209 sshd[26040]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
350 Jan 18 06:30:11 tools-webgrid-lighttpd-1209 sshd[26040]: Accepted hostbased for scfc from 10.68.17.228 port 57554 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
351 Jan 18 06:30:11 tools-webgrid-lighttpd-1209 sshd[26040]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
352 Jan 18 06:30:11 tools-webgrid-lighttpd-1209 sshd[26055]: Starting session: command for scfc from 10.68.17.228 port 57554
353 Jan 18 06:30:11 tools-webgrid-lighttpd-1209 sshd[26040]: pam_unix(sshd:session): session closed for user scfc
354 Jan 18 07:00:10 tools-webgrid-lighttpd-1209 sshd[28838]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
355 Jan 18 07:00:10 tools-webgrid-lighttpd-1209 sshd[28838]: Accepted hostbased for scfc from 10.68.17.228 port 33528 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
356 Jan 18 07:00:10 tools-webgrid-lighttpd-1209 sshd[28838]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
357 Jan 18 07:00:10 tools-webgrid-lighttpd-1209 sshd[28853]: Starting session: command for scfc from 10.68.17.228 port 33528
358 Jan 18 07:00:11 tools-webgrid-lighttpd-1209 sshd[28838]: pam_unix(sshd:session): session closed for user scfc
359 Jan 18 07:30:10 tools-webgrid-lighttpd-1209 sshd[31454]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
360 Jan 18 07:30:11 tools-webgrid-lighttpd-1209 sshd[31454]: Accepted hostbased for scfc from 10.68.17.228 port 37928 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
361 Jan 18 07:30:11 tools-webgrid-lighttpd-1209 sshd[31454]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
362 Jan 18 07:30:11 tools-webgrid-lighttpd-1209 sshd[31469]: Starting session: command for scfc from 10.68.17.228 port 37928
363 Jan 18 07:30:11 tools-webgrid-lighttpd-1209 sshd[31454]: pam_unix(sshd:session): session closed for user scfc
364 Jan 18 08:00:10 tools-webgrid-lighttpd-1209 sshd[1696]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
365 Jan 18 08:00:10 tools-webgrid-lighttpd-1209 sshd[1696]: Accepted hostbased for scfc from 10.68.17.228 port 42393 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
366 Jan 18 08:00:10 tools-webgrid-lighttpd-1209 sshd[1696]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
367 Jan 18 08:00:10 tools-webgrid-lighttpd-1209 sshd[1711]: Starting session: command for scfc from 10.68.17.228 port 42393
368 Jan 18 08:00:10 tools-webgrid-lighttpd-1209 sshd[1696]: pam_unix(sshd:session): session closed for user scfc
369 Jan 18 08:30:10 tools-webgrid-lighttpd-1209 sshd[4414]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
370 Jan 18 08:30:10 tools-webgrid-lighttpd-1209 sshd[4414]: Accepted hostbased for scfc from 10.68.17.228 port 46935 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
371 Jan 18 08:30:10 tools-webgrid-lighttpd-1209 sshd[4414]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
372 Jan 18 08:30:11 tools-webgrid-lighttpd-1209 sshd[4429]: Starting session: command for scfc from 10.68.17.228 port 46935
373 Jan 18 08:30:11 tools-webgrid-lighttpd-1209 sshd[4414]: pam_unix(sshd:session): session closed for user scfc
374 Jan 18 09:00:14 tools-webgrid-lighttpd-1209 sshd[7192]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
375 Jan 18 09:00:14 tools-webgrid-lighttpd-1209 sshd[7192]: Accepted hostbased for scfc from 10.68.17.228 port 51505 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
376 Jan 18 09:00:14 tools-webgrid-lighttpd-1209 sshd[7192]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
377 Jan 18 09:00:14 tools-webgrid-lighttpd-1209 sshd[7207]: Starting session: command for scfc from 10.68.17.228 port 51505
378 Jan 18 09:00:14 tools-webgrid-lighttpd-1209 sshd[7192]: pam_unix(sshd:session): session closed for user scfc
379 Jan 18 09:30:17 tools-webgrid-lighttpd-1209 sshd[10015]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
380 Jan 18 09:30:17 tools-webgrid-lighttpd-1209 sshd[10015]: Accepted hostbased for scfc from 10.68.17.228 port 56563 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
381 Jan 18 09:30:17 tools-webgrid-lighttpd-1209 sshd[10015]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
382 Jan 18 09:30:17 tools-webgrid-lighttpd-1209 sshd[10032]: Starting session: command for scfc from 10.68.17.228 port 56563
383 Jan 18 09:30:17 tools-webgrid-lighttpd-1209 sshd[10015]: pam_unix(sshd:session): session closed for user scfc
384 Jan 18 10:00:10 tools-webgrid-lighttpd-1209 sshd[12682]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
385 Jan 18 10:00:10 tools-webgrid-lighttpd-1209 sshd[12682]: Accepted hostbased for scfc from 10.68.17.228 port 32878 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
386 Jan 18 10:00:10 tools-webgrid-lighttpd-1209 sshd[12682]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
387 Jan 18 10:00:11 tools-webgrid-lighttpd-1209 sshd[12697]: Starting session: command for scfc from 10.68.17.228 port 32878
388 Jan 18 10:00:11 tools-webgrid-lighttpd-1209 sshd[12682]: pam_unix(sshd:session): session closed for user scfc
389 Jan 18 10:30:11 tools-webgrid-lighttpd-1209 sshd[15299]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
390 Jan 18 10:30:11 tools-webgrid-lighttpd-1209 sshd[15299]: Accepted hostbased for scfc from 10.68.17.228 port 37233 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
391 Jan 18 10:30:11 tools-webgrid-lighttpd-1209 sshd[15299]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
392 Jan 18 10:30:11 tools-webgrid-lighttpd-1209 sshd[15314]: Starting session: command for scfc from 10.68.17.228 port 37233
393 Jan 18 10:30:11 tools-webgrid-lighttpd-1209 sshd[15299]: pam_unix(sshd:session): session closed for user scfc
394 Jan 18 11:00:14 tools-webgrid-lighttpd-1209 sshd[17899]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
395 Jan 18 11:00:14 tools-webgrid-lighttpd-1209 sshd[17899]: Accepted hostbased for scfc from 10.68.17.228 port 41626 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
396 Jan 18 11:00:14 tools-webgrid-lighttpd-1209 sshd[17899]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
397 Jan 18 11:00:14 tools-webgrid-lighttpd-1209 sshd[17914]: Starting session: command for scfc from 10.68.17.228 port 41626
398 Jan 18 11:00:14 tools-webgrid-lighttpd-1209 sshd[17899]: pam_unix(sshd:session): session closed for user scfc
399 Jan 18 11:30:10 tools-webgrid-lighttpd-1209 sshd[20528]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
400 Jan 18 11:30:10 tools-webgrid-lighttpd-1209 sshd[20528]: Accepted hostbased for scfc from 10.68.17.228 port 46211 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
401 Jan 18 11:30:10 tools-webgrid-lighttpd-1209 sshd[20528]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
402 Jan 18 11:30:10 tools-webgrid-lighttpd-1209 sshd[20543]: Starting session: command for scfc from 10.68.17.228 port 46211
403 Jan 18 11:30:10 tools-webgrid-lighttpd-1209 sshd[20528]: pam_unix(sshd:session): session closed for user scfc
404 Jan 18 12:00:12 tools-webgrid-lighttpd-1209 sshd[23131]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
405 Jan 18 12:00:12 tools-webgrid-lighttpd-1209 sshd[23131]: Accepted hostbased for scfc from 10.68.17.228 port 50709 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
406 Jan 18 12:00:12 tools-webgrid-lighttpd-1209 sshd[23131]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
407 Jan 18 12:00:12 tools-webgrid-lighttpd-1209 sshd[23147]: Starting session: command for scfc from 10.68.17.228 port 50709
408 Jan 18 12:00:12 tools-webgrid-lighttpd-1209 sshd[23131]: pam_unix(sshd:session): session closed for user scfc
409 Jan 18 12:30:09 tools-webgrid-lighttpd-1209 sshd[25764]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
410 Jan 18 12:30:09 tools-webgrid-lighttpd-1209 sshd[25764]: Accepted hostbased for scfc from 10.68.17.228 port 56256 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
411 Jan 18 12:30:09 tools-webgrid-lighttpd-1209 sshd[25764]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
412 Jan 18 12:30:09 tools-webgrid-lighttpd-1209 sshd[25779]: Starting session: command for scfc from 10.68.17.228 port 56256
413 Jan 18 12:30:09 tools-webgrid-lighttpd-1209 sshd[25764]: pam_unix(sshd:session): session closed for user scfc
414 Jan 18 13:00:11 tools-webgrid-lighttpd-1209 sshd[28390]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
415 Jan 18 13:00:11 tools-webgrid-lighttpd-1209 sshd[28390]: Accepted hostbased for scfc from 10.68.17.228 port 37144 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
416 Jan 18 13:00:11 tools-webgrid-lighttpd-1209 sshd[28390]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
417 Jan 18 13:00:11 tools-webgrid-lighttpd-1209 sshd[28405]: Starting session: command for scfc from 10.68.17.228 port 37144
418 Jan 18 13:00:11 tools-webgrid-lighttpd-1209 sshd[28390]: pam_unix(sshd:session): session closed for user scfc
419 Jan 18 13:30:10 tools-webgrid-lighttpd-1209 sshd[31035]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
420 Jan 18 13:30:10 tools-webgrid-lighttpd-1209 sshd[31035]: Accepted hostbased for scfc from 10.68.17.228 port 60233 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
421 Jan 18 13:30:10 tools-webgrid-lighttpd-1209 sshd[31035]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
422 Jan 18 13:30:10 tools-webgrid-lighttpd-1209 sshd[31050]: Starting session: command for scfc from 10.68.17.228 port 60233
423 Jan 18 13:30:10 tools-webgrid-lighttpd-1209 sshd[31035]: pam_unix(sshd:session): session closed for user scfc
424 Jan 18 14:00:11 tools-webgrid-lighttpd-1209 sshd[1285]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
425 Jan 18 14:00:11 tools-webgrid-lighttpd-1209 sshd[1285]: Accepted hostbased for scfc from 10.68.17.228 port 34682 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
426 Jan 18 14:00:11 tools-webgrid-lighttpd-1209 sshd[1285]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
427 Jan 18 14:00:11 tools-webgrid-lighttpd-1209 sshd[1300]: Starting session: command for scfc from 10.68.17.228 port 34682
428 Jan 18 14:00:11 tools-webgrid-lighttpd-1209 sshd[1285]: pam_unix(sshd:session): session closed for user scfc
429 Jan 18 14:30:11 tools-webgrid-lighttpd-1209 sshd[3922]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
430 Jan 18 14:30:11 tools-webgrid-lighttpd-1209 sshd[3922]: Accepted hostbased for scfc from 10.68.17.228 port 40501 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
431 Jan 18 14:30:11 tools-webgrid-lighttpd-1209 sshd[3922]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
432 Jan 18 14:30:11 tools-webgrid-lighttpd-1209 sshd[3937]: Starting session: command for scfc from 10.68.17.228 port 40501
433 Jan 18 14:30:11 tools-webgrid-lighttpd-1209 sshd[3922]: pam_unix(sshd:session): session closed for user scfc
434 Jan 18 15:00:10 tools-webgrid-lighttpd-1209 sshd[6533]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
435 Jan 18 15:00:10 tools-webgrid-lighttpd-1209 sshd[6533]: Accepted hostbased for scfc from 10.68.17.228 port 51275 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
436 Jan 18 15:00:10 tools-webgrid-lighttpd-1209 sshd[6533]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
437 Jan 18 15:00:10 tools-webgrid-lighttpd-1209 sshd[6548]: Starting session: command for scfc from 10.68.17.228 port 51275
438 Jan 18 15:00:11 tools-webgrid-lighttpd-1209 sshd[6533]: pam_unix(sshd:session): session closed for user scfc
439 Jan 18 15:30:11 tools-webgrid-lighttpd-1209 sshd[9262]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
440 Jan 18 15:30:11 tools-webgrid-lighttpd-1209 sshd[9262]: Accepted hostbased for scfc from 10.68.17.228 port 54269 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
441 Jan 18 15:30:11 tools-webgrid-lighttpd-1209 sshd[9262]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
442 Jan 18 15:30:11 tools-webgrid-lighttpd-1209 sshd[9277]: Starting session: command for scfc from 10.68.17.228 port 54269
443 Jan 18 15:30:11 tools-webgrid-lighttpd-1209 sshd[9262]: pam_unix(sshd:session): session closed for user scfc
444 Jan 18 16:00:11 tools-webgrid-lighttpd-1209 sshd[12049]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
445 Jan 18 16:00:11 tools-webgrid-lighttpd-1209 sshd[12049]: Accepted hostbased for scfc from 10.68.17.228 port 58231 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
446 Jan 18 16:00:11 tools-webgrid-lighttpd-1209 sshd[12049]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
447 Jan 18 16:00:11 tools-webgrid-lighttpd-1209 sshd[12064]: Starting session: command for scfc from 10.68.17.228 port 58231
448 Jan 18 16:00:11 tools-webgrid-lighttpd-1209 sshd[12049]: pam_unix(sshd:session): session closed for user scfc
449 Jan 18 16:30:13 tools-webgrid-lighttpd-1209 sshd[14682]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
450 Jan 18 16:30:13 tools-webgrid-lighttpd-1209 sshd[14682]: Accepted hostbased for scfc from 10.68.17.228 port 42506 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
451 Jan 18 16:30:13 tools-webgrid-lighttpd-1209 sshd[14682]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
452 Jan 18 16:30:14 tools-webgrid-lighttpd-1209 sshd[14697]: Starting session: command for scfc from 10.68.17.228 port 42506
453 Jan 18 16:30:14 tools-webgrid-lighttpd-1209 sshd[14682]: pam_unix(sshd:session): session closed for user scfc
454 Jan 18 17:00:12 tools-webgrid-lighttpd-1209 sshd[17295]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
455 Jan 18 17:00:12 tools-webgrid-lighttpd-1209 sshd[17295]: Accepted hostbased for scfc from 10.68.17.228 port 35892 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
456 Jan 18 17:00:12 tools-webgrid-lighttpd-1209 sshd[17295]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
457 Jan 18 17:00:13 tools-webgrid-lighttpd-1209 sshd[17310]: Starting session: command for scfc from 10.68.17.228 port 35892
458 Jan 18 17:00:13 tools-webgrid-lighttpd-1209 sshd[17295]: pam_unix(sshd:session): session closed for user scfc
459 Jan 18 17:30:10 tools-webgrid-lighttpd-1209 sshd[19923]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
460 Jan 18 17:30:10 tools-webgrid-lighttpd-1209 sshd[19923]: Accepted hostbased for scfc from 10.68.17.228 port 46620 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
461 Jan 18 17:30:10 tools-webgrid-lighttpd-1209 sshd[19923]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
462 Jan 18 17:30:11 tools-webgrid-lighttpd-1209 sshd[19938]: Starting session: command for scfc from 10.68.17.228 port 46620
463 Jan 18 17:30:11 tools-webgrid-lighttpd-1209 sshd[19923]: pam_unix(sshd:session): session closed for user scfc
464 Jan 18 18:00:12 tools-webgrid-lighttpd-1209 sshd[22525]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
465 Jan 18 18:00:12 tools-webgrid-lighttpd-1209 sshd[22525]: Accepted hostbased for scfc from 10.68.17.228 port 52268 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
466 Jan 18 18:00:12 tools-webgrid-lighttpd-1209 sshd[22525]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
467 Jan 18 18:00:12 tools-webgrid-lighttpd-1209 sshd[22540]: Starting session: command for scfc from 10.68.17.228 port 52268
468 Jan 18 18:00:12 tools-webgrid-lighttpd-1209 sshd[22525]: pam_unix(sshd:session): session closed for user scfc
469 Jan 18 18:30:10 tools-webgrid-lighttpd-1209 sshd[25156]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
470 Jan 18 18:30:10 tools-webgrid-lighttpd-1209 sshd[25156]: Accepted hostbased for scfc from 10.68.17.228 port 55129 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
471 Jan 18 18:30:10 tools-webgrid-lighttpd-1209 sshd[25156]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
472 Jan 18 18:30:10 tools-webgrid-lighttpd-1209 sshd[25171]: Starting session: command for scfc from 10.68.17.228 port 55129
473 Jan 18 18:30:10 tools-webgrid-lighttpd-1209 sshd[25156]: pam_unix(sshd:session): session closed for user scfc
474 Jan 18 19:00:11 tools-webgrid-lighttpd-1209 sshd[27784]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
475 Jan 18 19:00:11 tools-webgrid-lighttpd-1209 sshd[27784]: Accepted hostbased for scfc from 10.68.17.228 port 57869 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
476 Jan 18 19:00:11 tools-webgrid-lighttpd-1209 sshd[27784]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
477 Jan 18 19:00:11 tools-webgrid-lighttpd-1209 sshd[27799]: Starting session: command for scfc from 10.68.17.228 port 57869
478 Jan 18 19:00:11 tools-webgrid-lighttpd-1209 sshd[27784]: pam_unix(sshd:session): session closed for user scfc
479 Jan 18 19:30:10 tools-webgrid-lighttpd-1209 sshd[30427]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
480 Jan 18 19:30:10 tools-webgrid-lighttpd-1209 sshd[30427]: Accepted hostbased for scfc from 10.68.17.228 port 60536 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
481 Jan 18 19:30:10 tools-webgrid-lighttpd-1209 sshd[30427]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
482 Jan 18 19:30:10 tools-webgrid-lighttpd-1209 sshd[30442]: Starting session: command for scfc from 10.68.17.228 port 60536
483 Jan 18 19:30:10 tools-webgrid-lighttpd-1209 sshd[30427]: pam_unix(sshd:session): session closed for user scfc
484 Jan 18 20:00:11 tools-webgrid-lighttpd-1209 sshd[592]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
485 Jan 18 20:00:11 tools-webgrid-lighttpd-1209 sshd[592]: Accepted hostbased for scfc from 10.68.17.228 port 34899 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
486 Jan 18 20:00:11 tools-webgrid-lighttpd-1209 sshd[592]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
487 Jan 18 20:00:11 tools-webgrid-lighttpd-1209 sshd[607]: Starting session: command for scfc from 10.68.17.228 port 34899
488 Jan 18 20:00:11 tools-webgrid-lighttpd-1209 sshd[592]: pam_unix(sshd:session): session closed for user scfc
489 Jan 18 20:30:10 tools-webgrid-lighttpd-1209 sshd[3313]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
490 Jan 18 20:30:10 tools-webgrid-lighttpd-1209 sshd[3313]: Accepted hostbased for scfc from 10.68.17.228 port 47623 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
491 Jan 18 20:30:10 tools-webgrid-lighttpd-1209 sshd[3313]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
492 Jan 18 20:30:10 tools-webgrid-lighttpd-1209 sshd[3328]: Starting session: command for scfc from 10.68.17.228 port 47623
493 Jan 18 20:30:10 tools-webgrid-lighttpd-1209 sshd[3313]: pam_unix(sshd:session): session closed for user scfc
494 Jan 18 21:00:12 tools-webgrid-lighttpd-1209 sshd[5927]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
495 Jan 18 21:00:12 tools-webgrid-lighttpd-1209 sshd[5927]: Accepted hostbased for scfc from 10.68.17.228 port 48176 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
496 Jan 18 21:00:12 tools-webgrid-lighttpd-1209 sshd[5927]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
497 Jan 18 21:00:12 tools-webgrid-lighttpd-1209 sshd[5942]: Starting session: command for scfc from 10.68.17.228 port 48176
498 Jan 18 21:00:12 tools-webgrid-lighttpd-1209 sshd[5927]: pam_unix(sshd:session): session closed for user scfc
499 Jan 18 21:30:10 tools-webgrid-lighttpd-1209 sshd[8673]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
500 Jan 18 21:30:10 tools-webgrid-lighttpd-1209 sshd[8673]: Accepted hostbased for scfc from 10.68.17.228 port 60843 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
501 Jan 18 21:30:10 tools-webgrid-lighttpd-1209 sshd[8673]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
502 Jan 18 21:30:10 tools-webgrid-lighttpd-1209 sshd[8688]: Starting session: command for scfc from 10.68.17.228 port 60843
503 Jan 18 21:30:10 tools-webgrid-lighttpd-1209 sshd[8673]: pam_unix(sshd:session): session closed for user scfc
504 Jan 18 22:00:11 tools-webgrid-lighttpd-1209 sshd[11428]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
505 Jan 18 22:00:11 tools-webgrid-lighttpd-1209 sshd[11428]: Accepted hostbased for scfc from 10.68.17.228 port 34590 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
506 Jan 18 22:00:11 tools-webgrid-lighttpd-1209 sshd[11428]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
507 Jan 18 22:00:11 tools-webgrid-lighttpd-1209 sshd[11443]: Starting session: command for scfc from 10.68.17.228 port 34590
508 Jan 18 22:00:11 tools-webgrid-lighttpd-1209 sshd[11428]: pam_unix(sshd:session): session closed for user scfc
509 Jan 18 22:30:10 tools-webgrid-lighttpd-1209 sshd[14120]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
510 Jan 18 22:30:10 tools-webgrid-lighttpd-1209 sshd[14120]: Accepted hostbased for scfc from 10.68.17.228 port 35073 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
511 Jan 18 22:30:10 tools-webgrid-lighttpd-1209 sshd[14120]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
512 Jan 18 22:30:10 tools-webgrid-lighttpd-1209 sshd[14135]: Starting session: command for scfc from 10.68.17.228 port 35073
513 Jan 18 22:30:10 tools-webgrid-lighttpd-1209 sshd[14120]: pam_unix(sshd:session): session closed for user scfc
514 Jan 18 22:33:29 tools-webgrid-lighttpd-1209 sshd[14181]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
515 Jan 18 22:33:30 tools-webgrid-lighttpd-1209 sshd[14181]: Accepted hostbased for scfc from 10.68.17.228 port 44228 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
516 Jan 18 22:33:30 tools-webgrid-lighttpd-1209 sshd[14181]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
517 Jan 18 22:33:30 tools-webgrid-lighttpd-1209 sshd[14196]: Starting session: command for scfc from 10.68.17.228 port 44228
518 Jan 18 22:33:30 tools-webgrid-lighttpd-1209 sshd[14181]: pam_unix(sshd:session): session closed for user scfc
519 Jan 18 23:00:10 tools-webgrid-lighttpd-1209 sshd[16746]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
520 Jan 18 23:00:10 tools-webgrid-lighttpd-1209 sshd[16746]: Accepted hostbased for scfc from 10.68.17.228 port 39811 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
521 Jan 18 23:00:10 tools-webgrid-lighttpd-1209 sshd[16746]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
522 Jan 18 23:00:10 tools-webgrid-lighttpd-1209 sshd[16761]: Starting session: command for scfc from 10.68.17.228 port 39811
523 Jan 18 23:00:10 tools-webgrid-lighttpd-1209 sshd[16746]: pam_unix(sshd:session): session closed for user scfc
524 Jan 18 23:30:09 tools-webgrid-lighttpd-1209 sshd[19346]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
525 Jan 18 23:30:09 tools-webgrid-lighttpd-1209 sshd[19346]: Accepted hostbased for scfc from 10.68.17.228 port 43068 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
526 Jan 18 23:30:09 tools-webgrid-lighttpd-1209 sshd[19346]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
527 Jan 18 23:30:09 tools-webgrid-lighttpd-1209 sshd[19361]: Starting session: command for scfc from 10.68.17.228 port 43068
528 Jan 18 23:30:09 tools-webgrid-lighttpd-1209 sshd[19346]: pam_unix(sshd:session): session closed for user scfc
529 Jan 19 00:00:10 tools-webgrid-lighttpd-1209 sshd[21970]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
530 Jan 19 00:00:10 tools-webgrid-lighttpd-1209 sshd[21970]: Accepted hostbased for scfc from 10.68.17.228 port 39017 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
531 Jan 19 00:00:10 tools-webgrid-lighttpd-1209 sshd[21970]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
532 Jan 19 00:00:10 tools-webgrid-lighttpd-1209 sshd[21985]: Starting session: command for scfc from 10.68.17.228 port 39017
533 Jan 19 00:00:10 tools-webgrid-lighttpd-1209 sshd[21970]: pam_unix(sshd:session): session closed for user scfc
534 Jan 19 00:30:14 tools-webgrid-lighttpd-1209 sshd[24583]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
535 Jan 19 00:30:14 tools-webgrid-lighttpd-1209 sshd[24583]: Accepted hostbased for scfc from 10.68.17.228 port 56254 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
536 Jan 19 00:30:14 tools-webgrid-lighttpd-1209 sshd[24583]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
537 Jan 19 00:30:14 tools-webgrid-lighttpd-1209 sshd[24598]: Starting session: command for scfc from 10.68.17.228 port 56254
538 Jan 19 00:30:14 tools-webgrid-lighttpd-1209 sshd[24583]: pam_unix(sshd:session): session closed for user scfc
539 Jan 19 01:00:15 tools-webgrid-lighttpd-1209 sshd[27204]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
540 Jan 19 01:00:15 tools-webgrid-lighttpd-1209 sshd[27204]: Accepted hostbased for scfc from 10.68.17.228 port 54802 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
541 Jan 19 01:00:15 tools-webgrid-lighttpd-1209 sshd[27204]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
542 Jan 19 01:00:15 tools-webgrid-lighttpd-1209 sshd[27219]: Starting session: command for scfc from 10.68.17.228 port 54802
543 Jan 19 01:00:15 tools-webgrid-lighttpd-1209 sshd[27204]: pam_unix(sshd:session): session closed for user scfc
544 Jan 19 01:30:11 tools-webgrid-lighttpd-1209 sshd[29818]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
545 Jan 19 01:30:11 tools-webgrid-lighttpd-1209 sshd[29818]: Accepted hostbased for scfc from 10.68.17.228 port 42642 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
546 Jan 19 01:30:11 tools-webgrid-lighttpd-1209 sshd[29818]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
547 Jan 19 01:30:11 tools-webgrid-lighttpd-1209 sshd[29834]: Starting session: command for scfc from 10.68.17.228 port 42642
548 Jan 19 01:30:11 tools-webgrid-lighttpd-1209 sshd[29818]: pam_unix(sshd:session): session closed for user scfc
549 Jan 19 02:00:09 tools-webgrid-lighttpd-1209 sshd[32427]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
550 Jan 19 02:00:09 tools-webgrid-lighttpd-1209 sshd[32427]: Accepted hostbased for scfc from 10.68.17.228 port 42710 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
551 Jan 19 02:00:09 tools-webgrid-lighttpd-1209 sshd[32427]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
552 Jan 19 02:00:09 tools-webgrid-lighttpd-1209 sshd[32442]: Starting session: command for scfc from 10.68.17.228 port 42710
553 Jan 19 02:00:09 tools-webgrid-lighttpd-1209 sshd[32427]: pam_unix(sshd:session): session closed for user scfc
554 Jan 19 02:30:11 tools-webgrid-lighttpd-1209 sshd[2663]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
555 Jan 19 02:30:11 tools-webgrid-lighttpd-1209 sshd[2663]: Accepted hostbased for scfc from 10.68.17.228 port 46546 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
556 Jan 19 02:30:11 tools-webgrid-lighttpd-1209 sshd[2663]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
557 Jan 19 02:30:11 tools-webgrid-lighttpd-1209 sshd[2688]: Starting session: command for scfc from 10.68.17.228 port 46546
558 Jan 19 02:30:11 tools-webgrid-lighttpd-1209 sshd[2663]: pam_unix(sshd:session): session closed for user scfc
559 Jan 19 03:00:10 tools-webgrid-lighttpd-1209 sshd[5285]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
560 Jan 19 03:00:10 tools-webgrid-lighttpd-1209 sshd[5285]: Accepted hostbased for scfc from 10.68.17.228 port 46731 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
561 Jan 19 03:00:10 tools-webgrid-lighttpd-1209 sshd[5285]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
562 Jan 19 03:00:10 tools-webgrid-lighttpd-1209 sshd[5300]: Starting session: command for scfc from 10.68.17.228 port 46731
563 Jan 19 03:00:10 tools-webgrid-lighttpd-1209 sshd[5285]: pam_unix(sshd:session): session closed for user scfc
564 Jan 19 03:30:09 tools-webgrid-lighttpd-1209 sshd[7992]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
565 Jan 19 03:30:09 tools-webgrid-lighttpd-1209 sshd[7992]: Accepted hostbased for scfc from 10.68.17.228 port 38177 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
566 Jan 19 03:30:09 tools-webgrid-lighttpd-1209 sshd[7992]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
567 Jan 19 03:30:09 tools-webgrid-lighttpd-1209 sshd[8007]: Starting session: command for scfc from 10.68.17.228 port 38177
568 Jan 19 03:30:09 tools-webgrid-lighttpd-1209 sshd[7992]: pam_unix(sshd:session): session closed for user scfc
569 Jan 19 04:00:51 tools-webgrid-lighttpd-1209 sshd[10758]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
570 Jan 19 04:00:51 tools-webgrid-lighttpd-1209 sshd[10758]: Accepted hostbased for scfc from 10.68.17.228 port 35906 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
571 Jan 19 04:00:51 tools-webgrid-lighttpd-1209 sshd[10758]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
572 Jan 19 04:00:51 tools-webgrid-lighttpd-1209 sshd[10773]: Starting session: command for scfc from 10.68.17.228 port 35906
573 Jan 19 04:00:51 tools-webgrid-lighttpd-1209 sshd[10758]: pam_unix(sshd:session): session closed for user scfc
574 Jan 19 04:30:09 tools-webgrid-lighttpd-1209 sshd[13405]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
575 Jan 19 04:30:09 tools-webgrid-lighttpd-1209 sshd[13405]: Accepted hostbased for scfc from 10.68.17.228 port 51490 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
576 Jan 19 04:30:09 tools-webgrid-lighttpd-1209 sshd[13405]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
577 Jan 19 04:30:09 tools-webgrid-lighttpd-1209 sshd[13420]: Starting session: command for scfc from 10.68.17.228 port 51490
578 Jan 19 04:30:09 tools-webgrid-lighttpd-1209 sshd[13405]: pam_unix(sshd:session): session closed for user scfc
579 Jan 19 05:00:09 tools-webgrid-lighttpd-1209 sshd[16023]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
580 Jan 19 05:00:09 tools-webgrid-lighttpd-1209 sshd[16023]: Accepted hostbased for scfc from 10.68.17.228 port 38742 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
581 Jan 19 05:00:09 tools-webgrid-lighttpd-1209 sshd[16023]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
582 Jan 19 05:00:09 tools-webgrid-lighttpd-1209 sshd[16038]: Starting session: command for scfc from 10.68.17.228 port 38742
583 Jan 19 05:00:09 tools-webgrid-lighttpd-1209 sshd[16023]: pam_unix(sshd:session): session closed for user scfc
584 Jan 19 05:30:11 tools-webgrid-lighttpd-1209 sshd[18643]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
585 Jan 19 05:30:11 tools-webgrid-lighttpd-1209 sshd[18643]: Accepted hostbased for scfc from 10.68.17.228 port 42061 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
586 Jan 19 05:30:11 tools-webgrid-lighttpd-1209 sshd[18643]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
587 Jan 19 05:30:11 tools-webgrid-lighttpd-1209 sshd[18659]: Starting session: command for scfc from 10.68.17.228 port 42061
588 Jan 19 05:30:11 tools-webgrid-lighttpd-1209 sshd[18643]: pam_unix(sshd:session): session closed for user scfc
589 Jan 19 06:00:10 tools-webgrid-lighttpd-1209 sshd[21266]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
590 Jan 19 06:00:10 tools-webgrid-lighttpd-1209 sshd[21266]: Accepted hostbased for scfc from 10.68.17.228 port 45164 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
591 Jan 19 06:00:10 tools-webgrid-lighttpd-1209 sshd[21266]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
592 Jan 19 06:00:10 tools-webgrid-lighttpd-1209 sshd[21283]: Starting session: command for scfc from 10.68.17.228 port 45164
593 Jan 19 06:00:10 tools-webgrid-lighttpd-1209 sshd[21266]: pam_unix(sshd:session): session closed for user scfc
594 Jan 19 06:30:11 tools-webgrid-lighttpd-1209 sshd[23970]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
595 Jan 19 06:30:11 tools-webgrid-lighttpd-1209 sshd[23970]: Accepted hostbased for scfc from 10.68.17.228 port 48145 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
596 Jan 19 06:30:11 tools-webgrid-lighttpd-1209 sshd[23970]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
597 Jan 19 06:30:11 tools-webgrid-lighttpd-1209 sshd[23985]: Starting session: command for scfc from 10.68.17.228 port 48145
598 Jan 19 06:30:11 tools-webgrid-lighttpd-1209 sshd[23970]: pam_unix(sshd:session): session closed for user scfc
599 Jan 19 07:00:10 tools-webgrid-lighttpd-1209 sshd[26775]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
600 Jan 19 07:00:10 tools-webgrid-lighttpd-1209 sshd[26775]: Accepted hostbased for scfc from 10.68.17.228 port 51561 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
601 Jan 19 07:00:10 tools-webgrid-lighttpd-1209 sshd[26775]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
602 Jan 19 07:00:10 tools-webgrid-lighttpd-1209 sshd[26790]: Starting session: command for scfc from 10.68.17.228 port 51561
603 Jan 19 07:00:10 tools-webgrid-lighttpd-1209 sshd[26775]: pam_unix(sshd:session): session closed for user scfc
604 Jan 19 07:30:10 tools-webgrid-lighttpd-1209 sshd[29391]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
605 Jan 19 07:30:10 tools-webgrid-lighttpd-1209 sshd[29391]: Accepted hostbased for scfc from 10.68.17.228 port 55134 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
606 Jan 19 07:30:10 tools-webgrid-lighttpd-1209 sshd[29391]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
607 Jan 19 07:30:10 tools-webgrid-lighttpd-1209 sshd[29406]: Starting session: command for scfc from 10.68.17.228 port 55134
608 Jan 19 07:30:10 tools-webgrid-lighttpd-1209 sshd[29391]: pam_unix(sshd:session): session closed for user scfc
609 Jan 19 08:00:11 tools-webgrid-lighttpd-1209 sshd[32018]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
610 Jan 19 08:00:11 tools-webgrid-lighttpd-1209 sshd[32018]: Accepted hostbased for scfc from 10.68.17.228 port 58559 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
611 Jan 19 08:00:11 tools-webgrid-lighttpd-1209 sshd[32018]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
612 Jan 19 08:00:12 tools-webgrid-lighttpd-1209 sshd[32033]: Starting session: command for scfc from 10.68.17.228 port 58559
613 Jan 19 08:00:12 tools-webgrid-lighttpd-1209 sshd[32018]: pam_unix(sshd:session): session closed for user scfc
614 Jan 19 08:30:09 tools-webgrid-lighttpd-1209 sshd[2357]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
615 Jan 19 08:30:09 tools-webgrid-lighttpd-1209 sshd[2357]: Accepted hostbased for scfc from 10.68.17.228 port 33512 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
616 Jan 19 08:30:09 tools-webgrid-lighttpd-1209 sshd[2357]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
617 Jan 19 08:30:09 tools-webgrid-lighttpd-1209 sshd[2372]: Starting session: command for scfc from 10.68.17.228 port 33512
618 Jan 19 08:30:10 tools-webgrid-lighttpd-1209 sshd[2357]: pam_unix(sshd:session): session closed for user scfc
619 Jan 19 09:00:09 tools-webgrid-lighttpd-1209 sshd[5075]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
620 Jan 19 09:00:09 tools-webgrid-lighttpd-1209 sshd[5075]: Accepted hostbased for scfc from 10.68.17.228 port 37055 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
621 Jan 19 09:00:09 tools-webgrid-lighttpd-1209 sshd[5075]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
622 Jan 19 09:00:10 tools-webgrid-lighttpd-1209 sshd[5090]: Starting session: command for scfc from 10.68.17.228 port 37055
623 Jan 19 09:00:10 tools-webgrid-lighttpd-1209 sshd[5075]: pam_unix(sshd:session): session closed for user scfc
624 Jan 19 09:30:10 tools-webgrid-lighttpd-1209 sshd[7808]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
625 Jan 19 09:30:10 tools-webgrid-lighttpd-1209 sshd[7808]: Accepted hostbased for scfc from 10.68.17.228 port 40694 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
626 Jan 19 09:30:10 tools-webgrid-lighttpd-1209 sshd[7808]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
627 Jan 19 09:30:10 tools-webgrid-lighttpd-1209 sshd[7823]: Starting session: command for scfc from 10.68.17.228 port 40694
628 Jan 19 09:30:10 tools-webgrid-lighttpd-1209 sshd[7808]: pam_unix(sshd:session): session closed for user scfc
629 Jan 19 10:00:10 tools-webgrid-lighttpd-1209 sshd[10545]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
630 Jan 19 10:00:10 tools-webgrid-lighttpd-1209 sshd[10545]: Accepted hostbased for scfc from 10.68.17.228 port 44256 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
631 Jan 19 10:00:10 tools-webgrid-lighttpd-1209 sshd[10545]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
632 Jan 19 10:00:10 tools-webgrid-lighttpd-1209 sshd[10560]: Starting session: command for scfc from 10.68.17.228 port 44256
633 Jan 19 10:00:10 tools-webgrid-lighttpd-1209 sshd[10545]: pam_unix(sshd:session): session closed for user scfc
634 Jan 19 10:30:13 tools-webgrid-lighttpd-1209 sshd[13214]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
635 Jan 19 10:30:13 tools-webgrid-lighttpd-1209 sshd[13214]: Accepted hostbased for scfc from 10.68.17.228 port 47592 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
636 Jan 19 10:30:13 tools-webgrid-lighttpd-1209 sshd[13214]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
637 Jan 19 10:30:13 tools-webgrid-lighttpd-1209 sshd[13229]: Starting session: command for scfc from 10.68.17.228 port 47592
638 Jan 19 10:30:13 tools-webgrid-lighttpd-1209 sshd[13214]: pam_unix(sshd:session): session closed for user scfc
639 Jan 19 11:00:12 tools-webgrid-lighttpd-1209 sshd[15822]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
640 Jan 19 11:00:12 tools-webgrid-lighttpd-1209 sshd[15822]: Accepted hostbased for scfc from 10.68.17.228 port 51134 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
641 Jan 19 11:00:12 tools-webgrid-lighttpd-1209 sshd[15822]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
642 Jan 19 11:00:12 tools-webgrid-lighttpd-1209 sshd[15837]: Starting session: command for scfc from 10.68.17.228 port 51134
643 Jan 19 11:00:12 tools-webgrid-lighttpd-1209 sshd[15822]: pam_unix(sshd:session): session closed for user scfc
644 Jan 19 11:30:11 tools-webgrid-lighttpd-1209 sshd[18453]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
645 Jan 19 11:30:11 tools-webgrid-lighttpd-1209 sshd[18453]: Accepted hostbased for scfc from 10.68.17.228 port 54627 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
646 Jan 19 11:30:11 tools-webgrid-lighttpd-1209 sshd[18453]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
647 Jan 19 11:30:11 tools-webgrid-lighttpd-1209 sshd[18468]: Starting session: command for scfc from 10.68.17.228 port 54627
648 Jan 19 11:30:11 tools-webgrid-lighttpd-1209 sshd[18453]: pam_unix(sshd:session): session closed for user scfc
649 Jan 19 12:00:11 tools-webgrid-lighttpd-1209 sshd[21075]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
650 Jan 19 12:00:11 tools-webgrid-lighttpd-1209 sshd[21075]: Accepted hostbased for scfc from 10.68.17.228 port 58464 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
651 Jan 19 12:00:11 tools-webgrid-lighttpd-1209 sshd[21075]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
652 Jan 19 12:00:12 tools-webgrid-lighttpd-1209 sshd[21090]: Starting session: command for scfc from 10.68.17.228 port 58464
653 Jan 19 12:00:12 tools-webgrid-lighttpd-1209 sshd[21075]: pam_unix(sshd:session): session closed for user scfc
654 Jan 19 12:30:11 tools-webgrid-lighttpd-1209 sshd[23710]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
655 Jan 19 12:30:11 tools-webgrid-lighttpd-1209 sshd[23710]: Accepted hostbased for scfc from 10.68.17.228 port 51773 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
656 Jan 19 12:30:11 tools-webgrid-lighttpd-1209 sshd[23710]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
657 Jan 19 12:30:11 tools-webgrid-lighttpd-1209 sshd[23725]: Starting session: command for scfc from 10.68.17.228 port 51773
658 Jan 19 12:30:11 tools-webgrid-lighttpd-1209 sshd[23710]: pam_unix(sshd:session): session closed for user scfc
659 Jan 19 13:00:10 tools-webgrid-lighttpd-1209 sshd[26331]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
660 Jan 19 13:00:10 tools-webgrid-lighttpd-1209 sshd[26331]: Accepted hostbased for scfc from 10.68.17.228 port 47675 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
661 Jan 19 13:00:10 tools-webgrid-lighttpd-1209 sshd[26331]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
662 Jan 19 13:00:11 tools-webgrid-lighttpd-1209 sshd[26346]: Starting session: command for scfc from 10.68.17.228 port 47675
663 Jan 19 13:00:11 tools-webgrid-lighttpd-1209 sshd[26331]: pam_unix(sshd:session): session closed for user scfc
664 Jan 19 13:30:10 tools-webgrid-lighttpd-1209 sshd[28950]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
665 Jan 19 13:30:10 tools-webgrid-lighttpd-1209 sshd[28950]: Accepted hostbased for scfc from 10.68.17.228 port 57628 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
666 Jan 19 13:30:10 tools-webgrid-lighttpd-1209 sshd[28950]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
667 Jan 19 13:30:10 tools-webgrid-lighttpd-1209 sshd[28965]: Starting session: command for scfc from 10.68.17.228 port 57628
668 Jan 19 13:30:11 tools-webgrid-lighttpd-1209 sshd[28950]: pam_unix(sshd:session): session closed for user scfc
669 Jan 19 14:00:11 tools-webgrid-lighttpd-1209 sshd[31554]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
670 Jan 19 14:00:11 tools-webgrid-lighttpd-1209 sshd[31554]: Accepted hostbased for scfc from 10.68.17.228 port 44940 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
671 Jan 19 14:00:11 tools-webgrid-lighttpd-1209 sshd[31554]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
672 Jan 19 14:00:11 tools-webgrid-lighttpd-1209 sshd[31569]: Starting session: command for scfc from 10.68.17.228 port 44940
673 Jan 19 14:00:11 tools-webgrid-lighttpd-1209 sshd[31554]: pam_unix(sshd:session): session closed for user scfc
674 Jan 19 14:30:09 tools-webgrid-lighttpd-1209 sshd[1818]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
675 Jan 19 14:30:09 tools-webgrid-lighttpd-1209 sshd[1818]: Accepted hostbased for scfc from 10.68.17.228 port 53529 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
676 Jan 19 14:30:09 tools-webgrid-lighttpd-1209 sshd[1818]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
677 Jan 19 14:30:09 tools-webgrid-lighttpd-1209 sshd[1833]: Starting session: command for scfc from 10.68.17.228 port 53529
678 Jan 19 14:30:10 tools-webgrid-lighttpd-1209 sshd[1818]: pam_unix(sshd:session): session closed for user scfc
679 Jan 19 15:00:11 tools-webgrid-lighttpd-1209 sshd[4423]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
680 Jan 19 15:00:11 tools-webgrid-lighttpd-1209 sshd[4423]: Accepted hostbased for scfc from 10.68.17.228 port 43299 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
681 Jan 19 15:00:11 tools-webgrid-lighttpd-1209 sshd[4423]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
682 Jan 19 15:00:11 tools-webgrid-lighttpd-1209 sshd[4438]: Starting session: command for scfc from 10.68.17.228 port 43299
683 Jan 19 15:00:11 tools-webgrid-lighttpd-1209 sshd[4423]: pam_unix(sshd:session): session closed for user scfc
684 Jan 19 15:30:09 tools-webgrid-lighttpd-1209 sshd[7074]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
685 Jan 19 15:30:09 tools-webgrid-lighttpd-1209 sshd[7074]: Accepted hostbased for scfc from 10.68.17.228 port 40101 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
686 Jan 19 15:30:09 tools-webgrid-lighttpd-1209 sshd[7074]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
687 Jan 19 15:30:10 tools-webgrid-lighttpd-1209 sshd[7099]: Starting session: command for scfc from 10.68.17.228 port 40101
688 Jan 19 15:30:10 tools-webgrid-lighttpd-1209 sshd[7074]: pam_unix(sshd:session): session closed for user scfc
689 Jan 19 16:00:10 tools-webgrid-lighttpd-1209 sshd[9832]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
690 Jan 19 16:00:10 tools-webgrid-lighttpd-1209 sshd[9832]: Accepted hostbased for scfc from 10.68.17.228 port 34347 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
691 Jan 19 16:00:10 tools-webgrid-lighttpd-1209 sshd[9832]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
692 Jan 19 16:00:10 tools-webgrid-lighttpd-1209 sshd[9857]: Starting session: command for scfc from 10.68.17.228 port 34347
693 Jan 19 16:00:10 tools-webgrid-lighttpd-1209 sshd[9832]: pam_unix(sshd:session): session closed for user scfc
694 Jan 19 16:30:13 tools-webgrid-lighttpd-1209 sshd[12568]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
695 Jan 19 16:30:13 tools-webgrid-lighttpd-1209 sshd[12568]: Accepted hostbased for scfc from 10.68.17.228 port 48960 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
696 Jan 19 16:30:13 tools-webgrid-lighttpd-1209 sshd[12568]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
697 Jan 19 16:30:13 tools-webgrid-lighttpd-1209 sshd[12583]: Starting session: command for scfc from 10.68.17.228 port 48960
698 Jan 19 16:30:13 tools-webgrid-lighttpd-1209 sshd[12568]: pam_unix(sshd:session): session closed for user scfc
699 Jan 19 17:00:11 tools-webgrid-lighttpd-1209 sshd[15249]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
700 Jan 19 17:00:11 tools-webgrid-lighttpd-1209 sshd[15249]: Accepted hostbased for scfc from 10.68.17.228 port 37180 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
701 Jan 19 17:00:11 tools-webgrid-lighttpd-1209 sshd[15249]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
702 Jan 19 17:00:11 tools-webgrid-lighttpd-1209 sshd[15264]: Starting session: command for scfc from 10.68.17.228 port 37180
703 Jan 19 17:00:11 tools-webgrid-lighttpd-1209 sshd[15249]: pam_unix(sshd:session): session closed for user scfc
704 Jan 19 17:30:10 tools-webgrid-lighttpd-1209 sshd[17860]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
705 Jan 19 17:30:10 tools-webgrid-lighttpd-1209 sshd[17860]: Accepted hostbased for scfc from 10.68.17.228 port 42197 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
706 Jan 19 17:30:10 tools-webgrid-lighttpd-1209 sshd[17860]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
707 Jan 19 17:30:11 tools-webgrid-lighttpd-1209 sshd[17875]: Starting session: command for scfc from 10.68.17.228 port 42197
708 Jan 19 17:30:11 tools-webgrid-lighttpd-1209 sshd[17860]: pam_unix(sshd:session): session closed for user scfc
709 Jan 19 18:00:11 tools-webgrid-lighttpd-1209 sshd[20466]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
710 Jan 19 18:00:11 tools-webgrid-lighttpd-1209 sshd[20466]: Accepted hostbased for scfc from 10.68.17.228 port 55286 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
711 Jan 19 18:00:11 tools-webgrid-lighttpd-1209 sshd[20466]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
712 Jan 19 18:00:11 tools-webgrid-lighttpd-1209 sshd[20481]: Starting session: command for scfc from 10.68.17.228 port 55286
713 Jan 19 18:00:12 tools-webgrid-lighttpd-1209 sshd[20466]: pam_unix(sshd:session): session closed for user scfc
714 Jan 19 18:30:10 tools-webgrid-lighttpd-1209 sshd[23098]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
715 Jan 19 18:30:10 tools-webgrid-lighttpd-1209 sshd[23098]: Accepted hostbased for scfc from 10.68.17.228 port 33573 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
716 Jan 19 18:30:10 tools-webgrid-lighttpd-1209 sshd[23098]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
717 Jan 19 18:30:11 tools-webgrid-lighttpd-1209 sshd[23113]: Starting session: command for scfc from 10.68.17.228 port 33573
718 Jan 19 18:30:11 tools-webgrid-lighttpd-1209 sshd[23098]: pam_unix(sshd:session): session closed for user scfc
719 Jan 19 19:00:11 tools-webgrid-lighttpd-1209 sshd[25739]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
720 Jan 19 19:00:11 tools-webgrid-lighttpd-1209 sshd[25739]: Accepted hostbased for scfc from 10.68.17.228 port 44431 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
721 Jan 19 19:00:11 tools-webgrid-lighttpd-1209 sshd[25739]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
722 Jan 19 19:00:11 tools-webgrid-lighttpd-1209 sshd[25754]: Starting session: command for scfc from 10.68.17.228 port 44431
723 Jan 19 19:00:11 tools-webgrid-lighttpd-1209 sshd[25739]: pam_unix(sshd:session): session closed for user scfc
724 Jan 19 19:30:11 tools-webgrid-lighttpd-1209 sshd[28456]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
725 Jan 19 19:30:11 tools-webgrid-lighttpd-1209 sshd[28456]: Accepted hostbased for scfc from 10.68.17.228 port 53529 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
726 Jan 19 19:30:11 tools-webgrid-lighttpd-1209 sshd[28456]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
727 Jan 19 19:30:11 tools-webgrid-lighttpd-1209 sshd[28471]: Starting session: command for scfc from 10.68.17.228 port 53529
728 Jan 19 19:30:11 tools-webgrid-lighttpd-1209 sshd[28456]: pam_unix(sshd:session): session closed for user scfc
729 Jan 19 20:00:11 tools-webgrid-lighttpd-1209 sshd[31641]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
730 Jan 19 20:00:11 tools-webgrid-lighttpd-1209 sshd[31641]: Accepted hostbased for scfc from 10.68.17.228 port 34459 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
731 Jan 19 20:00:11 tools-webgrid-lighttpd-1209 sshd[31641]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
732 Jan 19 20:00:11 tools-webgrid-lighttpd-1209 sshd[31656]: Starting session: command for scfc from 10.68.17.228 port 34459
733 Jan 19 20:00:11 tools-webgrid-lighttpd-1209 sshd[31641]: pam_unix(sshd:session): session closed for user scfc
734 Jan 19 20:30:10 tools-webgrid-lighttpd-1209 sshd[3601]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
735 Jan 19 20:30:10 tools-webgrid-lighttpd-1209 sshd[3601]: Accepted hostbased for scfc from 10.68.17.228 port 55701 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
736 Jan 19 20:30:10 tools-webgrid-lighttpd-1209 sshd[3601]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
737 Jan 19 20:30:11 tools-webgrid-lighttpd-1209 sshd[3620]: Starting session: command for scfc from 10.68.17.228 port 55701
738 Jan 19 20:30:11 tools-webgrid-lighttpd-1209 sshd[3601]: pam_unix(sshd:session): session closed for user scfc
739 Jan 19 21:00:11 tools-webgrid-lighttpd-1209 sshd[6786]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
740 Jan 19 21:00:11 tools-webgrid-lighttpd-1209 sshd[6786]: Accepted hostbased for scfc from 10.68.17.228 port 54678 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
741 Jan 19 21:00:11 tools-webgrid-lighttpd-1209 sshd[6786]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
742 Jan 19 21:00:11 tools-webgrid-lighttpd-1209 sshd[6801]: Starting session: command for scfc from 10.68.17.228 port 54678
743 Jan 19 21:00:11 tools-webgrid-lighttpd-1209 sshd[6786]: pam_unix(sshd:session): session closed for user scfc
744 Jan 19 21:30:11 tools-webgrid-lighttpd-1209 sshd[10503]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
745 Jan 19 21:30:11 tools-webgrid-lighttpd-1209 sshd[10503]: Accepted hostbased for scfc from 10.68.17.228 port 52204 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
746 Jan 19 21:30:11 tools-webgrid-lighttpd-1209 sshd[10503]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
747 Jan 19 21:30:11 tools-webgrid-lighttpd-1209 sshd[10518]: Starting session: command for scfc from 10.68.17.228 port 52204
748 Jan 19 21:30:11 tools-webgrid-lighttpd-1209 sshd[10503]: pam_unix(sshd:session): session closed for user scfc
749 Jan 19 22:00:10 tools-webgrid-lighttpd-1209 sshd[14313]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
750 Jan 19 22:00:11 tools-webgrid-lighttpd-1209 sshd[14313]: Accepted hostbased for scfc from 10.68.17.228 port 50620 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
751 Jan 19 22:00:11 tools-webgrid-lighttpd-1209 sshd[14313]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
752 Jan 19 22:00:11 tools-webgrid-lighttpd-1209 sshd[14328]: Starting session: command for scfc from 10.68.17.228 port 50620
753 Jan 19 22:00:11 tools-webgrid-lighttpd-1209 sshd[14313]: pam_unix(sshd:session): session closed for user scfc
754 Jan 19 22:30:10 tools-webgrid-lighttpd-1209 sshd[17070]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
755 Jan 19 22:30:10 tools-webgrid-lighttpd-1209 sshd[17070]: Accepted hostbased for scfc from 10.68.17.228 port 45198 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
756 Jan 19 22:30:10 tools-webgrid-lighttpd-1209 sshd[17070]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
757 Jan 19 22:30:10 tools-webgrid-lighttpd-1209 sshd[17085]: Starting session: command for scfc from 10.68.17.228 port 45198
758 Jan 19 22:30:10 tools-webgrid-lighttpd-1209 sshd[17070]: pam_unix(sshd:session): session closed for user scfc
759 Jan 19 23:00:13 tools-webgrid-lighttpd-1209 sshd[19941]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
760 Jan 19 23:00:13 tools-webgrid-lighttpd-1209 sshd[19941]: Accepted hostbased for scfc from 10.68.17.228 port 48683 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
761 Jan 19 23:00:13 tools-webgrid-lighttpd-1209 sshd[19941]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
762 Jan 19 23:00:13 tools-webgrid-lighttpd-1209 sshd[19956]: Starting session: command for scfc from 10.68.17.228 port 48683
763 Jan 19 23:00:13 tools-webgrid-lighttpd-1209 sshd[19941]: pam_unix(sshd:session): session closed for user scfc
764 Jan 19 23:02:07 tools-webgrid-lighttpd-1209 sshd[19990]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
765 Jan 19 23:02:07 tools-webgrid-lighttpd-1209 sshd[19990]: Accepted hostbased for scfc from 10.68.17.228 port 52565 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
766 Jan 19 23:02:07 tools-webgrid-lighttpd-1209 sshd[19990]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
767 Jan 19 23:02:07 tools-webgrid-lighttpd-1209 sshd[20005]: Starting session: command for scfc from 10.68.17.228 port 52565
768 Jan 19 23:02:07 tools-webgrid-lighttpd-1209 sshd[19990]: pam_unix(sshd:session): session closed for user scfc
769 Jan 19 23:30:10 tools-webgrid-lighttpd-1209 sshd[22682]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
770 Jan 19 23:30:10 tools-webgrid-lighttpd-1209 sshd[22682]: Accepted hostbased for scfc from 10.68.17.228 port 38550 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
771 Jan 19 23:30:10 tools-webgrid-lighttpd-1209 sshd[22682]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
772 Jan 19 23:30:10 tools-webgrid-lighttpd-1209 sshd[22697]: Starting session: command for scfc from 10.68.17.228 port 38550
773 Jan 19 23:30:10 tools-webgrid-lighttpd-1209 sshd[22682]: pam_unix(sshd:session): session closed for user scfc
774 Jan 20 00:00:11 tools-webgrid-lighttpd-1209 sshd[18434]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
775 Jan 20 00:00:11 tools-webgrid-lighttpd-1209 sshd[18434]: Accepted hostbased for scfc from 10.68.17.228 port 39999 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
776 Jan 20 00:00:11 tools-webgrid-lighttpd-1209 sshd[18434]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
777 Jan 20 00:00:11 tools-webgrid-lighttpd-1209 sshd[18449]: Starting session: command for scfc from 10.68.17.228 port 39999
778 Jan 20 00:00:11 tools-webgrid-lighttpd-1209 sshd[18434]: pam_unix(sshd:session): session closed for user scfc
779 Jan 20 00:30:11 tools-webgrid-lighttpd-1209 sshd[25012]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
780 Jan 20 00:30:12 tools-webgrid-lighttpd-1209 sshd[25012]: Accepted hostbased for scfc from 10.68.17.228 port 41382 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
781 Jan 20 00:30:12 tools-webgrid-lighttpd-1209 sshd[25012]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
782 Jan 20 00:30:12 tools-webgrid-lighttpd-1209 sshd[25038]: Starting session: command for scfc from 10.68.17.228 port 41382
783 Jan 20 00:30:12 tools-webgrid-lighttpd-1209 sshd[25012]: pam_unix(sshd:session): session closed for user scfc
784 Jan 20 01:00:17 tools-webgrid-lighttpd-1209 sshd[25772]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
785 Jan 20 01:00:17 tools-webgrid-lighttpd-1209 sshd[25772]: Accepted hostbased for scfc from 10.68.17.228 port 59542 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
786 Jan 20 01:00:17 tools-webgrid-lighttpd-1209 sshd[25772]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
787 Jan 20 01:00:17 tools-webgrid-lighttpd-1209 sshd[25787]: Starting session: command for scfc from 10.68.17.228 port 59542
788 Jan 20 01:00:17 tools-webgrid-lighttpd-1209 sshd[25772]: pam_unix(sshd:session): session closed for user scfc
789 Jan 20 01:30:09 tools-webgrid-lighttpd-1209 sshd[25165]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
790 Jan 20 01:30:09 tools-webgrid-lighttpd-1209 sshd[25165]: Accepted hostbased for scfc from 10.68.17.228 port 40268 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
791 Jan 20 01:30:09 tools-webgrid-lighttpd-1209 sshd[25165]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
792 Jan 20 01:30:09 tools-webgrid-lighttpd-1209 sshd[25190]: Starting session: command for scfc from 10.68.17.228 port 40268
793 Jan 20 01:30:09 tools-webgrid-lighttpd-1209 sshd[25165]: pam_unix(sshd:session): session closed for user scfc
794 Jan 20 02:00:10 tools-webgrid-lighttpd-1209 sshd[406]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
795 Jan 20 02:00:10 tools-webgrid-lighttpd-1209 sshd[406]: Accepted hostbased for scfc from 10.68.17.228 port 35878 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
796 Jan 20 02:00:10 tools-webgrid-lighttpd-1209 sshd[406]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
797 Jan 20 02:00:10 tools-webgrid-lighttpd-1209 sshd[421]: Starting session: command for scfc from 10.68.17.228 port 35878
798 Jan 20 02:00:11 tools-webgrid-lighttpd-1209 sudo: scfc : TTY=unknown ; PWD=/home/scfc ; USER=root ; COMMAND=/bin/kill -HUP 22935
799 Jan 20 02:00:11 tools-webgrid-lighttpd-1209 sudo: scfc : TTY=unknown ; PWD=/home/scfc ; USER=root ; COMMAND=/bin/kill -HUP 22935
800 Jan 20 02:00:11 tools-webgrid-lighttpd-1209 sshd[406]: pam_unix(sshd:session): session closed for user scfc
801 Jan 20 02:30:09 tools-webgrid-lighttpd-1209 sshd[3228]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
802 Jan 20 02:30:09 tools-webgrid-lighttpd-1209 sshd[3228]: Accepted hostbased for scfc from 10.68.17.228 port 34954 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
803 Jan 20 02:30:09 tools-webgrid-lighttpd-1209 sshd[3228]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
804 Jan 20 02:30:09 tools-webgrid-lighttpd-1209 sshd[3243]: Starting session: command for scfc from 10.68.17.228 port 34954
805 Jan 20 02:30:10 tools-webgrid-lighttpd-1209 sshd[3228]: pam_unix(sshd:session): session closed for user scfc
806 Jan 20 03:00:09 tools-webgrid-lighttpd-1209 sshd[6001]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
807 Jan 20 03:00:09 tools-webgrid-lighttpd-1209 sshd[6001]: Accepted hostbased for scfc from 10.68.17.228 port 34723 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
808 Jan 20 03:00:09 tools-webgrid-lighttpd-1209 sshd[6001]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
809 Jan 20 03:00:09 tools-webgrid-lighttpd-1209 sshd[6016]: Starting session: command for scfc from 10.68.17.228 port 34723
810 Jan 20 03:00:09 tools-webgrid-lighttpd-1209 sshd[6001]: pam_unix(sshd:session): session closed for user scfc
811 Jan 20 03:30:10 tools-webgrid-lighttpd-1209 sshd[9347]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
812 Jan 20 03:30:10 tools-webgrid-lighttpd-1209 sshd[9347]: Accepted hostbased for scfc from 10.68.17.228 port 35066 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
813 Jan 20 03:30:10 tools-webgrid-lighttpd-1209 sshd[9347]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
814 Jan 20 03:30:10 tools-webgrid-lighttpd-1209 sshd[9363]: Starting session: command for scfc from 10.68.17.228 port 35066
815 Jan 20 03:30:10 tools-webgrid-lighttpd-1209 sshd[9347]: pam_unix(sshd:session): session closed for user scfc
816 Jan 20 04:00:08 tools-webgrid-lighttpd-1209 sshd[12526]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
817 Jan 20 04:00:08 tools-webgrid-lighttpd-1209 sshd[12526]: Accepted hostbased for scfc from 10.68.17.228 port 55157 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
818 Jan 20 04:00:08 tools-webgrid-lighttpd-1209 sshd[12526]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
819 Jan 20 04:00:08 tools-webgrid-lighttpd-1209 sshd[12541]: Starting session: command for scfc from 10.68.17.228 port 55157
820 Jan 20 04:00:08 tools-webgrid-lighttpd-1209 sshd[12526]: pam_unix(sshd:session): session closed for user scfc
821 Jan 20 04:30:09 tools-webgrid-lighttpd-1209 sshd[15668]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
822 Jan 20 04:30:09 tools-webgrid-lighttpd-1209 sshd[15668]: Accepted hostbased for scfc from 10.68.17.228 port 40075 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
823 Jan 20 04:30:09 tools-webgrid-lighttpd-1209 sshd[15668]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
824 Jan 20 04:30:09 tools-webgrid-lighttpd-1209 sshd[15683]: Starting session: command for scfc from 10.68.17.228 port 40075
825 Jan 20 04:30:09 tools-webgrid-lighttpd-1209 sshd[15668]: pam_unix(sshd:session): session closed for user scfc
826 Jan 20 05:00:09 tools-webgrid-lighttpd-1209 sshd[18533]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
827 Jan 20 05:00:09 tools-webgrid-lighttpd-1209 sshd[18533]: Accepted hostbased for scfc from 10.68.17.228 port 48744 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
828 Jan 20 05:00:09 tools-webgrid-lighttpd-1209 sshd[18533]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
829 Jan 20 05:00:09 tools-webgrid-lighttpd-1209 sshd[18548]: Starting session: command for scfc from 10.68.17.228 port 48744
830 Jan 20 05:00:09 tools-webgrid-lighttpd-1209 sshd[18533]: pam_unix(sshd:session): session closed for user scfc
831 Jan 20 05:30:09 tools-webgrid-lighttpd-1209 sshd[21348]: Accepted ECDSA public key 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57 from scfc@tools-bastion-01.tools.eqiad.wmflabs
832 Jan 20 05:30:09 tools-webgrid-lighttpd-1209 sshd[21348]: Accepted hostbased for scfc from 10.68.17.228 port 57235 ssh2: ECDSA 80:37:58:71:84:99:54:e7:17:dd:c4:be:54:48:41:57, client user "scfc", client host "tools-bastion-01.tools.eqiad.wmflabs"
833 Jan 20 05:30:09 tools-webgrid-lighttpd-1209 sshd[21348]: pam_unix(sshd:session): session opened for user scfc by (uid=0)
834 Jan 20 05:30:09 tools-webgrid-lighttpd-1209 sshd[21363]: Starting session: command for scfc from 10.68.17.228 port 57235
835 Jan 20 05:30:09 tools-webgrid-lighttpd-1209 sshd[21348]: pam_unix(sshd:session): session closed for user scfc
836root@labcontrol1001:~# salt 'tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs' cmd.run "date"
837tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs:
838tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs:
839 top - 14:12:25 up 21 days, 10:55, 0 users, load average: 106.68, 106.35, 105.32
840 Tasks: 500 total, 1 running, 499 sleeping, 0 stopped, 0 zombie
841 Cpu(s): 34.4%us, 3.0%sy, 0.0%ni, 61.7%id, 0.1%wa, 0.0%hi, 0.4%si, 0.5%st
842 Mem: 8178496k total, 5856324k used, 2322172k free, 248684k buffers
843 Swap: 25034744k total, 0k used, 25034744k free, 3513728k cached
844
845 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
846 1418 root 20 0 12176 1460 848 R 4 0.0 0:00.02 top
847 1 root 20 0 20064 2224 1284 S 0 0.0 0:07.86 init
848 2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
849 3 root 20 0 0 0 0 S 0 0.0 0:50.55 ksoftirqd/0
850 4 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/0:0
851tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs:
852 top - 14:12:59 up 21 days, 10:55, 0 users, load average: 106.38, 106.31, 105.3
853 Tasks: 500 total, 1 running, 499 sleeping, 0 stopped, 0 zombie
854 Cpu(s): 34.4%us, 3.0%sy, 0.0%ni, 61.7%id, 0.1%wa, 0.0%hi, 0.4%si, 0.5%st
855 Mem: 8178496k total, 5856628k used, 2321868k free, 248684k buffers
856 Swap: 25034744k total, 0k used, 25034744k free, 3513732k cached
857
858 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
859 1438 root 20 0 12180 1484 856 R 6 0.0 0:00.04 top -c -b -n 1
860 1061 tools.sd 20 0 443m 25m 5368 S 2 0.3 57:06.66 python -m src.serve
861 1 root 20 0 20064 2224 1284 S 0 0.0 0:07.86 /sbin/init
862 2 root 20 0 0 0 0 S 0 0.0 0:00.00 [kthreadd]
863 3 root 20 0 0 0 0 S 0 0.0 0:50.55 [ksoftirqd/0]
864 4 root 20 0 0 0 0 S 0 0.0 0:00.00 [kworker/0:0]
865 5 root 20 0 0 0 0 S 0 0.0 0:00.08 [kworker/u:0]
866 6 root RT 0 0 0 0 S 0 0.0 0:04.11 [migration/0]
867 7 root RT 0 0 0 0 S 0 0.0 0:11.52 [watchdog/0]
868 8 root RT 0 0 0 0 S 0 0.0 0:04.01 [migration/1]
869 10 root 20 0 0 0 0 S 0 0.0 4:03.34 [ksoftirqd/1]
870 11 root 20 0 0 0 0 S 0 0.0 3:11.62 [kworker/0:1]
871 12 root RT 0 0 0 0 S 0 0.0 0:11.47 [watchdog/1]
872 13 root RT 0 0 0 0 S 0 0.0 0:04.94 [migration/2]
873 15 root 20 0 0 0 0 S 0 0.0 0:49.39 [ksoftirqd/2]
874 16 root RT 0 0 0 0 S 0 0.0 0:10.08 [watchdog/2]
875 17 root RT 0 0 0 0 S 0 0.0 0:03.62 [migration/3]
876 18 root 20 0 0 0 0 S 0 0.0 0:00.00 [kworker/3:0]
877 19 root 20 0 0 0 0 S 0 0.0 0:40.28 [ksoftirqd/3]
878 20 root RT 0 0 0 0 S 0 0.0 0:09.35 [watchdog/3]
879 21 root 0 -20 0 0 0 S 0 0.0 0:00.00 [cpuset]
880 22 root 0 -20 0 0 0 S 0 0.0 0:00.00 [khelper]
881 23 root 20 0 0 0 0 S 0 0.0 0:00.00 [kdevtmpfs]
882 24 root 0 -20 0 0 0 S 0 0.0 0:00.00 [netns]
883 25 root 20 0 0 0 0 S 0 0.0 0:00.00 [kworker/u:1]
884 26 root 20 0 0 0 0 S 0 0.0 0:05.62 [sync_supers]
885 27 root 20 0 0 0 0 S 0 0.0 0:00.30 [bdi-default]
886 28 root 0 -20 0 0 0 S 0 0.0 0:00.00 [kintegrityd]
887 29 root 0 -20 0 0 0 S 0 0.0 0:00.00 [kblockd]
888 30 root 0 -20 0 0 0 S 0 0.0 0:00.00 [ata_sff]
889 31 root 20 0 0 0 0 S 0 0.0 0:00.00 [khubd]
890 32 root 0 -20 0 0 0 S 0 0.0 0:00.00 [md]
891 33 root 20 0 0 0 0 S 0 0.0 1:25.62 [kworker/2:1]
892 35 root 20 0 0 0 0 S 0 0.0 0:01.63 [khungtaskd]
893 36 root 20 0 0 0 0 S 0 0.0 0:03.19 [kswapd0]
894 37 root 25 5 0 0 0 S 0 0.0 0:00.00 [ksmd]
895 38 root 39 19 0 0 0 S 0 0.0 0:00.00 [khugepaged]
896 39 root 20 0 0 0 0 S 0 0.0 0:00.00 [fsnotify_mark]
897 40 root 20 0 0 0 0 S 0 0.0 0:00.00 [ecryptfs-kthrea]
898 41 root 0 -20 0 0 0 S 0 0.0 0:00.00 [crypto]
899 49 root 0 -20 0 0 0 S 0 0.0 0:00.00 [kthrotld]
900 50 root 20 0 0 0 0 S 0 0.0 0:00.00 [scsi_eh_0]
901 51 root 20 0 0 0 0 S 0 0.0 0:00.00 [scsi_eh_1]
902 72 root 0 -20 0 0 0 S 0 0.0 0:00.00 [devfreq_wq]
903 214 root 20 0 0 0 0 S 0 0.0 48:15.13 [kworker/1:1]
904 215 root 20 0 0 0 0 S 0 0.0 2:28.47 [kworker/3:1]
905 221 root 0 -20 0 0 0 S 0 0.0 0:00.00 [kdmflush]
906 228 root 0 -20 0 0 0 S 0 0.0 0:00.00 [kdmflush]
907 236 root 20 0 0 0 0 S 0 0.0 0:41.24 [jbd2/vda1-8]
908 237 root 0 -20 0 0 0 S 0 0.0 0:00.00 [ext4-dio-unwrit]
909 320 root 20 0 17228 636 456 S 0 0.0 0:00.06 upstart-udev-bridge
910 331 root 20 0 15208 1264 740 S 0 0.0 0:00.05 /sbin/udevd --daemo
911 338 root 20 0 19196 1008 724 S 0 0.0 0:03.73 rpcbind -w
912 367 statd 20 0 27336 1980 1288 S 0 0.0 0:00.00 rpc.statd -L
913 406 root 20 0 35860 1188 856 S 0 0.0 0:00.00 CRON
914 407 root 20 0 4396 608 508 S 0 0.0 0:00.00 /bin/sh -c [ -x /
915 412 root 20 0 38908 1000 852 S 0 0.0 0:00.00 find /var/lib/php5/
916 413 root 20 0 31124 828 680 D 0 0.0 0:00.30 fuser -s ./sess_j3a
917 458 root 20 0 0 0 0 S 0 0.0 1:54.59 [vballoon]
918 464 root 20 0 15204 852 324 S 0 0.0 0:00.00 /sbin/udevd --daemo
919 465 root 20 0 15184 832 324 S 0 0.0 0:00.00 /sbin/udevd --daemo
920 474 root 0 -20 0 0 0 S 0 0.0 0:00.00 [kpsmoused]
921 487 root 20 0 0 0 0 S 0 0.0 0:05.64 [jbd2/dm-1-8]
922 488 root 0 -20 0 0 0 S 0 0.0 0:00.00 [ext4-dio-unwrit]
923 568 root 20 0 15184 384 196 S 0 0.0 0:00.00 upstart-socket-brid
924 608 root 20 0 15032 1084 544 S 0 0.0 0:00.07 dhclient3 -e IF_MET
925 661 root 0 -20 0 0 0 S 0 0.0 0:00.00 [rpciod]
926 678 root 0 -20 0 0 0 S 0 0.0 0:00.00 [nfsiod]
927 682 root 20 0 0 0 0 S 0 0.0 0:00.00 [nfsv4.0-svc]
928 684 root 20 0 0 0 0 S 0 0.0 0:35.37 [flush-253:0]
929 732 ntp 20 0 39360 2832 1972 S 0 0.0 3:15.12 /usr/sbin/ntpd -p /
930 748 diamond 20 0 570m 25m 5300 S 0 0.3 30:44.64 /usr/bin/python /us
931 802 syslog 20 0 241m 5152 1068 S 0 0.1 2:21.40 rsyslogd -c5
932 804 messageb 20 0 19580 684 452 S 0 0.0 0:00.02 dbus-daemon --syste
933 866 root 20 0 33404 880 748 S 0 0.0 0:00.00 /sbin/getty -8 3840
934 872 root 20 0 33404 876 748 S 0 0.0 0:00.00 /sbin/getty -8 3840
935 874 root 20 0 588m 47m 9528 S 0 0.6 15:12.54 /usr/bin/python /us
936 878 root 20 0 33404 880 748 S 0 0.0 0:00.00 /sbin/getty -8 3840
937 879 root 20 0 33404 880 748 S 0 0.0 0:00.00 /sbin/getty -8 3840
938 882 root 20 0 33404 876 748 S 0 0.0 0:00.00 /sbin/getty -8 3840
939 886 root 20 0 52568 2956 2336 S 0 0.0 0:01.22 /usr/sbin/sshd -D
940 887 daemon 20 0 10572 352 220 S 0 0.0 0:00.05 atd
941 888 root 20 0 12776 932 724 S 0 0.0 0:07.13 cron
942 909 root 20 0 15976 700 524 S 0 0.0 4:07.85 /usr/sbin/irqbalanc
943 933 sgeadmin 20 0 22548 1988 1544 S 0 0.0 0:00.00 /usr/lib/gridengine
944 935 tools.po 20 0 52300 2936 1576 D 0 0.0 5:48.38 /usr/sbin/lighttpd
945 936 Debian-e 20 0 45392 1148 732 S 0 0.0 0:01.98 /usr/sbin/exim4 -oP
946 937 sgeadmin 20 0 22548 1996 1544 S 0 0.0 0:00.00 /usr/lib/gridengine
947 939 tools.sd 20 0 52152 2920 1684 D 0 0.0 1:37.51 /usr/sbin/lighttpd
948 940 sgeadmin 20 0 22548 1988 1544 S 0 0.0 0:00.00 /usr/lib/gridengine
949 942 tools.wi 20 0 52332 3156 1660 D 0 0.0 1:33.50 /usr/sbin/lighttpd
950 959 root 20 0 47044 1528 944 S 0 0.0 1:45.51 /usr/sbin/lldpd
951 960 _lldpd 20 0 42804 676 204 S 0 0.0 0:39.60 /usr/sbin/lldpd
952 966 root 20 0 6992 384 216 S 0 0.0 0:00.00 /usr/sbin/mcelog --
953 975 tools.po 20 0 307m 13m 8868 S 0 0.2 0:00.02 /usr/bin/php-cgi
954 981 nagios 20 0 23340 1108 616 S 0 0.0 4:31.24 /usr/sbin/nrpe -c /
955 995 tools.po 20 0 307m 13m 8880 S 0 0.2 0:00.38 /usr/bin/php-cgi
956 1001 tools.wi 20 0 307m 13m 8868 S 0 0.2 0:00.03 /usr/bin/php-cgi
957 1003 root 20 0 942m 3896 1656 S 0 0.0 4:24.71 /usr/sbin/nscd
958 1004 tools.wi 20 0 307m 5320 324 S 0 0.1 0:00.00 /usr/bin/php-cgi
959 1005 tools.wi 20 0 307m 5320 324 S 0 0.1 0:00.00 /usr/bin/php-cgi
960 1007 tools.po 20 0 17092 1388 1132 D 0 0.0 137:34.48 /usr/lib/gamin/gam_
961 1008 tools.wi 20 0 307m 13m 8868 S 0 0.2 0:00.03 /usr/bin/php-cgi
962 1017 tools.wi 20 0 307m 5320 324 S 0 0.1 0:00.00 /usr/bin/php-cgi
963 1024 tools.wi 20 0 307m 5320 324 S 0 0.1 0:00.00 /usr/bin/php-cgi
964 1035 tools.wi 20 0 16972 1280 1108 D 0 0.0 11:42.46 /usr/lib/gamin/gam_
965 1039 nslcd 20 0 437m 6964 1868 S 0 0.1 4:20.22 /usr/sbin/nslcd
966 1050 tools.sd 20 0 307m 13m 8868 S 0 0.2 0:00.02 /usr/bin/php-cgi
967 1052 tools.sd 20 0 307m 5316 324 S 0 0.1 0:00.00 /usr/bin/php-cgi
968 1053 tools.sd 20 0 307m 5316 324 S 0 0.1 0:00.00 /usr/bin/php-cgi
969 1054 tools.sd 20 0 307m 13m 8868 S 0 0.2 0:00.02 /usr/bin/php-cgi
970 1056 tools.sd 20 0 307m 5320 324 S 0 0.1 0:00.00 /usr/bin/php-cgi
971 1057 tools.sd 20 0 307m 5320 324 S 0 0.1 0:00.00 /usr/bin/php-cgi
972 1058 tools.sd 20 0 4392 604 508 S 0 0.0 0:00.00 /bin/sh /data/proje
973 1063 tools.sd 20 0 16972 1288 1112 D 0 0.0 2:06.84 /usr/lib/gamin/gam_
974 1066 root 20 0 8768 524 408 S 0 0.0 0:00.53 /usr/sbin/inetd
975 1105 sgeadmin 20 0 62768 8076 2208 D 0 0.1 515:39.36 /usr/lib/gridengine
976 1182 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd: scfc [priv]
977 1183 sshd 20 0 62672 1340 628 S 0 0.0 0:00.00 sshd: scfc [net]
978 1276 www-data 20 0 44332 1048 292 S 0 0.0 1:22.55 /usr/sbin/lighttpd
979 1304 root 20 0 34320 816 704 S 0 0.0 0:00.00 /sbin/getty -8 3840
980 1305 root 20 0 7332 716 604 S 0 0.0 0:00.00 /sbin/getty -L 1152
981 1354 root 20 0 35860 1188 856 S 0 0.0 0:00.00 CRON
982 1355 root 20 0 4396 612 508 S 0 0.0 0:00.00 /bin/sh -c [ -x /
983 1360 root 20 0 38908 1000 852 S 0 0.0 0:00.00 find /var/lib/php5/
984 1361 root 20 0 31124 832 680 D 0 0.0 0:00.33 fuser -s ./sess_j3a
985 1436 root 20 0 588m 39m 1372 S 0 0.5 0:00.00 /usr/bin/python /us
986 1437 root 20 0 4396 608 504 S 0 0.0 0:00.00 /bin/sh -c top -c -
987 3104 tools.ca 20 0 314m 12m 4376 S 0 0.2 0:13.14 /usr/bin/php-cgi
988 3136 tools.po 20 0 308m 8184 2668 S 0 0.1 0:01.56 /usr/bin/php-cgi
989 3302 tools.wi 20 0 339m 9928 3380 S 0 0.1 0:02.84 /usr/bin/php-cgi
990 5686 tools.po 20 0 308m 8304 2668 S 0 0.1 0:01.56 /usr/bin/php-cgi
991 5766 tools.wi 20 0 339m 9.8m 3324 S 0 0.1 0:02.54 /usr/bin/php-cgi
992 6414 tools.wi 20 0 338m 8856 3284 S 0 0.1 0:00.10 /usr/bin/php-cgi
993 6703 sgeadmin 20 0 22548 2000 1552 S 0 0.0 0:00.00 /usr/lib/gridengine
994 6705 51058 20 0 53044 3796 1692 D 0 0.0 1:44.01 /usr/sbin/lighttpd
995 6729 51058 20 0 307m 13m 8868 S 0 0.2 0:00.01 /usr/bin/php-cgi
996 6731 51058 20 0 307m 5316 324 S 0 0.1 0:00.00 /usr/bin/php-cgi
997 6732 51058 20 0 307m 5316 324 S 0 0.1 0:00.00 /usr/bin/php-cgi
998 6733 51058 20 0 307m 13m 8868 S 0 0.2 0:00.02 /usr/bin/php-cgi
999 6735 51058 20 0 307m 5316 324 S 0 0.1 0:00.00 /usr/bin/php-cgi
1000 6736 51058 20 0 307m 5316 324 S 0 0.1 0:00.00 /usr/bin/php-cgi
1001 6738 51058 20 0 17316 1624 1144 D 0 0.0 719:19.58 /usr/lib/gamin/gam_
1002 6956 sgeadmin 20 0 22548 1996 1548 S 0 0.0 0:00.00 /usr/lib/gridengine
1003 6958 tools.ca 20 0 52532 3268 1588 D 0 0.0 9:51.10 /usr/sbin/lighttpd
1004 6988 tools.ca 20 0 307m 13m 8880 S 0 0.2 0:00.33 /usr/bin/php-cgi
1005 6992 tools.ca 20 0 307m 13m 8880 S 0 0.2 0:01.05 /usr/bin/php-cgi
1006 6997 tools.ca 20 0 16972 1284 1108 D 0 0.0 5:16.09 /usr/lib/gamin/gam_
1007 7011 sgeadmin 20 0 22548 2008 1552 S 0 0.0 0:00.00 /usr/lib/gridengine
1008 7013 52421 20 0 52188 2804 1556 D 0 0.0 1:36.84 /usr/sbin/lighttpd
1009 7033 52421 20 0 307m 13m 8868 S 0 0.2 0:00.02 /usr/bin/php-cgi
1010 7035 52421 20 0 312m 9436 2936 S 0 0.1 0:00.26 /usr/bin/php-cgi
1011 7036 52421 20 0 312m 9464 2924 S 0 0.1 0:00.24 /usr/bin/php-cgi
1012 7037 52421 20 0 307m 13m 8868 S 0 0.2 0:00.02 /usr/bin/php-cgi
1013 7039 52421 20 0 312m 9552 2964 S 0 0.1 0:08.39 /usr/bin/php-cgi
1014 7040 52421 20 0 312m 9540 2952 S 0 0.1 0:08.17 /usr/bin/php-cgi
1015 7042 52421 20 0 16972 1284 1112 D 0 0.0 7:39.02 /usr/lib/gamin/gam_
1016 7056 sgeadmin 20 0 22548 2000 1552 S 0 0.0 0:00.00 /usr/lib/gridengine
1017 7058 tools.al 20 0 52072 2796 1628 D 0 0.0 1:36.00 /usr/sbin/lighttpd
1018 7082 tools.al 20 0 307m 13m 8868 S 0 0.2 0:00.01 /usr/bin/php-cgi
1019 7084 tools.al 20 0 307m 5316 324 S 0 0.1 0:00.00 /usr/bin/php-cgi
1020 7085 tools.al 20 0 307m 5316 324 S 0 0.1 0:00.00 /usr/bin/php-cgi
1021 7086 tools.al 20 0 307m 13m 8868 S 0 0.2 0:00.02 /usr/bin/php-cgi
1022 7088 tools.al 20 0 307m 5320 324 S 0 0.1 0:00.00 /usr/bin/php-cgi
1023 7089 tools.al 20 0 307m 5320 324 S 0 0.1 0:00.00 /usr/bin/php-cgi
1024 7091 tools.al 20 0 16972 1288 1112 D 0 0.0 9:44.89 /usr/lib/gamin/gam_
1025 7092 sgeadmin 20 0 22548 1992 1548 S 0 0.0 0:00.00 /usr/lib/gridengine
1026 7094 tools.ti 20 0 52168 2844 1636 S 0 0.0 1:35.91 /usr/sbin/lighttpd
1027 7118 tools.ti 20 0 307m 13m 8868 S 0 0.2 0:00.02 /usr/bin/php-cgi
1028 7120 tools.ti 20 0 307m 5320 324 S 0 0.1 0:00.00 /usr/bin/php-cgi
1029 7121 tools.ti 20 0 307m 5320 324 S 0 0.1 0:00.00 /usr/bin/php-cgi
1030 7122 tools.ti 20 0 307m 13m 8868 S 0 0.2 0:00.01 /usr/bin/php-cgi
1031 7124 tools.ti 20 0 307m 5320 324 S 0 0.1 0:00.00 /usr/bin/php-cgi
1032 7125 tools.ti 20 0 307m 5320 324 S 0 0.1 0:00.00 /usr/bin/php-cgi
1033 7127 tools.ti 20 0 16972 1288 1112 D 0 0.0 34:08.19 /usr/lib/gamin/gam_
1034 7189 sgeadmin 20 0 22548 2000 1548 S 0 0.0 0:00.00 /usr/lib/gridengine
1035 7191 tools.ex 20 0 52076 2500 1468 D 0 0.0 1:35.43 /usr/sbin/lighttpd
1036 7211 tools.ex 20 0 307m 13m 8868 S 0 0.2 0:00.01 /usr/bin/php-cgi
1037 7213 tools.ex 20 0 307m 5320 324 S 0 0.1 0:00.00 /usr/bin/php-cgi
1038 7214 tools.ex 20 0 307m 5320 324 S 0 0.1 0:00.00 /usr/bin/php-cgi
1039 7215 tools.ex 20 0 307m 13m 8868 S 0 0.2 0:00.01 /usr/bin/php-cgi
1040 7217 tools.ex 20 0 307m 5316 324 S 0 0.1 0:00.00 /usr/bin/php-cgi
1041 7218 tools.ex 20 0 307m 5316 324 S 0 0.1 0:00.00 /usr/bin/php-cgi
1042 7220 tools.ex 20 0 16972 1284 1112 D 0 0.0 44:15.11 /usr/lib/gamin/gam_
1043 7379 sgeadmin 20 0 22548 1992 1548 S 0 0.0 0:00.00 /usr/lib/gridengine
1044 7381 tools.po 20 0 52460 3172 1552 D 0 0.0 1:35.47 /usr/sbin/lighttpd

kern.log

P2494 salt 'tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs' cmd.run "cat /var/log/kern.log"
1ools-webgrid-lighttpd-1209.tools.eqiad.wmflabs:
2 Jan 17 14:20:47 tools-webgrid-lighttpd-1209 kernel: [1595022.260547] php-cgi[3080]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
3 Jan 17 22:08:57 tools-webgrid-lighttpd-1209 kernel: [1623111.495594] php-cgi[10714]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
4 Jan 18 04:51:27 tools-webgrid-lighttpd-1209 kernel: [1647261.956719] php-cgi[12795]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
5 Jan 18 04:51:27 tools-webgrid-lighttpd-1209 kernel: [1647262.065589] php-cgi[17958]: segfault at 1000701 ip 00002b91093d9c0c sp 00007fff793626c0 error 4 in libc-2.15.so[2b910939f000+1b4000]
6 Jan 18 04:53:08 tools-webgrid-lighttpd-1209 kernel: [1647362.914089] php-cgi[17972]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
7 Jan 18 05:56:49 tools-webgrid-lighttpd-1209 kernel: [1651183.355985] php-cgi[20606]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
8 Jan 18 07:28:54 tools-webgrid-lighttpd-1209 kernel: [1656708.349900] php-cgi[25790]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
9 Jan 18 15:57:36 tools-webgrid-lighttpd-1209 kernel: [1687230.932884] php-cgi[9063]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
10 Jan 18 20:58:38 tools-webgrid-lighttpd-1209 kernel: [1705292.668192] php-cgi[434]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
11 Jan 19 01:01:34 tools-webgrid-lighttpd-1209 kernel: [1719869.063518] php-cgi[22073]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
12 Jan 19 01:03:15 tools-webgrid-lighttpd-1209 kernel: [1719969.527141] php-cgi[27239]: segfault at 80001000321 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
13 Jan 19 01:53:42 tools-webgrid-lighttpd-1209 kernel: [1722996.475524] php-cgi[27266]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
14 Jan 19 01:55:30 tools-webgrid-lighttpd-1209 kernel: [1723104.503873] php-cgi[11306]: segfault at 1000721 ip 00000000006ab997 sp 00007fff79363c40 error 4 in php5-cgi[400000+74b000]
15 Jan 19 01:59:17 tools-webgrid-lighttpd-1209 kernel: [1723331.816459] php-cgi[24515]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
16 Jan 19 08:01:12 tools-webgrid-lighttpd-1209 kernel: [1745046.887606] php-cgi[26645]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
17 Jan 19 08:02:25 tools-webgrid-lighttpd-1209 kernel: [1745119.560187] php-cgi[32056]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
18 Jan 19 08:52:24 tools-webgrid-lighttpd-1209 kernel: [1748118.784763] php-cgi[2542]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
19 Jan 19 08:53:36 tools-webgrid-lighttpd-1209 kernel: [1748190.890766] php-cgi[32073]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
20 Jan 19 09:30:44 tools-webgrid-lighttpd-1209 kernel: [1750418.531413] php-cgi[4974]: segfault at 1000721 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
21 Jan 19 12:00:36 tools-webgrid-lighttpd-1209 kernel: [1759410.723578] php-cgi[13369]: segfault at 1000701 ip 00002b91093d9c0c sp 00007fff793626c0 error 4 in libc-2.15.so[2b910939f000+1b4000]
22 Jan 19 15:19:50 tools-webgrid-lighttpd-1209 kernel: [1771364.560906] php-cgi[4221]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
23 Jan 19 15:21:33 tools-webgrid-lighttpd-1209 kernel: [1771467.407169] php-cgi[6904]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
24 Jan 19 21:28:41 tools-webgrid-lighttpd-1209 kernel: [1793495.817149] php-cgi[7199]: segfault at 1000701 ip 00002b91093d9c0c sp 00007fff79360b40 error 4 in libc-2.15.so[2b910939f000+1b4000]
25 Jan 19 21:30:25 tools-webgrid-lighttpd-1209 kernel: [1793599.748935] php-cgi[6966]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d65f0 error 4 in libc-2.15.so[2b21cb637000+1b4000]
26 Jan 19 22:01:41 tools-webgrid-lighttpd-1209 kernel: [1795475.309218] php-cgi[9824]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
27 Jan 19 22:11:30 tools-webgrid-lighttpd-1209 kernel: [1796064.724207] php-cgi[14367]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
28 Jan 19 22:56:20 tools-webgrid-lighttpd-1209 kernel: [1798754.486805] php-cgi[10544]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
29 Jan 19 22:57:31 tools-webgrid-lighttpd-1209 kernel: [1798825.411214] php-cgi[19884]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d8170 error 4 in libc-2.15.so[2b21cb637000+1b4000]
30 Jan 20 01:35:29 tools-webgrid-lighttpd-1209 kernel: [1808304.162852] php-cgi[22936]: segfault at 1000721 ip 00000000006ab997 sp 00007fff85536ca0 error 4 in php5-cgi[400000+74b000]
31 Jan 20 03:23:16 tools-webgrid-lighttpd-1209 kernel: [1814770.619007] php-cgi[8847]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d65f0 error 4 in libc-2.15.so[2b21cb637000+1b4000]
32 Jan 20 03:24:28 tools-webgrid-lighttpd-1209 kernel: [1814842.372966] php-cgi[3024]: segfault at 1000701 ip 00002b21cb671c0c sp 00007fffbf7d65f0 error 4 in libc-2.15.so[2b21cb637000+1b4000]
33 Jan 20 04:03:29 tools-webgrid-lighttpd-1209 kernel: [1817183.390569] php-cgi[9103]: segfault at 1000621 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]
34 Jan 20 04:04:51 tools-webgrid-lighttpd-1209 kernel: [1817265.527446] php-cgi[9128]: segfault at 1000621 ip 00000000006ab997 sp 00007fffbf7d96f0 error 4 in php5-cgi[400000+74b000]

looking at the grid master

root@tools-grid-master:~# qhost | grep 'webgrid-lighttpd-1209'
tools-webgrid-lighttpd-1209.eqiad.wmflabs lx26-amd64      4  0.43    7.8G    2.1G   23.9G     0.0

looking at the SGE state of jobs on the frozen host

P2495 root@tools-grid-master:~# qstat -u '*' -xml | grep -A 5 1209
1 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
2 <slots>1</slots>
3 </job_list>
4 <job_list state="running">
5 <JB_job_number>609</JB_job_number>
6 <JAT_prio>0.34984</JAT_prio>
7--
8 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
9 <slots>1</slots>
10 </job_list>
11 <job_list state="running">
12 <JB_job_number>658</JB_job_number>
13 <JAT_prio>0.34984</JAT_prio>
14--
15 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
16 <slots>1</slots>
17 </job_list>
18 <job_list state="running">
19 <JB_job_number>662</JB_job_number>
20 <JAT_prio>0.34984</JAT_prio>
21--
22 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
23 <slots>1</slots>
24 </job_list>
25 <job_list state="running">
26 <JB_job_number>681</JB_job_number>
27 <JAT_prio>0.34983</JAT_prio>
28--
29 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
30 <slots>1</slots>
31 </job_list>
32 <job_list state="running">
33 <JB_job_number>764</JB_job_number>
34 <JAT_prio>0.34983</JAT_prio>
35--
36 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
37 <slots>1</slots>
38 </job_list>
39 <job_list state="running">
40 <JB_job_number>784</JB_job_number>
41 <JAT_prio>0.34983</JAT_prio>
42--
43 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
44 <slots>1</slots>
45 </job_list>
46 <job_list state="running">
47 <JB_job_number>813</JB_job_number>
48 <JAT_prio>0.34983</JAT_prio>
49--
50 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
51 <slots>1</slots>
52 </job_list>
53 <job_list state="running">
54 <JB_job_number>819</JB_job_number>
55 <JAT_prio>0.34983</JAT_prio>
56--
57 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
58 <slots>1</slots>
59 </job_list>
60 <job_list state="running">
61 <JB_job_number>822</JB_job_number>
62 <JAT_prio>0.34983</JAT_prio>
63--
64 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
65 <slots>1</slots>
66 </job_list>
67 <job_list state="running">
68 <JB_job_number>826</JB_job_number>
69 <JAT_prio>0.34983</JAT_prio>
70--
71 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
72 <slots>1</slots>
73 </job_list>
74 <job_list state="running">
75 <JB_job_number>859</JB_job_number>
76 <JAT_prio>0.34983</JAT_prio>
77--
78 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
79 <slots>1</slots>
80 </job_list>
81 <job_list state="running">
82 <JB_job_number>881</JB_job_number>
83 <JAT_prio>0.34983</JAT_prio>
84--
85 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
86 <slots>1</slots>
87 </job_list>
88 <job_list state="running">
89 <JB_job_number>979</JB_job_number>
90 <JAT_prio>0.34983</JAT_prio>
91--
92 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
93 <slots>1</slots>
94 </job_list>
95 <job_list state="running">
96 <JB_job_number>985</JB_job_number>
97 <JAT_prio>0.34983</JAT_prio>
98--
99 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
100 <slots>1</slots>
101 </job_list>
102 <job_list state="running">
103 <JB_job_number>1115</JB_job_number>
104 <JAT_prio>0.34982</JAT_prio>
105--
106 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
107 <slots>1</slots>
108 </job_list>
109 <job_list state="running">
110 <JB_job_number>1116</JB_job_number>
111 <JAT_prio>0.34982</JAT_prio>
112--
113 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
114 <slots>1</slots>
115 </job_list>
116 <job_list state="running">
117 <JB_job_number>1200</JB_job_number>
118 <JAT_prio>0.34982</JAT_prio>
119--
120 <JB_job_number>1209</JB_job_number>
121 <JAT_prio>0.34982</JAT_prio>
122 <JB_name>lighttpd-urlinktranslator</JB_name>
123 <JB_owner>tools.urlinktranslator</JB_owner>
124 <state>r</state>
125 <JAT_start_time>2015-12-30T04:18:17</JAT_start_time>
126--
127 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
128 <slots>1</slots>
129 </job_list>
130 <job_list state="running">
131 <JB_job_number>1213</JB_job_number>
132 <JAT_prio>0.34982</JAT_prio>
133--
134 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
135 <slots>1</slots>
136 </job_list>
137 <job_list state="running">
138 <JB_job_number>1264</JB_job_number>
139 <JAT_prio>0.34982</JAT_prio>
140--
141 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
142 <slots>1</slots>
143 </job_list>
144 <job_list state="running">
145 <JB_job_number>1265</JB_job_number>
146 <JAT_prio>0.34982</JAT_prio>
147--
148 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
149 <slots>1</slots>
150 </job_list>
151 <job_list state="running">
152 <JB_job_number>1278</JB_job_number>
153 <JAT_prio>0.34982</JAT_prio>
154--
155 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
156 <slots>1</slots>
157 </job_list>
158 <job_list state="running">
159 <JB_job_number>1320</JB_job_number>
160 <JAT_prio>0.34982</JAT_prio>
161--
162 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
163 <slots>1</slots>
164 </job_list>
165 <job_list state="running">
166 <JB_job_number>1355</JB_job_number>
167 <JAT_prio>0.34982</JAT_prio>
168--
169 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
170 <slots>1</slots>
171 </job_list>
172 <job_list state="running">
173 <JB_job_number>1362</JB_job_number>
174 <JAT_prio>0.34982</JAT_prio>
175--
176 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
177 <slots>1</slots>
178 </job_list>
179 <job_list state="running">
180 <JB_job_number>1399</JB_job_number>
181 <JAT_prio>0.34982</JAT_prio>
182--
183 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
184 <slots>1</slots>
185 </job_list>
186 <job_list state="running">
187 <JB_job_number>1433</JB_job_number>
188 <JAT_prio>0.34981</JAT_prio>
189--
190 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
191 <slots>1</slots>
192 </job_list>
193 <job_list state="running">
194 <JB_job_number>1447</JB_job_number>
195 <JAT_prio>0.34981</JAT_prio>
196--
197 <queue_name>continuous@tools-exec-1209.eqiad.wmflabs</queue_name>
198 <slots>1</slots>
199 </job_list>
200 <job_list state="running">
201 <JB_job_number>7973</JB_job_number>
202 <JAT_prio>0.38146</JAT_prio>
203--
204 <JB_job_number>31209</JB_job_number>
205 <JAT_prio>0.37999</JAT_prio>
206 <JB_name>php_wbStringValueAsRedirect_20151201_zhwiki_p225_part_5_10</JB_name>
207 <JB_owner>tools.liangent-php</JB_owner>
208 <state>r</state>
209 <JAT_start_time>2015-12-17T05:06:53</JAT_start_time>
210--
211 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
212 <slots>1</slots>
213 </job_list>
214 <job_list state="running">
215 <JB_job_number>36142</JB_job_number>
216 <JAT_prio>0.62624</JAT_prio>
217--
218 <queue_name>task@tools-exec-1209.eqiad.wmflabs</queue_name>
219 <slots>1</slots>
220 </job_list>
221 <job_list state="running">
222 <JB_job_number>156134</JB_job_number>
223 <JAT_prio>0.65320</JAT_prio>
224--
225 <queue_name>continuous@tools-exec-1209.eqiad.wmflabs</queue_name>
226 <slots>1</slots>
227 </job_list>
228 <job_list state="running">
229 <JB_job_number>189858</JB_job_number>
230 <JAT_prio>0.33891</JAT_prio>
231--
232 <queue_name>task@tools-exec-1209.eqiad.wmflabs</queue_name>
233 <slots>1</slots>
234 </job_list>
235 <job_list state="running">
236 <JB_job_number>459132</JB_job_number>
237 <JAT_prio>0.63330</JAT_prio>
238--
239 <queue_name>continuous@tools-exec-1209.eqiad.wmflabs</queue_name>
240 <slots>1</slots>
241 </job_list>
242 <job_list state="running">
243 <JB_job_number>608235</JB_job_number>
244 <JAT_prio>0.52377</JAT_prio>
245--
246 <queue_name>continuous@tools-exec-1209.eqiad.wmflabs</queue_name>
247 <slots>1</slots>
248 </job_list>
249 <job_list state="running">
250 <JB_job_number>988870</JB_job_number>
251 <JAT_prio>0.49532</JAT_prio>
252--
253 <queue_name>continuous@tools-exec-1209.eqiad.wmflabs</queue_name>
254 <slots>1</slots>
255 </job_list>
256 <job_list state="running">
257 <JB_job_number>1915894</JB_job_number>
258 <JAT_prio>0.66293</JAT_prio>
259--
260 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
261 <slots>1</slots>
262 </job_list>
263 <job_list state="running">
264 <JB_job_number>2231271</JB_job_number>
265 <JAT_prio>0.31960</JAT_prio>
266--
267 <queue_name>task@tools-exec-1209.eqiad.wmflabs</queue_name>
268 <slots>1</slots>
269 </job_list>
270 <job_list state="running">
271 <JB_job_number>2425791</JB_job_number>
272 <JAT_prio>0.30626</JAT_prio>
273--
274 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
275 <slots>1</slots>
276 </job_list>
277 <job_list state="running">
278 <JB_job_number>2493921</JB_job_number>
279 <JAT_prio>0.30137</JAT_prio>
280--
281 <queue_name>continuous@tools-exec-1209.eqiad.wmflabs</queue_name>
282 <slots>1</slots>
283 </job_list>
284 <job_list state="running">
285 <JB_job_number>2496926</JB_job_number>
286 <JAT_prio>0.30115</JAT_prio>
287--
288 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
289 <slots>1</slots>
290 </job_list>
291 <job_list state="running">
292 <JB_job_number>2499386</JB_job_number>
293 <JAT_prio>0.30097</JAT_prio>
294--
295 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
296 <slots>1</slots>
297 </job_list>
298 <job_list state="running">
299 <JB_job_number>2501443</JB_job_number>
300 <JAT_prio>0.30082</JAT_prio>
301--
302 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
303 <slots>1</slots>
304 </job_list>
305 <job_list state="running">
306 <JB_job_number>2501498</JB_job_number>
307 <JAT_prio>0.30081</JAT_prio>
308--
309 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
310 <slots>1</slots>
311 </job_list>
312 <job_list state="running">
313 <JB_job_number>2501712</JB_job_number>
314 <JAT_prio>0.30080</JAT_prio>
315--
316 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
317 <slots>1</slots>
318 </job_list>
319 <job_list state="running">
320 <JB_job_number>2502115</JB_job_number>
321 <JAT_prio>0.30077</JAT_prio>
322--
323 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
324 <slots>1</slots>
325 </job_list>
326 <job_list state="running">
327 <JB_job_number>2502127</JB_job_number>
328 <JAT_prio>0.30076</JAT_prio>
329--
330 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
331 <slots>1</slots>
332 </job_list>
333 <job_list state="running">
334 <JB_job_number>2502357</JB_job_number>
335 <JAT_prio>0.30075</JAT_prio>
336--
337 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
338 <slots>1</slots>
339 </job_list>
340 <job_list state="running">
341 <JB_job_number>2502515</JB_job_number>
342 <JAT_prio>0.30074</JAT_prio>
343--
344 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
345 <slots>1</slots>
346 </job_list>
347 <job_list state="running">
348 <JB_job_number>2502613</JB_job_number>
349 <JAT_prio>0.30073</JAT_prio>
350--
351 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
352 <slots>1</slots>
353 </job_list>
354 <job_list state="running">
355 <JB_job_number>2503197</JB_job_number>
356 <JAT_prio>0.30069</JAT_prio>
357--
358 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
359 <slots>1</slots>
360 </job_list>
361 <job_list state="running">
362 <JB_job_number>2503470</JB_job_number>
363 <JAT_prio>0.30067</JAT_prio>
364--
365 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
366 <slots>1</slots>
367 </job_list>
368 <job_list state="running">
369 <JB_job_number>2503638</JB_job_number>
370 <JAT_prio>0.30065</JAT_prio>
371--
372 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
373 <slots>1</slots>
374 </job_list>
375 <job_list state="running">
376 <JB_job_number>2504419</JB_job_number>
377 <JAT_prio>0.30059</JAT_prio>
378--
379 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
380 <slots>1</slots>
381 </job_list>
382 <job_list state="running">
383 <JB_job_number>2504691</JB_job_number>
384 <JAT_prio>0.30057</JAT_prio>
385--
386 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
387 <slots>1</slots>
388 </job_list>
389 <job_list state="running">
390 <JB_job_number>2504790</JB_job_number>
391 <JAT_prio>0.30057</JAT_prio>
392--
393 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
394 <slots>1</slots>
395 </job_list>
396 <job_list state="running">
397 <JB_job_number>2504909</JB_job_number>
398 <JAT_prio>0.30056</JAT_prio>
399--
400 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
401 <slots>1</slots>
402 </job_list>
403 <job_list state="running">
404 <JB_job_number>2505023</JB_job_number>
405 <JAT_prio>0.30054</JAT_prio>
406--
407 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
408 <slots>1</slots>
409 </job_list>
410 <job_list state="running">
411 <JB_job_number>2505040</JB_job_number>
412 <JAT_prio>0.30054</JAT_prio>
413--
414 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
415 <slots>1</slots>
416 </job_list>
417 <job_list state="running">
418 <JB_job_number>2505083</JB_job_number>
419 <JAT_prio>0.30054</JAT_prio>
420--
421 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
422 <slots>1</slots>
423 </job_list>
424 <job_list state="running">
425 <JB_job_number>2505560</JB_job_number>
426 <JAT_prio>0.30051</JAT_prio>
427--
428 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
429 <slots>1</slots>
430 </job_list>
431 <job_list state="running">
432 <JB_job_number>2505736</JB_job_number>
433 <JAT_prio>0.30050</JAT_prio>
434--
435 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
436 <slots>1</slots>
437 </job_list>
438 <job_list state="running">
439 <JB_job_number>2506022</JB_job_number>
440 <JAT_prio>0.30047</JAT_prio>
441--
442 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
443 <slots>1</slots>
444 </job_list>
445 <job_list state="running">
446 <JB_job_number>2506167</JB_job_number>
447 <JAT_prio>0.30046</JAT_prio>
448--
449 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
450 <slots>1</slots>
451 </job_list>
452 <job_list state="running">
453 <JB_job_number>2506256</JB_job_number>
454 <JAT_prio>0.30046</JAT_prio>
455--
456 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
457 <slots>1</slots>
458 </job_list>
459 <job_list state="running">
460 <JB_job_number>2506465</JB_job_number>
461 <JAT_prio>0.30044</JAT_prio>
462--
463 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
464 <slots>1</slots>
465 </job_list>
466 <job_list state="running">
467 <JB_job_number>2506469</JB_job_number>
468 <JAT_prio>0.30044</JAT_prio>
469--
470 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
471 <slots>1</slots>
472 </job_list>
473 <job_list state="running">
474 <JB_job_number>2507336</JB_job_number>
475 <JAT_prio>0.30038</JAT_prio>
476--
477 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
478 <slots>1</slots>
479 </job_list>
480 <job_list state="running">
481 <JB_job_number>2507467</JB_job_number>
482 <JAT_prio>0.30037</JAT_prio>
483--
484 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
485 <slots>1</slots>
486 </job_list>
487 <job_list state="running">
488 <JB_job_number>2507648</JB_job_number>
489 <JAT_prio>0.30035</JAT_prio>
490--
491 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
492 <slots>1</slots>
493 </job_list>
494 <job_list state="running">
495 <JB_job_number>2508128</JB_job_number>
496 <JAT_prio>0.30032</JAT_prio>
497--
498 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
499 <slots>1</slots>
500 </job_list>
501 <job_list state="running">
502 <JB_job_number>2508654</JB_job_number>
503 <JAT_prio>0.30028</JAT_prio>
504--
505 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
506 <slots>1</slots>
507 </job_list>
508 <job_list state="running">
509 <JB_job_number>2508957</JB_job_number>
510 <JAT_prio>0.30025</JAT_prio>
511--
512 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
513 <slots>1</slots>
514 </job_list>
515 <job_list state="running">
516 <JB_job_number>2509147</JB_job_number>
517 <JAT_prio>0.30024</JAT_prio>
518--
519 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
520 <slots>1</slots>
521 </job_list>
522 <job_list state="running">
523 <JB_job_number>2509371</JB_job_number>
524 <JAT_prio>0.30022</JAT_prio>
525--
526 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
527 <slots>1</slots>
528 </job_list>
529 <job_list state="running">
530 <JB_job_number>2509808</JB_job_number>
531 <JAT_prio>0.30020</JAT_prio>
532--
533 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
534 <slots>1</slots>
535 </job_list>
536 <job_list state="running">
537 <JB_job_number>2509970</JB_job_number>
538 <JAT_prio>0.30018</JAT_prio>
539--
540 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
541 <slots>1</slots>
542 </job_list>
543 <job_list state="running">
544 <JB_job_number>2510310</JB_job_number>
545 <JAT_prio>0.30016</JAT_prio>
546--
547 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
548 <slots>1</slots>
549 </job_list>
550 <job_list state="running">
551 <JB_job_number>2510969</JB_job_number>
552 <JAT_prio>0.30011</JAT_prio>
553--
554 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
555 <slots>1</slots>
556 </job_list>
557 <job_list state="running">
558 <JB_job_number>2510971</JB_job_number>
559 <JAT_prio>0.30011</JAT_prio>
560--
561 <queue_name>task@tools-exec-1209.eqiad.wmflabs</queue_name>
562 <slots>1</slots>
563 </job_list>
564 <job_list state="running">
565 <JB_job_number>2510988</JB_job_number>
566 <JAT_prio>0.30011</JAT_prio>
567--
568 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
569 <slots>1</slots>
570 </job_list>
571 <job_list state="running">
572 <JB_job_number>2512156</JB_job_number>
573 <JAT_prio>0.30001</JAT_prio>
574root@tools-grid-master:~# qstat -u '*' -xml | grep -A 5 tools-webgrid-lighttpd-1209
575 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
576 <slots>1</slots>
577 </job_list>
578 <job_list state="running">
579 <JB_job_number>609</JB_job_number>
580 <JAT_prio>0.34984</JAT_prio>
581--
582 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
583 <slots>1</slots>
584 </job_list>
585 <job_list state="running">
586 <JB_job_number>658</JB_job_number>
587 <JAT_prio>0.34984</JAT_prio>
588--
589 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
590 <slots>1</slots>
591 </job_list>
592 <job_list state="running">
593 <JB_job_number>662</JB_job_number>
594 <JAT_prio>0.34984</JAT_prio>
595--
596 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
597 <slots>1</slots>
598 </job_list>
599 <job_list state="running">
600 <JB_job_number>681</JB_job_number>
601 <JAT_prio>0.34984</JAT_prio>
602--
603 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
604 <slots>1</slots>
605 </job_list>
606 <job_list state="running">
607 <JB_job_number>764</JB_job_number>
608 <JAT_prio>0.34983</JAT_prio>
609--
610 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
611 <slots>1</slots>
612 </job_list>
613 <job_list state="running">
614 <JB_job_number>784</JB_job_number>
615 <JAT_prio>0.34983</JAT_prio>
616--
617 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
618 <slots>1</slots>
619 </job_list>
620 <job_list state="running">
621 <JB_job_number>813</JB_job_number>
622 <JAT_prio>0.34983</JAT_prio>
623--
624 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
625 <slots>1</slots>
626 </job_list>
627 <job_list state="running">
628 <JB_job_number>819</JB_job_number>
629 <JAT_prio>0.34983</JAT_prio>
630--
631 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
632 <slots>1</slots>
633 </job_list>
634 <job_list state="running">
635 <JB_job_number>822</JB_job_number>
636 <JAT_prio>0.34983</JAT_prio>
637--
638 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
639 <slots>1</slots>
640 </job_list>
641 <job_list state="running">
642 <JB_job_number>826</JB_job_number>
643 <JAT_prio>0.34983</JAT_prio>
644--
645 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
646 <slots>1</slots>
647 </job_list>
648 <job_list state="running">
649 <JB_job_number>859</JB_job_number>
650 <JAT_prio>0.34983</JAT_prio>
651--
652 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
653 <slots>1</slots>
654 </job_list>
655 <job_list state="running">
656 <JB_job_number>881</JB_job_number>
657 <JAT_prio>0.34983</JAT_prio>
658--
659 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
660 <slots>1</slots>
661 </job_list>
662 <job_list state="running">
663 <JB_job_number>979</JB_job_number>
664 <JAT_prio>0.34983</JAT_prio>
665--
666 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
667 <slots>1</slots>
668 </job_list>
669 <job_list state="running">
670 <JB_job_number>985</JB_job_number>
671 <JAT_prio>0.34983</JAT_prio>
672--
673 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
674 <slots>1</slots>
675 </job_list>
676 <job_list state="running">
677 <JB_job_number>1115</JB_job_number>
678 <JAT_prio>0.34982</JAT_prio>
679--
680 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
681 <slots>1</slots>
682 </job_list>
683 <job_list state="running">
684 <JB_job_number>1116</JB_job_number>
685 <JAT_prio>0.34982</JAT_prio>
686--
687 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
688 <slots>1</slots>
689 </job_list>
690 <job_list state="running">
691 <JB_job_number>1200</JB_job_number>
692 <JAT_prio>0.34982</JAT_prio>
693--
694 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
695 <slots>1</slots>
696 </job_list>
697 <job_list state="running">
698 <JB_job_number>1213</JB_job_number>
699 <JAT_prio>0.34982</JAT_prio>
700--
701 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
702 <slots>1</slots>
703 </job_list>
704 <job_list state="running">
705 <JB_job_number>1264</JB_job_number>
706 <JAT_prio>0.34982</JAT_prio>
707--
708 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
709 <slots>1</slots>
710 </job_list>
711 <job_list state="running">
712 <JB_job_number>1265</JB_job_number>
713 <JAT_prio>0.34982</JAT_prio>
714--
715 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
716 <slots>1</slots>
717 </job_list>
718 <job_list state="running">
719 <JB_job_number>1278</JB_job_number>
720 <JAT_prio>0.34982</JAT_prio>
721--
722 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
723 <slots>1</slots>
724 </job_list>
725 <job_list state="running">
726 <JB_job_number>1320</JB_job_number>
727 <JAT_prio>0.34982</JAT_prio>
728--
729 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
730 <slots>1</slots>
731 </job_list>
732 <job_list state="running">
733 <JB_job_number>1355</JB_job_number>
734 <JAT_prio>0.34982</JAT_prio>
735--
736 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
737 <slots>1</slots>
738 </job_list>
739 <job_list state="running">
740 <JB_job_number>1362</JB_job_number>
741 <JAT_prio>0.34982</JAT_prio>
742--
743 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
744 <slots>1</slots>
745 </job_list>
746 <job_list state="running">
747 <JB_job_number>1399</JB_job_number>
748 <JAT_prio>0.34982</JAT_prio>
749--
750 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
751 <slots>1</slots>
752 </job_list>
753 <job_list state="running">
754 <JB_job_number>1433</JB_job_number>
755 <JAT_prio>0.34982</JAT_prio>
756--
757 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
758 <slots>1</slots>
759 </job_list>
760 <job_list state="running">
761 <JB_job_number>1447</JB_job_number>
762 <JAT_prio>0.34982</JAT_prio>
763--
764 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
765 <slots>1</slots>
766 </job_list>
767 <job_list state="running">
768 <JB_job_number>36142</JB_job_number>
769 <JAT_prio>0.62624</JAT_prio>
770--
771 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
772 <slots>1</slots>
773 </job_list>
774 <job_list state="running">
775 <JB_job_number>2231271</JB_job_number>
776 <JAT_prio>0.31960</JAT_prio>
777--
778 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
779 <slots>1</slots>
780 </job_list>
781 <job_list state="running">
782 <JB_job_number>2493921</JB_job_number>
783 <JAT_prio>0.30137</JAT_prio>
784--
785 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
786 <slots>1</slots>
787 </job_list>
788 <job_list state="running">
789 <JB_job_number>2499386</JB_job_number>
790 <JAT_prio>0.30097</JAT_prio>
791--
792 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
793 <slots>1</slots>
794 </job_list>
795 <job_list state="running">
796 <JB_job_number>2501443</JB_job_number>
797 <JAT_prio>0.30082</JAT_prio>
798--
799 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
800 <slots>1</slots>
801 </job_list>
802 <job_list state="running">
803 <JB_job_number>2501498</JB_job_number>
804 <JAT_prio>0.30081</JAT_prio>
805--
806 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
807 <slots>1</slots>
808 </job_list>
809 <job_list state="running">
810 <JB_job_number>2501712</JB_job_number>
811 <JAT_prio>0.30080</JAT_prio>
812--
813 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
814 <slots>1</slots>
815 </job_list>
816 <job_list state="running">
817 <JB_job_number>2502115</JB_job_number>
818 <JAT_prio>0.30077</JAT_prio>
819--
820 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
821 <slots>1</slots>
822 </job_list>
823 <job_list state="running">
824 <JB_job_number>2502127</JB_job_number>
825 <JAT_prio>0.30077</JAT_prio>
826--
827 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
828 <slots>1</slots>
829 </job_list>
830 <job_list state="running">
831 <JB_job_number>2502357</JB_job_number>
832 <JAT_prio>0.30075</JAT_prio>
833--
834 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
835 <slots>1</slots>
836 </job_list>
837 <job_list state="running">
838 <JB_job_number>2502515</JB_job_number>
839 <JAT_prio>0.30074</JAT_prio>
840--
841 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
842 <slots>1</slots>
843 </job_list>
844 <job_list state="running">
845 <JB_job_number>2502613</JB_job_number>
846 <JAT_prio>0.30073</JAT_prio>
847--
848 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
849 <slots>1</slots>
850 </job_list>
851 <job_list state="running">
852 <JB_job_number>2503197</JB_job_number>
853 <JAT_prio>0.30069</JAT_prio>
854--
855 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
856 <slots>1</slots>
857 </job_list>
858 <job_list state="running">
859 <JB_job_number>2503470</JB_job_number>
860 <JAT_prio>0.30067</JAT_prio>
861--
862 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
863 <slots>1</slots>
864 </job_list>
865 <job_list state="running">
866 <JB_job_number>2503638</JB_job_number>
867 <JAT_prio>0.30066</JAT_prio>
868--
869 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
870 <slots>1</slots>
871 </job_list>
872 <job_list state="running">
873 <JB_job_number>2504419</JB_job_number>
874 <JAT_prio>0.30059</JAT_prio>
875--
876 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
877 <slots>1</slots>
878 </job_list>
879 <job_list state="running">
880 <JB_job_number>2504691</JB_job_number>
881 <JAT_prio>0.30057</JAT_prio>
882--
883 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
884 <slots>1</slots>
885 </job_list>
886 <job_list state="running">
887 <JB_job_number>2504790</JB_job_number>
888 <JAT_prio>0.30057</JAT_prio>
889--
890 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
891 <slots>1</slots>
892 </job_list>
893 <job_list state="running">
894 <JB_job_number>2504909</JB_job_number>
895 <JAT_prio>0.30056</JAT_prio>
896--
897 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
898 <slots>1</slots>
899 </job_list>
900 <job_list state="running">
901 <JB_job_number>2505023</JB_job_number>
902 <JAT_prio>0.30055</JAT_prio>
903--
904 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
905 <slots>1</slots>
906 </job_list>
907 <job_list state="running">
908 <JB_job_number>2505040</JB_job_number>
909 <JAT_prio>0.30055</JAT_prio>
910--
911 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
912 <slots>1</slots>
913 </job_list>
914 <job_list state="running">
915 <JB_job_number>2505083</JB_job_number>
916 <JAT_prio>0.30055</JAT_prio>
917--
918 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
919 <slots>1</slots>
920 </job_list>
921 <job_list state="running">
922 <JB_job_number>2505560</JB_job_number>
923 <JAT_prio>0.30051</JAT_prio>
924--
925 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
926 <slots>1</slots>
927 </job_list>
928 <job_list state="running">
929 <JB_job_number>2505736</JB_job_number>
930 <JAT_prio>0.30050</JAT_prio>
931--
932 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
933 <slots>1</slots>
934 </job_list>
935 <job_list state="running">
936 <JB_job_number>2506022</JB_job_number>
937 <JAT_prio>0.30048</JAT_prio>
938--
939 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
940 <slots>1</slots>
941 </job_list>
942 <job_list state="running">
943 <JB_job_number>2506167</JB_job_number>
944 <JAT_prio>0.30047</JAT_prio>
945--
946 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
947 <slots>1</slots>
948 </job_list>
949 <job_list state="running">
950 <JB_job_number>2506256</JB_job_number>
951 <JAT_prio>0.30046</JAT_prio>
952--
953 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
954 <slots>1</slots>
955 </job_list>
956 <job_list state="running">
957 <JB_job_number>2506465</JB_job_number>
958 <JAT_prio>0.30044</JAT_prio>
959--
960 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
961 <slots>1</slots>
962 </job_list>
963 <job_list state="running">
964 <JB_job_number>2506469</JB_job_number>
965 <JAT_prio>0.30044</JAT_prio>
966--
967 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
968 <slots>1</slots>
969 </job_list>
970 <job_list state="running">
971 <JB_job_number>2507336</JB_job_number>
972 <JAT_prio>0.30038</JAT_prio>
973--
974 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
975 <slots>1</slots>
976 </job_list>
977 <job_list state="running">
978 <JB_job_number>2507467</JB_job_number>
979 <JAT_prio>0.30037</JAT_prio>
980--
981 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
982 <slots>1</slots>
983 </job_list>
984 <job_list state="running">
985 <JB_job_number>2507648</JB_job_number>
986 <JAT_prio>0.30035</JAT_prio>
987--
988 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
989 <slots>1</slots>
990 </job_list>
991 <job_list state="running">
992 <JB_job_number>2508128</JB_job_number>
993 <JAT_prio>0.30032</JAT_prio>
994--
995 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
996 <slots>1</slots>
997 </job_list>
998 <job_list state="running">
999 <JB_job_number>2508654</JB_job_number>
1000 <JAT_prio>0.30028</JAT_prio>
1001--
1002 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1003 <slots>1</slots>
1004 </job_list>
1005 <job_list state="running">
1006 <JB_job_number>2508957</JB_job_number>
1007 <JAT_prio>0.30026</JAT_prio>
1008--
1009 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1010 <slots>1</slots>
1011 </job_list>
1012 <job_list state="running">
1013 <JB_job_number>2509147</JB_job_number>
1014 <JAT_prio>0.30024</JAT_prio>
1015--
1016 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1017 <slots>1</slots>
1018 </job_list>
1019 <job_list state="running">
1020 <JB_job_number>2509371</JB_job_number>
1021 <JAT_prio>0.30023</JAT_prio>
1022--
1023 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1024 <slots>1</slots>
1025 </job_list>
1026 <job_list state="running">
1027 <JB_job_number>2509808</JB_job_number>
1028 <JAT_prio>0.30020</JAT_prio>
1029--
1030 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1031 <slots>1</slots>
1032 </job_list>
1033 <job_list state="running">
1034 <JB_job_number>2509970</JB_job_number>
1035 <JAT_prio>0.30018</JAT_prio>
1036--
1037 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1038 <slots>1</slots>
1039 </job_list>
1040 <job_list state="running">
1041 <JB_job_number>2510310</JB_job_number>
1042 <JAT_prio>0.30016</JAT_prio>
1043--
1044 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1045 <slots>1</slots>
1046 </job_list>
1047 <job_list state="running">
1048 <JB_job_number>2510969</JB_job_number>
1049 <JAT_prio>0.30011</JAT_prio>
1050--
1051 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1052 <slots>1</slots>
1053 </job_list>
1054 <job_list state="running">
1055 <JB_job_number>2510971</JB_job_number>
1056 <JAT_prio>0.30011</JAT_prio>
1057--
1058 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1059 <slots>1</slots>
1060 </job_list>
1061 <job_list state="running">
1062 <JB_job_number>2512156</JB_job_number>
1063 <JAT_prio>0.30002</JAT_prio>
1064root@tools-grid-master:~# qstat -u '*' -xml | grep -A 10 tools-webgrid-lighttpd-1209
1065 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1066 <slots>1</slots>
1067 </job_list>
1068 <job_list state="running">
1069 <JB_job_number>609</JB_job_number>
1070 <JAT_prio>0.34984</JAT_prio>
1071 <JB_name>lighttpd-pagecounts</JB_name>
1072 <JB_owner>tools.pagecounts</JB_owner>
1073 <state>r</state>
1074 <JAT_start_time>2015-12-30T04:08:04</JAT_start_time>
1075 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1404.eqiad.wmflabs</queue_name>
1076--
1077 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1078 <slots>1</slots>
1079 </job_list>
1080 <job_list state="running">
1081 <JB_job_number>658</JB_job_number>
1082 <JAT_prio>0.34984</JAT_prio>
1083 <JB_name>lighttpd-add-information</JB_name>
1084 <JB_owner>tools.add-information</JB_owner>
1085 <state>r</state>
1086 <JAT_start_time>2015-12-30T04:09:04</JAT_start_time>
1087 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1409.eqiad.wmflabs</queue_name>
1088--
1089 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1090 <slots>1</slots>
1091 </job_list>
1092 <job_list state="running">
1093 <JB_job_number>662</JB_job_number>
1094 <JAT_prio>0.34984</JAT_prio>
1095 <JB_name>lighttpd-geograph2commons</JB_name>
1096 <JB_owner>tools.geograph2commons</JB_owner>
1097 <state>r</state>
1098 <JAT_start_time>2015-12-30T04:09:10</JAT_start_time>
1099 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1401.eqiad.wmflabs</queue_name>
1100--
1101 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1102 <slots>1</slots>
1103 </job_list>
1104 <job_list state="running">
1105 <JB_job_number>681</JB_job_number>
1106 <JAT_prio>0.34984</JAT_prio>
1107 <JB_name>lighttpd-wikipedia-contributor-locations</JB_name>
1108 <JB_owner>tools.wikipedia-contributor-locations</JB_owner>
1109 <state>r</state>
1110 <JAT_start_time>2015-12-30T04:09:34</JAT_start_time>
1111 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1208.eqiad.wmflabs</queue_name>
1112--
1113 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1114 <slots>1</slots>
1115 </job_list>
1116 <job_list state="running">
1117 <JB_job_number>764</JB_job_number>
1118 <JAT_prio>0.34983</JAT_prio>
1119 <JB_name>lighttpd-wd-art</JB_name>
1120 <JB_owner>tools.wd-art</JB_owner>
1121 <state>r</state>
1122 <JAT_start_time>2015-12-30T04:10:32</JAT_start_time>
1123 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1208.eqiad.wmflabs</queue_name>
1124--
1125 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1126 <slots>1</slots>
1127 </job_list>
1128 <job_list state="running">
1129 <JB_job_number>784</JB_job_number>
1130 <JAT_prio>0.34983</JAT_prio>
1131 <JB_name>lighttpd-video2commons</JB_name>
1132 <JB_owner>tools.video2commons</JB_owner>
1133 <state>r</state>
1134 <JAT_start_time>2015-12-30T04:11:00</JAT_start_time>
1135 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1401.eqiad.wmflabs</queue_name>
1136--
1137 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1138 <slots>1</slots>
1139 </job_list>
1140 <job_list state="running">
1141 <JB_job_number>813</JB_job_number>
1142 <JAT_prio>0.34983</JAT_prio>
1143 <JB_name>lighttpd-languageproofing-ui</JB_name>
1144 <JB_owner>tools.languageproofing-ui</JB_owner>
1145 <state>r</state>
1146 <JAT_start_time>2015-12-30T04:11:28</JAT_start_time>
1147 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1410.eqiad.wmflabs</queue_name>
1148--
1149 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1150 <slots>1</slots>
1151 </job_list>
1152 <job_list state="running">
1153 <JB_job_number>819</JB_job_number>
1154 <JAT_prio>0.34983</JAT_prio>
1155 <JB_name>lighttpd-wm-bot</JB_name>
1156 <JB_owner>tools.wm-bot</JB_owner>
1157 <state>r</state>
1158 <JAT_start_time>2015-12-30T04:11:36</JAT_start_time>
1159 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1203.eqiad.wmflabs</queue_name>
1160--
1161 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1162 <slots>1</slots>
1163 </job_list>
1164 <job_list state="running">
1165 <JB_job_number>822</JB_job_number>
1166 <JAT_prio>0.34983</JAT_prio>
1167 <JB_name>uwsgi-python-watroles</JB_name>
1168 <JB_owner>tools.watroles</JB_owner>
1169 <state>r</state>
1170 <JAT_start_time>2015-12-30T04:11:41</JAT_start_time>
1171 <queue_name>webgrid-generic@tools-webgrid-generic-1401.eqiad.wmflabs</queue_name>
1172--
1173 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1174 <slots>1</slots>
1175 </job_list>
1176 <job_list state="running">
1177 <JB_job_number>826</JB_job_number>
1178 <JAT_prio>0.34983</JAT_prio>
1179 <JB_name>lighttpd-steinsplitter</JB_name>
1180 <JB_owner>tools.steinsplitter</JB_owner>
1181 <state>r</state>
1182 <JAT_start_time>2015-12-30T04:11:47</JAT_start_time>
1183 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1210.eqiad.wmflabs</queue_name>
1184--
1185 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1186 <slots>1</slots>
1187 </job_list>
1188 <job_list state="running">
1189 <JB_job_number>859</JB_job_number>
1190 <JAT_prio>0.34983</JAT_prio>
1191 <JB_name>lighttpd-yellowbot</JB_name>
1192 <JB_owner>tools.yellowbot</JB_owner>
1193 <state>r</state>
1194 <JAT_start_time>2015-12-30T04:12:13</JAT_start_time>
1195 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1410.eqiad.wmflabs</queue_name>
1196--
1197 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1198 <slots>1</slots>
1199 </job_list>
1200 <job_list state="running">
1201 <JB_job_number>881</JB_job_number>
1202 <JAT_prio>0.34983</JAT_prio>
1203 <JB_name>uwsgi-python-fab-proxy</JB_name>
1204 <JB_owner>tools.fab-proxy</JB_owner>
1205 <state>r</state>
1206 <JAT_start_time>2015-12-30T04:12:46</JAT_start_time>
1207 <queue_name>webgrid-generic@tools-webgrid-generic-1403.eqiad.wmflabs</queue_name>
1208--
1209 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1210 <slots>1</slots>
1211 </job_list>
1212 <job_list state="running">
1213 <JB_job_number>979</JB_job_number>
1214 <JAT_prio>0.34983</JAT_prio>
1215 <JB_name>lighttpd-fc-importer</JB_name>
1216 <JB_owner>tools.fc-importer</JB_owner>
1217 <state>Rr</state>
1218 <JAT_start_time>2016-01-19T14:32:08</JAT_start_time>
1219 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1411.tools.eqiad.wmflabs</queue_name>
1220--
1221 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1222 <slots>1</slots>
1223 </job_list>
1224 <job_list state="running">
1225 <JB_job_number>985</JB_job_number>
1226 <JAT_prio>0.34983</JAT_prio>
1227 <JB_name>lighttpd-wikidata-primary-sources</JB_name>
1228 <JB_owner>tools.wikidata-primary-sources</JB_owner>
1229 <state>r</state>
1230 <JAT_start_time>2015-12-30T04:14:53</JAT_start_time>
1231 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1406.eqiad.wmflabs</queue_name>
1232--
1233 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1234 <slots>1</slots>
1235 </job_list>
1236 <job_list state="running">
1237 <JB_job_number>1115</JB_job_number>
1238 <JAT_prio>0.34982</JAT_prio>
1239 <JB_name>lighttpd-inactiveadmins</JB_name>
1240 <JB_owner>tools.inactiveadmins</JB_owner>
1241 <state>r</state>
1242 <JAT_start_time>2015-12-30T04:16:29</JAT_start_time>
1243 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1244 <slots>1</slots>
1245 </job_list>
1246 <job_list state="running">
1247 <JB_job_number>1116</JB_job_number>
1248 <JAT_prio>0.34982</JAT_prio>
1249 <JB_name>lighttpd-wikidata-todo</JB_name>
1250 <JB_owner>tools.wikidata-todo</JB_owner>
1251 <state>r</state>
1252 <JAT_start_time>2015-12-30T04:16:30</JAT_start_time>
1253 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1402.eqiad.wmflabs</queue_name>
1254--
1255 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1256 <slots>1</slots>
1257 </job_list>
1258 <job_list state="running">
1259 <JB_job_number>1200</JB_job_number>
1260 <JAT_prio>0.34982</JAT_prio>
1261 <JB_name>lighttpd-joanjoc</JB_name>
1262 <JB_owner>tools.joanjoc</JB_owner>
1263 <state>r</state>
1264 <JAT_start_time>2015-12-30T04:18:04</JAT_start_time>
1265 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1409.eqiad.wmflabs</queue_name>
1266--
1267 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1268 <slots>1</slots>
1269 </job_list>
1270 <job_list state="running">
1271 <JB_job_number>1213</JB_job_number>
1272 <JAT_prio>0.34982</JAT_prio>
1273 <JB_name>lighttpd-matvaretabellen</JB_name>
1274 <JB_owner>tools.matvaretabellen</JB_owner>
1275 <state>r</state>
1276 <JAT_start_time>2015-12-30T04:18:24</JAT_start_time>
1277 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1203.eqiad.wmflabs</queue_name>
1278--
1279 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1280 <slots>1</slots>
1281 </job_list>
1282 <job_list state="running">
1283 <JB_job_number>1264</JB_job_number>
1284 <JAT_prio>0.34982</JAT_prio>
1285 <JB_name>lighttpd-igloo</JB_name>
1286 <JB_owner>tools.igloo</JB_owner>
1287 <state>r</state>
1288 <JAT_start_time>2015-12-30T04:19:21</JAT_start_time>
1289 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1290 <slots>1</slots>
1291 </job_list>
1292 <job_list state="running">
1293 <JB_job_number>1265</JB_job_number>
1294 <JAT_prio>0.34982</JAT_prio>
1295 <JB_name>lighttpd-plantel2wiki</JB_name>
1296 <JB_owner>tools.plantel2wiki</JB_owner>
1297 <state>r</state>
1298 <JAT_start_time>2015-12-30T04:19:22</JAT_start_time>
1299 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1205.eqiad.wmflabs</queue_name>
1300--
1301 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1302 <slots>1</slots>
1303 </job_list>
1304 <job_list state="running">
1305 <JB_job_number>1278</JB_job_number>
1306 <JAT_prio>0.34982</JAT_prio>
1307 <JB_name>lighttpd-npp</JB_name>
1308 <JB_owner>tools.npp</JB_owner>
1309 <state>r</state>
1310 <JAT_start_time>2015-12-30T04:19:42</JAT_start_time>
1311 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1401.eqiad.wmflabs</queue_name>
1312--
1313 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1314 <slots>1</slots>
1315 </job_list>
1316 <job_list state="running">
1317 <JB_job_number>1320</JB_job_number>
1318 <JAT_prio>0.34982</JAT_prio>
1319 <JB_name>lighttpd-serviceawards</JB_name>
1320 <JB_owner>tools.serviceawards</JB_owner>
1321 <state>r</state>
1322 <JAT_start_time>2015-12-30T04:20:06</JAT_start_time>
1323 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1205.eqiad.wmflabs</queue_name>
1324--
1325 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1326 <slots>1</slots>
1327 </job_list>
1328 <job_list state="running">
1329 <JB_job_number>1355</JB_job_number>
1330 <JAT_prio>0.34982</JAT_prio>
1331 <JB_name>lighttpd-convert</JB_name>
1332 <JB_owner>tools.convert</JB_owner>
1333 <state>r</state>
1334 <JAT_start_time>2015-12-30T04:20:30</JAT_start_time>
1335 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1401.eqiad.wmflabs</queue_name>
1336--
1337 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1338 <slots>1</slots>
1339 </job_list>
1340 <job_list state="running">
1341 <JB_job_number>1362</JB_job_number>
1342 <JAT_prio>0.34982</JAT_prio>
1343 <JB_name>lighttpd-heritage</JB_name>
1344 <JB_owner>tools.heritage</JB_owner>
1345 <state>r</state>
1346 <JAT_start_time>2015-12-30T04:20:42</JAT_start_time>
1347 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1205.eqiad.wmflabs</queue_name>
1348--
1349 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1350 <slots>1</slots>
1351 </job_list>
1352 <job_list state="running">
1353 <JB_job_number>1399</JB_job_number>
1354 <JAT_prio>0.34982</JAT_prio>
1355 <JB_name>lighttpd-personabot</JB_name>
1356 <JB_owner>tools.personabot</JB_owner>
1357 <state>r</state>
1358 <JAT_start_time>2015-12-30T04:21:27</JAT_start_time>
1359 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1205.eqiad.wmflabs</queue_name>
1360--
1361 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1362 <slots>1</slots>
1363 </job_list>
1364 <job_list state="running">
1365 <JB_job_number>1433</JB_job_number>
1366 <JAT_prio>0.34982</JAT_prio>
1367 <JB_name>lighttpd-deskana</JB_name>
1368 <JB_owner>tools.deskana</JB_owner>
1369 <state>r</state>
1370 <JAT_start_time>2015-12-30T04:22:04</JAT_start_time>
1371 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1409.eqiad.wmflabs</queue_name>
1372--
1373 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1374 <slots>1</slots>
1375 </job_list>
1376 <job_list state="running">
1377 <JB_job_number>1447</JB_job_number>
1378 <JAT_prio>0.34982</JAT_prio>
1379 <JB_name>lighttpd-superzerocool</JB_name>
1380 <JB_owner>tools.superzerocool</JB_owner>
1381 <state>r</state>
1382 <JAT_start_time>2015-12-30T04:22:27</JAT_start_time>
1383 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1410.eqiad.wmflabs</queue_name>
1384--
1385 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1386 <slots>1</slots>
1387 </job_list>
1388 <job_list state="running">
1389 <JB_job_number>36142</JB_job_number>
1390 <JAT_prio>0.62624</JAT_prio>
1391 <JB_name>rahu</JB_name>
1392 <JB_owner>tools.asurabot</JB_owner>
1393 <state>Rr</state>
1394 <JAT_start_time>2015-12-30T03:54:44</JAT_start_time>
1395 <queue_name>continuous@tools-exec-1219.eqiad.wmflabs</queue_name>
1396--
1397 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1398 <slots>1</slots>
1399 </job_list>
1400 <job_list state="running">
1401 <JB_job_number>2231271</JB_job_number>
1402 <JAT_prio>0.31960</JAT_prio>
1403 <JB_name>mg</JB_name>
1404 <JB_owner>tools.giftbot</JB_owner>
1405 <state>r</state>
1406 <JAT_start_time>2016-01-12T03:56:37</JAT_start_time>
1407 <queue_name>continuous@tools-exec-1221.tools.eqiad.wmflabs</queue_name>
1408--
1409 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1410 <slots>1</slots>
1411 </job_list>
1412 <job_list state="running">
1413 <JB_job_number>2493921</JB_job_number>
1414 <JAT_prio>0.30137</JAT_prio>
1415 <JB_name>kasparbot_nt2</JB_name>
1416 <JB_owner>tools.kasparbot</JB_owner>
1417 <state>r</state>
1418 <JAT_start_time>2016-01-20T00:00:32</JAT_start_time>
1419 <queue_name>task@tools-exec-1207.eqiad.wmflabs</queue_name>
1420--
1421 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1422 <slots>1</slots>
1423 </job_list>
1424 <job_list state="running">
1425 <JB_job_number>2499386</JB_job_number>
1426 <JAT_prio>0.30097</JAT_prio>
1427 <JB_name>lighttpd-osm4wiki</JB_name>
1428 <JB_owner>tools.osm4wiki</JB_owner>
1429 <state>r</state>
1430 <JAT_start_time>2016-01-20T04:07:09</JAT_start_time>
1431 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1208.eqiad.wmflabs</queue_name>
1432--
1433 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1434 <slots>1</slots>
1435 </job_list>
1436 <job_list state="running">
1437 <JB_job_number>2501443</JB_job_number>
1438 <JAT_prio>0.30082</JAT_prio>
1439 <JB_name>lighttpd-ftl</JB_name>
1440 <JB_owner>tools.ftl</JB_owner>
1441 <state>r</state>
1442 <JAT_start_time>2016-01-20T05:40:43</JAT_start_time>
1443 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1444 <slots>1</slots>
1445 </job_list>
1446 <job_list state="running">
1447 <JB_job_number>2501498</JB_job_number>
1448 <JAT_prio>0.30081</JAT_prio>
1449 <JB_name>lighttpd-toolschecker</JB_name>
1450 <JB_owner>tools.toolschecker</JB_owner>
1451 <state>dt</state>
1452 <JAT_start_time>2016-01-20T14:03:55</JAT_start_time>
1453 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1454 <slots>1</slots>
1455 </job_list>
1456 <job_list state="running">
1457 <JB_job_number>2501712</JB_job_number>
1458 <JAT_prio>0.30080</JAT_prio>
1459 <JB_name>twitter.sh</JB_name>
1460 <JB_owner>tools.avicbot</JB_owner>
1461 <state>t</state>
1462 <JAT_start_time>2016-01-20T14:05:02</JAT_start_time>
1463 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1464 <slots>1</slots>
1465 </job_list>
1466 <job_list state="running">
1467 <JB_job_number>2502115</JB_job_number>
1468 <JAT_prio>0.30077</JAT_prio>
1469 <JB_name>simple_cr</JB_name>
1470 <JB_owner>tools.russbot</JB_owner>
1471 <state>t</state>
1472 <JAT_start_time>2016-01-20T14:08:02</JAT_start_time>
1473 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1474 <slots>1</slots>
1475 </job_list>
1476 <job_list state="running">
1477 <JB_job_number>2502127</JB_job_number>
1478 <JAT_prio>0.30077</JAT_prio>
1479 <JB_name>sdab_procs</JB_name>
1480 <JB_owner>tools.dplbot</JB_owner>
1481 <state>dt</state>
1482 <JAT_start_time>2016-01-20T14:04:02</JAT_start_time>
1483 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1484 <slots>1</slots>
1485 </job_list>
1486 <job_list state="running">
1487 <JB_job_number>2502357</JB_job_number>
1488 <JAT_prio>0.30075</JAT_prio>
1489 <JB_name>lighttpd-dplbot</JB_name>
1490 <JB_owner>tools.dplbot</JB_owner>
1491 <state>r</state>
1492 <JAT_start_time>2016-01-20T06:24:19</JAT_start_time>
1493 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1205.eqiad.wmflabs</queue_name>
1494--
1495 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1496 <slots>1</slots>
1497 </job_list>
1498 <job_list state="running">
1499 <JB_job_number>2502515</JB_job_number>
1500 <JAT_prio>0.30074</JAT_prio>
1501 <JB_name>opentask</JB_name>
1502 <JB_owner>tools.suggestbot</JB_owner>
1503 <state>t</state>
1504 <JAT_start_time>2016-01-20T14:06:01</JAT_start_time>
1505 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1506 <slots>1</slots>
1507 </job_list>
1508 <job_list state="running">
1509 <JB_job_number>2502613</JB_job_number>
1510 <JAT_prio>0.30073</JAT_prio>
1511 <JB_name>subster-de</JB_name>
1512 <JB_owner>tools.asurabot</JB_owner>
1513 <state>r</state>
1514 <JAT_start_time>2016-01-20T06:36:08</JAT_start_time>
1515 <queue_name>task@tools-exec-1406.eqiad.wmflabs</queue_name>
1516--
1517 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1518 <slots>1</slots>
1519 </job_list>
1520 <job_list state="running">
1521 <JB_job_number>2503197</JB_job_number>
1522 <JAT_prio>0.30069</JAT_prio>
1523 <JB_name>lighttpd-random-featured</JB_name>
1524 <JB_owner>tools.random-featured</JB_owner>
1525 <state>r</state>
1526 <JAT_start_time>2016-01-20T07:00:35</JAT_start_time>
1527 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1205.eqiad.wmflabs</queue_name>
1528--
1529 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1530 <slots>1</slots>
1531 </job_list>
1532 <job_list state="running">
1533 <JB_job_number>2503470</JB_job_number>
1534 <JAT_prio>0.30067</JAT_prio>
1535 <JB_name>opentask</JB_name>
1536 <JB_owner>tools.suggestbot</JB_owner>
1537 <state>t</state>
1538 <JAT_start_time>2016-01-20T14:05:02</JAT_start_time>
1539 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1540 <slots>1</slots>
1541 </job_list>
1542 <job_list state="running">
1543 <JB_job_number>2503638</JB_job_number>
1544 <JAT_prio>0.30066</JAT_prio>
1545 <JB_name>log-mineral</JB_name>
1546 <JB_owner>tools.taxonbot</JB_owner>
1547 <state>r</state>
1548 <JAT_start_time>2016-01-20T07:22:05</JAT_start_time>
1549 <queue_name>task@tools-exec-1212.eqiad.wmflabs</queue_name>
1550--
1551 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1552 <slots>1</slots>
1553 </job_list>
1554 <job_list state="running">
1555 <JB_job_number>2504419</JB_job_number>
1556 <JAT_prio>0.30059</JAT_prio>
1557 <JB_name>random-featured-update</JB_name>
1558 <JB_owner>tools.random-featured</JB_owner>
1559 <state>t</state>
1560 <JAT_start_time>2016-01-20T14:05:19</JAT_start_time>
1561 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1562 <slots>1</slots>
1563 </job_list>
1564 <job_list state="running">
1565 <JB_job_number>2504691</JB_job_number>
1566 <JAT_prio>0.30057</JAT_prio>
1567 <JB_name>uwsgi-python-citationhunt</JB_name>
1568 <JB_owner>tools.citationhunt</JB_owner>
1569 <state>r</state>
1570 <JAT_start_time>2016-01-20T08:12:44</JAT_start_time>
1571 <queue_name>webgrid-generic@tools-webgrid-generic-1401.eqiad.wmflabs</queue_name>
1572--
1573 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1574 <slots>1</slots>
1575 </job_list>
1576 <job_list state="running">
1577 <JB_job_number>2504790</JB_job_number>
1578 <JAT_prio>0.30057</JAT_prio>
1579 <JB_name>helperbot11</JB_name>
1580 <JB_owner>tools.helperbot</JB_owner>
1581 <state>r</state>
1582 <JAT_start_time>2016-01-20T08:16:42</JAT_start_time>
1583 <queue_name>continuous@tools-exec-1206.eqiad.wmflabs</queue_name>
1584--
1585 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1586 <slots>1</slots>
1587 </job_list>
1588 <job_list state="running">
1589 <JB_job_number>2504909</JB_job_number>
1590 <JAT_prio>0.30056</JAT_prio>
1591 <JB_name>lighttpd-russbot</JB_name>
1592 <JB_owner>tools.russbot</JB_owner>
1593 <state>dt</state>
1594 <JAT_start_time>2016-01-20T14:07:48</JAT_start_time>
1595 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1596 <slots>1</slots>
1597 </job_list>
1598 <job_list state="running">
1599 <JB_job_number>2505023</JB_job_number>
1600 <JAT_prio>0.30055</JAT_prio>
1601 <JB_name>twitter.sh</JB_name>
1602 <JB_owner>tools.avicbot</JB_owner>
1603 <state>t</state>
1604 <JAT_start_time>2016-01-20T14:05:09</JAT_start_time>
1605 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1606 <slots>1</slots>
1607 </job_list>
1608 <job_list state="running">
1609 <JB_job_number>2505040</JB_job_number>
1610 <JAT_prio>0.30055</JAT_prio>
1611 <JB_name>veblenbot-size</JB_name>
1612 <JB_owner>tools.veblenbot</JB_owner>
1613 <state>t</state>
1614 <JAT_start_time>2016-01-20T14:05:14</JAT_start_time>
1615 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1616 <slots>1</slots>
1617 </job_list>
1618 <job_list state="running">
1619 <JB_job_number>2505083</JB_job_number>
1620 <JAT_prio>0.30055</JAT_prio>
1621 <JB_name>rdallquote</JB_name>
1622 <JB_owner>tools.avicbot</JB_owner>
1623 <state>r</state>
1624 <JAT_start_time>2016-01-20T08:30:22</JAT_start_time>
1625 <queue_name>task@tools-exec-1215.eqiad.wmflabs</queue_name>
1626--
1627 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1628 <slots>1</slots>
1629 </job_list>
1630 <job_list state="running">
1631 <JB_job_number>2505560</JB_job_number>
1632 <JAT_prio>0.30051</JAT_prio>
1633 <JB_name>uwsgi-python-copyvios</JB_name>
1634 <JB_owner>tools.copyvios</JB_owner>
1635 <state>r</state>
1636 <JAT_start_time>2016-01-20T08:52:20</JAT_start_time>
1637 <queue_name>webgrid-generic@tools-webgrid-generic-1401.eqiad.wmflabs</queue_name>
1638--
1639 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1640 <slots>1</slots>
1641 </job_list>
1642 <job_list state="running">
1643 <JB_job_number>2505736</JB_job_number>
1644 <JAT_prio>0.30050</JAT_prio>
1645 <JB_name>dewikinews-rss-update</JB_name>
1646 <JB_owner>tools.dewikinews-rss</JB_owner>
1647 <state>t</state>
1648 <JAT_start_time>2016-01-20T14:05:19</JAT_start_time>
1649 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1650 <slots>1</slots>
1651 </job_list>
1652 <job_list state="running">
1653 <JB_job_number>2506022</JB_job_number>
1654 <JAT_prio>0.30048</JAT_prio>
1655 <JB_name>sdab_procs</JB_name>
1656 <JB_owner>tools.dplbot</JB_owner>
1657 <state>dt</state>
1658 <JAT_start_time>2016-01-20T14:04:02</JAT_start_time>
1659 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1660 <slots>1</slots>
1661 </job_list>
1662 <job_list state="running">
1663 <JB_job_number>2506167</JB_job_number>
1664 <JAT_prio>0.30047</JAT_prio>
1665 <JB_name>twitter.sh</JB_name>
1666 <JB_owner>tools.avicbot</JB_owner>
1667 <state>t</state>
1668 <JAT_start_time>2016-01-20T14:05:09</JAT_start_time>
1669 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1670 <slots>1</slots>
1671 </job_list>
1672 <job_list state="running">
1673 <JB_job_number>2506256</JB_job_number>
1674 <JAT_prio>0.30046</JAT_prio>
1675 <JB_name>lighttpd-whois</JB_name>
1676 <JB_owner>tools.whois</JB_owner>
1677 <state>r</state>
1678 <JAT_start_time>2016-01-20T09:24:14</JAT_start_time>
1679 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1410.eqiad.wmflabs</queue_name>
1680--
1681 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1682 <slots>1</slots>
1683 </job_list>
1684 <job_list state="running">
1685 <JB_job_number>2506465</JB_job_number>
1686 <JAT_prio>0.30044</JAT_prio>
1687 <JB_name>lighttpd-xtools</JB_name>
1688 <JB_owner>tools.xtools</JB_owner>
1689 <state>dt</state>
1690 <JAT_start_time>2016-01-20T14:05:09</JAT_start_time>
1691 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1692 <slots>1</slots>
1693 </job_list>
1694 <job_list state="running">
1695 <JB_job_number>2506469</JB_job_number>
1696 <JAT_prio>0.30044</JAT_prio>
1697 <JB_name>lighttpd-xtools-ec</JB_name>
1698 <JB_owner>tools.xtools-ec</JB_owner>
1699 <state>r</state>
1700 <JAT_start_time>2016-01-20T09:35:09</JAT_start_time>
1701 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1205.eqiad.wmflabs</queue_name>
1702--
1703 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1704 <slots>1</slots>
1705 </job_list>
1706 <job_list state="running">
1707 <JB_job_number>2507336</JB_job_number>
1708 <JAT_prio>0.30038</JAT_prio>
1709 <JB_name>sdab_procs</JB_name>
1710 <JB_owner>tools.dplbot</JB_owner>
1711 <state>dt</state>
1712 <JAT_start_time>2016-01-20T14:04:01</JAT_start_time>
1713 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1714 <slots>1</slots>
1715 </job_list>
1716 <job_list state="running">
1717 <JB_job_number>2507467</JB_job_number>
1718 <JAT_prio>0.30037</JAT_prio>
1719 <JB_name>twitter.sh</JB_name>
1720 <JB_owner>tools.avicbot</JB_owner>
1721 <state>t</state>
1722 <JAT_start_time>2016-01-20T14:05:04</JAT_start_time>
1723 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1724 <slots>1</slots>
1725 </job_list>
1726 <job_list state="running">
1727 <JB_job_number>2507648</JB_job_number>
1728 <JAT_prio>0.30035</JAT_prio>
1729 <JB_name>twitter.sh</JB_name>
1730 <JB_owner>tools.avicbot</JB_owner>
1731 <state>t</state>
1732 <JAT_start_time>2016-01-20T14:05:02</JAT_start_time>
1733 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1734 <slots>1</slots>
1735 </job_list>
1736 <job_list state="running">
1737 <JB_job_number>2508128</JB_job_number>
1738 <JAT_prio>0.30032</JAT_prio>
1739 <JB_name>lighttpd-catscan2</JB_name>
1740 <JB_owner>tools.catscan2</JB_owner>
1741 <state>r</state>
1742 <JAT_start_time>2016-01-20T10:50:07</JAT_start_time>
1743 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1412.tools.eqiad.wmflabs</queue_name>
1744--
1745 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1746 <slots>1</slots>
1747 </job_list>
1748 <job_list state="running">
1749 <JB_job_number>2508654</JB_job_number>
1750 <JAT_prio>0.30028</JAT_prio>
1751 <JB_name>twitter.sh</JB_name>
1752 <JB_owner>tools.avicbot</JB_owner>
1753 <state>t</state>
1754 <JAT_start_time>2016-01-20T14:05:03</JAT_start_time>
1755 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1756 <slots>1</slots>
1757 </job_list>
1758 <job_list state="running">
1759 <JB_job_number>2508957</JB_job_number>
1760 <JAT_prio>0.30026</JAT_prio>
1761 <JB_name>twitter.sh</JB_name>
1762 <JB_owner>tools.avicbot</JB_owner>
1763 <state>t</state>
1764 <JAT_start_time>2016-01-20T14:05:09</JAT_start_time>
1765 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1766 <slots>1</slots>
1767 </job_list>
1768 <job_list state="running">
1769 <JB_job_number>2509147</JB_job_number>
1770 <JAT_prio>0.30024</JAT_prio>
1771 <JB_name>lighttpd-phetools</JB_name>
1772 <JB_owner>tools.phetools</JB_owner>
1773 <state>dt</state>
1774 <JAT_start_time>2016-01-20T14:03:55</JAT_start_time>
1775 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1776 <slots>1</slots>
1777 </job_list>
1778 <job_list state="running">
1779 <JB_job_number>2509371</JB_job_number>
1780 <JAT_prio>0.30023</JAT_prio>
1781 <JB_name>aan</JB_name>
1782 <JB_owner>tools.taxonbot</JB_owner>
1783 <state>r</state>
1784 <JAT_start_time>2016-01-20T11:48:08</JAT_start_time>
1785 <queue_name>task@tools-exec-1213.eqiad.wmflabs</queue_name>
1786--
1787 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1788 <slots>1</slots>
1789 </job_list>
1790 <job_list state="running">
1791 <JB_job_number>2509808</JB_job_number>
1792 <JAT_prio>0.30020</JAT_prio>
1793 <JB_name>d-rfdarchiver</JB_name>
1794 <JB_owner>tools.hazard-bot</JB_owner>
1795 <state>r</state>
1796 <JAT_start_time>2016-01-20T12:06:05</JAT_start_time>
1797 <queue_name>task@tools-exec-1406.eqiad.wmflabs</queue_name>
1798--
1799 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1800 <slots>1</slots>
1801 </job_list>
1802 <job_list state="running">
1803 <JB_job_number>2509970</JB_job_number>
1804 <JAT_prio>0.30018</JAT_prio>
1805 <JB_name>cron-pers-bek-to-db-bot</JB_name>
1806 <JB_owner>tools.spbot</JB_owner>
1807 <state>r</state>
1808 <JAT_start_time>2016-01-20T12:15:13</JAT_start_time>
1809 <queue_name>task@tools-exec-1208.eqiad.wmflabs</queue_name>
1810--
1811 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1812 <slots>1</slots>
1813 </job_list>
1814 <job_list state="running">
1815 <JB_job_number>2510310</JB_job_number>
1816 <JAT_prio>0.30016</JAT_prio>
1817 <JB_name>d-rfbotstatus</JB_name>
1818 <JB_owner>tools.hazard-bot</JB_owner>
1819 <state>r</state>
1820 <JAT_start_time>2016-01-20T12:30:18</JAT_start_time>
1821 <queue_name>task@tools-exec-1406.eqiad.wmflabs</queue_name>
1822--
1823 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1824 <slots>1</slots>
1825 </job_list>
1826 <job_list state="running">
1827 <JB_job_number>2510969</JB_job_number>
1828 <JAT_prio>0.30011</JAT_prio>
1829 <JB_name>dewikinews-rss-update</JB_name>
1830 <JB_owner>tools.dewikinews-rss</JB_owner>
1831 <state>t</state>
1832 <JAT_start_time>2016-01-20T14:05:20</JAT_start_time>
1833 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1834 <slots>1</slots>
1835 </job_list>
1836 <job_list state="running">
1837 <JB_job_number>2510971</JB_job_number>
1838 <JAT_prio>0.30011</JAT_prio>
1839 <JB_name>snapshots-updateSnaphots</JB_name>
1840 <JB_owner>tools.snapshots</JB_owner>
1841 <state>r</state>
1842 <JAT_start_time>2016-01-20T13:00:20</JAT_start_time>
1843 <queue_name>task@tools-exec-1209.eqiad.wmflabs</queue_name>
1844--
1845 <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
1846 <slots>1</slots>
1847 </job_list>
1848 <job_list state="running">
1849 <JB_job_number>2512156</JB_job_number>
1850 <JAT_prio>0.30002</JAT_prio>
1851 <JB_name>php_cleanupILH_DOM_arwiki</JB_name>
1852 <JB_owner>tools.liangent-php</JB_owner>
1853 <state>r</state>
1854 <JAT_start_time>2016-01-20T13:58:04</JAT_start_time>
1855 <queue_name>task@tools-exec-1405.eqiad.wmflabs</queue_name>

I see for example:

     <queue_name>webgrid-lighttpd@tools-webgrid-lighttpd-1209.eqiad.wmflabs</queue_name>
     <slots>1</slots>
   </job_list>
   <job_list state="running">
     <JB_job_number>2512156</JB_job_number>
<JAT_prio>0.30001</JAT_prio>

Looking at that job:

root@tools-grid-master:~# qstat -j 2512156
==============================================================
job_number:                 2512156
exec_file:                  job_scripts/2512156
submission_time:            Wed Jan 20 13:58:04 2016
owner:                      tools.liangent-php
uid:                        51117
group:                      tools.liangent-php
gid:                        51117
sge_o_home:                 /data/project/liangent-php
sge_o_log_name:             tools.liangent-php
sge_o_path:                 /data/project/liangent-shared/bin/crontab:/usr/local/bin:/usr/bin:/bin
sge_o_shell:                /bin/sh
sge_o_workdir:              /data/project/liangent-php
sge_o_host:                 tools-submit
account:                    sge
stderr_path_list:           NONE:NONE:/data/project/liangent-php/php_cleanupILH_DOM_arwiki.err
hard resource_list:         release=trusty,h_vmem=2097152k
mail_list:                  tools.liangent-php@tools.wmflabs.org
notify:                     FALSE
job_name:                   php_cleanupILH_DOM_arwiki
stdout_path_list:           NONE:NONE:/data/project/liangent-php/php_cleanupILH_DOM_arwiki.out
jobshare:                   0
hard_queue_list:            task
env_list:
job_args:                   /data/project/liangent-php/mw/maintenance/cleanupILH_DOM.php,--wiki=arwiki,--random-count=60,--inclusion
script_file:                /usr/bin/php5
usage    1:                 cpu=00:01:47, mem=36.55970 GBs, io=0.15175, vmem=395.570M, maxvmem=785.641M
scheduling info:            queue instance "webgrid-lighttpd@tools-webgrid-lighttpd-1201.eqiad.wmflabs" dropped because it is temporarily not available
                            queue instance "continuous@tools-exec-1216.eqiad.wmflabs" dropped because it is overloaded: np_load_avg=2.307500 (= 2.307500 + 0.50 * 0.000000 with nproc=4) >= 1.75
                            queue instance "mailq@tools-exec-1216.eqiad.wmflabs" dropped because it is overloaded: np_load_avg=2.307500 (= 2.307500 + 0.50 * 0.000000 with nproc=4) >= 2.25
                            queue instance "task@tools-exec-1216.eqiad.wmflabs" dropped because it is overloaded: np_load_avg=2.307500 (= 2.307500 + 0.50 * 0.000000 with nproc=4) >= 1.75
                            queue instance "continuous@tools-exec-1402.eqiad.wmflabs" dropped because it is disabled
                            queue instance "continuous@tools-exec-1203.eqiad.wmflabs" dropped because it is disabled
                            queue instance "mailq@tools-exec-1402.eqiad.wmflabs" dropped because it is disabled
                            queue instance "mailq@tools-exec-1203.eqiad.wmflabs" dropped because it is disabled
                            queue instance "task@tools-exec-1402.eqiad.wmflabs" dropped because it is disabled
                            queue instance "task@tools-exec-1203.eqiad.wmflabs" dropped because it is disabled

AFAICT SGE thinks the jobs are doing fine over on tools-webgrid-lighttpd-1209.eqiad.wmflabs, now I know that load is a native scheduling weight for SGE but I' not sure what happens when load increases as far as existing jobs. So I would like to find that out, but it seemed like jobs that were supporsed to be on the frozen host were OK according to the grid master. However, even though SGE at least thinks it has an idea of job state it seemed not able to manage them as reported in T124169. This is consistent with previous behavior we have seen where a stuck host will ignore a qdel, adn I cannot recall if qdel -f has been successful or not.

At this point I heavily suspect hung IO and that leads me to suspect NFS.

Looking at basic operations:

root@labcontrol1001:~# salt 'tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs' cmd.run "time timeout 10s touch /home/foo"
tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs:
    Command exited with non-zero status 124
    0.00user 0.00system 0:10.00elapsed 0%CPU (0avgtext+0avgdata 2800maxresident)k
    0inputs+16outputs (0major+448minor)pagefaults 0swaps

I can't even touch a file on an NFS mount. (timeout is used to report real remote failure instead of a hung proc and salt timeout)

I can however touch a file on the local disk

root@labcontrol1001:~# salt 'tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs' cmd.run "time timeout 10s touch /root/foo"
tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs:
    0.00user 0.00system 0:00.00elapsed 0%CPU (0avgtext+0avgdata 2800maxresident)k
    0inputs+0outputs (0major+462minor)pagefaults 0swaps

Again NFS file touch failure:

root@labcontrol1001:~# salt 'tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs' cmd.run "time timeout 10s touch /data/project/cbench/testing && time timeout 10s ls -al /data/project/cbench/testing"
tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs:
    Command exited with non-zero status 124
    0.00user 0.00system 0:10.00elapsed 0%CPU (0avgtext+0avgdata 2784maxresident)k
    0inputs+8outputs (0major+449minor)pagefaults 0swaps
root@labcontrol1001:~# salt 'tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs' cmd.run "time timeout 10s touch /data/project/cbench/testing && time timeout 10s ls -al /data/project/cbench/testing"
tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs:
    Command exited with non-zero status 124
    0.00user 0.00system 0:10.00elapsed 0%CPU (0avgtext+0avgdata 2784maxresident)k
    0inputs+8outputs (0major+450minor)pagefaults 0swaps

and local disk success

root@labcontrol1001:~# salt 'tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs' cmd.run "time timeout 10s touch /etc/foo"tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs:
    0.00user 0.00system 0:00.00elapsed 0%CPU (0avgtext+0avgdata 2784maxresident)k
    0inputs+0outputs (0major+460minor)pagefaults 0swaps

comparison for dealing with NFS files on another host during the outage period for tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs

root@tools-webgrid-lighttpd-1412:~# time timeout 10s touch /data/project/cbench/testing && ls -al /data/project/cbench/testing
-rw-r--r-- 1 root tools.cbench 0 Jan 20 14:22 /data/project/cbench/testing
real	0m0.005s
root@tools-webgrid-lighttpd-1412:~# time timeout 10s touch /data/project/cbench/testing2

real	0m0.029s
user	0m0.000s
sys	0m0.003s

When doing basic touch NFS operations on both the hung host and a similar but fine grid host I was able to see no tcp traffic sent to the server from the hung host. So it timed out on an operation for the NFS mount without trying to contact the server AFAICT. I believe this may be related to the hard mount retry behavior that has increasing backoffs. But similar operations on grid hosts that were not hung contacted the server as expected.

I was looking with a simple

137  tcpdump host 10.68.16.40
568  tcpdump host 10.68.17.152
569  tcpdump host 10.68.17.222

looking for hung proc notification

salt 'tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs' cmd.run "grep -Ri hung /var/log/*"

none found

fstab at the time of failure

# HEADER: This file was autogenerated at Fri Nov 20 21:23:12 +0000 2015
# HEADER: by puppet.  While it can still be managed manually, it
# HEADER: is definitely not recommended.
# /etc/fstab: static file system information.
# <file system>                                 <mount point>   <type>  <options>       <dump>  <pass>
proc	/proc	proc	defaults	0	0
/dev/vda1	/	ext4	defaults	0	0
/dev/vda2	swap	swap	defaults	0	0
labstore.svc.eqiad.wmnet:/project/tools/project	/data/project	nfs	rw,vers=4,bg,hard,intr,sec=sys,proto=tcp,port=0,noatime,lookupcache=none,nofsc	0	0
labstore.svc.eqiad.wmnet:/project/tools/home	/home	nfs	rw,vers=4,bg,hard,intr,sec=sys,proto=tcp,port=0,noatime,lookupcache=none,nofsc	0	0
labstore1003.eqiad.wmnet:/dumps	/public/dumps	nfs	ro,vers=4,bg,hard,intr,sec=sys,proto=tcp,port=0,noatime,lookupcache=none,nofsc	0	0
labstore.svc.eqiad.wmnet:/scratch	/data/scratch	nfs	rw,vers=4,bg,hard,intr,sec=sys,proto=tcp,port=0,noatime,lookupcache=none,nofsc	0	0
/data/project/.system/gridengine	/var/lib/gridengine	none	rw,bind	0	0
/dev/vd/big	-	swap	defaults	0	0
/dev/vd/separate-tmp	/tmp	ext4	nosuid,noexec,nodev,rw	0	0

trying to get lsof to tell me what files are held open on NFS things doesn't work out

``root@labcontrol1001:~# salt 'tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs' cmd.run "lsof -b | grep /data/project"

^CExiting on Ctrl-C
This job's jid is:
20160120143043908371```

lsof on another non hung host:

lsof -b | grep /data/project

<snip>

lighttpd   3040           tools.limesmap    3w      REG               0,23      28678   87762920 /data/project/limesmap/error.log
lighttpd   3040           tools.limesmap    5w      REG               0,23   13951808   87762921 /data/project/limesmap/access.log
php-cgi    3078           tools.limesmap    2w      REG               0,23      28678   87762920 /data/project/limesmap/error.log
php-cgi    3080           tools.limesmap    2w      REG               0,23      28678   87762920 /data/project/limesmap/error.log
php-cgi    3081           tools.limesmap    2w      REG               0,23      28678   87762920 /data/project/limesmap/error.log
php-cgi    3084           tools.limesmap    2w      REG               0,23      28678   87762920 /data/project/limesmap/error.log
php-cgi    3086           tools.limesmap    2w      REG               0,23      28678   87762920 /data/project/limesmap/error.log
php-cgi    3087           tools.limesmap    2w      REG               0,23      28678   87762920 /data/project/limesmap/error.log
gam_serve  3089           tools.limesmap  cwd       DIR               0,23       4096   87752892 /data/project/limesmap
lighttpd   4988      tools.file-siblings  cwd       DIR               0,23       4096   87687461 /data/project/file-sibli
</snip>

and load is still very high on the frozen host.

14:36:57 up 21 days, 11:19,  0 users,  load average: 116.15, 112.97, 109.74

I'm omitting some of me trying to kill procs in D state and salt crashing and freaking out

...continued ....

scfc added a comment.Jan 20 2016, 9:01 PM

I haven't checked if it is accurate, but my mental picture for SGE is that if the master and the execds don't exchange messages, they assume that everything's unchanged. So high load → execd can't receive deletion command from master/can't kill job/can't report killing back to master → master lists the job as "waiting to be deleted" (forever); qdel -f bypasses that and directly deletes the job from the SGE database on the master even if the responsible execd doesn't answer.

scfc reopened subtask T122719: tools-webgrid-lighttpd-1201 webservices and ssh unaccessible as Open.Jan 20 2016, 9:04 PM
• chasemp added a comment.Jan 20 2016, 9:06 PM

At this point I think:

  • processes and jobs on tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs were hung in a state of IO sleep where they try to do a file operation and block forever
  • NFS operations that were completing on other non hung hosts were failing on tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs
  • Load as a consequence of a lot of D state procs was inflated
  • the host had some difficulties communicating with NFS (I did not include this output above but here it is retrans)
root@labcontrol1001:~# salt 'tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs' cmd.run 'nfsstat'
tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs:
    Server rpc stats:
    calls      badcalls   badclnt    badauth    xdrcall
    0          0          0          0          0

    Client rpc stats:
    calls      retrans    authrefrsh
    2105907856   22         2105966151

    Client nfs v4:
--
root@labcontrol1001:~# salt 'tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs' cmd.run 'nfsstat'
tools-webgrid-lighttpd-1209.tools.eqiad.wmflabs:
    Server rpc stats:
    calls      badcalls   badclnt    badauth    xdrcall
    0          0          0          0          0

    Client rpc stats:
    calls      retrans    authrefrsh
    2105907857   22         2105966151

    Client nfs v4:

it seems like a trivial amount of retrans but from what I can tell anything non-0 is worth considering in the world of NFS

For instance another VM:

root@tools-webgrid-lighttpd-1412:~# nfsstat
Server rpc stats:
calls      badcalls   badclnt    badauth    xdrcall
0          0          0          0          0

Client rpc stats:
calls      retrans    authrefrsh
9711439    0          9711467

I found various instances online of large file transfers to NFS causing this kind of mysterious lockup and lack of recovery so I tried a few test cases after it came back post-reboot:

root@tools-webgrid-lighttpd-1209:/data/project/cbench# time wget http://releases.ubuntu.com/14.04.3/ubuntu-14.04.3-desktop-amd64.iso
--2016-01-20 15:50:06--  http://releases.ubuntu.com/14.04.3/ubuntu-14.04.3-desktop-amd64.iso
Resolving releases.ubuntu.com (releases.ubuntu.com)... 91.189.92.151, 2001:67c:1360:8c01::1d
Connecting to releases.ubuntu.com (releases.ubuntu.com)|91.189.92.151|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1054867456 (1006M) [application/x-iso9660-image]
Saving to: `ubuntu-14.04.3-desktop-amd64.iso'

100%[=====================================================================================>] 1,054,867,456 2.78M/s   in 4m 31s

2016-01-20 15:54:39 (3.71 MB/s) - `ubuntu-14.04.3-desktop-amd64.iso' saved [1054867456/1054867456]


real	4m32.815s
user	0m0.964s
sys	0m9.029s

I downloaded a few ISO's and such without issue.

On another VM that was not stuck or having issues I also tried a bit more aggressively to get NFS to lockup:

(on /tmp for local baseline)

root@tools-webgrid-lighttpd-1412:/tmp# `dd if=/dev/zero of=my-large-file bs=1M count=100`
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.186015 s, 564 MB/s

root@tools-webgrid-lighttpd-1412:/tmp# `dd if=/dev/zero of=test-large-file-temp bs=1M count=100`
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.0860212 s, 1.2 GB/s

On NFS 

root@tools-webgrid-lighttpd-1412:/data/project/cbench# `dd if=/dev/zero of=test-large-file-temp bs=1M count=100`
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 1.44466 s, 72.6 MB/s


root@tools-webgrid-lighttpd-1412:/data/project/cbench# `dd if=/dev/zero of=test-large-file-temp bs=1M count=1000`
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 16.0165 s, 65.5 MB/s


root@tools-webgrid-lighttpd-1412:/data/project/cbench# time `dd if=/dev/zero of=test-large-file-temp bs=1M count=2048`
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 39.5538 s, 54.3 MB/s

real	0m51.924s
user	0m0.001s
sys	0m2.955s

time `dd if=/dev/zero of=test-large-file-temp bs=1M count=8048`
8048+0 records in
8048+0 records out
8438939648 bytes (8.4 GB) copied, 362.326 s, 23.3 MB/s

real	6m3.027s
user	0m0.009s
sys	0m7.945s
root@tools-webgrid-lighttpd-1412:/data/project/cbench# du -sh test-large-file-temp
7.9G	test-large-file-temp

I think the large files thought is not related

...continued...

• chasemp added a comment.EditedJan 20 2016, 10:18 PM

tools-webgrid-lighttpd-1209 graphs:

Screen Shot 2016-01-20 at 3.56.05 PM.png (507×729 px, 49 KB)

Screen Shot 2016-01-20 at 3.58.04 PM.png (503×733 px, 89 KB)

Screen Shot 2016-01-20 at 3.59.25 PM.png (494×720 px, 60 KB)

Screen Shot 2016-01-20 at 4.00.09 PM.png (512×753 px, 71 KB)

Labstore graphs

Screen Shot 2016-01-20 at 3.44.43 PM.png (528×1 px, 65 KB)

Screen Shot 2016-01-20 at 4.02.00 PM.png (510×753 px, 64 KB)

Screen Shot 2016-01-20 at 4.02.52 PM.png (535×727 px, 91 KB)

It seems like we have IO issues and tasks on a VM will go into D state and block (maybe forever). At the moment I believe https://phabricator.wikimedia.org/T124038#1943953 is most likely the same root issue and that is in fact IO related and can be traced back to NFS. This is related to the hard mount option behavior in NFS terms.

labstore.svc.eqiad.wmnet:/scratch /data/scratch nfs rw,vers=4,bg,hard,intr,sec=sys,proto=tcp,port=0,noatime,lookupcache=none,nofsc 0 0

With this option we expect processes to hang indefinitely until the underlying storage mechanism returns. In theory this is a short blip when it is one but in the case in question it flakes out and never seems to return.

It's worth noting that our setup is meant to be somewhat pragmatic with the application of the intr setting allowing sane process management in the case of a hung process. With intr in theory we can combine the persistence of hard mount with the ability to send signals to processes that go awry. But intr has been deprecated for a long time now:

intr / nointr

Selects whether to allow signals to interrupt file operations on this mount point. If neither option is specified (or if nointr is specified), signals do not interrupt NFS file operations. If intr is specified, system calls return EINTR if an in-progress NFS operation is interrupted by a signal.

Using the intr option is preferred to using the soft option because it is significantly less likely to result in data corruption.

The intr / nointr mount option is deprecated after kernel 2.6.25. Only SIGKILL can interrupt a pending NFS operation on these kernels, and if specified, this mount option is ignored to provide backwards compatibility with older kernels.

That means in our current setup we expect processes to freeze and not return on underlying IO failure and we expect those processes not to sanely handle any signal other than SIGKILL. Which I believe is what happens with a qdel -f $job but can only be done by certain users in the tools context.

I don't know if we are gaining anything at the moment depending on the failure mode here with hard mounts. It seems historically hard + intr was the best compromise in a reasonably stable NFS environment but was removed. My thought is that removal is because the mode may be less desirable than soft with a very long timeout period where processes can be notified and handle IO failure higher up. As it is we are hanging processes and there is no way for users to handle this failure mode on their own.

notes:

http://serverfault.com/questions/405210/can-high-load-cause-server-hang-and-error-blocked-for-more-than-120-seconds

http://oss.sgi.com/archives/xfs/2003-08/msg00742.html

http://www.gossamer-threads.com/lists/linux/kernel/117786

http://lkml.iu.edu/hypermail/linux/kernel/0305.0/1257.html

http://wiki.linuxservertech.com/index.php?action=artikel&cat=9&id=220&artlang=en

http://blog.kevac.org/2013/02/uninterruptible-sleep-d-state.html

http://linux.die.net/man/5/nfs

...continued...

• chasemp added a comment.Jan 21 2016, 12:11 AM

In looking at poor client behavior it seems prudent to poke at the server. A few weeks ago I added some instrumentation to get statistics from labstore1001 (a few general ones seen above). We have some high load there as far as I have always thought rule of thumb is load greater than len(cpu) * 3 is not a good situation. The actual NFS traffic to me doesn't seem extreme but then again this is an 8 core box with 32G of RAM and I'm not sure how we should expect to scale given its multifaceted role as NFS server, backup server, and odd job server for user credentials. It seems true it sits above 50% CPU usage. I/O is of course the obvious bottleneck to check but I'm not sure what to expect from this setup without historical numbers.

iops for our four main RAID blocks (I think)

Screen Shot 2016-01-20 at 5.36.55 PM.png (513×840 px, 145 KB)

PV         VG       Fmt  Attr PSize  PFree
/dev/md123 labstore lvm2 a--  10.91t 934.24g
/dev/md124 labstore lvm2 a--  10.91t      0
/dev/md125 labstore lvm2 a--  10.91t   1.94t
/dev/md126 labstore lvm2 a--  10.91t 934.25g

cpu totals stacked

Screen Shot 2016-01-20 at 5.36.49 PM.png (531×660 px, 99 KB)

last week load avg

Screen Shot 2016-01-20 at 5.36.43 PM.png (531×656 px, 81 KB)

NFS reads/writes bytes

Screen Shot 2016-01-20 at 5.36.37 PM.png (531×659 px, 137 KB)

Memory I'm looking at the server now at:

KiB Mem:  32936912 total, 32486236 used,   450676 free,  6436928 buffers
KiB Swap:  1950716 total,        0 used,  1950716 free. 19823940 cached Mem
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  1      0 763976 6442288 19944688    0    0  2567  2442    1    1  2 37 52  9  0

One element that I have been interested in is our allocated threads since it is based on a configured value (64 in our case). I don't actually trust the values here which come from /proc/net/rpc/nfsd (like nfsstat)

Screen Shot 2016-01-20 at 5.42.32 PM.png (308×587 px, 23 KB)

I think maybe there is another /proc to check that is somewhat more valuable

root@labstore1001:~# cat /proc/fs/nfsd/pool_stats
# pool packets-arrived sockets-enqueued threads-woken threads-timedout
0 81359530141 54826448 81305562732 173526

I have a suspicion that the value of sockets-enqueued may mean we would benefit from upping our thread count (if possible with the current server constraints).

sockets-enqueued
	Counts how many times an NFS transport is enqueued to wait for
	an nfsd thread to service it, i.e. no nfsd thread was considered
	available.

	The circumstance this statistic tracks indicates that there was NFS
	network-facing work to be done but it couldn't be done immediately,
	thus introducing a small delay in servicing NFS calls.  The ideal
	rate of change for this counter is zero; significantly non-zero
	values may indicate a performance limitation.

	This can happen because there are too few nfsd threads in the thread
	pool for the NFS workload (the workload is thread-limited), in which
	case configuring more nfsd threads will probably improve the
	performance of the NFS workload.

The conventional wisdom I have found for retrans being non-0 in cases of all things being equal (network is ok, other infrastructure is ok) is that it may mean there are basically too many clients for threads to handle them. i.e. as client count increases so must thread count. It's hard to find a good strategy for thread count AFAICT. For what I have seen we would be considered severely undermatched at the moment.

old school oracle

To set the number of NFS threads, take the maximum of the following three suggestions:

Use 2 NFS threads for each active client process.

A client workstation usually only has one active process. However, a time-shared system that is an NFS client may have many active processes.

Use 16 to 32 NFS threads for each CPU.

Use roughly 16 for a SPARCclassic or a SPARCstation 5 system. Use 32 NFS threads for a system with a 60 MHz SuperSPARC processor.

Use 16 NFS threads for each 10 Mbits of network capacity.

I'm not sure how relevant honestly, but the gist I gather is that the best-practice is at least one thread available per concurrent client.

If my figuring makes sense here then we are looking at:

ss -n -t -o state established '( dport = :2049 or sport = :2049 )' | grep -v Address | awk '{print $4}' | cut -d ':' -f 1 | sort | uniq | wc

271 concurrent clients (and I didn't see it go below 250)
64 threads

which is interesting as the vast majority of our tools hosts report retrans errors with NFS

P2502

Granted this isn't weighted and it's based on total uptime etc. I don't know how seriously to take it, but my instinct is it's symptomatic of the greater issue for sure. I especially don't have an answer for why in one case this would be dire and in another benign.

notes:

https://www.kernel.org/doc/Documentation/filesystems/nfs/knfsd-stats.txt
http://serverfault.com/questions/181926/how-to-choose-the-number-of-nfsd-threads
http://docs.oracle.com/cd/E19620-01/805-4448/z40000292837/index.html

• chasemp added a comment.Jan 21 2016, 12:19 AM

A few notes on tools-webgrid-lighttpd-1201.tools.eqiad.wmflabs

top - 00:17:20 up 21 days, 21:02,  0 users,  load average: 168.20, 168.12, 167.9

P2503 d state top
1root@labcontrol1001:~# cat /tmp/tools-webgrid-lighttpd-1201.tools.eqiad.wmflabs.topd | awk '{ print $8, $0}' | sort | egrep ^D
2D 10048 scfc 20 0 97832 4096 3076 D 0 0.1 0:00.01 sshd
3D 10126 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
4D 10196 scfc 20 0 97832 4104 3084 D 0 0.1 0:00.00 sshd
5D 10404 scfc 20 0 97832 4092 3072 D 0 0.1 0:00.06 sshd
6D 10629 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.00 sshd
7D 10756 root 20 0 10336 608 508 D 0 0.0 0:00.00 start-stop-daem
8D 1077 sgeadmin 20 0 52292 3508 1996 D 0 0.0 25:26.26 sge_execd
9D 10838 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
10D 11216 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
11D 11362 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.00 sshd
12D 11387 scfc 20 0 97832 4096 3076 D 0 0.1 0:00.00 sshd
13D 11534 52315 20 0 52072 2716 1532 D 0 0.0 0:04.50 lighttpd
14D 11595 52315 20 0 16972 1284 1108 D 0 0.0 0:14.48 gam_server
15D 11597 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.00 sshd
16D 11829 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
17D 12039 tools.co 20 0 52200 2888 1664 D 0 0.0 0:04.85 lighttpd
18D 12072 tools.co 20 0 17072 1340 1128 D 0 0.0 4:21.65 gam_server
19D 12088 tools.ur 20 0 52072 2756 1600 D 0 0.0 0:05.84 lighttpd
20D 12121 tools.ur 20 0 16972 1288 1112 D 0 0.0 0:16.45 gam_server
21D 12135 tools.tr 20 0 52172 2912 1584 D 0 0.0 0:09.03 lighttpd
22D 12164 tools.tr 20 0 16972 1284 1108 D 0 0.0 0:38.97 gam_server
23D 12169 tools.ic 20 0 52360 2880 1560 D 0 0.0 0:04.65 lighttpd
24D 1216 scfc 20 0 97832 4092 3072 D 0 0.1 0:00.01 sshd
25D 12198 tools.ic 20 0 16972 1288 1112 D 0 0.0 1:09.28 gam_server
26D 1222 scfc 20 0 97832 4096 3084 D 0 0.1 0:00.01 sshd
27D 12304 root 20 0 10336 608 504 D 0 0.0 0:00.00 start-stop-daem
28D 12428 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
29D 12573 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
30D 13005 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
31D 13165 scfc 20 0 97832 4096 3076 D 0 0.1 0:00.01 sshd
32D 13240 scfc 20 0 97832 4104 3084 D 0 0.1 0:00.01 sshd
33D 13584 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
34D 13738 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
35D 14170 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.00 sshd
36D 14194 scfc 20 0 97832 4092 3076 D 0 0.1 0:00.01 sshd
37D 14584 root 20 0 10336 608 504 D 0 0.0 0:00.00 start-stop-daem
38D 14755 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
39D 14890 scfc 20 0 97832 4092 3076 D 0 0.1 0:00.00 sshd
40D 14933 scfc 20 0 97832 4092 3080 D 0 0.1 0:00.00 sshd
41D 15326 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.02 sshd
42D 15566 scfc 20 0 97832 4092 3072 D 0 0.1 0:00.02 sshd
43D 15922 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
44D 16062 scfc 20 0 97832 4104 3084 D 0 0.1 0:00.00 sshd
45D 16084 root 20 0 10336 612 504 D 0 0.0 0:00.00 start-stop-daem
46D 16103 scfc 20 0 97832 4104 3084 D 0 0.1 0:00.00 sshd
47D 16494 scfc 20 0 97832 4096 3084 D 0 0.1 0:00.00 sshd
48D 17229 scfc 20 0 97832 4100 3084 D 0 0.1 0:00.01 sshd
49D 1730 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
50D 1754 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.02 sshd
51D 17691 scfc 20 0 97832 4104 3084 D 0 0.1 0:00.00 sshd
52D 17713 scfc 20 0 97832 4088 3072 D 0 0.0 0:00.01 sshd
53D 17815 scfc 20 0 97832 4092 3076 D 0 0.1 0:00.01 sshd
54D 17907 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
55D 18425 scfc 20 0 97832 4104 3084 D 0 0.1 0:00.01 sshd
56D 18429 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.00 sshd
57D 18627 root 20 0 10336 608 508 D 0 0.0 0:00.00 start-stop-daem
58D 18811 scfc 20 0 97832 4088 3076 D 0 0.0 0:00.01 sshd
59D 19413 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
60D 19973 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
61D 20003 root 20 0 10336 612 504 D 0 0.0 0:00.00 start-stop-daem
62D 20232 scfc 20 0 97832 4088 3072 D 0 0.0 0:00.00 sshd
63D 20707 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
64D 20747 scfc 20 0 97832 4096 3076 D 0 0.1 0:00.00 sshd
65D 21139 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.00 sshd
66D 21143 root 20 0 10336 608 504 D 0 0.0 0:00.00 start-stop-daem
67D 21730 scfc 20 0 97832 4088 3076 D 0 0.0 0:00.00 sshd
68D 21871 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
69D 22465 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
70D 22673 root 20 0 10336 608 504 D 0 0.0 0:00.00 start-stop-daem
71D 22986 scfc 20 0 97832 4088 3072 D 0 0.0 0:00.00 sshd
72D 23024 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.00 sshd
73D 23656 scfc 20 0 97832 4096 3076 D 0 0.1 0:00.00 sshd
74D 2381 scfc 20 0 97832 4096 3076 D 0 0.1 0:00.03 sshd
75D 24031 root 20 0 10336 608 504 D 0 0.0 0:00.00 start-stop-daem
76D 24376 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
77D 24617 scfc 20 0 97832 4100 3084 D 0 0.1 0:00.00 sshd
78D 24810 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
79D 24955 root 20 0 10336 608 504 D 0 0.0 0:00.00 start-stop-daem
80D 25223 scfc 20 0 97832 4092 3076 D 0 0.1 0:00.00 sshd
81D 26501 scfc 20 0 97832 4092 3076 D 0 0.1 0:00.00 sshd
82D 27105 root 20 0 10336 608 508 D 0 0.0 0:00.00 start-stop-daem
83D 27668 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
84D 27820 root 20 0 10336 608 504 D 0 0.0 0:00.00 start-stop-daem
85D 28015 scfc 20 0 97832 4084 3072 D 0 0.0 0:00.01 sshd
86D 28100 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
87D 28662 scfc 20 0 97832 4088 3072 D 0 0.0 0:00.01 sshd
88D 28841 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
89D 28955 root 20 0 10336 608 504 D 0 0.0 0:00.00 start-stop-daem
90D 29264 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
91D 29471 scfc 20 0 97832 4096 3076 D 0 0.1 0:00.01 sshd
92D 29998 scfc 20 0 97832 4100 3084 D 0 0.1 0:00.00 sshd
93D 30496 scfc 20 0 97832 4092 3072 D 0 0.1 0:00.03 sshd
94D 30505 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
95D 30734 scfc 20 0 97832 4092 3076 D 0 0.1 0:00.00 sshd
96D 30798 root 20 0 10336 608 508 D 0 0.0 0:00.00 start-stop-daem
97D 31391 scfc 20 0 97832 4092 3076 D 0 0.1 0:00.00 sshd
98D 31637 root 20 0 10336 612 504 D 0 0.0 0:00.00 start-stop-daem
99D 31724 scfc 20 0 97832 4100 3084 D 0 0.1 0:00.01 sshd
100D 32164 scfc 20 0 97832 4092 3072 D 0 0.1 0:00.01 sshd
101D 32458 scfc 20 0 97832 4100 3084 D 0 0.1 0:00.00 sshd
102D 3278 root 39 19 4328 700 540 D 0 0.0 0:00.02 pidof
103D 4307 tools.ge 20 0 309m 10m 4120 D 0 0.1 0:08.10 php-cgi
104D 4310 tools.ph 20 0 53492 4300 1660 D 0 0.1 0:00.37 lighttpd
105D 4349 tools.ph 20 0 16972 1284 1112 D 0 0.0 0:01.31 gam_server
106D 4410 tools.ge 20 0 309m 9.9m 4128 D 0 0.1 0:05.59 php-cgi
107D 4482 root 20 0 10336 604 504 D 0 0.0 0:00.00 start-stop-daem
108D 4633 tools.ge 20 0 309m 9.9m 4116 D 0 0.1 0:01.23 php-cgi
109D 4665 tools.ph 20 0 0 0 0 D 0 0.0 0:00.22 python
110D 4675 tools.ge 20 0 309m 9.8m 4048 D 0 0.1 0:00.47 php-cgi
111D 4765 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
112D 501 root 20 0 10336 612 504 D 0 0.0 0:00.00 start-stop-daem
113D 5063 tools.ph 20 0 97832 4100 3084 D 0 0.1 0:00.01 sshd
114D 5065 tools.ph 20 0 97832 4100 3084 D 0 0.1 0:00.01 sshd
115D 5175 valhalla 20 0 97832 4100 3084 D 0 0.1 0:00.00 sshd
116D 5177 valhalla 20 0 97832 4100 3084 D 0 0.1 0:00.01 sshd
117D 5205 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
118D 5207 root 20 0 99048 4252 3216 D 0 0.1 0:00.02 sshd
119D 5323 valhalla 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
120D 5380 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
121D 5816 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.00 sshd
122D 5818 tools.ph 20 0 97832 4100 3084 D 0 0.1 0:00.01 sshd
123D 5829 scfc 20 0 97832 4096 3076 D 0 0.1 0:00.00 sshd
124D 620 scfc 20 0 97832 4092 3076 D 0 0.1 0:00.00 sshd
125D 6537 scfc 20 0 97832 4104 3084 D 0 0.1 0:00.00 sshd
126D 6960 root 20 0 10336 612 504 D 0 0.0 0:00.00 start-stop-daem
127D 6969 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
128D 7001 scfc 20 0 97832 4104 3084 D 0 0.1 0:00.01 sshd
129D 7328 scfc 20 0 97832 4088 3072 D 0 0.0 0:00.01 sshd
130D 7565 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
131D 7628 scfc 20 0 97832 4088 3072 D 0 0.0 0:00.01 sshd
132D 7702 scfc 20 0 97832 4100 3084 D 0 0.1 0:00.00 sshd
133D 7898 scfc 20 0 97832 4104 3084 D 0 0.1 0:00.01 sshd
134D 8051 scfc 20 0 97832 4092 3072 D 0 0.1 0:00.01 sshd
135D 8134 scfc 20 0 97832 4096 3084 D 0 0.1 0:00.00 sshd
136D 8314 root 20 0 10336 608 504 D 0 0.0 0:00.00 start-stop-daem
137D 8732 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
138D 8870 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.00 sshd
139D 8985 tools.po 20 0 52192 2808 1576 D 0 0.0 0:20.89 lighttpd
140D 9020 tools.po 20 0 17092 1384 1132 D 0 0.0 8:05.48 gam_server
141D 9036 tools.de 20 0 52076 2728 1556 D 0 0.0 0:04.19 lighttpd
142D 9057 scfc 20 0 97832 4088 3072 D 0 0.0 0:00.02 sshd
143D 9065 tools.de 20 0 17104 1284 1108 D 0 0.0 9:08.58 gam_server
144D 9123 52348 20 0 52192 2932 1588 D 0 0.0 0:08.64 lighttpd
145D 9152 52348 20 0 16972 1280 1108 D 0 0.0 0:34.77 gam_server
146D 9166 tools.ca 20 0 52072 2780 1616 D 0 0.0 0:04.55 lighttpd
147D 9202 tools.ca 20 0 16972 1284 1112 D 0 0.0 0:36.71 gam_server
148D 9220 tools.wi 20 0 52076 2496 1468 D 0 0.0 0:07.86 lighttpd
149D 9251 tools.wi 20 0 16972 1284 1112 D 0 0.0 0:16.56 gam_server
150D 9267 tools.sd 20 0 52152 2928 1684 D 0 0.0 0:04.60 lighttpd
151D 9332 tools.to 20 0 104m 4132 2580 D 0 0.1 0:05.15 lighttpd
152D 9343 root 20 0 175m 87m 4104 D 0 1.1 0:08.06 puppet
153D 9365 tools.to 20 0 16972 1288 1112 D 0 0.0 0:52.42 gam_server
154D 9370 tools.ya 20 0 52072 2708 1564 D 0 0.0 0:04.04 lighttpd
155D 9384 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.00 sshd
156D 9422 tools.tr 20 0 52076 2492 1468 D 0 0.0 0:06.10 lighttpd
157D 9451 tools.tr 20 0 16972 1284 1108 D 0 0.0 0:36.84 gam_server
158D 9467 tools.ge 20 0 57352 8148 1752 D 0 0.1 10:25.52 lighttpd
159D 9500 tools.ge 20 0 16972 1284 1112 D 0 0.0 0:21.31 gam_server
160D 9525 tools.he 20 0 52076 2500 1464 D 0 0.0 0:04.57 lighttpd
161D 9554 tools.he 20 0 16972 1280 1108 D 0 0.0 0:10.31 gam_server
162D 9570 tools.bl 20 0 52076 2492 1468 D 0 0.0 0:04.55 lighttpd
163D 9603 tools.bl 20 0 16972 1288 1112 D 0 0.0 0:44.65 gam_server
164D 9622 tools.be 20 0 52076 2728 1556 D 0 0.0 0:05.62 lighttpd
165D 9651 tools.be 20 0 16972 1284 1112 D 0 0.0 0:30.61 gam_server
166D 9689 tools.wi 20 0 52336 3020 1628 D 0 0.0 0:04.59 lighttpd
167D 9720 tools.wi 20 0 16972 1284 1108 D 0 0.0 0:34.42 gam_server
168D 9751 tools.ca 20 0 52076 2504 1468 D 0 0.0 0:04.72 lighttpd
169D 9780 tools.ca 20 0 16972 1284 1108 D 0 0.0 0:14.14 gam_server

Same behavior with local and nfs mounts:

root@labcontrol1001:~# salt 'tools-webgrid-lighttpd-1201.tools.eqiad.wmflabs' cmd.run "time timeout 10s touch /home/foo"
tools-webgrid-lighttpd-1201.tools.eqiad.wmflabs:
    Command exited with non-zero status 124
    0.00user 0.00system 0:10.00elapsed 0%CPU (0avgtext+0avgdata 2800maxresident)k
    0inputs+8outputs (0major+450minor)pagefaults 0swaps

root@labcontrol1001:~# salt 'tools-webgrid-lighttpd-1201.tools.eqiad.wmflabs' cmd.run "time timeout 10s touch /root/foo"
tools-webgrid-lighttpd-1201.tools.eqiad.wmflabs:
    0.00user 0.00system 0:00.00elapsed 0%CPU (0avgtext+0avgdata 2800maxresident)k
    0inputs+0outputs (0major+463minor)pagefaults 0swaps

root@labcontrol1001:~# salt 'tools-webgrid-lighttpd-1201.tools.eqiad.wmflabs' cmd.run "time timeout 10s touch /data/project/cbench/foobar"
tools-webgrid-lighttpd-1201.tools.eqiad.wmflabs:
    Command exited with non-zero status 124
    0.00user 0.00system 0:10.00elapsed 0%CPU (0avgtext+0avgdata 2784maxresident)k
    0inputs+8outputs (0major+449minor)pagefaults 0swaps

P2504 tools-webgrid-lighttpd-1201.tools.eqiad.wmflabs (high load and top)
1tools-webgrid-lighttpd-1201.tools.eqiad.wmflabs:
2 top - 00:17:20 up 21 days, 21:02, 0 users, load average: 168.20, 168.12, 167.9
3 Tasks: 661 total, 1 running, 621 sleeping, 0 stopped, 39 zombie
4 Cpu(s): 0.5%us, 0.1%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.1%st
5 Mem: 8178496k total, 2849760k used, 5328736k free, 242388k buffers
6 Swap: 25034744k total, 0k used, 25034744k free, 1435052k cached
7
8 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9 10213 root 20 0 12308 1592 848 R 4 0.0 0:00.04 top
10 1 root 20 0 20184 2220 1284 S 0 0.0 0:02.19 init
11 2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
12 3 root 20 0 0 0 0 S 0 0.0 0:20.39 ksoftirqd/0
13 5 root 20 0 0 0 0 S 0 0.0 0:00.08 kworker/u:0
14 6 root RT 0 0 0 0 S 0 0.0 0:00.96 migration/0
15 7 root RT 0 0 0 0 S 0 0.0 0:13.72 watchdog/0
16 8 root RT 0 0 0 0 S 0 0.0 0:00.82 migration/1
17 9 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/1:0
18 10 root 20 0 0 0 0 S 0 0.0 0:59.68 ksoftirqd/1
19 11 root 20 0 0 0 0 S 0 0.0 1:01.40 kworker/0:1
20 12 root RT 0 0 0 0 S 0 0.0 0:12.84 watchdog/1
21 13 root RT 0 0 0 0 S 0 0.0 0:01.75 migration/2
22 15 root 20 0 0 0 0 S 0 0.0 0:19.79 ksoftirqd/2
23 16 root RT 0 0 0 0 S 0 0.0 0:12.34 watchdog/2
24 17 root RT 0 0 0 0 S 0 0.0 0:01.09 migration/3
25 18 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/3:0
26 19 root 20 0 0 0 0 S 0 0.0 0:16.30 ksoftirqd/3
27 20 root RT 0 0 0 0 S 0 0.0 0:11.97 watchdog/3
28 21 root 0 -20 0 0 0 S 0 0.0 0:00.00 cpuset
29 22 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper
30 23 root 20 0 0 0 0 S 0 0.0 0:00.00 kdevtmpfs
31 24 root 0 -20 0 0 0 S 0 0.0 0:00.00 netns
32 25 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/u:1
33 26 root 20 0 0 0 0 S 0 0.0 0:08.59 sync_supers
34 27 root 20 0 0 0 0 S 0 0.0 0:00.44 bdi-default
35 28 root 0 -20 0 0 0 S 0 0.0 0:00.00 kintegrityd
36 29 root 0 -20 0 0 0 S 0 0.0 0:00.00 kblockd
37 30 root 0 -20 0 0 0 S 0 0.0 0:00.00 ata_sff
38 31 root 20 0 0 0 0 S 0 0.0 0:00.00 khubd
39 32 root 0 -20 0 0 0 S 0 0.0 0:00.00 md
40 33 root 20 0 0 0 0 S 0 0.0 0:43.19 kworker/3:1
41 35 root 20 0 0 0 0 S 0 0.0 0:21.39 khungtaskd
42 36 root 20 0 0 0 0 S 0 0.0 0:00.00 kswapd0
43 37 root 25 5 0 0 0 S 0 0.0 0:00.00 ksmd
44 38 root 39 19 0 0 0 S 0 0.0 0:00.00 khugepaged
45 39 root 20 0 0 0 0 S 0 0.0 0:00.00 fsnotify_mark
46 40 root 20 0 0 0 0 S 0 0.0 0:00.00 ecryptfs-kthrea
47 41 root 0 -20 0 0 0 S 0 0.0 0:00.00 crypto
48 49 root 0 -20 0 0 0 S 0 0.0 0:00.00 kthrotld
49 50 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_0
50 51 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_1
51 53 root 20 0 0 0 0 S 0 0.0 6:19.15 kworker/1:1
52 73 root 0 -20 0 0 0 S 0 0.0 0:00.00 devfreq_wq
53 210 root 20 0 0 0 0 S 0 0.0 0:14.93 kworker/2:1
54 211 root 0 -20 0 0 0 S 0 0.0 0:00.00 kdmflush
55 219 root 0 -20 0 0 0 S 0 0.0 0:00.00 kdmflush
56 227 root 20 0 0 0 0 S 0 0.0 0:50.64 jbd2/vda1-8
57 228 root 0 -20 0 0 0 S 0 0.0 0:00.00 ext4-dio-unwrit
58 310 root 20 0 17228 640 456 S 0 0.0 0:00.05 upstart-udev-br
59 315 root 20 0 15384 1400 740 S 0 0.0 0:00.04 udevd
60 329 root 20 0 19196 1008 724 S 0 0.0 0:06.11 rpcbind
61 335 statd 20 0 27336 1984 1288 S 0 0.0 0:00.00 rpc.statd
62 398 root 20 0 15260 912 344 S 0 0.0 0:00.00 udevd
63 399 root 20 0 15260 884 312 S 0 0.0 0:00.00 udevd
64 413 root 20 0 35860 1184 856 S 0 0.0 0:00.00 cron
65 416 root 20 0 4396 608 508 S 0 0.0 0:00.00 sh
66 418 root 20 0 4296 604 496 S 0 0.0 0:00.00 run-parts
67 453 root 20 0 0 0 0 S 0 0.0 0:03.23 jbd2/dm-1-8
68 454 root 0 -20 0 0 0 S 0 0.0 0:00.00 ext4-dio-unwrit
69 476 root 20 0 0 0 0 S 0 0.0 0:48.36 vballoon
70 477 root 0 -20 0 0 0 S 0 0.0 0:00.00 kpsmoused
71 492 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/0:2
72 495 root 20 0 4396 608 508 S 0 0.0 0:00.00 exim4-base
73 500 root 20 0 31396 612 516 S 0 0.0 0:00.00 xargs
74 501 root 20 0 10336 612 504 D 0 0.0 0:00.00 start-stop-daem
75 577 root 20 0 15032 1088 544 S 0 0.0 0:00.11 dhclient3
76 602 root 20 0 15184 396 192 S 0 0.0 0:00.01 upstart-socket-
77 605 root 0 -20 0 0 0 S 0 0.0 0:00.00 rpciod
78 620 scfc 20 0 97832 4092 3076 D 0 0.1 0:00.00 sshd
79 621 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
80 640 root 0 -20 0 0 0 S 0 0.0 0:00.00 nfsiod
81 641 root 20 0 0 0 0 S 0 0.0 0:26.40 kworker/2:2
82 649 root 20 0 0 0 0 S 0 0.0 0:00.00 nfsv4.0-svc
83 652 root 20 0 0 0 0 S 0 0.0 1:02.39 flush-253:0
84 703 ntp 20 0 39360 2836 1972 S 0 0.0 3:05.82 ntpd
85 720 diamond 20 0 506m 24m 5300 S 0 0.3 39:51.93 diamond
86 770 messageb 20 0 19580 676 452 S 0 0.0 0:00.00 dbus-daemon
87 776 syslog 20 0 241m 5188 1064 S 0 0.1 2:55.82 rsyslogd
88 838 root 20 0 33404 876 748 S 0 0.0 0:00.00 getty
89 843 root 20 0 33404 880 748 S 0 0.0 0:00.00 getty
90 847 root 20 0 588m 47m 9528 S 0 0.6 16:50.28 salt-minion
91 850 root 20 0 33404 876 748 S 0 0.0 0:00.00 getty
92 851 root 20 0 33404 876 748 S 0 0.0 0:00.00 getty
93 853 root 20 0 33404 876 748 S 0 0.0 0:00.00 getty
94 857 root 20 0 52568 2960 2336 S 0 0.0 0:01.10 sshd
95 867 daemon 20 0 10572 352 220 S 0 0.0 0:00.06 atd
96 869 root 20 0 12776 924 724 S 0 0.0 0:10.89 cron
97 899 root 20 0 15976 700 524 S 0 0.0 5:22.58 irqbalance
98 908 Debian-e 20 0 45388 1148 732 S 0 0.0 0:03.31 exim4
99 929 root 20 0 47044 1564 984 S 0 0.0 2:04.67 lldpd
100 931 _lldpd 20 0 42804 672 204 S 0 0.0 0:53.08 lldpd
101 940 root 20 0 6992 384 216 S 0 0.0 0:00.00 mcelog
102 953 nagios 20 0 23340 1108 616 S 0 0.0 5:07.04 nrpe
103 976 root 20 0 876m 3096 1576 S 0 0.0 4:15.34 nscd
104 1011 nslcd 20 0 437m 6428 1868 S 0 0.1 5:20.64 nslcd
105 1038 root 20 0 8768 520 408 S 0 0.0 0:00.01 inetd
106 1077 sgeadmin 20 0 52292 3508 1996 D 0 0.0 25:26.37 sge_execd
107 1216 scfc 20 0 97832 4092 3072 D 0 0.1 0:00.01 sshd
108 1217 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
109 1222 scfc 20 0 97832 4096 3084 D 0 0.1 0:00.01 sshd
110 1223 sshd 20 0 62672 1332 628 S 0 0.0 0:00.00 sshd
111 1248 www-data 20 0 44332 1044 292 S 0 0.0 1:28.46 lighttpd
112 1276 root 20 0 34320 820 704 S 0 0.0 0:00.00 getty
113 1277 root 20 0 7332 712 604 S 0 0.0 0:00.00 getty
114 1686 52348 20 0 307m 6900 1804 S 0 0.1 0:00.04 php-cgi
115 1701 52348 20 0 307m 6900 1804 S 0 0.1 0:00.05 php-cgi
116 1730 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
117 1731 sshd 20 0 62672 1340 628 S 0 0.0 0:00.00 sshd
118 1754 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.02 sshd
119 1755 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
120 1830 tools.tr 20 0 308m 7304 1924 S 0 0.1 0:00.21 php-cgi
121 1844 tools.tr 20 0 308m 7304 1924 S 0 0.1 0:00.22 php-cgi
122 2381 scfc 20 0 97832 4096 3076 D 0 0.1 0:00.03 sshd
123 2382 sshd 20 0 62672 1340 628 S 0 0.0 0:00.01 sshd
124 2887 root 20 0 35860 1184 856 S 0 0.0 0:00.00 cron
125 2890 root 20 0 4396 608 508 S 0 0.0 0:00.00 sh
126 2892 root 20 0 4296 604 496 S 0 0.0 0:00.00 run-parts
127 2944 root 20 0 4396 712 576 S 0 0.0 0:00.02 apt
128 3206 root 39 19 114m 45m 27m S 0 0.6 34:29.26 unattended-upgr
129 3235 root 39 19 119m 26m 8772 S 0 0.3 1:09.30 unattended-upgr
130 3273 root 39 19 41072 5620 932 S 0 0.1 0:00.03 dpkg
131 3274 root 39 19 79740 11m 2484 S 0 0.1 0:00.16 frontend
132 3276 root 39 19 36348 1464 1208 S 0 0.0 0:00.00 libssl1.0.0:amd
133 3278 root 39 19 4328 700 540 D 0 0.0 0:00.02 pidof
134 4307 tools.ge 20 0 309m 10m 4120 D 0 0.1 0:08.10 php-cgi
135 4308 sgeadmin 20 0 22548 2004 1548 S 0 0.0 0:00.00 sge_shepherd
136 4310 tools.ph 20 0 53492 4300 1660 D 0 0.1 0:00.37 lighttpd
137 4340 tools.ph 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
138 4342 tools.ph 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
139 4343 tools.ph 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
140 4344 tools.ph 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
141 4346 tools.ph 20 0 307m 7192 2008 S 0 0.1 0:00.00 php-cgi
142 4347 tools.ph 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
143 4349 tools.ph 20 0 16972 1284 1112 D 0 0.0 0:01.31 gam_server
144 4398 root 20 0 35860 1184 856 S 0 0.0 0:00.00 cron
145 4400 root 20 0 4396 612 508 S 0 0.0 0:00.00 sh
146 4403 root 20 0 4296 608 496 S 0 0.0 0:00.00 run-parts
147 4410 tools.ge 20 0 309m 9.9m 4128 D 0 0.1 0:05.59 php-cgi
148 4476 root 20 0 4396 612 508 S 0 0.0 0:00.00 exim4-base
149 4481 root 20 0 31396 608 516 S 0 0.0 0:00.00 xargs
150 4482 root 20 0 10336 604 504 D 0 0.0 0:00.00 start-stop-daem
151 4633 tools.ge 20 0 309m 9.9m 4116 D 0 0.1 0:01.23 php-cgi
152 4635 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.25 python <defunct>
153 4636 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.25 python <defunct>
154 4637 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.26 python <defunct>
155 4638 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.24 python <defunct>
156 4639 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.27 python <defunct>
157 4640 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.23 python <defunct>
158 4641 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.24 python <defunct>
159 4642 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.23 python <defunct>
160 4643 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.26 python <defunct>
161 4644 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.23 python <defunct>
162 4645 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.23 python <defunct>
163 4646 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.23 python <defunct>
164 4647 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.25 python <defunct>
165 4649 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.24 python <defunct>
166 4650 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.23 python <defunct>
167 4651 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.23 python <defunct>
168 4652 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.23 python <defunct>
169 4653 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.25 python <defunct>
170 4654 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.26 python <defunct>
171 4655 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.25 python <defunct>
172 4656 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.22 python <defunct>
173 4657 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.24 python <defunct>
174 4658 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.22 python <defunct>
175 4659 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.24 python <defunct>
176 4660 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.24 python <defunct>
177 4661 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.24 python <defunct>
178 4662 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.25 python <defunct>
179 4663 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.23 python <defunct>
180 4664 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.25 python <defunct>
181 4665 tools.ph 20 0 0 0 0 D 0 0.0 0:00.22 python
182 4666 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.23 python <defunct>
183 4667 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.23 python <defunct>
184 4668 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.26 python <defunct>
185 4669 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.23 python <defunct>
186 4670 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.23 python <defunct>
187 4671 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.25 python <defunct>
188 4672 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.26 python <defunct>
189 4673 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.23 python <defunct>
190 4674 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.23 python <defunct>
191 4675 tools.ge 20 0 309m 9.8m 4048 D 0 0.1 0:00.47 php-cgi
192 4676 tools.ph 20 0 0 0 0 Z 0 0.0 0:00.14 python <defunct>
193 4765 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
194 4766 sshd 20 0 62672 1336 628 S 0 0.0 0:00.01 sshd
195 5063 tools.ph 20 0 97832 4100 3084 D 0 0.1 0:00.01 sshd
196 5064 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
197 5065 tools.ph 20 0 97832 4100 3084 D 0 0.1 0:00.01 sshd
198 5066 sshd 20 0 62672 1332 628 S 0 0.0 0:00.01 sshd
199 5175 valhalla 20 0 97832 4100 3084 D 0 0.1 0:00.00 sshd
200 5176 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
201 5177 valhalla 20 0 97832 4100 3084 D 0 0.1 0:00.01 sshd
202 5178 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
203 5205 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
204 5206 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
205 5207 root 20 0 99048 4252 3216 D 0 0.1 0:00.02 sshd
206 5323 valhalla 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
207 5324 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
208 5380 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
209 5381 sshd 20 0 62672 1332 628 S 0 0.0 0:00.00 sshd
210 5816 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.00 sshd
211 5817 sshd 20 0 62672 1336 628 S 0 0.0 0:00.01 sshd
212 5818 tools.ph 20 0 97832 4100 3084 D 0 0.1 0:00.01 sshd
213 5819 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
214 5829 scfc 20 0 97832 4096 3076 D 0 0.1 0:00.00 sshd
215 5830 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
216 6309 root 20 0 0 0 0 S 0 0.0 0:01.98 flush-252:1
217 6537 scfc 20 0 97832 4104 3084 D 0 0.1 0:00.00 sshd
218 6538 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
219 6615 root 20 0 35860 1184 856 S 0 0.0 0:00.00 cron
220 6618 root 20 0 4396 612 508 S 0 0.0 0:00.00 sh
221 6620 root 20 0 4296 604 496 S 0 0.0 0:00.00 run-parts
222 6954 root 20 0 4396 604 504 S 0 0.0 0:00.00 exim4-base
223 6959 root 20 0 31396 608 516 S 0 0.0 0:00.00 xargs
224 6960 root 20 0 10336 612 504 D 0 0.0 0:00.00 start-stop-daem
225 6969 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
226 6970 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
227 7001 scfc 20 0 97832 4104 3084 D 0 0.1 0:00.01 sshd
228 7002 sshd 20 0 62672 1340 628 S 0 0.0 0:00.01 sshd
229 7328 scfc 20 0 97832 4088 3072 D 0 0.0 0:00.01 sshd
230 7329 sshd 20 0 62672 1336 632 S 0 0.0 0:00.00 sshd
231 7565 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
232 7566 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
233 7628 scfc 20 0 97832 4088 3072 D 0 0.0 0:00.01 sshd
234 7629 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
235 7702 scfc 20 0 97832 4100 3084 D 0 0.1 0:00.00 sshd
236 7703 sshd 20 0 62672 1332 628 S 0 0.0 0:00.00 sshd
237 7898 scfc 20 0 97832 4104 3084 D 0 0.1 0:00.01 sshd
238 7899 sshd 20 0 62672 1340 628 S 0 0.0 0:00.00 sshd
239 8051 scfc 20 0 97832 4092 3072 D 0 0.1 0:00.01 sshd
240 8052 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
241 8134 scfc 20 0 97832 4096 3084 D 0 0.1 0:00.00 sshd
242 8135 sshd 20 0 62672 1336 632 S 0 0.0 0:00.01 sshd
243 8231 root 20 0 35860 1184 856 S 0 0.0 0:00.00 cron
244 8234 root 20 0 4396 608 504 S 0 0.0 0:00.00 sh
245 8236 root 20 0 4296 608 496 S 0 0.0 0:00.00 run-parts
246 8308 root 20 0 4396 612 508 S 0 0.0 0:00.00 exim4-base
247 8313 root 20 0 31396 608 516 S 0 0.0 0:00.00 xargs
248 8314 root 20 0 10336 608 504 D 0 0.0 0:00.00 start-stop-daem
249 8732 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
250 8733 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
251 8870 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.00 sshd
252 8871 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
253 8983 sgeadmin 20 0 22548 1992 1544 S 0 0.0 0:00.00 sge_shepherd
254 8985 tools.po 20 0 52192 2808 1576 D 0 0.0 0:20.89 lighttpd
255 9011 tools.po 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
256 9013 tools.po 20 0 308m 8168 2664 S 0 0.1 0:01.26 php-cgi
257 9014 tools.po 20 0 308m 8304 2664 S 0 0.1 0:01.23 php-cgi
258 9015 tools.po 20 0 307m 13m 8880 S 0 0.2 0:00.03 php-cgi
259 9020 tools.po 20 0 17092 1384 1132 D 0 0.0 8:05.48 gam_server
260 9034 sgeadmin 20 0 22548 1996 1548 S 0 0.0 0:00.00 sge_shepherd
261 9036 tools.de 20 0 52076 2728 1556 D 0 0.0 0:04.19 lighttpd
262 9056 tools.de 20 0 307m 13m 8868 S 0 0.2 0:00.00 php-cgi
263 9057 scfc 20 0 97832 4088 3072 D 0 0.0 0:00.02 sshd
264 9058 tools.de 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
265 9059 tools.de 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
266 9060 tools.de 20 0 307m 13m 8868 S 0 0.2 0:00.03 php-cgi
267 9061 sshd 20 0 62672 1332 628 S 0 0.0 0:00.01 sshd
268 9062 tools.de 20 0 334m 7352 2220 S 0 0.1 0:00.14 php-cgi
269 9063 tools.de 20 0 334m 7304 2172 S 0 0.1 0:00.13 php-cgi
270 9065 tools.de 20 0 17104 1284 1108 D 0 0.0 9:08.58 gam_server
271 9121 sgeadmin 20 0 22548 1996 1548 S 0 0.0 0:00.00 sge_shepherd
272 9123 52348 20 0 52192 2932 1588 D 0 0.0 0:08.64 lighttpd
273 9143 52348 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
274 9145 52348 20 0 307m 6904 1804 S 0 0.1 0:00.02 php-cgi
275 9146 52348 20 0 318m 15m 2684 S 0 0.2 0:00.41 php-cgi
276 9147 52348 20 0 307m 13m 8880 S 0 0.2 0:00.01 php-cgi
277 9152 52348 20 0 16972 1280 1108 D 0 0.0 0:34.77 gam_server
278 9164 sgeadmin 20 0 22548 2000 1548 S 0 0.0 0:00.00 sge_shepherd
279 9166 tools.ca 20 0 52072 2780 1616 D 0 0.0 0:04.55 lighttpd
280 9192 tools.ca 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
281 9194 tools.ca 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
282 9195 tools.ca 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
283 9196 tools.ca 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
284 9198 tools.ca 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
285 9199 tools.ca 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
286 9200 tools.ca 20 0 4392 608 512 S 0 0.0 0:00.00 server.sh
287 9202 tools.ca 20 0 16972 1284 1112 D 0 0.0 0:36.71 gam_server
288 9205 tools.ca 20 0 382m 15m 3596 S 0 0.2 3:51.61 python
289 9218 sgeadmin 20 0 22548 2008 1552 S 0 0.0 0:00.00 sge_shepherd
290 9220 tools.wi 20 0 52076 2496 1468 D 0 0.0 0:07.86 lighttpd
291 9242 tools.wi 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
292 9244 tools.wi 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
293 9245 tools.wi 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
294 9246 tools.wi 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
295 9248 tools.wi 20 0 307m 6712 1648 S 0 0.1 0:00.00 php-cgi
296 9249 tools.wi 20 0 307m 6708 1648 S 0 0.1 0:00.00 php-cgi
297 9251 tools.wi 20 0 16972 1284 1112 D 0 0.0 0:16.56 gam_server
298 9265 sgeadmin 20 0 22548 2000 1548 S 0 0.0 0:00.00 sge_shepherd
299 9267 tools.sd 20 0 52152 2928 1684 D 0 0.0 0:04.60 lighttpd
300 9285 root 20 0 35860 1184 856 S 0 0.0 0:00.00 cron
301 9286 root 20 0 4396 608 508 S 0 0.0 0:00.00 sh
302 9287 root 20 0 36260 1460 1248 S 0 0.0 0:00.00 puppet-run
303 9295 tools.sd 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
304 9297 tools.sd 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
305 9298 tools.sd 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
306 9299 tools.sd 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
307 9301 tools.sd 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
308 9302 tools.sd 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
309 9303 tools.sd 20 0 4392 608 512 S 0 0.0 0:00.00 server.sh
310 9306 tools.sd 20 0 443m 24m 5364 S 0 0.3 6:36.83 python
311 9308 tools.sd 20 0 16972 1288 1112 S 0 0.0 0:00.00 gam_server
312 9330 sgeadmin 20 0 22548 2000 1548 S 0 0.0 0:00.00 sge_shepherd
313 9332 tools.to 20 0 104m 4132 2580 D 0 0.1 0:05.15 lighttpd
314 9342 root 20 0 35380 664 544 S 0 0.0 0:00.00 timeout
315 9343 root 20 0 175m 87m 4104 D 0 1.1 0:08.06 puppet
316 9356 tools.to 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
317 9358 tools.to 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
318 9359 tools.to 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
319 9360 tools.to 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
320 9362 tools.to 20 0 309m 7828 2476 S 0 0.1 0:00.00 php-cgi
321 9363 tools.to 20 0 308m 8136 2700 S 0 0.1 0:00.00 php-cgi
322 9365 tools.to 20 0 16972 1288 1112 D 0 0.0 0:52.42 gam_server
323 9368 sgeadmin 20 0 22548 1996 1552 S 0 0.0 0:00.00 sge_shepherd
324 9370 tools.ya 20 0 52072 2708 1564 D 0 0.0 0:04.04 lighttpd
325 9384 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.00 sshd
326 9385 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
327 9392 tools.ya 20 0 536m 54m 5756 S 0 0.7 4:54.68 main.py
328 9393 tools.ya 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
329 9395 tools.ya 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
330 9396 tools.ya 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
331 9397 tools.ya 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
332 9399 tools.ya 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
333 9400 tools.ya 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
334 9402 tools.ya 20 0 16972 1288 1112 S 0 0.0 0:00.00 gam_server
335 9420 sgeadmin 20 0 22548 2000 1548 S 0 0.0 0:00.00 sge_shepherd
336 9422 tools.tr 20 0 52076 2492 1468 D 0 0.0 0:06.10 lighttpd
337 9442 tools.tr 20 0 307m 13m 8868 S 0 0.2 0:00.03 php-cgi
338 9444 tools.tr 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
339 9445 tools.tr 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
340 9446 tools.tr 20 0 307m 13m 8868 S 0 0.2 0:00.07 php-cgi
341 9448 tools.tr 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
342 9449 tools.tr 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
343 9451 tools.tr 20 0 16972 1284 1108 D 0 0.0 0:36.84 gam_server
344 9465 sgeadmin 20 0 22548 2000 1548 S 0 0.0 0:00.00 sge_shepherd
345 9467 tools.ge 20 0 57352 8148 1752 D 0 0.1 10:25.52 lighttpd
346 9491 tools.ge 20 0 307m 13m 8880 S 0 0.2 0:00.24 php-cgi
347 9495 tools.ge 20 0 307m 13m 8880 S 0 0.2 0:00.40 php-cgi
348 9500 tools.ge 20 0 16972 1284 1112 D 0 0.0 0:21.31 gam_server
349 9523 sgeadmin 20 0 22548 1992 1548 S 0 0.0 0:00.00 sge_shepherd
350 9525 tools.he 20 0 52076 2500 1464 D 0 0.0 0:04.57 lighttpd
351 9545 tools.he 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
352 9547 tools.he 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
353 9548 tools.he 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
354 9549 tools.he 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
355 9551 tools.he 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
356 9552 tools.he 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
357 9554 tools.he 20 0 16972 1280 1108 D 0 0.0 0:10.31 gam_server
358 9568 sgeadmin 20 0 22548 2000 1552 S 0 0.0 0:00.00 sge_shepherd
359 9570 tools.bl 20 0 52076 2492 1468 D 0 0.0 0:04.55 lighttpd
360 9594 tools.bl 20 0 307m 13m 8868 S 0 0.2 0:00.03 php-cgi
361 9596 tools.bl 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
362 9597 tools.bl 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
363 9598 tools.bl 20 0 307m 13m 8868 S 0 0.2 0:00.08 php-cgi
364 9600 tools.bl 20 0 307m 7280 2088 S 0 0.1 0:00.23 php-cgi
365 9601 tools.bl 20 0 307m 7280 2088 S 0 0.1 0:00.21 php-cgi
366 9603 tools.bl 20 0 16972 1288 1112 D 0 0.0 0:44.65 gam_server
367 9620 sgeadmin 20 0 22548 2000 1548 S 0 0.0 0:00.00 sge_shepherd
368 9622 tools.be 20 0 52076 2728 1556 D 0 0.0 0:05.62 lighttpd
369 9642 tools.be 20 0 307m 13m 8868 S 0 0.2 0:00.04 php-cgi
370 9644 tools.be 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
371 9645 tools.be 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
372 9646 tools.be 20 0 307m 13m 8868 S 0 0.2 0:00.06 php-cgi
373 9648 tools.be 20 0 311m 7560 2324 S 0 0.1 0:00.00 php-cgi
374 9649 tools.be 20 0 311m 7560 2324 S 0 0.1 0:00.00 php-cgi
375 9651 tools.be 20 0 16972 1284 1112 D 0 0.0 0:30.61 gam_server
376 9687 sgeadmin 20 0 22548 1992 1548 S 0 0.0 0:00.00 sge_shepherd
377 9689 tools.wi 20 0 52336 3020 1628 D 0 0.0 0:04.59 lighttpd
378 9711 tools.wi 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
379 9713 tools.wi 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
380 9714 tools.wi 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
381 9715 tools.wi 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
382 9717 tools.wi 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
383 9718 tools.wi 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
384 9720 tools.wi 20 0 16972 1284 1108 D 0 0.0 0:34.42 gam_server
385 9749 sgeadmin 20 0 22548 1992 1548 S 0 0.0 0:00.00 sge_shepherd
386 9751 tools.ca 20 0 52076 2504 1468 D 0 0.0 0:04.72 lighttpd
387 9771 tools.ca 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
388 9773 tools.ca 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
389 9774 tools.ca 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
390 9775 tools.ca 20 0 307m 13m 8868 S 0 0.2 0:00.06 php-cgi
391 9777 tools.ca 20 0 307m 7036 1916 S 0 0.1 0:00.00 php-cgi
392 9778 tools.ca 20 0 307m 7044 1920 S 0 0.1 0:00.00 php-cgi
393 9780 tools.ca 20 0 16972 1284 1108 D 0 0.0 0:14.14 gam_server
394 10048 scfc 20 0 97832 4096 3076 D 0 0.1 0:00.01 sshd
395 10049 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
396 10126 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
397 10127 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
398 10196 scfc 20 0 97832 4104 3084 D 0 0.1 0:00.00 sshd
399 10197 sshd 20 0 62672 1340 628 S 0 0.0 0:00.00 sshd
400 10211 root 20 0 588m 39m 1372 S 0 0.5 0:00.00 salt-minion
401 10212 root 20 0 4396 612 508 S 0 0.0 0:00.00 sh
402 10404 scfc 20 0 97832 4092 3072 D 0 0.1 0:00.06 sshd
403 10405 sshd 20 0 62672 1336 628 S 0 0.0 0:00.02 sshd
404 10543 root 20 0 35860 1184 856 S 0 0.0 0:00.01 cron
405 10545 root 20 0 4396 608 508 S 0 0.0 0:00.00 sh
406 10548 root 20 0 4296 608 496 S 0 0.0 0:00.00 run-parts
407 10629 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.00 sshd
408 10630 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
409 10750 root 20 0 4396 608 508 S 0 0.0 0:00.00 exim4-base
410 10755 root 20 0 31396 608 516 S 0 0.0 0:00.00 xargs
411 10756 root 20 0 10336 608 508 D 0 0.0 0:00.00 start-stop-daem
412 10838 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
413 10839 sshd 20 0 62672 1336 628 S 0 0.0 0:00.01 sshd
414 11216 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
415 11217 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
416 11362 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.00 sshd
417 11363 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
418 11387 scfc 20 0 97832 4096 3076 D 0 0.1 0:00.00 sshd
419 11388 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
420 11532 sgeadmin 20 0 22548 1996 1548 S 0 0.0 0:00.00 sge_shepherd
421 11534 52315 20 0 52072 2716 1532 D 0 0.0 0:04.50 lighttpd
422 11577 52315 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
423 11580 52315 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
424 11581 52315 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
425 11585 52315 20 0 307m 13m 8868 S 0 0.2 0:00.03 php-cgi
426 11590 52315 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
427 11591 52315 20 0 307m 5316 324 S 0 0.1 0:00.00 php-cgi
428 11595 52315 20 0 16972 1284 1108 D 0 0.0 0:14.48 gam_server
429 11597 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.00 sshd
430 11598 sshd 20 0 62672 1336 628 S 0 0.0 0:00.01 sshd
431 11829 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
432 11830 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
433 12037 sgeadmin 20 0 22548 2000 1548 S 0 0.0 0:00.00 sge_shepherd
434 12039 tools.co 20 0 52200 2888 1664 D 0 0.0 0:04.85 lighttpd
435 12063 tools.co 20 0 307m 13m 8868 S 0 0.2 0:00.03 php-cgi
436 12065 tools.co 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
437 12066 tools.co 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
438 12067 tools.co 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
439 12069 tools.co 20 0 309m 8540 2820 S 0 0.1 0:00.01 php-cgi
440 12070 tools.co 20 0 308m 7552 2080 S 0 0.1 0:00.01 php-cgi
441 12072 tools.co 20 0 17072 1340 1128 D 0 0.0 4:21.65 gam_server
442 12086 sgeadmin 20 0 22548 2000 1548 S 0 0.0 0:00.00 sge_shepherd
443 12088 tools.ur 20 0 52072 2756 1600 D 0 0.0 0:05.84 lighttpd
444 12112 tools.ur 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
445 12114 tools.ur 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
446 12115 tools.ur 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
447 12116 tools.ur 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
448 12118 tools.ur 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
449 12119 tools.ur 20 0 307m 5320 324 S 0 0.1 0:00.00 php-cgi
450 12121 tools.ur 20 0 16972 1288 1112 D 0 0.0 0:16.45 gam_server
451 12133 sgeadmin 20 0 22548 1992 1548 S 0 0.0 0:00.00 sge_shepherd
452 12135 tools.tr 20 0 52172 2912 1584 D 0 0.0 0:09.03 lighttpd
453 12155 tools.tr 20 0 307m 13m 8868 S 0 0.2 0:00.02 php-cgi
454 12157 tools.tr 20 0 308m 7304 1924 S 0 0.1 0:00.05 php-cgi
455 12158 tools.tr 20 0 307m 7288 1916 S 0 0.1 0:00.02 php-cgi
456 12159 tools.tr 20 0 307m 13m 8880 S 0 0.2 0:00.02 php-cgi
457 12164 tools.tr 20 0 16972 1284 1108 D 0 0.0 0:38.97 gam_server
458 12167 sgeadmin 20 0 22548 2000 1552 S 0 0.0 0:00.00 sge_shepherd
459 12169 tools.ic 20 0 52360 2880 1560 D 0 0.0 0:04.65 lighttpd
460 12189 tools.ic 20 0 307m 13m 8868 S 0 0.2 0:00.01 php-cgi
461 12191 tools.ic 20 0 334m 9596 3812 S 0 0.1 0:00.40 php-cgi
462 12192 tools.ic 20 0 334m 9592 3812 S 0 0.1 0:00.44 php-cgi
463 12193 tools.ic 20 0 307m 13m 8868 S 0 0.2 0:00.03 php-cgi
464 12195 tools.ic 20 0 334m 9716 3876 S 0 0.1 0:01.68 php-cgi
465 12196 tools.ic 20 0 335m 9740 3884 S 0 0.1 0:02.19 php-cgi
466 12198 tools.ic 20 0 16972 1288 1112 D 0 0.0 1:09.28 gam_server
467 12221 root 20 0 35860 1184 856 S 0 0.0 0:00.00 cron
468 12223 root 20 0 4396 612 508 S 0 0.0 0:00.01 sh
469 12224 root 20 0 4296 604 496 S 0 0.0 0:00.01 run-parts
470 12298 root 20 0 4396 612 508 S 0 0.0 0:00.00 exim4-base
471 12303 root 20 0 31396 608 516 S 0 0.0 0:00.00 xargs
472 12304 root 20 0 10336 608 504 D 0 0.0 0:00.00 start-stop-daem
473 12428 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
474 12429 sshd 20 0 62672 1336 628 S 0 0.0 0:00.01 sshd
475 12573 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
476 12574 sshd 20 0 62672 1340 628 S 0 0.0 0:00.00 sshd
477 13005 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
478 13006 sshd 20 0 62672 1336 632 S 0 0.0 0:00.00 sshd
479 13165 scfc 20 0 97832 4096 3076 D 0 0.1 0:00.01 sshd
480 13166 sshd 20 0 62672 1340 628 S 0 0.0 0:00.00 sshd
481 13240 scfc 20 0 97832 4104 3084 D 0 0.1 0:00.01 sshd
482 13241 sshd 20 0 62672 1340 628 S 0 0.0 0:00.00 sshd
483 13584 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
484 13585 sshd 20 0 62672 1340 628 S 0 0.0 0:00.00 sshd
485 13738 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
486 13739 sshd 20 0 62672 1332 628 S 0 0.0 0:00.00 sshd
487 14170 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.00 sshd
488 14171 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
489 14194 scfc 20 0 97832 4092 3076 D 0 0.1 0:00.01 sshd
490 14195 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
491 14346 root 20 0 35860 1184 856 S 0 0.0 0:00.00 cron
492 14349 root 20 0 4396 608 504 S 0 0.0 0:00.01 sh
493 14351 root 20 0 4296 604 496 S 0 0.0 0:00.01 run-parts
494 14578 root 20 0 4396 608 508 S 0 0.0 0:00.00 exim4-base
495 14583 root 20 0 31396 604 516 S 0 0.0 0:00.00 xargs
496 14584 root 20 0 10336 608 504 D 0 0.0 0:00.00 start-stop-daem
497 14755 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
498 14756 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
499 14890 scfc 20 0 97832 4092 3076 D 0 0.1 0:00.00 sshd
500 14891 sshd 20 0 62672 1332 628 S 0 0.0 0:00.00 sshd
501 14933 scfc 20 0 97832 4092 3080 D 0 0.1 0:00.00 sshd
502 14934 sshd 20 0 62672 1336 632 S 0 0.0 0:00.00 sshd
503 15326 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.02 sshd
504 15327 sshd 20 0 62672 1340 628 S 0 0.0 0:00.00 sshd
505 15566 scfc 20 0 97832 4092 3072 D 0 0.1 0:00.02 sshd
506 15567 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
507 15922 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
508 15923 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
509 15999 root 20 0 35860 1184 856 S 0 0.0 0:00.00 cron
510 16001 root 20 0 4396 612 508 S 0 0.0 0:00.00 sh
511 16003 root 20 0 4296 608 496 S 0 0.0 0:00.00 run-parts
512 16062 scfc 20 0 97832 4104 3084 D 0 0.1 0:00.00 sshd
513 16063 sshd 20 0 62672 1340 628 S 0 0.0 0:00.00 sshd
514 16078 root 20 0 4396 612 508 S 0 0.0 0:00.00 exim4-base
515 16083 root 20 0 31396 608 516 S 0 0.0 0:00.00 xargs
516 16084 root 20 0 10336 612 504 D 0 0.0 0:00.00 start-stop-daem
517 16103 scfc 20 0 97832 4104 3084 D 0 0.1 0:00.00 sshd
518 16104 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
519 16494 scfc 20 0 97832 4096 3084 D 0 0.1 0:00.00 sshd
520 16495 sshd 20 0 62672 1332 628 S 0 0.0 0:00.00 sshd
521 17229 scfc 20 0 97832 4100 3084 D 0 0.1 0:00.01 sshd
522 17230 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
523 17691 scfc 20 0 97832 4104 3084 D 0 0.1 0:00.00 sshd
524 17692 sshd 20 0 62672 1340 628 S 0 0.0 0:00.00 sshd
525 17713 scfc 20 0 97832 4088 3072 D 0 0.0 0:00.01 sshd
526 17714 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
527 17815 scfc 20 0 97832 4092 3076 D 0 0.1 0:00.01 sshd
528 17816 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
529 17907 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
530 17908 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
531 18257 root 20 0 35860 1184 856 S 0 0.0 0:00.01 cron
532 18259 root 20 0 4396 612 508 S 0 0.0 0:00.00 sh
533 18262 root 20 0 4296 608 496 S 0 0.0 0:00.00 run-parts
534 18425 scfc 20 0 97832 4104 3084 D 0 0.1 0:00.01 sshd
535 18426 sshd 20 0 62672 1336 628 S 0 0.0 0:00.01 sshd
536 18429 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.00 sshd
537 18430 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
538 18621 root 20 0 4396 612 508 S 0 0.0 0:00.00 exim4-base
539 18626 root 20 0 31396 608 516 S 0 0.0 0:00.00 xargs
540 18627 root 20 0 10336 608 508 D 0 0.0 0:00.00 start-stop-daem
541 18811 scfc 20 0 97832 4088 3076 D 0 0.0 0:00.01 sshd
542 18812 sshd 20 0 62672 1336 632 S 0 0.0 0:00.00 sshd
543 19413 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
544 19414 sshd 20 0 62672 1332 628 S 0 0.0 0:00.00 sshd
545 19918 root 20 0 35860 1184 856 S 0 0.0 0:00.01 cron
546 19920 root 20 0 4396 612 508 S 0 0.0 0:00.00 sh
547 19922 root 20 0 4296 604 496 S 0 0.0 0:00.00 run-parts
548 19973 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
549 19974 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
550 19997 root 20 0 4396 608 508 S 0 0.0 0:00.00 exim4-base
551 20002 root 20 0 31396 612 516 S 0 0.0 0:00.00 xargs
552 20003 root 20 0 10336 612 504 D 0 0.0 0:00.00 start-stop-daem
553 20232 scfc 20 0 97832 4088 3072 D 0 0.0 0:00.00 sshd
554 20233 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
555 20707 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
556 20708 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
557 20747 scfc 20 0 97832 4096 3076 D 0 0.1 0:00.00 sshd
558 20748 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
559 21058 root 20 0 35860 1184 856 S 0 0.0 0:00.00 cron
560 21060 root 20 0 4396 608 508 S 0 0.0 0:00.00 sh
561 21062 root 20 0 4296 604 496 S 0 0.0 0:00.00 run-parts
562 21135 root 20 0 4396 608 508 S 0 0.0 0:00.00 exim4-base
563 21139 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.00 sshd
564 21140 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
565 21142 root 20 0 31396 608 516 S 0 0.0 0:00.00 xargs
566 21143 root 20 0 10336 608 504 D 0 0.0 0:00.00 start-stop-daem
567 21730 scfc 20 0 97832 4088 3076 D 0 0.0 0:00.00 sshd
568 21731 sshd 20 0 62672 1336 632 S 0 0.0 0:00.00 sshd
569 21871 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
570 21872 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
571 22280 root 20 0 35860 1184 856 S 0 0.0 0:00.00 cron
572 22283 root 20 0 4396 612 508 S 0 0.0 0:00.00 sh
573 22285 root 20 0 4296 608 496 S 0 0.0 0:00.00 run-parts
574 22465 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
575 22466 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
576 22667 root 20 0 4396 608 508 S 0 0.0 0:00.00 exim4-base
577 22672 root 20 0 31396 612 516 S 0 0.0 0:00.00 xargs
578 22673 root 20 0 10336 608 504 D 0 0.0 0:00.00 start-stop-daem
579 22986 scfc 20 0 97832 4088 3072 D 0 0.0 0:00.00 sshd
580 22987 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
581 23024 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.00 sshd
582 23025 sshd 20 0 62672 1336 628 S 0 0.0 0:00.01 sshd
583 23656 scfc 20 0 97832 4096 3076 D 0 0.1 0:00.00 sshd
584 23657 sshd 20 0 62672 1332 628 S 0 0.0 0:00.00 sshd
585 23947 root 20 0 35860 1184 856 S 0 0.0 0:00.00 cron
586 23950 root 20 0 4396 604 504 S 0 0.0 0:00.00 sh
587 23951 root 20 0 4296 608 496 S 0 0.0 0:00.00 run-parts
588 24025 root 20 0 4396 612 508 S 0 0.0 0:00.00 exim4-base
589 24030 root 20 0 31396 612 516 S 0 0.0 0:00.00 xargs
590 24031 root 20 0 10336 608 504 D 0 0.0 0:00.00 start-stop-daem
591 24376 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
592 24377 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
593 24617 scfc 20 0 97832 4100 3084 D 0 0.1 0:00.00 sshd
594 24618 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
595 24810 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
596 24811 sshd 20 0 62672 1340 628 S 0 0.0 0:00.00 sshd
597 24860 root 20 0 35860 1184 856 S 0 0.0 0:00.00 cron
598 24862 root 20 0 4396 608 508 S 0 0.0 0:00.00 sh
599 24864 root 20 0 4296 608 496 S 0 0.0 0:00.02 run-parts
600 24949 root 20 0 4396 612 508 S 0 0.0 0:00.00 exim4-base
601 24954 root 20 0 31396 608 516 S 0 0.0 0:00.00 xargs
602 24955 root 20 0 10336 608 504 D 0 0.0 0:00.00 start-stop-daem
603 25223 scfc 20 0 97832 4092 3076 D 0 0.1 0:00.00 sshd
604 25224 sshd 20 0 62672 1336 632 S 0 0.0 0:00.00 sshd
605 26305 root 20 0 35860 1184 856 S 0 0.0 0:00.00 cron
606 26307 root 20 0 4396 608 508 S 0 0.0 0:00.00 sh
607 26310 root 20 0 4296 604 496 S 0 0.0 0:00.00 run-parts
608 26413 root 0 -20 21004 8724 3392 S 0 0.1 5:14.99 atop
609 26501 scfc 20 0 97832 4092 3076 D 0 0.1 0:00.00 sshd
610 26502 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
611 27099 root 20 0 4396 612 508 S 0 0.0 0:00.00 exim4-base
612 27104 root 20 0 31396 612 516 S 0 0.0 0:00.00 xargs
613 27105 root 20 0 10336 608 508 D 0 0.0 0:00.00 start-stop-daem
614 27668 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
615 27669 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
616 27737 root 20 0 35860 1184 856 S 0 0.0 0:00.00 cron
617 27739 root 20 0 4396 608 504 S 0 0.0 0:00.00 sh
618 27742 root 20 0 4296 600 496 S 0 0.0 0:00.00 run-parts
619 27814 root 20 0 4396 612 508 S 0 0.0 0:00.00 exim4-base
620 27819 root 20 0 31396 612 516 S 0 0.0 0:00.00 xargs
621 27820 root 20 0 10336 608 504 D 0 0.0 0:00.00 start-stop-daem
622 28015 scfc 20 0 97832 4084 3072 D 0 0.0 0:00.01 sshd
623 28016 sshd 20 0 62672 1332 628 S 0 0.0 0:00.00 sshd
624 28100 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
625 28101 sshd 20 0 62672 1332 628 S 0 0.0 0:00.01 sshd
626 28662 scfc 20 0 97832 4088 3072 D 0 0.0 0:00.01 sshd
627 28663 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
628 28841 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.01 sshd
629 28842 sshd 20 0 62672 1340 628 S 0 0.0 0:00.00 sshd
630 28859 root 20 0 35860 1184 856 S 0 0.0 0:00.01 cron
631 28861 root 20 0 4396 608 504 S 0 0.0 0:00.00 sh
632 28864 root 20 0 4296 608 496 S 0 0.0 0:00.00 run-parts
633 28949 root 20 0 4396 612 508 S 0 0.0 0:00.00 exim4-base
634 28954 root 20 0 31396 608 516 S 0 0.0 0:00.00 xargs
635 28955 root 20 0 10336 608 504 D 0 0.0 0:00.00 start-stop-daem
636 29264 scfc 20 0 97832 4100 3080 D 0 0.1 0:00.00 sshd
637 29265 sshd 20 0 62672 1336 628 S 0 0.0 0:00.01 sshd
638 29471 scfc 20 0 97832 4096 3076 D 0 0.1 0:00.01 sshd
639 29472 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
640 29998 scfc 20 0 97832 4100 3084 D 0 0.1 0:00.00 sshd
641 29999 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
642 30349 root 20 0 35860 1184 856 S 0 0.0 0:00.00 cron
643 30351 root 20 0 4396 608 504 S 0 0.0 0:00.00 sh
644 30353 root 20 0 4296 604 496 S 0 0.0 0:00.00 run-parts
645 30496 scfc 20 0 97832 4092 3072 D 0 0.1 0:00.03 sshd
646 30497 sshd 20 0 62672 1340 628 S 0 0.0 0:00.00 sshd
647 30505 scfc 20 0 97832 4096 3080 D 0 0.1 0:00.01 sshd
648 30506 sshd 20 0 62672 1336 628 S 0 0.0 0:00.01 sshd
649 30734 scfc 20 0 97832 4092 3076 D 0 0.1 0:00.00 sshd
650 30735 sshd 20 0 62672 1332 628 S 0 0.0 0:00.00 sshd
651 30792 root 20 0 4396 612 508 S 0 0.0 0:00.00 exim4-base
652 30797 root 20 0 31396 608 516 S 0 0.0 0:00.00 xargs
653 30798 root 20 0 10336 608 508 D 0 0.0 0:00.00 start-stop-daem
654 31391 scfc 20 0 97832 4092 3076 D 0 0.1 0:00.00 sshd
655 31392 sshd 20 0 62672 1336 628 S 0 0.0 0:00.01 sshd
656 31418 tools.po 20 0 308m 8176 2668 S 0 0.1 0:00.84 php-cgi
657 31507 tools.po 20 0 308m 8176 2668 S 0 0.1 0:00.82 php-cgi
658 31537 root 20 0 35860 1184 856 S 0 0.0 0:00.01 cron
659 31540 root 20 0 4396 608 508 S 0 0.0 0:00.00 sh
660 31541 root 20 0 4296 608 496 S 0 0.0 0:00.00 run-parts
661 31631 root 20 0 4396 612 508 S 0 0.0 0:00.00 exim4-base
662 31636 root 20 0 31396 612 516 S 0 0.0 0:00.00 xargs
663 31637 root 20 0 10336 612 504 D 0 0.0 0:00.00 start-stop-daem
664 31724 scfc 20 0 97832 4100 3084 D 0 0.1 0:00.01 sshd
665 31725 sshd 20 0 62672 1336 628 S 0 0.0 0:00.00 sshd
666 32164 scfc 20 0 97832 4092 3072 D 0 0.1 0:00.01 sshd
667 32165 sshd 20 0 62672 1336 628 S 0 0.0 0:00.01 sshd
668 32458 scfc 20 0 97832 4100 3084 D 0 0.1 0:00.00 sshd
669 32459 sshd 20 0 62672 1332 628 S 0 0.0 0:00.00 sshd

• chasemp added a comment.Jan 21 2016, 12:21 AM

tools-webgrid-lighttpd-1201 did not respond to soft reboot via salt and I rebooted it the hard way

• chasemp closed subtask T122719: tools-webgrid-lighttpd-1201 webservices and ssh unaccessible as Resolved.Jan 21 2016, 12:23 AM
• chasemp added a comment.Jan 21 2016, 5:18 PM

have we had non-webgrid examples?

• chasemp closed subtask T124304: tools-webgrid-lighttpd-1412 is not accessible by ssh as Resolved.Jan 21 2016, 5:19 PM
Dereckson mentioned this in T124538: Maps-warper instance very slow application start up times, passenger timeout.Jan 23 2016, 7:49 PM
valhallasw added subtasks: T121998: Instances locking up randomly, T124875: tools-webgrid-lighttpd-1206 half-dead, webservices and ssh unaccessible.Jan 27 2016, 12:48 PM
scfc added a subtask: T125360: toolsbeta-puppetmaster3 instance has "BUG: soft lockup - CPU#0 stuck for 23s! [git:1366]".Feb 1 2016, 4:15 AM
scfc closed subtask T125360: toolsbeta-puppetmaster3 instance has "BUG: soft lockup - CPU#0 stuck for 23s! [git:1366]" as Resolved.Feb 1 2016, 5:56 PM
valhallasw added subtasks: T126141: tools-exec-1213 looks dead, T125770: tools-webgrid-lighttpd-1208.eqiad.wmflabs hangs.Feb 7 2016, 5:00 PM
valhallasw subscribed.Feb 7 2016, 5:02 PM
In T124133#1952488, @chasemp wrote:

have we had non-webgrid examples?

Yes, tools-exec-1213 is currently locked up (webgrid-lighttpd-1208 as well).

zhuyifei1999 subscribed.Feb 7 2016, 5:06 PM
valhallasw added a comment.Feb 7 2016, 5:23 PM

@chasemp: if you have time to look at this, could you try to ptrace sshd to see why sshd is hanging? Having a working (root) ssh login would make it much easier to figure out what is going on. I don't think I can run salt, as I don't have access to labcontrol1001 (maybe we can have the salt credentials for tool hosts on some tools host as well?)

• chasemp added a comment.Feb 9 2016, 12:00 AM

thanks @valhallasw. I did look at this, no game changer conclusions yet but some more details I think.

I will update in the morning as it is late, but a few thoughts ---

Where NFS dropped out:

http://graphite.wmflabs.org/render/?width=925&height=580&_salt=1454963569.272&target=tools.tools-webgrid-lighttpd-1208.nfsiostat.labstore.ops&from=-7d

http://graphite.wmflabs.org/render/?width=925&height=580&_salt=1454963894.684&target=tools.tools-webgrid-lighttpd-1208.nfsiostat.mounts.data_project.write.kilobytes&target=tools.tools-webgrid-lighttpd-1208.nfsiostat.mounts.*.read.kilobytes&from=-10d

Load as a consequence of NFS stalling and putting procs in D wait (including the many sshd login attempts due to stat() on user home I imagine)

http://graphite.wmflabs.org/render/?width=925&height=580&_salt=1454965360.339&from=00%3A00_20160202&until=23%3A59_20160208&target=removeAboveValue%28tools.tools-webgrid-lighttpd-1208.nfsiostat.mounts.data_project.read.kilobytes%2C500%29&target=tools.tools-webgrid-lighttpd-1208.loadavg.05

Not really that many retrans considering

http://graphite.wmflabs.org/render/?width=925&height=580&_salt=1454966001.304&from=00%3A00_20160202&until=23%3A59_20160208&target=sum%28tools.tools-webgrid-lighttpd-1208.nfsiostat.mounts.*.write.retrans%29&target=sum%28tools.tools-webgrid-lighttpd-1208.nfsiostat.mounts.*.read.retrans%29

I see no traffic from the node with NFS obviously still trying. I enabled rpcdebug and caught:

Feb  8 22:53:54 tools-webgrid-lighttpd-1208 kernel: [1566503.303254] NFS: nfs_lookup_revalidate(///) is valid
 Feb  8 22:53:54 tools-webgrid-lighttpd-1208 kernel: [1566503.303267] NFS: revalidating (0:16/2)
 Feb  8 22:53:54 tools-webgrid-lighttpd-1208 kernel: [1566503.304790] RPC:       new task initialized, procpid 25991
 Feb  8 22:53:54 tools-webgrid-lighttpd-1208 kernel: [1566503.304792] RPC:       allocated task ffff88009b8a7100
 Feb  8 22:53:54 tools-webgrid-lighttpd-1208 kernel: [1566503.304824] RPC:   417 __rpc_execute flags=0x80
 Feb  8 22:53:54 tools-webgrid-lighttpd-1208 kernel: [1566503.304827] RPC:   417 call_start nfs4 proc GETATTR (sync)
 Feb  8 22:53:54 tools-webgrid-lighttpd-1208 kernel: [1566503.304829] RPC:   417 call_reserve (status 0)
 Feb  8 22:53:54 tools-webgrid-lighttpd-1208 kernel: [1566503.304837] RPC:   417 failed to lock transport ffff880232046000
 Feb  8 22:53:54 tools-webgrid-lighttpd-1208 kernel: [1566503.304839] RPC:   417 sleep_on(queue "xprt_sending" time 4686518122)
 Feb  8 22:53:54 tools-webgrid-lighttpd-1208 kernel: [1566503.305069] RPC:   417 added to queue ffff880232046190 "xprt_sending"
 Feb  8 22:53:54 tools-webgrid-lighttpd-1208 kernel: [1566503.305074] RPC:   417 sync task going to sleep
 Feb  8 23:15:58 tools-webgrid-lighttpd-1208 kernel: [1567827.311178] NFS call  lookup project
 Feb  8 23:15:58 tools-webgrid-lighttpd-1208 kernel: [1567827.311203] RPC:       new task initialized, procpid 27449
 Feb  8 23:15:58 tools-webgrid-lighttpd-1208 kernel: [1567827.311205] RPC:       allocated task ffff88006e27e500
 Feb  8 23:15:58 tools-webgrid-lighttpd-1208 kernel: [1567827.311209] RPC:   418 __rpc_execute flags=0x80
 Feb  8 23:15:58 tools-webgrid-lighttpd-1208 kernel: [1567827.311212] RPC:   418 call_start nfs4 proc LOOKUP (sync)
 Feb  8 23:15:58 tools-webgrid-lighttpd-1208 kernel: [1567827.311214] RPC:   418 call_reserve (status 0)
 Feb  8 23:15:58 tools-webgrid-lighttpd-1208 kernel: [1567827.311221] RPC:   418 failed to lock transport ffff880232046000
 Feb  8 23:15:58 tools-webgrid-lighttpd-1208 kernel: [1567827.311223] RPC:   418 sleep_on(queue "xprt_sending" time 4686849123)
 Feb  8 23:15:58 tools-webgrid-lighttpd-1208 kernel: [1567827.313598] RPC:   418 added to queue ffff880232046190 "xprt_sending"
 Feb  8 23:15:58 tools-webgrid-lighttpd-1208 kernel: [1567827.313912] RPC:   418 sync task going to sleep
 Feb  8 23:23:16 tools-webgrid-lighttpd-1208 kernel: [1568265.762392] RPC:   417 got signal
 Feb  8 23:23:16 tools-webgrid-lighttpd-1208 kernel: [1568265.762400] RPC:   417 __rpc_wake_up_task (now 4686958736)
 Feb  8 23:23:16 tools-webgrid-lighttpd-1208 kernel: [1568265.762403] RPC:   417 removed from queue ffff880232046190 "xprt_sending"
 Feb  8 23:23:16 tools-webgrid-lighttpd-1208 kernel: [1568265.762477] RPC:       __rpc_wake_up_task done
 Feb  8 23:23:16 tools-webgrid-lighttpd-1208 kernel: [1568265.762479] RPC:   417 sync task resuming
 Feb  8 23:23:16 tools-webgrid-lighttpd-1208 kernel: [1568265.762482] RPC:   417 return -512, status -512
 Feb  8 23:23:16 tools-webgrid-lighttpd-1208 kernel: [1568265.762483] RPC:   417 release task
 Feb  8 23:23:16 tools-webgrid-lighttpd-1208 kernel: [1568265.762493] RPC:       rpc_release_client(ffff8802315fc000)
 Feb  8 23:23:16 tools-webgrid-lighttpd-1208 kernel: [1568265.762495] RPC:   417 freeing task
 Feb  8 23:23:16 tools-webgrid-lighttpd-1208 kernel: [1568265.762781] nfs_revalidate_inode: (0:16/2) getattr failed, error=-512
 Feb  8 23:25:11 tools-webgrid-lighttpd-1208 kernel: [1568380.242705] NFS call  lookup home
 Feb  8 23:25:11 tools-webgrid-lighttpd-1208 kernel: [1568380.242727] RPC:       new task initialized, procpid 27733
 Feb  8 23:25:11 tools-webgrid-lighttpd-1208 kernel: [1568380.242729] RPC:       allocated task ffff88017b42cb00
 Feb  8 23:25:11 tools-webgrid-lighttpd-1208 kernel: [1568380.242744] RPC:   419 __rpc_execute flags=0x80
 Feb  8 23:25:11 tools-webgrid-lighttpd-1208 kernel: [1568380.242753] RPC:   419 call_start nfs4 proc LOOKUP (sync)
 Feb  8 23:25:11 tools-webgrid-lighttpd-1208 kernel: [1568380.242756] RPC:   419 call_reserve (status 0)
 Feb  8 23:25:11 tools-webgrid-lighttpd-1208 kernel: [1568380.242769] RPC:   419 failed to lock transport ffff880232046000
 Feb  8 23:25:11 tools-webgrid-lighttpd-1208 kernel: [1568380.242772] RPC:   419 sleep_on(queue "xprt_sending" time 4686987356)
 Feb  8 23:25:11 tools-webgrid-lighttpd-1208 kernel: [1568380.243036] RPC:   419 added to queue ffff880232046190 "xprt_sending"
 Feb  8 23:25:11 tools-webgrid-lighttpd-1208 kernel: [1568380.243039] RPC:   419 sync task going to sleep

Really weird, I could use nc to test connectivity fine, and I was able to mount / on labstore1001 to /test/nfs without issue (I saw traffic at that point) and could unmount it. During none of this did existing mounts start working. remount has no effect.

I think this has to be an NFS client bug but I'm not sure what the deal is.

• chasemp closed subtask T126141: tools-exec-1213 looks dead as Resolved.Feb 9 2016, 12:04 AM
• chasemp closed subtask T125039: tools-worker-1002 locked up as Resolved.
valhallasw mentioned this in T126323: broken labs instances (ssh or perms), do we care?.Feb 9 2016, 12:48 PM
• chasemp added a comment.Feb 12 2016, 3:57 PM

re: 'tools-webgrid-lighttpd-1208.tools.eqiad.wmflabs

It all ties back to NFS but some details.

top - 20:06:16 up 18 days, 20 min,  0 users,  load average: 157.00, 156.93, 156.
Tasks: 596 total,   1 running, 592 sleeping,   0 stopped,   3 zombie
Cpu(s):  2.1%us,  2.2%sy,  0.0%ni, 86.7%id,  6.6%wa,  0.0%hi,  0.4%si,  2.0%st
Mem:   8178496k total,  7962704k used,   215792k free,   166652k buffers
Swap: 25034744k total,        0k used, 25034744k free,  6320816k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
  985 root      20   0  942m 4128 1648 S    2  0.1   7:14.62 nscd
 1031 nslcd     20   0  442m 6588 1896 S    2  0.1   6:37.32 nslcd
21813 root      20   0 12308 1540  840 R    1  0.0   0:00.09 top
  673 root      20   0     0    0    0 S    0  0.0   0:57.79 flush-253:0
 1097 sgeadmin  20   0 62372 4264 2072 D    0  0.1 463:27.08 sge_execd
    1 root      20   0 20060 2168 1236 S    0  0.0   0:11.45 init
    2 root      20   0     0    0    0 S    0  0.0   0:00.01 kthreadd

including all of the SSH attempts tried while the NFS mounts were in a bad state:

5474 scfc      20   0  100m 4100 3084 D    0  0.1   0:00.04 sshd
5475 sshd      20   0 62672 1336  628 S    0  0.0   0:00.02 sshd
5638 scfc      20   0  100m 4092 3076 D    0  0.1   0:00.02 sshd
5639 sshd      20   0 62672 1336  632 S    0  0.0   0:00.02 sshd
5911 scfc      20   0  100m 4104 3084 D    0  0.1   0:00.03 sshd
5912 sshd      20   0 62672 1340  628 S    0  0.0   0:00.01 sshd

It's interesting that it is not just that jobs are stuck in D state but so is sge_execd which seems to effectively neuter SGE's ability to handle the failure case (as much as it would anyway).

A note this is not a node we tried to pilot NFS traffic shaping on:

webgrid-lighttpd-1208.tools.eqiad.wmflabs:
    qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
     Sent 1051235695172 bytes 2492611125 pkt (dropped 0, overlimits 0 requeues 3)
     backlog 0b 0p requeues 3

Looking at dmesg, it's interesting I see the same segfault behavior from php-cgi. Cause or effect I don't know, no other events seemed to correlate directly.

[107389.515750] php-cgi[2617]: segfault at 1000701 ip 00002ab4ef654c0c sp 00007fff1d5022d0 error 4 in libc-2.15.so[2ab4ef61a000+1b4000]
[136196.772812] php-cgi[29841]: segfault at 1000701 ip 00002b5c6b35ac0c sp 00007fff81dd1ac0 error 4 in libc-2.15.so[2b5c6b320000+1b4000]
[337778.849270] php-cgi[18080]: segfault at 1000721 ip 00000000006ab997 sp 00007fff71140e90 error 4 in php5-cgi[400000+74b000]
[337795.940746] php-cgi[18076] general protection ip:6ac3c7 sp:7fff8a166860 error:0 in php5-cgi[400000+74b000]
[337797.310784] php-cgi[18077] general protection ip:6ac3c7 sp:7fff8a166860 error:0 in php5-cgi[400000+74b000]
[337800.504064] php-cgi[18081]: segfault at 1000721 ip 00000000006ab997 sp 00007fff71140e90 error 4 in php5-cgi[400000+74b000]

NFS related troubles:

[420265.952060] nfs: server labstore.svc.eqiad.wmnet not responding, still trying
[420265.953186] nfs: server labstore.svc.eqiad.wmnet OK
[420644.576157] nfs: server labstore.svc.eqiad.wmnet not responding, still trying
[420644.576172] nfs: server labstore.svc.eqiad.wmnet not responding, still trying
[420647.664994] nfs: server labstore.svc.eqiad.wmnet OK
[420647.665721] nfs: server labstore.svc.eqiad.wmnet OK

Which is followed by stuck procs from D state and indications of more NFS trouble:

[421080.492146] INFO: task python2.7:26388 blocked for more than 120 seconds.
 [421080.493035] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
 [421080.493960] python2.7       D ffffffff81806200     0 26388  26387 0x00000000
 [421080.493965]  ffff8801c3041be8 0000000000000086 ffff8801bc50d570 ffff8801bc50d69c
 [421080.493969]  ffff8801c3041fd8 ffff8801c3041fd8 ffff8801c3041fd8 0000000000012700
 [421080.493972]  ffff8802321296f0 ffff8800aec9c4d0 ffff88018e81f300 ffff8801bc50d610
 [421080.493996] Call Trace:
 [421080.494037]  [<ffffffff8165c3ff>] schedule+0x3f/0x60
 [421080.494040]  [<ffffffff8165d1e7>] __mutex_lock_slowpath+0xd7/0x150
 [421080.494043]  [<ffffffff8165cda2>] mutex_lock+0x22/0x40
 [421080.494060]  [<ffffffff81177522>] do_truncate+0x52/0xa0
 [421080.494075]  [<ffffffff8129ebac>] ? security_inode_permission+0x1c/0x30
 [421080.494082]  [<ffffffff811851aa>] ? inode_permission+0x4a/0x110
 [421080.494099]  [<ffffffff812d9326>] ? apparmor_path_truncate+0x56/0x60
 [421080.494101]  [<ffffffff8118557b>] handle_truncate+0x6b/0xa0
 [421080.494105]  [<ffffffff81188c14>] do_last+0x414/0x720
 [421080.494107]  [<ffffffff81189121>] path_openat+0xd1/0x3f0
 [421080.494117]  [<ffffffff8113c0b6>] ? free_pgtables+0xd6/0x120
 [421080.494119]  [<ffffffff81189562>] do_filp_open+0x42/0xa0
 [421080.494129]  [<ffffffff8131b7e1>] ? strncpy_from_user+0x31/0x40
 [421080.494132]  [<ffffffff8118486a>] ? do_getname+0x10a/0x180
 [421080.494164]  [<ffffffff8165e2de>] ? _raw_spin_lock+0xe/0x20
 [421080.494171]  [<ffffffff81196677>] ? alloc_fd+0xf7/0x150
 [421080.494174]  [<ffffffff81178b68>] do_sys_open+0xf8/0x240
 [421080.494177]  [<ffffffff81178cd0>] sys_open+0x20/0x30
 [421080.494183]  [<ffffffff81666982>] system_call_fastpath+0x16/0x1b
 [421080.494187] INFO: task python2.7:26396 blocked for more than 120 seconds.
 [421080.499271] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
 [421080.500224] python2.7       D 0000000000000001     0 26396  26395 0x00000000
 [421080.500228]  ffff880199eabbe8 0000000000000082 ffff8801bc50d570 ffff8801bc50d69c
 [421080.500231]  ffff880199eabfd8 ffff880199eabfd8 ffff880199eabfd8 0000000000012700
 [421080.500234]  ffff8800aec9c4d0 ffff8800aec9ade0 ffff88018e81f180 ffff8801bc50d610
 [421080.500237] Call Trace:
 [421080.500242]  [<ffffffff8165c3ff>] schedule+0x3f/0x60
 [421080.500248]  [<ffffffff8165d1e7>] __mutex_lock_slowpath+0xd7/0x150
 [421080.500253]  [<ffffffff8165cda2>] mutex_lock+0x22/0x40
 [421080.500257]  [<ffffffff81177522>] do_truncate+0x52/0xa0
 [421080.500262]  [<ffffffff8129ebac>] ? security_inode_permission+0x1c/0x30
 [421080.500267]  [<ffffffff811851aa>] ? inode_permission+0x4a/0x110
 [421080.500274]  [<ffffffff812d9326>] ? apparmor_path_truncate+0x56/0x60
 [421080.500278]  [<ffffffff8118557b>] handle_truncate+0x6b/0xa0
 [421080.500283]  [<ffffffff81188c14>] do_last+0x414/0x720
 [421080.500288]  [<ffffffff81189121>] path_openat+0xd1/0x3f0
 [421080.500301]  [<ffffffff8130dbad>] ? cpumask_any_but+0x2d/0x40
 [421080.500307]  [<ffffffff8113b8dc>] ? tlb_flush_mmu+0x6c/0x90
 [421080.500312]  [<ffffffff81189562>] do_filp_open+0x42/0xa0
 [421080.500317]  [<ffffffff8131b7e1>] ? strncpy_from_user+0x31/0x40
 [421080.500321]  [<ffffffff8118486a>] ? do_getname+0x10a/0x180
 [421080.500326]  [<ffffffff8165e2de>] ? _raw_spin_lock+0xe/0x20
 [421080.500331]  [<ffffffff81196677>] ? alloc_fd+0xf7/0x150
 [421080.500335]  [<ffffffff81178b68>] do_sys_open+0xf8/0x240
 [421080.500340]  [<ffffffff81178cd0>] sys_open+0x20/0x30
 [421080.500345]  [<ffffffff81666982>] system_call_fastpath+0x16/0x1b
 [421087.200129] nfs: server labstore.svc.eqiad.wmnet not responding, still trying
 [421087.200136] nfs: server labstore.svc.eqiad.wmnet not responding, still trying
 [421087.201520] nfs: server labstore.svc.eqiad.wmnet OK
 [421087.201686] nfs: server labstore.svc.eqiad.wmnet OK
 [421267.424174] nfs: server labstore.svc.eqiad.wmnet not responding, still trying
 [421320.500332] INFO: task python2.7:26282 blocked for more than 120 seconds.
 [421320.501292] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
 [421320.504416] python2.7       D 0000000000000002     0 26282  26281 0x00000000
 [421320.504423]  ffff880205513c78 0000000000000086 ffff88023fffbe00 ffffffffa0090a14
 [421320.504428]  ffff880205513fd8 ffff880205513fd8 ffff880205513fd8 0000000000012700
 [421320.504432]  ffff8800aec9ade0 ffff8800aec98000 ffffffff81c3b040 ffff8801bc50d610

A long history of trouble contacting NFS:

[1049098.208909] nfs: server labstore.svc.eqiad.wmnet not responding, still trying
[1049098.208918] nfs: server labstore.svc.eqiad.wmnet not responding, still trying
[1049098.208934] nfs: server labstore.svc.eqiad.wmnet not responding, still trying
[1049103.210050] nfs: server labstore.svc.eqiad.wmnet OK
[1049119.868554] nfs: server labstore.svc.eqiad.wmnet OK
[1049119.868559] nfs: server labstore.svc.eqiad.wmnet OK
[1049218.272867] nfs: server labstore.svc.eqiad.wmnet not responding, still trying
[1049218.272887] nfs: server labstore.svc.eqiad.wmnet not responding, still trying
[1049218.272893] nfs: server labstore.svc.eqiad.wmnet not responding, still trying
[1049218.272917] nfs: server labstore.svc.eqiad.wmnet not responding, still trying
[1049218.274346] nfs: server labstore.svc.eqiad.wmnet OK
[1049218.274740] nfs: server labstore.svc.eqiad.wmnet OK
[1049218.274891] nfs: server labstore.svc.eqiad.wmnet OK
[1049218.274900] nfs: server labstore.svc.eqiad.wmnet OK
[1120350.944703] nfs: server labstore.svc.eqiad.wmnet not responding, still trying
[1120350.944797] nfs: server labstore.svc.eqiad.wmnet not responding, still trying
[1120350.947103] nfs: server labstore.svc.eqiad.wmnet OK
[1120350.947419] nfs: server labstore.svc.eqiad.wmnet OK

Pidstat -l -d

P2604

pstree (what was grid engine running)

P2605

still high load

21:04:31 up 18 days,  1:19,  0 users,  load average: 159.00, 158.92, 158.40

This host could still resolve DNS and get out to the net (note: mounts for NFS are by hostname not IP atm)

root@labcontrol1001:~# salt 'tools-webgrid-lighttpd-1208.tools.eqiad.wmflabs' cmd.run 'ping -c 2 google.com'
tools-webgrid-lighttpd-1208.tools.eqiad.wmflabs:
    PING google.com (173.194.68.139) 56(84) bytes of data.
    64 bytes from 173.194.68.139: icmp_req=1 ttl=49 time=8.67 ms
    64 bytes from 173.194.68.139: icmp_req=2 ttl=49 time=11.2 ms

cat /proc/mounts (NFS portions)

labstore.svc.eqiad.wmnet:/project/tools/home /home nfs4 rw,noatime,vers=4,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.68.16.239,minorversion=0,lookupcache=none,local_lock=none,addr=10.64.37.10 0 0
labstore1003.eqiad.wmnet:/dumps /public/dumps nfs4 ro,noatime,vers=4,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.68.16.239,minorversion=0,lookupcache=none,local_lock=none,addr=10.64.4.10 0 0
labstore.svc.eqiad.wmnet:/project/tools/project /data/project nfs4 rw,noatime,vers=4,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.68.16.239,minorversion=0,lookupcache=none,local_lock=none,addr=10.64.37.10 0 0
labstore.svc.eqiad.wmnet:/project/tools/project/.system/gridengine /var/lib/gridengine nfs4 rw,noatime,vers=4,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.68.16.239,minorversion=0,lookupcache=none,local_lock=none,addr=10.64.37.10 0 0
labstore.svc.eqiad.wmnet:/scratch /data/scratch nfs4 rw,noatime,vers=4,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.68.16.239,minorversion=0,lookupcache=none,local_lock=none,addr=10.64.37.10 0 0
rpc_pipefs /run/rpc_pipefs rpc_pipefs rw,relatime 0 0

Showing NFS retrans:

root@labcontrol1001:~# salt 'tools-webgrid-lighttpd-1208.tools.eqiad.wmflabs' cmd.run 'nfsstat -rc'
tools-webgrid-lighttpd-1208.tools.eqiad.wmflabs:
    Client rpc stats:
    calls      retrans    authrefrsh
    1834672431   529        1834782370

turning on NFS and RPC debugging info

salt 'tools-webgrid-lighttpd-1208.tools.eqiad.wmflabs' cmd.run 'which rpcdebug rpcdebug -m nfs all'

 salt 'tools-webgrid-lighttpd-1208.tools.eqiad.wmflabs' cmd.run 'rpcdebug -m nfs all'

nfs vfs dircache lookupcache pagecache proc xdr file root callback client mount

the last NFS message regarding connectivity to labstore is from

/var/log/kern.log:Feb  3 18:58:01 tools-webgrid-lighttpd-1208 kernel: [1120350.947419] nfs: server labstore.svc.eqiad.wmnet OK

yet I see no traffic hitting labstore and faidon was able to take a look and said no traffic was leaving the VM related to NFS at all. Total lockup yet...

/var/log/kern.log:Feb  8 22:06:35 tools-webgrid-lighttpd-1208 kernel: [1563665.045559]   400 0080    -11 ffff8802315fc000   (null)        0 ffffffffa00a7580 nfsv4 GETATTR a:call_reserveresult q:xprt_sending
 /var/log/kern.log:Feb  8 22:06:35 tools-webgrid-lighttpd-1208 kernel: [1563665.045571] 65480 0081    -11 ffff88022b6e1000   (null)        0 ffffffffa0154ea0 nfsv4 CLOSE a:call_reserveresult q:xprt_sending
 /var/log/kern.log:Feb  8 22:06:35 tools-webgrid-lighttpd-1208 kernel: [1563665.045585] 65492 0081      0 ffff88022b6e1000   (null)        0 ffffffffa0154c30 nfsv4 OPEN a:rpc_prepare_task q:Seqid_waitqueue
 /var/log/kern.log:Feb  8 22:06:35 tools-webgrid-lighttpd-1208 kernel: [1563665.045595] 65499 0081      0 ffff88022b6e1000   (null)        0 ffffffffa0154c30 nfsv4 OPEN a:rpc_prepare_task q:Seqid_waitqueue
 /var/log/kern.log:Feb  8 22:06:35 tools-webgrid-lighttpd-1208 kernel: [1563665.045601] 65514 0081      0 ffff88022b6e1000   (null)        0 ffffffffa0154c30 nfsv4 OPEN a:rpc_prepare_task q:Seqid_waitqueue
 /var/log/kern.log:Feb  8 22:06:35 tools-webgrid-lighttpd-1208 kernel: [1563665.045610] 65529 0081      0 ffff88022b6e1000   (null)        0 ffffffffa0154c30 nfsv4 OPEN a:rpc_prepare_task q:Seqid_waitqueue
 /var/log/kern.log:Feb  8 22:06:35 tools-webgrid-lighttpd-1208 kernel: [1563665.045616] 65535 0081      0 ffff88022b6e1000   (null)        0 ffffffffa0154c30 nfsv4 OPEN a:rpc_prepare_task q:Seqid_waitqueue
 /var/log/kern.log:Feb  8 22:06:35 tools-webgrid-lighttpd-1208 kernel: [1563665.045622]     1 0081      0 ffff88022b6e1000   (null)        0 ffffffffa0154c30 nfsv4 OPEN a:rpc_prepare_task q:Seqid_waitqueue
 /var/log/kern.log:Feb  8 22:06:35 tools-webgrid-lighttpd-1208 kernel: [1563665.045630]    15 0081      0 ffff88022b6e1000   (null)        0 ffffffffa0154c30 nfsv4 OPEN a:rpc_prepare_task q:Seqid_waitqueue
 /var/log/kern.log:Feb  8 22:06:35 tools-webgrid-lighttpd-1208 kernel: [1563665.045636]    16 0081      0 ffff88022b6e1000   (null)        0 ffffffffa0154c30 nfsv4 OPEN a:rpc_prepare_task q:Seqid_waitqueue

NFS is actively trying to perform operations but everything is stuck in a waitqueue, and later I see failures to gain a lock to send. i.e. NFS is caught in a blocking state that makes no sense and sends nothing but continually thinks it is trying.

I tried some NFS ops to see if it would generate traffic or have a different outcome:

root@labcontrol1001:/tmp# salt 'tools-webgrid-lighttpd-1208.tools.eqiad.wmflabs' cmd.run 'timeout 10s touch /home/rush/test; echo $?'
tools-webgrid-lighttpd-1208.tools.eqiad.wmflabs:

124

no dice.

Looking at diff from a few minutes apart of mountstats:

--- /home/rush/tools-webgrid-lighttpd-1208.mountstats1  2016-02-08 21:52:40.083270784 +0000
+++ /home/rush/tools-webgrid-lighttpd-1208.mountstats2  2016-02-08 21:58:48.944162616 +0000
@@ -14,7 +14,7 @@
     device /dev/mapper/vd-separate--tmp mounted on /tmp with fstype ext4
     device labstore.svc.eqiad.wmnet:/project/tools/home mounted on /home with fstype nfs4 statvers=1.0
        opts:   rw,vers=4,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.68.16.239,minorversion=0,lookupcache=none,local_lock=none
-       age:    1562797
+       age:    1563165
        caps:   caps=0xffff,wtmult=512,dtsize=32768,bsize=0,namlen=255
        nfsv4:  bm0=0xfdffbfff,bm1=0xf9be3e,acl=0x3,pnfs=not configured
        sec:    flavor=1,pseudoflavor=1
@@ -77,7 +77,7 @@

     device labstore1003.eqiad.wmnet:/dumps mounted on /public/dumps with fstype nfs4 statvers=1.0
        opts:   rw,vers=4,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.68.16.239,minorversion=0,lookupcache=none,local_lock=none
-       age:    1562796
+       age:    1563165
        caps:   caps=0xffff,wtmult=512,dtsize=32768,bsize=0,namlen=255
        nfsv4:  bm0=0xfdffbfff,bm1=0xf9be3e,acl=0x3,pnfs=not configured
        sec:    flavor=1,pseudoflavor=1
@@ -140,7 +140,7 @@

     device labstore.svc.eqiad.wmnet:/project/tools/project mounted on /data/project with fstype nfs4 statvers=1.0
        opts:   rw,vers=4,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.68.16.239,minorversion=0,lookupcache=none,local_lock=none
-       age:    1562797
+       age:    1563165
        caps:   caps=0xffff,wtmult=512,dtsize=32768,bsize=0,namlen=255
        nfsv4:  bm0=0xfdffbfff,bm1=0xf9be3e,acl=0x3,pnfs=not configured
        sec:    flavor=1,pseudoflavor=1
@@ -203,7 +203,7 @@

     device labstore.svc.eqiad.wmnet:/project/tools/project/.system/gridengine mounted on /var/lib/gridengine with fstype nfs4 statvers=1.0
        opts:   rw,vers=4,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.68.16.239,minorversion=0,lookupcache=none,local_lock=none
-       age:    1562797
+       age:    1563165
        caps:   caps=0xffff,wtmult=512,dtsize=32768,bsize=0,namlen=255
        nfsv4:  bm0=0xfdffbfff,bm1=0xf9be3e,acl=0x3,pnfs=not configured
        sec:    flavor=1,pseudoflavor=1
@@ -266,11 +266,11 @@

     device labstore.svc.eqiad.wmnet:/scratch mounted on /data/scratch with fstype nfs4 statvers=1.0
        opts:   rw,vers=4,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.68.16.239,minorversion=0,lookupcache=none,local_lock=none
-       age:    1562795
+       age:    1563164
        caps:   caps=0xffff,wtmult=512,dtsize=32768,bsize=0,namlen=255
        nfsv4:  bm0=0xfdffbfff,bm1=0xf9be3e,acl=0x3,pnfs=not configured
        sec:    flavor=1,pseudoflavor=1
-       events: 789 837 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
+       events: 790 838 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
        bytes:  0 0 0 0 0 0 0 0
        RPC iostats version: 1.0  p/v: 100003/4 (nfs)
        xprt:   tcp 787 1 202 0 0 1834786765 1834781918 461 3201365979506 31345

this matches up with the logs and lack of traffic, nfs is there constantly churning on failed to send and requeing of ops.

P2607

So I tried a new mount to see if it would succeed:

salt 'tools-webgrid-lighttpd-1208.tools.eqiad.wmflabs' cmd.run 'mkdir -p /nfs/test'
root@labcontrol1001:/tmp# salt 'tools-webgrid-lighttpd-1208.tools.eqiad.wmflabs' cmd.run 'mount -t nfs4 -o rw,soft,proto=tcp,port=2049 10.64.37.10:/ /nfs/test; echo $?'
tools-webgrid-lighttpd-1208.tools.eqiad.wmflabs:

0

root@labcontrol1001:/tmp# salt 'tools-webgrid-lighttpd-1208.tools.eqiad.wmflabs' cmd.run 'ls /nfs/test' tools-webgrid-lighttpd-1208.tools.eqiad.wmflabs:

others
project
scratch

works fine. I can see traffic to labstore. So the issue isn't just with NFS, it's with the old NFS mounts being a stale state and a new NFS mount is fine.

log_post_sysrq_t_trigger

logger t_to_sysrq_trigger && echo t > /proc/sysrq-trigger

P2608

logger w_to_sysrq_trigger && echo w > /proc/sysrq-trigger

P2609

not included is a lot of trouble with salt truncating logs etc. hard to get get information remotely, we really need to find a reliable console access solution for VM's to debug more effectively.

The overview here seems to be the same behavior. I haven't gotten a look at a trusty host that is exhibiting this, so far hosts I have gotten a chance to investigate are all precise. It seems like there is an NFS connectivity issue, whether related to the server being unresponsive, the path being flaky, or the client having issues of its own manifesting it is unclear. I did some testing on using hostnames in fstab instead of IP's and was only able to see resolution occur on new mounts, it seems cached for the duration of the mount afaict. I'm suspicious of this though as we have been having intermitten DNS troubles related most likely to someone inadvertently ddos'ing our resolver. Whatever the cause of the alarms for failure to communicate with labstore the local NFS instance is getting itself into a bad state which is not affected by the recovery of communication. A new mount to the same endpoint works fine all the while existing mounts are frozen. The other issue is that SGE is stuck and cannot manage the jobs on the host, and the variety of jobs in D state skyrocket load and eventually the VM has general issues. A mount -f -l has no effect on frozen mountpoints while this is happening. I did not try to unload and reload the nfs module itself, which I wish I would have. If we are using hard mount as a data integrity insurance measure we are mostly poking ourselves in the eye here and restricting the normal course of business for SGE. We could have reboot on panic behavior but that seems very undesirable, but it's bad news this node was stuck for several days before anyone noticed. Our NFS setup is overloaded. It is undersized for our deployment and we now recognize that and are in the process of pursuing more hardware. I am taking other measures to try to even out our overload in the short term and we will have to continue to be mindful of this issue.

• chasemp closed subtask T125770: tools-webgrid-lighttpd-1208.eqiad.wmflabs hangs as Resolved.Feb 12 2016, 3:58 PM
Andrew added a comment.Feb 22 2016, 1:49 PM

This has I believe just happened to tools-webgrid-lighttpd-1206.eqiad.wmflabs

valhallasw created subtask T127936: tools-webgrid-lighttpd-1206.eqiad.wmflabs hanging.Feb 24 2016, 9:52 AM
• chasemp closed subtask T127936: tools-webgrid-lighttpd-1206.eqiad.wmflabs hanging as Resolved.Feb 24 2016, 1:58 PM
valhallasw added a subtask: T136481: tools-exec-1207 hanging.May 28 2016, 2:46 PM
valhallasw closed subtask T136481: tools-exec-1207 hanging as Resolved.May 28 2016, 4:31 PM
valhallasw mentioned this in T137683: several stucks of tools-bastion-03.Jun 13 2016, 8:40 AM
hashar mentioned this in T129891: phlogiston-2 hangs every week.Jul 4 2016, 12:58 PM
hashar mentioned this in T138281: Some Jenkins jobs tend to be stuck and never times out.
Paladox subscribed.Jul 4 2016, 12:59 PM
hashar subscribed.Jul 4 2016, 1:02 PM

We had that issue on some Jessie instances on CI since at least mid june. See T138281 for some debugging session. At least dmesg on the labvirt looked clear.

We have a whole console dump at P3278 and it looks like a kernel softlock while attempting to access /dev/vda . Maybe it is a kernel bug ? :( We had:

Linux version 3.16.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 4.8.4 (Debian 4.8.4-1) ) #1 SMP Debian 3.16.7-ckt25-2
TerraCodes subscribed.Jul 4 2016, 1:08 PM
hashar mentioned this in T140256: etcd hosts hanging with kernel hang.Jul 18 2016, 2:44 PM
hashar added a comment.Jul 18 2016, 3:01 PM

I havent heard of such stall Jenkins jobs last week. Would have to grep in the build history to be sure though by looking at job that took more than max timeout / got aborted.

yuvipanda closed subtask T121998: Instances locking up randomly as Resolved.Jul 29 2016, 11:57 PM
yuvipanda mentioned this in T141673: Track labs instances hanging .
hashar added a subtask: T138281: Some Jenkins jobs tend to be stuck and never times out.Jul 30 2016, 10:34 PM
hashar closed subtask T138281: Some Jenkins jobs tend to be stuck and never times out as Resolved.Jul 30 2016, 10:36 PM
valhallasw mentioned this in T141720: tools-docker-builder-03 hanging .Jul 31 2016, 2:13 PM
• chasemp renamed this task from Labs instances sometimes freeze to NFS overload is causing instances to freeze.Aug 3 2016, 3:00 AM
• chasemp added a comment.Aug 3 2016, 3:07 AM

This task encompassed a particular failure pattern we were seeing really often in late 2015 and early 2016 all involving NFS and client failure. This affected primarily precise instances with NFS mounts (and very occasionally may still). At first this was treated as a general issue but all of this historical investigation and details are specific to the mode with which we were failing and the avenues of mitigation then. I am closing this as it keeps getting looped into separate issues. The symptom of an instance "freezing" or becoming unavailable is very general and it is not meaningful to confuse things with this tasks history. There are some current issues that have a similar outcome being tracked in T141673, T141017, and T140256.

• AlexMonk-WMF closed this task as Resolved.Aug 14 2016, 11:57 PM
• AlexMonk-WMF subscribed.
In T124133#2517843, @chasemp wrote:

This task encompassed a particular failure pattern we were seeing really often in late 2015 and early 2016 all involving NFS and client failure. This affected primarily precise instances with NFS mounts (and very occasionally may still). At first this was treated as a general issue but all of this historical investigation and details are specific to the mode with which we were failing and the avenues of mitigation then. I am closing this as it keeps getting looped into separate issues. The symptom of an instance "freezing" or becoming unavailable is very general and it is not meaningful to confuse things with this tasks history. There are some current issues that have a similar outcome being tracked in T141673, T141017, and T140256.

Actually closing :)

• Bstorm mentioned this in T217086: Investigate why the new Son of Grid Engine grid landed in a worse state when NFS was filled than the old Sun Grid Engine grid did.Feb 25 2019, 11:08 PM
Liuxinyu970226 moved this task from Tag to Transition completed / Archived on the Tracking-Neverending board.Mar 16 2019, 1:34 PM
• Bstorm mentioned this in T217280: LDAP server running out of memory frequently and disrupting Cloud VPS clients.Mar 22 2019, 2:57 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits