tools-exec-1218 appears to be having problems, stuck job 11474, can't even delete it. https://tools.wmflabs.org/nagf/?project=tools#h_tools-exec-1218 shows way more wait I/O than other instances
Description
Description
Event Timeline
Comment Actions
Ok, so gridengine says that job is running there, but is totally false - that job is running nowhere...
Comment Actions
The host itself might be ok - it has other jobs running. qdel -f the jobid and re-submit it?
Comment Actions
On the host, the job was stuck on device I/O:
scfc@tools-exec-1218:~$ ps auxfwww USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND […] sgeadmin 4212 0.0 0.0 22556 2044 ? S Sep01 0:01 \_ /usr/lib/gridengine/sge_shepherd -bg 51454 4214 0.3 1.1 404756 94044 ? Ds Sep01 5:17 | \_ /usr/bin/php5 DatabaseReportBot.php InvalidNavbarLinks 51454 5649 0.0 0.0 22892 3556 ? S 15:23 0:00 | \_ /usr/bin/perl /usr/local/bin/jobkill 4214 […] scfc@tools-exec-1218:~$
And lsof said (note the first two warnings):
scfc@tools-exec-1218:~$ sudo lsof -p 4214
lsof: WARNING: can't stat() nfs4 file system /home (deleted)
Output information may be incomplete.
lsof: WARNING: can't stat() nfs4 file system /data/project (deleted)
Output information may be incomplete.
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
php5 4214 tools.bambots cwd DIR 0,20 4096 111479292 /data/project/bambots/Bambots/src/cli
php5 4214 tools.bambots rtd DIR 253,1 4096 2 /
php5 4214 tools.bambots txt REG 253,1 8100184 416203 /usr/bin/php5
php5 4214 tools.bambots mem REG 253,1 149280 921745 /lib/x86_64-linux-gnu/ld-2.15.so
php5 4214 tools.bambots mem REG 253,1 43288 921736 /lib/x86_64-linux-gnu/libcrypt-2.15.so
php5 4214 tools.bambots mem REG 253,1 92720 917941 /lib/x86_64-linux-gnu/libz.so.1.2.3.4
php5 4214 tools.bambots mem REG 253,1 105288 921749 /lib/x86_64-linux-gnu/libresolv-2.15.so
php5 4214 tools.bambots mem REG 253,1 146584 400393 /usr/lib/x86_64-linux-gnu/libedit.so.2.11
php5 4214 tools.bambots mem REG 253,1 1518928 400387 /usr/lib/x86_64-linux-gnu/libdb-5.1.so
php5 4214 tools.bambots mem REG 253,1 66784 917879 /lib/x86_64-linux-gnu/libbz2.so.1.0.4
php5 4214 tools.bambots mem REG 253,1 247896 919986 /lib/x86_64-linux-gnu/libpcre.so.3.12.1
php5 4214 tools.bambots mem REG 253,1 1030512 921741 /lib/x86_64-linux-gnu/libm-2.15.so
php5 4214 tools.bambots mem REG 253,1 14768 921756 /lib/x86_64-linux-gnu/libdl-2.15.so
php5 4214 tools.bambots mem REG 253,1 383072 932864 /lib/x86_64-linux-gnu/libssl.so.1.0.0
php5 4214 tools.bambots mem REG 253,1 1930648 932865 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
php5 4214 tools.bambots mem REG 253,1 1421464 400472 /usr/lib/x86_64-linux-gnu/libxml2.so.2.7.8
php5 4214 tools.bambots mem REG 253,1 1807032 921752 /lib/x86_64-linux-gnu/libc-2.15.so
php5 4214 tools.bambots mem REG 253,1 47624 917878 /lib/x86_64-linux-gnu/libbsd.so.0.3.0
php5 4214 tools.bambots mem REG 253,1 159200 917933 /lib/x86_64-linux-gnu/libtinfo.so.5.9
php5 4214 tools.bambots mem REG 253,1 135366 921743 /lib/x86_64-linux-gnu/libpthread-2.15.so
php5 4214 tools.bambots mem REG 253,1 617365 405784 /usr/lib/php5/20090626/xdebug.so
php5 4214 tools.bambots mem REG 253,1 71968 416199 /usr/lib/php5/20090626/curl.so
php5 4214 tools.bambots mem REG 253,1 381512 399769 /usr/lib/x86_64-linux-gnu/libcurl.so.4.2.0
php5 4214 tools.bambots mem REG 253,1 207312 400424 /usr/lib/x86_64-linux-gnu/libidn.so.11.6.6
php5 4214 tools.bambots mem REG 253,1 55528 395759 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.1
php5 4214 tools.bambots mem REG 253,1 314936 395758 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.1
php5 4214 tools.bambots mem REG 253,1 31752 921746 /lib/x86_64-linux-gnu/librt-2.15.so
php5 4214 tools.bambots mem REG 253,1 252720 399813 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
php5 4214 tools.bambots mem REG 253,1 105656 400459 /usr/lib/x86_64-linux-gnu/librtmp.so.0
php5 4214 tools.bambots mem REG 253,1 109288 400460 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
php5 4214 tools.bambots mem REG 253,1 252312 400416 /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
php5 4214 tools.bambots mem REG 253,1 766856 400782 /usr/lib/x86_64-linux-gnu/libgnutls.so.26.21.8
php5 4214 tools.bambots mem REG 253,1 520152 917672 /lib/x86_64-linux-gnu/libgcrypt.so.11.7.0
php5 4214 tools.bambots mem REG 253,1 844648 399803 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
php5 4214 tools.bambots mem REG 253,1 158168 399799 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
php5 4214 tools.bambots mem REG 253,1 14696 917635 /lib/x86_64-linux-gnu/libcom_err.so.2.1
php5 4214 tools.bambots mem REG 253,1 31200 399815 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
php5 4214 tools.bambots mem REG 253,1 27624 400422 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
php5 4214 tools.bambots mem REG 253,1 547488 400428 /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
php5 4214 tools.bambots mem REG 253,1 655152 400375 /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
php5 4214 tools.bambots mem REG 253,1 209184 400420 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
php5 4214 tools.bambots mem REG 253,1 86024 400457 /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
php5 4214 tools.bambots mem REG 253,1 67864 403419 /usr/lib/x86_64-linux-gnu/libtasn1.so.3.1.12
php5 4214 tools.bambots mem REG 253,1 72752 400437 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
php5 4214 tools.bambots mem REG 253,1 14352 917894 /lib/x86_64-linux-gnu/libgpg-error.so.0.8.0
php5 4214 tools.bambots mem REG 253,1 14360 917897 /lib/x86_64-linux-gnu/libkeyutils.so.1.4
php5 4214 tools.bambots mem REG 253,1 166096 400468 /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
php5 4214 tools.bambots mem REG 253,1 61264 400421 /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
php5 4214 tools.bambots mem REG 253,1 300704 400423 /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
php5 4214 tools.bambots mem REG 253,1 664504 399834 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
php5 4214 tools.bambots mem REG 253,1 116792 416200 /usr/lib/php5/20090626/gd.so
php5 4214 tools.bambots mem REG 253,1 270400 400402 /usr/lib/x86_64-linux-gnu/libgd.so.2.0.0
php5 4214 tools.bambots mem REG 253,1 294416 399788 /usr/lib/libt1.so.5.1.2
php5 4214 tools.bambots mem REG 253,1 68288 400369 /usr/lib/x86_64-linux-gnu/libXpm.so.4.11.0
php5 4214 tools.bambots mem REG 253,1 260880 400426 /usr/lib/x86_64-linux-gnu/libjpeg.so.8.0.2
php5 4214 tools.bambots mem REG 253,1 220896 400397 /usr/lib/x86_64-linux-gnu/libfontconfig.so.1.4.4
php5 4214 tools.bambots mem REG 253,1 637160 410140 /usr/lib/x86_64-linux-gnu/libfreetype.so.6.8.0
php5 4214 tools.bambots mem REG 253,1 162776 917921 /lib/x86_64-linux-gnu/libpng12.so.0.46.0
php5 4214 tools.bambots mem REG 253,1 1265056 400918 /usr/lib/x86_64-linux-gnu/libX11.so.6.3.0
php5 4214 tools.bambots mem REG 253,1 170024 927588 /lib/x86_64-linux-gnu/libexpat.so.1.5.2
php5 4214 tools.bambots mem REG 253,1 121232 400471 /usr/lib/x86_64-linux-gnu/libxcb.so.1.1.0
php5 4214 tools.bambots mem REG 253,1 10328 400358 /usr/lib/x86_64-linux-gnu/libXau.so.6.0.0
php5 4214 tools.bambots mem REG 253,1 22504 400362 /usr/lib/x86_64-linux-gnu/libXdmcp.so.6.0.0
php5 4214 tools.bambots mem REG 253,1 420480 404051 /usr/lib/php5/20090626/imagick.so
php5 4214 tools.bambots mem REG 253,1 1134104 403989 /usr/lib/libMagickWand.so.4.0.1
php5 4214 tools.bambots mem REG 253,1 2499664 403987 /usr/lib/libMagickCore.so.4.0.1
php5 4214 tools.bambots mem REG 253,1 55888 403917 /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0
php5 4214 tools.bambots mem REG 253,1 215520 403919 /usr/lib/x86_64-linux-gnu/liblcms.so.1.0.19
php5 4214 tools.bambots mem REG 253,1 80064 403985 /usr/lib/liblqr-1.so.0.3.1
php5 4214 tools.bambots mem REG 253,1 69504 399744 /usr/lib/x86_64-linux-gnu/libXext.so.6.4.0
php5 4214 tools.bambots mem REG 253,1 39184 403921 /usr/lib/x86_64-linux-gnu/libltdl.so.7.3.0
php5 4214 tools.bambots mem REG 253,1 1000472 917893 /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.4
php5 4214 tools.bambots mem REG 253,1 176376 403882 /usr/lib/php5/20090626/intl.so
php5 4214 tools.bambots mem REG 253,1 1862400 405732 /usr/lib/libicui18n.so.48.1.1
php5 4214 tools.bambots mem REG 253,1 1465096 405728 /usr/lib/libicuuc.so.48.1.1
php5 4214 tools.bambots mem REG 253,1 88384 917891 /lib/x86_64-linux-gnu/libgcc_s.so.1
php5 4214 tools.bambots mem REG 253,1 962656 400463 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
php5 4214 tools.bambots mem REG 253,1 18282384 405730 /usr/lib/libicudata.so.48.1.1
php5 4214 tools.bambots mem REG 253,1 47832 405770 /usr/lib/php5/20090626/mcrypt.so
php5 4214 tools.bambots mem REG 253,1 187752 404623 /usr/lib/libmcrypt.so.4.4.8
php5 4214 tools.bambots mem REG 253,1 55360 416196 /usr/lib/php5/20090626/mysql.so
php5 4214 tools.bambots mem REG 253,1 3420424 399920 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
php5 4214 tools.bambots mem REG 253,1 139960 416197 /usr/lib/php5/20090626/mysqli.so
php5 4214 tools.bambots mem REG 253,1 104808 416207 /usr/lib/php5/20090626/pdo.so
php5 4214 tools.bambots mem REG 253,1 31056 405586 /usr/lib/php5/20090626/pdo_mysql.so
php5 4214 tools.bambots mem REG 253,1 39344 416201 /usr/lib/php5/20090626/pdo_pgsql.so
php5 4214 tools.bambots mem REG 253,1 178616 415678 /usr/lib/libpq.so.5.4
php5 4214 tools.bambots mem REG 253,1 26960 416204 /usr/lib/php5/20090626/pdo_sqlite.so
php5 4214 tools.bambots mem REG 253,1 120896 416202 /usr/lib/php5/20090626/pgsql.so
php5 4214 tools.bambots mem REG 253,1 243328 395365 /usr/lib/php5/20090626/redis.so
php5 4214 tools.bambots mem REG 253,1 48480 416205 /usr/lib/php5/20090626/sqlite3.so
php5 4214 tools.bambots mem REG 253,1 34872 416198 /usr/lib/php5/20090626/xsl.so
php5 4214 tools.bambots mem REG 253,1 84928 403461 /usr/lib/x86_64-linux-gnu/libexslt.so.0.8.15
php5 4214 tools.bambots mem REG 253,1 245600 403460 /usr/lib/x86_64-linux-gnu/libxslt.so.1.1.26
php5 4214 tools.bambots mem REG 253,1 26792 396194 /usr/lib/libnss_db-2.2.3.so
php5 4214 tools.bambots mem REG 253,1 52120 921744 /lib/x86_64-linux-gnu/libnss_files-2.15.so
php5 4214 tools.bambots mem REG 253,1 217032 133899 /var/cache/nscd/services
php5 4214 tools.bambots mem REG 253,1 4141032 133897 /var/cache/nscd/hosts
php5 4214 tools.bambots 0r CHR 1,3 0t0 1029 /dev/null
php5 4214 tools.bambots 1w REG 0,20 1559795 87692860 /data/project/bambots/DatabaseReportBot.out
php5 4214 tools.bambots 2w REG 0,20 0 87692861 /data/project/bambots/DatabaseReportBot.err
php5 4214 tools.bambots 3u IPv4 56961106 0t0 TCP tools-exec-1218.tools.eqiad.wmflabs:55616->text-lb.eqiad.wikimedia.org:https (CLOSE_WAIT)
php5 4214 tools.bambots 4u sock 0,7 0t0 56853588 can't identify protocol
php5 4214 tools.bambots 5u sock 0,7 0t0 56853589 can't identify protocol
php5 4214 tools.bambots 6u sock 0,7 0t0 56851327 can't identify protocol
scfc@tools-exec-1218:~$In my session, I could access /data/project and /home just fine.
I have now qdel -f 11474 and the job has disappeared from qstat, but the process still remains:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND […] sgeadmin 4212 0.0 0.0 22556 2044 ? S Sep01 0:01 \_ /usr/lib/gridengine/sge_shepherd -bg 51454 4214 0.3 1.1 404756 94044 ? Ds Sep01 5:17 | \_ /usr/bin/php5 DatabaseReportBot.php InvalidNavbarLinks […]
and I assume that it will stay that way until the instance is rebooted.