Page MenuHomePhabricator

Cannot enter configuration mode on cr2-drmrs
Closed, ResolvedPublic

Description

Noticed a bit of an issue on cr2-drmrs when trying to enter configuration mode manually.

Once a user enters the "configure" command in the CLI their session just hangs. Further if you try to force logout that user session from another one the second session also times out. The same behaviour is observable even when connected via serial console as root user, however in that instance you can background the hung cli session with CTRL+Z, and then kill the hung process with 'kill %1'.

Investigating what might be the cause I came across some entries on the Juniper support site which are different but have some similarities:

https://supportportal.juniper.net/s/article/Junos-Error-during-commit-error-configuration-database-locked-by?language=en_US

https://supportportal.juniper.net/s/article/Junos-No-response-with-the-CLI-after-executing-the-command-commit-confirmed-x-twice?language=en_US

Certainly when looking for 'mgd' processes in state 'lockfs' there are many on the device:

root@cr2-drmrs> show system processes wide detail | match "PID|mgd"
  PID   UID  PPID CPU PRI  NI    RSS WCHAN    STARTED TT  STAT        TIME COMMAND
99205     0     1   0 103   0  48736 -        22Dec23  -  Rs   18891:19.58 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
 8901     0 23577   0  20   0  44756 select   11:42    -  Ss       0:00.06 mgd: (mgd) (root)/dev/ttyu0 (mgd)
  280     0     1   0  20   0  46228 lockf    Thu20    -  Is       0:00.25 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
  314     0 23577   0  20   0  43792 lockf    Thu20    -  I        0:00.00 /usr/sbin/mgd -N
  333     0 23577   0  20   0  43792 lockf    Sat11    -  I        0:00.00 /usr/sbin/mgd -N
  953     0 23577   0  20   0  43792 lockf    Mon02    -  I        0:00.00 /usr/sbin/mgd -N
 1267     0     1   0  20   0  46228 lockf    08:51    -  Is       0:00.26 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
 1302     0 23577   0  20   0  43792 lockf    08:52    -  I        0:00.01 /usr/sbin/mgd -N
 1389     0 23577   0  20   0  43792 lockf    Tue17    -  I        0:00.00 /usr/sbin/mgd -N
 1666     0 23577   0  20   0  43792 lockf    22Dec23  -  I        0:00.00 /usr/sbin/mgd -N
 1793     0 23577   0  20   0  43792 lockf    24Dec23  -  I        0:00.00 /usr/sbin/mgd -N
 1946     0 23577   0  20   0  43792 lockf    08:56    -  I        0:00.01 /usr/sbin/mgd -N
 2023     0 23577   0  20   0  43792 lockf    25Dec23  -  I        0:00.00 /usr/sbin/mgd -N
 2214     0 23577   0  20   0  43792 lockf    27Dec23  -  I        0:00.00 /usr/sbin/mgd -N
 2653     0 23577   0  20   0  43792 lockf    Thu21    -  I        0:00.00 /usr/sbin/mgd -N
 3109     0 23577   0  20   0  43792 lockf    Sat12    -  I        0:00.00 /usr/sbin/mgd -N
 3758     0 23577   0  20   0  43792 lockf    Mon03    -  I        0:00.00 /usr/sbin/mgd -N
 4194     0 23577   0  20   0  43792 lockf    Tue18    -  I        0:00.00 /usr/sbin/mgd -N
 4419     0 23577   0  20   0  43792 lockf    22Dec23  -  I        0:00.00 /usr/sbin/mgd -N
 4577     0 23577   0  20   0  43792 lockf    24Dec23  -  I        0:00.00 /usr/sbin/mgd -N
 4734     0 23577   0  20   0  43792 lockf    10:01    -  I        0:00.01 /usr/sbin/mgd -N
 4780     0 23577   0  20   0  43792 lockf    25Dec23  -  I        0:00.00 /usr/sbin/mgd -N
 5023     0 23577   0  20   0  43792 lockf    27Dec23  -  I        0:00.00 /usr/sbin/mgd -N
 5457     0 23577   0  20   0  43792 lockf    Thu22    -  I        0:00.00 /usr/sbin/mgd -N
 5894     0 23577   0  20   0  43792 lockf    Sat13    -  I        0:00.00 /usr/sbin/mgd -N
 6579     0 23577   0  20   0  43792 lockf    Mon05    -  I        0:00.00 /usr/sbin/mgd -N
 6818     0 23577   0  20   0  45024 lockf    11:00    -  Is       0:00.08 mgd: (mgd) (cmooney)/dev/pts/0 (mgd)
 6888     0 23577   0  20   0  44752 lockf    11:02    -  Is       0:00.04 mgd: (mgd) (cmooney)/dev/pts/1 (mgd)
 6988     0 23577   0  20   0  43792 lockf    Tue20    -  I        0:00.00 /usr/sbin/mgd -N
 7222     0 23577   0  20   0  43792 lockf    22Dec23  -  I        0:00.00 /usr/sbin/mgd -N
 7390     0 23577   0  20   0  43792 lockf    24Dec23  -  I        0:00.00 /usr/sbin/mgd -N
 7582     0 23577   0  20   0  43792 lockf    25Dec23  -  I        0:00.00 /usr/sbin/mgd -N
 7631     0 23577   0  20   0  43792 lockf    11:07    -  I        0:00.01 /usr/sbin/mgd -N
 7783     0     1   0  20   0  46228 lockf    27Dec23  -  Is       0:00.25 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
 7785     0 23577   0  20   0  43792 lockf    27Dec23  -  I        0:00.00 /usr/sbin/mgd -N
 7819     0 23577   0  20   0  43792 lockf    27Dec23  -  I        0:00.00 /usr/sbin/mgd -N
 8243     0 23577   0  20   0  43792 lockf    Thu23    -  I        0:00.00 /usr/sbin/mgd -N
 8520     0 23577   0  22   0  44740 lockf    11:33    -  Is       0:00.04 mgd: (mgd) (root)/dev/ttyu0 (mgd)
 8703     0 23577   0  20   0  43792 lockf    Sat15    -  I        0:00.00 /usr/sbin/mgd -N
 8735     0     1   0  20   0  46228 lockf    Tue20    -  Is       0:00.26 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
 8780     0 23577   0  20   0  43792 lockf    Tue20    -  I        0:00.00 /usr/sbin/mgd -N
 8842     0 23577   0  20   0  44648 lockf    11:41    -  Is       0:00.04 mgd: (mgd) (root)/dev/ttyu0 (mgd)
 9122     0 23577   0  20   0  43792 lockf    11:47    -  I        0:00.01 /usr/sbin/mgd -N
 9125     0  8901   0  37   0   3184 -        11:48    -  R        0:00.18 /bin/ps -X -axrOuid,ppid,cpu,pri,nice,rss,wchan,start --libxo xml,pretty
 9358     0 23577   0  20   0  43792 lockf    Mon06    -  I        0:00.00 /usr/sbin/mgd -N
 9793     0 23577   0  20   0  43792 lockf    Tue21    -  I        0:00.00 /usr/sbin/mgd -N
10004     0 23577   0  20   0  43792 lockf    22Dec23  -  I        0:00.00 /usr/sbin/mgd -N
10158     0 23577   0  20   0  43792 lockf    24Dec23  -  I        0:00.00 /usr/sbin/mgd -N
10332     0 23577   0  20   0  43792 lockf    25Dec23  -  I        0:00.00 /usr/sbin/mgd -N
10582     0 23577   0  20   0  43792 lockf    27Dec23  -  I        0:00.00 /usr/sbin/mgd -N
11075     0 23577   0  20   0  43792 lockf    Fri01    -  I        0:00.00 /usr/sbin/mgd -N
11507     0 23577   0  20   0  43792 lockf    Sat16    -  I        0:00.00 /usr/sbin/mgd -N
12115     0 23577   0  20   0  43792 lockf    Mon07    -  I        0:00.00 /usr/sbin/mgd -N
12571     0 23577   0  20   0  43792 lockf    Tue22    -  I        0:00.00 /usr/sbin/mgd -N
12806     0 23577   0  20   0  43792 lockf    22Dec23  -  I        0:00.00 /usr/sbin/mgd -N
12971     0 23577   0  20   0  43792 lockf    24Dec23  -  I        0:00.00 /usr/sbin/mgd -N
13137     0 23577   0  20   0  43792 lockf    25Dec23  -  I        0:00.00 /usr/sbin/mgd -N
13460     0 23577   0  20   0  43792 lockf    27Dec23  -  I        0:00.00 /usr/sbin/mgd -N
13878     0 23577   0  20   0  43792 lockf    Fri02    -  I        0:00.00 /usr/sbin/mgd -N
14331     0 23577   0  20   0  43792 lockf    Sat17    -  I        0:00.00 /usr/sbin/mgd -N
14937     0 23577   0  20   0  43792 lockf    Mon08    -  I        0:00.00 /usr/sbin/mgd -N
14968     0     1   0  20   0  46228 lockf    25Dec23  -  Is       0:00.25 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
15004     0 23577   0  20   0  43792 lockf    25Dec23  -  I        0:00.00 /usr/sbin/mgd -N
15388     0 23577   0  20   0  43792 lockf    Tue23    -  I        0:00.00 /usr/sbin/mgd -N
15611     0 23577   0  20   0  43792 lockf    22Dec23  -  I        0:00.00 /usr/sbin/mgd -N
15774     0 23577   0  20   0  43792 lockf    24Dec23  -  I        0:00.00 /usr/sbin/mgd -N
15976     0 23577   0  20   0  43792 lockf    25Dec23  -  I        0:00.00 /usr/sbin/mgd -N
16145     0     1   0  20   0  46228 lockf    Mon08    -  Is       0:00.26 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
16180     0 23577   0  20   0  43792 lockf    Mon08    -  I        0:00.00 /usr/sbin/mgd -N
16261     0 23577   0  20   0  43792 lockf    27Dec23  -  I        0:00.00 /usr/sbin/mgd -N
16702     0 23577   0  20   0  43792 lockf    Fri03    -  I        0:00.00 /usr/sbin/mgd -N
17125     0 23577   0  20   0  43792 lockf    Sat18    -  I        0:00.00 /usr/sbin/mgd -N
17734     0 23577   0  20   0  43792 lockf    Mon09    -  I        0:00.00 /usr/sbin/mgd -N
18170     0 23577   0  20   0  43792 lockf    Wed00    -  I        0:00.00 /usr/sbin/mgd -N
18404     0 23577   0  20   0  43792 lockf    22Dec23  -  I        0:00.00 /usr/sbin/mgd -N
18554     0 23577   0  20   0  43792 lockf    24Dec23  -  I        0:00.00 /usr/sbin/mgd -N
18747     0 23577   0  20   0  43792 lockf    25Dec23  -  I        0:00.00 /usr/sbin/mgd -N
19041     0 23577   0  20   0  43792 lockf    27Dec23  -  I        0:00.00 /usr/sbin/mgd -N
19516     0 23577   0  20   0  43792 lockf    Fri04    -  I        0:00.00 /usr/sbin/mgd -N
19905     0 23577   0  20   0  43792 lockf    Sat19    -  I        0:00.00 /usr/sbin/mgd -N
20506     0 23577   0  20   0  43792 lockf    Mon10    -  I        0:00.00 /usr/sbin/mgd -N
20954     0 23577   0  20   0  43792 lockf    Wed01    -  I        0:00.00 /usr/sbin/mgd -N
21150     0 23577   0  20   0  43792 lockf    22Dec23  -  I        0:00.00 /usr/sbin/mgd -N
21348     0 23577   0  20   0  43792 lockf    24Dec23  -  I        0:00.00 /usr/sbin/mgd -N
21522     0 23577   0  20   0  43792 lockf    25Dec23  -  I        0:00.00 /usr/sbin/mgd -N
21836     0 23577   0  20   0  43792 lockf    27Dec23  -  I        0:00.00 /usr/sbin/mgd -N
22288     0 23577   0  20   0  43792 lockf    Fri05    -  I        0:00.00 /usr/sbin/mgd -N
22501     0     1   0  20   0  46228 lockf    24Dec23  -  Is       0:00.25 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
22547     0 23577   0  20   0  43792 lockf    24Dec23  -  I        0:00.00 /usr/sbin/mgd -N
22679     0 23577   0  20   0  43792 lockf    Sat20    -  I        0:00.00 /usr/sbin/mgd -N
23285     0 23577   0  20   0  43792 lockf    Mon11    -  I        0:00.00 /usr/sbin/mgd -N
23577     0 23462   0  20   0  43792 select    7Mar22  -  I        2:01.69 /usr/sbin/mgd -N
23622     0 23462   0  52   0   3644 select    7Mar22  -  I        0:00.01 /usr/sbin/inetd -p /var/run/inetd_4.pid -N -JU __juniper_private4__ /etc/inetd
23635     0 23462   0  20   0  51152 select    7Mar22  -  I        0:03.39 /usr/libexec32/bbe-smgd -b -N
23637     0 23462   0  20   0   8348 select    7Mar22  -  I        0:03.39 /usr/sbin/appidd -N
23658     0 23462   0  20   0   6836 select    7Mar22  -  I        0:00.05 /usr/sbin/mgd-api -N
23671     0     1   0  20   0  46228 lockf    Sat20    -  Is       0:00.26 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
23716     0 23577   0  20   0  43792 lockf    Sat20    -  I        0:00.00 /usr/sbin/mgd -N
23816     0 23577   0  20   0  43792 lockf    Wed02    -  I        0:00.00 /usr/sbin/mgd -N
23969     0 23577   0  20   0  43792 lockf    22Dec23  -  I        0:00.00 /usr/sbin/mgd -N
24158     0 23577   0  20   0  43792 lockf    24Dec23  -  I        0:00.00 /usr/sbin/mgd -N
24356     0 24245   0  20   0  44812 select    7Mar22  -  I       45:02.51 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (root) (mgd)
24403     0 23577   0  20   0  43792 lockf    26Dec23  -  I        0:00.00 /usr/sbin/mgd -N
24673     0 23577   0  20   0  43792 lockf    27Dec23  -  I        0:00.00 /usr/sbin/mgd -N
25155     0 23577   0  20   0  43792 lockf    Fri06    -  I        0:00.00 /usr/sbin/mgd -N
25574     0 23577   0  20   0  43792 lockf    Sat21    -  I        0:00.00 /usr/sbin/mgd -N
26200     0 23577   0  20   0  43792 lockf    Mon12    -  I        0:00.00 /usr/sbin/mgd -N
26653     0 23577   0  20   0  43792 lockf    Wed03    -  I        0:00.00 /usr/sbin/mgd -N
26772     0 23577   0  20   0  43792 lockf    22Dec23  -  I        0:00.00 /usr/sbin/mgd -N
27006     0 23577   0  20   0  43792 lockf    24Dec23  -  I        0:00.00 /usr/sbin/mgd -N
27145     0 23577   0  20   0  43792 lockf    26Dec23  -  I        0:00.00 /usr/sbin/mgd -N
27462     0 23577   0  20   0  43792 lockf    27Dec23  -  I        0:00.00 /usr/sbin/mgd -N
27917     0 23577   0  20   0  43792 lockf    Fri07    -  I        0:00.00 /usr/sbin/mgd -N
28347     0 23577   0  20   0  43792 lockf    Sat22    -  I        0:00.00 /usr/sbin/mgd -N
28977     0 23577   0  20   0  43792 lockf    Mon13    -  I        0:00.00 /usr/sbin/mgd -N
29452     0 23577   0  20   0  43792 lockf    Wed04    -  I        0:00.00 /usr/sbin/mgd -N
29560     0 23577   0  20   0  43792 lockf    22Dec23  -  I        0:00.00 /usr/sbin/mgd -N
29748     0 23577   0  20   0  43792 lockf    24Dec23  -  I        0:00.00 /usr/sbin/mgd -N
29945     0 23577   0  20   0  43792 lockf    26Dec23  -  I        0:00.00 /usr/sbin/mgd -N
30198     0 23577   0  20   0  43792 lockf    27Dec23  -  I        0:00.00 /usr/sbin/mgd -N
30303     0     1   0  20   0  46228 lockf    22Dec23  -  Is       0:00.25 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
30337     0 23577   0  20   0  43792 lockf    22Dec23  -  I        0:00.00 /usr/sbin/mgd -N
30709     0 23577   0  20   0  43792 lockf    Fri08    -  I        0:00.00 /usr/sbin/mgd -N
31159     0     1   0  20   0  46228 lockf    Fri08    -  Is       0:00.26 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
31166     0 23577   0  20   0  43792 lockf    Sat23    -  I        0:00.00 /usr/sbin/mgd -N
31194     0 23577   0  20   0  43792 lockf    Fri08    -  I        0:00.00 /usr/sbin/mgd -N
31779     0 23577   0  20   0  43792 lockf    Mon14    -  I        0:00.00 /usr/sbin/mgd -N
32209     0 23577   0  20   0  43792 lockf    Wed05    -  I        0:00.00 /usr/sbin/mgd -N
32347     0 23577   0  20   0  43792 lockf    22Dec23  -  I        0:00.00 /usr/sbin/mgd -N
32566     0 23577   0  20   0  43792 lockf    24Dec23  -  I        0:00.00 /usr/sbin/mgd -N
32740     0 23577   0  20   0  43792 lockf    26Dec23  -  I        0:00.00 /usr/sbin/mgd -N
32993     0 23577   0  20   0  43792 lockf    27Dec23  -  I        0:00.00 /usr/sbin/mgd -N
33488     0 23577   0  20   0  43792 lockf    Fri09    -  I        0:00.00 /usr/sbin/mgd -N
33948     0 23577   0  20   0  43792 lockf    Sun00    -  I        0:00.00 /usr/sbin/mgd -N
34551     0 23577   0  20   0  43792 lockf    Mon15    -  I        0:00.00 /usr/sbin/mgd -N
35006     0 23577   0  20   0  43792 lockf    Wed06    -  I        0:00.00 /usr/sbin/mgd -N
35116     0 23577   0  20   0  43792 lockf    22Dec23  -  I        0:00.00 /usr/sbin/mgd -N
35300     0 23577   0  20   0  43792 lockf    24Dec23  -  I        0:00.00 /usr/sbin/mgd -N
35487     0 23577   0  20   0  43792 lockf    26Dec23  -  I        0:00.00 /usr/sbin/mgd -N
35790     0 23577   0  20   0  43792 lockf    27Dec23  -  I        0:00.00 /usr/sbin/mgd -N
36281     0 23577   0  20   0  43792 lockf    Fri10    -  I        0:00.00 /usr/sbin/mgd -N
36748     0 23577   0  20   0  43792 lockf    Sun01    -  I        0:00.00 /usr/sbin/mgd -N
37343     0 23577   0  20   0  43792 lockf    Mon16    -  I        0:00.00 /usr/sbin/mgd -N
37781     0 23577   0  20   0  43792 lockf    Wed07    -  I        0:00.00 /usr/sbin/mgd -N
37878     0 23577   0  20   0  43792 lockf    22Dec23  -  I        0:00.00 /usr/sbin/mgd -N
38069     0 23577   0  20   0  43792 lockf    24Dec23  -  I        0:00.00 /usr/sbin/mgd -N
38279     0 23577   0  20   0  43792 lockf    26Dec23  -  I        0:00.00 /usr/sbin/mgd -N
38560     0 23577   0  20   0  43792 lockf    27Dec23  -  I        0:00.00 /usr/sbin/mgd -N
38718     0     1   0  20   0  46228 lockf    27Dec23  -  Is       0:00.25 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
38752     0 23577   0  20   0  43792 lockf    27Dec23  -  I        0:00.00 /usr/sbin/mgd -N
39029     0 23577   0  20   0  43792 lockf    Fri11    -  I        0:00.00 /usr/sbin/mgd -N
39526     0 23577   0  20   0  43792 lockf    Sun02    -  I        0:00.00 /usr/sbin/mgd -N
39598     0     1   0  20   0  46228 lockf    Wed08    -  Is       0:00.26 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
39633     0 23577   0  20   0  43792 lockf    Wed08    -  I        0:00.00 /usr/sbin/mgd -N
40094     0 23577   0  20   0  43792 lockf    Mon18    -  I        0:00.00 /usr/sbin/mgd -N
40610     0 23577   0  20   0  43792 lockf    Wed09    -  I        0:00.01 /usr/sbin/mgd -N
40652     0 23577   0  20   0  43792 lockf    23Dec23  -  I        0:00.00 /usr/sbin/mgd -N
40857     0 23577   0  20   0  43792 lockf    24Dec23  -  I        0:00.00 /usr/sbin/mgd -N
41025     0 23577   0  20   0  43792 lockf    26Dec23  -  I        0:00.00 /usr/sbin/mgd -N
41358     0 23577   0  20   0  43792 lockf    27Dec23  -  I        0:00.00 /usr/sbin/mgd -N
41827     0 23577   0  20   0  43792 lockf    Fri12    -  I        0:00.00 /usr/sbin/mgd -N
42323     0 23577   0  20   0  43792 lockf    Sun04    -  I        0:00.00 /usr/sbin/mgd -N
42913     0 23577   0  20   0  43792 lockf    Mon19    -  I        0:00.00 /usr/sbin/mgd -N
43360     0 23577   0  20   0  43792 lockf    Wed10    -  I        0:00.01 /usr/sbin/mgd -N
43416     0 23577   0  20   0  43792 lockf    23Dec23  -  I        0:00.00 /usr/sbin/mgd -N
43599     0 23577   0  20   0  43792 lockf    24Dec23  -  I        0:00.00 /usr/sbin/mgd -N
43812     0 23577   0  20   0  43792 lockf    26Dec23  -  I        0:00.00 /usr/sbin/mgd -N
44102     0 23577   0  20   0  43792 lockf    27Dec23  -  I        0:00.00 /usr/sbin/mgd -N
44599     0 23577   0  20   0  43792 lockf    Fri14    -  I        0:00.00 /usr/sbin/mgd -N
45098     0 23577   0  20   0  43792 lockf    Sun05    -  I        0:00.00 /usr/sbin/mgd -N
45660     0 23577   0  20   0  43792 lockf    Mon20    -  I        0:00.00 /usr/sbin/mgd -N
45722     0     1   0  20   0  46228 lockf    26Dec23  -  Is       0:00.25 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
45757     0 23577   0  20   0  43792 lockf    26Dec23  -  I        0:00.00 /usr/sbin/mgd -N
46154     0 23577   0  20   0  43792 lockf    Wed11    -  I        0:00.01 /usr/sbin/mgd -N
46213     0 23577   0  20   0  43792 lockf    23Dec23  -  I        0:00.00 /usr/sbin/mgd -N
46419     0 23577   0  20   0  43792 lockf    24Dec23  -  I        0:00.00 /usr/sbin/mgd -N
46566     0 23577   0  20   0  43792 lockf    26Dec23  -  I        0:00.00 /usr/sbin/mgd -N
46984     0 23577   0  20   0  43792 lockf    28Dec23  -  I        0:00.00 /usr/sbin/mgd -N
47060     0     1   0  20   0  46228 lockf    Mon20    -  Is       0:00.26 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
47105     0 23577   0  20   0  43792 lockf    Mon20    -  I        0:00.00 /usr/sbin/mgd -N
47389     0 23577   0  20   0  43792 lockf    Fri15    -  I        0:00.00 /usr/sbin/mgd -N
47894     0 23577   0  20   0  43792 lockf    Sun06    -  I        0:00.00 /usr/sbin/mgd -N
48435     0 23577   0  20   0  43792 lockf    Mon21    -  I        0:00.00 /usr/sbin/mgd -N
48926     0 23577   0  20   0  43792 lockf    12:19    -  I        0:00.01 /usr/sbin/mgd -N
48977     0 23577   0  20   0  43792 lockf    23Dec23  -  I        0:00.00 /usr/sbin/mgd -N
49212     0 23577   0  20   0  43792 lockf    24Dec23  -  I        0:00.00 /usr/sbin/mgd -N
49376     0 23577   0  20   0  43792 lockf    26Dec23  -  I        0:00.00 /usr/sbin/mgd -N
49761     0 23577   0  20   0  43792 lockf    28Dec23  -  I        0:00.00 /usr/sbin/mgd -N
50159     0 23577   0  20   0  43792 lockf    Fri16    -  I        0:00.00 /usr/sbin/mgd -N
50660     0 23577   0  20   0  43792 lockf    Sun07    -  I        0:00.00 /usr/sbin/mgd -N
51227     0 23577   0  20   0  43792 lockf    Mon22    -  I        0:00.00 /usr/sbin/mgd -N
51679     0 23577   0  20   0  43792 lockf    13:24    -  I        0:00.01 /usr/sbin/mgd -N
51748     0 23577   0  20   0  43792 lockf    23Dec23  -  I        0:00.00 /usr/sbin/mgd -N
51974     0 23577   0  20   0  43792 lockf    24Dec23  -  I        0:00.00 /usr/sbin/mgd -N
52108     0 23577   0  20   0  43792 lockf    26Dec23  -  I        0:00.00 /usr/sbin/mgd -N
52529     0 23577   0  20   0  43792 lockf    28Dec23  -  I        0:00.00 /usr/sbin/mgd -N
52926     0 23577   0  20   0  43792 lockf    Fri17    -  I        0:00.00 /usr/sbin/mgd -N
53378     0     1   0  20   0  46228 lockf    24Dec23  -  Is       0:00.25 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
53412     0 23577   0  20   0  43792 lockf    24Dec23  -  I        0:00.00 /usr/sbin/mgd -N
53436     0 23577   0  20   0  43792 lockf    Sun08    -  I        0:00.00 /usr/sbin/mgd -N
53977     0 23577   0  20   0  43792 lockf    Mon23    -  I        0:00.00 /usr/sbin/mgd -N
54411     0     1   0  20   0  46228 lockf    Sun08    -  Is       0:00.26 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
54446     0 23577   0  20   0  43792 lockf    Sun08    -  I        0:00.00 /usr/sbin/mgd -N
54477     0 23577   0  20   0  43792 lockf    14:29    -  I        0:00.01 /usr/sbin/mgd -N
54538     0 23577   0  20   0  43792 lockf    23Dec23  -  I        0:00.00 /usr/sbin/mgd -N
54750     0 23577   0  20   0  43792 lockf    24Dec23  -  I        0:00.00 /usr/sbin/mgd -N
54888     0 23577   0  20   0  43792 lockf    26Dec23  -  I        0:00.00 /usr/sbin/mgd -N
55307     0 23577   0  20   0  43792 lockf    28Dec23  -  I        0:00.00 /usr/sbin/mgd -N
55699     0 23577   0  20   0  43792 lockf    Fri18    -  I        0:00.00 /usr/sbin/mgd -N
56210     0 23577   0  20   0  43792 lockf    Sun09    -  I        0:00.00 /usr/sbin/mgd -N
56801     0 23577   0  20   0  43792 lockf    Tue00    -  I        0:00.00 /usr/sbin/mgd -N
57273     0 23577   0  20   0  43792 lockf    15:34    -  I        0:00.01 /usr/sbin/mgd -N
57308     0 23577   0  20   0  43792 lockf    23Dec23  -  I        0:00.00 /usr/sbin/mgd -N
57512     0 23577   0  20   0  43792 lockf    24Dec23  -  I        0:00.00 /usr/sbin/mgd -N
57654     0 23577   0  20   0  43792 lockf    26Dec23  -  I        0:00.00 /usr/sbin/mgd -N
58100     0 23577   0  20   0  43792 lockf    28Dec23  -  I        0:00.00 /usr/sbin/mgd -N
58463     0 23577   0  20   0  43792 lockf    Fri19    -  I        0:00.00 /usr/sbin/mgd -N
59008     0 23577   0  20   0  43792 lockf    Sun10    -  I        0:00.00 /usr/sbin/mgd -N
59582     0 23577   0  20   0  43792 lockf    Tue01    -  I        0:00.00 /usr/sbin/mgd -N
60040     0 23577   0  20   0  43792 lockf    16:39    -  I        0:00.01 /usr/sbin/mgd -N
60068     0 23577   0  20   0  43792 lockf    23Dec23  -  I        0:00.00 /usr/sbin/mgd -N
60280     0 23577   0  20   0  43792 lockf    24Dec23  -  I        0:00.00 /usr/sbin/mgd -N
60421     0 23577   0  20   0  43792 lockf    26Dec23  -  I        0:00.00 /usr/sbin/mgd -N
60876     0 23577   0  20   0  43792 lockf    28Dec23  -  I        0:00.00 /usr/sbin/mgd -N
60935     0     1   0  20   0  46228 lockf    23Dec23  -  Is       0:00.25 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
60982     0 23577   0  20   0  43792 lockf    23Dec23  -  I        0:00.00 /usr/sbin/mgd -N
61258     0 23577   0  20   0  43792 lockf    Fri20    -  I        0:00.00 /usr/sbin/mgd -N
61800     0 23577   0  20   0  43792 lockf    Sun11    -  I        0:00.00 /usr/sbin/mgd -N
61948     0     1   0  20   0  46228 lockf    Fri20    -  Is       0:00.25 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
61993     0 23577   0  20   0  43792 lockf    Fri20    -  I        0:00.00 /usr/sbin/mgd -N
62356     0 23577   0  20   0  43792 lockf    Tue02    -  I        0:00.00 /usr/sbin/mgd -N
62817     0 23577   0  20   0  43792 lockf    17:44    -  I        0:00.01 /usr/sbin/mgd -N
62836     0 23577   0  20   0  43792 lockf    23Dec23  -  I        0:00.00 /usr/sbin/mgd -N
63072     0 23577   0  20   0  43792 lockf    25Dec23  -  I        0:00.00 /usr/sbin/mgd -N
63212     0 23577   0  20   0  43792 lockf    26Dec23  -  I        0:00.00 /usr/sbin/mgd -N
63664     0 23577   0  20   0  43792 lockf    28Dec23  -  I        0:00.00 /usr/sbin/mgd -N
64034     0 23577   0  20   0  43792 lockf    Fri21    -  I        0:00.00 /usr/sbin/mgd -N
64579     0 23577   0  20   0  43792 lockf    Sun12    -  I        0:00.00 /usr/sbin/mgd -N
65161     0 23577   0  20   0  43792 lockf    Tue03    -  I        0:00.00 /usr/sbin/mgd -N
65590     0 23577   0  20   0  43792 lockf    18:49    -  I        0:00.01 /usr/sbin/mgd -N
65625     0 23577   0  20   0  43792 lockf    23Dec23  -  I        0:00.00 /usr/sbin/mgd -N
65836     0 23577   0  20   0  43792 lockf    25Dec23  -  I        0:00.00 /usr/sbin/mgd -N
65978     0 23577   0  20   0  43792 lockf    26Dec23  -  I        0:00.00 /usr/sbin/mgd -N
66439     0 23577   0  20   0  43792 lockf    28Dec23  -  I        0:00.00 /usr/sbin/mgd -N
66503     0 23577   0  20   0  44664 lockf    23Dec23  -  Is       0:00.02 mgd: (mgd) (ayounsi)/dev/pts/0 (mgd)
66825     0 23577   0  20   0  43792 lockf    Fri22    -  I        0:00.00 /usr/sbin/mgd -N
67352     0 23577   0  20   0  43792 lockf    Sun13    -  I        0:00.00 /usr/sbin/mgd -N
67932     0 23577   0  20   0  43792 lockf    Tue04    -  I        0:00.00 /usr/sbin/mgd -N
68363     0 23577   0  20   0  43792 lockf    19:53    -  I        0:00.01 /usr/sbin/mgd -N
68392     0 23577   0  20   0  43792 lockf    23Dec23  -  I        0:00.00 /usr/sbin/mgd -N
68609     0 23577   0  20   0  43792 lockf    25Dec23  -  I        0:00.00 /usr/sbin/mgd -N
68765     0 23577   0  20   0  43792 lockf    26Dec23  -  I        0:00.00 /usr/sbin/mgd -N
69204     0 23577   0  20   0  43792 lockf    28Dec23  -  I        0:00.00 /usr/sbin/mgd -N
69398     0     1   0  20   0  46228 lockf    28Dec23  -  Is       0:00.26 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
69433     0 23577   0  20   0  43792 lockf    28Dec23  -  I        0:00.00 /usr/sbin/mgd -N
69601     0 23577   0  20   0  43792 lockf    Fri23    -  I        0:00.00 /usr/sbin/mgd -N
70127     0 23577   0  20   0  43792 lockf    Sun14    -  I        0:00.00 /usr/sbin/mgd -N
70452     0     1   0  20   0  46228 lockf    20:53    -  Is       0:00.26 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
70497     0 23577   0  20   0  43792 lockf    20:54    -  I        0:00.01 /usr/sbin/mgd -N
70710     0 23577   0  20   0  43792 lockf    Tue05    -  I        0:00.00 /usr/sbin/mgd -N
71111     0 23577   0  20   0  43792 lockf    20:58    -  I        0:00.01 /usr/sbin/mgd -N
71169     0 23577   0  20   0  43792 lockf    23Dec23  -  I        0:00.00 /usr/sbin/mgd -N
71376     0 23577   0  20   0  43792 lockf    25Dec23  -  I        0:00.00 /usr/sbin/mgd -N
71509     0 23577   0  20   0  43792 lockf    26Dec23  -  I        0:00.00 /usr/sbin/mgd -N
71983     0 23577   0  20   0  43792 lockf    28Dec23  -  I        0:00.00 /usr/sbin/mgd -N
72401     0 23577   0  20   0  43792 lockf    Sat00    -  I        0:00.00 /usr/sbin/mgd -N
72918     0 23577   0  20   0  43792 lockf    Sun15    -  I        0:00.00 /usr/sbin/mgd -N
73506     0 23577   0  20   0  43792 lockf    Tue07    -  I        0:00.00 /usr/sbin/mgd -N
73953     0 23577   0  20   0  43792 lockf    22:05    -  I        0:00.01 /usr/sbin/mgd -N
73957     0 23577   0  20   0  43792 lockf    23Dec23  -  I        0:00.00 /usr/sbin/mgd -N
74147     0 23577   0  20   0  43792 lockf    25Dec23  -  I        0:00.00 /usr/sbin/mgd -N
74271     0 23577   0  20   0  43792 lockf    26Dec23  -  I        0:00.00 /usr/sbin/mgd -N
74746     0 23577   0  20   0  43792 lockf    28Dec23  -  I        0:00.00 /usr/sbin/mgd -N
75175     0 23577   0  20   0  43792 lockf    Sat01    -  I        0:00.00 /usr/sbin/mgd -N
75691     0 23577   0  20   0  43792 lockf    Sun17    -  I        0:00.00 /usr/sbin/mgd -N
76278     0 23577   0  20   0  43792 lockf    Tue08    -  I        0:00.00 /usr/sbin/mgd -N
76485     0     1   0  20   0  46228 lockf    26Dec23  -  Is       0:00.25 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
76519     0 23577   0  20   0  43792 lockf    26Dec23  -  I        0:00.00 /usr/sbin/mgd -N
76699     0 23577   0  20   0  43792 lockf    23Dec23  -  I        0:00.00 /usr/sbin/mgd -N
76774     0 23577   0  20   0  43792 lockf    23:10    -  I        0:00.01 /usr/sbin/mgd -N
76912     0 23577   0  20   0  43792 lockf    25Dec23  -  I        0:00.00 /usr/sbin/mgd -N
77054     0 23577   0  20   0  43792 lockf    26Dec23  -  I        0:00.00 /usr/sbin/mgd -N
77516     0 23577   0  20   0  43792 lockf    Thu12    -  I        0:00.00 /usr/sbin/mgd -N
77754     0     1   0  20   0  46228 lockf    Tue08    -  Is       0:00.26 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
77789     0 23577   0  20   0  43792 lockf    Tue08    -  I        0:00.00 /usr/sbin/mgd -N
77972     0 23577   0  20   0  43792 lockf    Sat03    -  I        0:00.00 /usr/sbin/mgd -N
78482     0 23577   0  20   0  43792 lockf    Sun18    -  I        0:00.00 /usr/sbin/mgd -N
79025     0 23577   0  20   0  43792 lockf    Tue09    -  I        0:00.00 /usr/sbin/mgd -N
79486     0 23577   0  20   0  43792 lockf    23Dec23  -  I        0:00.00 /usr/sbin/mgd -N
79557     0 23577   0  20   0  43792 lockf    00:16    -  I        0:00.01 /usr/sbin/mgd -N
79657     0 23577   0  20   0  43792 lockf    25Dec23  -  I        0:00.00 /usr/sbin/mgd -N
79807     0 23577   0  20   0  43792 lockf    26Dec23  -  I        0:00.00 /usr/sbin/mgd -N
80310     0 23577   0  20   0  43792 lockf    Thu13    -  I        0:00.00 /usr/sbin/mgd -N
80781     0 23577   0  20   0  43792 lockf    Sat04    -  I        0:00.00 /usr/sbin/mgd -N
81258     0 23577   0  20   0  43792 lockf    Sun19    -  I        0:00.00 /usr/sbin/mgd -N
81818     0 23577   0  20   0  43792 lockf    Tue10    -  I        0:00.00 /usr/sbin/mgd -N
82276     0 23577   0  20   0  43792 lockf    23Dec23  -  I        0:00.00 /usr/sbin/mgd -N
82356     0 23577   0  20   0  43792 lockf    01:21    -  I        0:00.01 /usr/sbin/mgd -N
82468     0 23577   0  20   0  43792 lockf    25Dec23  -  I        0:00.00 /usr/sbin/mgd -N
82572     0 23577   0  20   0  43792 lockf    26Dec23  -  I        0:00.00 /usr/sbin/mgd -N
83073     0 23577   0  20   0  43792 lockf    Thu14    -  I        0:00.00 /usr/sbin/mgd -N
83576     0 23577   0  20   0  43792 lockf    Sat05    -  I        0:00.00 /usr/sbin/mgd -N
83994     0 23577   0  20   0  43792 lockf    Sun20    -  I        0:00.00 /usr/sbin/mgd -N
84007     0     1   0  20   0  46228 lockf    25Dec23  -  Is       0:00.25 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
84042     0 23577   0  20   0  43792 lockf    25Dec23  -  I        0:00.00 /usr/sbin/mgd -N
84566     0 23577   0  20   0  43792 lockf    Tue11    -  I        0:00.00 /usr/sbin/mgd -N
85035     0 23577   0  20   0  43792 lockf    23Dec23  -  I        0:00.00 /usr/sbin/mgd -N
85126     0 23577   0  20   0  43792 lockf    02:26    -  I        0:00.01 /usr/sbin/mgd -N
85239     0     1   0  20   0  46228 lockf    Sun20    -  Is       0:00.26 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
85245     0 23577   0  20   0  43792 lockf    25Dec23  -  I        0:00.00 /usr/sbin/mgd -N
85285     0 23577   0  20   0  43792 lockf    Sun20    -  I        0:00.00 /usr/sbin/mgd -N
85366     0 23577   0  20   0  43792 lockf    27Dec23  -  I        0:00.00 /usr/sbin/mgd -N
85846     0 23577   0  20   0  43792 lockf    Thu15    -  I        0:00.00 /usr/sbin/mgd -N
86326     0 23577   0  20   0  43792 lockf    Sat06    -  I        0:00.00 /usr/sbin/mgd -N
86926     0 23577   0  20   0  43792 lockf    Sun21    -  I        0:00.00 /usr/sbin/mgd -N
87366     0 23577   0  20   0  43792 lockf    Tue12    -  I        0:00.00 /usr/sbin/mgd -N
87810     0 23577   0  20   0  43792 lockf    23Dec23  -  I        0:00.00 /usr/sbin/mgd -N
87906     0 23577   0  20   0  43792 lockf    03:31    -  I        0:00.01 /usr/sbin/mgd -N
88065     0 23577   0  20   0  43792 lockf    25Dec23  -  I        0:00.00 /usr/sbin/mgd -N
88199     0 23577   0  20   0  43792 lockf    27Dec23  -  I        0:00.00 /usr/sbin/mgd -N
88636     0 23577   0  20   0  43792 lockf    Thu16    -  I        0:00.00 /usr/sbin/mgd -N
89116     0 23577   0  20   0  43792 lockf    Sat07    -  I        0:00.00 /usr/sbin/mgd -N
89722     0 23577   0  20   0  43792 lockf    Sun22    -  I        0:00.00 /usr/sbin/mgd -N
90145     0 23577   0  20   0  43792 lockf    Tue13    -  I        0:00.00 /usr/sbin/mgd -N
90596     0 23577   0  20   0  43792 lockf    23Dec23  -  I        0:00.00 /usr/sbin/mgd -N
90704     0 23577   0  20   0  43792 lockf    04:36    -  I        0:00.01 /usr/sbin/mgd -N
90833     0 23577   0  20   0  43792 lockf    25Dec23  -  I        0:00.00 /usr/sbin/mgd -N
90975     0 23577   0  20   0  43792 lockf    27Dec23  -  I        0:00.00 /usr/sbin/mgd -N
91408     0 23577   0  20   0  43792 lockf    Thu17    -  I        0:00.00 /usr/sbin/mgd -N
91660     0     1   0  20   0  46228 lockf    23Dec23  -  Is       0:00.25 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
91694     0 23577   0  20   0  43792 lockf    23Dec23  -  I        0:00.00 /usr/sbin/mgd -N
91862     0 23577   0  20   0  43792 lockf    Sat08    -  I        0:00.00 /usr/sbin/mgd -N
92493     0 23577   0  20   0  43792 lockf    Sun23    -  I        0:00.00 /usr/sbin/mgd -N
92640     0     1   0  20   0  46228 lockf    Sat08    -  Is       0:00.26 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
92675     0 23577   0  20   0  43792 lockf    Sat08    -  I        0:00.00 /usr/sbin/mgd -N
92938     0 23577   0  20   0  43792 lockf    Tue14    -  I        0:00.00 /usr/sbin/mgd -N
93346     0 23577   0  20   0  43792 lockf    23Dec23  -  I        0:00.00 /usr/sbin/mgd -N
93486     0 23577   0  20   0  43792 lockf    05:41    -  I        0:00.01 /usr/sbin/mgd -N
93603     0 23577   0  20   0  43792 lockf    25Dec23  -  I        0:00.00 /usr/sbin/mgd -N
93772     0 23577   0  20   0  43792 lockf    27Dec23  -  I        0:00.00 /usr/sbin/mgd -N
94204     0 23577   0  20   0  43792 lockf    Thu18    -  I        0:00.00 /usr/sbin/mgd -N
94686     0 23577   0  20   0  43792 lockf    Sat09    -  I        0:00.00 /usr/sbin/mgd -N
95289     0 23577   0  20   0  43792 lockf    Mon00    -  I        0:00.00 /usr/sbin/mgd -N
95711     0 23577   0  20   0  43792 lockf    Tue15    -  I        0:00.00 /usr/sbin/mgd -N
96162     0 23577   0  20   0  43792 lockf    23Dec23  -  I        0:00.00 /usr/sbin/mgd -N
96281     0 23577   0  20   0  43792 lockf    06:46    -  I        0:00.01 /usr/sbin/mgd -N
96375     0 23577   0  20   0  43792 lockf    25Dec23  -  I        0:00.00 /usr/sbin/mgd -N
96545     0 23577   0  20   0  43792 lockf    27Dec23  -  I        0:00.00 /usr/sbin/mgd -N
96976     0 23577   0  20   0  43792 lockf    Thu19    -  I        0:00.00 /usr/sbin/mgd -N
97461     0 23577   0  20   0  43792 lockf    Sat10    -  I        0:00.00 /usr/sbin/mgd -N
98045     0 23577   0  20   0  43792 lockf    Mon01    -  I        0:00.00 /usr/sbin/mgd -N
98507     0 23577   0  20   0  43792 lockf    Tue16    -  I        0:00.00 /usr/sbin/mgd -N
98915     0 23577   0  20   0  43792 lockf    23Dec23  -  I        0:00.00 /usr/sbin/mgd -N
99033     0 23577   0  20   0  43792 lockf    07:51    -  I        0:00.01 /usr/sbin/mgd -N
99129     0 23577   0  20   0  43792 lockf    25Dec23  -  I        0:00.00 /usr/sbin/mgd -N
99209     0     1   0  42   0  48140 lockf    22Dec23  -  Is       0:00.25 mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (homer) (mgd)
99247     0 23577   0  20   0  43792 lockf    22Dec23  -  I        0:00.00 /usr/sbin/mgd -N
99342     0 23577   0  20   0  43792 lockf    27Dec23  -  I        0:00.00 /usr/sbin/mgd -N
99768     0 23577   0  20   0  43792 lockf    Thu20    -  I        0:00.00 /usr/sbin/mgd -N

Perhaps the answer here is to issue kill from the root shell for all the mgd processes in state 'lockf', as reccomended on the Juniper support site for those other problems.

Also wondering what the root cause of it is. The earliest processes in this state are from Dec 22nd, which is the day after we enabled homer diff check timer on cumin1002, but failed to disable it on cumin1001. Nothing more than that co-incidence to go on but I suspect it may be related, possibly two sessions trying to enter configuration mode at once has conflicted and tied up the system.

Event Timeline

cmooney triaged this task as Medium priority.Jan 4 2024, 12:20 PM
cmooney created this task.

Change 987767 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/dns@master] Depool drmrs for cr2 maintenance

https://gerrit.wikimedia.org/r/987767

Change 987767 merged by Ayounsi:

[operations/dns@master] Depool drmrs for cr2 maintenance

https://gerrit.wikimedia.org/r/987767

Mentioned in SAL (#wikimedia-operations) [2024-01-04T13:02:29Z] <XioNoX> depool drmrs for router work - T354340

Change 987727 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/dns@master] Repool drmrs after cr2 maintenance

https://gerrit.wikimedia.org/r/987727

Change 987727 merged by Ayounsi:

[operations/dns@master] Repool drmrs after cr2 maintenance

https://gerrit.wikimedia.org/r/987727

Problem has resolved following device reboot.

It looked like killing the mgd processes in "lockf" state was working, but I made an error and killed the active mgd process as well (not showing 'lockf'), which meant we had no choice to reboot.