labstore1005 load following 2017-06-29 kernel upgrade
ActivePublic
Actions

Authored by bd808 on Jun 29 2017, 10:35 PM.

Tags

Referenced Files

	F8575458: labstore1005 load following 2017-06-29 kernel upgrade
	Jun 30 2017, 1:00 AM

	F8575448: labstore1005 load following 2017-06-29 kernal upgrade
	Jun 30 2017, 12:56 AM

	F8575223:
	Jun 29 2017, 11:34 PM

	F8575209:
	Jun 29 2017, 11:28 PM

	F8575158:
	Jun 29 2017, 11:14 PM

	F8575153:
	Jun 29 2017, 11:12 PM

	F8575115:
	Jun 29 2017, 10:51 PM

	F8575075:
	Jun 29 2017, 10:35 PM

Subscribers

None

	bd808$ while /bin/true; do w\|head -1; sleep 60; done
	21:48:22 up 6:26, 5 users, load average: 22.48, 21.74, 24.36
	21:49:22 up 6:27, 5 users, load average: 20.15, 21.54, 24.15
	21:50:22 up 6:28, 5 users, load average: 15.01, 19.54, 23.28
	21:51:22 up 6:29, 5 users, load average: 23.41, 21.89, 23.90
	21:52:22 up 6:30, 5 users, load average: 9.67, 18.17, 22.49
	21:53:22 up 6:31, 5 users, load average: 5.26, 15.47, 21.30
	21:54:22 up 6:32, 5 users, load average: 3.68, 13.05, 20.09
	21:55:22 up 6:33, 5 users, load average: 9.31, 12.55, 19.46
	21:56:22 up 6:34, 5 users, load average: 22.45, 16.02, 20.23
	21:57:22 up 6:35, 5 users, load average: 16.71, 15.48, 19.78
	21:58:22 up 6:36, 5 users, load average: 10.81, 14.23, 19.09
	21:59:22 up 6:37, 5 users, load average: 4.34, 11.72, 17.92
	22:00:22 up 6:38, 5 users, load average: 20.87, 13.74, 18.15
	22:01:22 up 6:39, 5 users, load average: 107.12, 40.87, 27.27
	22:02:22 up 6:40, 5 users, load average: 79.89, 46.28, 30.06
	22:03:22 up 6:41, 5 users, load average: 39.44, 40.80, 29.19
	22:04:22 up 6:42, 5 users, load average: 23.71, 35.94, 28.24
	22:05:22 up 6:43, 5 users, load average: 23.53, 33.20, 27.75
	22:06:22 up 6:44, 5 users, load average: 23.09, 31.63, 27.57
	22:07:22 up 6:45, 5 users, load average: 13.49, 27.30, 26.33
	22:08:22 up 6:46, 5 users, load average: 17.68, 25.54, 25.76
	22:09:22 up 6:47, 5 users, load average: 18.33, 24.19, 25.28
	22:10:23 up 6:48, 5 users, load average: 33.64, 26.41, 25.92
	22:11:23 up 6:49, 5 users, load average: 42.27, 31.18, 27.65
	22:12:23 up 6:50, 5 users, load average: 28.28, 29.27, 27.22
	22:13:23 up 6:51, 5 users, load average: 23.89, 28.23, 27.01
	22:14:23 up 6:52, 6 users, load average: 19.23, 26.27, 26.42
	22:15:23 up 6:53, 6 users, load average: 29.90, 27.27, 26.71
	# <chasemp> !log set cfq scheduler on labstore1005
	22:16:23 up 6:54, 6 users, load average: 39.06, 30.90, 28.03
	22:17:23 up 6:55, 6 users, load average: 29.66, 29.78, 27.83
	22:18:23 up 6:56, 6 users, load average: 28.39, 29.13, 27.72
	22:19:23 up 6:57, 6 users, load average: 19.96, 26.63, 26.95
	22:20:23 up 6:58, 6 users, load average: 17.86, 24.48, 26.17
	22:21:23 up 6:59, 6 users, load average: 31.95, 27.74, 27.21
	22:22:23 up 7:00, 6 users, load average: 24.14, 26.34, 26.77
	22:23:23 up 7:01, 6 users, load average: 18.88, 24.37, 26.05
	22:24:23 up 7:02, 6 users, load average: 24.18, 24.08, 25.80
	22:25:23 up 7:03, 6 users, load average: 40.34, 29.36, 27.55
	22:26:23 up 7:04, 6 users, load average: 36.21, 30.58, 28.11
	22:27:23 up 7:05, 6 users, load average: 22.02, 27.42, 27.17
	22:28:23 up 7:06, 6 users, load average: 17.21, 24.87, 26.29
	22:29:23 up 7:07, 6 users, load average: 9.84, 21.39, 25.01
	22:30:23 up 7:08, 6 users, load average: 15.81, 20.20, 24.33
	22:31:23 up 7:09, 6 users, load average: 42.33, 27.60, 26.66
	22:32:23 up 7:10, 6 users, load average: 24.94, 25.29, 25.92
	22:33:23 up 7:11, 6 users, load average: 15.22, 22.62, 24.98
	22:34:23 up 7:12, 6 users, load average: 6.63, 18.73, 23.48
	22:35:23 up 7:13, 6 users, load average: 11.32, 17.40, 22.70
	22:36:23 up 7:14, 6 users, load average: 20.44, 19.36, 23.07
	22:37:23 up 7:15, 6 users, load average: 9.59, 16.41, 21.82
	22:38:23 up 7:16, 6 users, load average: 5.33, 13.94, 20.63
	22:39:23 up 7:17, 6 users, load average: 4.71, 12.06, 19.56
	22:40:23 up 7:18, 6 users, load average: 18.87, 14.60, 19.95
	22:41:23 up 7:19, 6 users, load average: 18.17, 15.34, 19.88
	22:42:23 up 7:20, 6 users, load average: 17.02, 15.53, 19.66
	22:43:23 up 7:21, 6 users, load average: 9.19, 13.71, 18.78
	22:44:23 up 7:22, 6 users, load average: 7.81, 12.37, 18.00
	22:45:23 up 7:23, 6 users, load average: 15.55, 13.35, 17.96
	22:46:23 up 7:24, 6 users, load average: 29.23, 17.72, 19.18
	22:47:23 up 7:25, 6 users, load average: 22.96, 18.22, 19.27
	22:48:24 up 7:26, 6 users, load average: 18.69, 17.75, 19.04
	22:49:24 up 7:27, 6 users, load average: 17.10, 17.72, 18.96
	22:50:24 up 7:28, 6 users, load average: 12.75, 16.20, 18.35
	22:51:24 up 7:29, 6 users, load average: 18.27, 17.72, 18.77
	22:52:24 up 7:30, 6 users, load average: 10.41, 15.61, 17.98
	22:53:24 up 7:31, 6 users, load average: 6.43, 13.67, 17.18
	22:54:24 up 7:32, 6 users, load average: 4.77, 11.82, 16.31
	22:55:24 up 7:33, 6 users, load average: 12.29, 12.35, 16.20
	22:56:24 up 7:34, 6 users, load average: 19.81, 14.69, 16.77
	22:57:24 up 7:35, 6 users, load average: 15.23, 14.24, 16.48
	22:58:24 up 7:36, 6 users, load average: 7.96, 12.50, 15.75
	22:59:24 up 7:37, 6 users, load average: 4.49, 10.58, 14.88
	# Re-enabled Kubernetes scheduler
	23:00:24 up 7:38, 6 users, load average: 30.33, 15.23, 16.12
	23:01:24 up 7:39, 6 users, load average: 77.63, 32.98, 22.25
	23:02:24 up 7:40, 6 users, load average: 49.11, 33.68, 23.22
	23:03:24 up 7:41, 6 users, load average: 31.96, 31.74, 23.21
	23:04:24 up 7:42, 6 users, load average: 23.84, 29.24, 22.87
	23:05:24 up 7:43, 6 users, load average: 25.82, 28.46, 22.99
	23:06:24 up 7:44, 6 users, load average: 26.25, 28.69, 23.44
	23:07:24 up 7:45, 6 users, load average: 14.66, 24.97, 22.49
	23:08:24 up 7:46, 6 users, load average: 13.73, 22.73, 21.87
	23:09:24 up 7:47, 6 users, load average: 9.77, 20.01, 20.99
	23:10:24 up 7:48, 6 users, load average: 25.70, 21.49, 21.37
	23:11:24 up 7:49, 6 users, load average: 24.64, 22.67, 21.83
	23:12:24 up 7:50, 6 users, load average: 16.08, 20.23, 21.02
	23:13:24 up 7:51, 6 users, load average: 15.49, 19.19, 20.61
	23:14:24 up 7:52, 6 users, load average: 13.61, 18.20, 20.19
	23:15:24 up 7:53, 6 users, load average: 22.57, 19.67, 20.55
	23:16:24 up 7:54, 6 users, load average: 35.45, 23.99, 21.99
	23:17:24 up 7:55, 6 users, load average: 29.25, 23.99, 22.10
	23:18:24 up 7:56, 6 users, load average: 31.99, 25.49, 22.72
	23:19:24 up 7:57, 6 users, load average: 39.21, 28.43, 23.89
	23:20:24 up 7:58, 6 users, load average: 36.44, 29.37, 24.49
	23:21:24 up 7:59, 6 users, load average: 43.42, 32.92, 26.03
	23:22:24 up 8:00, 6 users, load average: 43.83, 35.04, 27.19
	23:23:24 up 8:01, 6 users, load average: 30.10, 32.89, 26.96
	23:24:24 up 8:02, 6 users, load average: 35.34, 33.66, 27.58
	23:25:24 up 8:03, 6 users, load average: 53.45, 38.55, 29.60
	23:26:25 up 8:04, 6 users, load average: 42.10, 38.54, 30.19
	23:27:25 up 8:05, 6 users, load average: 27.22, 34.98, 29.49
	23:28:25 up 8:06, 6 users, load average: 19.43, 31.20, 28.52
	23:29:25 up 8:07, 6 users, load average: 16.17, 28.22, 27.67
	23:30:25 up 8:08, 6 users, load average: 34.74, 30.24, 28.34
	23:31:25 up 8:09, 6 users, load average: 55.58, 37.68, 31.05
	23:32:25 up 8:10, 6 users, load average: 38.48, 36.08, 30.92
	23:33:25 up 8:11, 6 users, load average: 30.98, 34.18, 30.58
	23:34:25 up 8:12, 6 users, load average: 24.63, 31.89, 30.03
	23:36:25 up 8:14, 6 users, load average: 34.26, 33.03, 30.64
	23:37:25 up 8:15, 6 users, load average: 24.90, 30.52, 29.93
	23:38:25 up 8:16, 6 users, load average: 21.08, 28.21, 29.16
	23:39:25 up 8:17, 6 users, load average: 15.94, 25.44, 28.14
	23:40:25 up 8:18, 6 users, load average: 23.06, 25.17, 27.85
	23:41:25 up 8:19, 6 users, load average: 25.25, 26.07, 28.03
	23:42:25 up 8:20, 6 users, load average: 20.00, 24.12, 27.22
	23:43:25 up 8:21, 6 users, load average: 15.96, 22.19, 26.36
	23:44:25 up 8:22, 6 users, load average: 23.61, 23.32, 26.49
	23:45:25 up 8:23, 6 users, load average: 30.87, 24.97, 26.83
	23:46:25 up 8:24, 6 users, load average: 37.74, 28.79, 28.07
	23:47:25 up 8:25, 6 users, load average: 29.97, 27.89, 27.79
	23:48:25 up 8:26, 6 users, load average: 29.88, 28.12, 27.86
	23:49:25 up 8:27, 6 users, load average: 31.49, 28.37, 27.93
	23:50:25 up 8:28, 6 users, load average: 24.30, 27.03, 27.51
	23:51:25 up 8:29, 6 users, load average: 20.32, 25.68, 27.03
	23:52:25 up 8:30, 6 users, load average: 17.82, 23.77, 26.28
	23:53:25 up 8:31, 6 users, load average: 15.61, 22.18, 25.58
	23:54:25 up 8:32, 6 users, load average: 14.75, 20.46, 24.76
	23:55:25 up 8:33, 6 users, load average: 30.94, 23.56, 25.52
	23:56:25 up 8:34, 6 users, load average: 37.85, 27.34, 26.73
	23:57:25 up 8:35, 6 users, load average: 31.53, 27.32, 26.75
	23:58:25 up 8:36, 6 users, load average: 21.11, 25.36, 26.13
	23:59:25 up 8:37, 5 users, load average: 11.26, 21.81, 24.86
	00:00:25 up 8:38, 5 users, load average: 30.35, 24.46, 25.53
	00:01:25 up 8:39, 5 users, load average: 128.66, 53.90, 35.60
	00:02:26 up 8:40, 5 users, load average: 118.68, 65.93, 40.96
	00:03:26 up 8:41, 5 users, load average: 68.00, 61.41, 41.01
	00:04:26 up 8:42, 5 users, load average: 48.45, 57.09, 40.80
	00:05:26 up 8:43, 5 users, load average: 30.11, 50.17, 39.43
	00:06:26 up 8:44, 5 users, load average: 37.95, 48.86, 39.66
	00:07:26 up 8:45, 5 users, load average: 25.56, 43.11, 38.25
	00:08:26 up 8:46, 5 users, load average: 17.92, 38.16, 36.88
	00:09:26 up 8:47, 4 users, load average: 11.29, 32.67, 35.07
	00:10:26 up 8:48, 4 users, load average: 32.73, 33.91, 35.29
	00:11:26 up 8:49, 4 users, load average: 35.13, 34.68, 35.48
	00:12:26 up 8:50, 4 users, load average: 31.50, 33.41, 34.98
	00:13:26 up 8:51, 4 users, load average: 26.91, 31.79, 34.33
	00:14:26 up 8:52, 4 users, load average: 29.68, 31.24, 33.95
	00:15:26 up 8:53, 4 users, load average: 33.28, 32.27, 34.15
	00:16:26 up 8:54, 4 users, load average: 42.20, 34.76, 34.88
	00:17:26 up 8:55, 4 users, load average: 26.46, 31.80, 33.86
	00:18:26 up 8:56, 4 users, load average: 24.92, 30.23, 33.18
	00:19:26 up 8:57, 4 users, load average: 21.84, 28.31, 32.33
	00:20:26 up 8:58, 4 users, load average: 27.89, 28.66, 32.19
	00:21:26 up 8:59, 4 users, load average: 33.64, 30.41, 32.58
	00:22:26 up 9:00, 4 users, load average: 30.08, 30.01, 32.31
	00:23:26 up 9:01, 4 users, load average: 23.74, 28.01, 31.47
	00:24:26 up 9:02, 4 users, load average: 51.83, 35.55, 33.86
	00:25:26 up 9:03, 4 users, load average: 92.31, 48.55, 38.34
	00:26:26 up 9:04, 4 users, load average: 124.97, 67.08, 45.42
	00:27:26 up 9:05, 4 users, load average: 126.32, 76.00, 49.73
	00:28:26 up 9:06, 4 users, load average: 165.14, 96.48, 58.43
	00:29:26 up 9:07, 4 users, load average: 132.22, 100.43, 62.23
	00:30:26 up 9:08, 4 users, load average: 98.44, 96.11, 63.12
	00:31:26 up 9:09, 4 users, load average: 113.44, 99.98, 66.47
	00:32:26 up 9:10, 4 users, load average: 91.29, 96.83, 67.53
	00:33:26 up 9:11, 4 users, load average: 67.54, 88.47, 66.46
	00:34:26 up 9:12, 4 users, load average: 90.20, 91.37, 68.85
	00:35:26 up 9:13, 4 users, load average: 66.07, 84.52, 67.92
	00:36:26 up 9:14, 4 users, load average: 65.21, 81.17, 67.82
	00:37:27 up 9:15, 4 users, load average: 37.02, 70.30, 64.92
	00:38:27 up 9:16, 4 users, load average: 23.21, 60.12, 61.75
	00:39:27 up 9:17, 4 users, load average: 14.69, 50.88, 58.46
	00:40:27 up 9:18, 4 users, load average: 18.34, 44.83, 55.88
	00:41:27 up 9:19, 4 users, load average: 24.19, 42.21, 54.31
	00:42:27 up 9:20, 4 users, load average: 19.45, 37.63, 51.98
	00:43:27 up 9:21, 4 users, load average: 39.88, 40.26, 51.99
	00:44:27 up 9:22, 4 users, load average: 35.88, 38.82, 50.74
	00:45:27 up 9:23, 4 users, load average: 30.76, 36.33, 49.12
	00:46:27 up 9:24, 4 users, load average: 50.64, 41.56, 50.14
	00:47:27 up 9:25, 4 users, load average: 53.48, 44.15, 50.50
	00:48:27 up 9:26, 4 users, load average: 50.56, 44.74, 50.29
	00:49:27 up 9:27, 4 users, load average: 35.02, 41.29, 48.75
	00:50:27 up 9:28, 4 users, load average: 38.36, 40.51, 47.98
	00:51:27 up 9:29, 4 users, load average: 46.91, 42.62, 48.25
	00:52:27 up 9:30, 4 users, load average: 28.36, 38.41, 46.47
	00:53:27 up 9:31, 4 users, load average: 20.75, 34.56, 44.65
	00:54:27 up 9:32, 4 users, load average: 29.30, 34.09, 43.83
	00:55:27 up 9:33, 4 users, load average: 41.25, 36.48, 44.02
	00:56:27 up 9:34, 4 users, load average: 36.59, 36.44, 43.56
	00:57:27 up 9:35, 4 users, load average: 28.67, 34.31, 42.39
	00:58:27 up 9:36, 4 users, load average: 36.35, 35.43, 42.26
	00:59:27 up 9:37, 4 users, load average: 39.51, 36.75, 42.30
	01:00:27 up 9:38, 4 users, load average: 53.77, 39.30, 42.72

Event Timeline

bd808 created this paste.Jun 29 2017, 10:35 PM

bd808 edited the content of this paste. (Show Details)Jun 29 2017, 10:51 PM

bd808 updated the paste's language from autodetect to shell.Jun 29 2017, 11:12 PM

bd808 edited the content of this paste. (Show Details)

bd808 edited the content of this paste. (Show Details)Jun 29 2017, 11:28 PM

bd808 edited the content of this paste. (Show Details)Jun 29 2017, 11:34 PM

bd808 changed the title of this paste from untitled to labstore1005 load following 2017-06-29 kernal upgrade.Jun 30 2017, 12:56 AM

bd808 edited the content of this paste. (Show Details)

bd808 added projects: Cloud-Services, Toolforge.

bd808 changed the title of this paste from labstore1005 load following 2017-06-29 kernal upgrade to labstore1005 load following 2017-06-29 kernel upgrade.Jun 30 2017, 1:00 AM

bd808 edited the content of this paste. (Show Details)

https://graphite.wikimedia.org/render/?width=873&height=491&_salt=1498766901.038&target=servers.labstore1005.loadavg.15&target=servers.labstore1004.loadavg.15&from=-1d

Back on labstore1004 with the older 4.4.2-3+wmf8 kernel:

labstore1004:~
bd808$ while true; do w|head -1; sleep 60; done
 01:28:03 up  3:45,  8 users,  load average: 9.00, 5.18, 1.99
 01:29:03 up  3:46,  8 users,  load average: 5.47, 4.95, 2.11
 01:30:03 up  3:47,  8 users,  load average: 2.47, 4.19, 2.03
 01:31:03 up  3:48,  8 users,  load average: 2.15, 3.82, 2.03
 01:32:03 up  3:49,  5 users,  load average: 0.96, 3.18, 1.92
 01:33:03 up  3:50,  4 users,  load average: 0.86, 2.73, 1.85
 01:34:03 up  3:51,  4 users,  load average: 0.67, 2.34, 1.77
 01:35:03 up  3:52,  4 users,  load average: 0.50, 1.97, 1.68
 01:36:03 up  3:53,  4 users,  load average: 0.63, 1.75, 1.62
 01:37:03 up  3:54,  4 users,  load average: 0.33, 1.46, 1.52
 01:38:03 up  3:55,  4 users,  load average: 0.73, 1.35, 1.48
 01:39:03 up  3:56,  4 users,  load average: 0.27, 1.10, 1.38
 01:40:03 up  3:57,  4 users,  load average: 0.36, 0.96, 1.32
 01:41:03 up  3:58,  4 users,  load average: 0.28, 0.83, 1.25
 01:42:03 up  3:59,  4 users,  load average: 0.40, 0.77, 1.20
 01:43:03 up  4:00,  4 users,  load average: 0.47, 0.73, 1.16
 01:44:03 up  4:01,  4 users,  load average: 0.71, 0.76, 1.14
 01:45:03 up  4:02,  4 users,  load average: 0.38, 0.66, 1.08
 01:46:03 up  4:03,  4 users,  load average: 0.48, 0.62, 1.04
 01:47:03 up  4:04,  4 users,  load average: 0.46, 0.59, 1.00
 01:48:03 up  4:05,  4 users,  load average: 0.24, 0.50, 0.94
 01:49:03 up  4:06,  4 users,  load average: 0.23, 0.47, 0.90
 01:50:03 up  4:07,  4 users,  load average: 0.45, 0.48, 0.88
 01:51:03 up  4:08,  4 users,  load average: 0.44, 0.48, 0.85
 01:52:03 up  4:09,  4 users,  load average: 0.46, 0.48, 0.83
 01:53:03 up  4:10,  4 users,  load average: 0.57, 0.53, 0.82
 01:54:03 up  4:11,  4 users,  load average: 0.43, 0.51, 0.80
 01:55:03 up  4:12,  4 users,  load average: 0.64, 0.57, 0.80
 01:56:03 up  4:13,  4 users,  load average: 0.56, 0.57, 0.79
 01:57:03 up  4:14,  4 users,  load average: 0.43, 0.54, 0.77
 01:58:03 up  4:15,  4 users,  load average: 0.44, 0.53, 0.75
 01:59:03 up  4:16,  4 users,  load average: 0.39, 0.49, 0.72
 02:00:03 up  4:17,  4 users,  load average: 0.37, 0.46, 0.70
 02:01:03 up  4:18,  4 users,  load average: 1.43, 0.76, 0.79
 02:02:03 up  4:19,  4 users,  load average: 1.27, 0.82, 0.81
 02:03:03 up  4:20,  4 users,  load average: 0.90, 0.80, 0.80
 02:04:03 up  4:21,  4 users,  load average: 0.72, 0.76, 0.79
 02:05:03 up  4:22,  4 users,  load average: 0.86, 0.79, 0.80
 02:06:03 up  4:23,  4 users,  load average: 0.78, 0.78, 0.79
 02:07:03 up  4:24,  4 users,  load average: 0.49, 0.70, 0.76
 02:08:03 up  4:25,  4 users,  load average: 0.52, 0.69, 0.76
 02:09:03 up  4:26,  4 users,  load average: 0.47, 0.64, 0.73
 02:10:03 up  4:27,  4 users,  load average: 0.33, 0.57, 0.70
 02:11:03 up  4:28,  4 users,  load average: 0.56, 0.57, 0.69
 02:12:03 up  4:29,  4 users,  load average: 0.74, 0.61, 0.70
 02:13:03 up  4:30,  4 users,  load average: 0.47, 0.56, 0.67
 02:14:04 up  4:31,  4 users,  load average: 0.45, 0.54, 0.66
 02:15:04 up  4:32,  4 users,  load average: 0.63, 0.57, 0.66
 02:16:04 up  4:33,  4 users,  load average: 1.07, 0.68, 0.69
 02:17:04 up  4:34,  4 users,  load average: 0.74, 0.66, 0.69
 02:18:04 up  4:35,  4 users,  load average: 0.65, 0.65, 0.68
 02:19:04 up  4:36,  4 users,  load average: 0.82, 0.70, 0.69
 02:20:04 up  4:37,  4 users,  load average: 0.68, 0.69, 0.69
 02:21:04 up  4:38,  4 users,  load average: 0.76, 0.71, 0.70
 02:22:04 up  4:39,  4 users,  load average: 0.45, 0.65, 0.68
 02:23:04 up  4:40,  4 users,  load average: 0.49, 0.62, 0.67
 02:24:04 up  4:41,  4 users,  load average: 0.52, 0.61, 0.66
 02:25:04 up  4:42,  4 users,  load average: 0.24, 0.52, 0.62
 02:26:04 up  4:43,  4 users,  load average: 0.41, 0.52, 0.62

bd808 mentioned this in T169289: Tool Labs 2017-06-29 Labstore100[45] kernel upgrade issues.Jun 30 2017, 2:38 AM

labstore1005 load following 2017-06-29 kernel upgradeActivePublicActions

Event Timeline

labstore1005 load following 2017-06-29 kernel upgrade
ActivePublic
Actions