Page MenuHomePhabricator

webservice restart with k8s backend does not apply CPU or memory arguments
Open, Needs TriagePublicBUG REPORT

Description

Steps to Reproduce:

  1. Start a webservice with webservice --backend=kubernetes start
  2. Use kubectl get pods and kubectl describe pod <pod id> to view the memory and CPU requests.
  3. Attempt to change the memory and CPU request values with webservice -m 512Mi -c 500m
  4. Use kubectl get pods and kubectl describe pod <pod id> to view the new memory and CPU requests.

Actual Results:

  • The CPU and memory requests do not change after using webservice restart.

Expected Results:

  • The restarted webservice has the new CPU and memory requests applied OR
  • webservice gives an error.

Workaround:

  1. Stop the webservice using webservice stop
  2. Start the webservice using webservice --backend=kubernetes -m 512Mi -c 500m start

1$ webservice --backend=kubernetes start
2Starting webservice.....
3$ kubectl get pods
4NAME READY STATUS RESTARTS AGE
5anticompositetest-7fb9869f5c-rdghd 1/1 Running 0 63s
6$ kubectl describe pod anticompositetest-7fb9869f5c-rdghd
7Name: anticompositetest-7fb9869f5c-rdghd
8Namespace: tool-anticompositetest
9Priority: 0
10Node: tools-k8s-worker-46/172.16.1.80
11Start Time: Mon, 15 Mar 2021 18:11:06 +0000
12Labels: name=anticompositetest
13 pod-template-hash=7fb9869f5c
14 toolforge=tool
15 tools.wmflabs.org/webservice=true
16 tools.wmflabs.org/webservice-version=1
17Annotations: cni.projectcalico.org/podIP: 192.168.46.208/32
18 cni.projectcalico.org/podIPs: 192.168.46.208/32
19 kubernetes.io/limit-ranger: LimitRanger plugin set: cpu, memory request for container webservice; cpu, memory limit for container webservice
20 kubernetes.io/psp: tool-anticompositetest-psp
21 podpreset.admission.kubernetes.io/podpreset-mount-toolforge-vols: 77696750
22 seccomp.security.alpha.kubernetes.io/pod: runtime/default
23Status: Running
24IP: 192.168.46.208
25IPs:
26 IP: 192.168.46.208
27Controlled By: ReplicaSet/anticompositetest-7fb9869f5c
28Containers:
29 webservice:
30 Container ID: docker://13468fbc14e5907b4f39cadfefa6cee0ac67c5cd7fa58be04bc24c6d3ce86104
31 Image: docker-registry.tools.wmflabs.org/toolforge-php73-sssd-web:latest
32 Image ID: docker-pullable://docker-registry.tools.wmflabs.org/toolforge-php73-sssd-web@sha256:aa92d3a0e5c7b0dc6786248dabd50c089c453e94b7e088fff8b5f8c115b8e25e
33 Port: 8000/TCP
34 Host Port: 0/TCP
35 Command:
36 /usr/bin/webservice-runner
37 --type
38 lighttpd
39 --port
40 8000
41 State: Running
42 Started: Mon, 15 Mar 2021 18:11:16 +0000
43 Ready: True
44 Restart Count: 0
45 Limits:
46 cpu: 500m
47 memory: 512Mi
48 Requests:
49 cpu: 150m
50 memory: 256Mi
51 Environment:
52 HOME: /data/project/anticompositetest
53 Mounts:
54 /data/project from home (rw)
55 /data/scratch from scratch (rw)
56 /etc/ldap.conf from etcldap-conf (ro)
57 /etc/ldap.yaml from etcldap-yaml (ro)
58 /etc/novaobserver.yaml from etcnovaobserver-yaml (ro)
59 /etc/wmcs-project from wmcs-project (ro)
60 /mnt/nfs/dumps-labstore1006.wikimedia.org from dumpsrc2 (ro)
61 /mnt/nfs/dumps-labstore1007.wikimedia.org from dumpsrc1 (ro)
62 /public/dumps from dumps (ro)
63 /var/lib/sss/pipes from sssd-pipes (rw)
64 /var/run/secrets/kubernetes.io/serviceaccount from default-token-gwcjv (ro)
65Conditions:
66 Type Status
67 Initialized True
68 Ready True
69 ContainersReady True
70 PodScheduled True
71Volumes:
72 dumps:
73 Type: HostPath (bare host directory volume)
74 Path: /public/dumps
75 HostPathType: Directory
76 dumpsrc1:
77 Type: HostPath (bare host directory volume)
78 Path: /mnt/nfs/dumps-labstore1007.wikimedia.org
79 HostPathType: Directory
80 dumpsrc2:
81 Type: HostPath (bare host directory volume)
82 Path: /mnt/nfs/dumps-labstore1006.wikimedia.org
83 HostPathType: Directory
84 home:
85 Type: HostPath (bare host directory volume)
86 Path: /data/project
87 HostPathType: Directory
88 wmcs-project:
89 Type: HostPath (bare host directory volume)
90 Path: /etc/wmcs-project
91 HostPathType: File
92 scratch:
93 Type: HostPath (bare host directory volume)
94 Path: /data/scratch
95 HostPathType: Directory
96 etcldap-conf:
97 Type: HostPath (bare host directory volume)
98 Path: /etc/ldap.conf
99 HostPathType: File
100 etcldap-yaml:
101 Type: HostPath (bare host directory volume)
102 Path: /etc/ldap.yaml
103 HostPathType: File
104 etcnovaobserver-yaml:
105 Type: HostPath (bare host directory volume)
106 Path: /etc/novaobserver.yaml
107 HostPathType: File
108 sssd-pipes:
109 Type: HostPath (bare host directory volume)
110 Path: /var/lib/sss/pipes
111 HostPathType: Directory
112 default-token-gwcjv:
113 Type: Secret (a volume populated by a Secret)
114 SecretName: default-token-gwcjv
115 Optional: false
116QoS Class: Burstable
117Node-Selectors: <none>
118Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
119 node.kubernetes.io/unreachable:NoExecute for 300s
120Events:
121 Type Reason Age From Message
122 ---- ------ ---- ---- -------
123 Normal Scheduled 71s default-scheduler Successfully assigned tool-anticompositetest/anticompositetest-7fb9869f5c-rdghd to tools-k8s-worker-46
124 Normal Pulling 61s kubelet, tools-k8s-worker-46 Pulling image "docker-registry.tools.wmflabs.org/toolforge-php73-sssd-web:latest"
125 Normal Pulled 61s kubelet, tools-k8s-worker-46 Successfully pulled image "docker-registry.tools.wmflabs.org/toolforge-php73-sssd-web:latest"
126 Normal Created 61s kubelet, tools-k8s-worker-46 Created container webservice
127 Normal Started 61s kubelet, tools-k8s-worker-46 Started container webservice
128$ webservice -m 512Mi -c 500m restart
129Restarting...
130$ kubectl get pods
131NAME READY STATUS RESTARTS AGE
132anticompositetest-7fb9869f5c-fw6ws 1/1 Running 0 61s
133$ kubectl describe pod anticompositetest-7fb9869f5c-fw6ws
134Name: anticompositetest-7fb9869f5c-fw6ws
135Namespace: tool-anticompositetest
136Priority: 0
137Node: tools-k8s-worker-53/172.16.1.128
138Start Time: Mon, 15 Mar 2021 18:14:06 +0000
139Labels: name=anticompositetest
140 pod-template-hash=7fb9869f5c
141 toolforge=tool
142 tools.wmflabs.org/webservice=true
143 tools.wmflabs.org/webservice-version=1
144Annotations: cni.projectcalico.org/podIP: 192.168.25.164/32
145 cni.projectcalico.org/podIPs: 192.168.25.164/32
146 kubernetes.io/limit-ranger: LimitRanger plugin set: cpu, memory request for container webservice; cpu, memory limit for container webservice
147 kubernetes.io/psp: tool-anticompositetest-psp
148 podpreset.admission.kubernetes.io/podpreset-mount-toolforge-vols: 77696750
149 seccomp.security.alpha.kubernetes.io/pod: runtime/default
150Status: Running
151IP: 192.168.25.164
152IPs:
153 IP: 192.168.25.164
154Controlled By: ReplicaSet/anticompositetest-7fb9869f5c
155Containers:
156 webservice:
157 Container ID: docker://1a09587f33171077f26698131bb5035756869b6a5041dc761fad1bc56ac5f045
158 Image: docker-registry.tools.wmflabs.org/toolforge-php73-sssd-web:latest
159 Image ID: docker-pullable://docker-registry.tools.wmflabs.org/toolforge-php73-sssd-web@sha256:aa92d3a0e5c7b0dc6786248dabd50c089c453e94b7e088fff8b5f8c115b8e25e
160 Port: 8000/TCP
161 Host Port: 0/TCP
162 Command:
163 /usr/bin/webservice-runner
164 --type
165 lighttpd
166 --port
167 8000
168 State: Running
169 Started: Mon, 15 Mar 2021 18:14:08 +0000
170 Ready: True
171 Restart Count: 0
172 Limits:
173 cpu: 500m
174 memory: 512Mi
175 Requests:
176 cpu: 150m
177 memory: 256Mi
178 Environment:
179 HOME: /data/project/anticompositetest
180 Mounts:
181 /data/project from home (rw)
182 /data/scratch from scratch (rw)
183 /etc/ldap.conf from etcldap-conf (ro)
184 /etc/ldap.yaml from etcldap-yaml (ro)
185 /etc/novaobserver.yaml from etcnovaobserver-yaml (ro)
186 /etc/wmcs-project from wmcs-project (ro)
187 /mnt/nfs/dumps-labstore1006.wikimedia.org from dumpsrc2 (ro)
188 /mnt/nfs/dumps-labstore1007.wikimedia.org from dumpsrc1 (ro)
189 /public/dumps from dumps (ro)
190 /var/lib/sss/pipes from sssd-pipes (rw)
191 /var/run/secrets/kubernetes.io/serviceaccount from default-token-gwcjv (ro)
192Conditions:
193 Type Status
194 Initialized True
195 Ready True
196 ContainersReady True
197 PodScheduled True
198Volumes:
199 dumps:
200 Type: HostPath (bare host directory volume)
201 Path: /public/dumps
202 HostPathType: Directory
203 dumpsrc1:
204 Type: HostPath (bare host directory volume)
205 Path: /mnt/nfs/dumps-labstore1007.wikimedia.org
206 HostPathType: Directory
207 dumpsrc2:
208 Type: HostPath (bare host directory volume)
209 Path: /mnt/nfs/dumps-labstore1006.wikimedia.org
210 HostPathType: Directory
211 home:
212 Type: HostPath (bare host directory volume)
213 Path: /data/project
214 HostPathType: Directory
215 wmcs-project:
216 Type: HostPath (bare host directory volume)
217 Path: /etc/wmcs-project
218 HostPathType: File
219 scratch:
220 Type: HostPath (bare host directory volume)
221 Path: /data/scratch
222 HostPathType: Directory
223 etcldap-conf:
224 Type: HostPath (bare host directory volume)
225 Path: /etc/ldap.conf
226 HostPathType: File
227 etcldap-yaml:
228 Type: HostPath (bare host directory volume)
229 Path: /etc/ldap.yaml
230 HostPathType: File
231 etcnovaobserver-yaml:
232 Type: HostPath (bare host directory volume)
233 Path: /etc/novaobserver.yaml
234 HostPathType: File
235 sssd-pipes:
236 Type: HostPath (bare host directory volume)
237 Path: /var/lib/sss/pipes
238 HostPathType: Directory
239 default-token-gwcjv:
240 Type: Secret (a volume populated by a Secret)
241 SecretName: default-token-gwcjv
242 Optional: false
243QoS Class: Burstable
244Node-Selectors: <none>
245Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
246 node.kubernetes.io/unreachable:NoExecute for 300s
247Events:
248 Type Reason Age From Message
249 ---- ------ ---- ---- -------
250 Normal Scheduled 76s default-scheduler Successfully assigned tool-anticompositetest/anticompositetest-7fb9869f5c-fw6ws to tools-k8s-worker-53
251 Normal Pulling 75s kubelet, tools-k8s-worker-53 Pulling image "docker-registry.tools.wmflabs.org/toolforge-php73-sssd-web:latest"
252 Normal Pulled 75s kubelet, tools-k8s-worker-53 Successfully pulled image "docker-registry.tools.wmflabs.org/toolforge-php73-sssd-web:latest"
253 Normal Created 75s kubelet, tools-k8s-worker-53 Created container webservice
254 Normal Started 66s kubelet, tools-k8s-worker-53 Started container webservice
255$ webservice stop
256Stopping webservice
257$ webservice --backend=kubernetes -m 512Mi -c 500m start
258Starting webservice.......
259$ kubectl get pods
260NAME READY STATUS RESTARTS AGE
261anticompositetest-7f6f49bbc8-ckrnm 1/1 Running 0 94s
262$ kubectl describe pod anticompositetest-7f6f49bbc8-ckrnm
263Name: anticompositetest-7f6f49bbc8-ckrnm
264Namespace: tool-anticompositetest
265Priority: 0
266Node: tools-k8s-worker-46/172.16.1.80
267Start Time: Mon, 15 Mar 2021 18:19:46 +0000
268Labels: name=anticompositetest
269 pod-template-hash=7f6f49bbc8
270 toolforge=tool
271 tools.wmflabs.org/webservice=true
272 tools.wmflabs.org/webservice-version=1
273Annotations: cni.projectcalico.org/podIP: 192.168.46.239/32
274 cni.projectcalico.org/podIPs: 192.168.46.239/32
275 kubernetes.io/psp: tool-anticompositetest-psp
276 podpreset.admission.kubernetes.io/podpreset-mount-toolforge-vols: 77696750
277 seccomp.security.alpha.kubernetes.io/pod: runtime/default
278Status: Running
279IP: 192.168.46.239
280IPs:
281 IP: 192.168.46.239
282Controlled By: ReplicaSet/anticompositetest-7f6f49bbc8
283Containers:
284 webservice:
285 Container ID: docker://1842793a7c137881c0de2e72c39971fe55dfdcaba7655b8997d1590c4ee94248
286 Image: docker-registry.tools.wmflabs.org/toolforge-php73-sssd-web:latest
287 Image ID: docker-pullable://docker-registry.tools.wmflabs.org/toolforge-php73-sssd-web@sha256:aa92d3a0e5c7b0dc6786248dabd50c089c453e94b7e088fff8b5f8c115b8e25e
288 Port: 8000/TCP
289 Host Port: 0/TCP
290 Command:
291 /usr/bin/webservice-runner
292 --type
293 lighttpd
294 --port
295 8000
296 State: Running
297 Started: Mon, 15 Mar 2021 18:19:54 +0000
298 Ready: True
299 Restart Count: 0
300 Limits:
301 cpu: 500m
302 memory: 512Mi
303 Requests:
304 cpu: 250m
305 memory: 268435456
306 Environment:
307 HOME: /data/project/anticompositetest
308 Mounts:
309 /data/project from home (rw)
310 /data/scratch from scratch (rw)
311 /etc/ldap.conf from etcldap-conf (ro)
312 /etc/ldap.yaml from etcldap-yaml (ro)
313 /etc/novaobserver.yaml from etcnovaobserver-yaml (ro)
314 /etc/wmcs-project from wmcs-project (ro)
315 /mnt/nfs/dumps-labstore1006.wikimedia.org from dumpsrc2 (ro)
316 /mnt/nfs/dumps-labstore1007.wikimedia.org from dumpsrc1 (ro)
317 /public/dumps from dumps (ro)
318 /var/lib/sss/pipes from sssd-pipes (rw)
319 /var/run/secrets/kubernetes.io/serviceaccount from default-token-gwcjv (ro)
320Conditions:
321 Type Status
322 Initialized True
323 Ready True
324 ContainersReady True
325 PodScheduled True
326Volumes:
327 dumps:
328 Type: HostPath (bare host directory volume)
329 Path: /public/dumps
330 HostPathType: Directory
331 dumpsrc1:
332 Type: HostPath (bare host directory volume)
333 Path: /mnt/nfs/dumps-labstore1007.wikimedia.org
334 HostPathType: Directory
335 dumpsrc2:
336 Type: HostPath (bare host directory volume)
337 Path: /mnt/nfs/dumps-labstore1006.wikimedia.org
338 HostPathType: Directory
339 home:
340 Type: HostPath (bare host directory volume)
341 Path: /data/project
342 HostPathType: Directory
343 wmcs-project:
344 Type: HostPath (bare host directory volume)
345 Path: /etc/wmcs-project
346 HostPathType: File
347 scratch:
348 Type: HostPath (bare host directory volume)
349 Path: /data/scratch
350 HostPathType: Directory
351 etcldap-conf:
352 Type: HostPath (bare host directory volume)
353 Path: /etc/ldap.conf
354 HostPathType: File
355 etcldap-yaml:
356 Type: HostPath (bare host directory volume)
357 Path: /etc/ldap.yaml
358 HostPathType: File
359 etcnovaobserver-yaml:
360 Type: HostPath (bare host directory volume)
361 Path: /etc/novaobserver.yaml
362 HostPathType: File
363 sssd-pipes:
364 Type: HostPath (bare host directory volume)
365 Path: /var/lib/sss/pipes
366 HostPathType: Directory
367 default-token-gwcjv:
368 Type: Secret (a volume populated by a Secret)
369 SecretName: default-token-gwcjv
370 Optional: false
371QoS Class: Burstable
372Node-Selectors: <none>
373Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
374 node.kubernetes.io/unreachable:NoExecute for 300s
375Events:
376 Type Reason Age From Message
377 ---- ------ ---- ---- -------
378 Normal Scheduled 108s default-scheduler Successfully assigned tool-anticompositetest/anticompositetest-7f6f49bbc8-ckrnm to tools-k8s-worker-46
379 Normal Pulling 102s kubelet, tools-k8s-worker-46 Pulling image "docker-registry.tools.wmflabs.org/toolforge-php73-sssd-web:latest"
380 Normal Pulled 101s kubelet, tools-k8s-worker-46 Successfully pulled image "docker-registry.tools.wmflabs.org/toolforge-php73-sssd-web:latest"
381 Normal Created 101s kubelet, tools-k8s-worker-46 Created container webservice
382 Normal Started 101s kubelet, tools-k8s-worker-46 Started container webservice

Related Objects