Page MenuHomePhabricator
Create Task
Maniphest T176623

Some docker slave still have old containers using old images
Closed, ResolvedPublic
Actions

Description

These containers should probably be cleaned up, but we should also investigate why they are still hanging around?

addshore@integration-saltmaster:~$ sudo salt -v '*slave-docker*' cmd.run 'sudo docker rmi wmfreleng/operations-puppet:0.0.1 wmfreleng/operations-puppet:0.1.0'
Executing job with jid 20170925123621098148
-------------------------------------------

integration-slave-docker-1705.integration.eqiad.wmflabs:
    Error response from daemon: No such image: wmfreleng/operations-puppet:0.0.1
    Error response from daemon: No such image: wmfreleng/operations-puppet:0.1.0
integration-slave-docker-1004.integration.eqiad.wmflabs:
    Error response from daemon: No such image: wmfreleng/operations-puppet:0.0.1
    Error response from daemon: conflict: unable to remove repository reference "wmfreleng/operations-puppet:0.1.0" (must force) - container ceab580a3ea3 is using its referenced image 4566820b2150
integration-slave-docker-1001.integration.eqiad.wmflabs:
    Untagged: wmfreleng/operations-puppet:0.1.0
    Untagged: wmfreleng/operations-puppet@sha256:9870d9a8c2292741a1f29e3c702f15dc93ce870c7f81ea8cf14fada2ea0255b6
    Error response from daemon: conflict: unable to remove repository reference "wmfreleng/operations-puppet:0.0.1" (must force) - container ec602051f28e is using its referenced image de312a0380db
integration-slave-docker-1002.integration.eqiad.wmflabs:
    Error response from daemon: No such image: wmfreleng/operations-puppet:0.0.1
    Error response from daemon: conflict: unable to remove repository reference "wmfreleng/operations-puppet:0.1.0" (must force) - container c8d1b2fe4e19 is using its referenced image 4566820b2150
integration-slave-docker-1003.integration.eqiad.wmflabs:
    Untagged: wmfreleng/operations-puppet:0.1.0
    Untagged: wmfreleng/operations-puppet@sha256:9870d9a8c2292741a1f29e3c702f15dc93ce870c7f81ea8cf14fada2ea0255b6
    Deleted: sha256:4566820b2150d5eac35f504b016f0a2e3d61b36c5b9a65665ead23943dbb6e6f
    Deleted: sha256:ecc4166089d933ab8f964653a9ca2e548fff677bd67bb76cc9b80f77351bbb8a
    Deleted: sha256:bdc95489cd2d6e7c50bac5b77f67dc4d12c4e6e60d21a91d1ff21a7e43320212
    Deleted: sha256:5e7418c974018e07fb3816448f960e8cee70c1a5a1e9cf5089cea762f7e770af
    Deleted: sha256:8de6fc8dc301a1cbb0fd4e7adac6270b03593e912b0061b6be6b0c10ceeee4c1
    Error response from daemon: conflict: unable to remove repository reference "wmfreleng/operations-puppet:0.0.1" (must force) - container 022b681d6a72 is using its referenced image de312a0380db

And listing the containers

addshore@integration-saltmaster:~$ sudo salt -v '*slave-docker*' cmd.run 'sudo docker container ls -a |grep operations-puppet'
Executing job with jid 20170925123916273790
-------------------------------------------

integration-slave-docker-1004.integration.eqiad.wmflabs:
    b3b09f60f300        wmfreleng/operations-puppet:v2017.09.13.09.23   "/bin/sh -c '/bin/..."   3 days ago          Exited (0) 3 days ago                        gigantic_hypatia
    8276db21f222        wmfreleng/operations-puppet:v2017.09.13.09.23   "/bin/sh -c '/bin/..."   5 days ago          Exited (0) 5 days ago                        romantic_dubinsky
    82b40a382b35        wmfreleng/operations-puppet:0.1.0               "/bin/sh -c '/bin/..."   2 weeks ago         Exited (0) 2 weeks ago                       high_nobel
    bc807196b509        wmfreleng/operations-puppet:0.1.0               "/bin/sh -c '/bin/..."   2 weeks ago         Exited (0) 2 weeks ago                       modest_bell
    3c8ae5f1a3cf        wmfreleng/operations-puppet:0.1.0               "/bin/sh -c '/bin/..."   2 weeks ago         Exited (0) 2 weeks ago                       trusting_spence
    501f2db46e84        wmfreleng/operations-puppet:0.1.0               "/bin/sh -c '/bin/..."   2 weeks ago         Exited (0) 2 weeks ago                       focused_spence
    102c20ba7ae4        wmfreleng/operations-puppet:0.1.0               "/bin/sh -c '/bin/..."   2 weeks ago         Exited (0) 2 weeks ago                       desperate_albattani
    7f9c9bfe7a64        wmfreleng/operations-puppet:0.1.0               "/bin/sh -c '/bin/..."   2 weeks ago         Exited (0) 2 weeks ago                       focused_jepsen
    489469d7e38a        wmfreleng/operations-puppet:0.1.0               "/bin/sh -c '/bin/..."   3 weeks ago         Exited (0) 3 weeks ago                       goofy_elion
    b72dbb9c19b5        wmfreleng/operations-puppet:0.1.0               "/bin/sh -c '/bin/..."   3 weeks ago         Exited (0) 3 weeks ago                       dreamy_raman
    61ac622090af        wmfreleng/operations-puppet:0.1.0               "/bin/sh -c '/bin/..."   4 weeks ago         Exited (0) 4 weeks ago                       sharp_booth
    ceab580a3ea3        wmfreleng/operations-puppet:0.1.0               "/bin/sh -c '/bin/..."   4 weeks ago         Exited (0) 4 weeks ago                       infallible_kalam
    4d66aeb69e69        wmfreleng/operations-puppet:0.1.0               "/bin/sh -c '/bin/..."   5 weeks ago         Exited (0) 5 weeks ago                       elegant_murdock
    4404d33f1791        wmfreleng/operations-puppet:0.1.0               "/bin/sh -c '/bin/..."   5 weeks ago         Exited (0) 5 weeks ago                       stupefied_wilson
integration-slave-docker-1705.integration.eqiad.wmflabs:
integration-slave-docker-1003.integration.eqiad.wmflabs:
    8fd046a2c330        wmfreleng/operations-puppet:v2017.09.13.09.23   "/bin/sh -c '/bin/..."   5 days ago          Created                                       kickass_bhabha
    c6c8509102ae        wmfreleng/operations-puppet:v2017.09.13.09.23   "/bin/sh -c '/bin/..."   12 days ago         Exited (0) 12 days ago                        elated_payne
    c163e156742c        wmfreleng/operations-puppet:0.0.1               "/bin/sh -c '/bin/..."   7 weeks ago         Exited (0) 7 weeks ago                        silly_tesla
    022b681d6a72        wmfreleng/operations-puppet:0.0.1               "/bin/sh -c '/bin/..."   8 weeks ago         Exited (0) 8 weeks ago                        admiring_engelbart
    c650aca478e6        wmfreleng/operations-puppet:0.0.1               "/bin/sh -c '/bin/..."   2 months ago        Exited (0) 2 months ago                       ecstatic_curran
integration-slave-docker-1002.integration.eqiad.wmflabs:
    6086da38b8bd        wmfreleng/operations-puppet:v2017.09.13.09.23                               "/bin/sh -c '/bin/..."   3 days ago          Exited (0) 3 days ago                         lonely_gates
    80de8ac34830        wmfreleng/operations-puppet:v2017.09.13.09.23                               "/bin/sh -c '/bin/..."   5 days ago          Exited (0) 5 days ago                         elated_heisenberg
    ba6508839d0d        wmfreleng/operations-puppet:0.1.0                                           "/bin/sh -c '/bin/..."   12 days ago         Created                                       backstabbing_lamarr
    1cbcaf93dce5        wmfreleng/operations-puppet:0.1.0                                           "/bin/sh -c '/bin/..."   13 days ago         Exited (0) 13 days ago                        infallible_shaw
    c7b3fa0e54c2        wmfreleng/operations-puppet:0.1.0                                           "/bin/sh -c '/bin/..."   2 weeks ago         Exited (0) 2 weeks ago                        small_agnesi
    b0fe2c64ddbc        wmfreleng/operations-puppet:0.1.0                                           "/bin/sh -c '/bin/..."   2 weeks ago         Exited (0) 2 weeks ago                        cocky_mirzakhani
    c6251799edcc        wmfreleng/operations-puppet:0.1.0                                           "/bin/sh -c '/bin/..."   2 weeks ago         Exited (0) 2 weeks ago                        grave_swartz
    c8d1b2fe4e19        wmfreleng/operations-puppet:0.1.0                                           "/bin/sh -c '/bin/..."   2 weeks ago         Exited (0) 2 weeks ago                        peaceful_swirles
    8f8143cfdc24        wmfreleng/operations-puppet:0.1.0                                           "/bin/sh -c '/bin/..."   2 weeks ago         Exited (0) 2 weeks ago                        furious_pasteur
    b53bab81433d        wmfreleng/operations-puppet:0.1.0                                           "/bin/sh -c '/bin/..."   4 weeks ago         Exited (0) 4 weeks ago                        jovial_ride
    e01a12704dc3        wmfreleng/operations-puppet:0.1.0                                           "/bin/sh -c '/bin/..."   7 weeks ago         Exited (0) 7 weeks ago                        cocky_feynman
    3bb7e5b067c5        ci-staging-docker-registry01.wmflabs.org/contint/operations-puppet:v0.0.4   "/bin/sh -c '/bin/..."   2 months ago        Exited (0) 2 months ago                       lonely_torvalds
    264be3244f13        ci-staging-docker-registry01.wmflabs.org/contint/operations-puppet:v0.0.4   "/bin/sh -c '/bin/..."   2 months ago        Exited (0) 2 months ago                       angry_carson
    2463b5432714        ci-staging-docker-registry01.wmflabs.org/contint/operations-puppet:v0.0.4   "/bin/sh -c '/bin/..."   2 months ago        Exited (0) 2 months ago                       silly_euler
    2f3b1ac8b2d6        ci-staging-docker-registry01.wmflabs.org/contint/operations-puppet:v0.0.4   "/bin/sh -c '/bin/..."   2 months ago        Exited (0) 2 months ago                       boring_cray
    286a5257a285        ci-staging-docker-registry01.wmflabs.org/contint/operations-puppet:v0.0.4   "/bin/sh -c '/bin/..."   2 months ago        Exited (0) 2 months ago                       cranky_austin
    e21bff3b74e0        ci-staging-docker-registry01.wmflabs.org/contint/operations-puppet:v0.0.4   "/bin/sh -c '/bin/..."   2 months ago        Exited (0) 2 months ago                       silly_einstein
    e8f641ae4ea5        ci-staging-docker-registry01.wmflabs.org/contint/operations-puppet:v0.0.4   "/bin/sh -c '/bin/..."   2 months ago        Exited (0) 2 months ago                       cocky_roentgen
    00028939627c        ci-staging-docker-registry01.wmflabs.org/contint/operations-puppet:v0.0.3   "/bin/sh -c '/bin/..."   2 months ago        Exited (0) 2 months ago                       backstabbing_jang
    1142dbb8aa66        ci-staging-docker-registry01.wmflabs.org/contint/operations-puppet:v0.0.3   "/bin/sh -c '/bin/..."   3 months ago        Exited (0) 3 months ago                       berserk_hugle
    a6e66409ac9f        ci-staging-docker-registry01.wmflabs.org/contint/operations-puppet:v0.0.3   "/bin/sh -c '/bin/..."   3 months ago        Exited (0) 3 months ago                       dreamy_cray
integration-slave-docker-1001.integration.eqiad.wmflabs:
    1cd5930f219e        wmfreleng/operations-puppet:v2017.09.13.09.23                               "/bin/sh -c '/bin/..."   3 days ago          Exited (0) 3 days ago                         zen_feynman
    6a8a40bacb3a        wmfreleng/operations-puppet:0.0.1                                           "/bin/sh -c '/bin/..."   7 weeks ago         Exited (0) 7 weeks ago                        hungry_babbage
    2523d4d8abc6        wmfreleng/operations-puppet:0.0.1                                           "/bin/sh -c '/bin/..."   7 weeks ago         Exited (0) 7 weeks ago                        amazing_kalam
    57fdc673540c        wmfreleng/operations-puppet:0.0.1                                           "/bin/sh -c '/bin/..."   7 weeks ago         Exited (0) 7 weeks ago                        small_wozniak
    ced290461a89        wmfreleng/operations-puppet:0.0.1                                           "/bin/sh -c '/bin/..."   7 weeks ago         Exited (0) 7 weeks ago                        focused_austin
    de1fd75ca47c        wmfreleng/operations-puppet:0.0.1                                           "/bin/sh -c '/bin/..."   7 weeks ago         Exited (0) 7 weeks ago                        mad_nobel
    74a8b2efd1bb        wmfreleng/operations-puppet:0.0.1                                           "/bin/sh -c '/bin/..."   7 weeks ago         Exited (0) 7 weeks ago                        compassionate_shockley
    43df207338cf        wmfreleng/operations-puppet:0.0.1                                           "/bin/sh -c '/bin/..."   7 weeks ago         Exited (0) 7 weeks ago                        sick_ptolemy
    ec602051f28e        wmfreleng/operations-puppet:0.0.1                                           "/bin/sh -c '/bin/..."   7 weeks ago         Exited (0) 7 weeks ago                        prickly_bhabha
    b7e2f0f3f8bd        wmfreleng/operations-puppet:0.0.1                                           "/bin/sh -c '/bin/..."   7 weeks ago         Exited (0) 7 weeks ago                        tiny_austin
    cc133c7b2519        wmfreleng/operations-puppet:0.0.1                                           "/bin/sh -c '/bin/..."   8 weeks ago         Exited (0) 8 weeks ago                        zen_chandrasekhar
    b35388e994cd        wmfreleng/operations-puppet:0.0.1                                           "/bin/sh -c '/bin/..."   8 weeks ago         Exited (0) 8 weeks ago                        tiny_lamport
    5cb29c152eed        wmfreleng/operations-puppet:0.0.1                                           "/bin/sh -c '/bin/..."   8 weeks ago         Exited (0) 8 weeks ago                        gloomy_kirch
    65594e18dc5a        wmfreleng/operations-puppet:0.0.1                                           "/bin/sh -c '/bin/..."   8 weeks ago         Exited (0) 8 weeks ago                        sad_stallman
    56637917548a        wmfreleng/operations-puppet:0.0.1                                           "/bin/sh -c '/bin/..."   8 weeks ago         Exited (0) 8 weeks ago                        naughty_leavitt
    664f5276409a        wmfreleng/operations-puppet:0.0.1                                           "/bin/sh -c '/bin/..."   8 weeks ago         Exited (0) 8 weeks ago                        lonely_mclean
    961bb93db771        wmfreleng/operations-puppet:0.0.1                                           "/bin/sh -c '/bin/..."   2 months ago        Exited (0) 2 months ago                       silly_lamarr
    8984222a1e7b        wmfreleng/operations-puppet:0.0.1                                           "/bin/sh -c '/bin/..."   2 months ago        Exited (0) 2 months ago                       sad_brahmagupta
    39ccf77bad33        wmfreleng/operations-puppet:0.0.1                                           "/bin/sh -c '/bin/..."   2 months ago        Exited (0) 2 months ago                       grave_fermat
    96d5ee09aa82        wmfreleng/operations-puppet:0.0.1                                           "/bin/sh -c '/bin/..."   2 months ago        Created                                       sick_swanson
    6b353965f1cd        wmfreleng/operations-puppet:0.0.1                                           "/bin/sh -c '/bin/..."   2 months ago        Exited (0) 2 months ago                       mad_ramanujan
    9a2143d4a902        wmfreleng/operations-puppet:0.0.1                                           "/bin/sh -c '/bin/..."   2 months ago        Exited (0) 2 months ago                       sick_borg
    937d2eb7e9ff        ci-staging-docker-registry01.wmflabs.org/contint/operations-puppet:v0.0.4   "/bin/sh -c '/bin/..."   2 months ago        Exited (0) 2 months ago                       agitated_shirley
    2cbc8e5f8875        ci-staging-docker-registry01.wmflabs.org/contint/operations-puppet:v0.0.4   "/bin/sh -c '/bin/..."   2 months ago        Exited (0) 2 months ago                       thirsty_fermi
    c651ea2f92a4        ci-staging-docker-registry01.wmflabs.org/contint/operations-puppet:v0.0.4   "/bin/sh -c '/bin/..."   2 months ago        Exited (0) 2 months ago                       jolly_einstein
    3ab671acdea5        ci-staging-docker-registry01.wmflabs.org/contint/operations-puppet:v0.0.4   "/bin/sh -c '/bin/..."   2 months ago        Exited (0) 2 months ago                       sick_brahmagupta
    abb7d02b2233        ci-staging-docker-registry01.wmflabs.org/contint/operations-puppet:v0.0.3   "/bin/sh -c '/bin/..."   2 months ago        Exited (0) 2 months ago                       determined_sammet
    679f84cf86c6        ci-staging-docker-registry01.wmflabs.org/contint/operations-puppet:v0.0.3   "/bin/sh -c '/bin/..."   2 months ago        Exited (0) 2 months ago                       angry_wescoff
    70e0eebcb013        ci-staging-docker-registry01.wmflabs.org/contint/operations-puppet:v0.0.3   "/bin/sh -c '/bin/..."   2 months ago        Exited (0) 2 months ago                       drunk_mestorf
    d7c6a7dd0966        ci-staging-docker-registry01.wmflabs.org/contint/operations-puppet:v0.0.3   "/bin/sh -c '/bin/..."   3 months ago        Exited (0) 3 months ago                       gigantic_meitner
    a2c2d7c8097a        ci-staging-docker-registry01.wmflabs.org/contint/operations-puppet:v0.0.3   "/bin/sh -c '/bin/..."   3 months ago        Exited (0) 3 months ago                       pedantic_hopper
    165e5c31ad65        ci-staging-docker-registry01.wmflabs.org/contint/operations-puppet:v0.0.2   "/bin/bash"              3 months ago        Exited (0) 3 months ago                       admiring_elion
    6a910b99ece3        ci-staging-docker-registry01.wmflabs.org/contint/operations-puppet          "/bin/bash"              3 months ago        Exited (0) 3 months ago                       small_elion
    f77c089003a5        ci-staging-docker-registry01.wmflabs.org/contint/operations-puppet          "/bin/bash"              3 months ago        Exited (0) 3 months ago                       tiny_williams

Are these just left over from manual testing or is jenkins somehow leaving some of these behind?
Do we need a script or check to make sure this doesn't keep happening / mount up / become a problem?

Related Objects

Event Timeline

Addshore created this task.Sep 25 2017, 12:41 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 25 2017, 12:41 PM
Addshore added a comment.Sep 26 2017, 1:18 PM

When jobs are killed due to log execution time such as https://integration.wikimedia.org/ci/job/lintr-docker/24/console this can happen

addshore@integration-saltmaster:~$ sudo salt -v '*slave-docker*' cmd.run 'sudo docker container ls -a |grep lintr'
Executing job with jid 20170926131739720098
-------------------------------------------

integration-slave-docker-1004.integration.eqiad.wmflabs:
integration-slave-docker-1705.integration.eqiad.wmflabs:
integration-slave-docker-1002.integration.eqiad.wmflabs:
integration-slave-docker-1003.integration.eqiad.wmflabs:
integration-slave-docker-1001.integration.eqiad.wmflabs:
    5fde92afe692        wmfreleng/lintr:v2017.09.26.12.04                                           "/bin/sh -c '/bin/..."   6 minutes ago       Up 6 minutes                                  inspiring_mcclintock

    a5df1bb6813b        wmfreleng/lintr:v2017.09.26.12.04                                           "/bin/sh -c '/bin/..."   8 minutes ago       Up 7 minutes                                  sad_lamarr
    e3b4a7ee9c33        wmfreleng/lintr:v2017.09.26.12.04                                           "/bin/sh -c '/bin/..."   14 minutes ago      Up 14 minutes                                 nostalgic_hopper
Addshore added a comment.Sep 26 2017, 1:33 PM

It actually looks like these containers continue to run and use resources on the node after jenkins has killed the process that started them too.

  1  [||||||||||||||||||||||                                                                    22.2%]     Tasks: 69, 96 thr; 2 running
  2  [||||||||||||||||||||||||||||||||||||||||||||||||                                          48.4%]     Load average: 5.29 5.43 4.62
  Mem[||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||1925/2004MB]     Uptime: 96 days, 20:19:25
  Swp[||||||||||||||||||||||||||||||||||||||||||||||||||                                    263/510MB]

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
29804 nobody     20   0  181M 54252   912 D  8.8  2.6  0:55.75 /usr/lib/R/bin/exec/R --slave --no-restore --file=/lint.R --args
28294 nobody     20   0  185M 52388     0 D  5.1  2.6  2:09.14 /usr/lib/R/bin/exec/R --slave --no-restore --file=/lint.R --args
29411 nobody     20   0  183M 46000   980 D  4.7  2.2  1:35.37 /usr/lib/R/bin/exec/R --slave --no-restore --file=/lint.R --args

on timeouts the containers should probably forcibly be stopped?
Maybe the docker plugin handles this?

addshore@integration-slave-docker-1001:~$ sudo docker container ls
CONTAINER ID        IMAGE                               COMMAND                  CREATED             STATUS              PORTS               NAMES
5fde92afe692        wmfreleng/lintr:v2017.09.26.12.04   "/bin/sh -c '/bin/..."   19 minutes ago      Up 19 minutes                           inspiring_mcclintock
a5df1bb6813b        wmfreleng/lintr:v2017.09.26.12.04   "/bin/sh -c '/bin/..."   21 minutes ago      Up 21 minutes                           sad_lamarr
e3b4a7ee9c33        wmfreleng/lintr:v2017.09.26.12.04   "/bin/sh -c '/bin/..."   27 minutes ago      Up 27 minutes                           nostalgic_hopper

Once the container was stopped it actually seemed to automatically delete itself correctly (as it was created with --rm

addshore@integration-slave-docker-1001:~$ sudo docker container stop 5fde92afe692 a5df1bb6813b e3b4a7ee9c33
5fde92afe692
a5df1bb6813b
e3b4a7ee9c33
addshore@integration-slave-docker-1001:~$ sudo docker rm 5fde92afe692 a5df1bb6813b e3b4a7ee9c33
Error response from daemon: No such container: 5fde92afe692
Error response from daemon: No such container: a5df1bb6813b
Error response from daemon: No such container: e3b4a7ee9c33
Addshore mentioned this in T176747: When jenkins kills a build due to max execution time the docker containers stay running.Sep 26 2017, 1:42 PM

I have split the last 2 comments into T176747 as container continuing to run is a different issue to old containers not running but hanging around.

Addshore mentioned this in T177039: mediawiki-core-phpcs-docker jobs running on integration-slave-docker-1001 are running significantly slower than other slaves.Sep 29 2017, 9:14 AM
hashar edited projects, added Continuous-Integration-Infrastructure; removed Continuous-Integration-Infrastructure (shipyard).Dec 31 2018, 3:18 PM
hashar closed this task as Resolved.Apr 29 2020, 8:01 AM
hashar assigned this task to • dduvall.
hashar added subscribers: • dduvall, hashar.

That one got addressed by @dduvall which refactored the jobs to use docker stop to kill those containers instead of relying on signaling (which is hmm.. unreliable).

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL