Page MenuHomePhabricator

Undeploy electron service from WMF production
Closed, ResolvedPublic

Description

After we've switched all PDF render traffic to Proton, waited the appropriate amount of time for user complaints and got "go" from the product, we can undeploy pdfrender electron service from WMF production and clean up puppet.

  1. Remove it from LVS, confd, and icinga checks, files to modify:
      • (This step should be merged and orchestrated by an SRE)
      • conftool-data/discovery/services.yaml
      • conftool-data/node/codfw.yaml
      • conftool-data/node/eqiad.yaml
      • conftool-data/service/services.yaml
      • hieradata/common/discovery.yaml
      • hieradata/common/lvs/configuration.yaml
      • hieradata/role/common/restbase/dev_cluster.yaml
      • hieradata/role/common/restbase/production.yaml
      • hieradata/role/common/scb.yaml
    1. Remove from puppet, modify or delete:
      • modules/pdfrender/*
      • modules/profile/manifests/pdfrender.pp
      • modules/profile/manifests/restbase.pp
      • modules/role/manifests/pdfrender.pp
      • modules/role/manifests/scb.pp
      • modules/service/manifests/configuration.pp
    2. Remove from DNS (SRE)
    3. Manually stop and disable services from scb* hosts (SRE)

Event Timeline

Change 514226 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/puppet@production] Clean up configuration for pdfrender service.

https://gerrit.wikimedia.org/r/514226

Change 519526 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/puppet@production] Remove references to pdfrender from RESTBase.

https://gerrit.wikimedia.org/r/519526

Joe triaged this task as Normal priority.Jun 28 2019, 7:02 AM
Joe moved this task from Backlog to Next up on the serviceops board.
jijiki moved this task from Next up to Doing on the serviceops board.Jul 3 2019, 12:41 PM
jijiki added a subscriber: jijiki.Jul 8 2019, 10:04 AM
jijiki added a comment.EditedJul 8 2019, 10:13 AM

We are going to break this into steps:

  1. Remove it from LVS, confd, and icinga checks, files to modify:
    • (This step should be merged and orchestrated by an SRE)
    • conftool-data/discovery/services.yaml
    • conftool-data/node/codfw.yaml
    • conftool-data/node/eqiad.yaml
    • conftool-data/service/services.yaml
    • hieradata/common/discovery.yaml
    • hieradata/common/lvs/configuration.yaml
    • hieradata/role/common/restbase/dev_cluster.yaml
    • hieradata/role/common/restbase/production.yaml
    • hieradata/role/common/scb.yaml
  2. Remove from puppet, modify or delete:
    • modules/pdfrender/*
    • modules/profile/manifests/pdfrender.pp
    • modules/profile/manifests/restbase.pp
    • modules/role/manifests/pdfrender.pp
    • modules/role/manifests/scb.pp
    • modules/service/manifests/configuration.pp
  3. Remove from DNS (SRE)
  4. Manually stop and disable services from scb* hosts (SRE)

Change 521504 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/puppet@production] Undeploy pdfrender service. LVS/confd cleanup.

https://gerrit.wikimedia.org/r/521504

Change 521504 merged by Alexandros Kosiaris:
[operations/puppet@production] Undeploy pdfrender service. LVS/confd cleanup.

https://gerrit.wikimedia.org/r/521504

Mentioned in SAL (#wikimedia-operations) [2019-07-09T15:33:05Z] <akosiaris> restart pybal on lvs2006, lvs1016. Removal of pdfrender service T226675

Change 519526 merged by Alexandros Kosiaris:
[operations/puppet@production] Remove references to pdfrender from RESTBase.

https://gerrit.wikimedia.org/r/519526

Mentioned in SAL (#wikimedia-operations) [2019-07-09T15:38:53Z] <akosiaris> restart pybal on lvs2003, lvs1015. Removal of pdfrender service T226675

Change 521523 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/dns@master] pdfrender: Remove discovery records

https://gerrit.wikimedia.org/r/521523

Change 521523 merged by Alexandros Kosiaris:
[operations/dns@master] pdfrender: Remove discovery records

https://gerrit.wikimedia.org/r/521523

Change 514226 abandoned by Ppchelko:
Clean up configuration for pdfrender service.

https://gerrit.wikimedia.org/r/514226

Change 521550 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/puppet@production] Undeploy pdfrender: remove remaining classes.

https://gerrit.wikimedia.org/r/521550

Change 521550 merged by Dzahn:
[operations/puppet@production] Undeploy pdfrender: remove remaining classes.

https://gerrit.wikimedia.org/r/521550

Seems like only 2 items are remaining:

  • Remove from DNS (SRE)
  • Manually stop and disable services from scb* hosts (SRE)

@jijiki are you gonna take care of these?

Change 521582 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] remove pdfrender records

https://gerrit.wikimedia.org/r/521582

Mentioned in SAL (#wikimedia-operations) [2019-07-09T20:36:28Z] <mutante> scb2001 - sudo systemctl stop pdfrender (T226675)

Mentioned in SAL (#wikimedia-operations) [2019-07-09T20:37:41Z] <mutante> scb1001 - re-activate puppet, run puppet, stop pdfrender service, run puppet again (T226675)

Dzahn added a subscriber: Dzahn.EditedJul 9 2019, 8:40 PM

We are going to break this into steps:

Going through these steps to confirm. Merging https://gerrit.wikimedia.org/r/521550

  1. Remove it from LVS, confd, and icinga checks, files to modify:

Done in https://gerrit.wikimedia.org/r/c/operations/puppet/+/521504

    • (This step should be merged and orchestrated by an SRE)
    • conftool-data/discovery/services.yaml
    • conftool-data/node/codfw.yaml
    • conftool-data/node/eqiad.yaml
    • conftool-data/service/services.yaml
    • hieradata/common/discovery.yaml
    • hieradata/common/lvs/configuration.yaml
    • hieradata/role/common/restbase/dev_cluster.yaml
    • hieradata/role/common/restbase/production.yaml
    • hieradata/role/common/scb.yaml
  1. Remove from puppet, modify or delete:

done in https://gerrit.wikimedia.org/r/c/operations/puppet/+/521550

  • modules/pdfrender/*
  • modules/profile/manifests/pdfrender.pp
  • modules/profile/manifests/restbase.pp
  • modules/role/manifests/pdfrender.pp
  • modules/role/manifests/scb.pp
  • modules/service/manifests/configuration.pp
  1. Remove from DNS (SRE)

discovery records done in https://gerrit.wikimedia.org/r/c/operations/dns/+/521523
remaining records removed in https://gerrit.wikimedia.org/r/c/operations/dns/+/521582 (not merged yet)

  1. Manually stop and disable services from scb* hosts (SRE)

done (see SAL above)

  • remove remaining remnants such as /lib/systemd/system/pdfrender.service, /srv/log/pdfrender ?

/srv/log/pdfrender
/lib/systemd/system/pdfrender.service
/etc/sudoers.d/pdfrender-admin
/etc/systemd/system/multi-user.target.wants/pdfrender.service
/etc/firejail/pdfrender.profile
/etc/logrotate.d/pdfrender

(find -name pdfrender* -delete ?)

For what is worth there was 1 extra step (step 0 actually in the order) and it's Remove discovery records

jijiki claimed this task.Jul 10 2019, 2:40 PM

Change 521582 merged by Effie Mouzeli:
[operations/dns@master] remove pdfrender records

https://gerrit.wikimedia.org/r/521582

Guessing this means we can delete deployment-pdfrender02 in beta?

Guessing this means we can delete deployment-pdfrender02 in beta?

That is correct @thcipriani

Guessing this means we can delete deployment-pdfrender02 in beta?

That is correct @thcipriani

Deleted. Thanks!

jijiki closed this task as Resolved.Jul 22 2019, 12:13 PM
jijiki updated the task description. (Show Details)