Page MenuHomePhabricator
Create Task
Maniphest T255877

Move proton to use TLS only
Closed, ResolvedPublic
Actions

Description

  • Add TLS support to the deployment chart
  • Enable TLS on k8s in production
  • Add Additional LVS endpoint configuration
  • Switch services to use the TLS LVS
  • Remove non-TLS LVS endpoint configuration
  • Remove the non-TLS k8s service
  • Remove proton VMs
  • Remove all proton puppet configuration related to the old, non k8s, infra

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

JMeybohm created this task.Jun 19 2020, 3:42 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 19 2020, 3:42 PM
JMeybohm added a subscriber: akosiaris.Jul 7 2020, 7:36 AM

Proton has an TLS LVS already via fbee4a768b10e7e405e6ecf64ace2062004c5c36 / T225680

gerritbot added a comment.Jul 7 2020, 9:28 AM

Change 607536 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] proton: Switch restbase production to TLS

https://gerrit.wikimedia.org/r/607536

gerritbot added a project: Patch-For-Review.Jul 7 2020, 9:28 AM
jcrespo triaged this task as Medium priority.Jul 8 2020, 10:44 AM
gerritbot added a comment.Jul 9 2020, 9:18 AM

Change 607536 abandoned by Alexandros Kosiaris:
[operations/puppet@production] proton: Switch restbase production to TLS

Reason:
Done differently in https://gerrit.wikimedia.org/r/c/operations/puppet/ /610720

https://gerrit.wikimedia.org/r/607536

gerritbot added a comment.Jul 9 2020, 12:23 PM

Change 610789 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] proton: Set LVS level OpenAPI checks on TLS

https://gerrit.wikimedia.org/r/610789

gerritbot added a comment.Jul 9 2020, 12:27 PM

Change 610789 merged by Alexandros Kosiaris:
[operations/puppet@production] proton: Set LVS level OpenAPI checks on TLS

https://gerrit.wikimedia.org/r/610789

gerritbot added a comment.Jul 9 2020, 3:24 PM

Change 610855 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] proton: Amend prometheus-statsd config

https://gerrit.wikimedia.org/r/610855

gerritbot added a comment.Jul 10 2020, 7:09 AM

Change 610855 merged by jenkins-bot:
[operations/deployment-charts@master] proton: Amend prometheus-statsd config

https://gerrit.wikimedia.org/r/610855

akosiaris updated the task description. (Show Details)Jul 10 2020, 7:27 AM
Joe assigned this task to JMeybohm.Jul 20 2020, 7:08 AM
jijiki moved this task from Incoming 🐫 to 🔦Unused2 on the serviceops board.Aug 17 2020, 11:45 PM
gerritbot added a comment.Sep 15 2020, 3:29 PM

Change 627541 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/puppet@production] lvs: Remove proton non-TLS endpoint from LVS 1/2

https://gerrit.wikimedia.org/r/627541

gerritbot added a comment.Sep 15 2020, 3:29 PM

Change 627542 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/puppet@production] lvs: Remove proton non-TLS endpoint from LVS 2/2

https://gerrit.wikimedia.org/r/627542

Joe updated the task description. (Show Details)Sep 16 2020, 2:13 PM
gerritbot added a comment.Sep 16 2020, 2:56 PM

Change 627857 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] proton-http: stop monitoring the endpoint

https://gerrit.wikimedia.org/r/627857

gerritbot added a comment.Sep 16 2020, 2:56 PM

Change 627858 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] proton: remove non-https endpoint

https://gerrit.wikimedia.org/r/627858

gerritbot added a comment.Sep 16 2020, 2:56 PM

Change 627859 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] proton: remove conftool-data

https://gerrit.wikimedia.org/r/627859

gerritbot added a comment.Sep 16 2020, 2:56 PM

Change 627860 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] proton: remove the ganeti VMs from puppet

https://gerrit.wikimedia.org/r/627860

gerritbot added a comment.Sep 16 2020, 2:56 PM

Change 627861 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] proton: remove all puppet code, other references to the non-k8s service

https://gerrit.wikimedia.org/r/627861

gerritbot added a comment.Sep 22 2020, 1:39 PM

Change 627857 merged by Giuseppe Lavagetto:
[operations/puppet@production] proton-http: stop monitoring the endpoint

https://gerrit.wikimedia.org/r/627857

gerritbot added a comment.Sep 22 2020, 1:56 PM

Change 627541 abandoned by JMeybohm:
[operations/puppet@production] lvs: Remove proton non-TLS endpoint from LVS 1/2

Reason:
Done with https://gerrit.wikimedia.org/r/c/operations/puppet/ /627857

https://gerrit.wikimedia.org/r/627541

gerritbot added a comment.Sep 22 2020, 2:02 PM

Change 627858 merged by JMeybohm:
[operations/puppet@production] proton: remove non-https endpoint

https://gerrit.wikimedia.org/r/627858

Stashbot added a comment.Sep 22 2020, 2:05 PM

Mentioned in SAL (#wikimedia-operations) [2020-09-22T14:05:57Z] <jayme> running puppet on lvs servers - T255868 T255877

Stashbot mentioned this in T255868: Move citoid to use TLS only.Sep 22 2020, 2:05 PM
Stashbot added a comment.Sep 22 2020, 2:09 PM

Mentioned in SAL (#wikimedia-operations) [2020-09-22T14:09:15Z] <jayme> restarting pybal on lvs1016.eqiad.wmnet,lvs2010.codfw.wmnet - T255868 T255877

Stashbot added a comment.Sep 22 2020, 2:11 PM

Mentioned in SAL (#wikimedia-operations) [2020-09-22T14:11:21Z] <jayme> restarting pybal on lvs1015.eqiad.wmnet,lvs2009.codfw.wmnet - T255868 T255877

Stashbot added a comment.Sep 22 2020, 2:12 PM

Mentioned in SAL (#wikimedia-operations) [2020-09-22T14:12:01Z] <jayme> running ipvsadm -D -t 10.2.2.19:1970; ipvsadm -D -t 10.2.2.21:24766 on lvs1016.eqiad.wmnet,lvs1015.eqiad.wmnet - T255868 T255877

Stashbot added a comment.Sep 22 2020, 2:12 PM

Mentioned in SAL (#wikimedia-operations) [2020-09-22T14:12:40Z] <jayme> running ipvsadm -D -t 10.2.1.19:1970; ipvsadm -D -t 10.2.1.21:24766 on lvs2010.codfw.wmnet,lvs2009.codfw.wmnet - T255868 T255877

JMeybohm updated the task description. (Show Details)Sep 22 2020, 2:14 PM
gerritbot added a comment.Sep 22 2020, 2:19 PM

Change 627542 abandoned by JMeybohm:
[operations/puppet@production] lvs: Remove proton non-TLS endpoint from LVS 2/2

Reason:
done with https://gerrit.wikimedia.org/r/c/operations/puppet/ /627858

https://gerrit.wikimedia.org/r/627542

gerritbot added a comment.Oct 1 2020, 7:54 AM

Change 627859 merged by Alexandros Kosiaris:
[operations/puppet@production] proton: remove conftool-data

https://gerrit.wikimedia.org/r/627859

akosiaris updated the task description. (Show Details)Oct 1 2020, 7:56 AM
JMeybohm updated the task description. (Show Details)Oct 1 2020, 7:59 AM
JMeybohm updated the task description. (Show Details)Oct 1 2020, 8:12 AM
JMeybohm updated the task description. (Show Details)
gerritbot added a comment.Oct 1 2020, 8:14 AM

Change 627860 merged by Alexandros Kosiaris:
[operations/puppet@production] proton: remove the ganeti VMs from puppet

https://gerrit.wikimedia.org/r/627860

ops-monitoring-bot added a comment.Oct 1 2020, 8:22 AM

cookbooks.sre.hosts.decommission executed by akosiaris@cumin1001 for hosts: proton1001.eqiad.wmnet

  • proton1001.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
  • COMMON_STEPS (FAIL)
    • Failed to run the sre.dns.netbox cookbook: Cumin execution failed (exit_code=2)
    • Not all affected DC(s) have been migrated to automatic DNS, a manual patch to the operations/dns repository is required

ERROR: some step on some host failed, check the bolded items above

gerritbot added a comment.Oct 1 2020, 8:27 AM

Change 627861 merged by Alexandros Kosiaris:
[operations/puppet@production] proton: remove all puppet code, other references to the non-k8s service

https://gerrit.wikimedia.org/r/627861

ops-monitoring-bot added a comment.Oct 1 2020, 8:27 AM

cookbooks.sre.hosts.decommission executed by akosiaris@cumin1001 for hosts: proton1002.eqiad.wmnet

  • proton1002.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
  • COMMON_STEPS (FAIL)
    • Failed to run the sre.dns.netbox cookbook: Cumin execution failed (exit_code=2)
    • Not all affected DC(s) have been migrated to automatic DNS, a manual patch to the operations/dns repository is required

ERROR: some step on some host failed, check the bolded items above

gerritbot added a comment.Oct 1 2020, 8:36 AM

Change 631398 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/dns@master] Remove proton{1,2}00{1,2}

https://gerrit.wikimedia.org/r/631398

ops-monitoring-bot added a comment.Oct 1 2020, 8:37 AM

cookbooks.sre.hosts.decommission executed by akosiaris@cumin1001 for hosts: proton2001.codfw.wmnet

  • proton2001.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
  • COMMON_STEPS (WARN)
    • Not all affected DC(s) have been migrated to automatic DNS, a manual patch to the operations/dns repository is required
ops-monitoring-bot added a comment.Oct 1 2020, 8:53 AM

cookbooks.sre.hosts.decommission executed by akosiaris@cumin1001 for hosts: proton2002.codfw.wmnet

  • proton2002.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
  • COMMON_STEPS (WARN)
    • Not all affected DC(s) have been migrated to automatic DNS, a manual patch to the operations/dns repository is required
gerritbot added a comment.Oct 1 2020, 12:41 PM

Change 631398 merged by Alexandros Kosiaris:
[operations/dns@master] Remove proton{1,2}00{1,2}

https://gerrit.wikimedia.org/r/631398

akosiaris closed this task as Resolved.Oct 1 2020, 12:52 PM
akosiaris updated the task description. (Show Details)

All old stuff has been removed, I 'll resolve this.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL