Page MenuHomePhabricator
Create Task
Maniphest T322369

create aphlict2001 (Phabricator realtime notifications codfw)
Closed, ResolvedPublic
Actions

Description

aphlict1001 is a VM that only exists in eqiad.

it's like a sidekick to the physical phabricator host and runs the realtime notification service

the physical hosts are about to be upgraded from phab1001 to phab1004 and phab2001 to phab2002

to have parity in codfw for the Phabricator setup we would also need to have the codfw-equivalent of aphlict, so aphlict2001.

When the aphlict service was moved from the Phabricator hosts to dedicated VMs it also got it's own discovery DNS name.

So now there is:

[phab1004:~] $ host aphlict.discovery.wmnet
aphlict.discovery.wmnet is an alias for aphlict1001.eqiad.wmnet.


[phab1004:~] $ host aphlict1001.eqiad.wmnet
aphlict1001.eqiad.wmnet has address 10.64.48.39
aphlict1001.eqiad.wmnet has IPv6 address 2620:0:861:107:10:64:48:39

[phab1004:~] $ host aphlict2001.codfw.wmnet
Host aphlict2001.codfw.wmnet not found: 3(NXDOMAIN)

Ultimately this ticket should end with an edit to this in the DNS repo:

; misc services without multiple backends
aphlict               300 IN CNAME aphlict1001.eqiad.wmnet.

Because then it can be moved into the section "; misc web services with multiple backends but without geoip".

That's an upgrade in itself.

Details

SubjectRepoBranchLines +/-
Add aphlict role to new vm hostoperations/puppetproduction+3 -5
Assign insetup role to new aphlict vmoperations/puppetproduction+4 -0
Add dummy 'config_deploy_vars' for aphlictoperations/puppetproduction+46 -3
Add the aphlict config on aphlict2001.codfwoperations/puppetproduction+76 -59
Add puppet role to new aphlict VMoperations/puppetproduction+1 -5
Add insetup puppet role for aphlict vm in codfwoperations/puppetproduction+10 -1
remove phab1001-aphlict.eqiad.wmnetoperations/dnsmaster+0 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Dzahn created this task.Nov 3 2022, 8:03 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 3 2022, 8:03 PM
Dzahn renamed this task from create aphlict2001 to create aphlict2001 (Phabricator realtime notifications codfw).Nov 3 2022, 8:06 PM
Dzahn updated the task description. (Show Details)
gerritbot added a comment.Nov 3 2022, 8:55 PM

Change 853010 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/dns@master] remove phab1001-aphlict.eqiad.wmnet

https://gerrit.wikimedia.org/r/853010

gerritbot added a project: Patch-For-Review.Nov 3 2022, 8:55 PM
LSobanski moved this task from Incoming to Backlog on the collaboration-services board.Nov 4 2022, 3:42 PM
gerritbot added a comment.Nov 7 2022, 9:24 PM

Change 853010 merged by Dzahn:

[operations/dns@master] remove phab1001-aphlict.eqiad.wmnet

https://gerrit.wikimedia.org/r/853010

Maintenance_bot removed a project: Patch-For-Review.Nov 7 2022, 9:31 PM
• Wikiakbar1 removed a project: collaboration-services.Nov 11 2022, 7:20 AM
• Wikiakbar1 updated the task description. (Show Details)
• Wikiakbar1 updated the task description. (Show Details)
• Wikiakbar1 removed a project: Phabricator.
• Wikiakbar1 updated the task description. (Show Details)
• Wikiakbar1 edited subscribers, added: • Wikiakbar1; removed: Aklapper, Dzahn.
JJMC89 added projects: collaboration-services, Phabricator.Nov 11 2022, 7:29 AM
JJMC89 edited subscribers, added: Aklapper, Dzahn; removed: • Wikiakbar1.
Dzahn added a comment.Jan 13 2023, 8:04 PM

created VM with:

dzahn@cumin2002:~$ sudo cookbook sre.ganeti.makevm --vcpus 1 --memory 2 --disk 20 --network private --cluster codfw --group D aphlict2001
LSobanski triaged this task as Medium priority.Jan 17 2023, 3:57 PM
eoghan claimed this task.Feb 13 2023, 12:40 PM
gerritbot added a comment.Feb 13 2023, 12:56 PM

Change 888690 had a related patch set uploaded (by EoghanGaffney; author: EoghanGaffney):

[operations/puppet@production] Add puppet role for aphlict vm in codfw

https://gerrit.wikimedia.org/r/888690

gerritbot added a project: Patch-For-Review.Feb 13 2023, 12:56 PM
eoghan added a comment.Feb 13 2023, 1:12 PM

It seems that we do three things here, since the VM is already created and shut down:

  • Add puppet role (not many changes to make beyond adding the new host to site.pp)
  • Switch on VM, make sure it bootstraps correctly
  • Change DNS
eoghan moved this task from Backlog to Work in Progress on the collaboration-services board.Feb 13 2023, 3:51 PM
Dzahn added a comment.Feb 13 2023, 6:36 PM
  • add "insetup puppet role"
  • start VM
  • get console on VM
  • use "install_console" command from a puppetmaster to get intial root on VM (https://wikitech.wikimedia.org/wiki/Ganeti#Sign_Puppet_certs)
  • run puppet agent on VM first time, this creates a cert signing request
  • sign the request on the puppetmaster
  • run puppet agent on VM one more time.. a lot of things get installed all from the "base" module included by the "insetup" role
  • run puppet again until nothing new happens anymore
  • upload change that flips the role from "insetup" to production role, use puppet compiler to see what it does. does it hardcode the host name "1001" somewhere for example? does it start services by default but should they be started on the second instance? keep in mind this role has never had a second server before, be it active or passive.
  • apply puppet change if fine with it and unless other changes are needed
  • change the status of the VM in netbox to "active"
gerritbot added a comment.Feb 14 2023, 11:06 AM

Change 888690 merged by EoghanGaffney:

[operations/puppet@production] Add insetup puppet role for aphlict vm in codfw

https://gerrit.wikimedia.org/r/888690

Maintenance_bot removed a project: Patch-For-Review.Feb 14 2023, 11:10 AM
Dzahn added a comment.Feb 14 2023, 7:29 PM

This VM would not boot. After debugging we found out it's because it's not in DHCP server config.

Until recently we had to add the MAC address of a freshly created VM into DHCP config under the install_server module and without it it would not boot.

But now this has changed and the files are gone from the puppet repo.

It seems what happened here is that there were changes to how this is handled between the time I created this VM and now that @eoghan wans to install an OS on it.

Trying to use the "reimage" cookbook resulted in a "not found in netbox" error but as Eoghan points out it does exist in netbox. It's currently unclear why that happens.

It might be easiest to just delete this VM and create it again with the makevm cookbook and check if the MAC gets addded to DHCP then.

Currently only aphlict1001 but not aphlict2001 is in there:

[install2004:/etc/dhcp] $ grep -r aphlict *
linux-host-entries.ttyS0-115200:host aphlict1001 {
linux-host-entries.ttyS0-115200:    fixed-address aphlict1001.eqiad.wmnet;
[install2004:/etc/dhcp] $

The MAC address of aphlict2001 is aa:00:00:10:b5:6f.

ops-monitoring-bot added a comment.Feb 14 2023, 10:10 PM

Cookbook cookbooks.sre.ganeti.reimage was started by eoghan@cumin2002 for host aphlict2001.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 14 2023, 10:39 PM

Cookbook cookbooks.sre.ganeti.reimage started by eoghan@cumin2002 for host aphlict2001.codfw.wmnet with OS bullseye completed:

  • aphlict2001 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot to disk
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/ganeti/reimage/202302142210_eoghan_2528502_aphlict2001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
gerritbot added a comment.Feb 15 2023, 12:12 PM

Change 889531 had a related patch set uploaded (by EoghanGaffney; author: EoghanGaffney):

[operations/puppet@production] Add puppet role to new aphlict VM

https://gerrit.wikimedia.org/r/889531

gerritbot added a project: Patch-For-Review.Feb 15 2023, 12:12 PM
gerritbot added a comment.Feb 16 2023, 12:38 PM

Change 889531 merged by EoghanGaffney:

[operations/puppet@production] Add puppet role to new aphlict VM

https://gerrit.wikimedia.org/r/889531

Maintenance_bot removed a project: Patch-For-Review.Feb 16 2023, 1:11 PM
Dzahn added a comment.Feb 16 2023, 8:08 PM

The production role has been applied, which is great!

Though it looks like next we need a scap deployment to aphlict2001. Probably would be best to add this as a topic to the next sync meeting with releng., maybe we can do the deploy and also make sure the phab config only refers to the discovery DNS record and not a host name.

Once we have done these things we know we can failover aphlict by just changing a DNS record.

To be truly active-active would be another thing though.

brennen subscribed.Feb 16 2023, 8:15 PM
eoghan mentioned this in T329908: Deploy phabricator to aphlict2001.codfw.wmnet.Feb 16 2023, 11:57 PM
Dzahn mentioned this in T330393: puppet/scap: failing on aphlict2001.Feb 23 2023, 7:08 PM
Dzahn added a subtask: T330393: puppet/scap: failing on aphlict2001.
Dzahn added a comment.Feb 23 2023, 7:10 PM

T330393 was created because puppet fails on this machine but we know that and the scap deploy at T329908 should fix that.

Stashbot added a comment.Feb 23 2023, 7:50 PM

Mentioned in SAL (#wikimedia-operations) [2023-02-23T19:50:19Z] <mutante> aphlict2001 - manually created /etc/phabricator/config.yaml - empty file owned by root:phab-deploy to debug for T330393 T322369

Dzahn added a comment.Feb 23 2023, 9:06 PM

19:41 < mutante> scap has defaults to use gerrit as origin: modules/scap/lib/puppet/provider/scap_source/default.rb: "https://gerrit.wikimedia.org/r/#{repo_name}.git"
19:42 < mutante> so you cant use gitlab repos as origin for something deployed with scap
19:42 < mutante> this currently is a problem for phabricator repos which got moved from phabricator itself to gitlab

eoghan added a comment.Mar 7 2023, 1:26 PM

I split out the phabricator config into a separate module from the main phabricator module in https://gerrit.wikimedia.org/r/c/operations/puppet/+/891841 - next step is to apply this role to the aphlict machines. The catch is that because these machines are configured by hand, we might overwrite the configs that were originally deployed. We'll apply it to aphlict2001 to start with, see if we can make that work, and then apply it to aphlict1001.

Jelto subscribed.Mar 7 2023, 1:48 PM
gerritbot added a comment.Mar 8 2023, 12:04 PM

Change 895240 had a related patch set uploaded (by EoghanGaffney; author: EoghanGaffney):

[operations/puppet@production] Add the aphlict config on aphlict2001.codfw

https://gerrit.wikimedia.org/r/895240

gerritbot added a project: Patch-For-Review.Mar 8 2023, 12:04 PM
gerritbot added a comment.Mar 10 2023, 12:44 PM

Change 895240 merged by EoghanGaffney:

[operations/puppet@production] Add the aphlict config on aphlict2001.codfw

https://gerrit.wikimedia.org/r/895240

Maintenance_bot removed a project: Patch-For-Review.Mar 10 2023, 1:10 PM
eoghan added a comment.Mar 10 2023, 1:19 PM

I've tried deploying phabricator::config with an empty config to aphlict2001 and running scap after that, but scap needs some credentials in there. At a minimum, what it needs is the variables to perform:

13:05:31 Rendering config_file: /srv/deployment/phabricator/deployment-cache/revs/3f2dd1bf5761597244976c8a39822b08c0843199/phabricator/conf/local/local.json using /etc/phabricator/config.yaml
13:05:31 Rendering config_file: /srv/deployment/phabricator/deployment-cache/revs/3f2dd1bf5761597244976c8a39822b08c0843199/phabricator/conf/local/mail.json using /etc/phabricator/config.yaml
13:05:31 Rendering config_file: /srv/deployment/phabricator/deployment-cache/revs/3f2dd1bf5761597244976c8a39822b08c0843199/phabricator/conf/local/phd.json using /etc/phabricator/config.yaml
13:05:31 Rendering config_file: /srv/deployment/phabricator/deployment-cache/revs/3f2dd1bf5761597244976c8a39822b08c0843199/phabricator/conf/local/vcs.json using /etc/phabricator/config.yaml
13:05:31 Rendering config_file: /srv/deployment/phabricator/deployment-cache/revs/3f2dd1bf5761597244976c8a39822b08c0843199/phabricator/conf/local/www.json using /etc/phabricator/config.yaml
13:05:31 Rendering config_file: /srv/deployment/phabricator/deployment-cache/revs/3f2dd1bf5761597244976c8a39822b08c0843199/phabricator/support/preamble.php using /etc/phabricator/config.yaml
13:05:31 Rendering config_file: /srv/deployment/phabricator/deployment-cache/revs/3f2dd1bf5761597244976c8a39822b08c0843199/phabricator/support/redirect_config.json using /etc/phabricator/config.yaml

Putting a useful phabricator config (put in by @Dzahn recently to debug scap with @brennen) in place on the host let scap finish a deploy successfully, so I think we're very close to completion here.

I'll try adding these credentials to the aphlict config and see if that allows it to continue. In the mean time, I'm leaving puppet disabled on that host for the rest of the day.

Dzahn awarded a token.Mar 10 2023, 5:41 PM
gerritbot added a comment.Mar 13 2023, 11:39 AM

Change 897852 had a related patch set uploaded (by EoghanGaffney; author: EoghanGaffney):

[operations/puppet@production] Add dummy 'config_deploy_vars' for aphlict

https://gerrit.wikimedia.org/r/897852

gerritbot added a project: Patch-For-Review.Mar 13 2023, 11:39 AM
gerritbot added a comment.Mar 13 2023, 10:20 PM

Change 897852 merged by EoghanGaffney:

[operations/puppet@production] Add dummy 'config_deploy_vars' for aphlict

https://gerrit.wikimedia.org/r/897852

Maintenance_bot removed a project: Patch-For-Review.Mar 13 2023, 10:30 PM
eoghan added a comment.Mar 13 2023, 10:45 PM

It looks like the dummy config variables for the phabricator config at least got us as far as a successful puppet deployment, including a scap deploy of phabricator.

I currently see aphlict running on aphlict2001.codfw.wmnet. Before we add it to the DNS template, I'd like to find a way to test if this works correctly. I'll try come up with a good way of doing this soon.

Jelto awarded a token.Mar 14 2023, 10:16 AM
eoghan mentioned this in T331278: scap deploy-local fails on aphlict servers.Mar 14 2023, 11:58 AM
eoghan closed subtask T330393: puppet/scap: failing on aphlict2001 as Resolved.Mar 14 2023, 12:07 PM
brennen moved this task from To Triage to Infrastructure on the Phabricator board.Mar 20 2023, 10:26 PM
gerritbot added a comment.Mar 27 2023, 2:27 PM

Change 903264 had a related patch set uploaded (by EoghanGaffney; author: EoghanGaffney):

[operations/puppet@production] Assign insetup role to new aphlict vm

https://gerrit.wikimedia.org/r/903264

gerritbot added a project: Patch-For-Review.Mar 27 2023, 2:27 PM
gerritbot added a comment.Mar 27 2023, 8:32 PM

Change 903264 merged by EoghanGaffney:

[operations/puppet@production] Assign insetup role to new aphlict vm

https://gerrit.wikimedia.org/r/903264

Maintenance_bot removed a project: Patch-For-Review.Mar 27 2023, 9:10 PM
ops-monitoring-bot added a comment.Mar 28 2023, 12:09 PM

Cookbook cookbooks.sre.ganeti.reimage was started by eoghan@cumin1001 for host aphlict1002.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 28 2023, 12:36 PM

Cookbook cookbooks.sre.ganeti.reimage started by eoghan@cumin1001 for host aphlict1002.eqiad.wmnet with OS bullseye completed:

  • aphlict1002 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot to disk
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/ganeti/reimage/202303281209_eoghan_145080_aphlict1002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
gerritbot added a comment.Mar 28 2023, 12:52 PM

Change 903641 had a related patch set uploaded (by EoghanGaffney; author: EoghanGaffney):

[operations/puppet@production] Add aphlict role to new vm host

https://gerrit.wikimedia.org/r/903641

gerritbot added a project: Patch-For-Review.Mar 28 2023, 12:52 PM
gerritbot added a comment.Mar 29 2023, 12:10 PM

Change 903641 merged by EoghanGaffney:

[operations/puppet@production] Add aphlict role to new vm host

https://gerrit.wikimedia.org/r/903641

Maintenance_bot removed a project: Patch-For-Review.Mar 29 2023, 12:30 PM
eoghan moved this task from Work in Progress to Backlog on the collaboration-services board.Apr 11 2023, 8:37 PM
eoghan moved this task from Backlog to Work in Progress on the collaboration-services board.May 3 2023, 10:22 AM
eoghan moved this task from Work in Progress to Backlog on the collaboration-services board.
eoghan moved this task from Backlog to Work in Progress on the collaboration-services board.
ops-monitoring-bot added a comment.May 4 2023, 10:48 AM

Cookbook cookbooks.sre.ganeti.reimage was started by eoghan@cumin1001 for host aphlict2001.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 4 2023, 11:14 AM

Cookbook cookbooks.sre.ganeti.reimage started by eoghan@cumin1001 for host aphlict2001.codfw.wmnet with OS bullseye completed:

  • aphlict2001 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot to disk
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/ganeti/reimage/202305041048_eoghan_1906453_aphlict2001.out
    • Unable to run puppet on puppetmaster2001.codfw.wmnet,puppetmaster1001.eqiad.wmnet to update configmaster.wikimedia.org with the new host SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
eoghan closed this task as Resolved.May 4 2023, 7:14 PM

We now have aphlict2001 running in codfw, We haven't tested it since we don't allow cross-region traffic, but it uses the same puppet modules as aphlict1002 which was brought from bare install to production traffic without intervention so I'm confident it'll work.

I've also updated the DNS file to move aphlict into the ; misc web services with multiple backends but without geoip section.

eoghan closed subtask T333452: Replace existing aphlict1001 with puppet-managed bullseye host as Resolved.May 10 2023, 11:03 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL