Page MenuHomePhabricator
Create Task
Maniphest T217891

CloudVPS: rework codfw deployments
Closed, ResolvedPublic
Actions

Description

It's time to rework codfw Cloud VPS (openstack) deployments. View of our current deployments: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Deployments

We will be keeping codfw1dev (AKA labtestn) as a mirror setup of eqiad1.

List of affected servers and their plans:

Hosts with little changes (or no changes at all):

After these changes, this is how the deployments in codfw will look like:

codfw1dev
cloudcontrol2001-dev.codfw.wmnet (was spare)
cloudcontrol2003-dev.codfw.wmnet (was labtestcontrol2003.wikimedia.org)

cloudnet2002-dev.codfw.wmnet
cloudnet2003-dev.codfw.smnet (was labtestnet2003.codfw.wmnet)

cloudservices2002-dev.wikimedia.org (was labtestservices2002.wikimedia.org)

cloudweb2001-dev.wikimedia.org (was labtestnet2002.codfw.wmnet)
clouddb2001.dev.codfw.wmnet (was labtestmetal2001.codfw.wmnet)

cloudvirt2001-dev.codfw.wmnet (was spare)
cloudvirt2002-dev.codfw.wmnet (was spare)
cloudvirt2003-dev.codfw.wmnet (was spare)

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedaborreroT217891 CloudVPS: rework codfw deployments
 ResolvedaborreroT217055 labtestmetal2001.codfw.wmnet needs to be rebuilt
 ResolvedPapaulT218021 decommission: labtestcontrol2001.wikimedia.org
 ResolvedPapaulT218022 decommission: labtestservices2001.wikimedia.org
 ResolvedAndrewT218575 Reallocate LDAP database from labtestservices2001
 ResolvedAndrewT221106 cloudservices2002-dev: bootstrap
 ResolvedAndrewT221463 questions about standalone wmf-mariadb103
 ResolvedPapaulT218023 decommission: labtestvirt200[12].codfw.wmnet
 ResolvedPapaulT218024 decommmision: labtestweb2001.wikimedia.org
 ResolvedPapaulT218025 decommission: cloudnet2001-dev.codfw.wmnet
 ResolvedaborreroT218026 puppet: cleanup labtest puppet code
 ResolvedaborreroT218029 CloudVPS: evaluate convenience of having codfw openstack DBs in proper DB hosts
 DeclinedNoneT218569 Openstack codfw DBs: move to m5-master.eqiad.wmnet
 DeclinedNoneT218570 DB planning: include a writeable (?) misc DB cluster in codfw for WMCS
 DeclinedaborreroT218705 codfw: hardware plan for codfw2dev openstack deployment
Unknown Object (Task)
 ResolvedaborreroT219626 codfw1dev: bootstrap cloudcontrol servers in mitaka/stretch
 ResolvedAndrewT220096 codfw1dev: decide which DBs to reallocate to clouddb2001-dev
 OpenNoneT229559 CloudVPS: codfw1dev: database backup for clouddb2001-dev.codfw.wmnet
 ResolvedAndrewT224528 rack/setup codfw: cloudbackup2001.codfw.wmnet and cloudbackup2002.codfw.wmnet
 ResolvedaborreroT234683 Build, package bdsync for Buster
 ResolvedAndrewT245808 Ferm rules for cloudbackup2001/2001
 ResolvedAndrewT336060 cloudbackup2001 lockup on 2023-05-05
 ResolvedaborreroT220173 cloudcontrol2001-dev hourly cronspam
 ResolvedaborreroT219776 labtestnet2003.codfw.wmnet: rename to cloudnet2003-dev.codfw.wmnet and reimage to stretch
 ResolvedPapaulT219861 Server rename: labtestnet2003 to cloudnet2003-dev, update label and switch ports descriptions, etc
 ResolvedPapaulT220095 labtestcontrol2003: rename to cloudcontrol2003-dev
 ResolvedaborreroT220101 labtestservices2002: rename to cloudservices2002-dev and reimage to stretch in codfw1dev
 ResolvedaborreroT220129 labtestmetal2001.codfw.wmnet: rename to clouddb2001-dev and reimage to stretch in codfw1dev
 DeclinedaborreroT220203 labtestnet2002: reimage to stretch and rename to cloudnet2004-dev
 DuplicateaborreroT220215 Repurpose labtestmetal2001 as a db server for cloud testing
 ResolvedaborreroT220426 labtestnet2002: repurpose as cloudweb2001-dev.wikimedia.org
 ResolvedAndrewT239360 determine status of cloudweb2001-dev
 ResolvedaborreroT222057 labtestvirt2003.codfw.wmnet: reimage as spare stretch
 ResolvedaborreroT222060 labtestservices2003.wikimedia.org: reimage as spare stretch
 DeclinedNoneT222061 labtestpuppetmaster2001.wikimedia.org: use proper codfw1dev role
 ResolvedNoneT229441 CloudVPS: codfw1dev: missing bits
 ResolvedAndrewT228972 CloudVPS: codfw1dev: refresh glance images
 ResolvedAndrewT228974 CloudVPS: codfw1dev: proper DNS setup
 ResolvedCDanisT233236 Move labtestwikitech database to clouddb2001-dev
 ResolvedMarosteguiT236010 Drop labtestwikitech database from m5
 ResolvedAndrewT236145 processEchoEmailBatch.php failing for labtestwiki
 ResolvedaborreroT237971 Cron <www-data@cloudweb2001-dev> /usr/local/bin/mwscript extensions/TorBlock/maintenance/loadExitNodes.php --wiki=labswiki --force > /dev/null
 InvalidNoneT234518 codfw1dev: glance_rsync_images is failing
 ResolvedaborreroT239347 create a 'normal' network for codf1dev neutron w/public IPs
 ResolvedKrenairT242607 Create in-cloud puppetmaster for codfw1dev
 ResolvedaborreroT243556 Fix internal TLD in use in codfw1dev
 ResolvedjcrespoT256846 labtestpuppetmaster2001 is failing to backup
 ResolvedAndrewT242697 Fix LDAP config on codfw1dev instances
 ResolvedNoneT242976 public dns for codfw1dev vms
 ResolvedaborreroT244222 CloudVPS: hiera refactor
 DuplicatejbondT244933 Re-organize hiera lookups for cloud-vps instances
 ResolvedAndrewT265965 Ceph dev cluster in codfw1dev
Unknown Object (Task)
 ResolvedPapaulT250846 (Need By: TBD) rack/setup/install cloudceph200[123]-dev
 ResolvedAndrewT266257 Three ceph mon nodes for codfw1dev
 InvalidNoneT266259 Locate hardware for cloudcephosd nodes in codfw1dev
Unknown Object (Task)
 ResolvedPapaulT267378 (Need By: TBD) rack/setup/install cloudcephmon200[12]-dev
 ResolvedAndrewT266261 rearrange networking for cloudceph200[1-3]-dev and rename
 ResolvedayounsiT269457 Permit traffic between cloudcontrol2001/2003/2004 and cloudcephmon200[1-3]-dev

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
aborrero updated the task description. (Show Details)Mar 18 2019, 1:35 PM
aborrero changed the status of subtask T218022: decommission: labtestservices2001.wikimedia.org from Open to Stalled.Mar 18 2019, 1:38 PM
aborrero updated the task description. (Show Details)Mar 18 2019, 1:51 PM
aborrero updated the task description. (Show Details)Mar 18 2019, 2:01 PM
aborrero closed subtask T218029: CloudVPS: evaluate convenience of having codfw openstack DBs in proper DB hosts as Resolved.Mar 19 2019, 11:25 AM
aborrero changed the status of subtask T218024: decommmision: labtestweb2001.wikimedia.org from Open to Stalled.Mar 21 2019, 4:47 PM
aborrero updated the task description. (Show Details)
aborrero updated the task description. (Show Details)Mar 21 2019, 5:10 PM
• GTirloni added a project: Cloud-VPS.Mar 21 2019, 5:23 PM
aborrero updated the task description. (Show Details)Mar 25 2019, 12:02 PM
bd808 mentioned this in T212772: Track remaining trusty servers in production.Mar 26 2019, 8:11 PM
aborrero mentioned this in T215407: cloudvps: cloudcontrol support for mitaka/stretch.Mar 29 2019, 12:33 PM
aborrero updated the task description. (Show Details)
aborrero updated the task description. (Show Details)Mar 29 2019, 12:41 PM
aborrero updated the task description. (Show Details)Apr 1 2019, 11:03 AM
aborrero closed subtask T219776: labtestnet2003.codfw.wmnet: rename to cloudnet2003-dev.codfw.wmnet and reimage to stretch as Resolved.Apr 2 2019, 12:49 PM
Papaul closed subtask T218023: decommission: labtestvirt200[12].codfw.wmnet as Resolved.Apr 2 2019, 2:37 PM
aborrero mentioned this in T220095: labtestcontrol2003: rename to cloudcontrol2003-dev.Apr 4 2019, 12:30 PM
aborrero updated the task description. (Show Details)Apr 4 2019, 12:34 PM
aborrero updated the task description. (Show Details)Apr 4 2019, 12:55 PM
aborrero updated the task description. (Show Details)Apr 4 2019, 1:02 PM
aborrero added a subtask: T220129: labtestmetal2001.codfw.wmnet: rename to clouddb2001-dev and reimage to stretch in codfw1dev.Apr 4 2019, 4:55 PM
aborrero updated the task description. (Show Details)
aborrero updated the task description. (Show Details)Apr 4 2019, 4:57 PM
aborrero mentioned this in T220173: cloudcontrol2001-dev hourly cronspam.Apr 5 2019, 9:16 AM
aborrero updated the task description. (Show Details)Apr 5 2019, 1:24 PM
aborrero reassigned this task from aborrero to Andrew.EditedApr 5 2019, 1:27 PM
aborrero added a subscriber: bd808.

We may end with 3 cloudnet servers:

Before I move on with T220203, I would like to evaluate another option, which is to simply return labtestnet2002 to the spare pool.
I'm trying to avoid a future in which we require another server in codfw and we decide to reuse one of those cloudnets.

So options are 2:

By now, I will leave the server with role::spare in stretch, which is a common intermediate state for both options.

Please @bd808 and @Andrew comment (assigning task to andrew)

Andrew added a comment.Apr 5 2019, 3:24 PM

I don't think we need three network nodes, until/unless we start testing for a new region. So yeah, marking labtestnet2002 as a spare seems like the best option for now.

aborrero claimed this task.Apr 5 2019, 3:25 PM
aborrero closed subtask T220203: labtestnet2002: reimage to stretch and rename to cloudnet2004-dev as Declined.
aborrero updated the task description. (Show Details)
aborrero added a comment.Apr 5 2019, 4:06 PM

Per conversation with @Andrew on IRC:

  • labtestmetal2001 -> clouddb2001-dev
  • labtestnet2002 -> cloudweb2001-dev
  • labtestweb2001 -> decom
• MoritzMuehlenhoff reopened subtask T219776: labtestnet2003.codfw.wmnet: rename to cloudnet2003-dev.codfw.wmnet and reimage to stretch as Open.Apr 8 2019, 7:33 AM
aborrero closed subtask T219776: labtestnet2003.codfw.wmnet: rename to cloudnet2003-dev.codfw.wmnet and reimage to stretch as Resolved.Apr 8 2019, 1:03 PM
aborrero updated the task description. (Show Details)Apr 8 2019, 4:48 PM
aborrero updated the task description. (Show Details)Apr 8 2019, 4:54 PM
aborrero mentioned this in T220426: labtestnet2002: repurpose as cloudweb2001-dev.wikimedia.org.Apr 8 2019, 4:58 PM
aborrero updated the task description. (Show Details)
aborrero closed subtask T217055: labtestmetal2001.codfw.wmnet needs to be rebuilt as Resolved.Apr 8 2019, 5:01 PM
aborrero updated the task description. (Show Details)Apr 9 2019, 10:07 AM
aborrero updated the task description. (Show Details)Apr 9 2019, 10:10 AM
aborrero closed subtask T220101: labtestservices2002: rename to cloudservices2002-dev and reimage to stretch in codfw1dev as Resolved.Apr 9 2019, 11:28 AM
aborrero updated the task description. (Show Details)
aborrero updated the task description. (Show Details)Apr 9 2019, 12:52 PM
aborrero closed subtask T220129: labtestmetal2001.codfw.wmnet: rename to clouddb2001-dev and reimage to stretch in codfw1dev as Resolved.Apr 9 2019, 3:52 PM
aborrero closed subtask T220426: labtestnet2002: repurpose as cloudweb2001-dev.wikimedia.org as Resolved.Apr 11 2019, 9:28 AM
aborrero updated the task description. (Show Details)
aborrero changed the status of subtask T218024: decommmision: labtestweb2001.wikimedia.org from Stalled to Open.Apr 11 2019, 10:54 AM
aborrero changed the status of subtask T218021: decommission: labtestcontrol2001.wikimedia.org from Stalled to Open.Apr 12 2019, 10:46 AM
aborrero updated the task description. (Show Details)
aborrero updated the task description. (Show Details)Apr 12 2019, 11:01 AM
• Gilles subscribed.Apr 12 2019, 11:46 AM

FYI, deployed a mediawiki config change just now and got this:

11:39:41 Started sync-apaches
11:44:06 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/InitialiseSettings.php', 'mw1268.eqiad.wmnet', 'mw1314.eqiad.wmnet', 'mw2255.codfw.wmnet', 'mw2290.codfw.wmnet', 'mw2216.codfw.wmnet', 'mw1251.eqiad.wmnet', 'mw2188.codfw.wmnet', 'mw1320.eqiad.wmnet', 'mw1285.eqiad.wmnet'] on cloudweb2001-dev.wikimedia.org returned [255]: ssh: connect to host cloudweb2001-dev.wikimedia.org port 22: Connection timed out

sync-apaches: 100% (ok: 262; fail: 1; left: 0)
aborrero changed the status of subtask T218705: codfw: hardware plan for codfw2dev openstack deployment from Open to Stalled.Apr 15 2019, 11:09 AM
Papaul closed subtask T218025: decommission: cloudnet2001-dev.codfw.wmnet as Resolved.Apr 15 2019, 2:59 PM
aborrero closed subtask T219626: codfw1dev: bootstrap cloudcontrol servers in mitaka/stretch as Resolved.Apr 16 2019, 11:32 AM
aborrero updated the task description. (Show Details)Apr 16 2019, 12:23 PM
In T217891#5106883, @Gilles wrote:

FYI, deployed a mediawiki config change just now and got this:

11:39:41 Started sync-apaches
11:44:06 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--include', 'wmf-config', '--include', 'wmf-config/InitialiseSettings.php', 'mw1268.eqiad.wmnet', 'mw1314.eqiad.wmnet', 'mw2255.codfw.wmnet', 'mw2290.codfw.wmnet', 'mw2216.codfw.wmnet', 'mw1251.eqiad.wmnet', 'mw2188.codfw.wmnet', 'mw1320.eqiad.wmnet', 'mw1285.eqiad.wmnet'] on cloudweb2001-dev.wikimedia.org returned [255]: ssh: connect to host cloudweb2001-dev.wikimedia.org port 22: Connection timed out

sync-apaches: 100% (ok: 262; fail: 1; left: 0)

This should be T220931: deploy1001 cannot reach cloudweb2001-dev.wikimedia.org when running scap

aborrero updated the task description. (Show Details)Apr 16 2019, 3:19 PM
Dzahn closed subtask T218021: decommission: labtestcontrol2001.wikimedia.org as Resolved.Apr 17 2019, 8:17 PM
aborrero changed the status of subtask T218022: decommission: labtestservices2001.wikimedia.org from Stalled to Open.Apr 22 2019, 11:35 AM
aborrero updated the task description. (Show Details)Apr 22 2019, 12:25 PM
Papaul closed subtask T220095: labtestcontrol2003: rename to cloudcontrol2003-dev as Resolved.Apr 24 2019, 2:49 PM
aborrero updated the task description. (Show Details)Apr 25 2019, 11:17 AM
aborrero updated the task description. (Show Details)Apr 29 2019, 9:39 AM
aborrero updated the task description. (Show Details)Apr 29 2019, 10:15 AM
aborrero updated the task description. (Show Details)
aborrero closed subtask T222060: labtestservices2003.wikimedia.org: reimage as spare stretch as Resolved.Apr 30 2019, 11:07 AM
aborrero closed subtask T222057: labtestvirt2003.codfw.wmnet: reimage as spare stretch as Resolved.
aborrero updated the task description. (Show Details)
Papaul closed subtask T218024: decommmision: labtestweb2001.wikimedia.org as Resolved.May 15 2019, 2:46 PM
Papaul closed subtask T218022: decommission: labtestservices2001.wikimedia.org as Resolved.May 15 2019, 4:32 PM
aborrero closed subtask Unknown Object (Task) as Resolved.May 20 2019, 10:32 AM
aborrero closed subtask T218705: codfw: hardware plan for codfw2dev openstack deployment as Declined.
aborrero closed this task as Resolved.May 20 2019, 10:34 AM
aborrero closed subtask T218026: puppet: cleanup labtest puppet code as Resolved.

Closing this task now, since the only important subtask is T222061: labtestpuppetmaster2001.wikimedia.org: use proper codfw1dev role but that doesn't really affect the rework of the codfw deployments, which is actually completed now.

aborrero added a subtask: T228972: CloudVPS: codfw1dev: refresh glance images.Jul 25 2019, 10:04 AM
aborrero added a subtask: T229441: CloudVPS: codfw1dev: missing bits.Jul 31 2019, 4:18 PM
aborrero removed a subtask: T228972: CloudVPS: codfw1dev: refresh glance images.Aug 1 2019, 9:07 AM
aborrero removed a subtask: T228974: CloudVPS: codfw1dev: proper DNS setup.
Andrew closed subtask T222061: labtestpuppetmaster2001.wikimedia.org: use proper codfw1dev role as Declined.Mar 10 2020, 5:03 PM
Andrew closed subtask T229441: CloudVPS: codfw1dev: missing bits as Resolved.Jan 27 2023, 2:05 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits