Page MenuHomePhabricator
Create Task
Maniphest T147597

setup/deploy cobalt as gerrit warm standby/replacement
Closed, ResolvedPublic
Actions

Description

This task will track the setup/deployment for server cobalt(WMF4725)

  • - update mgmt dns for hostname
  • - update public dns record
  • - install_server update (raid10-lvm-ext4-srv)
  • - network port setup
  • - os install (jessie)
  • - puppet/salt acceptance
  • - restore/copy data from lead
  • - make needed changes to gerrit role (provide a way to specify slave mode, others?)
  • - add cobalt to contint ferm rules
  • - apply gerrit::server role
  • - remove backup::host include in site.pp, lead and cobalt identical in site.pp, remove lead from site.pp

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Oct 6 2016, 9:49 PM
Paladox unsubscribed.Oct 6 2016, 9:49 PM
Paladox subscribed.
gerritbot subscribed.Oct 6 2016, 9:51 PM

Change 314601 had a related patch set uploaded (by Dzahn):
add IPs for cobalt, using WMF4725

https://gerrit.wikimedia.org/r/314601

gerritbot added a project: Patch-For-Review.Oct 6 2016, 9:51 PM
RobH updated the task description. (Show Details)Oct 6 2016, 9:52 PM
RobH updated the task description. (Show Details)
gerritbot added a comment.Oct 6 2016, 9:59 PM

Change 314601 merged by Dzahn:
add IPs for cobalt, using WMF4725

https://gerrit.wikimedia.org/r/314601

Dzahn claimed this task.Oct 6 2016, 10:01 PM
Dzahn updated the task description. (Show Details)

[radon:~] $ host cobalt.mgmt.eqiad.wmnet
cobalt.mgmt.eqiad.wmnet has address 10.65.2.127

[radon:~] $ host cobalt.wikimedia.org
cobalt.wikimedia.org has address 208.80.154.81
cobalt.wikimedia.org has IPv6 address 2620:0:861:3:208:80:154:81

gerritbot added a comment.Oct 6 2016, 10:15 PM

Change 314609 had a related patch set uploaded (by Dzahn):
add cobalt to DHCP, set partman recipe

https://gerrit.wikimedia.org/r/314609

gerritbot added a comment.Oct 6 2016, 10:19 PM

Change 314609 merged by Dzahn:
add cobalt to DHCP, set partman recipe

https://gerrit.wikimedia.org/r/314609

gerritbot added a comment.Oct 6 2016, 10:32 PM

Change 314612 had a related patch set uploaded (by Dzahn):
add cobalt site.pp, comment gerrit role, access for admins

https://gerrit.wikimedia.org/r/314612

gerritbot added a comment.Oct 6 2016, 10:55 PM

Change 314612 merged by Dzahn:
add cobalt site.pp, comment gerrit role, access for gerrit-roots

https://gerrit.wikimedia.org/r/314612

Dzahn updated the task description. (Show Details)Oct 6 2016, 11:04 PM
Dzahn removed a project: Patch-For-Review.
gerritbot added a comment.Oct 6 2016, 11:16 PM

Change 314628 had a related patch set uploaded (by Chad):
Gerrit: Copy public IPs from lead to cobalt, we're reusing them

https://gerrit.wikimedia.org/r/314628

gerritbot added a project: Patch-For-Review.Oct 6 2016, 11:16 PM
Dzahn updated the task description. (Show Details)Oct 7 2016, 12:32 AM

OS installed, added to puppet, signed salt-key, gave access to gerrit-roots, gerrit server role commented out until tomorrow...

gerritbot added a comment.Oct 7 2016, 12:39 AM

Change 314638 had a related patch set uploaded (by Dzahn):
make cobalt a backup::host

https://gerrit.wikimedia.org/r/314638

gerritbot added a comment.Oct 7 2016, 12:41 AM

Change 314638 merged by Dzahn:
make cobalt a backup::host

https://gerrit.wikimedia.org/r/314638

Dzahn added a comment.EditedOct 7 2016, 12:50 AM

started bacula restore of lead data to cobalt /srv

Run Restore job
JobName: RestoreFiles
Bootstrap: /var/lib/bacula/helium.eqiad.wmnet.restore.2.bsr
Where: /srv
Replace: always
FileSet: srv-gerrit-git
Backup Client: lead.wikimedia.org-fd
Restore Client: cobalt.wikimedia.org-fd
Storage: helium-FileStorage1
When: 2016-10-07 00:46:22
Catalog: production
Priority: 1
Plugin Options: *None*
OK to run? (yes/mod/no): yes
Job queued. JobId=38854

Dzahn added a comment.Oct 7 2016, 12:51 AM

oops, since Where: is a prefix, this is restoring it as /srv/srv/gerrit but we can simply move it when done.. and then we'll rsync the diff tomorrow.

Dzahn updated the task description. (Show Details)Oct 7 2016, 12:55 AM
Dzahn updated the task description. (Show Details)
Dzahn updated the task description. (Show Details)Oct 7 2016, 12:58 AM
gerritbot added a comment.Oct 7 2016, 1:07 AM

Change 314641 had a related patch set uploaded (by Dzahn):
contint: allow ssh from cobalt, in addition to lead

https://gerrit.wikimedia.org/r/314641

gerritbot added a comment.Oct 7 2016, 2:55 PM

Change 314641 merged by Dzahn:
contint: allow ssh from cobalt, in addition to lead

https://gerrit.wikimedia.org/r/314641

gerritbot added a comment.Oct 7 2016, 5:16 PM

Change 314726 had a related patch set uploaded (by Dzahn):
gerrit: add rsyncd on cobalt for migrating data

https://gerrit.wikimedia.org/r/314726

gerritbot added a comment.Oct 7 2016, 5:18 PM

Change 314726 merged by Dzahn:
gerrit: add rsyncd on cobalt for migrating data

https://gerrit.wikimedia.org/r/314726

gerritbot added a comment.Oct 7 2016, 6:19 PM

Change 314734 had a related patch set uploaded (by Dzahn):
gerrit: activate gerrit::server role on cobalt

https://gerrit.wikimedia.org/r/314734

gerritbot added a comment.Oct 7 2016, 6:29 PM

Change 314628 merged by Dzahn:
Gerrit: Specify public IPs for eqiad, we're not changing them

https://gerrit.wikimedia.org/r/314628

gerritbot added a comment.Oct 7 2016, 6:31 PM

Change 314734 merged by Dzahn:
gerrit: activate gerrit::server role on cobalt

https://gerrit.wikimedia.org/r/314734

Dzahn updated the task description. (Show Details)EditedOct 7 2016, 9:11 PM

20:30 bblack: lead.wikimedia.org: replaced by cobalt functionally, please leave it untouched for now with puppet disabled!
19:46 mutante: deleted old /var/lib/gerrit2/ data on cobalt, syncing from lead
19:45 mutante: rsyncing /var/lib/gerrit2 from lead to cobalt
19:30 mutante: removed gerrit IPs from cobalt interfaces
19:29 mutante: disabled puppet on lead and cobalt
19:21 mutante: re-enabling puppet on cobalt
19:21 mutante: removed gerrit IP from lead's interface, v4 and v6
19:09 mutante: rsyncing gerrit data one more time from lead to cobalt
19:07 ostriches: stopped puppet on lead
19:07 mutante: stopping gerrit on lead
19:02 mutante: cobalt, disabled puppet, removed service IP from interface

gerritbot added a comment.Oct 7 2016, 9:15 PM

Change 314767 had a related patch set uploaded (by Dzahn):
gerrit: remove backup::host include from cobalt

https://gerrit.wikimedia.org/r/314767

Dzahn added a subtask: T147776: cronspam from cobalt after the Gerrit migration.Oct 11 2016, 4:02 PM
Dzahn closed subtask T147776: cronspam from cobalt after the Gerrit migration as Resolved.Oct 11 2016, 6:20 PM
gerritbot added a comment.Oct 11 2016, 8:49 PM

Change 314767 merged by Dzahn:
gerrit: remove backup::host, rsyncd include from cobalt

https://gerrit.wikimedia.org/r/314767

gerritbot added a comment.Oct 11 2016, 8:50 PM

Change 314768 had a related patch set uploaded (by Dzahn):
gerrit: mv standard incl to role, rm duplicate firewall

https://gerrit.wikimedia.org/r/314768

gerritbot added a comment.Oct 11 2016, 9:01 PM

Change 314768 merged by Dzahn:
gerrit: mv standard incl to role, rm duplicate firewall

https://gerrit.wikimedia.org/r/314768

gerritbot added a comment.Oct 11 2016, 9:14 PM

Change 315418 had a related patch set uploaded (by Dzahn):
gerrit: remove lead from site.pp, adjust comment

https://gerrit.wikimedia.org/r/315418

RobH unsubscribed.Oct 11 2016, 9:56 PM
gerritbot added a comment.Oct 11 2016, 11:06 PM

Change 315418 merged by Dzahn:
gerrit: remove lead from site.pp, adjust comment

https://gerrit.wikimedia.org/r/315418

Dzahn closed this task as Resolved.Oct 11 2016, 11:09 PM
Dzahn updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits