Page MenuHomePhabricator

Move users from stat1005 to stat1007
Closed, ResolvedPublic21 Story Points

Description

Once T203852 is completed, we'll have stat1007 ready to be used. List of things to do:

  • rsync /srv from stat1005 to stat1007
  • Announce the plan to move away users from stat1005 to stat1007 for T148843
  • Add motd to stat1005 with a brief explanation of the migration and a link to this task
  • Move report updater
  • Move dumps
  • Move ::statistics::wmde
  • Move ::statistics::discovery
  • Move ::geoip::data::archive
  • Move @ezachte's repositories
  • Make a list of users using cron and contact them
  • Fix documentation in Wikitech
  • Remove access to users from stat1005

Event Timeline

elukey triaged this task as Normal priority.Oct 1 2018, 12:04 PM
elukey created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 1 2018, 12:04 PM
Ottomata raised the priority of this task from Normal to High.Oct 4 2018, 5:09 PM
Ottomata moved this task from Incoming to Operational Excellence on the Analytics board.

Change 469422 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add role::statistics::private to stat1007

https://gerrit.wikimedia.org/r/469422

Change 469422 merged by Elukey:
[operations/puppet@production] Add role::statistics::private to stat1007

https://gerrit.wikimedia.org/r/469422

elukey claimed this task.Oct 24 2018, 2:45 PM
elukey added a project: Analytics-Kanban.
elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.
elukey updated the task description. (Show Details)Oct 24 2018, 2:50 PM

Change 469458 had a related patch set uploaded (by Elukey; owner: Elukey):
[analytics/refinery/scap@master] Add stat1007 to the list of refinery targets

https://gerrit.wikimedia.org/r/469458

Change 469458 merged by Elukey:
[analytics/refinery/scap@master] Add stat1007 to the list of refinery targets

https://gerrit.wikimedia.org/r/469458

Hi @mpopov! We are in the process of migrating everybody from stat1005 to stat1007 (not announced yet for users) but I am wondering if I could move the statistics::discovery stuff beforehand (afaics the wikimedia-discovery-golden cron and related things). What do you think?

Hi @Addshore! Same question for you but for ::statistics::wmde :)

elukey updated the task description. (Show Details)Oct 25 2018, 9:43 AM

Hi @mpopov! We are in the process of migrating everybody from stat1005 to stat1007 (not announced yet for users) but I am wondering if I could move the statistics::discovery stuff beforehand (afaics the wikimedia-discovery-golden cron and related things). What do you think?

Sounds good but I'll need to manually install the required R packages since even though the ability to do that via Puppet exists AFAIC we're still not allowed to use it on production machines, so wikimedia-discovery-golden won't be able to run until that I do that and test that the latest versions of packages still work.

Without disabling the cron on stat1005, are you able to provision stat1007 so it has everything else in statistics::discovery? I'll need to login to stat1007 and install R packages into <$::statistics::working_path>/discovery/r-library (see discovery.pp#L14)

The reason that golden cron needs to be turned off and turned on in the same day is because there's a query that collects Wikimedia Maps usage on wikis and that can't be backfilled since the query operates on the current state of the content db.

Change 469826 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::statistics::private: move reportupdater to stat1007

https://gerrit.wikimedia.org/r/469826

Change 469826 merged by Elukey:
[operations/puppet@production] role::statistics::private: move reportupdater to stat1007

https://gerrit.wikimedia.org/r/469826

elukey updated the task description. (Show Details)Oct 26 2018, 5:55 AM

Change 469854 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::statistics::private: move geoip archive to stat1007

https://gerrit.wikimedia.org/r/469854

Change 469854 merged by Elukey:
[operations/puppet@production] profile::statistics::private: move geoip archive to stat1007

https://gerrit.wikimedia.org/r/469854

elukey updated the task description. (Show Details)Oct 26 2018, 9:48 AM

Change 469855 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::statistics::private: move geoip to stat1007 - p2

https://gerrit.wikimedia.org/r/469855

Change 469855 merged by Elukey:
[operations/puppet@production] profile::statistics::private: move geoip to stat1007 - p2

https://gerrit.wikimedia.org/r/469855

Change 470837 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Move statistics::discovery to stat1007

https://gerrit.wikimedia.org/r/470837

Change 470837 merged by Elukey:
[operations/puppet@production] Move statistics::discovery to stat1007

https://gerrit.wikimedia.org/r/470837

elukey updated the task description. (Show Details)Oct 31 2018, 1:56 PM
elukey updated the task description. (Show Details)
elukey added a subscriber: ezachte.
elukey updated the task description. (Show Details)Oct 31 2018, 1:58 PM
elukey updated the task description. (Show Details)Oct 31 2018, 2:11 PM

@mpopov all moved, let me know if anything is missing! Cron removed from stat1005 too :)

Change 470840 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Move ::statistics::wmde to stat1007

https://gerrit.wikimedia.org/r/470840

Change 470840 merged by Elukey:
[operations/puppet@production] Move ::statistics::wmde to stat1007

https://gerrit.wikimedia.org/r/470840

elukey updated the task description. (Show Details)Oct 31 2018, 2:44 PM
elukey updated the task description. (Show Details)
elukey updated the task description. (Show Details)Oct 31 2018, 3:03 PM

Change 470850 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::statistics::private: add deprecation motd to stat1005

https://gerrit.wikimedia.org/r/470850

Change 470850 merged by Elukey:
[operations/puppet@production] role::statistics::private: add deprecation motd to stat1005

https://gerrit.wikimedia.org/r/470850

elukey updated the task description. (Show Details)Oct 31 2018, 3:24 PM
elukey updated the task description. (Show Details)Nov 2 2018, 7:50 AM

Change 471216 had a related patch set uploaded (by Elukey; owner: Elukey):
[analytics/refinery/scap@master] Remove stat1005 and promote stat1007 as canary

https://gerrit.wikimedia.org/r/471216

Change 471216 merged by Elukey:
[analytics/refinery/scap@master] Remove stat1005 and promote stat1007 as canary

https://gerrit.wikimedia.org/r/471216

elukey updated the task description. (Show Details)Nov 2 2018, 8:20 AM

Update: we are still waiting for the last people moving out of stat1005, so access is still allowed but we are not doing any more rsyncs of home dirs to stat1007 (to avoid impacting people already moved there). If you need to copy data over, please ping me in this task so we can coordinate.

Change 478020 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Move remaining stat1005 references to stat1007

https://gerrit.wikimedia.org/r/478020

Change 478020 merged by Elukey:
[operations/puppet@production] Move remaining stat1005 references to stat1007

https://gerrit.wikimedia.org/r/478020

Change 478933 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Apply role::statistics::gpu to stat1005

https://gerrit.wikimedia.org/r/478933

Change 478933 merged by Elukey:
[operations/puppet@production] Apply role::statistics::gpu to stat1005

https://gerrit.wikimedia.org/r/478933

elukey updated the task description. (Show Details)Dec 11 2018, 2:00 PM
elukey updated the task description. (Show Details)

The last item should be to reimage stat1005 to a clean state, so we can avoid worrying about PII data in there etc.. I'll do it after holidays :)

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['stat1005.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201901141523_elukey_253129.log.

Mentioned in SAL (#wikimedia-analytics) [2019-01-14T15:26:22Z] <elukey> reimage stat1005 - T205846

Completed auto-reimage of hosts:

['stat1005.eqiad.wmnet']

and were ALL successful.

elukey moved this task from In Progress to Done on the Analytics-Kanban board.Jan 14 2019, 4:01 PM
elukey set the point value for this task to 21.
Nuria closed this task as Resolved.Jan 17 2019, 1:50 PM