Page MenuHomePhabricator
Create Task
Maniphest T205846

Move users from stat1005 to stat1007
Closed, ResolvedPublic21 Estimated Story Points
Actions

Description

Once T203852 is completed, we'll have stat1007 ready to be used. List of things to do:

  • rsync /srv from stat1005 to stat1007
  • Announce the plan to move away users from stat1005 to stat1007 for T148843
  • Add motd to stat1005 with a brief explanation of the migration and a link to this task
  • Move report updater
  • Move dumps
  • Move ::statistics::wmde
  • Move ::statistics::discovery
  • Move ::geoip::data::archive
  • Move @ezachte's repositories
  • Make a list of users using cron and contact them
  • Fix documentation in Wikitech
  • Remove access to users from stat1005

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

elukey created this task.Oct 1 2018, 12:04 PM
elukey triaged this task as Medium priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 1 2018, 12:04 PM
Ottomata raised the priority of this task from Medium to High.Oct 4 2018, 5:09 PM
Ottomata moved this task from Incoming to Operational Excellence on the Analytics board.
gerritbot subscribed.Oct 24 2018, 12:43 PM

Change 469422 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add role::statistics::private to stat1007

https://gerrit.wikimedia.org/r/469422

gerritbot added a project: Patch-For-Review.Oct 24 2018, 12:43 PM
gerritbot added a comment.Oct 24 2018, 2:08 PM

Change 469422 merged by Elukey:
[operations/puppet@production] Add role::statistics::private to stat1007

https://gerrit.wikimedia.org/r/469422

elukey claimed this task.Oct 24 2018, 2:45 PM
elukey added a project: Analytics-Kanban.
elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.
elukey updated the task description. (Show Details)Oct 24 2018, 2:50 PM
gerritbot added a comment.Oct 24 2018, 4:35 PM

Change 469458 had a related patch set uploaded (by Elukey; owner: Elukey):
[analytics/refinery/scap@master] Add stat1007 to the list of refinery targets

https://gerrit.wikimedia.org/r/469458

gerritbot added a comment.Oct 24 2018, 4:36 PM

Change 469458 merged by Elukey:
[analytics/refinery/scap@master] Add stat1007 to the list of refinery targets

https://gerrit.wikimedia.org/r/469458

elukey added subscribers: mpopov, Addshore.Oct 25 2018, 9:41 AM

Hi @mpopov! We are in the process of migrating everybody from stat1005 to stat1007 (not announced yet for users) but I am wondering if I could move the statistics::discovery stuff beforehand (afaics the wikimedia-discovery-golden cron and related things). What do you think?

Hi @Addshore! Same question for you but for ::statistics::wmde :)

elukey updated the task description. (Show Details)Oct 25 2018, 9:43 AM
mpopov added a comment.Oct 25 2018, 4:32 PM
In T205846#4694101, @elukey wrote:

Hi @mpopov! We are in the process of migrating everybody from stat1005 to stat1007 (not announced yet for users) but I am wondering if I could move the statistics::discovery stuff beforehand (afaics the wikimedia-discovery-golden cron and related things). What do you think?

Sounds good but I'll need to manually install the required R packages since even though the ability to do that via Puppet exists AFAIC we're still not allowed to use it on production machines, so wikimedia-discovery-golden won't be able to run until that I do that and test that the latest versions of packages still work.

Without disabling the cron on stat1005, are you able to provision stat1007 so it has everything else in statistics::discovery? I'll need to login to stat1007 and install R packages into <$::statistics::working_path>/discovery/r-library (see discovery.pp#L14)

The reason that golden cron needs to be turned off and turned on in the same day is because there's a query that collects Wikimedia Maps usage on wikis and that can't be backfilled since the query operates on the current state of the content db.

gerritbot added a comment.Oct 26 2018, 5:26 AM

Change 469826 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::statistics::private: move reportupdater to stat1007

https://gerrit.wikimedia.org/r/469826

gerritbot added a comment.Oct 26 2018, 5:30 AM

Change 469826 merged by Elukey:
[operations/puppet@production] role::statistics::private: move reportupdater to stat1007

https://gerrit.wikimedia.org/r/469826

elukey updated the task description. (Show Details)Oct 26 2018, 5:55 AM
gerritbot added a comment.Oct 26 2018, 9:32 AM

Change 469854 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::statistics::private: move geoip archive to stat1007

https://gerrit.wikimedia.org/r/469854

gerritbot added a comment.Oct 26 2018, 9:45 AM

Change 469854 merged by Elukey:
[operations/puppet@production] profile::statistics::private: move geoip archive to stat1007

https://gerrit.wikimedia.org/r/469854

elukey updated the task description. (Show Details)Oct 26 2018, 9:48 AM
gerritbot added a comment.Oct 26 2018, 9:59 AM

Change 469855 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::statistics::private: move geoip to stat1007 - p2

https://gerrit.wikimedia.org/r/469855

gerritbot added a comment.Oct 26 2018, 10:00 AM

Change 469855 merged by Elukey:
[operations/puppet@production] profile::statistics::private: move geoip to stat1007 - p2

https://gerrit.wikimedia.org/r/469855

elukey added a subtask: T208028: Geoip data archive repository cause puppet to run for minutes.Oct 26 2018, 1:42 PM
• Nuria closed subtask T208028: Geoip data archive repository cause puppet to run for minutes as Resolved.Oct 30 2018, 7:14 PM
gerritbot added a comment.Oct 31 2018, 1:53 PM

Change 470837 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Move statistics::discovery to stat1007

https://gerrit.wikimedia.org/r/470837

gerritbot added a comment.Oct 31 2018, 1:54 PM

Change 470837 merged by Elukey:
[operations/puppet@production] Move statistics::discovery to stat1007

https://gerrit.wikimedia.org/r/470837

elukey updated the task description. (Show Details)Oct 31 2018, 1:56 PM
elukey updated the task description. (Show Details)
elukey added a subscriber: • ezachte.
elukey updated the task description. (Show Details)Oct 31 2018, 1:58 PM
elukey updated the task description. (Show Details)Oct 31 2018, 2:11 PM

@mpopov all moved, let me know if anything is missing! Cron removed from stat1005 too :)

gerritbot added a comment.Oct 31 2018, 2:13 PM

Change 470840 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Move ::statistics::wmde to stat1007

https://gerrit.wikimedia.org/r/470840

gerritbot added a comment.Oct 31 2018, 2:14 PM

Change 470840 merged by Elukey:
[operations/puppet@production] Move ::statistics::wmde to stat1007

https://gerrit.wikimedia.org/r/470840

elukey updated the task description. (Show Details)Oct 31 2018, 2:44 PM
elukey updated the task description. (Show Details)
elukey updated the task description. (Show Details)Oct 31 2018, 3:03 PM
gerritbot added a comment.Oct 31 2018, 3:16 PM

Change 470850 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::statistics::private: add deprecation motd to stat1005

https://gerrit.wikimedia.org/r/470850

gerritbot added a comment.Oct 31 2018, 3:18 PM

Change 470850 merged by Elukey:
[operations/puppet@production] role::statistics::private: add deprecation motd to stat1005

https://gerrit.wikimedia.org/r/470850

elukey updated the task description. (Show Details)Oct 31 2018, 3:24 PM
elukey added a subtask: T208490: Please copy the wmde* and wdcm directories to stat1007 /srv/published-datasets.Nov 1 2018, 1:52 PM
GoranSMilovanovic closed subtask T208490: Please copy the wmde* and wdcm directories to stat1007 /srv/published-datasets as Resolved.Nov 1 2018, 10:08 PM
elukey updated the task description. (Show Details)Nov 2 2018, 7:50 AM
gerritbot added a comment.Nov 2 2018, 8:08 AM

Change 471216 had a related patch set uploaded (by Elukey; owner: Elukey):
[analytics/refinery/scap@master] Remove stat1005 and promote stat1007 as canary

https://gerrit.wikimedia.org/r/471216

gerritbot added a comment.Nov 2 2018, 8:09 AM

Change 471216 merged by Elukey:
[analytics/refinery/scap@master] Remove stat1005 and promote stat1007 as canary

https://gerrit.wikimedia.org/r/471216

elukey updated the task description. (Show Details)Nov 2 2018, 8:20 AM
elukey added a comment.Nov 20 2018, 1:59 PM

Update: we are still waiting for the last people moving out of stat1005, so access is still allowed but we are not doing any more rsyncs of home dirs to stat1007 (to avoid impacting people already moved there). If you need to copy data over, please ping me in this task so we can coordinate.

gerritbot added a comment.Dec 6 2018, 2:51 PM

Change 478020 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Move remaining stat1005 references to stat1007

https://gerrit.wikimedia.org/r/478020

gerritbot added a comment.Dec 11 2018, 1:31 PM

Change 478020 merged by Elukey:
[operations/puppet@production] Move remaining stat1005 references to stat1007

https://gerrit.wikimedia.org/r/478020

gerritbot added a comment.Dec 11 2018, 1:41 PM

Change 478933 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Apply role::statistics::gpu to stat1005

https://gerrit.wikimedia.org/r/478933

gerritbot added a comment.Dec 11 2018, 1:45 PM

Change 478933 merged by Elukey:
[operations/puppet@production] Apply role::statistics::gpu to stat1005

https://gerrit.wikimedia.org/r/478933

elukey updated the task description. (Show Details)Dec 11 2018, 2:00 PM
elukey updated the task description. (Show Details)
elukey added a comment.Dec 21 2018, 6:55 AM

The last item should be to reimage stat1005 to a clean state, so we can avoid worrying about PII data in there etc.. I'll do it after holidays :)

ops-monitoring-bot subscribed.Jan 14 2019, 3:23 PM

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['stat1005.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201901141523_elukey_253129.log.

Stashbot subscribed.Jan 14 2019, 3:26 PM

Mentioned in SAL (#wikimedia-analytics) [2019-01-14T15:26:22Z] <elukey> reimage stat1005 - T205846

ops-monitoring-bot added a comment.Jan 14 2019, 3:59 PM

Completed auto-reimage of hosts:

['stat1005.eqiad.wmnet']

and were ALL successful.

elukey moved this task from In Progress to Done on the Analytics-Kanban board.Jan 14 2019, 4:01 PM
elukey set the point value for this task to 21.
• Nuria closed this task as Resolved.Jan 17 2019, 1:50 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits