Page MenuHomePhabricator
Create Task
Maniphest T188726

make sure all datasets in xmldatadumps/public/other on dataset1001 are accounted for on new labs boxes
Closed, ResolvedPublic
Actions

Description

Going to list here everything in that subdirectory, indicate whether there it's old static data or there's a job that updates it from somewhere.
As each of the items that gets updates has its job moved, it can be checked off.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
fix up source dir for sync of phab dumps to public webserveroperations/puppetproduction+1 -1
pull phabricator dumps from phab server to dumps web serveroperations/puppetproduction+42 -10
analytics: Allow labstore1006|7 to rsync from stat*operations/puppetproduction+2 -0
dumps: Fold distribution related rsync config to profile/ pathoperations/puppetproduction+46 -107
dumps: Split up rsync config to base, mirrors, and datasetsoperations/puppetproduction+41 -21
dumps: Move rsyncer to distribution profile path and renameoperations/puppetproduction+3 -4
dumps: Switch rsyncer profile to use host settings from hieraoperations/puppetproduction+8 -5
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolvedbd808T166402 Program 7 Outcome 3: data services
 ResolvedArielGlennT182540 get datset1001, ms1001 ready for decommission
 Resolved• madhuvishyT168486 Migrate customer-facing Dumps endpoints to Cloud Services
 Resolved• madhuvishyT171508 Investigate and implement alternative for showmount based check at instance boot time
 Resolved• madhuvishyT174590 Revert patch that adds a temporary exception to the block-for-export check for the testlabs project
 Resolved• madhuvishyT171539 Puppetize and setup initial lvms and directory structures for labstore1006|7
 ResolvedArielGlennT171541 Setup periodic rsync jobs from dumps generation hosts to labstore1006|7
 ResolvedArielGlennT188726 make sure all datasets in xmldatadumps/public/other on dataset1001 are accounted for on new labs boxes
 Resolved• ezachteT189283 Replace cron jobs from EZachte's home directory on stat1005 with rsync fetches
 ResolvedArielGlennT189284 Stop serving slowparse logs from dumps distribution servers
 Resolved• madhuvishyT181431 Setup NFS on dumps servers
 Resolved• madhuvishyT171540 Figure out how NFS failovers will work for the dumps servers - labstore1006|7
 Resolved• madhuvishyT188073 Maintain symlinks for WMCS NFS Dumps users with new directory structure
 Resolved• madhuvishyT188641 Set up the web service that serves dumps.wikimedia.org
 Resolved• madhuvishyT188642 Set up labstore1006|7 as the source for rsync mirror sites
 Resolved• madhuvishyT188643 Migrate Dumps WMCS NFS users from labstore1003 to labstore1006/7
 Resolved• madhuvishyT188644 Migrate the stat* mount from dataset1001 to labstore1006/7
 Resolved• madhuvishyT188645 Get all the rsync mirror sites to to switch over to labstore1006,7
 Resolved• madhuvishyT188646 Point dumps.wikimedia.org to labstore1006/7
 Resolved• madhuvishyT185101 Labstore1006/7 profile for meltdown kernel
 Resolved• madhuvishyT188647 Announce/Communicate dumps migration to labstore1006|7 to stakeholders
Restricted Task

Event Timeline

ArielGlenn created this task.Mar 2 2018, 12:32 PM
ArielGlenn triaged this task as Medium priority.
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptMar 2 2018, 12:32 PM
ArielGlenn added a comment.EditedMar 2 2018, 1:16 PM
DIRECTORYSTATICJOBMOVED
analyticsYpuppet-
androidY--
articlefeedbackY--
bugzillaY--
categoriesrdf"misc" dumpsame job
cirrussearch"misc" dumpsame job
clickstreamdumps::web::fetches::statssame job
contenttranslation"misc" dumpsame job
diffdbY--
docsY--
experimentalY--
fundraisingY--
fundraising.20120213Y--
globalblocks"misc" dumpsame job
imageinfo"misc" dumpsame job
imageinfo_archiveY--
incr"misc" dumpsame job
incrsY--
index.htmlYpuppet-
index.html.20120301Y--
iOSY--
kiwixdumps::web::fetches::kiwixsame job
mediaEZachtedumps::web::fetches::stat_dumps
mediacountsdumps::web::fetches::statssame job
mediatitles"misc" dumpssame job
mediawikiY--
miscprofile::phabricator::maindumps::web::fetches::phab
multistreamY--
openzimY--
pagecounts-all-sitesY--
pagecounts-ezEZachtedumps::web::fetches::stat_dumps
pagecounts-newY--
pagecounts-rawY--
pagetitles"misc" dumpssame job
pageviewsdumps::web::fetches::statssame job
PlayBookY--
potyY--
scansetY--
searchY--
shorturlsY--
slow-parserole::logging::mediawiki::udp2logdiscontinued
staticY--
static_html_dumpsY--
surveysY--
testfilesY--
toolsY--
unique_devicesdumps::web::fetches::statssame job
wepY--
wikibase"misc" dumpssame job
wikichallengeY--
wikidata"misc" dumpssame job
wikitechdumps::web::fetches::wikitech_dumpssame job
win8Y--
  • "misc" dumps are everything except xml/sql dumps, run off of the 'misc dumps cron' snapshot host. they should already be available on labstore6,7.
  • "EZachte": cron jobs from EZachte's home directory on stat1005, see separate list below
  • "puppet": these html files are managed by puppet.
  • "dumps::web::fetches::stats": pulls via rsync from stat1005.eqiad.wmnet::hdfs-archive
  • "dumps::web::fetches::kiwix": pulls via rsync from download.kiwix.org
  • "dumps::web::fetches::wikitech_dumps": pulls via wget from https://wikitech.wikimedia.org/dumps/
  • "profile::phabricator::main": rsync from phab1001
  • "role::logging::mediawiki::udp2log": rsync from mwlog1001.eqiad.wmnet
  • "-" indicates there's no job, as these folders are not updated.

Notes on EZachte's jobs

  • wikistats/dumps/bash/count_editors_yoy.sh -- rsyncs to dataset1001=dataset1001.wikimedia.org::pagecounts-ez/wikistats/
  • wikistats/dumps/bash/rsync.sh -- rsyncs to dataset1001::pagecounts-ez/projectcounts
  • wikistats/image_sets/bash/publish_zips.sh -- rsyncs to dataset1001.wikimedia.org::media/contest_winners/WLM
  • wikistats/dammit.lt/bash/dammit_compact_daily.sh -- rsyncs to dataset1001.wikimedia.org::pagecounts-ez/merged/
  • wikistats/dammit.lt/bash/dammit_compact_monthly.sh -- rsyncs to dataset1001.wikimedia.org::pagecounts-ez/merged/
  • wikistats/dammit.lt/bash/dammit_projectviews_monthly.sh -- rsyncs to dataset1001.wikimedia.org::pagecounts-ez/projectviews
  • wikistats/dammit.lt/bash/dammit_published_merged.sh -- rsyncs to dataset1001.wikimedia.org::pagecounts-ez/merged/dataset1001.wikimedia.org::pagecounts-ez
  • wikistats/dammit.lt/bash/t2.sh -- rsyncs to dataset1001.wikimedia.org::pagecounts-ez/merged/
  • wikistats/dammit.lt/bash/t.sh -- rsyncs to dataset1001.wikimedia.org::pagecounts-ez/merged/
  • wikistats/dammit.lt/bash/dammit_sync.sh -- rsyncs to

Marking off which jobs have been moved or terminated, there is still at least one that has not been.
The job for other/media explicitly covers only the contest-winners subdirectory.

bd808 moved this task from Backlog to Dumps on the Data-Services board.Mar 2 2018, 1:42 PM
• madhuvishy mentioned this in T171541: Setup periodic rsync jobs from dumps generation hosts to labstore1006|7.Mar 5 2018, 6:01 PM
gerritbot added a comment.Mar 6 2018, 11:00 PM

Change 416863 had a related patch set uploaded (by Madhuvishy; owner: Madhuvishy):
[operations/puppet@production] dumps: Switch rsyncer profile to use host settings from hiera

https://gerrit.wikimedia.org/r/416863

gerritbot added a project: Patch-For-Review.Mar 6 2018, 11:00 PM
gerritbot added a comment.Mar 6 2018, 11:12 PM

Change 416866 had a related patch set uploaded (by Madhuvishy; owner: Madhuvishy):
[operations/puppet@production] dumps: Move rsyncer to distribution profile path and rename

https://gerrit.wikimedia.org/r/416866

gerritbot added a comment.Mar 6 2018, 11:29 PM

Change 416869 had a related patch set uploaded (by Madhuvishy; owner: Madhuvishy):
[operations/puppet@production] dumps: Split up rsync config to base, mirrors, and datasets

https://gerrit.wikimedia.org/r/416869

gerritbot added a comment.Mar 7 2018, 7:31 AM

Change 416901 had a related patch set uploaded (by Madhuvishy; owner: Madhuvishy):
[operations/puppet@production] dumps: Fold distribution related rsync config to profile/ path

https://gerrit.wikimedia.org/r/416901

gerritbot added a comment.Mar 7 2018, 12:49 PM

Change 416863 merged by ArielGlenn:
[operations/puppet@production] dumps: Switch rsyncer profile to use host settings from hiera

https://gerrit.wikimedia.org/r/416863

gerritbot added a comment.Mar 7 2018, 1:05 PM

Change 416866 merged by ArielGlenn:
[operations/puppet@production] dumps: Move rsyncer to distribution profile path and rename

https://gerrit.wikimedia.org/r/416866

gerritbot added a comment.Mar 7 2018, 1:21 PM

Change 416869 merged by ArielGlenn:
[operations/puppet@production] dumps: Split up rsync config to base, mirrors, and datasets

https://gerrit.wikimedia.org/r/416869

gerritbot added a comment.Mar 7 2018, 1:46 PM

Change 416901 merged by ArielGlenn:
[operations/puppet@production] dumps: Fold distribution related rsync config to profile/ path

https://gerrit.wikimedia.org/r/416901

• madhuvishy mentioned this in T168486: Migrate customer-facing Dumps endpoints to Cloud Services.Mar 7 2018, 10:01 PM
• madhuvishy mentioned this in T188647: Announce/Communicate dumps migration to labstore1006|7 to stakeholders.
gerritbot added a comment.Mar 16 2018, 5:13 PM

Change 420078 had a related patch set uploaded (by Madhuvishy; owner: Madhuvishy):
[operations/puppet@production] analytics: Allow labstore1006|7 to rsync from stat*

https://gerrit.wikimedia.org/r/420078

gerritbot added a comment.Mar 16 2018, 5:15 PM

Change 420078 merged by Madhuvishy:
[operations/puppet@production] analytics: Allow labstore1006|7 to rsync from stat*

https://gerrit.wikimedia.org/r/420078

• madhuvishy added a comment.Mar 20 2018, 2:08 AM

Note: Slowparse in other/ logs are being deprecated (T189284)

ArielGlenn closed subtask T189284: Stop serving slowparse logs from dumps distribution servers as Resolved.Mar 29 2018, 2:41 PM
• ezachte closed subtask T189283: Replace cron jobs from EZachte's home directory on stat1005 with rsync fetches as Resolved.Apr 17 2018, 6:31 PM
• madhuvishy closed this task as Resolved.Apr 20 2018, 3:28 AM
• madhuvishy claimed this task.
ArielGlenn reopened this task as Open.Apr 25 2018, 12:45 PM
ArielGlenn added a comment.Apr 25 2018, 1:01 PM

T188149 is open and there's a changeset (wrong for our purposes but at least gives us the info we need) here: https://gerrit.wikimedia.org/r/#/c/428540/

ArielGlenn mentioned this in T188149: Phlogiston reports don't have new data since mid-February.Apr 25 2018, 8:10 PM
gerritbot added a comment.Apr 26 2018, 2:05 PM

Change 429197 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] pull phabricator dumps from phab server to dumps web server

https://gerrit.wikimedia.org/r/429197

bd808 reassigned this task from • madhuvishy to ArielGlenn.Apr 26 2018, 8:01 PM
bd808 subscribed.

If assigning this to you is the wrong thing to do @ArielGlenn, let me know what help is needed and I'll try to round up another person.

gerritbot added a comment.Apr 29 2018, 5:26 PM

Change 429197 merged by ArielGlenn:
[operations/puppet@production] pull phabricator dumps from phab server to dumps web server

https://gerrit.wikimedia.org/r/429197

gerritbot added a comment.Apr 29 2018, 7:08 PM

Change 429666 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] fix up source dir for sync of phab dumps to public webserver

https://gerrit.wikimedia.org/r/429666

gerritbot added a comment.Apr 29 2018, 7:16 PM

Change 429666 merged by ArielGlenn:
[operations/puppet@production] fix up source dir for sync of phab dumps to public webserver

https://gerrit.wikimedia.org/r/429666

ArielGlenn closed this task as Resolved.Apr 30 2018, 6:59 AM

All datasets are now accounted for. Closing for good!

ArielGlenn moved this task from Backlog to Done on the Datasets-General-or-Unknown board.Jul 2 2018, 12:36 PM
ArielGlenn moved this task from Short-term backlog to Done on the User-ArielGlenn board.Aug 7 2018, 10:00 AM
Hydriz mentioned this in T117956: Make mediacounts available in Wikimedia Labs.Nov 28 2018, 10:10 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits