Page MenuHomePhabricator
Create Task
Maniphest T327689

Use rsync instead of NFS for wdqs data reload cookbook
Closed, DeclinedPublic
Actions

Description

In T222349 , we determined that there was a speed and stale data problem with reloading WDQS via HTTP. In T325114 , we decided to use NFS to avoid the problem. However, we have been advised not to rely on NFS when alternatives exist (rsync). Creating this ticket to:

  • Update the WDQS data reload cookbook to use rsync
  • Remove current mounts/ferm rules etc needed for NFS

Related Objects

Event Timeline

bking created this task.Jan 23 2023, 7:44 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 23 2023, 7:44 PM
bking mentioned this in T222349: Do not rate limit dumps from internal network.Jan 23 2023, 7:47 PM
ArielGlenn subscribed.Jan 24 2023, 11:31 AM
Gehel triaged this task as High priority.Jan 30 2023, 4:32 PM
Gehel moved this task from needs triage to Ops / SRE on the Discovery-Search board.
Gehel subscribed.

We need to understand better the failure modes around data reload before moving forward on this. See T323096

Gehel edited projects, added Wikidata-Query-Service; removed Discovery-Search.Mar 16 2023, 7:30 PM
Gehel moved this task from Incoming to Operations/SRE on the Wikidata-Query-Service board.
Maintenance_bot added a project: Wikidata.Mar 16 2023, 7:45 PM
Arian_Bozorg moved this task from incoming to monitoring on the Wikidata board.Mar 24 2023, 10:02 AM
bking closed this task as Declined.Thu, Apr 4, 4:57 PM
bking moved this task from monitoring to needs discussion or investigation on the Wikidata board.

This ticket resurfaced due to recent discussions around the data reload process (see T349069 and T359062 , among others). At this point, it seems unlikely that we'll change to rsync; it's more likely we'll use a different process that doesn't involve the dumps server. As such, I'm closing this one out.

bking added a project: Data-Platform-SRE.Thu, Apr 4, 5:01 PM
bking edited projects, added Data-Platform-SRE (2024.03.25 - 2024.04.14); removed Data-Platform-SRE.
bking moved this task from Backlog to Done on the Data-Platform-SRE (2024.03.25 - 2024.04.14) board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL