Page MenuHomePhabricator

Use rsync instead of NFS for wdqs data reload cookbook
Closed, DeclinedPublic


In T222349 , we determined that there was a speed and stale data problem with reloading WDQS via HTTP. In T325114 , we decided to use NFS to avoid the problem. However, we have been advised not to rely on NFS when alternatives exist (rsync). Creating this ticket to:

  • Update the WDQS data reload cookbook to use rsync
  • Remove current mounts/ferm rules etc needed for NFS

Event Timeline

Gehel triaged this task as High priority.Jan 30 2023, 4:32 PM
Gehel moved this task from needs triage to Ops / SRE on the Discovery-Search board.
Gehel subscribed.

We need to understand better the failure modes around data reload before moving forward on this. See T323096

bking moved this task from monitoring to needs discussion or investigation on the Wikidata board.

This ticket resurfaced due to recent discussions around the data reload process (see T349069 and T359062 , among others). At this point, it seems unlikely that we'll change to rsync; it's more likely we'll use a different process that doesn't involve the dumps server. As such, I'm closing this one out.