Page MenuHomePhabricator

[Explorer] Fix database storage exceedance
Open, HighPublic

Description

Context:
After switching the explorer to wmcloud, we are still running into storage issues every few weeks. We are manually clearing disk space from log data to make enough space for new data and Wikibase registrations coming in. Then we have to manually check log files elsewhere (i.e. Docker) to identify new Wikibases.

Problem:
We sometimes lose track of new Wikibase registrations because there is not enough disk space left on wmcloud. Assumption: The database is constantly flooded with unnecessary log data.

Goal:

  • Identify what is causing the storage exceedance - scope - try to work out how to get Leif data this month. If a deeper investigation is warranted, we can decide this at the end of the spike.
  • Define solution(s) to navigate or solve the issue completely - create separate tickets if required.

Timebox
1 day

Details

Other Assignee
Jacob_WMDE

Event Timeline

Leif_WMDE triaged this task as High priority.
Leif_WMDE moved this task from Preparation to Ready for Sprint on the Wikibase Suite Team board.

Update from this week:
We are running into the same issue again. I don't see any new instances and scrapes are failing. @RickiJay-WMDE Can you investigate this in the next sprint and potentially solve the issue? LMK: If you need product input on anything.

Lucyfediachambers renamed this task from [Explorer] Fix database storage exceedance to [SPIKE - check 28/09][Explorer] Fix database storage exceedance.Thu, Sep 24, 9:28 AM
Lucyfediachambers updated the task description. (Show Details)

Some more context @RickiJay-WMDE :

Example:

  • run mutation 3 ( updateAllQuantityData)
  • results in:
{
  "data": {
    "updateAllQuantityData": {
      "failure": 2385,
      "success": 72,
      "total": 2457
    }
  }
}
Lucyfediachambers renamed this task from [SPIKE - check 28/09][Explorer] Fix database storage exceedance to [Explorer] Fix database storage exceedance.Mon, Sep 28, 9:26 AM