Page MenuHomePhabricator

Beta Cluster uploads (new and viewing existing files/thumbnails, including captchas) broken due to WMF Labs NFS outage
Closed, ResolvedPublic

Description

The UploadWizard smoke test fails due to the NFS outage

https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/

======================================================================
ERROR: testUploadImageUsingWizardWorkflow (__main__.TestUploadWizardAPICalls)
Test that basic api calls used by the UploadWizard are working as expected
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/api/upload-wizard_tests.py", line 146, in testUploadImageUsingWizardWorkflow
  File "tests/api/upload-wizard_tests.py", line 81, in uploadImage
  File "/mnt/jenkins-workspace/workspace/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/DEV/local/lib/python2.7/site-packages/wikitools/api.py", line 165, in query
    raise APIError(data['error']['code'], data['error']['info'])
APIError: (u'stashfilestorage', u'Could not store upload in the stash: Stashing temporary file failed: UploadStashFileException Error storing file in \'/tmp/8blCX4\': Could not create directory "mwstore://local-backend/local-temp/d/d5".')

Event Timeline

hashar created this task.Jun 18 2015, 3:16 PM
hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added subscribers: zhuyifei1999, Krenair, hashar and 2 others.
Krenair renamed this task from UploadWizard-api-commons.wikimedia.beta.wmflabs.org smoke test fails to Beta cluster uploads (new and viewing existing files/thumbnails) broken due to NFS.Jun 18 2015, 3:31 PM
Krenair set Security to None.

Also captcha stuff (not sure if this should be a new ticket or not) would be at /data/project/upload7/private/captcha
MediaWiki exception logs are full of "Ran out of captcha images"

Bugreporter triaged this task as High priority.Jun 18 2015, 4:32 PM
Bugreporter added a subscriber: Bugreporter.

Also captcha stuff (not sure if this should be a new ticket or not) would be at /data/project/upload7/private/captcha
MediaWiki exception logs are full of "Ran out of captcha images"

Tasks are cheap! UploadWizard stash apparently tries to write to /tmp which is probably. But maybe that is under /data/project as well.

greg renamed this task from Beta cluster uploads (new and viewing existing files/thumbnails) broken due to NFS to Beta Cluster uploads (new and viewing existing files/thumbnails) broken due to WMF Labs NFS outage.Jun 19 2015, 4:03 PM
Krenair renamed this task from Beta Cluster uploads (new and viewing existing files/thumbnails) broken due to WMF Labs NFS outage to Beta Cluster uploads (new and viewing existing files/thumbnails, including captchas) broken due to WMF Labs NFS outage.Jun 19 2015, 4:09 PM
thcipriani closed this task as Resolved.Jun 22 2015, 7:35 PM
thcipriani claimed this task.
thcipriani added a subscriber: thcipriani.

All tests passing now

By NFS being up, yes... This task should probably remain open until it no longer depends on NFS or is closed as a duplicate of something like T64835 or T84950.