Page MenuHomePhabricator

Beta Cluster uploads (new and viewing existing files/thumbnails, including captchas) broken due to WMF Labs NFS outage
Closed, ResolvedPublic

Description

The UploadWizard smoke test fails due to the NFS outage

https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/

======================================================================
ERROR: testUploadImageUsingWizardWorkflow (__main__.TestUploadWizardAPICalls)
Test that basic api calls used by the UploadWizard are working as expected
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/api/upload-wizard_tests.py", line 146, in testUploadImageUsingWizardWorkflow
  File "tests/api/upload-wizard_tests.py", line 81, in uploadImage
  File "/mnt/jenkins-workspace/workspace/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/DEV/local/lib/python2.7/site-packages/wikitools/api.py", line 165, in query
    raise APIError(data['error']['code'], data['error']['info'])
APIError: (u'stashfilestorage', u'Could not store upload in the stash: Stashing temporary file failed: UploadStashFileException Error storing file in \'/tmp/8blCX4\': Could not create directory "mwstore://local-backend/local-temp/d/d5".')

Event Timeline

hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added subscribers: zhuyifei1999, Krenair, hashar and 2 others.
Krenair renamed this task from UploadWizard-api-commons.wikimedia.beta.wmflabs.org smoke test fails to Beta cluster uploads (new and viewing existing files/thumbnails) broken due to NFS.Jun 18 2015, 3:31 PM
Krenair set Security to None.

Also captcha stuff (not sure if this should be a new ticket or not) would be at /data/project/upload7/private/captcha
MediaWiki exception logs are full of "Ran out of captcha images"

Also captcha stuff (not sure if this should be a new ticket or not) would be at /data/project/upload7/private/captcha
MediaWiki exception logs are full of "Ran out of captcha images"

Tasks are cheap! UploadWizard stash apparently tries to write to /tmp which is probably. But maybe that is under /data/project as well.

greg renamed this task from Beta cluster uploads (new and viewing existing files/thumbnails) broken due to NFS to Beta Cluster uploads (new and viewing existing files/thumbnails) broken due to WMF Labs NFS outage.Jun 19 2015, 4:03 PM
Krenair renamed this task from Beta Cluster uploads (new and viewing existing files/thumbnails) broken due to WMF Labs NFS outage to Beta Cluster uploads (new and viewing existing files/thumbnails, including captchas) broken due to WMF Labs NFS outage.Jun 19 2015, 4:09 PM
thcipriani claimed this task.
thcipriani subscribed.

All tests passing now

By NFS being up, yes... This task should probably remain open until it no longer depends on NFS or is closed as a duplicate of something like T64835 or T84950.