We manually do this now, but then @CCogdill_WMF has to intervene on the mailing house servers to trigger an off-schedule import. If export works but FTP fails, try at least one more time.
Description
Description
Details
Details
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Try upload 3 times | wikimedia/fundraising/tools | master | +10 -1 | |
| Split silverpop export from upload | wikimedia/fundraising/tools | master | +50 -13 |
Event Timeline
Comment Actions
Would be great! I can move back the time of the Silverpop nightly import if it makes sense to give this time to rerun.
Comment Actions
I've emailed Sisolak about bzipping the files before we send. The upload size would decrease from 1.3GB to 360MB, with that much less chance of being interrupted.
Comment Actions
Does it make sense to have a separate upload job? ie. if the upload succeeds it could output that file & a job on jenkins could keep trying to upload it if it doesn't find the file.
Comment Actions
Change 325360 had a related patch set uploaded (by Awight):
Split silverpop export from upload
Comment Actions
One thing that got creepy is, the IBM import job was running only 3 hours after our export begins--which means we were always running on day-old data, since our export takes about 4 hours to complete.
I've pushed the export job to 2 hours earlier, so it starts at 06:00 UTC.
Comment Actions
New split export/upload is confirmed working, we'll see in time whether the retry functionality helps with our original issue.