Page MenuHomePhabricator
Create Task
Maniphest T104004

Can't download large datasets from datasets.wikimedia.org
Closed, ResolvedPublic
Actions

Description

I can't seem to get the download to start for large datasets on datasets.wikimedia.org. To my knowledge this is somewhat recently broken, but it could have been months.

See example here: http://datasets.wikimedia.org/public-datasets/enwiki/etc/session_revisions.20131105.tsv.gz

Expected: Download starts right after click

Actual: No download will start.

@Ottomata suspects that it could be due to varnish trying to add the dataset to its cache.

[11:00:46] <ottomata> ha, halfak, misc varnishes only have 8G memory allocated to them
[11:00:49] <ottomata> that file is 9G

Details

SubjectRepoBranchLines +/-
cache_misc: stream and hit-for-pass for large objectsoperations/puppetproduction+19 -0
varnish-misc: fix syntax error for datasets configoperations/puppetproduction+1 -1
Don't cache datasets.wikimedia.orgoperations/puppetproduction+5 -1
Customize query in gerrit

Related Objects

Event Timeline

Halfak created this task.Jun 26 2015, 4:08 PM
Halfak raised the priority of this task from to Needs Triage.
Halfak updated the task description. (Show Details)
Halfak added projects: Analytics-Clusters, acl*sre-team.
Halfak added subscribers: Halfak, Ottomata.
Restricted Application added subscribers: Matanya, Aklapper. · View Herald TranscriptJun 26 2015, 4:08 PM
gerritbot subscribed.Jun 26 2015, 4:09 PM

Change 221139 had a related patch set uploaded (by Ottomata):
Don't cache datasets.wikimedia.org

https://gerrit.wikimedia.org/r/221139

gerritbot added a project: Patch-For-Review.Jun 26 2015, 4:09 PM
gerritbot added a comment.Jun 26 2015, 7:20 PM

Change 221139 merged by Dzahn:
Don't cache datasets.wikimedia.org

https://gerrit.wikimedia.org/r/221139

Dzahn mentioned this in rOPUP91c0d05c339f: Don't cache datasets.wikimedia.org.Jun 26 2015, 7:20 PM
gerritbot added a comment.Jun 26 2015, 7:27 PM

Change 221177 had a related patch set uploaded (by Dzahn):
varnish-misc: fix syntax error for datasets config

https://gerrit.wikimedia.org/r/221177

gerritbot added a comment.Jun 26 2015, 7:28 PM

Change 221177 merged by Dzahn:
varnish-misc: fix syntax error for datasets config

https://gerrit.wikimedia.org/r/221177

Dzahn mentioned this in rOPUPebd227f0aa53: varnish-misc: fix syntax error for datasets config.Jun 26 2015, 7:28 PM
Halfak added a comment.Jun 26 2015, 7:39 PM

The patch should have been merged by now, but the problem persists.

fgiunchedi subscribed.Jul 20 2015, 10:33 AM

confirmed this is still a problem, I think what's happening is that we're no longer caching in varnish but it will still try to fetch the whole file from the backend. IOW we are not setting beresp.do_stream in vcl_fetch here I think:

sub vcl_fetch {
        /* Don't cache private, no-cache, no-store objects */
        if (beresp.http.Cache-Control ~ "(private|no-cache|no-store)") {
                set beresp.ttl = 0s;
                /* This should be translated into hit_for_pass later */
        }
fgiunchedi triaged this task as Medium priority.Jul 20 2015, 10:33 AM
fgiunchedi set Security to None.
fgiunchedi removed a project: Patch-For-Review.Jul 20 2015, 2:31 PM
Halfak mentioned this in T120091: 503 errors on datasets.wikimedia.org.Dec 3 2015, 3:43 PM
BBlack claimed this task.Dec 3 2015, 3:57 PM
BBlack merged a task: T120091: 503 errors on datasets.wikimedia.org.
BBlack added subscribers: ArielGlenn, StudiesWorld.
BBlack subscribed.

Yeah this is all the same issue and still present. I think @fgiunchedi is on the right track here about streaming, I'm going to write up a generic misc-cluster patch to work around this for datasets and other services there that might run into similar issues. Probably the recent misc-cluster switches to dual-layer and dual-tier have exacerbated this problem compared to the past.

gerritbot added a comment.Dec 3 2015, 4:15 PM

Change 256705 had a related patch set uploaded (by BBlack):
cache_misc: stream and hit-for-pass for large objects

https://gerrit.wikimedia.org/r/256705

gerritbot added a project: Patch-For-Review.Dec 3 2015, 4:15 PM
BBlack added a project: Traffic.Dec 3 2015, 4:17 PM
gerritbot added a comment.Dec 3 2015, 4:41 PM

Change 256705 merged by BBlack:
cache_misc: stream and hit-for-pass for large objects

https://gerrit.wikimedia.org/r/256705

BBlack closed this task as Resolved.Dec 3 2015, 4:50 PM

This should be resolved now with the change above applied. I've tested the files from this ticket and the more-recent merged dupe ticket, and they both started streaming quickly and got through several hundred megs before I aborted the transfer (which would otherwise take forever!). Reopen if there's still an issue!

fgiunchedi added a comment.Dec 4 2015, 11:44 AM

confirmed, thanks @BBlack!

BBlack moved this task from Backlog to Done on the Traffic board.Dec 11 2015, 4:23 PM
Halfak mentioned this in T277609: Generate dump of scored-revisions from 2018-2020 for English Wikipedia.Apr 1 2021, 5:14 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits