Tool Labs CDNjs mirror not up-to-date
Closed, ResolvedPublic

Description

Our mirror of CDNjs seems to be out of sync. Specifically, although TensorFlow.js was added in April (see https://github.com/cdnjs/cdnjs/search?o=asc&q=tensorflow&s=committer-date&type=Commits) and is available on the main mirror (https://cdnjs.com/libraries/tensorflow) but not in ours. Could there be a problem with the syncing?

mpopov created this task.Jun 7 2018, 5:40 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 7 2018, 5:40 PM
mpopov updated the task description. (Show Details)Jun 7 2018, 5:41 PM
Bstorm added a comment.Jun 7 2018, 5:53 PM

It is running into a memory error during the sync job. Need to work around that.

Andrew triaged this task as Normal priority.Jun 8 2018, 3:12 PM
Andrew assigned this task to Bstorm.
Bstorm added a comment.Jun 8 2018, 4:38 PM

So the issue is that memory tends to run out when deserializing JSON to python objects. I'll test just asking for more memory from gridengine, but it is likely that a pretty big rework is needed. The python streaming deserialization story is not great. I've done this in golang with little difficulty, but here, the libraries are a bit weird and often depend on C libs. The tricky thing is sorting by Github stars, since that sort of depends on having them all in memory. It's easy to just dump to files as the stream comes in. It may be a matter of constructing the html files during streaming, while determining the sort by using a much smaller data structure of stubs--or using something to store the data in temporarily like Redis.

If I can just ask for 3GB of memory, and mess with the ulimits, we might be fine.

mpopov awarded a token.Jun 8 2018, 5:03 PM
Bstorm added a comment.Jun 8 2018, 5:31 PM

Good news! Asking for more memory from gridengine did succeed in not getting a memory error so far (got past the whole JSON deserialization). However, I now get:
UnicodeEncodeError: 'ascii' codec can't encode character '\u2019' in position 14224560: ordinal not in range(128)

Somebody introduced a fancy quote character. That's an easy fix, at least.

Good news! Asking for more memory from gridengine did succeed in not getting a memory error so far (got past the whole JSON deserialization).

Yay! Hm, I wonder if the memory limit be an occasionally-recurring issue as more and more libraries (and their versions) are added to CDNjs over time.

Thank you so much for looking into this and fixing it, @Bstorm! :)

Change 439380 had a related patch set uploaded (by Bstorm; owner: Brooke Storm):
[labs/tools/cdnjs-index@master] cdnjs-index: force encoding on writing files

https://gerrit.wikimedia.org/r/439380

Bstorm added a comment.Jun 9 2018, 1:25 AM

I'll Just merge that patch, since I'm already running it via local hack :-p

Change 439380 merged by Bstorm:
[labs/tools/cdnjs-index@master] cdnjs-index: force encoding on writing files

https://gerrit.wikimedia.org/r/439380

Bstorm added a comment.Jun 9 2018, 5:01 AM

Also, I see tensorflow is now on the index page 💥

mpopov closed this task as Resolved.Jun 9 2018, 10:18 PM

@Bstorm: thank you very much!!!

Vvjjkkii renamed this task from Tool Labs CDNjs mirror not up-to-date to bgbaaaaaaa.Sun, Jul 1, 1:06 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii raised the priority of this task from Normal to High.
Vvjjkkii removed Bstorm as the assignee of this task.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed subscribers: gerritbot, Aklapper.
CommunityTechBot assigned this task to Bstorm.
CommunityTechBot lowered the priority of this task from High to Normal.
CommunityTechBot closed this task as Resolved.
CommunityTechBot renamed this task from bgbaaaaaaa to Tool Labs CDNjs mirror not up-to-date.
CommunityTechBot added subscribers: gerritbot, Aklapper.