Not sure who maintains the "cdnjs" mirror tool

https://tools.wmflabs.org/admin/tool/cdnjs

The files are all there, but the job that updates the page has started failing.

tools.cdnjs@tools-sgebastion-08:~$ source venv/bin/python
-bash: source: venv/bin/python: cannot execute binary file

Someone messed with the venv I think. I'll recreate it. I suspect someone may have mistaken this for needing a venv rebuild in a container, which it does not. The venv is only for grid jobs.

Mentioned in SAL (#wikimedia-cloud) [2020-03-17T15:09:03Z] <bstorm_> rebuilding the venv on a bastion T246096

Mentioned in SAL (#wikimedia-cloud) [2020-03-17T15:11:10Z] <bstorm_> cleared the logs T246096

Mentioned in SAL (#wikimedia-cloud) [2020-03-17T15:12:10Z] <bstorm_> running jsub -l h_vmem=3145728k /data/project/cdnjs/update-index.sh T246096

That'll take a little while :)

Ok, I still got a traceback, so something has changed upstream or something as well.

I see it fetched most of the github starcounts, but it might have timed out or something in the process.

Running it locally with a lot of debug stuff to see where it breaks. I don't immediately see why it would have an issue, but I should be able to make it pop up.

Found it. The cdnjs upstream changed their fields a bit.

https://cdnjs.discourse.group/t/asset-in-api-is-not-loading/291

I have working code that I'm doing a final test on.

Change 580438 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[labs/tools/cdnjs-index@master] upstream-changes: get assets from each individual library

https://gerrit.wikimedia.org/r/580438

Change 580438 merged by Bstorm:
[labs/tools/cdnjs-index@master] upstream-changes: get assets from each individual library

https://gerrit.wikimedia.org/r/580438

Running the job again. It'll be a while.

Ugh. It appears to have hit something weird even now. Fixing that.

I've set up some weird debugging inside the template to see what the problem data is coming out of the cdnjs API. Once I've got that info, I can fix the rest.

Change 580596 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[labs/tools/cdnjs-index@master] breakfix: skip blank assets

https://gerrit.wikimedia.org/r/580596

Change 580596 merged by Bstorm:
[labs/tools/cdnjs-index@master] breakfix: skip blank assets

https://gerrit.wikimedia.org/r/580596

Still crashed. There's something else strange in the API now. First, there's listed libraries that show up as 404 if you try to list files or assets, and there's apparently one more element that's giving me trouble. I'm guessing it's an assets object with an empty files section. I'll find out if it is that whenever the next run finishes.

Mentioned in SAL (#wikimedia-cloud) [2020-03-18T16:34:11Z] <bstorm_> hacked in a possible workaround with extra logging T246096

Change 581027 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[labs/tools/cdnjs-index@master] breakfix: replace "None" value from the JSON expressly

https://gerrit.wikimedia.org/r/581027

Change 581027 merged by Bstorm:
[labs/tools/cdnjs-index@master] breakfix: replace "None" value from the JSON expressly

https://gerrit.wikimedia.org/r/581027

I think I now have found all the places where the API changes have broken this. There is only one array that I'm not yet checking, and I'll add that logic if this fails yet again. It seems like that array, which is nested in another that I am checking, couldn't possibly be null.

Running yet another manual run.