The Toolforge cdnjs mirror does not list any packages which were updated within the past few months (for example, the latest version of twitter-bootstrap is 5.1.3, but the latest version shown on toolforge is 5.0.1). Could someone please update this?
Description
Details
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Guard against bad json responses | labs/tools/cdnjs-index | master | +8 -4 |
Event Timeline
Based on timestamps in the tool's public_html directory I think its update job last ran successfully on 2021-05-29.
I hacked up a copy of the generation script and was able to see that generation is currently failing while processing the "Iosevka" entry.
Processing Iosevka...
Traceback (most recent call last):
File "generate2.py", line 124, in <module>
main()
File "generate2.py", line 81, in main
lib["assets"] = resp.json()["assets"]
KeyError: 'assets'The http response being processed is:
{"error":true,"status":404,"message":"Library not found"}This script needs some hardening help to better deal with this kind of data processing error.
Change 755832 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):
[labs/tools/cdnjs-index@master] Guard against bad json responses
Change 755832 merged by jenkins-bot:
[labs/tools/cdnjs-index@master] Guard against bad json responses
The next scheduled job will run in about 3 hours, so I will check back sometime in my tomorrow to see if the content files have actually updated.
Things got much further, but then came to an abrupt halt with:
Traceback (most recent call last):
File "generate.py", line 145, in <module>
main()
File "generate.py", line 141, in main
modal_file.write(helper.render({"lib": lib}))
File "/data/project/cdnjs/venv/lib/python3.5/site-packages/jinja2/environment.py", line 1090, in render
self.environment.handle_exception()
File "/data/project/cdnjs/venv/lib/python3.5/site-packages/jinja2/environment.py", line 832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "/data/project/cdnjs/venv/lib/python3.5/site-packages/jinja2/_compat.py", line 28, in reraise
raise value.with_traceback(tb)
MemoryErrorThe job was running with h_vmem=3145728k (3GiB) which is kind of a wow amount of ram I thought.
Mentioned in SAL (#wikimedia-cloud) [2022-01-21T15:32:12Z] <bd808> Manually running update job with -mem 4096m to see if that is enough space to generate the ridiculous index file (T299723)
Bumping the memory along with the added error handling seems to have worked. For posterity, the lack of an updated index never prevented folks from requesting newer libraries and versions through the service as it is a transparent proxy at that level of operation.