| Krinkle | |
| Mar 30 2025, 5:38 AM |
| F58946814: Screenshot 2025-03-30 at 00.59.08.png | |
| Mar 30 2025, 7:59 AM |
https://cvn.wmcloud.org/api.php?users=KrinkleSock
{ "users": {}, "lastUpdate": 1676305503 }
Which translates to:
$ date --date='@1676305503' Mon Feb 13 16:25:03 UTC 2023
And indeed I can confirm when I instruct one of the CVNBot instance on Libera Chat (e.g. #cvn-wp-nl, #cvn-sw, or #cvn-bots) that new additions do not show up even after 5-15 minutes.
/cc @HideonRosie
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| docroot: Add real-time API status check | labs/countervandalism/cvn-infrastructure | master | +98 -4 |
The CVN API is hosted as part of the cvn-apache service:
https://meta.wikimedia.org/wiki/Countervandalism_Network/Infrastructure#cvn-apache
The current server for that service is cvn-apache10.cvn.eqiad1.wikimedia.cloud, per:
https://openstack-browser.toolforge.org/project/cvn
This server's provisioning is documented at:
https://gerrit.wikimedia.org/r/plugins/gitiles/labs/countervandalism/cvn-infrastructure/+/2f8a73199f4f3a4450dcb09e7613f0422fda3f53/setup.yaml#107
The source code for this in this repo:
https://gerrit.wikimedia.org/g/labs/countervandalism/cvn-api/
I don't think the source code of the API itself needs any changes, as in that case it would have been broken before as well, or present some kind of error. The API is "working", but is using an outdated copy of the database. Most likely something went went wrong with one of two things:
Both of these cronjobs are in the infra repo:
- The cronjob on the CVNBot server that copies the latest database to NFS (shared disk volume).
krinkle@cvn-app11:/data/project/cvn-common/dropbox$ ls -halF … -rw------- 1 root project-cvn 71M Sep 29 2023 .nfs000000000006002e00000001 -rw------- 1 root project-cvn 71M Sep 29 2023 .nfs000000000006002f00000002 -rw------- 1 root project-cvn 71M Sep 29 2023 .nfs000000000006003100000004 … -rw-r--r-- 1 root root 71M Mar 30 05:45 cvn-api-db.sqlite
krinkle@cvn-apache10:/data/project/cvn-common/dropbox$ ls -halF … -rw-r--r-- 1 root root 71M Mar 30 05:45 cvn-api-db.sqlite
Okay, so the cronjob that copies from cvn-app11 (CVNBot14) to shared storage is working.
And, both servers have a working NFS mount of our "dropbox" directory.
- The cronjob on the Apache server that copies the latest database from there to the API.
krinkle@cvn-apache10:/srv/cvn/git/cvn-api/data$ ls -halF … -rw-r--r-- 1 root project-cvn 71M Feb 13 2023 Lists.sqlite
And this indeed hasn't gotten updated in a long time.
Mentioned in SAL (#countervandalism) [2025-03-30T06:55:14Z] <Krinkle> krinkle@cvn-apache10: Run sudo chmod 644 /srv/cvn/git/infrastructure/crontab-config/*.cron, per T390415
Looking at /var/log/syslog on cvn-apache10 reveals why:
… Mar 30 06:40:01 cvn-apache10 cron[705]: (*system*cvndb-pull) INSECURE MODE (group/other writable) (/etc/cron.d/cvndb-pull) … Mar 30 06:41:01 cvn-apache10 cron[705]: (*system*cvndb-pull) INSECURE MODE (group/other writable) (/etc/cron.d/cvndb-pull) Mar 30 06:42:01 cvn-apache10 cron[705]: (*system*cvndb-pull) INSECURE MODE (group/other writable) (/etc/cron.d/cvndb-pull) Mar 30 06:43:01 cvn-apache10 cron[705]: (*system*cvndb-pull) INSECURE MODE (group/other writable) (/etc/cron.d/cvndb-pull) Mar 30 06:45:01 cvn-apache10 cron[705]: (*system*cvndb-pull) INSECURE MODE (group/other writable) (/etc/cron.d/cvndb-pull) …
And indeed, contrary to the provisioning at
https://gerrit.wikimedia.org/r/plugins/gitiles/labs/countervandalism/cvn-infrastructure/+/2f8a73199f4f3a4450dcb09e7613f0422fda3f53/setup.yaml#71
sudo chmod 644 /srv/cvn/git/infrastructure/crontab-config/*.cron
... the /etc/cron.d/cvndb-pull file on cvn-apache10 actually has mode 664 instead of 644.
I'm guessing during a server replacement we forgot to set it up correctly, and it takes a while to realize that it isn't working. Either that, or its mode changed for some other reason.
Anyway, I've run the provision steps again and it works correctly now.
… Mar 30 06:46:01 cvn-apache10 cron[705]: (*system*cvndb-pull) RELOAD (/etc/cron.d/cvndb-pull) … Mar 30 06:50:01 cvn-apache10 CRON[2875683]: (root) CMD (/srv/cvn/git/infrastructure/bin/cvndb-pull > /dev/null) … Mar 30 06:55:01 cvn-apache10 CRON[2876589]: (root) CMD (/srv/cvn/git/infrastructure/bin/cvndb-pull > /dev/null) …
It now has a recent timestamp, and includes actual data that I submitted to CVNBot14 earlier today:
{ "users": { "KrinkleSock": { "type": "whitelist", "comment": "No reason given", "expiry": 1745112769, "adder": "Krinkle" } }, "lastUpdate": 1743317403 }
Change #1132150 had a related patch set uploaded (by Krinkle; author: Krinkle):
[labs/countervandalism/cvn-infrastructure@master] docroot: Add real-time API status check
Change #1132150 merged by Krinkle:
[labs/countervandalism/cvn-infrastructure@master] docroot: Add real-time API status check
