Page MenuHomePhabricator

cronspam from cobalt after the Gerrit migration
Closed, ResolvedPublic

Description

Cronspam email:

subject:

Cron <gerrit2@cobalt> /usr/bin/java -jar /var/lib/gerrit2/review_site/bin/gerrit.war gsql -d /var/lib/gerrit2/review_site/ --format JSON_SINGLE -c "'SELECT changes.change_id AS change_id, COUNT(DISTINCT patch_set_approvals.account_id) AS reviewer_count FROM changes LEFT JOIN patch_set_approvals ON (changes.change_id = patch_set_approvals.change_id) GROUP BY changes.change_id'" > /var/www/reviewer-counts.json

body: /bin/sh: 1: cannot create /var/www/reviewer-counts.json: Permission denied

The cron and file got introduced as part of T54329

Event Timeline

I am pretty sure that is (was?) used for our community metrics tool at http://korma.wmflabs.org/browser/ with some cron fetching something like https://gerrit.wikimedia.org/reviewer-counts.json

Maybe we can get puppet to touch the file with gerrit2 permissions?

On lead.wikimedia.org:

$ ls -l /var/www/reviewer-counts.json
-rw-r--r-- 1 gerrit2 root 258 Oct 10 01:59 /var/www/reviewer-counts.json

root@cobalt:/var/www# touch reviewer-counts.json
root@cobalt:/var/www# chown gerrit2 reviewer-counts.json

puppetizing will follow tomorrow

Oh, it turns out it's not just permissions on the file, it's also that the DB part must have changed.

We now see on both, old and new server, that:

{"type":"error","message":"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''SELECT changes.change_id AS change_id, COUNT(DISTINCT patch_set_approvals.accou' at line 1"}

Change 315296 had a related patch set uploaded (by Dzahn):
gerrit: disable reviewer-counts cron job

https://gerrit.wikimedia.org/r/315296

Why the syntax error? We just moved gerrit, we did not upgrade the version. If anything i'd expect a permissions issue, but syntax ??

Change 315300 had a related patch set uploaded (by Paladox):
Gerrit: Fix reviewer-counts.json cronspam by removing \ and "

https://gerrit.wikimedia.org/r/315300

Change 315296 merged by Dzahn:
gerrit: disable reviewer-counts cron job

https://gerrit.wikimedia.org/r/315296

Why the syntax error? We just moved gerrit, we did not upgrade the version. If anything i'd expect a permissions issue, but syntax ??

It was using double quotes, so by removing \ and " it fixed the problem.

Yes, but how did it ever work before and now suddenly fail on both, old and new server? Or was it broken all the time and only noticed now by coincidence?

@Dzahn probably it was broken all the time. Just wasn't noticed until now.

Change 315300 merged by Dzahn:
Gerrit: Fix reviewer-counts.json cronspam by removing \ and "

https://gerrit.wikimedia.org/r/315300

Dzahn claimed this task.

11:18 < mutante> !log lead (old gerrit) manually remove reviewer-count cron, puppet is disabled

11:19 < mutante> !log cobalt (new gerrit) run reviewer-count cron, works now

11:20 < paladox> it works

Change 315596 had a related patch set uploaded (by Dzahn):
gerrit: puppetize reviewer-counts.json

https://gerrit.wikimedia.org/r/315596

Change 315596 merged by Dzahn:
gerrit: puppetize reviewer-counts.json

https://gerrit.wikimedia.org/r/315596