Page MenuHomePhabricator

cronspam from cobalt after the Gerrit migration
Closed, ResolvedPublic

Description

Cronspam email:

subject:

Cron <gerrit2@cobalt> /usr/bin/java -jar /var/lib/gerrit2/review_site/bin/gerrit.war gsql -d /var/lib/gerrit2/review_site/ --format JSON_SINGLE -c "'SELECT changes.change_id AS change_id, COUNT(DISTINCT patch_set_approvals.account_id) AS reviewer_count FROM changes LEFT JOIN patch_set_approvals ON (changes.change_id = patch_set_approvals.change_id) GROUP BY changes.change_id'" > /var/www/reviewer-counts.json

body: /bin/sh: 1: cannot create /var/www/reviewer-counts.json: Permission denied

The cron and file got introduced as part of T54329

Event Timeline

elukey created this task.Oct 10 2016, 7:58 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 10 2016, 7:58 AM

I am pretty sure that is (was?) used for our community metrics tool at http://korma.wmflabs.org/browser/ with some cron fetching something like https://gerrit.wikimedia.org/reviewer-counts.json

Maybe we can get puppet to touch the file with gerrit2 permissions?

On lead.wikimedia.org:

$ ls -l /var/www/reviewer-counts.json
-rw-r--r-- 1 gerrit2 root 258 Oct 10 01:59 /var/www/reviewer-counts.json
hashar updated the task description. (Show Details)Oct 10 2016, 8:08 AM
Dzahn added a comment.Oct 10 2016, 1:20 PM

root@cobalt:/var/www# touch reviewer-counts.json
root@cobalt:/var/www# chown gerrit2 reviewer-counts.json

puppetizing will follow tomorrow

Dzahn added a comment.Oct 11 2016, 4:11 PM

Oh, it turns out it's not just permissions on the file, it's also that the DB part must have changed.

We now see on both, old and new server, that:

{"type":"error","message":"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''SELECT changes.change_id AS change_id, COUNT(DISTINCT patch_set_approvals.accou' at line 1"}
Dzahn added a subscriber: demon.Oct 11 2016, 4:12 PM

Change 315296 had a related patch set uploaded (by Dzahn):
gerrit: disable reviewer-counts cron job

https://gerrit.wikimedia.org/r/315296

Dzahn added a comment.Oct 11 2016, 4:18 PM

Why the syntax error? We just moved gerrit, we did not upgrade the version. If anything i'd expect a permissions issue, but syntax ??

Change 315300 had a related patch set uploaded (by Paladox):
Gerrit: Fix reviewer-counts.json cronspam by removing \ and "

https://gerrit.wikimedia.org/r/315300

Change 315296 merged by Dzahn:
gerrit: disable reviewer-counts cron job

https://gerrit.wikimedia.org/r/315296

Why the syntax error? We just moved gerrit, we did not upgrade the version. If anything i'd expect a permissions issue, but syntax ??

It was using double quotes, so by removing \ and " it fixed the problem.

Dzahn added a comment.Oct 11 2016, 5:27 PM

Yes, but how did it ever work before and now suddenly fail on both, old and new server? Or was it broken all the time and only noticed now by coincidence?

@Dzahn probably it was broken all the time. Just wasn't noticed until now.

Change 315300 merged by Dzahn:
Gerrit: Fix reviewer-counts.json cronspam by removing \ and "

https://gerrit.wikimedia.org/r/315300

Dzahn closed this task as Resolved.Oct 11 2016, 6:20 PM
Dzahn claimed this task.

11:18 < mutante> !log lead (old gerrit) manually remove reviewer-count cron, puppet is disabled

11:19 < mutante> !log cobalt (new gerrit) run reviewer-count cron, works now

11:20 < paladox> it works

Change 315596 had a related patch set uploaded (by Dzahn):
gerrit: puppetize reviewer-counts.json

https://gerrit.wikimedia.org/r/315596

Change 315596 merged by Dzahn:
gerrit: puppetize reviewer-counts.json

https://gerrit.wikimedia.org/r/315596