Page MenuHomePhabricator

Gerrit perfsite bot is flaky
Closed, DeclinedPublic

Description

It seems to rarely update and when it does, it incorrectly uses fully defined paths instead of relative ones.

Event Timeline

Gilles triaged this task as Medium priority.May 27 2019, 4:31 AM

First thing that looks quite wrong, it hits a 401 error on gerrit:

Traceback (most recent call last):
  File "/home/gilles/checkupdate.py", line 59, in <module>
    if check_if_existing_change():
  File "/home/gilles/checkupdate.py", line 54, in check_if_existing_change
    results = gerrit_api.get('/changes/?q=topic:%s+is:open' % topic)
  File "/home/gilles/.local/lib/python2.7/site-packages/pygerrit2/rest/__init__.py", line 165, in get
    decoded_response = _decode_response(response)
  File "/home/gilles/.local/lib/python2.7/site-packages/pygerrit2/rest/__init__.py", line 60, in _decode_response
    response.raise_for_status()
  File "/home/gilles/.local/lib/python2.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://gerrit.wikimedia.org/r/a/changes/?q=topic:perfsite-autoupdate+is:open

Possibly due to the attack mitigations that have been put in place on our gerrit instance? Accessing that URL locally is fine, but fails with that 401 from the WMCS machine. Could be throttling of some kind.

Hmm yes, it appears that it's become impossible to use Basic auth with our gerrit instance: T218750: Re-enable use of Gerrit HTTP token to push patchsets I don't see the option for an HTTP password anymore. And interacting with gerrit has become an SSH-only affair as of late. This is unfortunate as I don't think the pygerrit2 library supports SSH auth.

I've disabled the bot for now. I don't think it'e worth working hard on making it function over SSH, as long as re-enabling HTTP tokens for gerrit is still considered.

Gilles changed the task status from Open to Stalled.May 27 2019, 5:00 AM
Gilles lowered the priority of this task from Medium to Low.

@Gilles you can use the ldap password for the bot (for http auth). Just the ability to use a random generated pass was switched off.

@Paladox doesn't seem like I can log into the account with the actual username, only the "real name" of the bot. Which unfortunately contains spaces in my case. Which python doesn't seem to like in .netrc files, which is what my bot needs to authenticate to gerrit.

Yes, looks like it! Thanks for linking to that task.

Krinkle renamed this task from Perf site bot is flaky to Gerrit perfsite bot is flaky.Jul 1 2019, 8:32 PM

It doesn't look like this functionality is going to be returning any time soon, I've deleted the perfbot VM on WMCS. We'll just have to keep committing the site updates manually.

@Gilles http passwords is back in gerrit (you can regenerate one now!)

We re enabled it after a serious of updates to the feature.

@Paladox I tried with my bot and it still failed, I don't think it's the same kind of http password. Anyway the VM has been nuked already. This bot was a nice-to-have but not that important.

@Gilles did you generate the password through gerrit's ui?

Not a new one, this was using the one that was defined when the feature was still available in the UI months ago. Did re-enabling the feature nuke the old passwords?

We revoked all passwords. So you will need to regenerate a new one.