Page MenuHomePhabricator

Coal graphs empty on performance.wikimedia.org
Closed, ResolvedPublic

Description

The endpoint returns empty results:

Screenshot 2020-11-25 at 09.54.39.png (255×239 px, 22 KB)

I imagine it's probably fallout from the Python 3 migration.

Event Timeline

Change 643440 had a related patch set uploaded (by Gilles; owner: Gilles):
[performance/coal@master] Convert message from bytes to str

https://gerrit.wikimedia.org/r/643440

Change 643440 merged by jenkins-bot:
[performance/coal@master] Convert message from bytes to str

https://gerrit.wikimedia.org/r/643440

Mentioned in SAL (#wikimedia-operations) [2020-11-25T11:27:49Z] <gilles@deploy1001> Started deploy [performance/coal@468bc50]: T268724

Mentioned in SAL (#wikimedia-operations) [2020-11-25T11:27:56Z] <gilles@deploy1001> Finished deploy [performance/coal@468bc50]: T268724 (duration: 00m 06s)

This fixed the error, but now I see that the process can't write to graphite:

2020-11-25 11:35:28,581 [ERROR] (send_to_graphite:247) Failed to submit to graphite: coal.requestStart 130 1605677820

Unfortunately we currently swallow the exceptions resulting in that code branch, without outputting them.

Change 643486 had a related patch set uploaded (by Gilles; owner: Gilles):
[performance/coal@master] Socket send() expects bytes in Python 3

https://gerrit.wikimedia.org/r/643486

Change 643486 merged by jenkins-bot:
[performance/coal@master] Socket send() expects bytes in Python 3

https://gerrit.wikimedia.org/r/643486

Mentioned in SAL (#wikimedia-operations) [2020-11-25T11:48:56Z] <gilles@deploy1001> Started deploy [performance/coal@be167b2]: T268724

Mentioned in SAL (#wikimedia-operations) [2020-11-25T11:49:03Z] <gilles@deploy1001> Finished deploy [performance/coal@be167b2]: T268724 (duration: 00m 06s)

Gilles added a subscriber: dpifke.

Fixed, I'm currently backfilling the data that went missing during the outage.