Coal graphs empty on performance.wikimedia.org
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Gilles
	Nov 25 2020, 8:55 AM

Description

The endpoint returns empty results:

I imagine it's probably fallout from the Python 3 migration.

	Project	Branch	Lines +/-	Subject
	performance/coal	master	+1 -1	Socket send() expects bytes in Python 3
	performance/coal	master	+1 -1	Convert message from bytes to str

Change 643440 had a related patch set uploaded (by Gilles; owner: Gilles):
[performance/coal@master] Convert message from bytes to str

Change 643440 merged by jenkins-bot:
[performance/coal@master] Convert message from bytes to str

Mentioned in SAL (#wikimedia-operations) [2020-11-25T11:27:49Z] <gilles@deploy1001> Started deploy [performance/coal@468bc50]: T268724

Mentioned in SAL (#wikimedia-operations) [2020-11-25T11:27:56Z] <gilles@deploy1001> Finished deploy [performance/coal@468bc50]: T268724 (duration: 00m 06s)

This fixed the error, but now I see that the process can't write to graphite:

2020-11-25 11:35:28,581 [ERROR] (send_to_graphite:247) Failed to submit to graphite: coal.requestStart 130 1605677820

Unfortunately we currently swallow the exceptions resulting in that code branch, without outputting them.

Change 643486 had a related patch set uploaded (by Gilles; owner: Gilles):
[performance/coal@master] Socket send() expects bytes in Python 3

Change 643486 merged by jenkins-bot:
[performance/coal@master] Socket send() expects bytes in Python 3

Mentioned in SAL (#wikimedia-operations) [2020-11-25T11:48:56Z] <gilles@deploy1001> Started deploy [performance/coal@be167b2]: T268724

Mentioned in SAL (#wikimedia-operations) [2020-11-25T11:49:03Z] <gilles@deploy1001> Finished deploy [performance/coal@be167b2]: T268724 (duration: 00m 06s)

Fixed, I'm currently backfilling the data that went missing during the outage.

Data backfilled