Page MenuHomePhabricator

Require kafka acknowledgment from mediawiki logging pipeline (CirrusSearchRequestSet, ApiAction channels)
Closed, ResolvedPublic

Description

When kafka rejects a message (perhaps the message is too big, or some other error) our monolog handler never knows because it uses the default ack=0, which basically just throws messages into the wind and hopes they arrive. We should update this to ack=1 which requires the partition leader to acknowledge the request.

The kafka library we use supports this, we simply need to set the value. We also need to look at what the error responses look like and log them on the mediawiki side. Kafka also logs these, but it's good to get the logging on both ends.

Event Timeline

Restricted Application added subscribers: Zppix, Aklapper. · View Herald Transcript
Deskana moved this task from needs triage to Up Next on the Discovery-Search board.

Change 292511 had a related patch set uploaded (by EBernhardson):
kafka: Implement ack handling

https://gerrit.wikimedia.org/r/292511

Change 292513 had a related patch set uploaded (by EBernhardson):
cirrus: Require acknowledgement of kafka logging

https://gerrit.wikimedia.org/r/292513

Change 292514 had a related patch set uploaded (by EBernhardson):
logging: Require acknowledgment of kafka logging

https://gerrit.wikimedia.org/r/292514

we just need to swat the wmf-config patch now.

Change 292511 merged by jenkins-bot:
kafka: Implement ack handling

https://gerrit.wikimedia.org/r/292511

Change 292513 merged by jenkins-bot:
cirrus: Require acknowledgement of kafka logging

https://gerrit.wikimedia.org/r/292513

Change 305606 had a related patch set uploaded (by EBernhardson):
Appropriatly check KafkaHandler requireAck argument

https://gerrit.wikimedia.org/r/305606

Change 305606 abandoned by EBernhardson:
Appropriatly check KafkaHandler requireAck argument

Reason:
nevermind ... the defaults are applied in the constructor, so this works as it was supposed to.

https://gerrit.wikimedia.org/r/305606

Change 292514 merged by jenkins-bot:
logging: Require acknowledgment of kafka logging

https://gerrit.wikimedia.org/r/292514

Mentioned in SAL [2016-08-30T23:58:08Z] <catrope@tin> Synchronized wmf-config/logging.php: Require ack of kafka logging (T135159) (duration: 00m 47s)