Create scripts to estimate Kafka queue size per wiki
Closed, DeclinedPublic
Actions

Assigned To

Authored By

	• Pchelolo
	Dec 6 2017, 10:06 PM

Description

Until we figure out how to report per-wiki metrics without destroying graphite (T175952) we need a way to estimate the queue size per wiki. This is fairly easy using kafkacat, jq and some bash scripting.

Related Objects

Mentioned Here: P6443 check_queue.sh
T175952: Split ChangeProp metrics by wiki

Event Timeline

• Pchelolo created this task.Dec 6 2017, 10:06 PM

Restricted Application added a project: Analytics. · View Herald TranscriptDec 6 2017, 10:06 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

• Pchelolo added a project: Services (doing).Dec 6 2017, 10:07 PM

• Pchelolo updated the task description. (Show Details)

• Pchelolo added a subscriber: Joe.

Created a simple bash script that can be run on Kafka boxes to find out the Kafka JobQueue backlog per job type per wiki. It's available in my home dir on kafka1001 under /home/ppchelko/check_queue.sh. I'll let it bake for a while (it probably needs more parametrization etc) and then we can transfer it to some more appropriate location

{

P6443 check_queue.sh

1	#!/bin/bash
2
3	if [ "$1" = "-h" ] \|\| [ "$#" -lt 1 ]; then
4	echo "Usage: check_queue.sh [ -d domain ] jobtype"
5	exit 0
6	fi
7
8	if [ "$1" = "-d" ]; then
9	DOMAIN="$2"
10	JOB_TYPE="$3"
11	else
12	JOB_TYPE="$1"
13	fi
14
15	KAFKA_INFO="`kafka run-class kafka.admin.ConsumerGroupCommand \
16	--describe \
17	--group change-prop-${JOB_TYPE} \
18	--bootstrap-server localhost:9092 \
19	--new-consumer \| grep eqiad.mediawiki.job.${JOB_TYPE}`"
20
21	COMMITTED_OFFSET="$(echo ${KAFKA_INFO} \| tr ',' '\n' \| sed -n 4p \| tr -d '[:space:]')"
22	BACKLOG="$(echo ${KAFKA_INFO} \| tr ',' '\n' \| sed -n 6p \| tr -d '[:space:]')"
23
24	echo "Committed offset for ${JOB_TYPE}: ${COMMITTED_OFFSET}"
25	echo "Full backlog for ${JOB_TYPE}: ${BACKLOG}"
26
27	if [ "x$DOMAIN" = "x" ]; then
28	echo "Top 10 domains by job backlog count:"
29	kafkacat -b localhost:9092 \
30	-p 0 \
31	-t eqiad.mediawiki.job.${JOB_TYPE} \
32	-o ${COMMITTED_OFFSET} \
33	-c ${BACKLOG} 2> /dev/null \| jq -c '.meta.domain' \| sort \| uniq -ic \| sort -n -k 1 -r \| head -n 10
34	else
35	echo -n "Job count for ${DOMAIN}: "
36	kafkacat -b localhost:9092 \
37	-p 0 \
38	-t eqiad.mediawiki.job.${JOB_TYPE} \
39	-o ${COMMITTED_OFFSET} \
40	-c ${BACKLOG} 2> /dev/null \| jq -c "select(.meta.domain == \"${DOMAIN}\") \| ." \| wc -l
41	fi

}

• fdans moved this task from Incoming to Radar on the Analytics board.Dec 11 2017, 4:56 PM

• mobrovac added a project: Platform Team Workboards (Doing).Dec 20 2018, 12:42 PM

@Pchelolo let's add it to ops/puppet?

WDoranWMF edited projects, added Platform Team Legacy (Later); removed Platform Team Workboards (Doing), Services (doing).Jul 5 2019, 6:28 PM

• Pchelolo lowered the priority of this task from Medium to Low.Jul 17 2019, 4:48 PM

• Pchelolo edited projects, added Platform Engineering (Needs Cleaning - Services Operations); removed Platform Team Legacy (Later).

• Pchelolo closed this task as Declined.Nov 22 2019, 5:56 PM

Aklapper edited projects, added Analytics-Radar; removed Analytics.Jun 10 2020, 6:44 AM

Create scripts to estimate Kafka queue size per wikiClosed, DeclinedPublicActions

Description

Related Objects

Event Timeline

Create scripts to estimate Kafka queue size per wiki
Closed, DeclinedPublic
Actions