MediaWiki will soon be writing to a new topic in kafka, mediawiki_CirrusSearchRequestSet. This will be formatted with apache avro and needs to flow through camus and into hadoop.
Schema will only be registered in the mediawiki repo.
Search team will take care of creating hive tables.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Declined | None | T112846 Display automata and humans separately on zero results rate graph | |||
| Resolved | EBernhardson | T103505 Create analytics-centric Cirrus logs and have them import into HDFS | |||
| Resolved | • Nuria | T113521 Setup pipeline for search logs to travel through kafka and camus into hadoop {hawk} [55 pts] |
Change 243845 had a related patch set uploaded (by Madhuvishy):
[WIP] Add support to camus to consume schemaID-less avro
Change 243990 had a related patch set uploaded (by Madhuvishy):
[WIP] Add refinery-camus module
Change 243845 abandoned by Madhuvishy:
[WIP] Add support to camus to consume schemaID-less avro
Reason:
This patch is superseded by https://gerrit.wikimedia.org/r/#/c/243990/
Change 244594 had a related patch set uploaded (by Madhuvishy):
[WIP] Add properties file for importing mediawiki data
Change 244601 had a related patch set uploaded (by Madhuvishy):
[WIP] analytics:Add cron that schedules camus imports for mediawiki data
Change 244594 abandoned by Ottomata:
[WIP] Add properties file for importing mediawiki data
Reason:
I've moved camus properties files to puppet, and amended https://gerrit.wikimedia.org/r/#/c/244601/2 with this mediawiki properties file.
Hey @EBernhardson, your schema needs a timestamp field in order for Camus to import the data into proper partitions.
It should be a unix seconds value in UTC.
Change 244601 merged by Ottomata:
Add cron that schedules camus imports for mediawiki Avro Binary data