Page MenuHomePhabricator

Rewrite Avro schemas (ApiAction, CirrusSearchRequestSet) as JSONSchema and produce to EventGate
Closed, ResolvedPublic21 Estimated Story Points

Description

A Modern Event Platform goal this quarter is deploy an EventGate service and produce the Monolog+Avro based events to it. We'll design new JSONSchema-ed events that represent the same data as the existent Avro one's. We'll then modify the EventBus extension to produce these events.

https://github.com/wikimedia/mediawiki-event-schemas/tree/master/avro/mediawiki

This ticket does not encompass replacing the Hadoop based processing of this data, just producing the new ones. We'll have another ticket for using the new data, and for decommissioning the Avro stuff.

Details

SubjectRepoBranchLines +/-
operations/mediawiki-configmaster+6 -12
operations/mediawiki-configmaster+6 -0
operations/deployment-chartsmaster+95 -81
operations/mediawiki-configmaster+7 -1
operations/puppetproduction+26 -2
operations/mediawiki-configmaster+8 -3
mediawiki/extensions/CirrusSearchmaster+4 -0
mediawiki/event-schemasmaster+7 -0
operations/mediawiki-configmaster+17 -0
operations/puppetproduction+1 -1
mediawiki/extensions/CirrusSearchmaster+1 K -17
mediawiki/event-schemasmaster+251 -0
operations/mediawiki-configmaster+2 -0
operations/mediawiki-configmaster+1 -0
operations/mediawiki-configmaster+1 -0
operations/mediawiki-configmaster+2 -0
mediawiki/corewmf/1.33.0-wmf.21+9 -13
mediawiki/coremaster+9 -13
mediawiki/extensions/EventBuswmf/1.33.0-wmf.19+12 -4
mediawiki/extensions/EventBusmaster+12 -4
mediawiki/coremaster+10 -4
mediawiki/coremaster+36 -6
operations/deployment-chartsmaster+10 -33
mediawiki/event-schemasmaster+1 -0
mediawiki/event-schemasmaster+94 -0
mediawiki/event-schemasmaster+3 -0
Show related patches Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 485893 merged by jenkins-bot:
[mediawiki/event-schemas@master] Add mediawiki/api/request/0.0.1 schema

https://gerrit.wikimedia.org/r/485893

Change 492923 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/event-schemas@master] Add maxLength for meta.uri in api/request schema

https://gerrit.wikimedia.org/r/492923

Change 492923 merged by Ottomata:
[mediawiki/event-schemas@master] Add maxLength for meta.uri in api/request schema

https://gerrit.wikimedia.org/r/492923

Change 492925 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] Use schemas from docker image, configure api-request stream

https://gerrit.wikimedia.org/r/492925

Change 492925 merged by Alexandros Kosiaris:
[operations/deployment-charts@master] Use schemas from docker image, configure api-request stream

https://gerrit.wikimedia.org/r/492925

Change 491887 merged by Ottomata:
[mediawiki/core@master] Emit new style API action logs into Monolog.

https://gerrit.wikimedia.org/r/491887

Change 494305 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/extensions/EventBus@master] Send monolog $record['context'] as event, not entire $record

https://gerrit.wikimedia.org/r/494305

Change 494308 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/core@master] Set api-request log http.request_headers properly

https://gerrit.wikimedia.org/r/494308

Change 494308 merged by Ottomata:
[mediawiki/core@master] Set api-request log http.request_headers properly

https://gerrit.wikimedia.org/r/494308

Change 494305 merged by Ottomata:
[mediawiki/extensions/EventBus@master] Send monolog $record['context'] as event, not entire $record

https://gerrit.wikimedia.org/r/494305

Change 494368 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/extensions/EventBus@wmf/1.33.0-wmf.19] Send monolog $record['context'] as event, not entire $record

https://gerrit.wikimedia.org/r/494368

@akosiaris, I deployed the new 0.0.7 eventgate-analytics chart to prod k8s and did an ab test there. I can currently push around 800 events per second on our single instance[1]. Let's be conservative and assume 500. By the end of the quarter we'll move both of the topics still used in the Kafka analytics cluster to EventGate + kafka jumbo.

https://grafana.wikimedia.org/d/000000524/kafka-by-topic-graphite?refresh=5m&orgId=1

So, ~10K / second. Is 20 eventgate-analytics pods too much to ask?

[1] From deploy1001.eqiad.wmnet:

ab -n4000 -c400 -T 'application/json' -p /home/otto/test_event_0.0.2.json http://eventgate-analytics.discovery.wmnet:31192/v1/events
...
Requests per second:    808.79 [#/sec] (mean)

@akosiaris, I deployed the new 0.0.7 eventgate-analytics chart to prod k8s and did an ab test there. I can currently push around 800 events per second on our single instance[1]. Let's be conservative and assume 500. By the end of the quarter we'll move both of the topics still used in the Kafka analytics cluster to EventGate + kafka jumbo.

https://grafana.wikimedia.org/d/000000524/kafka-by-topic-graphite?refresh=5m&orgId=1

So, ~10K / second. Is 20 eventgate-analytics pods too much to ask?

[1] From deploy1001.eqiad.wmnet:

ab -n4000 -c400 -T 'application/json' -p /home/otto/test_event_0.0.2.json http://eventgate-analytics.discovery.wmnet:31192/v1/events
...
Requests per second:    808.79 [#/sec] (mean)

That's totally fine.

Awesome, with 20 replicas I can make a single ab with -c1000 push ~6400 msgs/second, but then was limited by single node ab to push farther. :)

Change 494368 abandoned by Ottomata:
Send monolog $record['context'] as event, not entire $record

Reason:
This will go out with wmf.20 train.

https://gerrit.wikimedia.org/r/494368

Change 495144 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/extensions/CirrusSearch@master] [WIP] Log search/requestset event

https://gerrit.wikimedia.org/r/495144

Change 495399 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/core@master] ApiMain.php api/request logging event changes

https://gerrit.wikimedia.org/r/495399

Change 495399 merged by Ottomata:
[mediawiki/core@master] ApiMain.php api/request logging event changes

https://gerrit.wikimedia.org/r/495399

Change 496197 had a related patch set uploaded (by Addshore; owner: Ottomata):
[mediawiki/core@wmf/1.33.0-wmf.21] ApiMain.php api/request logging event changes

https://gerrit.wikimedia.org/r/496197

Change 496197 merged by Addshore:
[mediawiki/core@wmf/1.33.0-wmf.21] ApiMain.php api/request logging event changes

https://gerrit.wikimedia.org/r/496197

Mentioned in SAL (#wikimedia-operations) [2019-03-13T16:36:32Z] <addshore@deploy1001> Synchronized php-1.33.0-wmf.21/includes/api/ApiMain.php: SWAT: T214080 T212529 ApiMain.php api/request logging event changes [[gerrit:496197]] (duration: 00m 57s)

Change 496235 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Enable api-request logging into kafka for group1

https://gerrit.wikimedia.org/r/496235

Change 496235 merged by Ottomata:
[operations/mediawiki-config@master] Enable api-request logging into kafka for group1

https://gerrit.wikimedia.org/r/496235

Change 498873 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Re-enable eventgate-analytics api-request logging for group0 wikis

https://gerrit.wikimedia.org/r/498873

Change 498873 merged by jenkins-bot:
[operations/mediawiki-config@master] Re-enable eventgate-analytics api-request logging for group0 wikis

https://gerrit.wikimedia.org/r/498873

Change 502292 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Enable eventgate-analytics api-request logging for group0 wikis

https://gerrit.wikimedia.org/r/502292

Change 502292 merged by Ottomata:
[operations/mediawiki-config@master] Enable eventgate-analytics api-request logging for group0 wikis

https://gerrit.wikimedia.org/r/502292

Mentioned in SAL (#wikimedia-operations) [2019-04-08T18:27:07Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable eventgate-analytics api-request logging for group0 wikis - T214080 (duration: 00m 56s)

Change 485885 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/event-schemas@master] Add mediawiki/cirrussearch/request/0.0.1 schema

https://gerrit.wikimedia.org/r/485885

Ok, I think https://gerrit.wikimedia.org/r/#/c/mediawiki/event-schemas/+/485885/ and https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/CirrusSearch/+/495144/ are good to go.

We renamed search/requestset to cirrussearch/request, as that is more appropriate. I also renamed the sub elasticsearch requests named appropriately.

Example mediawiki/cirrussearch/request event:

{
  "$schema": "/mediawiki/cirrussearch/request/0.0.1",
  "meta": {
    "request_id": "1e12a388360c781ac70aa631",
    "id": "bedfc021-51ae-11e9-98f7-0800278dc04d",
    "dt": "2019-03-28T23:10:58Z",
    "domain": "dev.wiki.local.wmftest.net",
    "stream": "mediawiki.cirrussearch-request"
  },
  "http": {
    "method": "GET",
    "client_ip": "10.0.2.2",
    "has_cookies": true,
    "request_headers": {
      "accept-language": "en-US,en;q=0.5",
      "referer": "http://dev.wiki.local.wmftest.net:8080/w/index.php?search=sdf&title=Special%3ASearch&go=Go",
      "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:67.0) Gecko/20100101 Firefox/67.0"
    }
  },
  "database": "wiki",
  "mediawiki_host": "mws",
  "search_token": "80ewxp4pl6t7vev6m8x99j21c",
  "source": "web",
  "identity": "fe15eff974aa3371f119f00167c5cd09",
  "request_time_ms": 1114,
  "all_elasticsearch_requests_cached": false,
  "hits": [
    {
      "page_title": "Sandbox",
      "page_id": 6,
      "index": "wiki_content_first",
      "score": 0.036319096
    }
  ],
  "elasticsearch_requests": [
    {
      "query": "sdf",
      "query_type": "near_match",
      "indices": [
        "wiki_content"
      ],
      "request_time_ms": 39,
      "search_time_ms": 2,
      "limit": 50,
      "hits_total": 0,
      "hits_returned": 0,
      "hits_offset": 0,
      "suggestion_requested": false,
      "syntax": [
        "near_match"
      ],
      "cached": false
    },
    {
      "query": "sdf",
      "query_type": "full_text",
      "indices": [
        "wiki_content"
      ],
      "request_time_ms": 35,
      "search_time_ms": 16,
      "limit": 21,
      "hits_total": 1,
      "hits_returned": 1,
      "hits_offset": 0,
      "suggestion_requested": true,
      "max_score": 0.036319096,
      "syntax": [
        "full_text",
        "full_text_simple_match"
      ],
      "cached": false,
      "hits": [
        {
          "page_title": "Sandbox",
          "page_id": 6,
          "index": "wiki_content_first",
          "score": 0.036319096
        }
      ]
    }
  ]
}

@EBernhardson @Pchelolo lookin good ya? Any obvious changes needed?

Change 502838 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Enable api-request EventGate logging for group1 wikis

https://gerrit.wikimedia.org/r/502838

Change 502838 merged by Ottomata:
[operations/mediawiki-config@master] Enable api-request EventGate logging for group1 wikis

https://gerrit.wikimedia.org/r/502838

Mentioned in SAL (#wikimedia-operations) [2019-04-10T19:48:49Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enabling api-request logging via eventgate-analytics for group1 wikis - T214080 (duration: 00m 59s)

Mentioned in SAL (#wikimedia-operations) [2019-04-10T20:27:43Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Revert - Enabling api-request logging via eventgate-analytics for group1 wikis - T214080 (duration: 01m 00s)

We are good to go on the CirrusSearch php extension change. I'd like to merge both patches, test in beta, and then let this go out with the train next week. Once the code is out we can start slowly enabling it via config.

@Pchelolo ok to merge https://gerrit.wikimedia.org/r/#/c/mediawiki/event-schemas/+/485885/ and https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/CirrusSearch/+/495144/ ?

Change 485885 merged by Ottomata:
[mediawiki/event-schemas@master] Add mediawiki/cirrussearch/request/0.0.1 schema

https://gerrit.wikimedia.org/r/485885

Change 495144 merged by Ottomata:
[mediawiki/extensions/CirrusSearch@master] Log mediawiki/cirrussearch/request event via the cirrussearch-request channel

https://gerrit.wikimedia.org/r/495144

Ottomata changed the point value for this task from 8 to 21.Apr 19 2019, 6:21 PM

Change 505280 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Enable cirrussearch-request logs via eventgate in beta

https://gerrit.wikimedia.org/r/505280

Change 505283 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Import mediawiki.api-request and mediawiki.cirrussearch-request via Camus into Hadoop

https://gerrit.wikimedia.org/r/505283

Change 505287 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] New Refine job to refine events using remote JSONSchemas

https://gerrit.wikimedia.org/r/505287

Change 505283 merged by Ottomata:
[operations/puppet@production] Import mediawiki.(api|cirrussearch)-request via Camus into Hadoop

https://gerrit.wikimedia.org/r/505283

Change 505280 merged by Ottomata:
[operations/mediawiki-config@master] Enable cirrussearch-request logs via eventgate in beta

https://gerrit.wikimedia.org/r/505280

Yahoo we have cirrussearch-request events in beta!

Change 505308 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/event-schemas@master] cirrussearch-request - Add http query params

https://gerrit.wikimedia.org/r/505308

Change 505309 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/extensions/CirrusSearch@master] Add query params to cirrussearch-request

https://gerrit.wikimedia.org/r/505309

Change 505308 merged by jenkins-bot:
[mediawiki/event-schemas@master] cirrussearch-request - Add http query params

https://gerrit.wikimedia.org/r/505308

Change 505309 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Add query params to cirrussearch-request

https://gerrit.wikimedia.org/r/505309

Change 505787 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Enable api-request logging to EventGate on all wikis

https://gerrit.wikimedia.org/r/505787

Change 505787 merged by Ottomata:
[operations/mediawiki-config@master] Enable api-request logging to EventGate on all wikis

https://gerrit.wikimedia.org/r/505787

Mentioned in SAL (#wikimedia-operations) [2019-04-23T16:38:20Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable api-request logging to eventgate-analytics for all wikis - T214080 (duration: 00m 53s)

Change 505287 merged by Ottomata:
[operations/puppet@production] New Refine job to refine events using remote JSONSchemas

https://gerrit.wikimedia.org/r/505287

Cool! The Hive table event.mediawiki_api_request is now being filled with data via eventgate, with map type support. E.g.

hive (event)> select params["action"], count(*) as cnt from event.mediawiki_api_request where year=2019 and month=4  and day=23 and hour=0 group by params["action"] order by cnt desc limit 10;

wbgetentities	567136
parsoid-batch	250268
wbsgetsuggestions	69589
compare	69533
wbsearchentities	65124
parse	55458
record-lint	51392
opensearch	49459
cirrus-config-dump	46613

@bd808 what do we need to do next to move toward deprecating the ApiAction avro stream?

@bd808 what do we need to do next to move toward deprecating the ApiAction avro stream?

Update https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/ApiAction to describe the new tables

Done.

Change the roll-up scripts that run as my user on stat1007 to use the new source tables OR get T137321: Run ETL for wmf_raw.ActionApi into wmf.action_* aggregate tables done

These scripts are here: https://github.com/bd808/action-api-analytics

@EvanProdromou Nuria suggested I ask you what to do here. The Analytics team is is very happy to help with T137321, but are not able to drive it. We do need to deprecate usages of ApiAction (hopefully early) this quarter. From looking at Bryan's Hive SQL scripts, I think it might be easy to modify them to use the new event.mediawiki_api_request table. Can your team check on the data in the event.mediawiki_api_request and give this a go?

Change 507099 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Enable cirrussearch-request logging to eventgate-analytics for group0 wikis

https://gerrit.wikimedia.org/r/507099

Mentioned in SAL (#wikimedia-operations) [2019-04-29T19:01:48Z] <ottomata> deploying config change to enable cirrusssearch-request logging to eventgate-analytics for group0 wikis - T214080

Change 507099 merged by Ottomata:
[operations/mediawiki-config@master] Enable cirrussearch-request logging to eventgate-analytics for group0 wikis

https://gerrit.wikimedia.org/r/507099

Mentioned in SAL (#wikimedia-operations) [2019-04-29T19:05:52Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable cirrussearch-request logging to eventgate-analytics for group0 wikis - T214080 (duration: 00m 53s)

Mentioned in SAL (#wikimedia-operations) [2019-04-29T19:07:16Z] <otto@deploy1001> sync-file aborted: Enable cirrussearch-request logging to eventgate-analytics for group0 wikis - T214080 (duration: 00m 02s)

Change 507403 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-analytics - Add cirrussearch-request to stream-config.yaml

https://gerrit.wikimedia.org/r/507403

Change 507403 merged by Ottomata:
[operations/deployment-charts@master] eventgate-analytics - Add cirrussearch-request to stream-config.yaml

https://gerrit.wikimedia.org/r/507403

Change 507550 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Enable cirrussearch-request logging to eventgate-analytics for group1 wikis

https://gerrit.wikimedia.org/r/507550

Change 507550 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable cirrussearch-request logging to eventgate-analytics for group1 wikis

https://gerrit.wikimedia.org/r/507550

Change 507709 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Enable cirrussearch-request logging to eventgate-analytics on all wikis

https://gerrit.wikimedia.org/r/507709

Change 507709 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable cirrussearch-request logging to eventgate-analytics on all wikis

https://gerrit.wikimedia.org/r/507709

Mentioned in SAL (#wikimedia-operations) [2019-05-02T18:18:33Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable cirrussearch-request logging to eventgate-analytics on all wikis (T214080) (duration: 00m 58s)

This comment was removed by Ottomata.