Page MenuHomePhabricator
Create Task
Maniphest T214080

Rewrite Avro schemas (ApiAction, CirrusSearchRequestSet) as JSONSchema and produce to EventGate
Closed, ResolvedPublic21 Estimated Story Points
Actions

Description

A Modern Event Platform goal this quarter is deploy an EventGate service and produce the Monolog+Avro based events to it. We'll design new JSONSchema-ed events that represent the same data as the existent Avro one's. We'll then modify the EventBus extension to produce these events.

https://github.com/wikimedia/mediawiki-event-schemas/tree/master/avro/mediawiki

This ticket does not encompass replacing the Hadoop based processing of this data, just producing the new ones. We'll have another ticket for using the new data, and for decommissioning the Avro stuff.

Details

SubjectRepoBranchLines +/-
Enable cirrussearch-request logging to eventgate-analytics on all wikisoperations/mediawiki-configmaster+6 -12
Enable cirrussearch-request logging to eventgate-analytics for group1 wikisoperations/mediawiki-configmaster+6 -0
eventgate-analytics - Add cirrussearch-request to stream-config.yamloperations/deployment-chartsmaster+95 -81
Enable cirrussearch-request logging to eventgate-analytics for group0 wikisoperations/mediawiki-configmaster+7 -1
New Refine job to refine events using remote JSONSchemasoperations/puppetproduction+26 -2
Enable api-request logging to EventGate on all wikisoperations/mediawiki-configmaster+8 -3
Add query params to cirrussearch-requestmediawiki/extensions/CirrusSearchmaster+4 -0
cirrussearch-request - Add http query paramsmediawiki/event-schemasmaster+7 -0
Enable cirrussearch-request logs via eventgate in betaoperations/mediawiki-configmaster+17 -0
Import mediawiki.(api|cirrussearch)-request via Camus into Hadoopoperations/puppetproduction+1 -1
Log mediawiki/cirrussearch/request event via the cirrussearch-request channelmediawiki/extensions/CirrusSearchmaster+1 K -17
Add mediawiki/cirrussearch/request/0.0.1 schemamediawiki/event-schemasmaster+251 -0
Enable api-request EventGate logging for group1 wikisoperations/mediawiki-configmaster+2 -0
Enable eventgate-analytics api-request logging for group0 wikisoperations/mediawiki-configmaster+1 -0
Re-enable eventgate-analytics api-request logging for group0 wikisoperations/mediawiki-configmaster+1 -0
Enable api-request logging into kafka for group1operations/mediawiki-configmaster+2 -0
ApiMain.php api/request logging event changesmediawiki/corewmf/1.33.0-wmf.21+9 -13
ApiMain.php api/request logging event changesmediawiki/coremaster+9 -13
Send monolog $record['context'] as event, not entire $recordmediawiki/extensions/EventBuswmf/1.33.0-wmf.19+12 -4
Send monolog $record['context'] as event, not entire $recordmediawiki/extensions/EventBusmaster+12 -4
Set api-request log http.request_headers properlymediawiki/coremaster+10 -4
Emit new style API action logs into Monolog.mediawiki/coremaster+36 -6
Use schemas from docker image, configure api-request streamoperations/deployment-chartsmaster+10 -33
Add maxLength for meta.uri in api/request schemamediawiki/event-schemasmaster+1 -0
Add mediawiki/api/request/0.0.1 schemamediawiki/event-schemasmaster+94 -0
Set maxLength on patternlike fields in tewst/event/0.0.2mediawiki/event-schemasmaster+3 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Ottomata mentioned this in T206785: Modern Event Platform: Stream Intake Service (EventGate): Implementation.Feb 21 2019, 7:59 PM
gerritbot added a comment.Feb 22 2019, 9:35 PM

Change 485893 merged by jenkins-bot:
[mediawiki/event-schemas@master] Add mediawiki/api/request/0.0.1 schema

https://gerrit.wikimedia.org/r/485893

gerritbot added a comment.Feb 25 2019, 10:04 PM

Change 492923 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/event-schemas@master] Add maxLength for meta.uri in api/request schema

https://gerrit.wikimedia.org/r/492923

gerritbot added a comment.Feb 25 2019, 10:05 PM

Change 492923 merged by Ottomata:
[mediawiki/event-schemas@master] Add maxLength for meta.uri in api/request schema

https://gerrit.wikimedia.org/r/492923

gerritbot added a comment.Feb 25 2019, 10:22 PM

Change 492925 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] Use schemas from docker image, configure api-request stream

https://gerrit.wikimedia.org/r/492925

Ottomata mentioned this in rDEPLOYCHARTS8faa30f1d181: Use schemas from docker image, configure api-request stream.Feb 25 2019, 10:24 PM
gerritbot added a comment.Feb 26 2019, 9:27 AM

Change 492925 merged by Alexandros Kosiaris:
[operations/deployment-charts@master] Use schemas from docker image, configure api-request stream

https://gerrit.wikimedia.org/r/492925

gerritbot added a comment.Feb 26 2019, 5:53 PM

Change 491887 merged by Ottomata:
[mediawiki/core@master] Emit new style API action logs into Monolog.

https://gerrit.wikimedia.org/r/491887

gerritbot added a comment.Mar 4 2019, 8:35 PM

Change 494305 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/extensions/EventBus@master] Send monolog $record['context'] as event, not entire $record

https://gerrit.wikimedia.org/r/494305

gerritbot added a comment.Mar 4 2019, 8:48 PM

Change 494308 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/core@master] Set api-request log http.request_headers properly

https://gerrit.wikimedia.org/r/494308

gerritbot added a comment.Mar 4 2019, 9:41 PM

Change 494308 merged by Ottomata:
[mediawiki/core@master] Set api-request log http.request_headers properly

https://gerrit.wikimedia.org/r/494308

gerritbot added a comment.Mar 4 2019, 10:01 PM

Change 494305 merged by Ottomata:
[mediawiki/extensions/EventBus@master] Send monolog $record['context'] as event, not entire $record

https://gerrit.wikimedia.org/r/494305

gerritbot added a comment.Mar 4 2019, 10:17 PM

Change 494368 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/extensions/EventBus@wmf/1.33.0-wmf.19] Send monolog $record['context'] as event, not entire $record

https://gerrit.wikimedia.org/r/494368

Ottomata moved this task from Estimated/ Discussed to In Progress Before Value Streams Kickoff (August 15th) on the Event-Platform board.Mar 4 2019, 10:29 PM
Ottomata added a project: Analytics-Kanban.
Ottomata moved this task from Next Up to In Code Review on the Analytics-Kanban board.
Ottomata added a subscriber: akosiaris.Mar 4 2019, 10:36 PM

@akosiaris, I deployed the new 0.0.7 eventgate-analytics chart to prod k8s and did an ab test there. I can currently push around 800 events per second on our single instance[1]. Let's be conservative and assume 500. By the end of the quarter we'll move both of the topics still used in the Kafka analytics cluster to EventGate + kafka jumbo.

https://grafana.wikimedia.org/d/000000524/kafka-by-topic-graphite?refresh=5m&orgId=1

So, ~10K / second. Is 20 eventgate-analytics pods too much to ask?

[1] From deploy1001.eqiad.wmnet:

ab -n4000 -c400 -T 'application/json' -p /home/otto/test_event_0.0.2.json http://eventgate-analytics.discovery.wmnet:31192/v1/events
...
Requests per second:    808.79 [#/sec] (mean)
akosiaris added a comment.Mar 5 2019, 6:30 AM
In T214080#4999866, @Ottomata wrote:

@akosiaris, I deployed the new 0.0.7 eventgate-analytics chart to prod k8s and did an ab test there. I can currently push around 800 events per second on our single instance[1]. Let's be conservative and assume 500. By the end of the quarter we'll move both of the topics still used in the Kafka analytics cluster to EventGate + kafka jumbo.

https://grafana.wikimedia.org/d/000000524/kafka-by-topic-graphite?refresh=5m&orgId=1

So, ~10K / second. Is 20 eventgate-analytics pods too much to ask?

[1] From deploy1001.eqiad.wmnet:

ab -n4000 -c400 -T 'application/json' -p /home/otto/test_event_0.0.2.json http://eventgate-analytics.discovery.wmnet:31192/v1/events
...
Requests per second:    808.79 [#/sec] (mean)

That's totally fine.

ReleaseTaggerBot added a project: MW-1.33-notes (1.33.0-wmf.20; 2019-03-05).Mar 5 2019, 11:02 AM
Ottomata added a comment.Mar 5 2019, 2:56 PM

Awesome, with 20 replicas I can make a single ab with -c1000 push ~6400 msgs/second, but then was limited by single node ab to push farther. :)

ReleaseTaggerBot edited projects, added MW-1.33-notes (1.33.0-wmf.21; 2019-03-12); removed MW-1.33-notes (1.33.0-wmf.20; 2019-03-05).Mar 5 2019, 5:03 PM
gerritbot added a comment.Mar 7 2019, 2:25 PM

Change 494368 abandoned by Ottomata:
Send monolog $record['context'] as event, not entire $record

Reason:
This will go out with wmf.20 train.

https://gerrit.wikimedia.org/r/494368

Ottomata moved this task from In Code Review to In Progress on the Analytics-Kanban board.Mar 7 2019, 10:21 PM
gerritbot added a comment.Mar 7 2019, 10:21 PM

Change 495144 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/extensions/CirrusSearch@master] [WIP] Log search/requestset event

https://gerrit.wikimedia.org/r/495144

gerritbot added a comment.Mar 8 2019, 9:52 PM

Change 495399 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/core@master] ApiMain.php api/request logging event changes

https://gerrit.wikimedia.org/r/495399

• Nuria closed subtask T214446: EventBus mediawiki extension should support multiple 'event service' endpoints as Resolved.Mar 11 2019, 6:14 PM
Ottomata closed subtask T216163: Add monolog adapters for Eventbus as Resolved.Mar 12 2019, 2:39 PM
gerritbot added a comment.Mar 12 2019, 5:23 PM

Change 495399 merged by Ottomata:
[mediawiki/core@master] ApiMain.php api/request logging event changes

https://gerrit.wikimedia.org/r/495399

gerritbot added a comment.Mar 13 2019, 4:17 PM

Change 496197 had a related patch set uploaded (by Addshore; owner: Ottomata):
[mediawiki/core@wmf/1.33.0-wmf.21] ApiMain.php api/request logging event changes

https://gerrit.wikimedia.org/r/496197

gerritbot added a comment.Mar 13 2019, 4:30 PM

Change 496197 merged by Addshore:
[mediawiki/core@wmf/1.33.0-wmf.21] ApiMain.php api/request logging event changes

https://gerrit.wikimedia.org/r/496197

Stashbot subscribed.Mar 13 2019, 4:36 PM

Mentioned in SAL (#wikimedia-operations) [2019-03-13T16:36:32Z] <addshore@deploy1001> Synchronized php-1.33.0-wmf.21/includes/api/ApiMain.php: SWAT: T214080 T212529 ApiMain.php api/request logging event changes [[gerrit:496197]] (duration: 00m 57s)

Stashbot mentioned this in T212529: Standardize datetimes/timestamps in the Data Lake.Mar 13 2019, 4:36 PM
gerritbot added a comment.Mar 13 2019, 6:39 PM

Change 496235 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Enable api-request logging into kafka for group1

https://gerrit.wikimedia.org/r/496235

gerritbot added a comment.Mar 13 2019, 6:42 PM

Change 496235 merged by Ottomata:
[operations/mediawiki-config@master] Enable api-request logging into kafka for group1

https://gerrit.wikimedia.org/r/496235

gerritbot added a comment.Mar 25 2019, 2:19 PM

Change 498873 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Re-enable eventgate-analytics api-request logging for group0 wikis

https://gerrit.wikimedia.org/r/498873

gerritbot added a comment.Mar 25 2019, 6:58 PM

Change 498873 merged by jenkins-bot:
[operations/mediawiki-config@master] Re-enable eventgate-analytics api-request logging for group0 wikis

https://gerrit.wikimedia.org/r/498873

Ottomata mentioned this in T219842: [Post-mortem] Kafka Jumbo cluster cannot accept connections.Apr 8 2019, 1:57 PM
gerritbot added a comment.Apr 8 2019, 5:59 PM

Change 502292 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Enable eventgate-analytics api-request logging for group0 wikis

https://gerrit.wikimedia.org/r/502292

gerritbot added a comment.Apr 8 2019, 6:20 PM

Change 502292 merged by Ottomata:
[operations/mediawiki-config@master] Enable eventgate-analytics api-request logging for group0 wikis

https://gerrit.wikimedia.org/r/502292

Stashbot added a comment.Apr 8 2019, 6:27 PM

Mentioned in SAL (#wikimedia-operations) [2019-04-08T18:27:07Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable eventgate-analytics api-request logging for group0 wikis - T214080 (duration: 00m 56s)

gerritbot added a comment.Apr 9 2019, 7:53 PM

Change 485885 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/event-schemas@master] Add mediawiki/cirrussearch/request/0.0.1 schema

https://gerrit.wikimedia.org/r/485885

Ottomata added a comment.Apr 9 2019, 8:20 PM

Ok, I think https://gerrit.wikimedia.org/r/#/c/mediawiki/event-schemas/+/485885/ and https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/CirrusSearch/+/495144/ are good to go.

We renamed search/requestset to cirrussearch/request, as that is more appropriate. I also renamed the sub elasticsearch requests named appropriately.

Example mediawiki/cirrussearch/request event:

{
  "$schema": "/mediawiki/cirrussearch/request/0.0.1",
  "meta": {
    "request_id": "1e12a388360c781ac70aa631",
    "id": "bedfc021-51ae-11e9-98f7-0800278dc04d",
    "dt": "2019-03-28T23:10:58Z",
    "domain": "dev.wiki.local.wmftest.net",
    "stream": "mediawiki.cirrussearch-request"
  },
  "http": {
    "method": "GET",
    "client_ip": "10.0.2.2",
    "has_cookies": true,
    "request_headers": {
      "accept-language": "en-US,en;q=0.5",
      "referer": "http://dev.wiki.local.wmftest.net:8080/w/index.php?search=sdf&title=Special%3ASearch&go=Go",
      "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:67.0) Gecko/20100101 Firefox/67.0"
    }
  },
  "database": "wiki",
  "mediawiki_host": "mws",
  "search_token": "80ewxp4pl6t7vev6m8x99j21c",
  "source": "web",
  "identity": "fe15eff974aa3371f119f00167c5cd09",
  "request_time_ms": 1114,
  "all_elasticsearch_requests_cached": false,
  "hits": [
    {
      "page_title": "Sandbox",
      "page_id": 6,
      "index": "wiki_content_first",
      "score": 0.036319096
    }
  ],
  "elasticsearch_requests": [
    {
      "query": "sdf",
      "query_type": "near_match",
      "indices": [
        "wiki_content"
      ],
      "request_time_ms": 39,
      "search_time_ms": 2,
      "limit": 50,
      "hits_total": 0,
      "hits_returned": 0,
      "hits_offset": 0,
      "suggestion_requested": false,
      "syntax": [
        "near_match"
      ],
      "cached": false
    },
    {
      "query": "sdf",
      "query_type": "full_text",
      "indices": [
        "wiki_content"
      ],
      "request_time_ms": 35,
      "search_time_ms": 16,
      "limit": 21,
      "hits_total": 1,
      "hits_returned": 1,
      "hits_offset": 0,
      "suggestion_requested": true,
      "max_score": 0.036319096,
      "syntax": [
        "full_text",
        "full_text_simple_match"
      ],
      "cached": false,
      "hits": [
        {
          "page_title": "Sandbox",
          "page_id": 6,
          "index": "wiki_content_first",
          "score": 0.036319096
        }
      ]
    }
  ]
}

@EBernhardson @Pchelolo lookin good ya? Any obvious changes needed?

gerritbot added a comment.Apr 10 2019, 5:30 PM

Change 502838 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Enable api-request EventGate logging for group1 wikis

https://gerrit.wikimedia.org/r/502838

gerritbot added a comment.Apr 10 2019, 7:46 PM

Change 502838 merged by Ottomata:
[operations/mediawiki-config@master] Enable api-request EventGate logging for group1 wikis

https://gerrit.wikimedia.org/r/502838

Stashbot added a comment.Apr 10 2019, 7:48 PM

Mentioned in SAL (#wikimedia-operations) [2019-04-10T19:48:49Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enabling api-request logging via eventgate-analytics for group1 wikis - T214080 (duration: 00m 59s)

Stashbot added a comment.Apr 10 2019, 8:27 PM

Mentioned in SAL (#wikimedia-operations) [2019-04-10T20:27:43Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Revert - Enabling api-request logging via eventgate-analytics for group1 wikis - T214080 (duration: 01m 00s)

Ottomata added a comment.Apr 18 2019, 1:45 PM

We are good to go on the CirrusSearch php extension change. I'd like to merge both patches, test in beta, and then let this go out with the train next week. Once the code is out we can start slowly enabling it via config.

@Pchelolo ok to merge https://gerrit.wikimedia.org/r/#/c/mediawiki/event-schemas/+/485885/ and https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/CirrusSearch/+/495144/ ?

• mobrovac closed subtask T221390: Make sure any monolog + eventbus fields with binary data are base64 encoded as Resolved.Apr 18 2019, 11:17 PM
gerritbot added a comment.Apr 19 2019, 5:47 PM

Change 485885 merged by Ottomata:
[mediawiki/event-schemas@master] Add mediawiki/cirrussearch/request/0.0.1 schema

https://gerrit.wikimedia.org/r/485885

gerritbot added a comment.Apr 19 2019, 5:47 PM

Change 495144 merged by Ottomata:
[mediawiki/extensions/CirrusSearch@master] Log mediawiki/cirrussearch/request event via the cirrussearch-request channel

https://gerrit.wikimedia.org/r/495144

Ottomata changed the point value for this task from 8 to 21.Apr 19 2019, 6:21 PM
gerritbot added a comment.Apr 19 2019, 6:21 PM

Change 505280 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Enable cirrussearch-request logs via eventgate in beta

https://gerrit.wikimedia.org/r/505280

gerritbot added a comment.Apr 19 2019, 6:51 PM

Change 505283 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Import mediawiki.api-request and mediawiki.cirrussearch-request via Camus into Hadoop

https://gerrit.wikimedia.org/r/505283

gerritbot added a comment.Apr 19 2019, 6:56 PM

Change 505287 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] New Refine job to refine events using remote JSONSchemas

https://gerrit.wikimedia.org/r/505287

gerritbot added a comment.Apr 19 2019, 6:59 PM

Change 505283 merged by Ottomata:
[operations/puppet@production] Import mediawiki.(api|cirrussearch)-request via Camus into Hadoop

https://gerrit.wikimedia.org/r/505283

gerritbot added a comment.Apr 19 2019, 7:28 PM

Change 505280 merged by Ottomata:
[operations/mediawiki-config@master] Enable cirrussearch-request logs via eventgate in beta

https://gerrit.wikimedia.org/r/505280

Ottomata added a comment.Apr 19 2019, 7:45 PM

Yahoo we have cirrussearch-request events in beta!

• Nuria added a comment.Apr 19 2019, 8:16 PM

Nice!

gerritbot added a comment.Apr 19 2019, 8:45 PM

Change 505308 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/event-schemas@master] cirrussearch-request - Add http query params

https://gerrit.wikimedia.org/r/505308

gerritbot added a comment.Apr 19 2019, 8:46 PM

Change 505309 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/extensions/CirrusSearch@master] Add query params to cirrussearch-request

https://gerrit.wikimedia.org/r/505309

gerritbot added a comment.Apr 19 2019, 9:03 PM

Change 505308 merged by jenkins-bot:
[mediawiki/event-schemas@master] cirrussearch-request - Add http query params

https://gerrit.wikimedia.org/r/505308

gerritbot added a comment.Apr 19 2019, 9:33 PM

Change 505309 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Add query params to cirrussearch-request

https://gerrit.wikimedia.org/r/505309

ReleaseTaggerBot added a project: MW-1.34-notes (1.34.0-wmf.3; 2019-04-30).Apr 19 2019, 10:00 PM
gerritbot added a comment.Apr 23 2019, 2:56 PM

Change 505787 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Enable api-request logging to EventGate on all wikis

https://gerrit.wikimedia.org/r/505787

gerritbot added a comment.Apr 23 2019, 4:35 PM

Change 505787 merged by Ottomata:
[operations/mediawiki-config@master] Enable api-request logging to EventGate on all wikis

https://gerrit.wikimedia.org/r/505787

Stashbot added a comment.Apr 23 2019, 4:38 PM

Mentioned in SAL (#wikimedia-operations) [2019-04-23T16:38:20Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable api-request logging to eventgate-analytics for all wikis - T214080 (duration: 00m 53s)

gerritbot added a comment.Apr 23 2019, 6:48 PM

Change 505287 merged by Ottomata:
[operations/puppet@production] New Refine job to refine events using remote JSONSchemas

https://gerrit.wikimedia.org/r/505287

Ottomata added a comment.Apr 23 2019, 8:59 PM

Cool! The Hive table event.mediawiki_api_request is now being filled with data via eventgate, with map type support. E.g.

hive (event)> select params["action"], count(*) as cnt from event.mediawiki_api_request where year=2019 and month=4  and day=23 and hour=0 group by params["action"] order by cnt desc limit 10;

wbgetentities	567136
parsoid-batch	250268
wbsgetsuggestions	69589
compare	69533
wbsearchentities	65124
parse	55458
record-lint	51392
opensearch	49459
cirrus-config-dump	46613

@bd808 what do we need to do next to move toward deprecating the ApiAction avro stream?

bd808 added a comment.Apr 26 2019, 2:41 AM
In T214080#5132742, @Ottomata wrote:

@bd808 what do we need to do next to move toward deprecating the ApiAction avro stream?

Ottomata added a subscriber: EvanProdromou.Apr 29 2019, 3:22 PM

Update https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/ApiAction to describe the new tables

Done.

Change the roll-up scripts that run as my user on stat1007 to use the new source tables OR get T137321: Run ETL for wmf_raw.ActionApi into wmf.action_* aggregate tables done

These scripts are here: https://github.com/bd808/action-api-analytics

@EvanProdromou Nuria suggested I ask you what to do here. The Analytics team is is very happy to help with T137321, but are not able to drive it. We do need to deprecate usages of ApiAction (hopefully early) this quarter. From looking at Bryan's Hive SQL scripts, I think it might be easy to modify them to use the new event.mediawiki_api_request table. Can your team check on the data in the event.mediawiki_api_request and give this a go?

gerritbot added a comment.Apr 29 2019, 6:18 PM

Change 507099 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Enable cirrussearch-request logging to eventgate-analytics for group0 wikis

https://gerrit.wikimedia.org/r/507099

Stashbot added a comment.Apr 29 2019, 7:01 PM

Mentioned in SAL (#wikimedia-operations) [2019-04-29T19:01:48Z] <ottomata> deploying config change to enable cirrusssearch-request logging to eventgate-analytics for group0 wikis - T214080

gerritbot added a comment.Apr 29 2019, 7:03 PM

Change 507099 merged by Ottomata:
[operations/mediawiki-config@master] Enable cirrussearch-request logging to eventgate-analytics for group0 wikis

https://gerrit.wikimedia.org/r/507099

Stashbot added a comment.Apr 29 2019, 7:05 PM

Mentioned in SAL (#wikimedia-operations) [2019-04-29T19:05:52Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable cirrussearch-request logging to eventgate-analytics for group0 wikis - T214080 (duration: 00m 53s)

Stashbot added a comment.Apr 29 2019, 7:07 PM

Mentioned in SAL (#wikimedia-operations) [2019-04-29T19:07:16Z] <otto@deploy1001> sync-file aborted: Enable cirrussearch-request logging to eventgate-analytics for group0 wikis - T214080 (duration: 00m 02s)

gerritbot added a comment.Apr 30 2019, 7:21 PM

Change 507403 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-analytics - Add cirrussearch-request to stream-config.yaml

https://gerrit.wikimedia.org/r/507403

gerritbot added a comment.Apr 30 2019, 7:23 PM

Change 507403 merged by Ottomata:
[operations/deployment-charts@master] eventgate-analytics - Add cirrussearch-request to stream-config.yaml

https://gerrit.wikimedia.org/r/507403

Ottomata mentioned this in rDEPLOYCHARTSfc24af5afd46: eventgate-analytics - Add cirrussearch-request to stream-config.yaml.Apr 30 2019, 7:28 PM
gerritbot added a comment.May 1 2019, 12:16 PM

Change 507550 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Enable cirrussearch-request logging to eventgate-analytics for group1 wikis

https://gerrit.wikimedia.org/r/507550

Ottomata added a subtask: T222268: Port usage of mediawiki_CirrusSearchRequestSet to mediawiki_cirrussearch_request.May 1 2019, 2:53 PM
gerritbot added a comment.May 1 2019, 4:14 PM

Change 507550 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable cirrussearch-request logging to eventgate-analytics for group1 wikis

https://gerrit.wikimedia.org/r/507550

gerritbot added a comment.May 1 2019, 9:34 PM

Change 507709 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Enable cirrussearch-request logging to eventgate-analytics on all wikis

https://gerrit.wikimedia.org/r/507709

gerritbot added a comment.May 2 2019, 6:15 PM

Change 507709 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable cirrussearch-request logging to eventgate-analytics on all wikis

https://gerrit.wikimedia.org/r/507709

Stashbot added a comment.May 2 2019, 6:18 PM

Mentioned in SAL (#wikimedia-operations) [2019-05-02T18:18:33Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable cirrussearch-request logging to eventgate-analytics on all wikis (T214080) (duration: 00m 58s)

Milimetric unsubscribed.May 2 2019, 7:59 PM
Ottomata added a comment.May 6 2019, 6:03 PM
This comment was removed by Ottomata.
Ottomata mentioned this in T97226: Include the request ID in API request logs.May 14 2019, 1:23 PM
Ottomata moved this task from In Progress to Done on the Analytics-Kanban board.May 14 2019, 3:44 PM
• Nuria closed this task as Resolved.May 17 2019, 11:29 PM
debt closed subtask T222268: Port usage of mediawiki_CirrusSearchRequestSet to mediawiki_cirrussearch_request as Resolved.Jun 21 2019, 2:16 PM
• Nuria closed subtask T225110: cirrussearch-request has some validation errors as Resolved.Jun 26 2019, 10:49 PM
• Nuria closed subtask T222267: Port usage of mediawiki_ApiAction to mediawiki_api_request as Resolved.
• Nuria closed subtask T226436: Remove Avro + Kafka support in Mediawiki, Refinery and Puppet as Resolved.Jun 26 2019, 10:51 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits