Page MenuHomePhabricator

Allow (almost?) all EventLogging events to go into MySQL in beta
Closed, ResolvedPublic3 Story Points

Description

We use the same whitelist in beta that we do in prod to keep events from going to MySQL. But there's nothing BUT MySQL for querying events in beta. We should allow events through! :)

Details

Related Gerrit Patches:

Event Timeline

Ottomata created this task.Oct 30 2018, 9:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 30 2018, 9:17 PM

Thanks for filing this, @Ottomata. I nested it under the Growth team's "Understanding first day" epic because it is blocking our ability to test our new EditorJourney schema.

Nuria added a subscriber: Nuria.Oct 30 2018, 10:32 PM

because it is blocking our ability to test our new EditorJourney schema.

It shouldn't, EditorJourney events on beta are available in kafka, and while having those in MYSQL will help, it is a nice-to-have, you can also consume directly from kafka in beta and see events and errors.

See:
https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/TestingOnBetaCluster#See_all_EventLogging_schema_Kafka_topics
https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/TestingOnBetaCluster#Validated_events

cc @Catrope

Thanks for the links, @Nuria. I should have been more clear -- it is not strictly blocking our ability to test (since our engineers can look at Kafka), but it is blocking our ability to test thoroughly with the code we intend to do the analysis with. We'd like to add this capability, which I think aligns with what the Better Use of Data working group has come up with around test-driven development for EventLogging.

Ottomata claimed this task.Oct 31 2018, 4:24 PM
Ottomata triaged this task as High priority.
Ottomata edited projects, added Analytics-Kanban; removed Analytics.
Ottomata set the point value for this task to 3.
Ottomata moved this task from Next Up to Ready to Deploy on the Analytics-Kanban board.

Change 470862 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Default to not using eventlogging-valid-mixed filter, this is set to true in prod

https://gerrit.wikimedia.org/r/470862

Change 470862 merged by Ottomata:
[operations/puppet@production] Default to not using eventlogging-valid-mixed filter

https://gerrit.wikimedia.org/r/470862

Ok! I just disabled the whitelist filter in beta. Produce some events and they *should* show up in the MySQL database on deployment-eventlog05.

hey hey look at that! :)

MariaDB [log]> show tables LIKE 'EditorJourney_%';
+---------------------------------+
| Tables_in_log (EditorJourney_%) |
+---------------------------------+
| EditorJourney_18504997          |
+---------------------------------+
1 row in set (0.00 sec)
MMiller_WMF closed this task as Resolved.Nov 7 2018, 11:18 PM