| nshahquinn-wmf | |
| Jul 22 2022, 11:53 PM |
| F35399388: Screen Shot 2022-08-05 at 9.19.11 AM.png | |
| Aug 5 2022, 1:20 PM |
Although the schemas are merged and the code deployed, we aren't actually receiving Wikistories events in the Data Lake because the streams haven't been registered in the MediaWiki configuration.
Full instructions for registration are at wikitech:Event Platform/Instrumentation How To#Deployment.
Change 817263 had a related patch set uploaded (by Sbisson; author: Sbisson):
[operations/mediawiki-config@master] Register Wikistories streams
@nshahquinn-wmf I'm not sure what the 'destination_event_service' should be... I put 'eventgate-analytics-external' in the patch.
I let Data Engineering know that the documentation doesn't explain how to choose the right value.
@SBisson here are the testing instructions for the deployment:
1 Connect to one of the analytics clients. You might be able to use other servers in production too, but I know these will work.
2 Confirm that you can consume events using the following command, which prints the latest 5 events from an already-implemented stream and exits:
kafkacat -C -b kafka-jumbo1001.eqiad.wmnet:9092 -t eqiad.mediawiki.web_ui_scroll -o -5 -e -q | jq .
3 Start listening for events in the consumption stream using the following command, which will continuing running until manually terminated:
kafkacat -C -b kafka-jumbo1001.eqiad.wmnet:9092 -t eqiad.mediawiki.wikistories_consumption_event -o 0 | jq .
You can do the same for the contribution stream by changing the value of the -t parameter. This only listens for events routed through eqiad; if your events might get routed through codfw, you can listen for those by substituting codfw for eqiad in the stream name.
Okay, I just reviewed the patch and it looks right! I did find a better location for our wgEventStreams entries, but it wouldn't cause any actual harm if it got deployed as-is.
Change 817263 merged by jenkins-bot:
[operations/mediawiki-config@master] Register Wikistories streams
Mentioned in SAL (#wikimedia-operations) [2022-07-28T20:18:21Z] <thcipriani@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:817263|Register Wikistories streams (T313633)]] (duration: 03m 24s)
Change 818457 had a related patch set uploaded (by Sbisson; author: Sbisson):
[mediawiki/extensions/Wikistories@master] Fix consumption event stream name
Change 818457 merged by jenkins-bot:
[mediawiki/extensions/Wikistories@master] Fix consumption event stream name
I'm seeing consumption events I generate on Indonesian Wikipedia end up in Kafka! (Although I think I found a bug with the session ID we're using: T314622)
Still no contribution events, though.
I can see contribution events being sent by the browser. Do you see that as well but nothing in kafka?
I had been looking carefully for them, but not seeing them. But I just tried again and did see them, so I probably just missed them.
And I just looked in Kafka in the codfw stream and I saw them there! I also found yours in the eqiad stream. I also found events from both streams in the Data Lake.
So it looks like all is well here!