We have extensive documentation about how to test eventlogging in beta using mysql, if we do away with the consumer in prod we have the option to keep it in beta and continue operating this way. This has the disadvantage that we are building /maintaining and deploying code that we do not really use in prod.
A probably better strategy is to remove the mysql storage in beta entirely and use other system for developers to be able to test their schemas (1 file per event with valid events?)
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Ottomata | T159170 Sunset MySQL data store for eventlogging | |||
| Resolved | mforns | T223414 Move reportupdater reports that pull data from eventlogging mysql to pull data from hadoop | |||
| Resolved | Ottomata | T223415 Strategy to be able to test eventlogging in beta in the absence of mysql |
How about https://www.npmjs.com/package/kafka-sse? or Even just making the mysql path optional in puppet?
considerations: privacy so that events can't be read by everyone. Andrew should weigh in as this solution could be shared by MEP
It would be better in my opinion to keep using the mysql database in deployment-prep as interim solution, and see if we can find an alternative solution in the meantime.. Event Gate will eventually replace eventlogging, so it is probably better to spend time on the former rather than the latter. If we are afraid of keeping mysql code/dependencies we could try to make them optional to remove them on eventlog1002 (in production) and keep them in deployment-prep. What do you think @Ottomata ?
Can we just use kafkatee and write kafka topics to individual per-schema files? I think folks really just need to verify that their events are flowing in as expected, not really query them in beta.
+1 even this solution is fine, I am ok with anything that it is easy and doesn't require more than a couple of hours to be completed :)
I think kafkatee will be more solid than our current mysql in beta which hands up frequently. We would just need to make sure to update our logpuller so disk does not get full and files do not live long, we probably do not need more than a week worth of files if at all.
Or, maybe we don't even need the files? Will people just log in and tail -f them? If so, we could just write a little wrapper script (or even quick simple GUI???) that wraps kafkacat (or eventlogging-consumer !). E.g.
eventlogging-tail <SchemaName>
@Ottomata I think what you and Luca are proposing would work well to address this task. But we could spend some time brainbouncing a very user-friendly way for people to see these logs. It would solve MEP's requirement to make it easier to debug instrumentation in beta and there's no reason we couldn't use it for EventLogging until everything migrates over to MEP.
ping @Ottomata Please correct me if I am wrong but we decided to leave things on beta as they are for the time coming correct?
Correct ya! I think we can re-evaluate this idea for both beta and prod in the future. It'd be nice to have some topics UI like http://kafka-topics-ui.demo.lenses.io/#/cluster/UAT/topic/n/nyc_yellow_taxi_trip_data/ one day (can't use that one though, it uses Confluent REST API).