@jijiki in discussing this with the team we want to brainstorm about it a bit. Some think there might be a better way. Give us until end of day tomorrow before you get too far with the varnish changes, is that ok?

Ok, data is swapped. @nettrom_WMF if you don't mind, could you check out both event.PrefUpdate and event_sanitized.PrefUpdate to triple check that they look ok and the data you want is there?

1. (Optional) Add support in service-template-node for talking to Cassandra. Since we are planning to move storage from RESTBase down to individual services, we might benefit from some shared library support. This step however is optional - perhaps we could just use Cassandra driver directly without additional abstractions. To be investigated.

I had to tackle some performance problems with the actual sanitization. Parking this here

Requests bearing the X-Wikimedia-Debug header passthrough the caches but they endup in varnishkafka and thus turnilo, just like you mentioned. The requests I am running have been extracted from webrequest_text, and I trust they look like pageviews. One run of 700k requests is a blip, but multiple ones, for an extended period of time, might be an issue I believe (correct me if I am wrong). Would be it too much work for an X-Analytics header like you propose?

I'm game to do it all together. The new log stream might be easier considering changing a hook's signature is involved since T240307. Having the log ids anywhere gets us to minimum viable, so it seems smart to try both and use whichever one's done first. I'll update the task description and get to work :)

So, why Go? Are we completely moving away from node? Will there be a new version of node-service-template?

Talked it over with the team, we think we should post this. So post away, anytime that's good for your schedule. We have some good ideas for follow-ups.

Why not just use event timestamp instead of log timestamp in the ongoing update? they should be within milliseconds from each other. Cause you can't switch from streaming reconstruction to old way seamlessly since different timestamps are used?

I'm sorry, I keep trying to bring it up at our team meetings but we've been busy end of quarter. I think if you have a lull and need to get a post out, you can go with it. It works perfectly well as an introduction and we can follow up on it with more in-depth examples of research.

In T263055#6468006, @Pchelolo wrote:

Could you expand a bit in what MySQL data are you trying to correlate the events to? It feels conceptually wrong to add ‘log_id’ to an event cause the log entry is not really a primary part of the action the even represents, it’s more of a derivative from the primary event. The page is moved regardless of the log about it being generated (conceptually, not in practice)

In T262203#6459797, @JAllemandou wrote:

database	rev_id	users	rev_id_count_gt_1
enwiki	977034865	[104.235.36.59, 96.33.68.122]	2

In T262261#6450148, @JAllemandou wrote:

Checks for discrenpencies (using previous comment setup):

• page-delete

...

I noticed here that events to the same page were criss-crossing and they're actually all really close in time, here they are split into two tables and re-ordered to match (btw if you keep tables outside of the code formatting they format nicely in phab):

I think it would be mutually beneficial to work on producing the events together, so we can help iterate on the schemas and implement the emit. We just need guidance on what edge cases we might be missing and creation of any new hooks if needed.

https://github.com/allegro/turnilo/pull/657

Oof, good point, I hadn't thought of the Druid version of this. I suppose I'll have to wipe and reload all of that as well, to apply the whitelist.

quick status update (tl;dr; we can announce it Monday):

In T262202#6451136, @jijiki wrote:

@Milimetric my question is, if I want to do a performance test from outside our network, and fire 700k requests towards our debug servers, does this affect and alter our analytics data? thank you!

potential easier way: require belonging to analytics-privatedata to log into stat1xxx.

x-wikimedia-debug shouldn't affect anything we do

Low priority for us, people can feel free to raise priority or grab it (should be relatively easy)

(we decided to deploy this next week, and we can do so by simply merging the change above (https://gerrit.wikimedia.org/r/626223))

https://github.com/wikimedia/KafkaSSE/pull/6

This has been deployed for a while, we just forgot it in the wrong column.

pull request for public endpoints is here: https://github.com/wikimedia/restbase/pull/1273

We did option 1

That would be nice. We'd have to upstream a patch, but it would be a simple patch... hm, I'm game. @elukey: your choice.

Ok, thanks @RhinosF1 for the help, here's the problem as I understand it:

@JAllemandou, responding while Marcel is gone. I looked through and found a few of the reports rely on data that's changing (so if lost, would be hard/impossible to recompute). Why don't we just back up everything, the RU output is teeny tiny.

I thought we excluded these from denormalized, and left them just in page history. I can see why we'd want to include them. Hm... it is very confusing. Maybe we should give them a negative page id?

Apologies this took so long. My plan is to apply the whitelist to the old data. I will do so by the end of the week, let me know if there are any outstanding concerns. Ping @nettrom_WMF.

manually merged into T168573, to make it easier to triage and decide on the future of Dashiki.

manually merged into T168573, to make it easier to triage and decide on the future of Dashiki.

manually merged into T168573, to make it easier to triage and decide on the future of Dashiki.

manually merged into T168573, to make it easier to triage and decide on the future of Dashiki.

manually merged into T168573, to make it easier to triage and decide on the future of Dashiki.

manually merged into T168573, to make it easier to triage and decide on the future of Dashiki.

manually merged into T168573, to make it easier to triage and decide on the future of Dashiki.

Do follow-up if somehow this is still relevant

manually merged into T168573, to make it easier to triage and decide on the future of Dashiki.

manually merged into T168573, to make it easier to triage and decide on the future of Dashiki.

manually merged into T168573, to make it easier to triage and decide on the future of Dashiki.

manually merged into T168573, to make it easier to triage and decide on the future of Dashiki.

manually merged into T168573, to make it easier to triage and decide on the future of Dashiki.

manually merged into T168573, to make it easier to triage and decide on the future of Dashiki.

manually merged into T168573, to make it easier to triage and decide on the future of Dashiki.

manually merged into T168573, to make it easier to triage and decide on the future of Dashiki.

manually merged into T168573, to make it easier to triage and decide on the future of Dashiki.

manually merged into T168573, to make it easier to triage and decide on the future of Dashiki.

outdated

(one idea would be to download an increased-resolution version)

we can/should do this in the event way

Haven't seen this in a while, maybe it went away!

We have to think more about how to accomplish this, taking into account all the security implications we've discovered on the first pass.

outdated now, though I still have the patches and I'll try to honor their spirit in a later refactor.

outdated, no further work on wikimetrics

Presto may not work here, unless we just poke a hole to a production cluster. We'll have it for reference.