@jijiki in discussing this with the team we want to brainstorm about it a bit. Some think there might be a better way. Give us until end of day tomorrow before you get too far with the varnish changes, is that ok?
Ok, data is swapped. @nettrom_WMF if you don't mind, could you check out both event.PrefUpdate and event_sanitized.PrefUpdate to triple check that they look ok and the data you want is there?
Tue, Sep 22
- (Optional) Add support in service-template-node for talking to Cassandra. Since we are planning to move storage from RESTBase down to individual services, we might benefit from some shared library support. This step however is optional - perhaps we could just use Cassandra driver directly without additional abstractions. To be investigated.
I had to tackle some performance problems with the actual sanitization. Parking this herefor the record. It's finishing up running now, I'll pair with someone to swap the data tomorrow.
Fri, Sep 18
Requests bearing the X-Wikimedia-Debug header passthrough the caches but they endup in varnishkafka and thus turnilo, just like you mentioned. The requests I am running have been extracted from webrequest_text, and I trust they look like pageviews. One run of 700k requests is a blip, but multiple ones, for an extended period of time, might be an issue I believe (correct me if I am wrong). Would be it too much work for an X-Analytics header like you propose?
Thu, Sep 17
I'm game to do it all together. The new log stream might be easier considering changing a hook's signature is involved since T240307. Having the log ids anywhere gets us to minimum viable, so it seems smart to try both and use whichever one's done first. I'll update the task description and get to work :)
So, why Go? Are we completely moving away from node? Will there be a new version of node-service-template?
Talked it over with the team, we think we should post this. So post away, anytime that's good for your schedule. We have some good ideas for follow-ups.
Why not just use event timestamp instead of log timestamp in the ongoing update? they should be within milliseconds from each other. Cause you can't switch from streaming reconstruction to old way seamlessly since different timestamps are used?
I'm sorry, I keep trying to bring it up at our team meetings but we've been busy end of quarter. I think if you have a lull and need to get a post out, you can go with it. It works perfectly well as an introduction and we can follow up on it with more in-depth examples of research.
Wed, Sep 16
Tue, Sep 15
I noticed here that events to the same page were criss-crossing and they're actually all really close in time, here they are split into two tables and re-ordered to match (btw if you keep tables outside of the code formatting they format nicely in phab):
I think it would be mutually beneficial to work on producing the events together, so we can help iterate on the schemas and implement the emit. We just need guidance on what edge cases we might be missing and creation of any new hooks if needed.
Fri, Sep 11
Oof, good point, I hadn't thought of the Druid version of this. I suppose I'll have to wipe and reload all of that as well, to apply the whitelist.
quick status update (tl;dr; we can announce it Monday):
Thu, Sep 10
potential easier way: require belonging to analytics-privatedata to log into stat1xxx.
x-wikimedia-debug shouldn't affect anything we do
Low priority for us, people can feel free to raise priority or grab it (should be relatively easy)
Wed, Sep 9
(we decided to deploy this next week, and we can do so by simply merging the change above (https://gerrit.wikimedia.org/r/626223))
This has been deployed for a while, we just forgot it in the wrong column.
Fri, Sep 4
pull request for public endpoints is here: https://github.com/wikimedia/restbase/pull/1273
We did option 1
Wed, Sep 2
Tue, Sep 1
Ok, thanks @RhinosF1 for the help, here's the problem as I understand it:
@JAllemandou, responding while Marcel is gone. I looked through and found a few of the reports rely on data that's changing (so if lost, would be hard/impossible to recompute). Why don't we just back up everything, the RU output is teeny tiny.
I thought we excluded these from denormalized, and left them just in page history. I can see why we'd want to include them. Hm... it is very confusing. Maybe we should give them a negative page id?
Mon, Aug 31
Do follow-up if somehow this is still relevant
(one idea would be to download an increased-resolution version)
we can/should do this in the event way
Haven't seen this in a while, maybe it went away!
We have to think more about how to accomplish this, taking into account all the security implications we've discovered on the first pass.
outdated now, though I still have the patches and I'll try to honor their spirit in a later refactor.
outdated, no further work on wikimetrics
Presto may not work here, unless we just poke a hole to a production cluster. We'll have it for reference.