We limited the release to Wikipedias because there was a compelling use case. Folks involved with grants and programs needed that data for their day-to-day work, and we were making it available to them in private for a long time before we were able to understand the privacy implications and release publicly. We would need a similarly compelling use case to broaden the release, because the trade-off of value vs. privacy risk is always a consideration. Happy to reconsider given such use cases.

I think it's worth thinking through a migration to a hypothetical v3, just to be careful. So I did that today and I reason that a v2 -> v3 migration wouldn't be too bad. Mostly because the v2 urls respect a single hierarchical rule (project[/area[/metric]]). So a new version could change or extend that rule in a controlled way without breaking the old URLs. The output that v2 renders could even be materialized as static HTML and served at the old urls without any performance hit on the new site.

Moved to done, Nuria likes to look these over before she resolves them. But yes, no more work as far as I know.

In T232659#5813403, @Nuria wrote:

As far i can see month, day_of_week were not added but those can be easily obtained from the date, was that the intention @Milimetric ?

I'm looking at this again, and I think @sahil505 was exactly right - best to define the variables in some root file and use them everywhere. Side note for myself to look at https://github.com/MadLittleMods/postcss-css-variables instead of the package @sahil505 already tried, just in case it's better somehow. Will try to pick up this work as we decide what to do about the vue 3 upgrade.

Yes, Yael said it was fine to delete.

@bcampbell that's good news! I'm reading New Vector's privacy policy and while they seem very nice and everything, they do host their VMs on Amazon's AWS (https://modular.im/terms-of-service#privacy-policy). Would this still be ok with the group of folks that want to try Matrix or do others feel strongly we should host our own servers?

I wrote to Yael, will wait for their reply.

Good point! It's kind of a bug but one that would be very costly to fix. I explained it in the FAQ because I'm sure others are wondering: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Wikistats2/Metrics/FAQ#Why_are_pageviews_numbers_sometimes_0_and_sometimes_undefined?

HDFS is fine to store events, given Flink is agnostic to fetching batches/streams. That's great.

Maybe I have rose-colored glasses on, but I read the facebook comments a couple of times and it seems to me we have plans to address all the concerns:

For most of the use cases there we can generate historical events and store them somewhere, that way we can replay them through our Flink compute layer whenever we want. So one decision is where to store those kinds of events.

ok deployed and tested in deployment-prep, all good, ready for deploy to prod.

The output is here: https://analytics.wikimedia.org/published/datasets/periodic/reports/metrics/structured-data/

A productive meeting with Connie and Neil resulted in the following draft proposal:

@kzimmerman, the aggregate data is available, yes, but Dan Foy wanted to analyze this for other reasons, it's much richer raw data. I'm not sure if he handed off those reasons during his off-boarding (doesn't seem like it). So that's why we're checking with you, just making sure before we delete potentially valuable data.

@JAllemandou is adding page_is_redirect ok with you? I can piggy-back it on the change @Nuria mentioned but we should agree on doing that versus waiting for a higher quality option.

interesting, thanks for following up, and let us know if it still doesn't work

Ok, I traced this to the source data not being present. I am rerunning the jobs now and this should update by itself.

By the way, you can filter by date in the hierarchical view, little date selector on the left. While answering that I noticed that reports stopped a month ago, this is the error, I'm investigating:

ping @Neil_P._Quinn_WMF and @cchen to strategize on this. In my opinion, the ideal is that we update the tables in canonical_data regularly and use those in all our jobs. Any reason not to do that? For the specific differences here we could:

ok, well we need this for T219928 so I'm going to upgrade and test, to take care of any snags before the next upgrade. Then we can delay deploying if we want to upgrade again after PR 221.

The problem was just that there was no traffic on that page in the time period you selected. There was one hit on January 1st this year, and if you pass that to the API it works fine:

Hi Erik, we took a closer look, here are our thoughts:

no longer the strategy we're following

De-prioritizing until cloud infrastructure can support monitoring similar to what we can do in production.

The only subtask, supporting arrays and maps, is separate from this main effort, declaring victory. Well done!

Well done!

Good job!

team action: review Joseph's latest pull request: https://github.com/wikimedia/hdfs-tools/pull/4/files/66b61430d20bb9f4bf468e6232152bc7945e39d7..12fcc8b807e13c5ce19539ab1e5ceec9d5434a5c

Happy New Year, can we work on getting this out this quarter? :)

We're targeting January to roll this out

resolved for now, with all of us totally in agreement that we want historical redirects.

ping @cchen we have this new dataset, do you have use cases for this being in Druid, so we know what columns we need? Table is wmf.mediarequest:

suspecting the class loader is loading different stopwatch classes (check using maven tree)

taking to research during my ops week

Already done in June by Adam, follow-up from this was to update docs.

@Nuria: what I thought we were doing is productionizing @matthiasmullie's approach from T239565#5728117. This seemed sensible to me, and the change I submitted for T239565 properly comments that we are not measuring MediaInfo entities with the query on wbc_entity_usage. If any of this is wrong or not what others were expecting, I'd like a clear explanation as the multiple threads have become complicated to resolve objectively.

Ok, sounds like people want the feature, but for it to continue to be considered it needs a team to adopt and resource it. Does one of the teams mentioned above want to move this forward?

May I know what dashboard specific changes you made? (as we are still in the testing phase, I might be refreshing the dashboard soon with custom changes)

closing while cleaning up the backlog, let us know if you still see problems with matomo itself after checking the instrumentation.

This bothered me so I looked into it more. It can be explained by the fact that Spark matches newlines with % and Hive does not:

This is actually tricky for performance reasons. Lowering priority, please raise back up if this is important or becomes important later.

re: namespacing, for the record, I think namespaces would be useful, but I vote we don't start them with org/, always looked ugly to me and wouldn't be variable in our case.

@Erik_Zachte we are enabling kerberos on the whole cluster, so there's another layer of authentication. The announcement went out on the Analytics list and it's happening today, that's why you'll get some nasty error messages that have somewhere in them something like you pasted above. To authenticate with kerberos, you'll need to ask Luca to enable it for your user, but I'm not sure you need that if we just want to archive stuff. Let me know, and I can point you in the right direction.

This is done, updated docs and deployment code, deleting the instances now.

@srishakatux: just a ping that this is done and I deployed the capability to your dashboard. I still want to update the Dashiki docs which are in a very sad state, but before I get to that. To use the feature, you have to change your script to output a single file, named {submetric}.{format} (in your case maybe something like edit_percent.tsv in the reports/metrics/wmcs/ folder). The format of the file should be like this:

@Iflorez: sorry for the confusion, edited-pages is just the root of our pages-related metrics on the api. So in this case, I pointed to the edited-pages/new metric which gives you pages created stats. The following parameters are {wiki}/{editor type}/{page type}/{granularity}/{start}/{end}. You can see that the "Edited Pages" metric is a different edited-pages/aggregate path: https://wikitech.wikimedia.org/wiki/Analytics/AQS/Wikistats_2#Edited_Pages_2

Ok, seems like some of this confusion is getting cleared up. For my part, here's what I'm planning to do next:

I tracked down the other zero files I saved for @DFoy: