This is done, updated docs and deployment code, deleting the instances now.

@srishakatux: just a ping that this is done. I still want to update the Dashiki docs which are in a very sad state, but before I get to that. To use the feature, you have to change your script to output a single file, named {submetric}.{format} (in your case maybe something like edit_percent.tsv in the reports/metrics/wmcs/ folder). The format of the file should be like this:

@Iflorez: sorry for the confusion, edited-pages is just the root of our pages-related metrics on the api. So in this case, I pointed to the edited-pages/new metric which gives you pages created stats. The following parameters are {wiki}/{editor type}/{page type}/{granularity}/{start}/{end}. You can see that the "Edited Pages" metric is a different edited-pages/aggregate path: https://wikitech.wikimedia.org/wiki/Analytics/AQS/Wikistats_2#Edited_Pages_2

Ok, seems like some of this confusion is getting cleared up. For my part, here's what I'm planning to do next:

I tracked down the other zero files I saved for @DFoy:

I translated T238878#5708511 to Hive to familiarize myself with it and get ahead of productionizing it. I got similar numbers first as a sanity check and then grouped the numbers by the month of the page_latest revision's timestamp. I was wondering if the numbers increase in some nice way that we could report on regardless of the overall total. There's a fairly clear trend towards more structured data. And this is not a clear way to show the trend because we're just looking at the latest revision not all revisions, but maybe it's useful to @Abit as she thinks about this metric:

Tasked this briefly, want to coordinate with T239589 and make T239136 not necessary. To that end,

I'll take care of this either tomorrow or early next week. I was going to do it tonight but it's rejecting my ssh connect, probably either have to wait for puppet or something else is wrong.

Moving to last call, @Krinkle or @Nikerabbit to incorporate comments from T220056#5644705.

@tstarling we wanted to assign this to you in today's techcom meeting, but not in absentia, so just pinging you. Are you ok with driving this forward? We only had four people today but we generally agreed with you and Gergo that forking when maintainers leave is a bad idea.

Yay, I get to work with @mpopov :) Ok, questions:

I noticed that three folks used rev_deleted = 0 to mean "revision not deleted", but this field means something completely different (https://www.mediawiki.org/wiki/Manual:Revision_table#rev_deleted). Basically, it's about parts of the revision that have been suppressed from different types of users in the interface (like hiding revision comments). Good news is that revisions in the revision table are by definition not deleted. When a page is deleted, its revisions all get inserted into the archive table and removed from the revision table. So when you did rev_deleted = 0 you were just filtering out revisions that had any kind of suppression applied. I think that's a small number and didn't seem to add to the confusion above, but should be included just in case.

doc looks great, copy-edited a bit as I went through it

We looked at this in tech com and honestly we've lost track of where this is at. Is it still being discussed? Is there a preference for a way forward and resourcing and this is waiting for us? We can help set up discussions and move this forward, but we're not sure which way forward is.

I'm also liking the idea of experimental, and I think experiment is actually really nice and concise. It is fun and easy and kind of the same thing as instrument but less specific so it stays more flexible.

I started out thinking Sentry was the gold standard and I now prefer the way you're going about it, Jason. Because even if we want all those weird corner cases it makes much more sense for us to build up to them as we need them than to not understand exactly what we're seeing and why. Also love that this is much easier to maintain and get through code review, and would bring value to core.

Will comment on architecture next, but just jotting down thoughts about code as I read it:

Sold!

That’s the ugly/cute part, since we’re copying and not moving / to /v1, all the urls will work, relative or absolute. This is totally fine with v2 because that’s a client-side single page app that won’t overwrite anything except index.html

@elukey - we can talk more tomorrow but this solution hides Wikistats 1's index.html, which is how most people navigated the old site. I think we should preserve it. My idea seemed simple to me, let's see what I'm missing:

Wikistats 1 is no longer maintained.

In the time since we made this task, the Product Analytics and Analytics Engineering teams have been working closely on this dataset. We made some quality improvements and continue to vet together. This task is therefore outdated.

@DannyS712: could you tweak the batch to not leave a comment? Otherwise subscribers get notified

I'm sorry I thought I confirmed this - yes, we were blissfully ignorant on top of the cloud db views. Our other private sqoops don't use these tables either.

In T234188#5602707, @MGerlach wrote:

Most sessions start in ns=1 (article), but for new users the percentage is slightly smaller.

@srishakatux what's left to do here? Need any support updating the config.yaml for example?

In T211881#5615716, @kaldari wrote:

I guess its not clear to me what exactly the decision to kill the server side component means. Are we replacing it with a different server side component? If so, how do we know the new component won't suffer from the same concerns that Graphoid has.

No, the idea is that we will no longer have any server-side component dependency. This decision is based on the fact that we don't have engineering capacity to support a dedicated server-side component. It's obviously not ideal, and may have various problems (as mentioned by @Yurik and @Yair_rand), but the alternative would be to retire the Graph extension entirely, which would be worse.

I'm fine with just two repos, and the way you outline using them. Two thoughts on naming:

In T235143#5597019, @Nuria wrote:

Yes, definitely, his aggregated data is available there and in CSV files,

Do we know where are these files?

We have archived all the old geowiki (old name for geoeditors data) data to the archive hive database, tables are:

Thanks for the context, @Pine. This issue is in our backlog, meaning it's behind all of our other priorities. It has very little chance of being picked up by itself. We do have the data now to group pageviews by page id, and to figure out all names that a particular page had over its history. So someone could pick this up at a hackathon and make a workable prototype of collecting pageviews by title or someone has to argue that this should be higher priority for our team to pick it up.

Just trying to channel help here, @tstarling said he could help review a couple years ago, and @Tgr knows this code. Would either of you be able to help now?

Status update:

Plan to correct, with draft of scripts needed (EDITED using feedback below):

@Ijon, this is only an issue for the 1 to 4 activity level, but it raises another question we should've asked before:

review from @Ottomata is appreciated. And btw, why is the mediawiki_history fetch disabled? Is there some (problem/decision to be made) with the (rsync/labs servers) that would affect this as well?

Moving to last call per decision in today's TechCom meeting, looking to approve in 3 weeks.

Removing TechCom-RFC for now, just add it back when this task is ready to go through the RFC process.

In T236223#5597290, @srishakatux wrote:

I've a few questions:

• Is metrics-by-layout choice the right one for us?

In T193613#5562950, @daniel wrote:

Per the policy, public methods are considered stable (safe to call) per default. However, public constructors are not considered stable. And public methods are not automatically considered fixed (safe to override).

• Erik compiled data from sampled logs which are no longer available

I tried to enable the vagrant role again to test and I got this error, so I figure either my vagrant is messed up or this will prevent me from having a clean test. I think if it works for other folks on a clean vagrant, it's fine, I'll try to test again when I look at the unit testing RFC next.

In T234649#5551056, @Gilles wrote:

Something missing in the announcements about the FAWG and this task is the possibility that none of the available options are good trade-offs/long-term investments. There is a maturity of the environment required to do such a migration and I don't know if the current frontend framework landscape has reached that stage yet.
For example, it's possible that the most useful parts of these frameworks are things that should be made part of the web platform itself and that resources and efforts in terms of long-term investments would be better made in the standards space (W3C, WHATWG, etc.).
By framing the discussion on "which framework should we pick", it might lock the discussion into a close-minded view in that respect.
If there is a need so common that most websites end up using similar frameworks to achieve it, it might be something that should be part of JS/CSS. We've seen that process from beginning to end with jQuery being less and less useful as features have made their way into JS itself. Now we're still paying the price of jQuery's weight because we're locked into it, it's completely baked into all our code, even though we use very little of it.
We might be early in such a cycle with frontend frameworks. It would be unfortunate to invest a lot of resources in the jQuery-like phase when we could be helping and pushing towards the endgame. I.e. that it becomes part of the web platform and using those features becomes a non-issue. Worse, we could make a heavy frontend framework a page load requirement that will be very difficult to undo, with a potential usefulness lifespan of a few years.

anticipating release, docs here: https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Edits/Geoeditors/Public

I wouldn't worry too much about how others are using this in their own mediawiki installs. Not that it's not important, just that we can't possibly guess as to how they might want to do that. Just having three repos with some common stuff will allow for plenty of flexibility and refactoring later on.

I like a single namespace, especially because having "common" as a root would be too vague. This might be useful:

I looked at this and it's a clever way to get some rough information to answer the main question. But I just wanted to point out: it's not by accident that this kind of connection is hard to make. We made a conscious decision a while back that we should not optimize accessing reading patterns of specific users. Being able to get this data is problematic for privacy reasons, and more so if we can get this data quickly. So, nice work, but I would hesitate before optimizing it too much more.