We took a look at this. It seems that the confusion here is that this metric excludes redirects. So, we ran a query and found if we include redirects, for all time, on sv.wikipedia, we have:

8469790 creations, 2531709 deletions

And if we exclude redirects, we have:

4175681 creations, 2197 deletions

So most of the deletions you're talking about seem to be deletions of redirect pages. And the last set of numbers seems to match closely what's available on Wikistats.

There is the issue of filtering out deletions. The description of the metric we have on Wikistats says we are NOT filtering out deletions, but the description on Wikitech says we ARE filtering them out. We'll have to create a separate issue to look at that, but in this case the number of deletions of non-redirect pages is so small as to not matter.

@Milimetric I wonder if there's a mistake somewhere in the data around what is a redirect. Even if we just look at the last seven years, that would average less than one deletion of a non-redirect per day. This couldn't possible be true even ignoring the clean-up project – we delete far more than this.

I've also looked at some samples from the clean-up project and none of the deleted pages I looked at were redirects.

@Johan thank you, indeed, I believe you just found a bug.

Ok, so when we load deleted pages, we query the archive table. This doesn't have redirect information, so we set page_is_redirect to null. And later, when we generate the dataset that Wikistats queries, we check for NOT page_is_redirect. And NOT NULL is FALSE. That means we exclude all of the deleted pages by accidentally labeling them as redirects!

This is an interesting bug, because we want to exclude deleted pages from our metrics, and so this kind of does that. The number on Wikistats ends up still being correct, as the bug does what the metric wants, just via a mistake.

I'll talk to the team and see what they want to do. But what does this mean for you? Does the data make sense now or is there something else that feels wrong nested inside here?

But what does this mean for you? Does the data make sense now or is there something else that feels wrong nested inside here?

Answering my question a bit, the problem is that, in labeling these as redirects, we erase them from history completely. So we don't show the historical page count properly. This is tricky because the redirect information is really only stored in the wikitext itself, or in the page_is_redirect field on the Page table. The latter is lost when the page is deleted. So to better rebuild the history of the metric we'd have to look at the content or get more information in the delete event in the logging table. This will take some doing.

@Milimetric Yeah, my problem here is that I don't get the development of Swedish Wikipedia. We're missing a significant portion of the now deleted articles in the Wikistats graph, where the number of articles should have peaked years ago at a far higher count than today.

In T313955#8115392, @Johan wrote:

@Milimetric Yeah, my problem here is that I don't get the development of Swedish Wikipedia. We're missing a significant portion of the now deleted articles in the Wikistats graph, where the number of articles should have peaked years ago at a far higher count than today.

Makes perfect sense. I'm going to wrestle this task and the related T313985 to reflect the bug.