Wonderful. I think bots can be excluded at all for this report. No need to make it optional.

group-bots vs name-bots, Ah I see, makes sense to makes this distinction, at least internally. Not sure if it's relevant to the end-user. AFAIK using the string 'bot' in user names is solely allowed for actual bots since long (users try to be funny with user names all the time, so not-really-a-bot names were a common occurrence but no longer). How about explicit(ly defined) vs implicit(ly defined) bots? My 2 cents.

Both bugs fixed.

I edited index.html and replaced en.wikipedia.org/wikistats/... with stats.wikimedia.org/...

@elukey yes, it works like a charm now, thanks so much :-)

Yes, it's me who issues the command.

Luca, the commands I use are in my first comment. So Both 'git pull' and 'git push' hang.
To be sure I checked my aliasses: nope these are the original commands.

Updating priority as I need to check-in bug fixes, which is way overdue. thanks.

@ChrisPins thanks for pointing this out.

No longer assignee.

@Aklapper I don't remember this bug at all.. As I'm wrapping up., this won't be for me. Will unhook. Thanks

I wonder: is there a link in Wikistats 2.0 to Wikistats 1.0 ? If not, why not?
After all most of the reports that contain the above-mentioned description have no counterpart in Wikistats 2.0 yet, and nothing is around the corner, right?
If people want detailed reports on content, editors and edits ( like those reports listed on https://www.mediawiki.org/wiki/Analytics/Wikistats/DumpReports/Future_per_report ), Wikistats 1.0 dump-based reports are often still their only resort, right?
Frankly I feel the above announcement which we added Dec 2017 better be reworded, or withdrawn.

Less than before. I am the only maintainer of Wikistats 1 and since July I work 30 hours per month, on my own request.

I see 4 colors (red, yellow, blue and green) which each contrast well with the background and none is grabbing more attention than the others. I like it too. Well done, @sahil505

@Reedy FYI I'm asking as I don't have server rights, so this is normal procedure.

Yes in my code it's CPAN as follows:

Reads a csv file with embedded commas in texts properly.

@Neil_P._Quinn_WMF for Wikistats a special cycle to generate stub dumps is started at beginning of new month. This takes a few days, with wp:en, commons and wikidata being the slowest.
Wikistats does indeed process a dump soon after completion, but each dump only once per month.

Oh BTW path other/pagecounts-ez/wikistats is obsolete, should be replaced by redirect.
On recent server migration we fixed the path, as it was counterintuitive (these csv files have nothing to do with pagecounts/pageviews)

The zip&publish step had not run yet. New zips are now online.

Aggregating data over all wikis and all projects has been done for the (now defunct) Report Card.
I just adapted that script to the new server environment (on stat1005) and ran it for the first time in a year.
Please see attached csv file.

All data look good to me now, and are being updated.

@Ottomata the daily merged files are copied now, thanks.

New daily pagecounts files exist at stat1005 in /srv/dumps/pagecounts-ez/merged/2018/2018-04

My bad! I totally forgot to follow up on this (and was behind on mail as well).

yup

Thanks Andrew, sounds good to me.

Am I bike-shedding? Perhaps.

published-datasets comes close, although images are no data, it would work for me
published-files ? [-files] doesn't add much
how about solely published? I can't login right now, but I vaguely rermember that already exists

Hmm. Imprecise naming can cause a lot of confusion, as it did already in recent years. (I did my part to add to this confusion, by misnaming things as well)

Thanks @Amitjoki , new version looks clean and quite intuitive to me.

@Nuria, could you please resend url to new map projection, I think you already did send but I can't find that mail (sorry).

I think this new feature underlines the need to present standard periods: click a button of 'one year' and get presented 12 months
not less, not more. There is some phabricator task (but I can't find it) on having the reports show such standardized period as the user asked for (be it a month, a year, or two years, not 18 months).

@Pamputt the url you give is about views not viewers. This is so often mixed up.

just a quick heads-up: I (finally) managed to regain access to stat1005, and will start cleanup tomorrow

Thanks @Ottomata, I just managed to ssh to stat1005 from Ubuntu. following advice on wikitech page

@MelodyKramer I also added some viauals to the Google doc.

"show daily unique devices instead of monthly unique devices on dashboard"

I found this in DammitSummarizeProjectviews.pl:

Both issues fixed in WikiCountsInput.pm and WikiCountsLog.pm

Stats for missing wikis were regenerated over the past days. Each days Wikipedia reports were refreshed. Yesterday the overhaul was completed.

I found two anomalies so far, but can't reproduce the problem.

@Nemo_bis, I'm still running Wikistats monthly till Wikistats 2.0 is taking over. As you know the later is in the UI design phase.

@Milimetric Ah OK, that's already as good as the current solution. The dumps don't contain redirect on a per revision level, only the current status. Thanks

I learned from @Neil_P._Quinn_WMF yesterday that the data lake doesn't know about redirects. If indeed that is the case, I'm curious: how do we discern countable pages (aka articles) from all the rest in so called 'content' namespaces? (In dumps there is the redirect tag, which isn't so easy to set, as #REDIRECT tag can be localized into many language versions). Thx

OK, I'll activate the patch in Wikistats to conform to this, and close once done

Ah, there is a commented line in my R input file, which I overlooked (and apparently is needed just once)

@Millimetric I'm not sure but I believe we said any change in the definition will always be reapplied to earlier months if feasible, right?

The one update I could apply is allowing redirects be included in edit counts.
I was wondering today, does it change anything that we need the internationalized tags for #redirect anyway (to keep redirects out of article counts)?
Unless I'm totally confused, we decided to counts edits to redirect pages purely for pragmatic reasons, but as we need that detection anyway for page counts, does that change anything?

I also responded on the talk page. And actually reconsidered while doing so. It seemed adding extensive code to detect internationalized redirects wasn't worth the trouble for filtering a few edits. But then I realized we need that detection anyway to filter redirect pages from article counts (or else English Wikipedia would have 13 million 'articles'). Hmm

@harej-NIOSH thanks

I'm really glad this work is happening. As for which columns to include here is some feedback:

all Wikistats files are up to date in git now

We have a monthly dataset [1] , for several years now, with title, month, views, not namespace but that could be inferred from the title I think

comScore data are also a thing of the past

Answers supplied to Tony Le (minus the links), with thanks to Quim and Andre

@PleaseStand kudoos for the in-depth analysis! This will be really useful also later on when the hadoop numbers for external links need to be vetted/understood.

Same process as above for fywiki yields for enwiki
19,635,379 external links

First let me state that the upcoming Wikistats 2.0 (parts of Wikistats migrated to hadoop) will be authoritative on this in a few months time, if not earlier.

Only the /wiki URLs are counted, right?

99% of the robots.txt queries on ru.wikipedia.org have same user agent string:
"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1)"
(so no clue here for bot detection)
and same ip address, which resolves to location in Amsterdam belonging to a top 5 global IT corporation

I collected all *non-bot* requests to ru.wikinews.org in May [1]
To my amazement I found robots.txt was the second highest requested page [2]. Isn't that odd?

With mobile traffic added for 'other projects' (commons, etc)
The average difference over 12 months is now 0.118% or 1/10 of a percent.

Ah good to see there is white list.

After adding Wp zero earlier, this week I added two more categories that were missing: mobile traffic to other projects than Wikipedia, and other 'Other Projects' than commons: wikidata, foundation, meta, species, incubator (desktop only, mobile to follow).

So my post from two days ago was a false lead. The issue of the rounding error is real, but also really small. More like 1 or 2 M rather than 400 M.

I should have said *part of the discrepancy* is as follows [..].

The discrepancy between [1] and [2] is as follows:
rightmost column in [2] is calculated from other columns, numbers which were already rounded to millions

Issues fixed. Thanks, elukey, and others.

This task is set to done, but my home dir hasn't been restored yet, and https://stats.wikimedia.org/cgi-bin/search_portal.pl?search=views produces an access error.

Count of links

Quote: "revert_info [can either be reverted_revision in case of sha1 equality, or if revert is present in comment (Not sure this - last one should be continued because of cross-language issues) ]"

Note that many users like puns or letter games and make up a nick with 'bot' in it just because they can.

Wikistats groups editors, edits and creates by user type, registered (and logged in) user, anonymous user or bot.