I'm afraid there's bugs that result RESTbase datas not synced well in Pageviews-API

In T172379#3499989, @Liuxinyu970226 wrote:

I'm afraid there's bugs that result RESTbase datas not synced well in Pageviews-API

Also, @elukey I would point to you that both was same before, err, 1 month ago.

AFAIK, MCS manipulates the data received from the Pageviews API, so perhaps that's why there is a difference? Could MCS people take a look?

MCS does some filtering.

• It removes the main page since that would always be the same.
• It removes pages that are not in the main-namespace. (We mainly want to get rid of Special pages. One could argue that Category pages might be OK. That might be up for discussion.)
• It removes pages where we think the traffic is bot-inflated.

Also I should mention that MCS responds with the previous days data. So, you'd want to compare https://zh.wikipedia.org/api/rest_v1/feed/featured/2017/08/02 with https://wikimedia.org/api/rest_v1/metrics/pageviews/top/zh.wikipedia/all-access/2017/08/01. That is because it's part of the feed data calculated on 2017/08/02. At that time (shortly after 00:00 UTC) it only knows what the top pages of the previous days were. It cannot predict the current day's top viewed articles.

I think @bearND explained it pretty well, data from pageview API is subjected to fluctuations due to bot traffic (not self-reported as such and thus labelled as "user") and also it includes Main_Page, which is not an interesting bit . Android app massages that data and creates a more interesting feed.

@Shizhao but where tells you that both should be the same?

@Liuxinyu970226 The views data of the same article is the same in same day, but one of the sources lacks data of "Wiki" and "維基媒體基金會".

In T172379#3509156, @Shizhao wrote:

@Liuxinyu970226 The views data of the same article is the same in same day, but one of the sources lacks data of "Wiki" and "維基媒體基金會".

If the answers above can't help you, then I'm not sure that what's your aim. Here is a @MusikAnimal tool called Tool-Pageviews where its FAQ says

Why can't I view data for today's date?

The Wikimedia pageviews API generally takes a full 24 hours to populate, sometimes longer. In some situations you may see data missing for yesterday's date as well, which will be left blank rather than showing a count of zero views.

Since you 2nd re-open, should I consider that you want to make such populate dynamically?

@Liuxinyu970226 I mean same data of the same day, but lacks data of "Wiki" and "維基媒體基金會".

https://zh.wikipedia.org/api/rest_v1/feed/featured/2017/08/02 is 20170801 data (not "Wiki" and "維基媒體基金會" data), "楚乔传" is 53127

https://wikimedia.org/api/rest_v1/metrics/pageviews/top/zh.wikipedia/all-access/2017/08/01 also is 20170801 data, "楚乔传" also is 53127

Although date of the url is not the same.(This is not yesterday's data! Yesterday is 20170807 )

@Shizao: Some pageviews that "appear" to be from users are actually from bots, some of those are removed on feeds endpoint but not on pageview api endpoint. That is the case for "wiki" and likely for "維基媒體基金會" (wikimedia foundation)

So those diffs are probably the reasons that both can't shown on feeds, but shown on pageviews (note that "維基媒體基金會", a zh-hant title, is a redirect to zh-hans "维基媒体基金会"):

At least one of non-bot edits that listed here are considered as (fake?) bot edits due to some reasons (they use AWB? Wikiplus? 3rd party scripts?)

Is there such this situation in other languages?

In T172379#3511168, @RazeSoldier wrote:

Is there such this situation in other languages?

Actually yes, except those which are rarely active. As using 2017-08-07 datas for example

• enwiki: shows "Main_Page" as the first article in pageviews, but "Disneyland_Railroad" as the first title in feed
• cebwiki: shows "Unang_Panid" as the first article in pageviews, but "Etnolinggwistiko" as the first title in feed
• svwiki: shows "Portal:Huvudsida" as the first article in pageviews, but "Armand_Duplantis" as the first title in feed
• dewiki: shows "Wikipedia:Hauptseite" as the first article in pageviews, but "Dachauer_Dora-Prozess" as the first title in feed
• nlwiki: shows "Hoofdpagina" as the first article in pageviews, but "Lieke_Martens" as the first title in feed
• frwiki: shows "Wikipédia:Accueil_principal" as the first article in pageviews, but "Bentley_Speed_8" as the first title in feed
• ruwiki: shows "Заглавная_страница" as the first article in pageviews, but "Бережная,_Ирина_Григорьевна" as the first title in feed
• itwiki: shows "Pagina_principale" as the first article in pageviews, but "Usain_Bolt" as the first title in feed
• eswiki: shows "Wikipedia:Portada" as the first article in pageviews, but "Usain_Bolt" as the first title in feed
• warwiki: shows "Syahan_nga_Pakli" as the first article in pageviews, but "Neolitiko" as the first title in feed
• plwiki: shows "Wikipedia:Strona_główna" as the first article in pageviews, but "Mistrzostwa_Świata_w_Lekkoatletyce_2017" as the first title in feed
• viwiki: shows "Trang_Chính" as the first article in pageviews, but "Phục_Hưng" as the first title in feed
• jawiki: shows "メインページ" as the first article in pageviews, but "市川雷蔵_(8代目)" as the first title in feed
• ptwiki: shows "Wikipédia:Página_principal" as the first article in pageviews, but "Tim_Maia" as the first title in feed
• ukwiki: shows "Головна_сторінка" as the first article in pageviews, but "Фуксман_Борис_Леонідович" as the first title in feed
• fawiki: shows "ماه‌گرفتگی" as the first article in pageviews, but "همسایه‌ها" as the first title in feed
• cawiki: shows "Portada" as the first article in pageviews, but "Ángel_Nieto_Roldán" as the first title in feed
• arwiki: shows "خاص:بحث" as the first article in pageviews, but "ناصر_الخليفي" as the first title in feed
• nowiki: shows "Portal:Forside" as the first article in pageviews, but "AFI's_100_Years…100_Movies" as the first title in feed

...
Due to those examples, I don't think that this task should remain open. It's expected that main pages and special pages are considered as articles in pageviews but not in feeds, and there's nothing to be resolved.

Also @Shizhao your comments are still foggy, at least

It removes pages where we think the traffic is bot-inflated.

only user data

Haven't you seen T172379#3510994? there's at least Antigng-bot and InternetArchiveBot available and you say those are "only user data"?!

Also @Shizhao your comments are still foggy, at least

It removes pages where we think the traffic is bot-inflated.

only user data

Haven't you seen T172379#3510994? there's at least Antigng-bot and InternetArchiveBot available and you say those are "only user data"?!

see https://tools.wmflabs.org/pageviews/?project=zh.wikipedia.org&platform=all-access&agent=user&range=latest-10&pages=Wiki

• source 1: https://wikimedia.org/api/rest_v1/metrics/pageviews/top/zh.wikipedia/all-access/2017/08/01
• source 2: https://zh.wikipedia.org/api/rest_v1/feed/featured/2017/08/02
• source 3: https://tools.wmflabs.org/pageviews/?project=zh.wikipedia.org&platform=all-access&agent=user&range=latest-10&pages={{article name}} (only user data from 20170801, not included bot and spider)

Why just source 2 lacks data of #1 and #2, and all other data is the same? At least source 2 and source 3 are data of only user.

Also @Liuxinyu970226

• enwiki: shows "Main_Page" as the first article in pageviews, but "Disneyland_Railroad" as the first title in feed
• cebwiki: shows "Unang_Panid" as the first article in pageviews, but "Etnolinggwistiko" as the first title in feed

..... ....

Your comparison is wrong. The first link data from 20170807, and the second link data from 20170806.

@Shizhao I can only reply one question for now

Why just source 2 lacks data of #1 and #2, and all other data is the same?

T172379#3509744 from nuria

In T172379#3511401, @Liuxinyu970226 wrote:

@Shizhao I can only reply one question for now

Why just source 2 lacks data of #1 and #2, and all other data is the same?

T172379#3509744 from nuria

if just remove some bot data. see "wiki", the views data is 66134 ( pageview api endpoint rank #1, include some bot data), "六四事件" is 6621 (feeds endpoint TOP30 rank #30, remove some bot data, "wiki" not in TOP30). so 66134 - 6621 = 59513. Does this mean that there bots has visited at least 59k times "wiki" page in 20170801? Is it really? 59k bot views data, I really can't believe it. Who can provide details of this 59k bot data? Just a bot, or a lot of bot?

I doubt the feed algorithm considers pageviews when removing a page and very likely pages that show up a lot (over and over) regardless of traffic will not get displayed.

Now, that being said the "wiki" article has a lot of bot traffic not identified as such, this user agent below is responsible for a lot of pageviews (~60.000) on August 1st.

{"browser_major":"9","os_family":"Windows 7","os_major":"-","device_family":"Other","browser_family":"IE","os_minor":"-","wmf_app_version":"-"}

In T172379#3514125, @Nuria wrote:

I doubt the feed algorithm considers pageviews when removing a page and very likely pages that show up a lot (over and over) regardless of traffic will not get displayed.

Now, that being said the "wiki" article has a lot of bot traffic not identified as such, this user agent below is responsible for a lot of pageviews (~60.000) on August 1st.

{"browser_major":"9","os_family":"Windows 7","os_major":"-","device_family":"Other","browser_family":"IE","os_minor":"-","wmf_app_version":"-"}

thx! It seems that "wiki" data was not normal for the last month: https://tools.wmflabs.org/pageviews/?project=zh.wikipedia.org&platform=all-access&agent=user&range=latest-60&pages=Wiki

"維基媒體基金會" seems to be?

Sorry but I do not understand what is the question, I think we have provided quite a bit of info about why things are different in those endpoints and will be closing ticket shortly.