Add alarms for high volume of views to pages with replacement characters
Open, MediumPublic
Actions

Assigned To

None

Authored By

	• Tbayer
	Nov 6 2015, 5:27 AM

Description

NOTE: This task was titled "Investigate anomalous views to pages with replacement characters".

Recently I generated a list of the top 200 most viewed pages (URLs) on Wikimedia sites during May-October 2015 from pageview_hourly.[1] It turns out that lot among them - 24 of 200 - contain "�"s in their page name.

"�" is the Unicode replacement character, "most commonly seen when a font does not contain a character, but is also seen when the data is invalid and does not match any character". It is actually disallowed in page names ("bad title"), so something is clearly wrong here either with how these views are recorded (hence filing this bug under Analytics) or with requests that readers' browsers send to our sites (hence the LangEng tag).

Wikipedias in non-Latin languages like Russian and Japanese seem most affected, but it happens on the English Wikipedia too, for example.[2]

Views recorded for these page names are well distributed geographically [3] and across operating systems [4], so it's very unlikely that this is just a few unrecognized spiders running amok, and it seems these numbers do correspond to human reader activity somehow. On the other hand, their rate has varied a lot in the time since May:

Views to pages with replacement characters, April-November 2015.png (371×600 px, 13 KB)

(Query:[5])

Or in a log scale:

Views to pages with replacement characters, April-November 2015 (log scale).png (371×600 px, 20 KB)

On June 3, 2015 for example, page names with "�" made up more than 8% of total non-spider views.

It looks like these spurious views are fairly even distributed among external, internal and unknown referrers[6], so we can also rule out that they are just caused by malformed external links.

I'm also quite certain it's not a problem with how the page titles were retrieved from the database on Hive. That was my first thought, but after trying with various access methods (using Hue, the Hive CLI outputting to a terminal and to a file, the new Beeline CLI - which I erroneously conjectured to work fine on #wikimedia-analytics, because I had happened to test a date range and project without anomalous results with Beeline) and looking at the distribution of pageviews, I think that that possibility can be ruled out.

Footnotes

[1]: top 200 May 1-Oct 23 2015.txt

SELECT CONCAT('https://',project,'.org/wiki/',page_title), SUM(view_count) AS views
FROM wmf.pageview_hourly
WHERE
   year = 2015
   AND agent_type = "user"
GROUP BY project, page_title
ORDER BY views DESC LIMIT 200;

https://en.wikipedia.org/wiki/Main_Page	3950733770
https://de.wikipedia.org/wiki/Wikipedia:Hauptseite	518079768
https://en.wikipedia.org/wiki/Special:Search	412911496
https://es.wikipedia.org/wiki/Wikipedia:Portada	301576053
https://it.wikipedia.org/wiki/Pagina_principale	294084319
https://fr.wikipedia.org/wiki/Wikipédia:Accueil_principal	271889431
https://ru.wikipedia.org/wiki/Заглавная_страница	235694286
https://en.wikipedia.org/wiki/-	169453279
https://pt.wikipedia.org/wiki/Wikipédia:Página_principal	109785321
https://id.wikipedia.org/wiki/Template:AdvancedSiteNotices/ajax	89707047
https://en.wikipedia.org/wiki/Special:RecentChanges	89328332
https://de.wikipedia.org/wiki/Spezial:Suche	76612196
https://en.wikipedia.org/wiki/Angelsberg	74498022
https://ja.wikipedia.org/wiki/メインページ	68849608
https://nl.wikipedia.org/wiki/Hoofdpagina	65984206
https://es.wikipedia.org/wiki/Especial:Buscar	65366727
https://fi.wikipedia.org/wiki/Wikipedia:Etusivu	57095356
https://zh.wikipedia.org/wiki/Wikipedia:首页	47913120
https://uk.wikipedia.org/wiki/Template:AdvancedSiteNotices/ajax	46411452
https://pl.wikipedia.org/wiki/Wikipedia:Strona_główna	44937302
https://ja.wikipedia.org/wiki/������������	41477439
https://en.wikipedia.org/wiki/Special:BlankPage	40173544
https://ru.wikipedia.org/wiki/Служебная:Поиск	35717349
https://zh.wikipedia.org/wiki/Special:最近更改	34860755
https://en.wikipedia.org/wiki/Special:Book	33353391
https://cs.wikipedia.org/wiki/Hlavní_strana	33111756
https://tr.wikipedia.org/wiki/Ana_Sayfa	31452102
https://en.wikipedia.org/wiki/User:GoogleAnalitycsRoman/google-api	30707913
https://es.wikipedia.org/wiki/-	29961995
https://fr.wikipedia.org/wiki/Spécial:Recherche	29103311
https://de.wikipedia.org/wiki/-	28686671
https://sv.wikipedia.org/wiki/Portal:Huvudsida	28455196
https://ja.wikipedia.org/wiki/-	28159010
https://en.wikipedia.org/wiki/Template:GeoTemplate	26438021
https://hu.wikipedia.org/wiki/Kezdőlap	24789961
https://commons.wikimedia.org/wiki/Special:Search	24282981
https://zh.wikipedia.org/wiki/-	24082799
https://ja.wikipedia.org/wiki/特別:検索	22578179
https://ru.wikipedia.org/wiki/Special:Search	22426302
https://ja.wikipedia.org/wiki/���������������	21354893
https://ja.wikipedia.org/wiki/���������	21045703
https://fr.wikipedia.org/wiki/Sp?cial:Search	20126196
https://ja.wikipedia.org/wiki/特別:最近の更新	19730989
https://en.wikipedia.org/wiki/Malware	19291150
https://ru.wikipedia.org/wiki/-	18893424
https://nl.wiktionary.org/wiki/Speciaal:Zoeken	18670611
https://ca.wikipedia.org/wiki/Internet	18124959
https://pt.wikipedia.org/wiki/Especial:Pesquisar	17780182
https://de.wikipedia.org/wiki/Spezial:Weblinksuche	17757280
https://en.wikibooks.org/wiki/LaTeX/Document_Structure	17728366
https://ar.wikipedia.org/wiki/الصفحة_الرئيسية	17660205
https://zh.wikipedia.org/wiki/Special:搜索	17136906
https://vi.wikipedia.org/wiki/Trang_Chính	16282348
https://ar.wikipedia.org/wiki/-	15818401
https://da.wikipedia.org/wiki/Forside	15656648
https://de.wikipedia.org/wiki/Special:Search	14939334
https://it.wikipedia.org/wiki/Speciale:Ricerca	14936157
https://id.wikipedia.org/wiki/Halaman_Utama	14921635
https://ja.wikipedia.org/wiki/������������������	14914357
https://fr.wikipedia.org/wiki/-	14597405
https://ro.wikipedia.org/wiki/Pagina_principală	14508668
https://zh.wikipedia.org/wiki/Special:链出更改	14458353
https://en.wikipedia.org/wiki/Deaths_in_2015	14396192
https://pt.wikipedia.org/wiki/-	13432687
https://en.wikipedia.org/wiki/Special:Watchlist	12814398
https://zh.wikipedia.org/wiki/���������	12590841
https://de.wikipedia.org/wiki/Spezial:Anmelden	12257873
https://en.wikipedia.org/wiki/Jurassic_World	11950116
https://ru.wikipedia.org/wiki/������	11216473
https://pl.wikipedia.org/wiki/Specjalna:Szukaj	10965266
https://ru.wikipedia.org/wiki/�����	10913472
https://ja.wikipedia.org/wiki/���������������������	10779241
https://de.wikipedia.org/wiki/Hauptseite	10330557
https://commons.wikimedia.org/wiki/Main_Page	9996226
https://en.wikipedia.org/wiki/Avengers:_Age_of_Ultron	9844236
https://ru.wikipedia.org/wiki/�������	9489397
https://ru.wikipedia.org/wiki/��������	9373099
https://en.wiktionary.org/wiki/Wiktionary:Main_Page	9333138
https://nl.wikipedia.org/wiki/Speciaal:Zoeken	9312610
https://en.wikipedia.org/wiki/Pablo_Escobar	9296618
https://ru.wikipedia.org/wiki/����	9258156
https://ja.wikipedia.org/wiki/������	9241579
https://ja.wikipedia.org/wiki/������������������������	9183241
https://en.wikipedia.org/wiki/Donald_Trump	9166835
https://en.wikipedia.org/wiki/List_of_Bollywood_films_of_2015	9134179
https://en.wikipedia.org/wiki/Facebook	9071720
https://kk.wikipedia.org/wiki/Template:AdvancedSiteNotices/ajax	9026941
https://en.wikipedia.org/wiki/Special:MobileMenu	8941355
https://en.wikipedia.org/wiki/Mad_Max:_Fury_Road	8887619
https://uk.wikipedia.org/wiki/Головна_сторінка	8794233
https://en.wikipedia.org/wiki/Game_of_Thrones	8745274
https://fa.wikipedia.org/wiki/ویژه:ورود_به_سامانه	8572793
https://id.wikipedia.org/wiki/-	8507302
https://en.wikipedia.org/wiki/A._P._J._Abdul_Kalam	8284113
https://en.wikipedia.org/wiki/Google	8242968
https://wikimediafoundation.org/wiki/Terms_of_Use	8023245
https://zh.wikipedia.org/wiki/������������	7998626
https://fr.wikipedia.org/wiki/Wikip��dia:Accueil_principal	7979428
https://ja.wikipedia.org/wiki/���������������������������	7880728
https://en.wikipedia.org/wiki/Nuclear_magnetic_resonance_spectroscopy	7867169
https://id.wikipedia.org/wiki/Istimewa:Pencarian	7813450
https://sr.wikipedia.org/wiki/Главна_страна	7803138
https://en.wikipedia.org/wiki/United_States	7675129
https://de.wikipedia.org/wiki/Spezial:Buch	7610434
https://en.wikipedia.org/wiki/Eazy-E	7471043
https://en.wikipedia.org/wiki/Ddd	7433941
https://en.wikipedia.org/wiki/Game_of_Thrones_(season_5)	7382274
https://it.wikipedia.org/wiki/-	7346806
https://en.wikipedia.org/wiki/Bajrangi_Bhaijaan	7266442
https://en.wikipedia.org/wiki/Floyd_Mayweather,_Jr.	7262380
https://meta.wikimedia.org/wiki/Main_Page	7257715
https://ru.wikipedia.org/wiki/Служебная:Ссылки_сюда	7170290
https://cs.wikipedia.org/wiki/Speciální:Hledání	7075394
https://ko.wikipedia.org/wiki/위키백과:대문	7004688
https://en.wikipedia.org/wiki/Stephen_Hawking	6979413
https://ar.wikipedia.org/wiki/خاص:بحث	6884847
https://ru.wikipedia.org/wiki/������������������_����������������	6838779
https://ja.wikipedia.org/wiki/������������������������������	6811375
https://en.wikipedia.org/wiki/Website	6786692
https://en.wikipedia.org/wiki/Special:CiteThisPage	6740468
https://da.wikipedia.org/wiki/Speciel:Seneste_ændringer	6648351
https://en.wikipedia.org/wiki/2015_in_film	6643983
https://pl.wikipedia.org/wiki/Special:Search	6626950
https://en.wikipedia.org/wiki/Orange_Is_the_New_Black	6593267
https://en.wikipedia.org/wiki/Chris_Kyle	6585441
https://en.wikipedia.org/wiki/Caitlyn_Jenner	6539879
https://ru.wikipedia.org/wiki/����������	6500017
https://en.wikipedia.org/wiki/Ronda_Rousey	6473199
https://en.wikipedia.org/wiki/Stephen_Curry	6408829
https://es.wikipedia.org/wiki/Especial:Entrar	6395071
https://de.wikipedia.org/wiki/Spezial:Beobachtungsliste	6297673
https://en.wikipedia.org/wiki/India	6224400
https://en.wikipedia.org/wiki/Islamic_State_of_Iraq_and_the_Levant	6208548
https://en.wikipedia.org/wiki/Portal:Current_events	6204018
https://fr.wikipedia.org/wiki/Spécial:Modifications_récentes	6187732
https://ru.wikipedia.org/wiki/���	6096976
https://no.wikipedia.org/wiki/Portal:Forside	6073743
https://bg.wikipedia.org/wiki/Начална_страница	6068727
https://en.wikipedia.org/wiki/Straight_Outta_Compton_(2015_film)	5971556
https://mediawiki.org/wiki/MediaWiki	5920023
https://en.wikipedia.org/wiki/Bernie_Sanders	5902370
https://en.wikipedia.org/wiki/Baahubali:_The_Beginning	5884765
https://en.wikipedia.org/wiki/Dr._Dre	5881774
https://en.wikipedia.org/wiki/Flags_of_the_Confederate_States_of_America	5866460
https://en.wikipedia.org/wiki/Ruby_Rose	5859660
https://en.wikipedia.org/wiki/N.W.A	5824111
https://en.wikipedia.org/wiki/Terminator_Genisys	5775976
https://he.wikipedia.org/wiki/עמוד_ראשי	5764826
https://en.wikipedia.org/wiki/List_of_Game_of_Thrones_episodes	5738637
https://fr.wikipedia.org/wiki/Accueil_(homonymie)	5687549
https://vi.wikipedia.org/wiki/-	5668948
https://ko.wikipedia.org/wiki/특수:검색	5617110
https://en.wikipedia.org/wiki/List_of_highest-grossing_films	5612829
https://zh.wikipedia.org/wiki/������	5597419
https://de.wikipedia.org/wiki/Template:GeoTemplate	5536877
https://ru.wikipedia.org/wiki/������������	5502537
https://en.wikipedia.org/wiki/Ant-Man_(film)	5496621
https://en.wikipedia.org/wiki/Academy_Awards	5476218
https://en.wikipedia.org/wiki/Lionel_Messi	5467386
https://ru.wikipedia.org/wiki/���������	5405438
https://fa.wikipedia.org/wiki/صفحهٔ_اصلی	5384935
https://en.wikipedia.org/wiki/Ice_Cube	5324917
https://de.wikipedia.org/wiki/Spezial:Letzte_Änderungen	5320086
https://en.wikipedia.org/wiki/Dwayne_Johnson	5301339
https://es.wikipedia.org/wiki/Special:Search	5261454
https://en.wikipedia.org/wiki/World_War_II	5260716
https://en.wikipedia.org/wiki/Suicide_Squad_(film)	5240753
https://pl.wikipedia.org/wiki/Template:GeoTemplate	5208262
https://en.wikipedia.org/wiki/The_Flash_(2014_TV_series)	5114356
https://en.wikipedia.org/wiki/Whitey_Bulger	5110246
https://en.wikipedia.org/wiki/True_Detective_(TV_series)	5066957
https://zh.wikipedia.org/wiki/Running_Man	5060715
https://en.wikipedia.org/wiki/Fear_the_Walking_Dead	5029503
https://en.wikipedia.org/wiki/Taylor_Swift	5002322
https://en.wikipedia.org/wiki/Barack_Obama	4990982
https://en.wikipedia.org/wiki/Cristiano_Ronaldo	4978517
https://en.wikipedia.org/wiki/2015_Copa_América	4975466
https://en.wikipedia.org/wiki/Elizabeth_II	4966913
https://pt.wikipedia.org/wiki/Wikipedia:Página_principal	4920049
https://en.wikipedia.org/wiki/Mr._Robot_(TV_series)	4909087
https://en.wikipedia.org/wiki/Help:IPA_for_English	4907615
https://en.wikipedia.org/wiki/Star_Wars:_The_Force_Awakens	4904135
https://en.wikipedia.org/wiki/The_Walking_Dead_(TV_series)	4889111
https://en.wikipedia.org/wiki/Kanye_West	4861871
https://en.wikipedia.org/wiki/Furious_7	4840048
https://wikimediafoundation.org/wiki/Home	4835328
https://en.wikipedia.org/wiki/Manny_Pacquiao	4820494
https://en.wikipedia.org/wiki/Fetty_Wap	4795556
https://en.wikipedia.org/wiki/Wikipedia	4743801
https://en.wikipedia.org/wiki/Captain_America:_Civil_War	4722014
https://ja.wikipedia.org/wiki/���������������������������������	4686784
https://en.wikipedia.org/wiki/Paul_Walker	4618488
https://en.wikipedia.org/wiki/Steve_Jobs	4617029
https://en.wikipedia.org/wiki/Suicide_Squad	4599728
https://en.wikipedia.org/wiki/Inside_Out_(2015_film)	4595928
https://en.wikipedia.org/wiki/Kim_Kardashian	4579288
https://es.wikipedia.org/wiki/Especial:Libro	4529801
https://en.wikipedia.org/wiki/Dulce_María	4527902
https://en.wikipedia.org/wiki/Tom_Hardy	4525020
https://en.wikipedia.org/wiki/Serena_Williams	4475772

(this list contains some other oddities that are worth investigating, I already looked at Angelsberg and might file a separate bug about it)

[2]

hive (wmf)> SELECT project, SUM(view_count) AS views FROM wmf.pageview_hourly WHERE year = 2015 and month = 6 AND page_title LIKE '%�%' GROUP BY project ORDER BY views DESC LIMIT 50;

project	views
ru.wikipedia	177741998
ja.wikipedia	107116573
es.wikipedia	44089839
zh.wikipedia	35606734
fr.wikipedia	28696604
en.wikipedia	25515914
de.wikipedia	16948577
pt.wikipedia	15066605
ar.wikipedia	11618348
tr.wikipedia	11529255
pl.wikipedia	10527791
ko.wikipedia	9308818
fa.wikipedia	8156094
uk.wikipedia	6095328
commons.wikimedia	5615428
cs.wikipedia	5473564
he.wikipedia	4971069
th.wikipedia	4887358
hu.wikipedia	4079865
vi.wikipedia	3794512
sv.wikipedia	2975373
it.wikipedia	2835094
sr.wikipedia	2655465
bg.wikipedia	2416994
ru.wiktionary	2308125
el.wikipedia	2234374
ro.wikipedia	2191295
en.wiktionary	1613834
be.wikipedia	1551227
fr.wiktionary	1434420
ca.wikipedia	1259693
sk.wikipedia	1229912
hi.wikipedia	1183813
fi.wikipedia	1042819
nl.wikipedia	1037110
bn.wikipedia	893571
da.wikipedia	874715
kk.wikipedia	809861
no.wikipedia	800887
ka.wikipedia	623034
lt.wikipedia	592751
hr.wikipedia	573796
he.wikisource	497302
hy.wikipedia	456730
ta.wikipedia	431583
mk.wikipedia	430303
fa.wikisource	413805
ko.wikisource	402373
lv.wikipedia	394519
fr.wikisource	391089

[3]

SELECT continent, SUM(view_count) as views FROM wmf.pageview_hourly WHERE year = 2015 AND month = 6 AND page_title = "������������" AND project = "ja.wikipedia" AND agent_type = "user" GROUP BY continent ORDER BY continent LIMIT 1000;

continent	views
Africa	2705
Asia	20113970
Europe	112468
North America	449616
Oceania	24136
South America	7862
Unknown	2935

[4]

SELECT user_agent_map['os_family'] AS OS_family, SUM(view_count) as views FROM wmf.pageview_hourly WHERE year = 2015 AND month = 6 AND page_title = "������������" AND project = "ja.wikipedia" AND agent_type = "user" GROUP BY user_agent_map['os_family'] ORDER BY views DESC LIMIT 10;

os_family	views
Windows 7	5800663
iOS	5434875
Android	4926276
Windows 8.1	1694772
Other	868088
Mac OS X	633296
Windows Vista	537291
Windows XP	400079
Linux	241776
Windows 8	97222

[5]

hive (default)> SELECT year, month, day, CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) AS date, SUM(view_count) as views FROM wmf.pageview_hourly WHERE year = 2015 AND page_title LIKE '%�%' AND agent_type = "user" GROUP BY year, month, day ORDER BY year, month, day LIMIT 1000;

[6]

SELECT referer_class, SUM(view_count) as views FROM wmf.pageview_hourly WHERE year = 2015 AND month = 6 AND page_title LIKE '%�%' AND agent_type = "user" GROUP BY referer_class;

referer_class	views
external	276431095
internal	124159749
unknown	97135216

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Declined		None	T159840 Alarm on data quality issues
		Open		None	T117945 Add alarms for high volume of views to pages with replacement characters

Event Timeline

• Tbayer created this task.Nov 6 2015, 5:27 AM

• Tbayer raised the priority of this task from to Needs Triage.

• Tbayer updated the task description. (Show Details)

• Tbayer added projects: Analytics, Language-Team.

• Tbayer subscribed.

Restricted Application added subscribers: StudiesWorld, revi, Aklapper. · View Herald TranscriptNov 6 2015, 5:27 AM

• kevinator edited projects, added Analytics-Backlog; removed Analytics.Nov 12 2015, 8:10 PM

• kevinator set Security to None.

• Tbayer mentioned this in T117343: Remove spider traffic from "top" results.Nov 17 2015, 12:22 AM

• Tbayer mentioned this in T118933: Wikimedia "top" pageviews API weirdness with the "Paul_Elio" article [5 pts] {slug}.Nov 18 2015, 7:48 AM

Nemo_bis added a project: Datasets-Webstatscollector.Nov 19 2015, 11:55 AM

Nemo_bis subscribed.

• Nuria triaged this task as Medium priority.Dec 2 2015, 9:16 PM

Milimetric moved this task from Incoming to Backlog on the Analytics-Backlog board.Dec 3 2015, 6:31 PM

We looked into this issue quite a bit as part of the data loading jobs for the Pageview API. The problem is pretty complicated, but we think the current pipeline handles titles relatively safely and well considering that complication. We'll prioritize this low for now, but if you guys have time to do a more thorough analysis and suggest specific improvements that we can make, please let us know. Thanks :)

Thanks Dan! I'm still curious about the actual reason and whether it could affect either the validity of our pageview data analyses, or the experience of readers accessing pages with non-ASCII titles. But it's reassuring to know that the issue is on your radar and that you have it under control.

As far as I could personally tell from the anecdotal analysis I did (as in not over 6 months of data but just over a few days), this almost doesn't affect overall statistics at all.

As far as experience of readers accessing non-ASCII titles, that's fully separate from any problems we have in the logging pipeline, and I'm not worried about those at all. I was able to make some pretty insane titles work as part of my checking (like, titles with zero-width spaces, all kinds of crazy control characters, etc.). So I think we're good for now, though this is one of those weird issues that may linger for a while.

OK, so do we now know what may be causing this problem? Good that it's not disrupting the logging pipeline, and not caused by the pipeline. But if these are faithful reflections of actual URLs requested by users' browsers, that would indicate that a lot of readers have not been getting the content that they want (and likely make this an issue for Language Engineering).

And 8% of total non-spider views (see above) did actually look like something that could affect overall statistics. So it would be good to have some cause to believe that this won't return to the levels seen in June-August.

Yeah, I agree, Language Engineering should have a look at this.

I am afraid that at least I have no good idea what could cause this.

One could be incorrectly truncated log entries, where a link is truncated in middle of utf-8 byte and then interpreted in wrong encoding. But that's not easy to confirm if the raw bytes are not present in the page view logs.

Milimetric raised the priority of this task from Medium to High.Dec 15 2015, 11:25 PM

• ggellerman edited projects, added Analytics; removed Analytics-Backlog.Jan 12 2016, 7:41 PM

Milimetric moved this task from Incoming to Event Platform on the Analytics board.Jan 12 2016, 7:41 PM

See also T108867: Research Spike: How do redirects affect pageviews [8 pts] {hawk} . Possibly related, too: T104755 ("Wikimedia's URL-routing logic straddles five layers ...")

mforns subscribed.Jan 21 2016, 12:57 PM

Milimetric moved this task from Event Platform to Wikistats on the Analytics board.Aug 29 2016, 3:56 PM

• Tbayer mentioned this in T145043: Special characters showing up as question marks in /pageviews/top endpoint.Sep 11 2016, 5:23 PM

It looks like @West.andrew.g just encountered the same or a similar issue, see https://en.wikipedia.org/wiki/User_talk:West.andrew.g/Popular_pages#Regarding_the_.EF.BF.BD_characters .

@West.andrew.g's FFFD count seems to match up with UTF-8 byte count (verified with '\u2013' in [[Obsessive��compulsive disorder]] and 'é' in Milowent's "Murder of JonBenét Ramsey" example; matches up with others). Therefore accidental, file-level truncation is likely not the cause for him. (UTF-8 is pretty resilient and self-synchronizing anyway. Per-byte truncation when decoding may still be the cause (Failed UTF-8/ASCII out-of-range). "Forced ASCII" locales like C (aka POSIX) in glibc can cause issues somewhere too.) There are also some ??? sequences in west.andrew's page that appear analogous to � in terms of byte count behavior.

For this issue, the https://fr.wikipedia.org/wiki/Wikip��dia:Accueil_principal example with an é in place of the FFFDs seems to show consistent behavior. But that's just one data point -- the hypothesis should be verified with more of such guessable FFFD's.

(Admittedly I am more interested in feeding the API with emojis or other Unicode SMP characters after learning it's written in Java.)

Restricted Application added a subscriber: Stang. · View Herald TranscriptJan 25 2017, 1:15 PM

• Tbayer updated the task description. (Show Details)Feb 6 2017, 10:19 PM

Liuxinyu970226 added a project: Chinese-Sites.Feb 18 2017, 2:10 AM

Liuxinyu970226 subscribed.

Liuxinyu970226 added a project: Bengali-Sites.Feb 18 2017, 2:12 AM

Liuxinyu970226 added projects: Hindi-Sites, Tamil-Sites.

Liuxinyu970226 added a project: Spanish-Sites.Feb 18 2017, 2:22 AM

Liuxinyu970226, this problem seems to affect all sites independent of language settings. Are you sure you are going to tag all languages that often use titles with non-ASCII characters?

Liuxinyu970226 added a comment.Feb 22 2017, 2:47 PM

This comment was removed by Liuxinyu970226.

In T117945#3046773, @Liuxinyu970226 wrote:

At least this isn't happened on eowiki, happy now?

That's good to hear. Do you think it is working normally (unlike other wikis) for a particular technical reason worthy of investigation?

PS: You also have a good journey of

Good luck replicating my hash-brown-table flame.

"LMPRNG (c.f. https://artoria2e5.github.io/PRCoords/demo )"

Fixed. Thanks for pointing that out, but did anyone force you to read that? Try to pick a better sample from this thread, like "Unicode SMP" or "ASCII". By the way, "cf." is written with one single 0x2E (cf. enwiki).

Liuxinyu970226 added a comment.Feb 22 2017, 10:45 PM

This comment was removed by Liuxinyu970226.

• Nuria moved this task from Wikistats to Dashiki on the Analytics board.Mar 13 2017, 5:09 PM

• Nuria added a project: Analytics-Data-Quality.Jul 13 2017, 4:14 PM

• Nuria moved this task from Dashiki to Backlog (Later) on the Analytics board.

Jayprakash12345 subscribed.Jul 13 2017, 4:41 PM

Restricted Application added subscribers: jeblad, Danmichaelo. · View Herald TranscriptOct 19 2017, 9:04 AM

• fdans edited projects, added Analytics-Kanban; removed Analytics.Oct 23 2017, 3:54 PM

mforns claimed this task.Dec 21 2017, 2:18 PM

mforns moved this task from Next Up to In Progress on the Analytics-Kanban board.Dec 21 2017, 2:22 PM

Hi all,

I've been looking into this for a while.
It seems that this issue is not happening any more, at least for the last 5 months.
See the following queries (BTW, thanks @Tbayer for posting the queries you used for the previous analysis!):

Timeseries of views to pages with �

SELECT
    CONCAT(year, LPAD(month, 2, '0'), LPAD(day, 2, '0')) AS date,
    SUM(view_count) AS views
FROM wmf.pageview_hourly
WHERE
    year = 2017 AND
    month > 6 AND
    month < 12 AND
    page_title LIKE '%�%'
GROUP BY
    CONCAT(year, LPAD(month, 2, '0'), LPAD(day, 2, '0'))
ORDER BY views DESC
LIMIT 1000
;

In the 2015 analysis, the values for this kind of pageviews were peaking between 30 to 40 millions per day, and had a lower base line just below 100K per day.
The current query shows no peaks and a base line of 200K-300K per day, which corresponds roughly to 0.06% of all wikimedia pageviews.
There has been an 2-3x increase in the base line since 2015, though. Maybe it is be due to an increase in edits and pageviews for wikis that use non-ascii characters extensively?

Top most viewed pages

SELECT
    CONCAT('https://', project, '.org/wiki/', page_title),
    SUM(view_count) AS views
FROM wmf.pageview_hourly
WHERE
    year = 2017 AND
    month = 11 AND
    agent_type = "user"
GROUP BY
    project,
    page_title
ORDER BY views DESC
LIMIT 200
;

https://en.wikipedia.org/wiki/Main_Page 509040015
https://en.wikipedia.org/wiki/Special:Search    61718561
https://de.wikipedia.org/wiki/Wikipedia:Hauptseite  49138976
https://en.wikipedia.org/wiki/- 41422306
https://ru.wikipedia.org/wiki/Заглавная_страница    24892411
https://it.wikipedia.org/wiki/Pagina_principale 22738884
https://fr.wikipedia.org/wiki/Wikipédia:Accueil_principal   22624751
https://wikidata.org/wiki/Special:ItemDisambiguation    20516058
https://es.wikipedia.org/wiki/Wikipedia:Portada 16368812
https://es.wikipedia.org/wiki/Especial:Buscar   13291957
https://ja.wikipedia.org/wiki/メインページ    12829089
https://de.wikipedia.org/wiki/Spezial:Suche 10701479
https://en.wikipedia.org/wiki/XHamster  9539132
https://en.wikipedia.org/wiki/Meghan_Markle 8325659
https://commons.wikimedia.org/wiki/Special:CreateAccount    7098274
https://fr.wikipedia.org/wiki/Spécial:Recherche 6232043
https://en.wikipedia.org/wiki/Special:CreateAccount 6074120
https://fr.wikipedia.org/wiki/Sp?cial:Search    5767256
https://es.wikipedia.org/wiki/- 5706664
https://zh.wikipedia.org/wiki/Wikipedia:首页  5689280
https://en.wikipedia.org/wiki/Justice_League_(film) 5501717
https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Asian_Month   5432802
https://nl.wikipedia.org/wiki/Hoofdpagina   5346360
https://en.wikipedia.org/wiki/Stranger_Things   5028333
https://pt.wikipedia.org/wiki/Wikipédia:Página_principal    4776336
https://ru.wikipedia.org/wiki/Служебная:Поиск   4727586
https://en.wikipedia.org/wiki/Charles_Manson    4534081
https://en.wikipedia.org/wiki/File:ITVBe_logo_2014-.svg 4031082
https://pt.wikipedia.org/wiki/Especial:Pesquisar    3933154
https://ar.wikipedia.org/wiki/خاص:بحث   3805110
https://pl.wikipedia.org/wiki/Wikipedia:Strona_główna   3780977
https://en.wikipedia.org/wiki/Thor:_Ragnarok    3753408
https://commons.wikimedia.org/wiki/Special:Search   3662952
https://it.wikipedia.org/wiki/File:Logo_Rai_1_2016.png  3522269
https://ja.wikipedia.org/wiki/特別:検索 3323110
https://en.wikipedia.org/wiki/Deaths_in_2017    3278163
https://kk.wikipedia.org/wiki/Template:AdvancedSiteNotices/ajax 3204185
https://commons.wikimedia.org/wiki/Main_Page    3173168
https://en.wikipedia.org/wiki/Lil_Peep  2928824
https://cs.wikipedia.org/wiki/Hlavní_strana 2865838
https://wikimediafoundation.org/wiki/Terms_of_Use   2857589
https://de.wikipedia.org/wiki/Special:MyPage/toolserverhelferleinconfig.js  2829252
https://en.wikipedia.org/wiki/David_Cassidy 2658645
https://de.wikipedia.org/wiki/- 2626317
https://zh.wikipedia.org/wiki/Special:搜索    2584862
https://en.wikipedia.org/wiki/Darth_Vader   2501870
https://fr.wikipedia.org/wiki/- 2442000
https://en.wikipedia.org/wiki/Millie_Bobby_Brown    2418394
https://en.wikipedia.org/wiki/Special:ElectronPdf   2415787
https://ar.wikipedia.org/wiki/الصفحة_الرئيسية   2392445
https://fi.wikipedia.org/wiki/Wikipedia:Etusivu 2389545
https://en.wikipedia.org/wiki/Krauss-Maffei_Wegmann 2298839
https://it.wikipedia.org/wiki/Speciale:Ricerca  2282656
https://en.wikipedia.org/wiki/Kevin_Spacey  2282320
https://sv.wikipedia.org/wiki/Portal:Huvudsida  2255130
https://en.wikipedia.org/wiki/XXX_(franchise)   2146256
https://en.wikipedia.org/wiki/Special:Watchlist 2128132
https://en.wikipedia.org/wiki/Bitcoin   2123997
https://en.wikipedia.org/wiki/Jason_Momoa   2116244
https://en.wikipedia.org/wiki/Manushi_Chhillar  2032722
https://ru.wikipedia.org/wiki/- 2004623
https://en.wikipedia.org/wiki/Survivor_Series_(2017)    1977502
https://ar.wikipedia.org/wiki/- 1909394
https://en.wikipedia.org/wiki/Prince_Harry  1899108
https://ru.wikipedia.org/wiki/Задорнов,_Михаил_Николаевич   1877089
https://en.wikipedia.org/wiki/Special:RecentChangesLinked   1787963
https://pl.wikipedia.org/wiki/Specjalna:Szukaj  1752979
https://en.wikipedia.org/wiki/Matt_Lauer    1704694
https://ru.wikipedia.org/wiki/Троцкий,_Лев_Давидович    1682487
https://en.wikipedia.org/wiki/Special:Book  1680602
https://en.wikipedia.org/wiki/The_Punisher_(TV_series)  1662210
https://en.wiktionary.org/wiki/Category:English_adjectives  1597964
https://en.wikipedia.org/wiki/Thanksgiving  1582239
https://en.wikipedia.org/wiki/Robert_Mugabe 1576851
https://de.wikipedia.org/wiki/Special:Search    1562212
https://zh.wikipedia.org/wiki/Wiki  1529575
https://nl.wikipedia.org/wiki/Speciaal:Zoeken   1518165
https://ja.wikipedia.org/wiki/鶴ひろみ  1515078
https://pt.wikipedia.org/wiki/- 1506364
https://en.wikipedia.org/wiki/Special:MobileMenu    1493252
https://en.wikipedia.org/wiki/Special:CiteThisPage  1487776
https://en.wikipedia.org/wiki/Donald_Trump  1484905
https://en.wikipedia.org/wiki/Roy_Halladay  1470667
https://cs.wikipedia.org/wiki/Speciální:Hledání 1457294
https://en.wikipedia.org/wiki/Murder_on_the_Orient_Express_(2017_film)  1407840
https://zh.wikipedia.org/wiki/Wikipedia:分类索引    1391512
https://en.wikipedia.org/wiki/Winona_Ryder  1383533
https://zh.wikipedia.org/wiki/Wikipedia:关于  1374186
https://zh.wikipedia.org/wiki/Wikipedia:分類索引    1372641
https://en.wikipedia.org/wiki/Roy_Moore 1362577
https://zh.wikipedia.org/wiki/Category:交通   1360586
https://zh.wikipedia.org/wiki/Category:交通组织 1359496
https://zh.wikipedia.org/wiki/Category:鐵路公司 1358931
https://zh.wikipedia.org/wiki/Category:路面電車公司   1358511
https://zh.wikipedia.org/wiki/維基媒體基金會   1358323
https://zh.wikipedia.org/wiki/Category:東京都交通局   1358110
https://zh.wikipedia.org/wiki/Category:東京都交通局路線模板   1357764
https://zh.wikipedia.org/wiki/Template:荒川線  1357479
https://es.wikipedia.org/wiki/Wikipedia:Mes_de_Asia 1351217
https://en.wikipedia.org/wiki/Gaten_Matarazzo   1344798
https://ja.wikipedia.org/wiki/- 1331552
https://en.wikipedia.org/wiki/Al-Batin  1325264
https://ru.wikipedia.org/wiki/Служебная:Ссылки_сюда 1319381
https://en.wikipedia.org/wiki/Elizabeth_II  1318184
https://en.wikipedia.org/wiki/Portal:Current_events 1308209
https://en.wikipedia.org/wiki/2018_FIFA_World_Cup   1295203
https://hu.wikipedia.org/wiki/Kezdőlap  1287702
https://tl.wikipedia.org/wiki/Natatangi:Maghanap    1276708
https://en.wikipedia.org/wiki/Natalia_Dyer  1262260
https://en.wikipedia.org/wiki/List_of_Bollywood_films_of_2017   1261276
https://zh.wikipedia.org/wiki/- 1261270
https://en.wikipedia.org/wiki/Black_Friday_(shopping)   1254166
https://en.wikipedia.org/wiki/United_States 1251293
https://en.wikipedia.org/wiki/Muscat_and_Oman   1229997
https://en.wikipedia.org/wiki/Mindhunter_(TV_series)    1229010
https://en.wikipedia.org/wiki/Jebel_Akhdar_War  1226283
https://en.wikipedia.org/wiki/Rani_Padmini  1223122
https://en.wikipedia.org/wiki/Avengers:_Infinity_War    1204309
https://ko.wikipedia.org/wiki/위키백과:대문   1203329
https://en.wikipedia.org/wiki/Elon_Musk 1200744
https://en.wiktionary.org/wiki/Wiktionary:Main_Page 1189114
https://en.wiktionary.org/wiki/Special:Search   1184840
https://en.wikipedia.org/wiki/Lyle_and_Erik_Menendez    1179651
https://en.wikipedia.org/wiki/Lisa_Bonet    1178981
https://en.wikipedia.org/wiki/XXX   1176431
https://en.wikipedia.org/wiki/Riverdale_(2017_TV_series)    1174326
https://ru.wikipedia.org/wiki/Хворостовский,_Дмитрий_Александрович  1172816
https://de.wikipedia.org/wiki/Hauptseite    1161961
https://en.wikipedia.org/wiki/XVideos   1160624
https://en.wikipedia.org/wiki/Malcolm_Young 1159670
https://en.wikipedia.org/wiki/Gal_Gadot 1152152
https://en.wikipedia.org/wiki/UFC_217   1151244
https://en.wikipedia.org/wiki/2018_FIFA_World_Cup_qualification 1145138
https://es.wikipedia.org/wiki/Stranger_Things   1144703
https://it.wikipedia.org/wiki/- 1126068
https://ko.wikipedia.org/wiki/특수:검색 1124864
https://en.wikipedia.org/wiki/List_of_Marvel_Cinematic_Universe_films   1109782
https://en.wikipedia.org/wiki/Special:WhatLinksHere 1094428
https://id.wikipedia.org/wiki/Istimewa:Pencarian    1082813
https://en.wikipedia.org/wiki/Lil_Pump  1076797
https://en.wikipedia.org/wiki/Padmavati_(film)  1071896
https://pt.wikipedia.org/wiki/Especial:Entrar   1045648
https://fa.wikipedia.org/wiki/ویژه:جستجو    1044638
https://en.wikipedia.org/wiki/Sharon_Tate   1037452
https://en.wikipedia.org/wiki/M2_Browning   1035157
https://en.wikipedia.org/wiki/Net_neutrality    1034073
https://en.wikipedia.org/wiki/M249_light_machine_gun    1024542
https://de.wikipedia.org/wiki/Spezial:Weblinksuche  1024448
https://en.wikipedia.org/wiki/Coco_(2017_film)  1022661
https://id.wikipedia.org/wiki/Candy_Crush_Saga  1019644
https://da.wikipedia.org/wiki/Forside   1017163
https://en.wikipedia.org/wiki/Finn_Wolfhard 1011023
https://en.wikipedia.org/wiki/Joe_Keery 1007206
https://en.wikipedia.org/wiki/Louis_C.K.    1005549
https://wikimediafoundation.org/wiki/Thank_You/en   993197
https://en.wikipedia.org/wiki/Reputation_(Taylor_Swift_album)   990984
https://bn.wikipedia.org/wiki/বিশেষ:অনুসন্ধান   990352
https://en.wikipedia.org/wiki/Jeffrey_Dahmer    969240
https://de.wikipedia.org/wiki/Spezial:Beobachtungsliste 969003
https://en.wikipedia.org/wiki/Miss_Universe_2017    960789
https://en.wikipedia.org/wiki/2017_in_film  955210
https://en.wikipedia.org/wiki/Millennials   953278
https://en.wikipedia.org/wiki/Miss_World_2017   952883
https://pt.wikipedia.org/wiki/Jogo_Duro_(reality_show)  951402
https://en.wikipedia.org/wiki/Special:MobileOptions 937199
https://en.wikipedia.org/wiki/Al_Franken    934756
https://en.wikipedia.org/wiki/Queen_Victoria    933050
https://en.wikipedia.org/wiki/Zimbabwe  927059
https://en.wikipedia.org/wiki/Ric_Flair 925136
https://en.wikipedia.org/wiki/Pablo_Escobar 923023
https://en.wikipedia.org/wiki/Corey_Haim    922328
https://en.wikipedia.org/wiki/India 921465
https://en.wikipedia.org/wiki/Andy_Kaufman  921014
https://en.wikipedia.org/wiki/World_War_II  920056
https://en.wikipedia.org/wiki/AMGTV 919873
https://en.wikipedia.org/wiki/Diana_Ross    913809
https://en.wikipedia.org/wiki/Ed_Sheeran    906752
https://en.wikipedia.org/wiki/Eminem    900335
https://en.wikipedia.org/wiki/Charlie_Heaton    899977
https://en.wikipedia.org/wiki/Lovebug   899400
https://wikimediafoundation.org/wiki/Privacy_policy 894473
https://en.wikipedia.org/wiki/Taylor_Swift  891298
https://en.wikipedia.org/wiki/Edmund_Kemper 878741
https://en.wikipedia.org/wiki/Murder_on_the_Orient_Express  877701
https://en.wikipedia.org/wiki/Cristiano_Ronaldo 869189
https://it.wikipedia.org/wiki/Stranger_Things   864606
https://id.wikipedia.org/wiki/Halaman_Utama 864069
https://en.wikipedia.org/wiki/Tate_murders  863414
https://en.wikipedia.org/wiki/Post_Malone   862213
https://en.wikipedia.org/wiki/Tommy_Wiseau  854853
https://en.wikipedia.org/wiki/Paradise_Papers   853190
https://en.wikipedia.org/wiki/Charlie_Rose  852492
https://en.wikipedia.org/wiki/Jigsaw_(Marvel_Comics)    843562
https://en.wikipedia.org/wiki/American_Horror_Story:_Cult   842887
https://fr.wikipedia.org/wiki/Spécial:ElectronPdf   840848
https://en.wikipedia.org/wiki/World_War_I   840172
https://en.wikipedia.org/wiki/Help:IPA/English  835976
https://es.wikipedia.org/wiki/Charles_Manson    834375
https://commons.wikimedia.org/wiki/Commons:Wiki_Loves_Africa_2017/en    833307
https://mediawiki.org/wiki/MediaWiki    833187
Time taken: 598.572 seconds, Fetched: 200 row(s)

There's no pages with � in the top 200 for November 2017.

Distribution among wikis

SELECT
    project,
    SUM(view_count) AS views
FROM wmf.pageview_hourly
WHERE
    year = 2017 AND
    month = 11 AND
    page_title LIKE '%�%'
GROUP BY project
ORDER BY views DESC
LIMIT 50
;

ru.wikipedia    3212784
zh.wikipedia    1576571
fr.wikipedia    666843
en.wikipedia    462291
de.wikipedia    181756
ru.wiktionary   156837
es.wikipedia    147440
sv.wikipedia    125222
ca.wikipedia    116221
cs.wikipedia    96272
hu.wikipedia    88226
pt.wikipedia    76488
fi.wikipedia    63053
sk.wikipedia    55489
da.wikipedia    54648
nl.wikipedia    45994
uk.wikipedia    41981
ms.wikipedia    37956
it.wikipedia    37720
ro.wikipedia    37437
fr.wiktionary   33495
vi.wikipedia    33084
de.wiktionary   31940
et.wikipedia    29779
pl.wikipedia    28118
es.wiktionary   26595
no.wikipedia    23845
tr.wikipedia    23409
en.wiktionary   20044
id.wikipedia    16825
el.wikipedia    13236
ru.wikisource   10193
nl.wiktionary   8823
sl.wikipedia    8742
ja.wiktionary   8640
bg.wikipedia    7394
sv.wiktionary   6382
hr.wikipedia    5598
ar.wikipedia    5534
pt.wiktionary   4600
tl.wikipedia    4524
commons.wikimedia   4481
he.wikipedia    4455
ja.wikipedia    4052
lt.wikipedia    3982
fr.wikisource   3843
ru.wikiquote    3149
de.wikibooks    2547
ca.wiktionary   2493
tr.wikisource   2211
Time taken: 175.995 seconds, Fetched: 50 row(s)

The distribution is similar to 2015, but the values are a lot lower.
In 2015-05 ru.wikipedia.org had 177+ million views with �, whereas in 2017-11 there were only 3 million for the same wiki.

@Tbayer @Nuria

I'd say it will be difficult to troubleshoot this issue in the past without being able to reproduce the error.
I would rather add this task as a subtask of T159840, and when we work on it, we create an alarm that can ping us in case this problem comes back.

mforns moved this task from In Progress to In Code Review on the Analytics-Kanban board.Dec 22 2017, 2:29 PM

Agreed @mforns

mforns added a parent task: T159840: Alarm on data quality issues .Jan 3 2018, 11:29 AM

This has become a subtask of T159840.
I will change the title from "Investigate anomalous views to pages with replacement characters" to "Add alerts for high volume of views to pages with replacement characters".

mforns renamed this task from Investigate anomalous views to pages with replacement characters to Add alarms for high volume of views to pages with replacement characters.Jan 3 2018, 11:36 AM

mforns updated the task description. (Show Details)

mforns edited projects, added Analytics; removed Analytics-Kanban, Spanish-Sites, Tamil-Sites, Hindi-Sites, Bengali-Sites, Chinese-Sites.

zhuyifei1999 subscribed.Jan 3 2018, 12:26 PM

• fdans moved this task from Backlog (Later) to Dashiki on the Analytics board.Jan 8 2018, 5:04 PM

Milimetric moved this task from Dashiki to Incoming on the Analytics board.Apr 2 2018, 3:32 PM

• Nuria moved this task from Incoming to Data Quality on the Analytics board.Apr 5 2018, 5:02 PM

mforns lowered the priority of this task from High to Medium.Apr 23 2018, 4:19 PM

• Tbayer mentioned this in T198218: Generate list of most used special pages.Jun 28 2018, 7:52 PM

Restricted Application added a subscriber: Urbanecm. · View Herald TranscriptNov 20 2018, 1:23 AM

I received another report that this is happening to he.wikipedia, see https://wikimedia.org/api/rest_v1/metrics/pageviews/top/he.wikipedia/all-access/2015/07/all-days

IKhitron subscribed.Nov 25 2018, 9:57 PM

AfroThundr3007730 subscribed.Dec 24 2018, 8:44 PM

mforns lowered the priority of this task from Medium to Low.Mar 11 2019, 3:40 PM

mforns removed mforns as the assignee of this task.May 20 2020, 6:52 PM

Restricted Application added a subscriber: Strainu. · View Herald TranscriptMay 20 2020, 6:52 PM

Strainu unsubscribed.May 21 2020, 5:55 AM

I believe Language-Team was tagged to provide input, which was given, so I am removing the tag. I also noticed that there was a comment saying that this task is resolved. If anyone can confirm that would be nice.

@Nikerabbit I think we still could work on this in the future.
However, as we have to prioritize it against other tasks and projects, we are not going to work on this soon.
And no problem removing Language-Team, we can ping you in the future if we need more help, thanks!

Milimetric raised the priority of this task from Low to Medium.Jul 20 2021, 6:51 PM

MusikAnimal unsubscribed.Jul 20 2021, 7:25 PM

odimitrijevic edited projects, added Data-Engineering; removed Analytics-Data-Quality.Nov 29 2021, 11:29 PM

odimitrijevic moved this task from Incoming (new tickets) to Datasets on the Data-Engineering board.

odimitrijevic removed a project: Analytics.Jan 12 2022, 12:20 AM

JArguello-WMF moved this task from Datasets to Data Products & Metrics on the Data-Engineering board.Jun 29 2023, 11:46 PM

lbowmaker moved this task from Data Products & Metrics to Icebox (not considered in current quarter) on the Data-Engineering board.Nov 10 2023, 2:25 PM

	F2921047: Views to pages with replacement characters, April-November 2015.png
	Nov 6 2015, 5:27 AM

	F2921050: Views to pages with replacement characters, April-November 2015 (log scale).png
	Nov 6 2015, 5:27 AM

	F11992325: chart (1).png
	Dec 22 2017, 2:21 PM

Add alarms for high volume of views to pages with replacement charactersOpen, MediumPublicActions