I mentioned https://de.wikipedia.org/wiki/Spezial:Einstellungen because testing the final solution should take a look at that page too.

The Hotfix is just a part of the story. There are still lowercase letters in https://de.wikipedia.org/wiki/Spezial:Einstellungen (or if you prefer english: https://de.wikipedia.org/wiki/Special:Preferences )

This is now 16 years old and still nasty.

The nesting can be seen in enWP too, Example: https://en.wikipedia.org/wiki/J._Golden_Kimball#Service_as_a_Seventy (just one of a lot of examples) However, in enWP I did not find an example where such a nesting is used for the first picture.

Real-Life example: https://de.wikipedia.org/wiki/Altersfreigabe?useskin=modern or an english page: https://en.wikipedia.org/wiki/Mestizo?useskin=monobook

At the very bottom of the window there is a scrollbar (not just below the gallery). Firefox 78.8.0esr openSuse 15.2 or Chromium 89.0.4389.90 (openSUSE Build) (64-Bit)

At least one occurrence of this error can be reproduced with a simple query like "Select page_title, page_title from page where page_id = 1", see T265155 or look at https://quarry.wmflabs.org/query/53652

Strange! Really strange!

Try the query

select * from externallinks where el_to like "//%" limit 10;

and you will see the so called "duplicates". el_to is duplicate, but el_index (and el_index_60) is different.

Yes, looks fine. I do not see that error message now. Thank you!

Happens also on tools-sgeexec-0952

Seems to happen at least on tools-sgeexec-0951 (does not happen on a lot of other machines in the cloud)

To see the effect, just move the mouse over the link to page "Demo Picture" on https://de.wikipedia.beta.wmflabs.org/wiki/Demo_Picture_link

Maybe related: Link to Wikidata-Object is missing too (only in MonoBook, all other skins I tried show that link)

@thiemowmde: I have NOT turned on the "Automatically enable all new beta features", I have NOT turned on any beta feature.

@Ammarpad: I agree that the beta feature may have some flaws.

Now the job for tool persondata vanished in some magical way. I did not touch it.

With wikihistory I seem to have a similar effect:

tools.wikihistory@tools-sgebastion-07:~$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
2935144 0.30307 dewiki_upd tools.wikihi r     12/12/2019 11:48:52 continuous@tools-sgeexec-0941.     1        
3981817 0.25421 dewiki_upd tools.wikihi r     01/03/2020 00:48:27 continuous@tools-sgeexec-0937.     1        
9532659 0.45207 dewiki_upd tools.wikihi Rr    10/23/2019 09:22:39 continuous@tools-sgeexec-0926.     1        
9532660 0.45207 dewiki_upd tools.wikihi r     10/07/2019 19:18:58 continuous@tools-sgeexec-0901.     1        
tools.wikihistory@tools-sgebastion-07:~$ qacct -j 9532659;qacct -j 9532660
error: job id 9532659 not found
error: job id 9532660 not found

No data from qacct?

In the logfile of that job, I see:
2019-12-06 00:15:24 Start process_templatedata.php

Another case:

The article includes template https://de.wikipedia.org/wiki/Vorlage:Metadaten_Einwohnerzahl_AT_Ortschaft multiple (81) times. I wonder, if some kind of cache can be implemented, so parsing of a template is done only once.

My comment from March 22nd is still valid. s51412__data is done.

okay, the empty data seems to be solvable, I got the following response from the API:
(showing just the relevant part)

{"pageid":9577707,"ns":0,"title":"Chambon (Charente-Maritime)","revisions":[{"revid":192865788,"parentid":192865756,"timestamp":"2019-10-05T07:14:27Z","sha1":"1e29e6f2c752a9e3dd172b2d2acfe1844eb58fa4","slotsmissing":true}]}

The full request and the full answer can be found in file /data/project/persondata/data/2019-10-05 07:14:29.txt

Seems to be my own fault. ==> closed

I am running a home brewn watchdog, see /data/project/persondata/diverses/watchdog.php with the logs dewiki_watchdog.out but there is nothing found there. However, this jobs runs at minute 28 and 58, which does not match the modification time which was in minute 12.

It is even worse!

sysbench is that tool for measuring high load on mysql? I bought my box in May 2009 and sysbench (version 1.0.11) says "total time: 10.0012s" and "execution time (avg/stddev): 9.9984" (2 Quad-Core AMD Opteron(tm) Processor 2382 at 2613 MHz, 5230.44 bogomips)

I do not understand why this one is merged?

It works as expected in deWP. thanks

It is strange, really strange. I have seen that slowness three times within a few minutes on my watchlist, but now I do not see it anymore? Did the servers take a break for a cup of tea?

Sorry to reopen that issue, but the behaviour is back :-(

@ema It was not a single timeout. A chunk of data (the start of the page) was rendered by the browser very fast, but it stuck. And from that moment the browser added line by line, sometimes character by character, sometimes larger chunks arrived. When I scrolled down to the end of the page, I could read the data faster than it arrived. As I said in the IRC channel: Slow like a acoustic coupler.

Agree to @Gestumblindi !

time curl -I 'https://de.wikipedia.org/wiki/Spezial:Leerseite' --resolve 'de.wikipedia.org:443:208.80.154.224'
HTTP/2 200
date: Wed, 19 Jun 2019 19:37:08 GMT
content-type: text/html; charset=UTF-8
server: mw1319.eqiad.wmnet
x-content-type-options: nosniff
p3p: CP="This is not a P3P policy! See https://de.wikipedia.org/wiki/Spezial:CentralAutoLogin/P3P for more info."
x-powered-by: HHVM/3.18.6-dev
content-language: de
expires: Thu, 01 Jan 1970 00:00:00 GMT
x-frame-options: DENY
backend-timing: D=74475 t=1560973028653859
vary: Accept-Encoding,Cookie,Authorization,X-Seven
content-encoding: gzip
x-varnish: 412364975, 230972406
via: 1.1 varnish (Varnish/5.1), 1.1 varnish (Varnish/5.1)
age: 0
x-cache: cp1089 pass, cp1079 pass
x-cache-status: pass
server-timing: cache;desc="pass"
strict-transport-security: max-age=106384710; includeSubDomains; preload
set-cookie: WMF-Last-Access=19-Jun-2019;Path=/;HttpOnly;secure;Expires=Sun, 21 Jul 2019 12:00:00 GMT
set-cookie: WMF-Last-Access-Global=19-Jun-2019;Path=/;Domain=.wikipedia.org;HttpOnly;secure;Expires=Sun, 21 Jul 2019 12:00:00 GMT
x-analytics: ns=-1;special=Blankpage;https=1;nocookies=1
x-client-ip: 178.202.72.113
cache-control: private, s-maxage=0, max-age=0, must-revalidate
set-cookie: GeoIP=DE:NW:M__nchengladbach:51.20:6.44:v4; Path=/; secure; Domain=.wikipedia.org

real 0m0,574s
user 0m0,048s
sys 0m0,001s

Last test: I added in my /etc/hosts file the following line:
208.80.154.224 de.wikipedia.org

Okay I did some more tests:

I did the following:
$ ping -c 100 -i 20 -s 1000 de.wikipedia.org
PING dyna.wikimedia.org (91.198.174.192) 1000(1028) bytes of data.
1008 bytes from text-lb.esams.wikimedia.org (91.198.174.192): icmp_seq=1 ttl=55 time=18.4 ms
1008 bytes from text-lb.esams.wikimedia.org (91.198.174.192): icmp_seq=2 ttl=55 time=16.1 ms
...
1008 bytes from text-lb.esams.wikimedia.org (91.198.174.192): icmp_seq=99 ttl=55 time=25.5 ms
1008 bytes from text-lb.esams.wikimedia.org (91.198.174.192): icmp_seq=100 ttl=55 time=17.9 ms

Krinkle, I refresh my watchlist maybe 100 times a day. It is usually fast, I cannot see the rebuild *blink* refreshed. But it started a few weeks ago
From IRC #wikimedia-cloud.log (local times, UTC+2:00)
Jun 07 00:24:57 <Wurgl> Really strange: Very often when I open a wiki-page it opens within a blink of an eye. But sometimes I can watch the bytes while they arrive. Slow like a acoustic coupler … (I have a 150 Mbit connection)

A really bad quality video of refreshing my watchlist. You see the watchlist and at 0:17 - 0:36 you see a slow refresh.
https://www.youtube.com/watch?v=W7r6DvmnfnE

Okay @Lea_WMDE, I thought/hoped it was deployed already on last Thursday. I tried on Beta and cannot reproduce any problems. So eagerly waiting for Thursday … If you like you can close it, I will not reopen any more.

Sorry to reopen this one, maybe there is some fix, but not on de-wiki and not for me.

Could this difference be caused by prevention of those design flaws called Spectre & Meltdown?

the last days:
2019-06-05 connected to labsdb1010, 1 large query "SELECT DISTINCT pl_title FROM pagelinks LEFT JOIN page ON page_title = pl_title AND page_namespace = pl_namespace WHERE pl_from_namespace = 0 AND pl_namespace = 0 AND page_id IS NULL" took ~55 Minutes which is okay.

Maybe important: The URL gets truncated to https://en.wikipedia.org/w/index.php once you select "Remember selection for future searches" which is accessible when you expand/open "Search in:". Actually not for the current search, but for all following. And this field is accessible only when you are logged in. See T224796

I will watch it for a few days and close this one when nothing unexpected happens.

Some background: There are a lot of "redlinks" in articles. Like [[Ac Aceca]] where an article [[AC Aceca]] exists, this is an example of uppercase/lowercase difference, the same is true for letters with accents and similar additions, like a redlink [[Diana Castano Sarrias]] with an existing [[Diana Castaño Sarrias]].

Explain when the slow query runs:

MariaDB [dewiki_p]> show explain FOR 248311371;
+------+-------------+-----------+--------+-------------------------------------+--------------+---------+---------------------------------+----------+-----------------------------------------------------+

id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra

+------+-------------+-----------+--------+-------------------------------------+--------------+---------+---------------------------------+----------+-----------------------------------------------------+

1	SIMPLE	pagelinks	ref	pl_namespace,pl_backlinks_namespace	pl_namespace	4	const	11090664	Using index condition; Using where; Using temporary
1	SIMPLE	page	eq_ref	name_title	name_title	261	const,dewiki.pagelinks.pl_title	1	Using where; Using index; Not exists; Distinct

+------+-------------+-----------+--------+-------------------------------------+--------------+---------+---------------------------------+----------+-----------------------------------------------------+
2 rows in set, 1 warning (0.01 sec)

Today (23 days later) there are 128 hits in dewiki_p (6 additional wrong pairs), 78 in frwiki_p (2 fewer), still none in eswiki_p, still 25 in itwiki_p.

This is already implemented. Closing Task

From my side (s51412__data) it is done. Thanks.

From the few words there, it sound like the same problem.

I was not aware of this one. Worked fine on trusty :-(

tools.persondata@tools-sgebastion-07:~$ webservice --backend=kubernetes start
Looks like you already have another webservice running, with a gridengine backend
You should stop that webservice by issuing:

webservice --backend=gridengine stop

And then start it again with backend kubernetes by issuing:

webservice --backend=kubernetes start

Hmm … seems to be my error.

Sorry, my mistake.

Well, I started all my jobs (for historical reasons) from crontab with -once. I have rewritten some job which does its work, sleeps some minutes and then runs again. So I do not have to start it over and over again.

I accessed it from tools-bastion-02 with some script. Could be reproduces with command line

After a few minutes the problem vanished.

Maybe a second account in the database would be a simpler solution to double the number of connections? (just my few cents)

I'm the happiest guy in the world!

A dump would be fine. I do not need all, some a really large!

It is already set.