Now the job for tool persondata vanished in some magical way. I did not touch it.

With wikihistory I seem to have a similar effect:

tools.wikihistory@tools-sgebastion-07:~$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
2935144 0.30307 dewiki_upd tools.wikihi r     12/12/2019 11:48:52 continuous@tools-sgeexec-0941.     1        
3981817 0.25421 dewiki_upd tools.wikihi r     01/03/2020 00:48:27 continuous@tools-sgeexec-0937.     1        
9532659 0.45207 dewiki_upd tools.wikihi Rr    10/23/2019 09:22:39 continuous@tools-sgeexec-0926.     1        
9532660 0.45207 dewiki_upd tools.wikihi r     10/07/2019 19:18:58 continuous@tools-sgeexec-0901.     1        
tools.wikihistory@tools-sgebastion-07:~$ qacct -j 9532659;qacct -j 9532660
error: job id 9532659 not found
error: job id 9532660 not found

No data from qacct?

In the logfile of that job, I see:
2019-12-06 00:15:24 Start process_templatedata.php

Another case:

The article includes template https://de.wikipedia.org/wiki/Vorlage:Metadaten_Einwohnerzahl_AT_Ortschaft multiple (81) times. I wonder, if some kind of cache can be implemented, so parsing of a template is done only once.

My comment from March 22nd is still valid. s51412__data is done.

okay, the empty data seems to be solvable, I got the following response from the API:
(showing just the relevant part)

{"pageid":9577707,"ns":0,"title":"Chambon (Charente-Maritime)","revisions":[{"revid":192865788,"parentid":192865756,"timestamp":"2019-10-05T07:14:27Z","sha1":"1e29e6f2c752a9e3dd172b2d2acfe1844eb58fa4","slotsmissing":true}]}

The full request and the full answer can be found in file /data/project/persondata/data/2019-10-05 07:14:29.txt

Seems to be my own fault. ==> closed

I am running a home brewn watchdog, see /data/project/persondata/diverses/watchdog.php with the logs dewiki_watchdog.out but there is nothing found there. However, this jobs runs at minute 28 and 58, which does not match the modification time which was in minute 12.

It is even worse!

sysbench is that tool for measuring high load on mysql? I bought my box in May 2009 and sysbench (version 1.0.11) says "total time: 10.0012s" and "execution time (avg/stddev): 9.9984" (2 Quad-Core AMD Opteron(tm) Processor 2382 at 2613 MHz, 5230.44 bogomips)

I do not understand why this one is merged?

It works as expected in deWP. thanks

It is strange, really strange. I have seen that slowness three times within a few minutes on my watchlist, but now I do not see it anymore? Did the servers take a break for a cup of tea?

Sorry to reopen that issue, but the behaviour is back :-(

@ema It was not a single timeout. A chunk of data (the start of the page) was rendered by the browser very fast, but it stuck. And from that moment the browser added line by line, sometimes character by character, sometimes larger chunks arrived. When I scrolled down to the end of the page, I could read the data faster than it arrived. As I said in the IRC channel: Slow like a acoustic coupler.

Agree to @Gestumblindi !

time curl -I 'https://de.wikipedia.org/wiki/Spezial:Leerseite' --resolve 'de.wikipedia.org:443:208.80.154.224'
HTTP/2 200
date: Wed, 19 Jun 2019 19:37:08 GMT
content-type: text/html; charset=UTF-8
server: mw1319.eqiad.wmnet
x-content-type-options: nosniff
p3p: CP="This is not a P3P policy! See https://de.wikipedia.org/wiki/Spezial:CentralAutoLogin/P3P for more info."
x-powered-by: HHVM/3.18.6-dev
content-language: de
expires: Thu, 01 Jan 1970 00:00:00 GMT
x-frame-options: DENY
backend-timing: D=74475 t=1560973028653859
vary: Accept-Encoding,Cookie,Authorization,X-Seven
content-encoding: gzip
x-varnish: 412364975, 230972406
via: 1.1 varnish (Varnish/5.1), 1.1 varnish (Varnish/5.1)
age: 0
x-cache: cp1089 pass, cp1079 pass
x-cache-status: pass
server-timing: cache;desc="pass"
strict-transport-security: max-age=106384710; includeSubDomains; preload
set-cookie: WMF-Last-Access=19-Jun-2019;Path=/;HttpOnly;secure;Expires=Sun, 21 Jul 2019 12:00:00 GMT
set-cookie: WMF-Last-Access-Global=19-Jun-2019;Path=/;Domain=.wikipedia.org;HttpOnly;secure;Expires=Sun, 21 Jul 2019 12:00:00 GMT
x-analytics: ns=-1;special=Blankpage;https=1;nocookies=1
x-client-ip: 178.202.72.113
cache-control: private, s-maxage=0, max-age=0, must-revalidate
set-cookie: GeoIP=DE:NW:M__nchengladbach:51.20:6.44:v4; Path=/; secure; Domain=.wikipedia.org
real 0m0,574s
user 0m0,048s
sys 0m0,001s

Last test: I added in my /etc/hosts file the following line:
208.80.154.224 de.wikipedia.org

Okay I did some more tests:

I did the following:
$ ping -c 100 -i 20 -s 1000 de.wikipedia.org
PING dyna.wikimedia.org (91.198.174.192) 1000(1028) bytes of data.
1008 bytes from text-lb.esams.wikimedia.org (91.198.174.192): icmp_seq=1 ttl=55 time=18.4 ms
1008 bytes from text-lb.esams.wikimedia.org (91.198.174.192): icmp_seq=2 ttl=55 time=16.1 ms
...
1008 bytes from text-lb.esams.wikimedia.org (91.198.174.192): icmp_seq=99 ttl=55 time=25.5 ms
1008 bytes from text-lb.esams.wikimedia.org (91.198.174.192): icmp_seq=100 ttl=55 time=17.9 ms

Krinkle, I refresh my watchlist maybe 100 times a day. It is usually fast, I cannot see the rebuild *blink* refreshed. But it started a few weeks ago
From IRC #wikimedia-cloud.log (local times, UTC+2:00)
Jun 07 00:24:57 <Wurgl> Really strange: Very often when I open a wiki-page it opens within a blink of an eye. But sometimes I can watch the bytes while they arrive. Slow like a acoustic coupler … (I have a 150 Mbit connection)

A really bad quality video of refreshing my watchlist. You see the watchlist and at 0:17 - 0:36 you see a slow refresh.
https://www.youtube.com/watch?v=W7r6DvmnfnE

Okay @Lea_WMDE, I thought/hoped it was deployed already on last Thursday. I tried on Beta and cannot reproduce any problems. So eagerly waiting for Thursday … If you like you can close it, I will not reopen any more.

Sorry to reopen this one, maybe there is some fix, but not on de-wiki and not for me.

Could this difference be caused by prevention of those design flaws called Spectre & Meltdown?

the last days:
2019-06-05 connected to labsdb1010, 1 large query "SELECT DISTINCT pl_title FROM pagelinks LEFT JOIN page ON page_title = pl_title AND page_namespace = pl_namespace WHERE pl_from_namespace = 0 AND pl_namespace = 0 AND page_id IS NULL" took ~55 Minutes which is okay.

Maybe important: The URL gets truncated to https://en.wikipedia.org/w/index.php once you select "Remember selection for future searches" which is accessible when you expand/open "Search in:". Actually not for the current search, but for all following. And this field is accessible only when you are logged in. See T224796

I will watch it for a few days and close this one when nothing unexpected happens.

Some background: There are a lot of "redlinks" in articles. Like [[Ac Aceca]] where an article [[AC Aceca]] exists, this is an example of uppercase/lowercase difference, the same is true for letters with accents and similar additions, like a redlink [[Diana Castano Sarrias]] with an existing [[Diana Castaño Sarrias]].

Explain when the slow query runs:

MariaDB [dewiki_p]> show explain FOR 248311371;
+------+-------------+-----------+--------+-------------------------------------+--------------+---------+---------------------------------+----------+-----------------------------------------------------+

id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra

+------+-------------+-----------+--------+-------------------------------------+--------------+---------+---------------------------------+----------+-----------------------------------------------------+

1	SIMPLE	pagelinks	ref	pl_namespace,pl_backlinks_namespace	pl_namespace	4	const	11090664	Using index condition; Using where; Using temporary
1	SIMPLE	page	eq_ref	name_title	name_title	261	const,dewiki.pagelinks.pl_title	1	Using where; Using index; Not exists; Distinct

+------+-------------+-----------+--------+-------------------------------------+--------------+---------+---------------------------------+----------+-----------------------------------------------------+
2 rows in set, 1 warning (0.01 sec)

Today (23 days later) there are 128 hits in dewiki_p (6 additional wrong pairs), 78 in frwiki_p (2 fewer), still none in eswiki_p, still 25 in itwiki_p.

This is already implemented. Closing Task

From my side (s51412__data) it is done. Thanks.

From the few words there, it sound like the same problem.

I was not aware of this one. Worked fine on trusty :-(

tools.persondata@tools-sgebastion-07:~$ webservice --backend=kubernetes start
Looks like you already have another webservice running, with a gridengine backend
You should stop that webservice by issuing:

webservice --backend=gridengine stop

And then start it again with backend kubernetes by issuing:

webservice --backend=kubernetes start

Hmm … seems to be my error.

Sorry, my mistake.

Well, I started all my jobs (for historical reasons) from crontab with -once. I have rewritten some job which does its work, sleeps some minutes and then runs again. So I do not have to start it over and over again.

I accessed it from tools-bastion-02 with some script. Could be reproduces with command line

After a few minutes the problem vanished.

Maybe a second account in the database would be a simpler solution to double the number of connections? (just my few cents)

I'm the happiest guy in the world!

A dump would be fine. I do not need all, some a really large!

It is already set.

Would be fine when the data of s51412__data could be saved/restored onto the new server too.

Generally a user limit is okay.

I think there is no need to run such a query by cron or similar.

Yes! I have even started some virtual machine which I did not use for month and there the errors show up too, I tried en.wikipedia.org and fr.wikipedia.org (for sure I never visitied fr.wikipedia.org, since I do not understand that language).

On my local machine here at home I have installed a version from december 2017, when I grep in the source-hell I find …

Just an example: https://de.wikipedia.org/wiki/Internet_Movie_Database or en-wp: https://en.wikipedia.org/wiki/Robert_W._Chambers

A different example: ref-tags are enclosed in nowiki-tags
https://de.wikipedia.org/w/index.php?title=Maschinenkanone&diff=prev&oldid=174409703

@MusikAnimal … If you redirect to https://tools.wmflabs.org/wikihistory?wiki=enwiki I will try to enhance that tool.

enwiki works the same way as dewiki does. There is just an additional parameter "wiki=", example: https://tools.wmflabs.org/wikihistory/wh.php?wiki=enwiki&page_title=Crooked+House Okay, I did not change the frontend https://tools.wmflabs.org/wikihistory/ this is still the same as before and works for the german WP only.

I actually changed it a little bit. I have 10 queues. When I started to maintain this tool, somewhen in September/October it had 4 queues, but at that time it served only de-wiki (and two very small ones: ndl and als). Now I serve en-wiki too. 9 queues are for different sizes of the page. The change I did already is polling fewer times. The processes for the large files poll now every 15 seconds since the computation takes a long time, so it does not matter if the user has to wait 5 minutes or 5 minutes and 15 seconds. The queue for the small pages now polls every 2 seconds (instead of 1 second). 1 second was not invented by me, that was the value when I started to maintain the tool.

I can reduce the number ob jobs to ~4 if that is enough.

In the webserver-logfile of the tool wikihistory I see 2208 entries similar to the following

2017-11-07 11:42:14: (mod_fastcgi.c.2673) FastCGI-stderr: PHP Warning:  mysql_connect(): Too many connections in /mnt/nfs/labstore-secondary-tools-project/wikihistory/db.inc.php on line 7
2017-11-07 14:32:16: (mod_fastcgi.c.2673) FastCGI-stderr: PHP Warning:  mysql_connect(): Too many connections in /mnt/nfs/labstore-secondary-tools-project/wikihistory/db.inc.php on line 7

Same here: