The current implementation contains a logic error, in that it tries to split a parameter value on a ',' without first checking for the presence of an '='. This is a bug, regardless of whether we think the correct implementation should or should not allow for {{nobots}} to have parameters.

As of today, there are 3,082 categories in [[:commons:Category:Non-empty category redirects]], which is populated by a template that uses {{PAGESINCAT}} to determine whether a category is empty. The vast majority of these categories (a) have a title that starts with a digit and (b) are actually empty, even though {{PAGESINCAT}} returns a non-zero value for them. (There are a couple of hundred categories whose names don't start with a digit, but most of these appear to be legitimately non-empty.) I can't help but wonder whether this has something to do with the non-quoted category title issue identified by CDanis in the original post ("This should probably instead be WHERE cat_title = '11'.")

In T115231#5834396, @zhuyifei1999 wrote:

Ok lets say in the current situation all SIGTERMs are sent by you.

In T115231#5791761, @russblau wrote:

2019-12-29 01:24:56: (server.c.1751) [note] graceful shutdown started
2019-12-29 01:24:56: (server.c.1828) server stopped by UID = 51290 PID = 11187
2019-12-29 01:25:18: (log.c.164) server started

2020-01-08 00:35:17: (server.c.1558) server stopped by UID = 0 PID = 0
2020-01-08 00:35:22: (log.c.164) server started

What about these?

Honestly I don’t know, except that based on the wiki I expected something different from the proxy’s error page. That is why I suggested the possibility that the problem might be with the documentation.

The SIGTERM is being sent when I run ‘webservice restart’ on one of the login bastions, which I do only _after_ receiving a 503 response.

I could really use some help trying to resolve this issue. The webservice is somewhat more stable now that I am running it with the Kubernetes backend instead of on the grid, but this just means that it fails every few hours instead of every few minutes, on average.

This issue is no longer relevant, since I have been able to start the webservice under kubernetes and no longer rely on the grid servers.

Thanks!

The webservice for dplbot now rarely stays up for more than 10-15 minutes at a time. There are still no error messages whatsoever in the logs giving any clue as to the cause of the outages. Note that, as reported originally, the lighttpd process continues to run during these outages, but all http requests return an error 503. (I have tried using webservice start --backend=kubernetes, but this results in all requests throwing an error 502.) I have also reopened T218915, a related issue.

This issue has resurfaced. /data/project/dplbot/service.log shows that the last attempted restart was at 2019-12-22T06:24:17.778139; it is now Tue Dec 24 15:49:50 and the webservice has been down for over 48 hours. I can restart it manually, but it will inevitably crash again sometime in the near future.

no longer an issue

It's done as far as s51290 is concerned; thanks!

As of this morning, all of the jobs listed in my previous comment are now gone.

As of this morning I was able to start the webservice again.

I have four unkillable jobs stuck on the stretch grid for two different tools:

@jcrespo please restore the table s51290__dpl_p.dab_hof ; thanks in advance!

Also, please try

Thanks. The tables that I'd like to recover from database s51290__dpl_p if possible are as follows:

Although I can probably recreate most of the tables in s51290__dpl_p from scratch, there are a few tables I’d like to recover if they turn out to be recoverable. (I’m mobile now but can come up with a list of tables sometime within the next 24 hours).

All working now.

Sorry for the delay in responding to this. I was able to update the list manually for you.

No one contacted me during this whole process, until after everything had been "resolved". I won't run the job with MEMORY tables again, so you can remove the filters.

To answer your question: We cannot ensure that titles in a user database are unique unless we can set a UNIQUE KEY on the entire column; two titles might be identical in the first 254 characters but differ in the 255th (in the most extreme possible case).

I can't reproduce the issue now, and the bot seems to be logging in as before.

User error :-) please disregard.

I have found an apparent replication issue on one of the new servers:

Is it possible to give s51290 read-only access to labsdb1001 until I have a chance to update the troublesome scripts?

What is the specific query that seems to be causing the problem now?

More detail ... in the commons family file (families/commons_family.py), the values in self.category_redirect_templates aren't being picked up when the Family object's .category_redirects() method is called.

ENGINE=MEMORY changed to ENGINE=InnoDB for all dplbot jobs.

Well, this came as quite a surprise, but I've gone ahead and converted all of dplbot's user databases to use InnoDB. (Hard to believe we were causing all this lag, since the lag was continuing to be a problem even when none of dplbot's jobs were running, but whatever.)

@jcrespo: Is there (or will there be) a way to access and create user databases on the new servers (as per https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Database#User_databases)?

Still not working; same error message as previously reported (ERROR 1045).

Is there any update on the status of the database? The erroneous entries reported back in May are still an issue. It has been nearly six months; is the reimport complete yet? If not, is there an ETA?

It is down again and not restarting. It just went down at approx. 10:20 GMT today (Thursday).

I've had to restart it manually at least once a day for the past several days, although not today (so far).

In T115231#2486162, @yuvipanda wrote:

Is there some custom code that's attempting to autorestart this tool?

It is currently down again. Shell shows the following:

It is down at the moment. "webservice status" says it is running, but "qstat" shows no server process running.

Yes, I had to manually restart it twice today. The automatic webservice restarter is not working.

Reopening; same symptoms are occurring again.

Sorry; it appears that I must have stopped reading before the end of the sentence. :-*>

Is there any update on the status of this? On 23 May, the revision table was in progress and was expected to take ~12 hours. The pagelinks table is about 3X larger and so might be expected to take ~36 hours. But this was eight days ago, so even if the estimates were off by a factor of 2 (or 4), the process could have completed by now.

This has been happening multiple times per month, sometimes more than once in a week. When it happens, it can be fixed simply by re-running the nightly job on the Tool Labs pywikibot account. However, this has to be done manually. It would be much better if the script could be modified to detect failures and start over.

No cron jobs have run for approximately the last six hours. The error messages I've been getting vary quite a bit; the most recent one was:

I'm not entirely sure because I have been mostly offline myself for several days, but I noticed it definitely on Saturday 10 Oct.