Page MenuHomePhabricator
Create Task
Maniphest T356561

Wikidata query service updater script seems to not close connections to wikibase after latest update
Closed, ResolvedPublicSecurity
Actions

Description

We recently updated the wdqs-updater following https://github.com/wmde/wikibase-release-pipeline/commit/21088b5e6030e3e3fd968a2dbc1ea71abf2c93bf
After that update, the updater script of the wikidata query service does not close connections to our wikibase instance. After a few minutes over 100 open connections can be seen on the wikibase host

apache2ctl status
Apache Server Status for localhost (via 127.0.0.1)

Server Version: Apache/2.4.57 (Debian) PHP/8.1.27
Server MPM: prefork
Server Built: 2023-04-13T03:26:51

-------------------------------------------------------------------------------

Current Time: Saturday, 03-Feb-2024 14:33:23 CET
Restart Time: Saturday, 03-Feb-2024 14:28:53 CET
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 4 minutes 29 seconds
Server load: 16.25 17.23 17.00
Total accesses: 920 - Total Traffic: 2.9 MB - Total Duration: 313546
CPU Usage: u44.45 s13.12 cu16.76 cs2.97 - 28.7% CPU load
3.42 requests/sec - 10.9 kB/second - 3257 B/request - 340.811 ms/request
104 requests currently being processed, 6 idle workers

WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWW_WWWWWW_WWWWWWWWWW____W..................
......................

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

Stopping the query service resolves the problem, and restarting it creates a lot of connections again.

Details

Risk Rating
Low
Author Affiliation
Wikimedia Communities

Related Objects

Event Timeline

Physikerwelt created this task.Feb 3 2024, 10:30 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 3 2024, 10:30 AM
Physikerwelt added a project: Wikibase (3rd party installations).Feb 3 2024, 10:30 AM
Physikerwelt closed this task as Invalid.Feb 3 2024, 10:35 AM

The problem was the '&' which I missed to escape. The reason for the fact the updater creates too many connections must be elsewhere

Physikerwelt added a comment.Feb 3 2024, 10:48 AM

By the way, I could fix the updater problem by adding the -s switch, which means no site links are created.

Physikerwelt renamed this task from Using Special:EntityData in with parameters flavor=dump and nocache creates many open connections to Wikidata query service updater script seems to not close connections to wikibase after latest update.Feb 3 2024, 1:40 PM
Physikerwelt reopened this task as Open.
Physikerwelt triaged this task as Medium priority.
Physikerwelt edited projects, added Wikidata-Query-Service; removed Wikibase (3rd party installations), Security, Security-Team.
Physikerwelt updated the task description. (Show Details)

@Aklapper, could you change the visibility of this ticket to public, please?

Physikerwelt added a comment.Feb 3 2024, 7:38 PM

Increasing the number of workers did not help as well:

pache2ctl status
Apache Server Status for localhost (via 127.0.0.1)

Server Version: Apache/2.4.57 (Debian) PHP/8.1.27
Server MPM: prefork
Server Built: 2023-04-13T03:26:51

-------------------------------------------------------------------------------

Current Time: Saturday, 03-Feb-2024 20:36:45 CET
Restart Time: Saturday, 03-Feb-2024 19:17:10 CET
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 1 hour 19 minutes 35 seconds
Server load: 22.57 23.22 27.67
Total accesses: 10395 - Total Traffic: 24.1 MB - Total Duration: 238091544
CPU Usage: u855.78 s267.58 cu2507.89 cs745.75 - 91.7% CPU load
2.18 requests/sec - 5.2 kB/second - 2430 B/request - 22904.4 ms/request
1117 requests currently being processed, 9 idle workers
Physikerwelt added a comment.Feb 3 2024, 10:02 PM

This seems to be related to https://phabricator.wikimedia.org/T151681 when tracking this further it seems the problem is the database writing to wb_changes

Physikerwelt closed this task as Resolved.Feb 4 2024, 11:28 AM
Physikerwelt claimed this task.

Oh, eventually, it turned out that setting $wgJobRunRate to 0 solved the problem.

 apache2ctl status
Apache Server Status for localhost (via 127.0.0.1)

Server Version: Apache/2.4.57 (Debian) PHP/8.1.27
Server MPM: prefork
Server Built: 2023-04-13T03:26:51

-------------------------------------------------------------------------------

Current Time: Sunday, 04-Feb-2024 12:26:12 CET
Restart Time: Saturday, 03-Feb-2024 21:38:18 CET
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 14 hours 47 minutes 53 seconds
Server load: 33.29 31.67 30.88
Total accesses: 39408 - Total Traffic: 170.7 MB - Total Duration: 55283205849
CPU Usage: u1183.75 s341.35 cu57163.7 cs15870 - 140% CPU load
.74 requests/sec - 3360 B/second - 4542 B/request - 1402840 ms/request
11 requests currently being processed, 6 idle workers

_KKKKK_._KK_KK_KW_..............................................
sbassett changed the visibility from "Custom Policy" to "Public (No Login Required)".Feb 5 2024, 4:25 PM
sbassett changed the edit policy from "Custom Policy" to "All Users".
sbassett changed Risk Rating from N/A to Low.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits