| jijiki | |
| Thu, Feb 5, 10:46 AM |
| F71689980: image.png | |
| Thu, Feb 5, 11:57 AM |
| F71688735: image.png | |
| Thu, Feb 5, 10:46 AM |
| F71688626: image.png | |
| Thu, Feb 5, 10:46 AM |
| F71688588: image.png | |
| Thu, Feb 5, 10:46 AM |
| F71688386: image.png | |
| Thu, Feb 5, 10:46 AM |
I have observed for quite sometime now, that almost every UTC midnight we have an increase in DB, memcached traffic, and busy php workers.
*potential reasons*
Here is the latest parsoid rollouts. From git log:
36e03fe94%an Thu Jan 29 23:19:32 2026 +0100 Prepare pplwiki fbae5b133%an Fri Jan 23 12:17:25 2026 -0500 Deploy PRV to 21 wikis + bump 3 top50 to 100% 183631378%an Mon Jan 26 16:13:49 2026 +0100 Prepare configuration for kajwiki 066dcf67e%an Mon Jan 5 19:23:04 2026 -0500 Deploy PRV to 27 wikis 0a095b7c5%an Thu Dec 18 08:33:16 2025 -0500 Turn on Parsoid Read Views on itwiki 1e6f68a2d%an Thu Dec 18 08:32:53 2025 -0500 Turn on Parsoid Read Views on nlwiki cad94ef8c%an Fri Nov 28 15:24:47 2025 -0500 Deploy Parsoid Read Views to 19 wikis c0f1be09f%an Sun Nov 16 17:38:22 2025 +0200 Initial configuration for tokwiki 6003495c5%an Wed Nov 19 16:16:24 2025 -0500 Deploy Parsoid Read Views to 18 wikis f229d315c%an Fri Nov 7 16:22:07 2025 -0500 Deploy Parsoid Read Views to 13 wikis 92839b8cf%an Wed Oct 29 20:05:24 2025 -0400 Deploy Parsoid Read Views to 7 wikis 4a51792ce%an Tue Oct 28 01:12:33 2025 +0100 Initial configuration for pcmwikiqoute 5e71f3c05%an Thu Oct 2 14:49:15 2025 -0400 Deploy Parsoid Read Views to 26 Wikipedias 6d3707a43%an Thu Sep 18 13:52:03 2025 -0400 Deploy Parsoid Read Views to 28 Wikipedias e8966c9b4%an Fri Sep 5 11:59:39 2025 +0200 Initial configuration for arbcom_plwiki 6248c1c54%an Wed Sep 17 00:13:28 2025 +0200 Initial configuration for thwikimedia 6fcb397f8%an Tue Sep 16 23:59:45 2025 +0200 Initial configuration for mswikiquote 2ba819cd1%an Thu Sep 11 15:53:26 2025 -0400 Deploy Parsoid Read Views to 23 Wikipedias d271193d8%an Tue Aug 19 17:06:57 2025 -0400 Deploy Parsoid Read Views to ~20 Wikipedias b2c68e7e4%an Wed Aug 20 22:48:40 2025 +0200 Initial configuration for bewwiktionary 364836718%an Mon Aug 11 22:13:08 2025 +0200 Initial configuration for zghwiktionary 327629d84%an Mon Aug 11 22:10:36 2025 +0200 Initial configuration for minwikibooks b9c8e424e%an Mon Aug 11 22:06:56 2025 +0200 Initial configuration for rkiwiki 47e6ee6c6%an Fri Jul 25 16:11:42 2025 -0400 Deploy Parsoid Read Views to 39 Wikipedias 008c6d854%an Fri Apr 25 10:04:35 2025 -0400 nupwiki: Enable Parsoid mode 5a61108b3%an Fri Apr 25 09:48:37 2025 -0400 Move wgParserMigrationEnableParsoidDiscussionTools and wgParserMigrationEnableParsoidArticlePages to a dblist
@jijiki out of curiosity, have you already looked at/excluded logrotate from the picture? AFAICT it runs fleetwide few seconds after midnight and it could be possible that some service upon getting notified of the log rotation does a less smooth reload?
Is the spike caused by more requests or more work per request?
It could simply be client behavior. If the volume of requests goes up, something is driving more traffic.
If the volume of requests stays relatively flat, but work per request is increasing, that would support a backend issue like cache expiration.
There is a lot of wikitext which refers to the date. By default all that content expires at midnight (UTC or local, depending on the exact wikitext used) so that it can be re-evaluated with a new date string.
A description of the problem and a proposal to address it are in T416616: Create new cache-friendly lua/parser function for "is today before X date" and "is today after X date".
Unfortunately Serviceops doesn't have the capacity to investigate further at this point in time. @cscott will the rootcausing continue in the task that you created, and if so can we dedupe?
Otherwise if anyone wants to pick this up please assign it to you.
Probably we should just resolve this as a dup of T416616, as that is our current working plan to address the issue.
