Well, that's just incorrect. You can do this:

.mw-content-subtitle > .cdx-message {
  whatever you like;
}

In T191156#10017518, @MBH wrote:

@Ladsgroup Russian Wikipedia community, as one of the biggest communities using FR on all articles, has a consensus that this change should be completely reverted: https://ru.wikipedia.org/wiki/Project:Форум/Технический#Новый_интерфейс_патрулирования . New design is unimaginably horrible and unacceptable, also buggy (dropdown lists don't disappear).

Regarding cross wiki reads and central table. You can use virtual domains now. Here is more details on it: https://www.mediawiki.org/wiki/Manual:$wgVirtualDomainsMapping

It is now ready for DE team.

On it

Overall looks good to go. But name of the mailing list is basically impossible to change in mm3. Can you pick a better name? like wikidata-for-wiki-projects@ wikidata-for-wikis@ wikidata-for-wikimedia@?

I might backport this change on Monday but not today.

My review of the design:

• I need to understand why we are going to have a cache table. Why memcached or something else is not working here?
• "1 query will be run every time a user views a non translatable wikitext page." is a decent chunk, can you cache the value in memcached for shorter period of time on top of the reads? (using page_touched as key to make invalidation work out of the box?)
• This should be in x1, it can be per-wiki. That's fine (we have a database for each wiki in x1). You can use virtual domains to make this work easily.

Not merged on master

I suggest merging the backport but not the patch on master. That'd make things easier and give you enough time to make proper fix.

Creating new tables needs to be signed off by a DBA beforehand: https://wikitech.wikimedia.org/wiki/Creating_new_tables

Before:

root@db2194:/srv/sqldata/loginwiki# ls -Ssh | grep -i cu_changes
193M cu_changes.ibd

Thank you so much!

Wrote the python script. Yup almost all of the updates to page table is to page_touched (964971 updates in 11 minutes). Then I cross checked it with SBR binlog of the same time. Almost all are coming from /* HTMLCacheUpdateJob::invalidateTitles */
Stuff like:

UPDATE /* HTMLCacheUpdateJob::invalidateTitles  */  `page` SET page_touched = '20240723090952' WHERE page_id IN (101998615,101998616,101998618,101998619,101998620,101998621,101998622,101998623,101998624,101998625,101998626,101998627,101998628,101998631,101998632,101998634,101998635,101998636,101998638,101998640,101998641,101998643,101998645,101998647,101998648,101998650,101998652,101998653,101998654,101998655,101998656,101998657,101998658,101998659,101998660,101998661,101998663,101998664,101998665,101998666,101998667,101998668,101998669,101998670,101998672,101998673,101998674,101998675,101998676,101998678,101998679,101998680,101998681,101998682,101998683,101998684,101998685,101998687,101998688,101998689,101998694,101998695,101998696,101998697,101998698,101998699,101998700,101998701,101998702,101998703,101998704,101998705,101998706,101998707,101998708,101998709,101998710,101998711,101998712,101998713,101998714,101998715,101998716,101998717,101998718,101998721,101998722,101998723,101998724,101998726,101998728,101998730,101998733,101998737,101998738,101998739,101998742,101998743,101998747,101998748)  AND (page_touched < '20240723085452')

This should have the map: https://fault-tolerance.toolforge.org/map?cluster=s1

Can be. I think a main driver is that many templates are being edited to make them compatible with dark mode. Combine that with many performance improvements, It's causing issues in the next bottleneck.

I'm repooling the replicas now.

We had a much shorter spike of writes and page for unavailability today. I spent some more time digging.

This has more information on the why: T299371: Migrate globalblocks table to use central ids instead of usernames

Ran it with replication on testwiki, mediawiki.org, officewiki.

After the improvements, I'm monitoring to see if any further issues show up.

In T241053#9996056, @Catrope wrote:

In T370378 we're discussing potentially broadening the scope of gil_to to any pages rather than just files, so maybe we could normalize gil_to to a linktarget instead, and solve both this task and that one at the same time.

Now it seems to be working. It's doing a lot of writes since it's deleting the old rows. I don't know when that ends but hopefully soon. Things should be better now (and get much better once lint jobs are done with deletes).

Still getting flood of writes (newer check shows 47% of the writes) and reading the code, the "enable" part won't seem to be impacting the db writes it seems. I even see this:

			if ( $errorInfo['type'] === 'inline-media-caption' ) {
				// Drop lints of this type for now
				continue;
			}

Some brain dump.

In T369855#9999857, @Jclark-ctr wrote:

Server is back up no errors on idrac

Lint writes are now gone. db1179 so far looks good (logs looks good, aria logs look good, replication hasn't broken, ...) and is catching up (now only 7 days left). Once it's fully caught up, I repool it to reduce the chance of x1 bringing everything down.

I'm also repooling db1179 (x1) now (after checking for errors since it just started after hardware crash: T369855: db1179 crashed - hardware issues) that would prevent the circuit breaking to kick in for x1 and reduce the impact of the issue.

In T370304#10001049, @Ladsgroup wrote:

I'm also repooling db1179 (x1) now (after checking for errors since it just started after hardware crash: T369855: db1179 crashed - hardware issues) that would prevent the circuit breaking to kick in for x1 and reduce the impact of the issue.

it is because s4 gets overloaded and it brings down x1 with itself and a lot of other places go down too.

In T370304#9997418, @Pppery wrote:

Why did s4 problems cause wikis not on s4 to throw errors?

I'm on it. It's s4 being exhausted.

Edits by temp users/logged out users are quite small in number. Recording information on all users edits in all wikis requires two orders of magnitude more writes and more rows (specially in the original idea of storing all of them in one table) and can and will cause issues for the databases if we are not very careful. The last 90 days won't help much (if anything, it adds even more simultanes writes and potential deadlocks and contentions.) I highly recommend you separate the global block and GUC tables. GUC for temp users is tiny compared the global autoblocks one (or non-temp GUC one) and can easily move forward. But I recommend these mitigations before doing anything on global autoblocks or non-temp GUC:

• Do not try to update the timestamp if it's updated within last minute (debounce it).
• (maybe) if the last update was within an hour, roll a dice and update the timestamp in 10% of the times only.
• Don't store any information on the bots
• Don't store information on WMCS CIDR ranges (we skip global block checks for those as they caused outages).
• (maybe) Don't store anything on wikidata and always separately make a query to rc table of it when GUC is being used.
• Be very careful about your data structures and DDLs. I suggest splitting IPv6 and IPv4 if you're planning to store IPs for non-temp users. T292623#7546780
• Make sure the writes happen in a job, not part of main transaction or deferred update.

It is valid. I want to get it out sometime so you would just clone schema changes repo (not dbtools) and directly run the python file. No extra hassle necessary but it also depends on how the long-term automation for db work is going to look like.

You still need to get the design and schema approved before creating it in production: https://wikitech.wikimedia.org/wiki/Creating_new_tables

In T370304#9992814, @Tgr wrote:

Besides the DB issue, there's also the minor issue that it would be nice if this got reported to the user with the real error message, not the cryptic "Exception caught inside exception handler".

That's weird, s1 is enwiki and that's group2. I don't know anything that would do a cross wiki read to enwiki from a group1 wiki not to mention writes. Do we have some samples on what was overwhelming s1?

In T191156#9992640, @UOzurumba wrote:

@Ladsgroup Re: Tech News - What wording would you suggest as the content, and When should it be included? Thanks!

Regarding the size of banner, I seriously and honestly think both sides are right but they are not mutually exclusive.

• It is true that this is it has gotten much bigger. In my localhost it went from 22px (height) to 58px. That's a 2.6x original size. This jump can be quite annoying and at least takes some getting used to.
• It is also true that the original design was way too small as well and completely random without any adherence to UX principals and should have switched to a proper codex message box.
  • I want to expand upon this. This message is being shown when the user is either reading an outdated version of the article (stable=1) or a version of article that has pending changes (+ call to action to review the change). This is an important note about the article you're reading but it has been hidden away in an extremely small box for decades.
  • To draw a parallel, it is literally like reading an old version of the article (?oldid=###) and that has a massive yellow box at the top which is 70px in my localhost (bigger than the message box we merged). Amboxes are also usually bigger and more noticeable as well.

This should be announced.

Thank you!

The codesearch patch should be automatically deployed in a day. If it doesn't let me know to kick the service manually.

Also noting that this is a candidate master.

Since this is a valid bug report, I reopened it but removed the CommTech tag, feel free to unsubscribe.

We cut the size of this extension to half. Sure more can be done but I don't see any obvious removals anymore.

I bumped a couple replicas that were obviously under-utilized. The rest were like this for a reason (e.g. being used as vslow/dump replica) so I think it's good for now.

https://lists.wikimedia.org/postorius/lists/wikisource-conf.lists.wikimedia.org/

rev_timestamp is empty in the database:

wikiadmin2023@10.64.16.46(simplewiki)> select * from revision where rev_id = 22329;
+--------+----------+----------------+-----------+----------------+----------------+-------------+---------+---------------+---------------------------------+
| rev_id | rev_page | rev_comment_id | rev_actor | rev_timestamp  | rev_minor_edit | rev_deleted | rev_len | rev_parent_id | rev_sha1                        |
+--------+----------+----------------+-----------+----------------+----------------+-------------+---------+---------------+---------------------------------+
|  22329 |     7870 |         113519 |     28592 |                |              0 |           0 |     205 |             0 | 3vcdufxmrq68yxp7cmap3cr50qvdqm5 |
+--------+----------+----------------+-----------+----------------+----------------+-------------+---------+---------------+---------------------------------+
1 row in set (0.001 sec)

Making these images pushed to docker registry via CI is actually quite complicated because codesearch pushes via blubber (I don't think there is any other way) and you can push only one image per repo. Not to mention that converting Dockerfile of frontend to a blubber file is quite a lot of work :( any ideas are welcome here.

In T268199#9905399, @Dzahn wrote:

One thing to be done here:

add a systemd unit for the frontend. See T367479#9904556

Hi @TheDJ, it seems Idbc2a309fbca15 broke RTL

We don't provide free wiki hosting, please see https://www.mediawiki.org/wiki/Hosting_services

Fixed, the docker container went on zombie mode. New changes should get it fixed. I'll run it to backfill old tickets too.

^ dropped the user in production on m5.

you could probably make a MR to phabbot https://gitlab.wikimedia.org/ladsgroup/Phabricator-maintenance-bot/-/blob/master/new_wikis_handler.py?ref_type=heads

This is intentional and a process change since last year, we create wikis with placeholder config to ease the creation process since way more people can change those config post-creation and it simplify the process of creating the wiki a lot. I sent an email about this to langcom last year.

i.e. please create a separate ticket or mention it in T360310: Post-creation work for bewwiki

Thank you!

Also, looked at the 10 day rolling average and removed spikes, these show replicas that are behind the pack.

Hi, I reviewed the original idea long time ago and this still sounds good to me (no changes since my review). Please let me know if you need anything further from me.

Confirming what Manuel has said: This is needed as it powers https://labtestwikitech.wikimedia.org, it's quite a snowflake and I would appreciate if it's removed but AFAIK it's used by WMCS for testing of integrations (See T310795: Revive Labtestwikitech (formerly: Abolish labtestwikitech) for more info). If WMCS thinks they don't need that anymore (idm is replacing some functionalities I think), then by all means kill it with fire.

That's a nice trick. Feel free to add this to the existing views.

We looked at this and will post an update soon!

@Physikerwelt Hi! This proposal (T18491#9950713) is really similar to how math extension works. I wonder if you have some insights about this.

In T368098#9951531, @xcollazo wrote:

That single row change would make the current prefetch() code miss most all the revisions from rev_page = 20003, because the code has to walk the prefetched XML all the way to rev_id= 621927320 and it can't walk it back. Changing this behavior would require prefetch() surgery to make it work via rev_timestamp instead, and also we'd have to suffer thru another month or two of no prefetch available. Not fun.

Done now. Here is the script: P65796

I wish we could use linktarget ids instead but given their complexity and the fact that page title won't be repeated much I think it's fine for now. It's good to go.

Much better:

+-------------------------------------------------+------------+------------+----------+
| linter_template                                 | linter_cat | linter_tag | count(*) |
+-------------------------------------------------+------------+------------+----------+
| Template:Self                                   |         22 |            | 13969317 |
| Template:Information                            |         23 |            |  7926719 |
| Template:Artwork                                |         22 |            |  7273742 |
| Template:Welcome                                |         23 |            |  6723401 |
| Template:Information                            |         22 |            |  6136551 |
| Template:Artwork                                |         23 |            |  5845934 |
| Template:FlickreviewR                           |         23 |            |  5356323 |
| Template:Wikidata_Infobox                       |         22 |            |  5044754 |
| Template:Book                                   |         22 |            |  4731527 |
| Template:Geograph_from_structured_data          |         22 |            |  4476655 |
| Template:NASA-image                             |         22 |            |  4225716 |
| User:FlickreviewR/reviewed-pass                 |         23 |            |  3896330 |
| Template:FlickreviewR                           |         22 |            |  2787045 |
| Template:Book                                   |         23 |            |  2506893 |
| Template:Wikidata_Infobox                       |         23 |            |  2480577 |
| Template:PD-scan                                |         22 |            |  2028593 |
| Template:Autotranslate                          |         22 |            |  2025964 |
| Template:Photograph                             |         22 |            |  1983395 |
| Template:PD-USGov-NASA                          |         23 |            |  1918599 |
| Template:Geograph                               |         22 |            |  1862598 |
| Template:NASA-image                             |         23 |            |  1854221 |
| Template:PD-scan                                |         23 |            |  1675291 |

In T368136#9935249, @fnegri wrote:

Can we somehow remove the data that is currently filtered at the view layer, and instead blank out this data at the sanitarium level?

To emphasize. It's be nice to have that function in quarry/superset itself and deal with it in those services but not on mariadb level.

I don't know if MariaDB could allow a function being defined but not used in WHERE (and only in SELECT for example). Not to mention that someone could do something like this:

SELECT CONCAT(domain_index_to_domain(el_to_domain_index), el_to_path) as url from externallinks WHERE url = 'google.com/'

And we are back to the pathological mode.

That's part of updating query page entries and by design it's slow (it'll fill a cache). The proper fix is to move it to hadoop T309738: Move Mediawiki QueryPages computation to Hadoop (which I've been slowly working on it) but in the mean time, we can reduce its frequency.