I feel like the intent of the code of conduct is that in the event of a conflict, it is considered superior to any local policies

Could we maybe just have a new separate view that contains user_id, user_name and user_is_emailable where user_is_emailable is a boolean field that checks that disablemail is not on and use email is authenticated?

Umm, which page creation log specificly do you mean? Can you give a url to where the log used to be?

Checked in betalabs (big watchlist (>700 pages) and very few updates happening) and production (few pages on the Watchlist and big volume of updates) - the new query works fast in both cases.

But yes, the best answer is to start using the new servers :-)

Ok, so suppress, spamblacklist and titleblacklist must not be in the view.

Btw, it seems the root cause was two things:

• straight_join preventing efficient queries in case the associated page has a few links (e.g. not the giant category use case)
• the query killer not recognizing union queries.

MediaWiki already has a lot of internal support for RSS (For history pages, watchlist, recentchanges), So i doubt it would be super onerous.

In T177026#3645652, @Aklapper wrote:

2017-09-28T20:55:51
A database query timeout has occurred.
Error: 2062 Read timeout is reached (10.64.32.25)
#0 /srv/mediawiki/php-1.31.0-wmf.1/includes/libs/rdbms/database/Database.php(979): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#1 /srv/mediawiki/php-1.31.0-wmf.1/includes/libs/rdbms/database/Database.php(1361): Wikimedia\Rdbms\Database->query(string, string)
#2 /srv/mediawiki/php-1.31.0-wmf.1/includes/specials/SpecialRecentchanges.php(358): Wikimedia\Rdbms\Database->select(array, array, array, string, array, array)
#3 /srv/mediawiki/php-1.31.0-wmf.1/includes/specialpage/ChangesListSpecialPage.php(746): SpecialRecentChanges->doMainQuery(array, array, array, array, array, FormOptions)
#4 /srv/mediawiki/php-1.31.0-wmf.1/includes/specialpage/ChangesListSpecialPage.php(543): ChangesListSpecialPage->getRows()
#5 /srv/mediawiki/php-1.31.0-wmf.1/includes/specials/SpecialRecentchanges.php(166): ChangesListSpecialPage->execute(NULL)
#6 /srv/mediawiki/php-1.31.0-wmf.1/includes/specialpage/SpecialPage.php(522): SpecialRecentChanges->execute(NULL)
#7 /srv/mediawiki/php-1.31.0-wmf.1/includes/specialpage/SpecialPageFactory.php(578): SpecialPage->run(NULL)
#8 /srv/mediawiki/php-1.31.0-wmf.1/includes/MediaWiki.php(287): SpecialPageFactory::executePath(Title, RequestContext)
#9 /srv/mediawiki/php-1.31.0-wmf.1/includes/MediaWiki.php(851): MediaWiki->performRequest()
#10 /srv/mediawiki/php-1.31.0-wmf.1/includes/MediaWiki.php(523): MediaWiki->main()
#11 /srv/mediawiki/php-1.31.0-wmf.1/index.php(43): MediaWiki->run()
#12 /srv/mediawiki/w/index.php(3): include(string)
#13 {main}

I think the idea is that people with complex needs should be encouraged to use ForeignDBRepoViaLB instead.

This probably should be using the tmp_3 index (which is on production but not in mediawiki - its on (rc_timestamp, rc_namespace)) But according to the explain on production its using the rc_timestamp index. [my quick test showed 3.48 seconds when FORCE INDEX for tmp_3 vs 42.65 for query as is. Running the faster query first, so any effects of warm cache would favour the slow one]

Its impossible to optimize this query further unless we denormalize namespace into the revision table (which is unlikely to happen).

It would be even better if we could also have a list of open changesets of newcomers submitted during this quarter. Real-time might be too much, but perhaps renewed each month? With the same goal: identify which projects and reviewers could help, and keep newcomers engaged and learning.

One could fill a book with the areas of wikimedia that nobody or everybody jointly (which is the same as nobody) is responsible for...

Whats the point of keeping newcomers engaged if they become totally ignored the minute they hit some sort of no-longer-a-newcomer class?

assuming that person was on IRC at the time :-)

There has also been discussions on Operations on whether we should alert if mediawiki goes into read only, but this can cause many many false positives that it might be paging every hours :-)

In theory most of the time, phan will already enforce type hints from the comment block, so I'm not sure it would change much in practise in terms of code reliability (Unless we also got better at declaring more specific types).

Usually these filters are pretty selective, but it's possible for them to not be. Thankfully, the kinds of rows that bloat the RC table (Wikidata and categorization) are also among the kinds of rows that don't have ORES scores associated with them. So for wikis with bloated RC tables, the filter will be selective, and typical selections for these filters ("only show bad edits") are quite selective too (<10% on enwiki IIRC). However, the user could choose the reverse filter ("only show good edits") and that wouldn't be very selective.

@awight Btw, some debugging tips for next time:

So I guess part of the problem with the alerting is its looking at individual lag, where the real issue (MediaWiki goes into read-only mode) comes up when all the slaves are lagged more than 6 seconds - which as it stands, MediaWiki doesn't even create a log entry for this situation.

It will still probably be better then the old query for people with large but not insane sized watchlists, but the savings just wont be as spectacular for big watchlists

In T176456#3681276, @jmatazzoni wrote:

Hey @awight, I think @Bawolff's brilliant suggestion just enabled @Catrope to totally fix this problem. Following the protocol above, even with Latest Revision on, I get a 2.19 second load time—with 30 days selected! Good job gang!

Following the protocols in T176445, I get:

==="Typical ORES filter set"
https://en.wikipedia.org/wiki/Special:Watchlist?highlight=1&limit=250&days=30&urlversion=2&hidepreviousrevisions=1&hidecategorization=1&hideWikibase=1&damaging=maybebad

2.71 secs at 7 days.

==="'Rare' ORES filter set"
https://en.wikipedia.org/wiki/Special:Watchlist?hidebots=1&hidecategorization=1&hideWikibase=1&hidelog=1&highlight=1&limit=50&days=7&urlversion=2&damaging__verylikelybad_color=c4&damaging=likelybad&goodfaith=likelygood

2.13 secs at 7 days.

@Etonkovidova, moving this to QA. @Trizek-WMF, if confirmed we should prominently announce that we've fixed the bug that caused ORES on watchlist to load very slowly. This bug predated the New Filters release, so anyone who has every dropped ORES on Watchlist in the past because of slowness should give it another try. (@Jdforrester-WMF, just pinging you because I think this will make you happy.)

Hi, There's been reports of high amounts of lag on commons causing read only mode, especially between 7:10-10:10 UTC today. (I filed T178094) Perhaps the rate of deletion of commons RC entries is too aggresive?

Oh, well in that case, if a particular filter is very selective it may make sense to allow query plans wherethe base table is ores_classification, with some sort of index like (oresc_model, oresc_class, oresc_probability). If we can narrow down the range of revisions (for options of the form only show last three days), putting oresc_rev at the end of that index may make sense too for index condition pushdown. This of course only makes sense if the filter is very selective as the required filesort has a high overhead.

I think it might be cool to have an extension that provides ?uselang=qqs.

Rv my tag change. Wrong bug.

I wonder if its related to deleting all the wikibase stuff in rc(?)

Well in general this sounds nice - there is a potential security issue here in the event that a content handler returns an unsafe mime that could execute scripts. E.g. returning text/html, application/svg+xml would be bad.Some browsers do a lot of sniffing on text/plain (not sure how much that still is the case). And there is also the potential for a format which is downloaded and executed unsafely (e.g. text/csv interpreted by excel. ) which may or not be an issue. Even if not executed could maybe cause a network request to be issued to a third party server which may be unacceptable privacy wise.

I dont think that index exists in production (not sure; i dont remember it one i was looking about an hour ago).

In T176456#3661739, @awight wrote:

I'll just admit right here that I'm not particularly good at query optimization. But still, I'll go ahead and make the obvious comment. I'm not sure we're using multi-column indexes correctly. For example, this is the only index that covers the oresc_probability column,

CREATE INDEX /*i*/oresc_model_class_prob ON /*_*/ores_classification (oresc_model, oresc_class, oresc_probability);

The order of columns matters here, so in my understanding, we can only filter by probability after we've already narrowed by model and class. Similarly, we can't filter by class until model is locked in. Something like that. So I'm imagining that our multi-column key is causing the conditions to be evaluated in a difficult order.

It would be great if someone with more DB chops could take a look at this.

Copying my comment from T171027#3677195 over here as I commented on the wrong bug, and its good to keep things together for reference:

In T164796#3564570, @awight wrote:

@jmatazzoni ah sorry, I wasn't suggesting that the query is any faster than you said. I was running on a dedicated analytics box, so the timings I gave only matter on a relative scale. The takeaway is that the query runs 30x faster when the specific unused clause I quoted is removed.

The bad news is that I didn't understand the change_tag clause when I looked at this problem yesterday. It's:

(SELECT Group_concat(ct_tag SEPARATOR ',') 
                             FROM   `change_tag` 
                             WHERE  ct_rc_id = rc_id)            AS `ts_tags`,

This isn't actually unused, it's returning tags which I assume are rendered in the results, somehow. The only quick fix that comes to mind is that we can fetch the tags in a second query, after limiting to a small number of revisions when paging the view.

I'd be happy to see a volunteer take this on, though I note that they'd need some support from us to get the code reviewed and a follow-up performance review undertaken to see whether it would be OK

In T107875#3668064, @mmodell wrote:

This is ancient history now.

Well wikidata is almost certainly a contributing factor, (particularly for russian) I would hesitate to blame it solely for slow ores on watchlist speeds, without more evidence. Especially on english wikipedia with its large recentchanges table and relatively low usage of wikidata.

Correction: the response is cached according to the maxage specified in the request; however, I’m not sure if this works if pages are edited or purged (as far as I can tell, the browser doesn’t validate its cached data), so it can be hard to choose the right maxage. (This workaround also requires that you parse and transform the canonical URI, which is ugly.)

My vote is redirect to #wikimedia-dev

Dropping irc channels is not really an operations thing. The channel founder would be the one who'd have to drop it.

In T171027#3676023, @jmatazzoni wrote:

In T171027#3673060, @Lydia_Pintscher wrote:

This is a hugely political issue. Let's please not do this unless necessary.

Right now, ORES is unusable on Watchlist for English, Russian and probably some other big wikis. It causes delays of 30 seconds to a minute when users have more than 3 days set. I'm told Wikidata is a likely culprit. If that's correct (I can't say it is or isn't), then by that standard, some kind of change appears to be "necessary," and, as Risker says, some kind of escalation appears both justified and required.

I personally find real time code reviews useful in some circumstances:

• if there is something really complicated or tricky happening, going over it in irc can help to make sure all the cases are covered (not really relavent to this ticket since that is rare and generally applies to experianced contribs who are easy to contact over irc)
• for very new contributors, where we are not really reviewing the patch so much as advising people how to contribute - real time communication can be very helpful.
• if there are problems with the patch, real time means a very small feedback loop. Issue is identified and fixed immediately instead of problem noted than fixed a week later and then relooked at 2 weeks later by which time the reviewer has totally forgotten context/doesnt care anymore. (Part of this is a cultural issue where reviewers are discouraged from "helping" get a patch to a ready state because they are then no longer qualified as a neutal reviewer)
• [i suspect] having code review office hours means time gets specificly set aside to do code review. Well this is a poor substitute for actually setting time aside, I suspect its better than the status quo.

In T177895#3674239, @Bawolff wrote:

From a security prespective, the main risk (that a per-user off switch could address) is that these features allow making an xss "permenant" (excluding the risk of malicious admin). An option by itself wouldnt prevent that as a malicious person could just turn it back on. Would need to make the user do something like reenter password before reenabling

From a security prespective, the main risk (that a per-user off switch could address) is that these features allow making an xss "permenant" (excluding the risk of malicious admin). An option by itself wouldnt prevent that as a malicious person could just turn it back on. Would need to make the user do something like reenter password before reenabling

In T177707#3673488, @Mike_Peel wrote:

"Open question: Where should the cut-off initially be?" - the problem I have here is that it's particularly useful to see changes to highly-used pages, as it's those same pages that need to be watched closer to make sure that vandalism is quickly reverted. E.g. if a country name is vandalised, that needs to be caught quickly. If that's not feasible, then I'm not sure I see the benefit of having a middle-region, perhaps it should only appear for the directly linked page... Or maybe it should show in RelatedChanges rather than RecentChanges.

Any implementation approach you wish to follow? Could you include this in the proposal? I mean how you would solve this problem.

In T171027#3667090, @jcrespo wrote:

The introduction of wikidata events on recentchanges has converted the "light" recentchanges table into a monolithical 500GB table:

commonswiki]> SHOW TABLE STATUS like 'recentchanges'\G
*************************** 1. row ***************************
           Name: recentchanges
           Rows: 617078906
 Avg_row_length: 566
    Data_length: 349283375366
   Index_length: 20766761170
 Auto_increment: 965208270

This was a) Not discussed previously with us DBAs b) created huge performance issues c) Made maintenance impossible d) break recentchanges functionality for users e) break watchlist functionality for users f) after report of breakage almost 3 months ago, no workaround/mitigation as been done g) we are close of running out of space on several main database servers, breaking all of Wikipedia h) propose to create even newer indexes, which only makes all of the above problems worse.

Given all of the above, I administratively will take the emergency decision (because "things are broken)" of finding out which functionality created this issue- revert it on code and purge all new rows created. Once that has been done, the owners of the functionality can requests it to be enabled again, after a strategy not breaking production is proposed. I am switching this to unbreak now to reflect how bad things are, and that I am taking leadership to get something finally done.

I'm curious if this would be better served by $wgEnableScaryTranscluding ?

In T173891#3645059, @MusikAnimal wrote:

The table is now live and populated on all wikis, so I've repurposed this ticket for creating the replicas view. Hope that is OK. Any update on that? Anything I can do to help? People have been asking about this :)

All the eval()s in the callstack in the first error is kind of weird though. I wonder if corrupted ResourceLoaderStorage is a possibility. But that wouldn't make sense with error being intermittent.

In T177105#3647663, @Yann wrote:

The problem is not ONE tool, but quite a lot of scripts do not load. And yes, it is the French Wikisource.

On the URL given above, I get

Use of "importScriptURI" is deprecated. Use mw.loader instead. load.php?debug=true&lang=fr&modules=jquery%2Cmediawiki|mediawiki.legacy.wikibits&only=scripts&skin=monobook&version=0cg1aij:10861 
get @ load.php?debug=true&lang=fr&modules=jquery%2Cmediawiki|mediawiki.legacy.wikibits&only=scripts&skin=monobook&version=0cg1aij:10861

Use of "mw.toolbar" is deprecated. 9load.php?debug=true&lang=fr&modules=jquery%2Cmediawiki|mediawiki.legacy.wikibits&only=scripts&skin=monobook&version=0cg1aij:10861 
get @ load.php?debug=true&lang=fr&modules=jquery%2Cmediawiki|mediawiki.legacy.wikibits&only=scripts&skin=monobook&version=0cg1aij:10861

Please, load LanguageConverter from [[meta:User:He7d3r/Tools/LanguageConverter.js]]. wikisource.org/w/index.php?title=User:He7d3r/Tools/LanguageConverter.js&action=raw&ctype=text/javascript:2 
(anonymous) @ wikisource.org/w/index.php?title=User:He7d3r/Tools/LanguageConverter.js&action=raw&ctype=text/javascript:2

Use of "wgNamespaceNumber" is deprecated. Use mw.config instead. load.php?debug=true&lang=fr&modules=jquery%2Cmediawiki|mediawiki.legacy.wikibits&only=scripts&skin=monobook&version=0cg1aij:10861 
get @ load.php?debug=true&lang=fr&modules=jquery%2Cmediawiki|mediawiki.legacy.wikibits&only=scripts&skin=monobook&version=0cg1aij:10861

Use of "mw.toolbar" is deprecated. 2load.php?debug=true&lang=fr&modules=jquery%2Cmediawiki|mediawiki.legacy.wikibits&only=scripts&skin=monobook&version=0cg1aij:10861
get @ load.php?debug=true&lang=fr&modules=jquery%2Cmediawiki|mediawiki.legacy.wikibits&only=scripts&skin=monobook&version=0cg1aij:10861

Realistically, this is probably the fault of those gadgets in question, and bugs should be filed with their maintainer.

In T176269#3644205, @Iniquity wrote:

In T176269#3639418, @Anomie wrote:

Err, what? That non sequitur sounds like pure FUD.

Hm, I think you have never tried to control the community's vandals and users who can broke everything when they start to work with design or code.

@TBolliger Is this something your team would be interested in taking on.

Assuming this works, special pages on commons should start updating again on oct 5.

With that in mind im going to close this bug. Having revdel kill all the logs automatically sounds like a very reasonable feature request, but its not a newsletter issue.

Also happening for me on T176055

At the very least i expect people will want it on their user pages because users seem to like making overly ornate user pages. I could also imagine people using it in the portal namrspace.

In T174126#3635393, @jcrespo wrote:

> This seems to have many hardcoded assumptions about lists being small (e.g. Reording requiring to submit the order of all items on a list, with the api unworkable if lists have > 1000 entries. Some queries not having LIMIT clauses. Paging using OFFSET clauses instead of range conditions in the WHERE. Some parts not even having paging at all).

I specifically said I will block any deployment to production if lists get larger than 100 items/rows as there was not allocated hardware for this functionality. I still stand by that statement, no matter if it has been ignored by the development team or not.

In T155725#3634976, @MarcoAurelio wrote:

I was thinking that, maybe, we could do a deploy of this extension to the Beta Cluster, where we're having a continuous spamming problem, once @Bawolff issues in the security review are addressed, so we can test it there and prepare for deployment on production wikis? According to https://www.mediawiki.org/wiki/User:Legoktm/StopForumSpam_effectiveness, more than 20% of IPs blacklisted on SFS are also blocked on Wikimedia wikis so it seems like a good resource in anti spamming activities.

IMO the RESTBase part doesn't require a security review: it's basically just a web proxy that converts between slightly different URL styles.

Review of a1b8a6cc70e. (Just the ReadingList extension. I haven't reviewed anything RESTBase related or client side related)

In T176533#3632822, @Zoranzoki21 wrote:

Which patch used for disabling (moving from true to false)?

We should log history of users' logins.

Log user agent string, time, IP of successful logins on ALL wikis.

Make it view-able in account info each wiki.

They're not fully redundant, since rc_type for Wikidata is RC_EXTERNAL (from core, thus not Wikidata-specific).