Note that sql.php should still work.

Unassigned, unless there is a clear maintainer to review that patch and do any future upkseep.

It finished.

Note that the backing store can be moved again later on, making it easy to use mcrouter first.

Running it again since the screen is gone...

Using env vars for now seems OK for now.

There could be a FileBackendTestBase with subclasses for each backend. The "proxy" backend classes (FileBackendMultiWrite) could just use MemoryFileBackend instances. The tests for MemoryFileBackend would not need any config. The FSFileBackend subclass could just use the tmp directory. The other FileBackendStore subclass would need site config pointing to a real backend...

Possibly related is https://gerrit.wikimedia.org/r/c/mediawiki/core/+/659617 (Last-Write-Wins updates for subkeys within a key).

The headers_sent() checks should handle those, though maybe something is checked in the wrong place.

Closing given the updates to git master and REL1_35

Keep in mind that, strictly speaking, some of these problems are not even solved in MediaWiki for "core" DB shards. These include the "main" s[1-8] shards and the "extension" x1 shard. For example, a web request might update an S1 (enwiki) and S7 (centralauth) in one "transaction round", which just means that each of the relevant DB connections are checked for connectivity (pinged if there was no activity < 1 sec ago), and, after that passes, then the COMMITs are made in rapid succession. It is still possible, though very unlikely, that a proper subset of the transactions fail. Also, some events might be triggered from onTransaction() callbacks or PRESEND deferred updates.

I rebased the patch above. Once this is merged, I can consider this task closed.

I finished running this on labs via:

While we gzip memcached values, it would still be larger than the 1 mb limit. Even if I let WANCache use BagOStuff::WRITE_SEGMENTABLE, that is still a lot of I/O (even with "pcTTL" enabled).

The second one alone should be enough for a quick fix.

About how large is the IP list that will be stored in cache?

Ideally the SqlBagOStuff hashing would use HashRing, though any naive transition would involve a lot of misses/churn at first.

Playing around with

mwscript shell.php aawiki

...I noticed that SHOW SLAVE STATUS is empty in eqiad for the 'pc3' slot server. Both have SHOW MASTER STATUS output and read_only = 0. Any reason the eqiad DBs are not listening to the codfw DB binlogs?

In T246594#6862999, @Reedy wrote:

Um @aaron... why are you changing the title back to blacklisting?

I see two places in ConvertibleTimestamp.php that recast generic "Exception" errors into "TimestampException", which would make convert() return false, which could cause this problem.

In T269324#6794021, @Marostegui wrote:

• @Krinkle If your team could check what would be the behaviour if we simply depool a host via dbctl? Would that stop writes nicely? Would that break the site? :-)

Thank you

In T269324#6815030, @Marostegui wrote:

Thanks @Kormat
@Krinkle @aaron - let's go for the x1 approach but with local masters being writable then?

Possibly triggered by MediaWiki\Auth\AuthManager->autoCreateUser calls, which I also see in the logging for the same reqId.

The only thing that currently updates the replication positions on HTTP GET, that is not an easily spotted entrypoint like rollback/createaccount/login (which can be routed like HTTP POST) are:

• UpdateHitCountWatcher from AbuseFilter on edit form views (this should use the main stash or least the job queue); does not need chronology-protector
• CentralAuthUser->lazyImportLocalNames(); this should be solved by the migration script (T150506)
• SpecialContentTranslation setting global preferences just to store "user has seen X" state; this should use the main stash (or the job queue at least)
• ShortUrlHooks on page views; this will be fixed by T256993

I wonder what output_buffering value is being used in php.ini here. If I set it to off (not my distro default), I can trigger the MW_SETUP_CALLBACK use of ob_start()/OutputHandler::handle(). I definitely see some mismatch between the logic of OutputHandler::handle and MediaWiki::outputResponsePayload.

In T235705#6787784, @Mainframe98 wrote:

It appears rMW57325ba3bdce: objectcache: add statsd key metrics to BagOStuff classes is causing notices about undefined indexes:

Notice: Undefined index: wiki:user-quicktouched:id:1 in /vagrant/mediawiki/includes/libs/objectcache/MediumSpecificBagOStuff.php on line 1083
(10.0.2.15)
Notice: Undefined index: wiki:user-quicktouched:id:1 in /vagrant/mediawiki/includes/libs/objectcache/MediumSpecificBagOStuff.php on line 1098
(10.0.2.15)

I can only reproduce this on MediaWiki-Vagrant, which comes with a Redis setup out of the box. I don't know if this means Vagrant is misconfigured, or the aforementioned change is incomplete.

In T269324#6786272, @Marostegui wrote:

In T269324#6785507, @aaron wrote:

In T269324#6772394, @Marostegui wrote:

This is all done - hosts are ready to start getting data.

I was thinking that these would be setup just like the pcxxxx servers (e.g. each server in eqiad having circular replication with a corresponding server in codfw). Doing so will allow for things like https://phabricator.wikimedia.org/T113916 to proceed, since some uses cases involve writes that can happen in either datacenter. The MediaWiki mainstash config will be similar to the parser cache config as well.

That wasn't my understanding when this was discussed at T212129. Also I didn't get that impression when we briefly spoke on IRC a few days ago, I thought we just wanted another x1 with eqiad <-> codfw replication between masters, not 3 more independent replication chains.
Handling parsercache, from an operational point of view, at the moment is really painful (ie: having to commit to MW for a depooling, having to always have 3 lines and just duplicate IPs on the array, having no spares...) and I would like to expand this by 3 more replication chains. It puts certainly lots of overhead when having to operate with them.

There's probably also some puppet work that would need to be done as we simply don't want to just create 3 new pc4, pc5 and pc6 as that would be confusing - I would prefer if @Kormat can estimate how much this could be.

In T269324#6772394, @Marostegui wrote:

This is all done - hosts are ready to start getting data.

Logstash no longer shows the errors after the deploy.

It would be interesting to see if the rate of occurrence changes after T266055 is deployed.

In T273006#6781053, @dancy wrote:

@aaron, is https://gerrit.wikimedia.org/r/658780 sufficient to take care of this problem or do I need https://gerrit.wikimedia.org/r/c/mediawiki/core/+/658781 as well?

I can repro with

./srv_paratest --filter BagOStuff --use-bagostuff=redis

It should use DB Domains, which can always be converted to wiki IDs (though not 100% the other way around in some messy legacy edge cases that do not effect WMF). This is also what LoadBalancer has always expected in it's methods.

In T272078#6763020, @jijiki wrote:

@Krinkle @aaron the gutter pool sets a max TTL of 600s to any key with a TTL over 600s, do you think it is fine to keep the gutter-pool substitute the missing server?

I don't think it would be worth using pt-heartbeat for LoadBalancer::waitFor() unless the precision was much higher (likely problematically high in terms of spammy heartbeat table updates).

In T269326#6675369, @tstarling wrote:

I'm still thinking about re-adding scoped critical sections, because otherwise every critical section needs to be wrapped in try/finally, which is annoying. Yes a fatal error will be generated if a critical section scope is destroyed during request shutdown, but maybe that is the least bad option.

I wonder if the scope object should have an explicit exit function, to make it a bit more obvious in calling code. So most timeout exceptions will be thrown from the exit() call, not from __destruct().

In T264604#6681125, @jijiki wrote:

@Krinkle @aaron do you think we are ready to move this forward?

At first, I suspected a timeout causing a failed deferred updated, but it seems that no failure was logged likely due to NullLogger being used by sub-backends of FileBackendMultiWrite.

A subset of the log entries are bogus though (should be DEBUG, not ERROR).

See also: T264735

In T265778#6665113, @tstarling wrote:

Does this happen even if $wgShellLocale is set to the default of C.UTF8? We call setlocale() in Setup.php, and I think it should always be C.UTF8 or en_US.UTF8, there's not much justification for changing it.

Related task: T269325

I'm seeing timeout associated with these entries, e.g.:

Looked like a config error.

This seems to have gotten better over the weeks. Not sure why.

What is the state of this now? Are there any query graphs specific to this table?

No objections here.

I see plenty of timeouts where the error is not logged twice. Those that happen twice seem to be about 200 ms apart.

It could be related if the problem has to do with regenerations (since it was intermitted). In any case, excessive regeneration is a problem in itself.

Searching for << +channel:memcached +message:/.*deps.*/ >> I still see this sometimes.

In T264604#6601576, @jijiki wrote:

@aaron is there a timeline as to when those patches will be merged?