Fri, Apr 19
Thu, Apr 18
Mon, Apr 15
I think that ::profile::services_proxy is lacking on these machines, @ArielGlenn would it be possible to add it?
So it's likely that snapshot machines are lacking the reverse proxy used for connection pooling needed to connect to elastic when using PHP, reopening.
I expect the dumps generated today to be correct, I'll keep an eye on it and re-open if needed.
Fri, Apr 12
Wed, Apr 10
Tue, Apr 9
Thu, Apr 4
Fixed by https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CirrusSearch/+/500611 (all cirrus scripts are broken because of this problem)
Wed, Apr 3
Not sure what to do here, the errors are legit since Elasticsearch was unreachable, should this more explicit in the error message?
Tue, Apr 2
Mon, Apr 1
path upstream: https://github.com/ruflin/Elastica/pull/1622
E.g. avoid queuing updates of this type or this size (possibly configurable), or run them differently, or to try it as today and then catch/suppress the failure - maybe logging a warning in its stead.
Thu, Mar 28
Backlog of updates is now completely absorbed, a script has been run to catchup lost updates, nothing we can do at this point except waiting for the maint script to stop moving to done.
Wed, Mar 27
patch upstream: https://github.com/ruflin/Elastica/pull/1621
The backlog of updates is being processed, once we catch up on these updates we will run a maint script to reindex lost updates.
Lowering to High as the immediate actions were taken, it now may take few days to fully sync the index and the database for the affected wikis.
Tue, Mar 26
Mon, Mar 25
@Joe no objections on my side, I'll be around to monitor the logs.
Mar 22 2019
Elastica is affected as well.
Mar 21 2019
Mar 19 2019
will SWAT the patch once ES6 is up and running everywhere?
Could it be caused by display_errors being set to true in INI settings? (T211488)
Mar 18 2019
Mar 15 2019
Mar 13 2019
Mar 12 2019
Mar 8 2019
Mar 1 2019
Feb 27 2019
The timeout problem was fixed by T216860 but I have no clue what cause the response from elastic to be displayed on site...