Sorry, this is puzzling me but I fail to reproduce...

I still can't reproduce:

• Open Special:Search@GermanWikipedia in advanced mode.
  • OK
• Enter the following search term: incategory:Vorlage:Fremdsprachenunterstützung insource:/invoke:Vorlage:lang\|full/
  • OK
• Make sure no particular namespace is specified.
  • What does this mean? I have "Standard" selected (which is Artikel apparently). Should I unselect "Standard"?
• Query.
  • OK
• 20 of 223 results should be shown.
  • I see 0 results here (Keeping Standard selected or removing it.)

@Aklapper thanks, but can you reproduce the bug after that: empty results when clicking "next page"?

@PerfektesChaos I cannot reproduce, searching for incategory:"Vorlage:Fremdsprachenunterstützung" insource:/invoke:Vorlage:lang\|full/ leads to no results.
It might be because you had set preferred namespaces to search by default? Could you check that it's the case and verify that the list of namespaces is properly carried on when you click on next page, the bug is perhaps here? Some saved namespaces are being lost when you click on such links?

Using more RAM by default might detrimental to most users given the fact that we set Xms to the value of Xmx. In other JAVA will always all the RAM we allow it to consume even if it's not needed.
I'd rather double check that the default we have is suited for "normal operations". As a CirrusSearch maintainer I'm fine tuning this myself by hand when needed.
Do you remember what kind of "elastic" work if any you were doing on this vagrant instance when it started to OOM?

@PerfektesChaos the URL attached to the task description leads to no results and thus it's impossible to check the behavior you describe as no search links are available.
Could you indicate the original search query you used and in which input box you typed it (if you used any of the new advanced input boxes).
While trying to search for insource:/\|NAME-AMTSSPRACHE/ I found nothing wrong with the generated links (more results/next|previous page), some part of the search query is URL encoded but this is expected as long as the original search query is kept intact in the search box that you see from the interface.

Many thanks for the investigation and the fixes!

No the data is not available in "searchable" way.

The few pages I checked seem to be invalid title strings to do exist in the DB but fails to load using Title::newFromDBKey(),
e.g. eswiki https://es.wikipedia.org/wiki/Discusi%C3%B3n:WP:MI which exists in the DB : https://es.wikipedia.org/w/api.php?action=query&prop=info&pageids=3214680&inprop=url

I think it's due to mobile_session_counts.tsv not being updated.
Trying to debug I found that mobile_session_counts.R#L64 is failing with

Error: Duplicate identifiers for rows (2, 3, 4)

to clarify the bug was affecting all users having set "classic" in their search preferences.

Thanks for the investigations!
See the attached patch for the cause.

There should be no need to fetch indices stats for this I think there was a misunderstanding in https://github.com/justwatchcom/elasticsearch_exporter/issues/115, the maintainer thought we wanted fine grained index statistics but we want node stats.
the /_nodes/_local/stats API endpoint should have all what we need.
Group stats are available when calling /_nodes/_local/stats?groups=_all
After a quick look it does not seem that the exporter is doing what we need.

done in https://github.com/o19s/elasticsearch-learning-to-rank/pull/197

I suggest converting the negative boosts to a positive boost and flip the filter condition to MUST_NOT, I think we can do this automatically within cirrus.

Existing indices will fail to load when ES6 will boot if the token filters they depend on are not available.

it seems to work well, the problems identified so far are:

• hotthread script (T209030)
• firewall config, mwmaint1002 is unable to talk to relforge:

sudo iptables -n -L | grep 10.64.16.77
ACCEPT     tcp  --  10.64.16.77          0.0.0.0/0            tcp dpt:9243

It's very probable that it's a specific thing to relforge but I suggest to see if we have other ad hoc rules that may need to be adjusted with new ports.

The merged patch only changed the "old" UI for Special:RecentChanges and Special:Watchlist, the new rcfilters UI is not affected by this change and perhaps should. (I failed to find from where the list of namespace is populated in rcfilters).