Change #1185884 had a related patch set uploaded (by Aklapper; author: Aklapper):

[operations/puppet@production] phabricator: remove defunct ElasticSearch backend settings

https://gerrit.wikimedia.org/r/1185884

Thanks for reporting this!

https://phabricator.wmcloud.org/config/issue/elastic.misconfigured/ still dreamed of an ElasticSearch backend.
Production instance has the same (unused) Local Config value (probably yet another side effect of the mess described in T398460: Phab Scap template vs actual value out of sync for numerous values) but a working local overwrite:

[
	{
		"type": "mysql",
		"roles": {
			"read": true,
			"write": true
		}
	}
]

I also set that as a Database Value on the test instance now at https://phabricator.wmcloud.org/config/edit/cluster.search/

I assume this was a result of content in https://phabricator.wikimedia.org/source/operations-puppet/browse/production/hieradata/role/eqiad/phabricator.yaml

We should remove the Local Config value so Phab instances would fall back to the Global Default value.

Change #1185884 merged by Dzahn:

[operations/puppet@production] phabricator: remove defunct ElasticSearch backend settings

https://gerrit.wikimedia.org/r/1185884

Change #1188815 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator: drop elasticsearch settings in codfw

https://gerrit.wikimedia.org/r/1188815

Change #1188815 merged by Dzahn:

[operations/puppet@production] phabricator: drop elasticsearch settings in codfw

https://gerrit.wikimedia.org/r/1188815

reverts do not automatically get linked to the tickets they belong to.. so leaving a comment.. I reverted the changes above.

additional code changes are needed.. if we just remove values without removing lookups trying to find these values then puppet fails

also: there are separate Hiera settings for a) eqiad b) codfw and c) devtools

Trying to reproduce the error I actually got this error:

Unhandled Exception ("PhutilAggregateException")
Fulltext Search failed. Use an application's Advanced Search (e.g. tasks only, or commits only) instead of the Global Search, or set a smaller search scope, and/or use more less common terms and put phrases into quotation marks, and/or use operators. See https://www.mediawiki.org/wiki/Phabricator/Help#Searching_for_items
    - AphrontMalformedRequestException: You are trying to save some data to permanent storage, but the request your browser made included an incorrect token. Reload the page and try again. You may need to clear your cookies.
      This was a Web request.
      This request had no CSRF token.

@Aklapper we can edit just hieradata/cloud/eqiad1/devtools/common.yaml and remove the phabricator_cluster_search there and it will only influence the test instance. that is safest. but we still need to make the parameter an optional thing. currently it is required.

I will look more at this.

Yea. .so... regardless if we use or not use elasticsearch in production.. the status is that the puppet class requires that we specify a search service / host.

And in cloud VPS devtools we don't have an search service / host.

In production it's set to a service name like search.svc.eqiad.wmnet and in the test setup it was only set to localhost so that puppet does not error out because there is no value for it.

We didn't have an expectation that this cluster search would work in the test setup.

We need to make the entire parameter optional or there is an error if we don't set it to something.

This phabricator_cluster_search setting is passed to the main class::phabricator which then just passes ALL the settings as a hash to the phabricator local settings json file.

So we need to look at upstream config options and see if it's optional too.

noticed while debugging:

The last Puppet run was at Fri Aug 22 22:08:26 UTC 2025 (36081 minutes ago). Puppet is disabled. apc settings busted for devtools instance - brennen - @brennen

The fact that puppet happens to be disabled is actually good for me right now.. since I wanted to test what happens if I just remove the cluster_search settings from the phabricator config ... without dealing with the puppet part right away.

I removed the cluster_search section directly from /etc/phabricator/config.yaml on the test instance.

root@phabricator-bullseye:/etc/phabricator# diff /root/config.yaml config.yaml 
36,46d35
<     cluster_search:
<     - hosts:
<       - host: localhost
<         protocol: http
<         roles:
<           read: true
<           write: true
<       path: "/phabricator"
<       port: 9200
<       type: elasticsearch
<       version: 5

@A_smart_kitten Could you let me know what happens now if you try again?

In T403948#11187729, @Dzahn wrote:

Trying to reproduce the error I actually got this error:

Unhandled Exception ("PhutilAggregateException")
Fulltext Search failed. Use an application's Advanced Search (e.g. tasks only, or commits only) instead of the Global Search, or set a smaller search scope, and/or use more less common terms and put phrases into quotation marks, and/or use operators. See https://www.mediawiki.org/wiki/Phabricator/Help#Searching_for_items
    - AphrontMalformedRequestException: You are trying to save some data to permanent storage, but the request your browser made included an incorrect token. Reload the page and try again. You may need to clear your cookies.
      This was a Web request.
      This request had no CSRF token.

Yeah, that's T404134#11178734. Just coincidental timing to discover (what seem to be) two separate bugs with global searching within a week of each other, lol

In T403948#11187791, @Dzahn wrote:

@A_smart_kitten Could you let me know what happens now if you try again?

Noting that I just removed the override @Aklapper added for https://phabricator.wmcloud.org/config/edit/cluster.search/ in T403948#11156998, so that we can more easily test whether this bug has been fixed on the puppet/hiera side of things.

After removing that local config override, I get the error reported in this task once again.
Maybe worth noting that https://phabricator.wmcloud.org/config/edit/cluster.search/ still shows the "Local Config" as being (what look like) the values you removed in T403948#11187774?

If removing config from /etc/phabricator/config.yaml and restarting everything does not change what phorge sees as local config then I am not sure what is needed. Probably a deployment.

we can more easily test whether this bug has been fixed on the puppet/hiera side of things.

Puppet is currently disabled and the file I edited directly is what puppet would edit. So you can and should ignore puppet/hiera for now. That would be for later once we know what config we actually want.

In T403948#11190631, @Dzahn wrote:

we can more easily test whether this bug has been fixed on the puppet/hiera side of things.

Puppet is currently disabled and the file I edited directly is what puppet would edit. So you can and should ignore puppet/hiera for now. That would be for later once we know what config we actually want.

Clumsy wording on my part, sorry :) I was probably looking for a way to convey something like 'whether the bug has been fixed by whatever's being changed on the backend of things right now'

There is still this:

/srv/deployment/phabricator/deployment$ sudo grep -r cluster_search *

scap/templates/phabricator/conf/local/local.json.j2:  "cluster.search": {{ phabricator.local.cluster_search | tojson }},

So.. can anyone tell if the original issue is fixed (even if there is another unrelated issue) now?

The original issue (ie., global searches on the test Phab instance failing with the error in the task description) has been fixed by the 'database value' added at https://phabricator.wmcloud.org/config/edit/cluster.search/ IIUC.

The issue was worked around by manually overwriting https://phabricator.wmcloud.org/config/edit/cluster.search/ after each deployment.
The proper fix would be not deploying a setting onto the test instance which defines non-existing ElasticSearch 5 machines as that instance's search backend.
In my understanding.

Currently https://phabricator.wmcloud.org/config/edit/cluster.search/ still shows Elasticsearch stuff under "Local Config". That should not be the case.

If something got fixed in the meantime, then I guess the way to find out would be deploying to that test instance. Maybe next Tuesday again.

Thank you both for the responses.

So.. this means to me the real questions are:

• If editing /etc/phabricator/config.yaml and restarting doesn't change things about the "Local Config" then what else needs to happen for that to sync to the Local Config? (I assume a proper deploy) (cc: @brennen )

• re: > The proper fix would be not deploying a setting onto the test instance ..

So which one is it.. the test instance config needs to be different from the prod instance OR .. cluster.search is not used anywhere and should be dropped from both configs?

side comment: puppet is still disabled there and I wanted to re-enable it but due to the APC size thing (--> T401157 , https://gerrit.wikimedia.org/r/c/operations/puppet/+/1180643) we can't right now. Because the test instance doesn't have the same amount of RAM as production yet we hard code that cache size. We need to Hiera'ize that value so that we can adjust it for the test instance.

If editing /etc/phabricator/config.yaml and restarting doesn't change things about the "Local Config" then what else needs to happen for that to sync to the Local Config? (I assume a proper deploy)

Yeah, the scap deploy process copies those values into the actual config files in /srv/phab/phabricator/conf/local/.

I made this to unblock the "re-enable puppet part" not directly related to this ticket but midly blocking it:

https://gerrit.wikimedia.org/r/c/operations/puppet/+/

In T403948#11220920, @brennen wrote:

Yeah, the scap deploy process copies those values into the actual config files in /srv/phab/phabricator/conf/local/.

Ok, thanks. that's what I was guessing. Let's do that then. Andre already suggested Tuesday.

We deployed to the test instance today.

We agreed the cluster_search config should be removed from both test and prod setup.

Change #1192636 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] phabricator: drop cluster_search config

https://gerrit.wikimedia.org/r/1192636

brennen opened https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/84

scap: remove cluster.search value from local.json.j2

brennen merged https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/84

scap: remove cluster.search value from local.json.j2

Using the top-right search bar to search for anything on the Phab test instance results in the following error:
Unhandled Exception ("PhutilAggregateException")

This is not the case any longer. I can search for test words now without seeing this exception.

@A_smart_kitten ok to close ?

Sure thing, it seems okay to me as well; and https://phabricator.wmcloud.org/config/edit/cluster.search/ no longer shows elasticsearch under 'Local Config' :) Thanks for checking in.

It seems like https://gerrit.wikimedia.org/r/1192636 is still pending here, but I assume that'll be dealt with as appropriate (and that it isn't a blocker to this being closed out).

Yea, that patch still needs to be deployed in production but the test instance is already fixed regardless.

Change #1192636 merged by Dzahn:

[operations/puppet@production] phabricator: drop cluster_search config

https://gerrit.wikimedia.org/r/1192636