It's just a matter of backing out upstream D11011. We'll lose the fix for T75743 but I think that's better than the status quo.

What if we use the default backend instead of Elasticsearch? The problems we are having currently are supposed to be fixed in Phabricator's native backend, and we are not gaining anything from having Elasticsearch. Upstream won't take bugs specific to the Elasticsearch backend.

At the beginning we decided to go for Elasticsearch under the assumption that we could leverage our expertise and obtain better results. However, in practice this puts all the pressure on top of exactly 1,5 people (mainly @chad, also @Manybubbles to a certain extent). These guys are hyperbusy in several critical Wikimedia missions, and I'm the first one not willing to ask them to put more of their time here. To be more precise, if I had a minute of Chad, I would put it first on the Diffusion / gitblit-deprecate project (which is what he is doing).

So what about this: install Phabricator's default search backend, and if there are any bugs we can report them upstream.

I thought the reason for it was that Elasticsearch supported things that Mysql just wouldn't support? I can't find the task offhand where it was originally requested.

In T75854#783942, @Qgil wrote:

I'm after a problem related to terms in task titles, but I haven't found the exact description of the problem yet. Basically, searching for one or more words in the title of a task brings poor results, and I even suspect no results (sometimes?). This needs specific tests, because words in the title are usually mentioned in the body as well. Has anybody tried to search for words appearing only in the title, not in the body?

Here is a recent example: I searched via the quick search for "sentry coverage". The only ticket actually containing those two words in its title is result number 62. None of the top 10 results have both words in their text (including body); I'm pretty sure none of the other results don't either because "sentry" is a pretty specific term and the few tickets related to it have nothing to do with coverage. Result number 4 does not include either word and seems to be related in absolutely no way to the query.

In general, searching based on anything but projects is beyond useless, and I have taken to the habit of listing all tasks of a project and then using the browser's builtin search (Ctrl-F) tp try tp find what I am looking for.

Is there a way to see what request is sent to elasticsearch and play with alternatives?

have to fall back to using old-bugzilla for searching for duplicates

Time to reconsider T934? (And some other old-bugzilla changes which I proposed somewhere, related to T240#833612.)

What if we use the default backend instead of Elasticsearch?

Seems a sensible short-term solution.

I don't think we are going to ever stop using old-bugzilla search, because phabricator is intrinsically and proudly less semantic. So, we must admit that using old-bugzilla is integral part of the phabricator workflow, and spend the little effort needed to make it easier to use.

In T75854#1019135, @Qgil wrote:

and we are not gaining anything from having Elasticsearch.

The default backend now supports stemming / substrings? I probably missed that change.

As per initial description, this task is about search results, not about the search UI. Clarified the summary.

Another recent example: searching for "boomerang" (a JS library that I remembered mentioning in some discussion but had otherwise no clue where that discussion happened). The actual ticket is around place 50; not a single one in the first 20 search results contains that word. Instead they have something remotely similarly sounding (like "zoom" or "book") in a prominent field (title or description). So I would guess the problem (or part of it) is that the search score cutoff for word similarity should be much sharper.

I'm having a great time with our Phabricator instance, it's really made work more pleasant and sane, but yeah this bug is turning it into a black hole for me... Especially since I don't have permissions to create tags!

In T75854#1020525, @Aklapper wrote:

In T75854#1019135, @Qgil wrote:

and we are not gaining anything from having Elasticsearch.

The default backend now supports stemming / substrings? I probably missed that change.

This is fair, and the task to follow this feature upstream is https://secure.phabricator.com/T6740

If you are not convinced about changing the backend (I have no solid arguments myself other than conservativism), then an option is to do what Chad said a couple of weeks ago:

In T75854#999306, @Chad wrote:

It's just a matter of backing out upstream D11011. We'll lose the fix for T75743 but I think that's better than the status quo.

Oh, I'm totally fine with temporarily changing the backend to the SQL-backed standard search. @Chad also proposes that in https://lists.wikimedia.org/pipermail/wikitech-l/2015-February/080592.html for the time being.

It's just not clear to me which steps are needed on our side to do that, and who will perform them.

Very sick today but in general I can do this, probably wed as it will
create an unknown period of search outage. I wanted to give @Springle a
heads up since any fallout likely affects him.

So sure let's do it but may be a day or two.

In T75854#1024713, @chasemp wrote:

Very sick today but in general I can do this, probably wed as it will
create an unknown period of search outage. I wanted to give @Springle a
heads up since any fallout likely affects him.

So sure let's do it but may be a day or two.

Did this and talked to @Springle a bit about it. We should probably raise the aria_pagecache_buffer_size but it requires restart and is not a dynamic change.

wOOT! Thanks for all the great work, search results look exceptionally healthy today.

Is the backend change live? I'm not sure I'm seeing any improvements with example searches yet. For example, this task starts with "Fix provided search". I would expect a search for those words even without quotes to yield this as the first result. However, the first set of results only match one of the words in the title, and aren't relevant to the search. So generally the ranking still seems to be in dire need of improvement.

Question to other users: How valuable are the commit results? I find that I most frequently search for tasks, and it might be reasonable to exclude commit searches from default search. But I wonder how very active devs are using the tool and how important commit searches are to day-to-day workflows.

In T75854#1035759, @Eloquence wrote:

Question to other users: How valuable are the commit results? I find that I most frequently search for tasks, and it might be reasonable to exclude commit searches from default search. But I wonder how very active devs are using the tool and how important commit searches are to day-to-day workflows.

That's T76273.

I was trying to find T69476 which is currently titled

Add wiki title case sensitivity flag (is_sensitive) to meta_p.wiki to support jbo.wp and wiktionary tools

Searching open tasks only for 'jbo wp' and 'jbo.wp' returns no results!

However 'jbo.wp case' returns results, but not T69476 , even after all 300 results:
https://phabricator.wikimedia.org/search/query/26monNitFr8X/?offset=300

After many iterations, and a few swear words, I found it at the bottom of the first page of 'jbo wp title case' .

So I went back and tried 'jbo wp case' to see if that includes it; yes, it shows it after 400 results:
https://phabricator.wikimedia.org/search/query/FDjUwEs7K.dB/?offset=400

Search has to be the worst aspect of Phab for people who use it a lot, and know what they are looking for. It is not a productivity tool; it is a time sink. ;-(
Is there a document somewhere that describes the search algorithm in depth, so we can adjust how we remember bugs and search for them accordingly.

The top maniphest adv search result for Contains Words: interwiki google search (open & stalled only) is this task (T75854).

I was searching for T28115 (Interwiki redirects via URL and other 301s are indexed by Google search), and it did not appear in the search results at all.

This task (T75854) doesnt mention Interwiki or Google at all.

In T75854#1039822, @jayvdb wrote:

I was searching for T28115 (Interwiki redirects via URL and other 301s are indexed by Google search), and it did not appear in the search results at all.

This task (T75854) doesnt mention Interwiki or Google at all.

It almost sounds like the search is doing the opposite of what it should, i.e. it ignores specific keywords and it promotes "stopwords" / very common words.

In T75854#1035759, @Eloquence wrote:

Question to other users: How valuable are the commit results? I find that I most frequently search for tasks, and it might be reasonable to exclude commit searches from default search. But I wonder how very active devs are using the tool and how important commit searches are to day-to-day workflows.

As a developer, my experience is the same. When I use Phabricator search, I am almost always searching for tasks (important note, this might change somewhat if/when we use Phabricator for code review).

If I do need to search for a commit message (or code in a commit), I normally use the git command line. I can also use Gerrit for searching commit messages (not great performance-wise), or explicitly use the Commit document type in Phabricator search.

Commit search in Phabricator is utterly useless as it is now. Commits are not tagged by project, so you can only search by author name and description, which means heaps of false positives. (You can filter by repo via the super-user-hostile "callsign" field in Diffusion search, but that's a different interface not affected by the default setting for general search.) Not to mention that commits only appear after they have been merged.

Once code review happens on Phabricator, there will be a new entity which is the rough equivalent of open changesets on Gerrit, so commits are not going to be the "tasks" of code review, and I will imagine they will remain useless. Either way, I don't think it makes sense to keep a setting that is confusing for new users just because it might become useful in a year or so.

In T75854#1040795, @Tgr wrote:

Commit search in Phabricator is utterly useless as it is now. Commits are not tagged by project, so you can only search by author name and description, which means heaps of false positives. (You can filter by repo via the super-user-hostile "callsign" field in Diffusion search, but that's a different interface not affected by the default setting for general search.) Not to mention that commits only appear after they have been merged.

Once code review happens on Phabricator, there will be a new entity which is the rough equivalent of open changesets on Gerrit, so commits are not going to be the "tasks" of code review, and I will imagine they will remain useless. Either way, I don't think it makes sense to keep a setting that is confusing for new users just because it might become useful in a year or so.

Agreed. I don't think commits need to be part of the default search.

Field-testing and inspection of the dozens of reports about search issues provided some information about the current search, which I tried to summarise in the docs: https://www.mediawiki.org/w/index.php?title=Phabricator%2FHelp&type=revision&diff=1650699&oldid=1650693
Please revise.

In T75854#1287277, @Nemo_bis wrote:

Field-testing and inspection of the dozens of reports about search issues provided some information about the current search, which I tried to summarise in the docs: https://www.mediawiki.org/w/index.php?title=Phabricator%2FHelp&type=revision&diff=1650699&oldid=1650693
Please revise.

Thank you, @Nemo_bis! Only thought: I'm not sure what "search keyword" means in this context and I'm afraid others might face the same problem. What is the difference to a "search term" entered?

What is the difference to a "search term" entered?

Dunno, I guess you're right that "search term" is the standard phrase in English.