2100 tasks cannot be found by fulltext search (due to exception when indexing "Other Assignee" custom field)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Aklapper
	Jun 30 2025, 12:45 PM

Description

aklapper@phab1004:/srv/phab/phabricator$ sudo ./bin/search index --force T397642
 NOTE  Run this workflow with "--background" to queue tasks for the daemon workers.
 SELECT  Selecting objects to index...
 INDEXING  Indexing 1 object(s).
[                                                                  ]   0.0%[2025-06-30 12:21:10] EXCEPTION: (PhutilProxyException) In-process task ("PhabricatorSearchWorker") failed permanently. {>} (PhabricatorWorkerPermanentFailureException) Failed to update search index for document "PHID-TASK-y76yxykzqrftt7kbs76l": strlen() expects parameter 1 to be string, array given at [<phorge>/src/applications/search/worker/PhabricatorSearchWorker.php:94]
arcanist(), ava(), phorge(), translations(), wmf-ext-misc()
  #0 <#3> PhabricatorSearchWorker::doWork() called at [<phorge>/src/infrastructure/daemon/workers/PhabricatorWorker.php:124]
  #1 <#3> PhabricatorWorker::executeTask() called at [<phorge>/src/infrastructure/daemon/workers/PhabricatorWorker.php:166]
  #2 phlog(PhutilProxyException) called at [<phorge>/src/infrastructure/daemon/workers/PhabricatorWorker.php:178]
  #3 PhabricatorWorker::scheduleTask(string, array, array) called at [<phorge>/src/applications/search/worker/PhabricatorSearchWorker.php:23]
  #4 PhabricatorSearchWorker::queueDocumentForIndexing(string, array, boolean) called at [<phorge>/src/applications/search/management/PhabricatorSearchManagementIndexWorkflow.php:228]
  #5 PhabricatorSearchManagementIndexWorkflow::execute(PhutilArgumentParser) called at [<arcanist>/src/parser/argument/PhutilArgumentParser.php:492]
  #6 PhutilArgumentParser::parseWorkflowsFull(array) called at [<arcanist>/src/parser/argument/PhutilArgumentParser.php:377]
  #7 PhutilArgumentParser::parseWorkflows(array) called at [<phorge>/scripts/search/manage_search.php:21]
Done.                                                                         
 DONE  Forced search index updates for 1 document(s).

While sudo ./bin/search index --force T397641 (previous ID) works fine:

aklapper@phab1004:/srv/phab/phabricator$ sudo ./bin/search index --force T397641
 NOTE  Run this workflow with "--background" to queue tasks for the daemon workers.
 SELECT  Selecting objects to index...
 INDEXING  Indexing 1 object(s).
Done.                                                                         
 DONE  Forced search index updates for 1 document(s).

Might be an underlying reason for wrong data in the fdocument DB table. See T397392: Searching by Document Status "Open" includes resolved tasks, T305065: Global fulltext search results miss some tasks.

But not T365128: Active user accounts don't show up in fulltext search when setting "Document Status" to "open" (and email verification is required), that seems to be an upstream issue.

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		Aklapper	T398399 Drop a bunch of Maniphest Custom Fields from fulltext indexing
Resolved		Aklapper	T397392 Searching by Document Status "Open" includes resolved tasks
Resolved	BUG REPORT	Aklapper	T305065 Global fulltext search results miss some tasks
Resolved		Aklapper	T398197 2100 tasks cannot be found by fulltext search (due to exception when indexing "Other Assignee" custom field)

Event Timeline

Aklapper created this task.Jun 30 2025, 12:45 PM

Aklapper triaged this task as Medium priority.

No clue where that strlen() call is even located; none of the files listed in the stacktrace have one.
Thus added debug output in https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/commit/edb4011c7a89f8d5160275066ea9021ae38602d7

Aklapper mentioned this in T398328: Deploy Phabricator/Phorge 2025-07-01.Jul 1 2025, 3:13 PM

My added debug output in https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/commit/edb4011c7a89f8d5160275066ea9021ae38602d7 is useless as it only outputs the generic error handler. Reverted.

But after giving this some more thoughts I can now reproduce locally. <3

Aklapper edited projects, added Phabricator (Upstream), Upstream; removed Phabricator (Search).Jul 1 2025, 3:52 PM

So in our instance, this should be an easy fix by changing one line in /config/edit/maniphest.custom-field-definitions/:

"train.backup": {
  "name": "Other Assignee",
  "fulltext": true, // THIS SHOULD BE FALSE INSTEAD

so Phab doesn't blow up running into the "Other Assignee" array, failing to handle a non-string.

Because having User PHIDs indexed in the fulltext search DB isn't particularly useful either:

MariaDB [phabricator_maniphest]> SELECT * FROM maniphest_task_ffield WHERE fieldKey = "!iGz";
+-------+------------+----------+---------------------------------------------------------------------+-------------------------------------------------------+-----------------------------------------------------------------+
| id    | documentID | fieldKey | rawCorpus                                                           | normalCorpus                                          | termCorpus                                                      |
+-------+------------+----------+---------------------------------------------------------------------+-------------------------------------------------------+-----------------------------------------------------------------+
| 11784 |       2939 | !iGz     | ["PHID-USER-pu3kb2cppwbdhc4gyzzt","PHID-USER-la2vg7r2ndxntntedjhj"] |  phid user pu3kb2cppwbdhc4gyzzt la2vg7r2ndxntntedjhj  |  PHID USER pu3kb2cppwbdhc4gyzzt PHID USER la2vg7r2ndxntntedjhj  |
+-------+------------+----------+---------------------------------------------------------------------+-------------------------------------------------------+-----------------------------------------------------------------+
1 row in set (0.016 sec)

But upstream should also handle this much more gracefully, filed https://we.phorge.it/T16139 / https://we.phorge.it/D26123 for that.

Aklapper added a parent task: T398399: Drop a bunch of Maniphest Custom Fields from fulltext indexing.Jul 2 2025, 7:43 AM

Aklapper renamed this task from Exception in "./bin/search index --force": "strlen() expects parameter 1 to be string, array given" to Exception in "./bin/search index --force": "strlen() expects parameter 1 to be string, array given" due to fulltext indexing Other Assignee custom field.Jul 3 2025, 2:16 PM

Aklapper renamed this task from Exception in "./bin/search index --force": "strlen() expects parameter 1 to be string, array given" due to fulltext indexing Other Assignee custom field to 2100 tasks cannot be found by fulltext search (due to exception when indexing "Other Assignee" custom field).Jul 3 2025, 2:34 PM

Edited https://phabricator.wikimedia.org/config/edit/maniphest.custom-field-definitions/ and set fulltext to false for "train.backup" (Other Assignee). (but also note T398460: Phab Scap template vs actual value out of sync for maniphest.custom-field-definitions)
Get all task IDs which have the "Other Assignee" custom field not empty: SELECT t.id FROM maniphest_customfieldstringindex csi INNER JOIN maniphest_task t ON t.phid = csi.objectPHID WHERE csi.indexKey = "iGzhZ5s19vey" ORDER BY t.id; (2096 results, these are tasks currently not correctly indexed because of this issue)
A/B test: Went to latest T398449, copied a random string from the task desc "New Oauth 2.0 key taken into account", put that into the fulltext query at https://phabricator.wikimedia.org/maniphest/query/66vg7qt8S98z/#R , get no results
On CLI, ran sudo /srv/phab/phabricator/bin/search index --force T398449
A/B test: Reloaded https://phabricator.wikimedia.org/maniphest/query/66vg7qt8S98z/#R , get the one result we expected
Fed all the task IDs into sudo ./bin/search index --force --background Tx Ty Tz
Watched PhabricatorSearchWorker doing stuff on https://phabricator.wikimedia.org/daemon/

Aklapper added a parent task: T397392: Searching by Document Status "Open" includes resolved tasks.Jul 3 2025, 2:52 PM

Aklapper mentioned this in T397392: Searching by Document Status "Open" includes resolved tasks.Jul 3 2025, 2:56 PM

Aklapper added a parent task: T305065: Global fulltext search results miss some tasks.

Aklapper mentioned this in T305065: Global fulltext search results miss some tasks.Jul 3 2025, 2:59 PM

Aklapper mentioned this in T398399: Drop a bunch of Maniphest Custom Fields from fulltext indexing.Jul 8 2025, 1:20 PM

2100 tasks cannot be found by fulltext search (due to exception when indexing "Other Assignee" custom field)Closed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

2100 tasks cannot be found by fulltext search (due to exception when indexing "Other Assignee" custom field)
Closed, ResolvedPublic
Actions

Related Objects
Search...