Page MenuHomePhabricator

Determine if we can run both elasticsearch and myisam fulltext backends in parallel
Closed, ResolvedPublic

Description

In order to properly evaluate elasticsearch we need to be able to run the same query against the same dataset using each engine to compare results. Ideally we can keep both innodb and elastic indexes updated in parallel and switch backends using a request parameter or feature flag.


Conclusion

Yes we can.

When elasticsearch is configured, phab uses it, when it's not configured it uses mysql. So we can run a separate indexer with PHABRICATOR_ENV=elasticsearch which will index into elastic.

Then we just need something like a feature flag to trigger the front end to query elasticsearch on demand (the feature flag is something I can easily implement in the phabricator request router)

Event Timeline

mmodell created this task.Nov 8 2016, 12:30 AM
Restricted Application added projects: Discovery, Discovery-Search. · View Herald TranscriptNov 8 2016, 12:30 AM

We could use phab-01 to test any changes that will allow us to use both MySQL full text and elasticsearch.

Also note we support elasticsearch2 now.

Deskana added a subscriber: Deskana.

Removing Discovery and Discovery-Search since we focus on search on the Wikimedia wikis.

@Deskana sorry, the projects were inherited from the parent task. Should probably remove them from that task as well.

@Deskana sorry, the projects were inherited from the parent task. Should probably remove them from that task as well.

No problem. I always think it's better to err on the side of including projects that may be relevant, as removing them is cheap. :-)

Oh no, I was wrong, I guess the projects were from a herald rule related to elasticsearch? Oh well, it seems to be all fixed now.

mmodell closed this task as Resolved.Nov 29 2016, 7:06 PM

Ok so this is possibly we just have to use config environments to provide the elasticsearch config to phabricator.

When elasticsearch is configured, phab uses it, when it's not configured it uses mysql... So we can run a separate indexer with PHABRICATOR_ENV=elasticsearch to trigger it to index into elastic. Then we just need something like a feature flag to trigger the front end to query elasticsearch on demand (the feature flag is something I can easily implement in the phabricator request router)

mmodell updated the task description. (Show Details)Nov 29 2016, 7:08 PM
mmodell moved this task from In-progress to Done on the Release-Engineering-Team board.