Page MenuHomePhabricator

Elasticsearch document versioning doesn't work in CirrusSearch
Closed, ResolvedPublic

Description

When we first implemented document versioning in CirrusSearch it was done incorrectly, we never actually set the version or version type on at a document parameter, only as values inside the document. As such elasticsearch has never used our version numbers. We typically ship updates using the super_detect_noop script, which requires usage of the update api. Per the docs:

External versioning (version types external & external_gte) is not supported by the update API as it would result in Elasticsearch version numbers being out of sync with the external system. Use the index API instead.

As such we will need to bring back https://gerrit.wikimedia.org/r/#/c/219118/ and deploy it, then update CirrusSearch to use that versioning support when available.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 26 2016, 5:40 PM
debt triaged this task as Normal priority.Sep 1 2016, 10:18 PM
debt moved this task from needs triage to Up Next on the Discovery-Search board.
debt moved this task from Up Next to Current work on the Discovery-Search board.
debt edited projects, added Discovery-Search (Current work); removed Discovery-Search.
debt added a subscriber: dcausse.

Change 219118 had a related patch set uploaded (by EBernhardson):
Document level noop detection with a version field

https://gerrit.wikimedia.org/r/219118

Change 308895 had a related patch set uploaded (by EBernhardson):
Add document versioning to FullyFeaturedConfig.php

https://gerrit.wikimedia.org/r/308895

Change 308896 had a related patch set uploaded (by EBernhardson):
Cirrus: Support document versioning

https://gerrit.wikimedia.org/r/308896

This has been split into multiple patches, due to the split between repositories and the order of deployment:

  • New handler in super_detect_noop script in the search-extra plugin
  • CirrusSearch support for new handler with feature flag to enable
  • CirrusSearch change to FullyFeaturedConfig.php which must be merged after vagrant has the new search-extra plugin version
  • wmf-config change to enable usage after production and beta cluster search clusters have been restarted with the new search-extra plugin

Change 219118 merged by jenkins-bot:
Document level noop detection with a version field

https://gerrit.wikimedia.org/r/219118

Change 308895 merged by jenkins-bot:
Add document versioning to FullyFeaturedConfig.php

https://gerrit.wikimedia.org/r/308895

this is still in needs-review as we need to do the full cluster restart and deploy the configuration change

debt closed this task as Resolved.Sep 23 2016, 9:02 PM
debt added a subscriber: debt.

This will be in production when T146210 goes out.

Change 308896 merged by jenkins-bot:
Cirrus: Support document versioning

https://gerrit.wikimedia.org/r/308896

Mentioned in SAL (#wikimedia-operations) [2016-10-05T23:09:07Z] <dereckson@tin> Synchronized wmf-config/CirrusSearch-common.php: Cirrus: Support document versioning (T144039) (duration: 00m 50s)