Page MenuHomePhabricator

Deprecation warning on elasticsearch 6
Open, HighPublic

Description

A large number of [2019-03-22T14:19:52,864][WARN ][org.elasticsearch.deprecation.common.ParseField] Deprecated field [_retry_on_conflict] used, expected [retry_on_conflict] instead messages are seen on the upgraded elasticsearch 6 nodes.

  • T209859 Setting a negative [weight] in Function Score Query is deprecated and will throw an error in the next major version
  • T219265: _retry_on_conflict -> retry_on_conflict
  • T219266: nested_path/nested_filter has been deprecated in favour of the [nested] parameter
  • T219267: Deprecated field [auto_generate_phrase_queries] used, replaced by [This setting is ignored, use [type=phrase] instead to make phrase queries out of all text that is within query operators, or use explicitly quoted strings if you need finer-grained control]
  • T219268: The [classic] similarity is now deprecated in favour of BM25, which is generally accepted as a better alternative. Use the [BM25] similarity or build a custom [scripted] similarity instead.

Related Objects

StatusSubtypeAssignedTask
OpenNone
OpenNone
OpenNone
ResolvedReedy
ResolvedReedy
Resolvedmatthiasmullie
InvalidNone
ResolvedReedy
OpenNone
OpenNone
OpenEBernhardson
ResolvedEBernhardson
Resolved dcausse
Resolved dcausse
Resolved dcausse
Resolved dcausse
Resolved dcausse
OpenNone
ResolvedEBernhardson
DuplicateNone
ResolvedEBernhardson
ResolvedEJoseph
ResolvedEBernhardson

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 498380 had a related patch set uploaded (by DCausse; owner: DCausse):
[search/MjoLniR@master] [elastic] fix deprecation warning on retry_on_conflict

https://gerrit.wikimedia.org/r/498380

Change 498388 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: hide deprecation warning for ParseField

https://gerrit.wikimedia.org/r/498388

Change 498388 merged by Gehel:
[operations/puppet@production] elasticsearch: hide deprecation warning for ParseField

https://gerrit.wikimedia.org/r/498388

disabling this logger for now, let's not forget to re-enable it once we've fixed the underlying issues!

Change 498395 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: hide deprecation warning for ParseField

https://gerrit.wikimedia.org/r/498395

Change 498395 merged by Gehel:
[operations/puppet@production] elasticsearch: hide deprecation warning for ParseField

https://gerrit.wikimedia.org/r/498395

Mentioned in SAL (#wikimedia-operations) [2019-03-22T15:26:58Z] <gehel> restarting elasticsearch on elastic1046 for logging configuration change - T218994

The elasticsearch security manager is preventing log4j2 to auto-reload it's configuration (more precisely, it can't restart the GELF appender, as socket access is denied). So we will require a full cluster restart to reload the logging configuration. This will be done next week, bundled with the JVM upgrade.

Change 498420 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/CirrusSearch@master] Rename _retry_on_conflict to retry_on_conflict

https://gerrit.wikimedia.org/r/498420

Can't we apply the setting directly to elasticsearch cluster settings?

We already have this in the cluster settings:

"logger" : {
  "org" : {
    "elasticsearch" : {
      "cluster" : {
        "routing" : {
          "allocation" : "INFO"
        }
      },
      "http" : "DEBUG",
      "transport" : "DEBUG"
    }
  },
  "io" : {
    "netty" : "DEBUG"
  },
  "transport" : "DEBUG"
}

I suspect the following would mute the error messages:

curl -XPUT https://search.svc.eqiad.wmnet:9243/_cluster/settings -d '{"transient":{"logger.org.elasticsearch.deprecation.common.ParseField": "ERROR"}}'

I suspect the following would mute the error messages:

curl -XPUT https://search.svc.eqiad.wmnet:9243/_cluster/settings -d '{"transient":{"logger.org.elasticsearch.deprecation.common.ParseField": "ERROR"}}'

So simple. And yes, we can!

Mentioned in SAL (#wikimedia-operations) [2019-03-25T10:40:00Z] <gehel> disable deprecation warnings on elasticsearch eqiad - T218994

dcausse renamed this task from Deprecation warning on elasticsearch 6 expected [retry_on_conflict] to Deprecation warning on elasticsearch 6 .Mar 25 2019, 7:42 PM
dcausse updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2019-03-25T19:48:14Z] <dcausse> elasticsearch search cluster: SET "logger.org.elasticsearch.deprecation.index.query.functionscore.ScoreFunctionBuilder" to "ERROR" to chi/psi/omega@eqiad (T218994)

Mentioned in SAL (#wikimedia-operations) [2019-03-25T19:54:31Z] <dcausse> elasticsearch search cluster: SET "logger.org.elasticsearch.common.logging.DeprecationLogger" to "ERROR" to psi/omega@eqiad (T218994)

Went through logstash and filtered one error at a time to collect a list of new warnings. Task description has been updated will all of the warning messages, transient cluster settings have updated the following channels to ERROR level on all six cirrus clusters

  • org.elasticsearch.deprecation.search.sort.GeoDistanceSortBuilder
  • org.elasticsearch.deprecation.common.ParseField
  • org.elasticsearch.common.logging.DeprecationLogger
  • org.elasticsearch.deprecation.index.query.functionscore.ScoreFunctionBuilder
  • org.elasticsearch.deprecation.index.similarity.SimilarityService

Once we have resolved the warnings in the task description these should all be reverted back to INFO.

Mentioned in SAL (#wikimedia-operations) [2019-03-25T20:32:29Z] <ebernhardson> T218994 set various deprecation channels on all six cirrus elasticsearch clusters to ERROR

dcausse renamed this task from Deprecation warning on elasticsearch 6 to Epic: Deprecation warning on elasticsearch 6 .Mar 26 2019, 9:47 AM
dcausse edited projects, added Epic; removed Patch-For-Review.
dcausse updated the task description. (Show Details)

Change 498380 merged by jenkins-bot:
[search/MjoLniR@master] [elastic] fix deprecation warning on retry_on_conflict

https://gerrit.wikimedia.org/r/498380

fgiunchedi triaged this task as Medium priority.Apr 9 2019, 8:38 AM
Gehel renamed this task from Epic: Deprecation warning on elasticsearch 6 to Deprecation warning on elasticsearch 6 .Sep 17 2020, 3:56 PM
Gehel raised the priority of this task from Medium to Needs Triage.
Gehel removed projects: Epic, SRE.
Gehel moved this task from [epic] to elastic / cirrus on the Discovery-Search board.

Most of the warnings have been muted in elastic logging, moving forward we need to unmute to start up the stream of deprecation errors again.

CBogen triaged this task as High priority.Sep 28 2020, 3:43 PM
CBogen moved this task from needs triage to elastic / cirrus on the Discovery-Search board.

Cleared out out custom settings for deprecation logging on -eqiad and -codfw clusters, i haven't changed cloudelastic. Two high volume deprecations are still going (and were re-muted), will see if other low volume deprecations show up over time:

  • T219265: Remove leading _ in _retry_on_conflict. Previously resolved but we haven't deployed the config to enable the fix. Will ship a config patch next week
  • T209859: negative weights wont be allowed any more, previously captured in ticket.

I've also setup cirrus running against 6.8.18 locally, added tickets for deprecations triggered by the integration tests:

  • T288764: Set include_type_name in all get requests
  • T288765: Always provide minimum_should_match in bool queries.

Change 787068 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/mediawiki-config@master] cirrus: Enable DeprecationLoggedHttps

https://gerrit.wikimedia.org/r/787068

Change 787068 merged by jenkins-bot:

[operations/mediawiki-config@master] cirrus: Enable DeprecationLoggedHttps

https://gerrit.wikimedia.org/r/787068

Change 787074 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/mediawiki-config@master] Forward CirrusSearchDeprecation logs to logstash

https://gerrit.wikimedia.org/r/787074

Change 787074 merged by jenkins-bot:

[operations/mediawiki-config@master] Forward CirrusSearchDeprecation logs to logstash

https://gerrit.wikimedia.org/r/787074

Mentioned in SAL (#wikimedia-operations) [2022-04-27T21:12:13Z] <ebernhardson@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:787074|Forward CirrusSearchDeprecation logs to logstash (T218994)]] (duration: 00m 56s)

Mentioned in SAL (#wikimedia-operations) [2022-04-27T21:13:33Z] <ebernhardson@deploy1002> Synchronized wmf-config/ProductionServices.php: Config: [[gerrit:787068|cirrus: Enable DeprecationLoggedHttps (T218994)]] (duration: 00m 54s)

Mentioned in SAL (#wikimedia-operations) [2022-04-27T21:16:17Z] <ebernhardson@deploy1002> Synchronized wmf-config/ProductionServices.php: Config: Revert: [[gerrit:787068|cirrus: Enable DeprecationLoggedHttps (T218994)]] (duration: 00m 58s)

Change 788282 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/mediawiki-config@master] cirrus: Enable DeprecationLoggedHttps in deployment-prep

https://gerrit.wikimedia.org/r/788282

Change 788282 merged by jenkins-bot:

[operations/mediawiki-config@master] cirrus: Enable DeprecationLoggedHttps in deployment-prep

https://gerrit.wikimedia.org/r/788282

Change 790408 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/mediawiki-config@master] cirrus: Enable DeprecationLoggedHttps

https://gerrit.wikimedia.org/r/790408

Change 790408 merged by jenkins-bot:

[operations/mediawiki-config@master] cirrus: Enable DeprecationLoggedHttps

https://gerrit.wikimedia.org/r/790408

Mentioned in SAL (#wikimedia-operations) [2022-05-09T20:36:25Z] <cjming@deploy1002> Synchronized wmf-config/ProductionServices.php: Config: [[gerrit:790408|cirrus: Enable DeprecationLoggedHttps (T218994)]] (duration: 00m 51s)

Reviewing production logs for the week since we turned on deprecation logging, there are very few remaining:

  • Should clauses in the filter context will no longer automatically set the minimum should match to 1 in the next major version.. We missed something when doing T288765.
  • ApiFeatureUsage triggers "'y' year should be replaced with 'u'. Use 'y' for year-of-era. Prefix your date format with '8' to use the new specifier.". When we undeploy ApiFeatureUsage these queries will stop being issued.

Change 793124 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/extensions/CirrusSearch@master] metastore: Update deprecated version parameters

https://gerrit.wikimedia.org/r/793124

Change 793124 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] metastore: Update deprecated version parameters

https://gerrit.wikimedia.org/r/793124

New warning in logstash I hadn't seen before, will need to look into how metastore is using versioning but i suspect this should be minor.

warning: 299 Elasticsearch-6.8.23-4f67856 "Usage of internal versioning for optimistic concurrency control is deprecated and will be removed. Please use the `if_seq_no` and `if_primary_term` parameters instead. (request for index [mw_cirrus_metastore], type [_doc])"

Related stack frames:

#6 /srv/mediawiki/php-1.39.0-wmf.13/vendor/ruflin/elastica/lib/Elastica/Index.php(194): Elastica\Client->addDocuments(array, array)
#7 /srv/mediawiki/php-1.39.0-wmf.13/extensions/CirrusSearch/includes/MetaStore/MetaSaneitizeJobStore.php(90): Elastica\Index->addDocuments(array)
#8 /srv/mediawiki/php-1.39.0-wmf.13/extensions/CirrusSearch/maintenance/SaneitizeJobs.php(348): CirrusSearch\MetaStore\MetaSaneitizeJobStore->update(Elastica\Document)
#9 /srv/mediawiki/php-1.39.0-wmf.13/extensions/CirrusSearch/maintenance/SaneitizeJobs.php(251): CirrusSearch\Maintenance\SaneitizeJobs->updateJob(Elastica\Document)
#10 /srv/mediawiki/php-1.39.0-wmf.13/extensions/CirrusSearch/maintenance/SaneitizeJobs.php(89): CirrusSearch\Maintenance\SaneitizeJobs->pushJobs()
#11 /srv/mediawiki/php-1.39.0-wmf.13/maintenance/includes/MaintenanceRunner.php(119): CirrusSearch\Maintenance\SaneitizeJobs->execute()
#12 /srv/mediawiki/php-1.39.0-wmf.13/maintenance/doMaintenance.php(84): MediaWiki\Maintenance\MaintenanceRunner->run()
#13 /srv/mediawiki/php-1.39.0-wmf.13/extensions/CirrusSearch/maintenance/SaneitizeJobs.php(419): require_once(string)

Change 802654 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/extensions/CirrusSearch@master] metastore: Don't use versioning semantics for saneitize

https://gerrit.wikimedia.org/r/802654

Change 802654 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] metastore: Don't use versioning semantics for saneitize

https://gerrit.wikimedia.org/r/802654

Checked logstash with the query channel:CirrusSearchDeprecation AND NOT exception.trace:"ApiFeatureUsage" and verified nothing is coming through

We had a few custom log levels set in the elasticsearch cluster state, specifically logger.org.elasticsearch.deprecation.common.ParseField and logger.org.elasticsearch.deprecation.index.query.functionscore.ScoreFunctionBuilder were muted. I've unmuted them and nothing seems to be complaining (and they used to be quite verbose). There is reindexing currently going on so if the reindexing process triggers any deprecations those should come up today. If nothing comes up this can be considered complete.

Looks like wikidata is still triggering the should clause warning (rarely):

https://www.wikidata.org/w/api.php?format=xml&prop=info%7Cpageprops&action=query&generator=search&ppprop=wikibase_item&gsrsearch=e&gsrsort=random

warning: 299 Elasticsearch-6.8.23-4f67856 "Should clauses in the filter context will no longer automatically set the minimum should match to 1 in the next major version. You should group them in a [filter] clause or explicitly set [minimum_should_match] to 1 to restore this behavior in the next major version."

Adjusting the query a bit, this only triggers the warning when using random sort mode (which makes sense, it puts the whole query into filter context).

Change 816849 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/extensions/WikibaseCirrusSearch@master] Apply minimum_should_match for elastic 7

https://gerrit.wikimedia.org/r/816849

Change 816849 merged by jenkins-bot:

[mediawiki/extensions/WikibaseCirrusSearch@master] Apply minimum_should_match for elastic 7

https://gerrit.wikimedia.org/r/816849

New batch of logs came out Aug 4th, but they don't really make much sense. I would expect this error message to come from a mapping update, but the stack trace clearly shows this was run while sending writes. As this is the ElasticaWrite job it's highly likely this is the cloudelastic cluster, but it could be one of the normal clusters. Will write up a quick script to iterate over all indices in all clusters and check if anything still has an all field defined.

warning: 299 Elasticsearch-6.8.23-4f67856 "[_all] is deprecated in 6.0+ and will be removed in 7.0. As a replacement, you can use [copy_to] on mapping fields to create your own catch all field."

from /srv/mediawiki/php-1.39.0-wmf.22/extensions/CirrusSearch/includes/Elastica/DeprecationLoggedHttp.php(32)
#0 [internal function]: CirrusSearch\Elastica\DeprecationLoggedHttp->CirrusSearch\Elastica\{closure}(resource, string)
#1 /srv/mediawiki/php-1.39.0-wmf.22/vendor/ruflin/elastica/lib/Elastica/Transport/Http.php(166): curl_exec(resource)
#2 /srv/mediawiki/php-1.39.0-wmf.22/vendor/ruflin/elastica/lib/Elastica/Request.php(194): Elastica\Transport\Http->exec(Elastica\Request, array)
#3 /srv/mediawiki/php-1.39.0-wmf.22/vendor/ruflin/elastica/lib/Elastica/Client.php(689): Elastica\Request->send()
#4 /srv/mediawiki/php-1.39.0-wmf.22/vendor/ruflin/elastica/lib/Elastica/Bulk.php(358): Elastica\Client->request(string, string, string, array, string)
#5 /srv/mediawiki/php-1.39.0-wmf.22/extensions/CirrusSearch/includes/DataSender.php(322): Elastica\Bulk->send()
#6 /srv/mediawiki/php-1.39.0-wmf.22/extensions/CirrusSearch/includes/Job/ElasticaWrite.php(160): CirrusSearch\DataSender->sendData(string, array)
#7 /srv/mediawiki/php-1.39.0-wmf.22/extensions/CirrusSearch/includes/Job/JobTraits.php(138): CirrusSearch\Job\ElasticaWrite->doJob()
#8 /srv/mediawiki/php-1.39.0-wmf.22/extensions/CirrusSearch/includes/Updater.php(446): CirrusSearch\Job\CirrusGenericJob->run()
#9 /srv/mediawiki/php-1.39.0-wmf.22/extensions/CirrusSearch/includes/Updater.php(221): CirrusSearch\Updater->pushElasticaWriteJobs(array, Closure)
#10 /srv/mediawiki/php-1.39.0-wmf.22/extensions/CirrusSearch/includes/Updater.php(79): CirrusSearch\Updater->updatePages(array, integer)
#11 /srv/mediawiki/php-1.39.0-wmf.22/extensions/CirrusSearch/includes/Job/LinksUpdate.php(51): CirrusSearch\Updater->updateFromTitle(Title)
#12 /srv/mediawiki/php-1.39.0-wmf.22/extensions/CirrusSearch/includes/Job/JobTraits.php(138): CirrusSearch\Job\LinksUpdate->doJob()
#13 /srv/mediawiki/php-1.39.0-wmf.22/extensions/EventBus/includes/JobExecutor.php(79): CirrusSearch\Job\CirrusTitleJob->run()
#14 /srv/mediawiki/rpc/RunSingleJob.php(77): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#15 {main}