Page MenuHomePhabricator

Epic: Deprecation warning on elasticsearch 6
Open, NormalPublic

Description

A large number of [2019-03-22T14:19:52,864][WARN ][org.elasticsearch.deprecation.common.ParseField] Deprecated field [_retry_on_conflict] used, expected [retry_on_conflict] instead messages are seen on the upgraded elasticsearch 6 nodes.

  • T209859 Setting a negative [weight] in Function Score Query is deprecated and will throw an error in the next major version
  • T219265: _retry_on_conflict -> retry_on_conflict
  • T219266: nested_path/nested_filter has been deprecated in favour of the [nested] parameter
  • T219267: Deprecated field [auto_generate_phrase_queries] used, replaced by [This setting is ignored, use [type=phrase] instead to make phrase queries out of all text that is within query operators, or use explicitly quoted strings if you need finer-grained control]
  • T219268: The [classic] similarity is now deprecated in favour of BM25, which is generally accepted as a better alternative. Use the [BM25] similarity or build a custom [scripted] similarity instead.

Event Timeline

Gehel created this task.Mar 22 2019, 2:21 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 22 2019, 2:21 PM

Change 498380 had a related patch set uploaded (by DCausse; owner: DCausse):
[search/MjoLniR@master] [elastic] fix deprecation warning on retry_on_conflict

https://gerrit.wikimedia.org/r/498380

Elastica is affected as well.

Restricted Application edited projects, added Discovery-Search; removed Discovery-Search (Current work). · View Herald TranscriptMar 22 2019, 2:24 PM

Change 498388 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: hide deprecation warning for ParseField

https://gerrit.wikimedia.org/r/498388

Change 498388 merged by Gehel:
[operations/puppet@production] elasticsearch: hide deprecation warning for ParseField

https://gerrit.wikimedia.org/r/498388

Gehel added a comment.Mar 22 2019, 2:29 PM

disabling this logger for now, let's not forget to re-enable it once we've fixed the underlying issues!

Change 498395 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: hide deprecation warning for ParseField

https://gerrit.wikimedia.org/r/498395

Change 498395 merged by Gehel:
[operations/puppet@production] elasticsearch: hide deprecation warning for ParseField

https://gerrit.wikimedia.org/r/498395

Mentioned in SAL (#wikimedia-operations) [2019-03-22T15:26:58Z] <gehel> restarting elasticsearch on elastic1046 for logging configuration change - T218994

Gehel added a comment.Mar 22 2019, 3:39 PM

The elasticsearch security manager is preventing log4j2 to auto-reload it's configuration (more precisely, it can't restart the GELF appender, as socket access is denied). So we will require a full cluster restart to reload the logging configuration. This will be done next week, bundled with the JVM upgrade.

Change 498420 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/CirrusSearch@master] Rename _retry_on_conflict to retry_on_conflict

https://gerrit.wikimedia.org/r/498420

EBernhardson added a subscriber: EBernhardson.EditedMar 22 2019, 5:48 PM

Can't we apply the setting directly to elasticsearch cluster settings?

We already have this in the cluster settings:

"logger" : {
  "org" : {
    "elasticsearch" : {
      "cluster" : {
        "routing" : {
          "allocation" : "INFO"
        }
      },
      "http" : "DEBUG",
      "transport" : "DEBUG"
    }
  },
  "io" : {
    "netty" : "DEBUG"
  },
  "transport" : "DEBUG"
}

I suspect the following would mute the error messages:

curl -XPUT https://search.svc.eqiad.wmnet:9243/_cluster/settings -d '{"transient":{"logger.org.elasticsearch.deprecation.common.ParseField": "ERROR"}}'

I suspect the following would mute the error messages:

curl -XPUT https://search.svc.eqiad.wmnet:9243/_cluster/settings -d '{"transient":{"logger.org.elasticsearch.deprecation.common.ParseField": "ERROR"}}'

So simple. And yes, we can!

Mentioned in SAL (#wikimedia-operations) [2019-03-25T10:40:00Z] <gehel> disable deprecation warnings on elasticsearch eqiad - T218994

dcausse renamed this task from Deprecation warning on elasticsearch 6 expected [retry_on_conflict] to Deprecation warning on elasticsearch 6 .Mar 25 2019, 7:42 PM
dcausse updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2019-03-25T19:48:14Z] <dcausse> elasticsearch search cluster: SET "logger.org.elasticsearch.deprecation.index.query.functionscore.ScoreFunctionBuilder" to "ERROR" to chi/psi/omega@eqiad (T218994)

Mentioned in SAL (#wikimedia-operations) [2019-03-25T19:54:31Z] <dcausse> elasticsearch search cluster: SET "logger.org.elasticsearch.common.logging.DeprecationLogger" to "ERROR" to psi/omega@eqiad (T218994)

dcausse updated the task description. (Show Details)Mar 25 2019, 7:55 PM
EBernhardson updated the task description. (Show Details)Mar 25 2019, 8:15 PM
EBernhardson updated the task description. (Show Details)Mar 25 2019, 8:22 PM
EBernhardson added a comment.EditedMar 25 2019, 8:27 PM

Went through logstash and filtered one error at a time to collect a list of new warnings. Task description has been updated will all of the warning messages, transient cluster settings have updated the following channels to ERROR level on all six cirrus clusters

  • org.elasticsearch.deprecation.search.sort.GeoDistanceSortBuilder
  • org.elasticsearch.deprecation.common.ParseField
  • org.elasticsearch.common.logging.DeprecationLogger
  • org.elasticsearch.deprecation.index.query.functionscore.ScoreFunctionBuilder
  • org.elasticsearch.deprecation.index.similarity.SimilarityService

Once we have resolved the warnings in the task description these should all be reverted back to INFO.

Mentioned in SAL (#wikimedia-operations) [2019-03-25T20:32:29Z] <ebernhardson> T218994 set various deprecation channels on all six cirrus elasticsearch clusters to ERROR

EBernhardson updated the task description. (Show Details)Mar 25 2019, 8:32 PM
dcausse renamed this task from Deprecation warning on elasticsearch 6 to Epic: Deprecation warning on elasticsearch 6 .Mar 26 2019, 9:47 AM
dcausse edited projects, added Epic; removed Patch-For-Review.
dcausse updated the task description. (Show Details)

Change 498380 merged by jenkins-bot:
[search/MjoLniR@master] [elastic] fix deprecation warning on retry_on_conflict

https://gerrit.wikimedia.org/r/498380

fgiunchedi triaged this task as Normal priority.Apr 9 2019, 8:38 AM
dcausse updated the task description. (Show Details)Jun 25 2019, 5:36 PM