Page MenuHomePhabricator

The export_queries_to_relforge dag is not compatible with opensearch
Closed, ResolvedPublic1 Estimated Story Points

Description

This dag is reponsible for shipping search queries to relforge.

After migrating the first relforge host to opensearch we saw the failure:

: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
        at org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:403)
        at org.elasticsearch.spark.sql.EsSparkSQL$.saveToEs(EsSparkSQL.scala:99)
        at org.elasticsearch.spark.sql.ElasticsearchRelation.insert(DefaultSource.scala:622)
        at org.elasticsearch.spark.sql.DefaultSource.createRelation(DefaultSource.scala:107)
        at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:90)
        at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180)
        at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)
        at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)
        at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:132)
        at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:131)
        at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:989)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
        at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
        at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:989)
        at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:438)
        at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:415)
        at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:301)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.lang.Thread.run(Thread.java:750)
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalStateException: Invalid major version [1.3.20]. Version is lower than minimum required version [6.x].
        at org.elasticsearch.hadoop.rest.RestClient.mainInfo(RestClient.java:769)
        at org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:393)
        ... 35 more

We should probably switch to the opensearch-hadoop.
This dag cannot work properly in mixed cluster state.

This dag has run for several years but I don't recall anyone from the search team actively using the indices it generates, it might be a good to simply get rid of it?

AC:

  • decide if it's worth keeping this dag
  • if yes adapt it to use opensearch compatible hadoop connector
  • if not drop it

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
search: drop export_queries_to_relforgerepos/data-engineering/airflow-dags!1148dcausseT386945-search-drop-export_queries_to_relforgemain
Drop export_queries_to_relforgerepos/search-platform/discolytics!50dcausseT386945-drop-export_queries_to_relforgemain
Customize query in GitLab

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Gehel triaged this task as High priority.Mar 3 2025, 2:41 PM
Gehel raised the priority of this task from High to Needs Triage.
Gehel moved this task from Incoming to Done on the Discovery-Search (2025.03.01 - 2025.03.21) board.
Gehel set the point value for this task to 1.
Gehel subscribed.

We're not using this data, so let's drop the DAG instead of fixing it.

Deployed the airflow side. Discolytics is merged but not deployed, since it's only code deletion it can arrive whenever we next ship discolytics.