Page MenuHomePhabricator

Upgrade to Druid 0.12.3
Closed, ResolvedPublic8 Story Points

Description

While reading info about the Kafka Indexing Service (for banner impression data) I noticed that all the Druid 0.12.x releases contain improvements and fixes for the extension. They also offer a lot of new interesting things, like the built in caffeine cache extension, improvements in security auth (kerberos and basic auth), SQL query performance, etc.. It would be great to upgrade the Analytics Druid cluster before starting any work with KIS in my opinion, especially since the upgrade procedure when a KIS is active is more elaborate.

Releases: https://github.com/apache/incubator-druid/releases

This restriction is now fine since we are running 0.11:

Rollback restrictions
Please note that after upgrading to 0.12.0, it is no longer possible to downgrade to a version older than 0.11.0, due to changes made in #4762. It is still possible to roll back to version 0.11.0.

Event Timeline

elukey triaged this task as Medium priority.Oct 12 2018, 8:52 AM
elukey created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 12 2018, 8:52 AM
elukey updated the task description. (Show Details)Oct 12 2018, 8:55 AM

Created the 0.12.3 debs on boron and deployed the new version in Labs, ready for testing!

elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.

+1 to this, thanks!

fdans added a subscriber: fdans.Oct 15 2018, 4:23 PM

Let's remember to test turnilo against this version.

elukey moved this task from Backlog to In Progress on the User-Elukey board.Oct 18 2018, 12:27 PM

Deployed turnilo on turnilo.eqiad.wmflabs, seems working fine. To test it:

ssh -L 9091:turnilo.eqiad.wmflabs:9091 turnilo.eqiad.wmflabs

This indexation example worked fine:

{
  "type" : "index_hadoop",
  "spec" : {
    "ioConfig" : {
      "type" : "hadoop",
      "inputSpec" : {
        "type" : "static",
        "inputFormat": "io.druid.data.input.parquet.DruidParquetInputFormat",
        "paths" : "hdfs:///wmf/data/wmf/webrequest/webrequest_source=text/year=2018/month=5/day=18/hour=0"
      }
    },
    "dataSchema" : {
      "dataSource" : "webrequest",
      "granularitySpec" : {
        "type" : "uniform",
        "segmentGranularity" : "hour",
        "queryGranularity" : "second",
        "intervals" : ["2018-05-18T00:00:00Z/2018-05-18T01:00:00Z"]
      },
      "parser" : {
        "type" : "parquet",
        "parseSpec" : {
          "format" : "timeAndDims",
          "dimensionsSpec" : {
            "dimensions" : [
            "uri_host",
            "uri_path"
            ]
          },
          "timestampSpec" : {
            "format" : "auto",
            "column" : "dt"
          }
        }
      },
      "metricsSpec" : [
        {
          "type" : "count",
          "name" : "events"
        }
      ]
    },
    "tuningConfig" : {
      "type" : "hadoop",
      "overwriteFiles": true,
      "partitionsSpec" : {
        "type" : "hashed",
        "numShards" : 1
      },
      "jobProperties" : {
        "parquet.avro.add-list-element-records":"false",
        "mapreduce.job.reduce.slowstart.completedmaps": "0.95",
        "mapreduce.job.queuename": "default"
      }
    }
  }
}
elukey set the point value for this task to 8.Oct 24 2018, 8:00 AM

druid100[1-3] upgraded today, we'll proceed with druid public on monday if no issue will be registered!

elukey moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Oct 25 2018, 3:30 PM

Druid public upgraded too!

Nuria closed this task as Resolved.Oct 25 2018, 4:25 PM