Page MenuHomePhabricator
Create Task
Maniphest T107949

upgrade RESTBase cluster to Cassandra 2.1.8
Closed, ResolvedPublic
Actions

Description

Cassandra 2.1.8 was released earlier this month, with a number of new bug fixes; No major regressions have since been reported.

One important new feature added is the logging of partitions that exceed a configurable threshold during routine compaction. This should come in handy in determining the source of our largest partitions (see: T94121).

Related Objects

Event Timeline

Eevans created this task.Aug 4 2015, 9:59 PM
Eevans raised the priority of this task from to Medium.
Eevans updated the task description. (Show Details)
Eevans added a project: RESTBase-Cassandra.
Eevans subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 4 2015, 9:59 PM
Eevans mentioned this in T94121: Understand and solve wide row issues for frequently edited and re-rendered pages.Aug 4 2015, 9:59 PM
Eevans set Security to None.
Eevans updated the task description. (Show Details)
Eevans added a subscriber: fgiunchedi.
• mobrovac awarded a token.Aug 5 2015, 12:02 AM
fgiunchedi added a project: acl*sre-team.Aug 5 2015, 10:46 AM

proposed plan:

  • upgrade cassandra to 2.1.8 via deb upgrades on the staging cluster
  • benchmark/stresstest
  • upload package to apt.w.o and upgrade production cluster
Restricted Application added a subscriber: Matanya. · View Herald TranscriptAug 5 2015, 10:46 AM
Eevans added a comment.Aug 5 2015, 2:06 PM
In T107949#1510274, @fgiunchedi wrote:

proposed plan:

  • upgrade cassandra to 2.1.8 via deb upgrades on the staging cluster
  • benchmark/stresstest
  • upload package to apt.w.o and upgrade production cluster

+1

• GWicke subscribed.Aug 5 2015, 6:27 PM

+1 from me as well.

fgiunchedi claimed this task.Aug 7 2015, 8:41 AM

the test cluster has been upgraded to cassandra 2.1.8 (+latest openjdk in T104888)

fgiunchedi added a comment.EditedAug 11 2015, 9:44 AM

upgrade plan, starting today:

  • upgrade row A machines, (restbase100[127]) with nodetool drain && sudo apt-get -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install cassandra
  • check regressions, http://grafana.wikimedia.org/#/dashboard/db/cassandra-restbase-eqiad
  • if no regressions proceed with row B (restbase[348]) on wed
  • if no regressions proceed with row C (restbase[569]) on thurs

edit: nodetool flush vs nodetool drain

fgiunchedi added a comment.Aug 11 2015, 9:46 AM
root@carbon:~# reprepro --noskipold --restrict cassandra update
aptmethod 'http' seems to have a obsoleted redirect handling which causes
reprepro to request files multiple times. Work-around activated, but better
only use it for targets not redirecting (or upgrade to apt >= 0.9.4 if
that is the http method from apt)!
Calculating packages to get...
Getting packages...
Installing (and possibly deleting) packages...
Exporting indices...
Deleting files no longer referenced...
root@carbon:~# reprepro list jessie-wikimedia cassandra
jessie-wikimedia|thirdparty|amd64: cassandra 2.1.8
jessie-wikimedia|thirdparty|source: cassandra 2.1.8
root@carbon:~#
fgiunchedi added a comment.Aug 11 2015, 10:35 AM

cosmetic issue output contains %s spotted while looking at the logs, benign

restbase1002:~$ grep %s /var/log/cassandra/system.log
INFO  [MemtableFlushWriter:1] 2015-08-11 10:31:36,380 Memtable.java:393 - Completed flushing %s; nothing needed to be retained.  Commitlog position was /var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-tmp-ka-1-Data.db
INFO  [MemtableFlushWriter:1] 2015-08-11 10:32:07,021 Memtable.java:393 - Completed flushing %s; nothing needed to be retained.  Commitlog position was /var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-tmp-ka-2-Data.db
INFO  [MemtableFlushWriter:2] 2015-08-11 10:33:07,032 Memtable.java:393 - Completed flushing %s; nothing needed to be retained.  Commitlog position was /var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-tmp-ka-3-Data.db
INFO  [MemtableFlushWriter:1] 2015-08-11 10:34:07,041 Memtable.java:393 - Completed flushing %s; nothing needed to be retained.  Commitlog position was /var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-tmp-ka-4-Data.db
Eevans added a comment.Aug 11 2015, 2:37 PM
In T107949#1526823, @fgiunchedi wrote:

upgrade plan, starting today:

  • upgrade row A machines, (restbase100[127]) with nodetool flush && sudo apt-get -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install cassandra
  • check regressions, http://grafana.wikimedia.org/#/dashboard/db/cassandra-restbase-eqiad
  • if no regressions proceed with row B (restbase[348]) on wed
  • if no regressions proceed with row C (restbase[569]) on thurs

very sensible; +1

Eevans added a comment.EditedAug 11 2015, 2:37 PM
In T107949#1526933, @fgiunchedi wrote:

cosmetic issue output contains %s spotted while looking at the logs, benign

restbase1002:~$ grep %s /var/log/cassandra/system.log
INFO  [MemtableFlushWriter:1] 2015-08-11 10:31:36,380 Memtable.java:393 - Completed flushing %s; nothing needed to be retained.  Commitlog position was /var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-tmp-ka-1-Data.db
INFO  [MemtableFlushWriter:1] 2015-08-11 10:32:07,021 Memtable.java:393 - Completed flushing %s; nothing needed to be retained.  Commitlog position was /var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-tmp-ka-2-Data.db
INFO  [MemtableFlushWriter:2] 2015-08-11 10:33:07,032 Memtable.java:393 - Completed flushing %s; nothing needed to be retained.  Commitlog position was /var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-tmp-ka-3-Data.db
INFO  [MemtableFlushWriter:1] 2015-08-11 10:34:07,041 Memtable.java:393 - Completed flushing %s; nothing needed to be retained.  Commitlog position was /var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-tmp-ka-4-Data.db

I'll follow up with upstream on this.

Eevans added a comment.Aug 11 2015, 2:42 PM
In T107949#1526823, @fgiunchedi wrote:

upgrade plan, starting today:

  • upgrade row A machines, (restbase100[127]) with nodetool flush && sudo apt-get -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install cassandra
  • check regressions, http://grafana.wikimedia.org/#/dashboard/db/cassandra-restbase-eqiad
  • if no regressions proceed with row B (restbase[348]) on wed
  • if no regressions proceed with row C (restbase[569]) on thurs

On minor nit here, you might consider using drain instead of flush here:

$ nodetool help drain
NAME
        nodetool drain - Drain the node (stop accepting writes and flush all
        column families)

SYNOPSIS
        ...

OPTIONS
   ...
Eevans added a comment.Aug 11 2015, 6:27 PM
In T107949#1527439, @Eevans wrote:
In T107949#1526933, @fgiunchedi wrote:

cosmetic issue output contains %s spotted while looking at the logs, benign

restbase1002:~$ grep %s /var/log/cassandra/system.log
INFO  [MemtableFlushWriter:1] 2015-08-11 10:31:36,380 Memtable.java:393 - Completed flushing %s; nothing needed to be retained.  Commitlog position was /var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-tmp-ka-1-Data.db
INFO  [MemtableFlushWriter:1] 2015-08-11 10:32:07,021 Memtable.java:393 - Completed flushing %s; nothing needed to be retained.  Commitlog position was /var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-tmp-ka-2-Data.db
INFO  [MemtableFlushWriter:2] 2015-08-11 10:33:07,032 Memtable.java:393 - Completed flushing %s; nothing needed to be retained.  Commitlog position was /var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-tmp-ka-3-Data.db
INFO  [MemtableFlushWriter:1] 2015-08-11 10:34:07,041 Memtable.java:393 - Completed flushing %s; nothing needed to be retained.  Commitlog position was /var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-tmp-ka-4-Data.db

I'll follow up with upstream on this.

This has already been reported, and will be fixed in 2.1.9

fgiunchedi added a comment.Aug 12 2015, 2:36 PM
In T107949#1527473, @Eevans wrote:
In T107949#1526823, @fgiunchedi wrote:

upgrade plan, starting today:

  • upgrade row A machines, (restbase100[127]) with nodetool flush && sudo apt-get -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install cassandra
  • check regressions, http://grafana.wikimedia.org/#/dashboard/db/cassandra-restbase-eqiad
  • if no regressions proceed with row B (restbase[348]) on wed
  • if no regressions proceed with row C (restbase[569]) on thurs

On minor nit here, you might consider using drain instead of flush here:

looks good, thanks! tomorrow I'll complete the upgrade

fgiunchedi closed this task as Resolved.Aug 13 2015, 9:12 AM

this is completed

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL