Page MenuHomePhabricator

Investigate AQS cassandra schema hash warninga
Closed, DeclinedPublic

Description

See here: https://logstash.wikimedia.org/app/kibana#/discover?_g=h@a98e110&_a=h@8a82733

Warning: Schema hash mismatch: {"_backend_version":1,"_config_version":0,"attributeIndexes":{},"attributes":{"_del":"timeuuid","_domain":"string","_tid":"timeuuid","access":"string","articles":"string","articlesJSON":"json","day":"string","month":"string","project":"string","year":"string"},"conversions":{"articlesJSON":{}},"iKeyMap":{"_domain":{"attribute":"_domain","type":"hash"},"_tid":{"attribute":"_tid","order":"desc","type":"range"},"access":{"attribute":"access","type":"hash"},"day":{"attribute":"day","type":"hash"},"month":{"attribute":"month","type":"hash"},"project":{"attribute":"project","type":"hash"},"year":{"attribute":"year","type":"hash"}},"iKeys":["_domain","project","access","year","month","day","_tid"],"index":[{"attribute":"_domain","type":"hash"},{"attribute":"project","type":"hash"},{"attribute":"access","type":"hash"},{"attribute":"year","type":"hash"},{"attribute":"month","type":"hash"},{"attribute":"day","type":"hash"},{"attribute":"_tid","order":"desc","type":"range"}],"options":{"updates":{"pattern":"random-update"}},"revisionRetentionPolicy":{"type":"all"},"secondaryIndexes":{},"staticKeyMap":{},"table":"top.pageviews","tid":"_tid","version":2,"versioned":false} != {"_backend_version":1,"_config_version":0,"attributeIndexes":{},"attributes":{"_del":"timeuuid","_domain":"string","_tid":"timeuuid","access":"string","articles":"string","articlesJSON":"json","day":"string","month":"string","project":"string","year":"string"},"conversions":{"articlesJSON":{}},"iKeyMap":{"_domain":{"attribute":"_domain","type":"hash"},"_tid":{"attribute":"_tid","order":"desc","type":"range"},"access":{"attribute":"access","type":"hash"},"day":{"attribute":"day","type":"hash"},"month":{"attribute":"month","type":"hash"},"project":{"attribute":"project","type":"hash"},"year":{"attribute":"year","type":"hash"}},"iKeys":["_domain","project","access","year","month","day","_tid"],"index":[{"attribute":"_domain","type":"hash"},{"attribute":"project","type":"hash"},{"attribute":"access","type":"hash"},{"attribute":"year","type":"hash"},{"attribute":"month","type":"hash"},{"attribute":"day","type":"hash"},{"attribute":"_tid","order":"desc","type":"range"}],"options":{"updates":{"pattern":"random-update"}},"revisionRetentionPolicy":{"type":"all"},"secondaryIndexes":{},"staticKeyMap":{},"table":"top.pageviews","tid":"_tid","version":3,"versioned":false}

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 23 2017, 7:06 PM
Nuria edited projects, added Analytics; removed Analytics-Kanban.Mar 22 2018, 4:12 PM
fdans triaged this task as Low priority.Mar 26 2018, 4:26 PM
fdans moved this task from Incoming to Operational Excellence on the Analytics board.
Milimetric added a subscriber: Milimetric.EditedOct 29 2018, 4:13 PM

Update: @JAllemandou says we made manual changes to one of the Cassandra models, meaning there's a mismatch in the hash made in the node AQS Cassandra module and the one in production, raising a warning. This task is about resolving the warning.

Milimetric raised the priority of this task from Low to Medium.Oct 29 2018, 4:15 PM
Milimetric moved this task from Incoming to Operational Excellence on the Analytics board.

Here are my findings:

  • The warning message happens for 2 schemas of AQS: pageviews.per.article.flat and top.pageviews. The difference between schema definitions are on the version field. The one store in database is 2 (verified in cassandra), the one coming from AQS definition is 3 (verified in code).
  • In this commit we have changed the version number of 3 schemas: pageviews.per.article.flat, top.pageviews and pageviews.per.project.v2. The 2 schemas raising warning messages are present, and there is the pageviews.per.project.v2 that doesn't raise warnings. Tis due to the database containing 3 for that schema version, which matches the AQS defined version number.
  • Looking at the restbase-mod-table-cassandra git repo, I found that schema migration are driven by changes in code-defined-configuration (as expected, a new schema line is stored in the cassandra meta table for the targetted keyspace), and that this new row is added only if a real schema change happens (see this function). Since we updated the version number without having actually changed the schema, no DB updates were made, hence throwing warnings (for more than a year :( Sadness).

My suggested solution for this is to downgrade the non-changed schema versions to 2.

I'm ok with downgrading. Initially we wanted to upgrade all of the versions so it would be uniform, but I don't think anyone felt very strongly one way or the other.

Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptDec 9 2019, 5:26 PM
mforns raised the priority of this task from Medium to Needs Triage.Dec 9 2019, 5:27 PM
mforns triaged this task as Medium priority.
Nuria moved this task from Ops Week to Deprioritized on the Analytics board.Feb 17 2020, 5:22 PM
Milimetric closed this task as Declined.Aug 31 2020, 5:09 PM

Haven't seen this in a while, maybe it went away!