Page MenuHomePhabricator

api feature logs should be sent to both eqiad and codfw clusters
Closed, ResolvedPublic

Description

As seen in T176335, api feature logs are sent only to the cirrus elasticsearch cluster. We should send them to both DC and read them from the active elasticsearch cluster.

Event Timeline

Gehel created this task.Sep 21 2017, 5:24 PM
Restricted Application edited projects, added Discovery-Search; removed Discovery-Search (Current work). · View Herald TranscriptSep 21 2017, 5:24 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Anomie reopened this task as Open.Sep 21 2017, 7:52 PM

Change 380507 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] logstash: api feature logs should be sent to both eqiad and codfw clusters

https://gerrit.wikimedia.org/r/380507

Change 380507 merged by Gehel:
[operations/puppet@production] logstash: api feature logs should be sent to both eqiad and codfw clusters

https://gerrit.wikimedia.org/r/380507

Mentioned in SAL (#wikimedia-operations) [2017-09-25T14:13:55Z] <gehel> rolling restart of logstash for config change - T176430

Gehel assigned this task to dcausse.Sep 25 2017, 2:21 PM

API Feature Usage logs are now sent to both codfw and eqiad. @dcausse could you make sure we have a mechanism in place to use the correct DC for reads as well?

Change 380517 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] logstash: fix index name for apifeatureusage

https://gerrit.wikimedia.org/r/380517

Change 380517 merged by Gehel:
[operations/puppet@production] logstash: fix index name for apifeatureusage

https://gerrit.wikimedia.org/r/380517

Mentioned in SAL (#wikimedia-operations) [2017-09-25T14:31:49Z] <gehel> rolling restart of logstash for config change - T176430

Mentioned in SAL (#wikimedia-operations) [2017-09-25T14:39:22Z] <gehel> dropping badly named apifeature index from cirrus elasticsearch codfw/eqiad - T176430

@Gehel the indices are properly created and have data in them, switching to codfw for api-feature-usage will be handled automatically by the logic in mediawiki-config. Changing the local DC will force api-feature-usage to use codfw, we can also force api-feature-usage to use codfw by tweaking mediawiki-config.
I'll update mediawiki-config with some comments to make this more obvious. I'm confident that this should just work, but I'm fine to run a quick test on production to be 100% sure it'll work when needed.

Change 380692 had a related patch set uploaded (by Gehel; owner: Guillaume Lederrey):
[operations/puppet@production] logstash: cleanup old api feature usage indices for all clusters

https://gerrit.wikimedia.org/r/380692

Change 380692 merged by Gehel:
[operations/puppet@production] logstash: cleanup old api feature usage indices for all clusters

https://gerrit.wikimedia.org/r/380692

debt closed this task as Resolved.Oct 2 2017, 2:13 PM