As seen in T176335, api feature logs are sent only to the cirrus elasticsearch cluster. We should send them to both DC and read them from the active elasticsearch cluster.
Description
Description
Details
Details
Customize query in gerrit
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | herron | T281266 Decommission old ELK5 Logstash cluster | |||
| Resolved | herron | T297239 Move logstash api-feature-usage output away from v5 cluster | |||
| Resolved | colewhite | T176335 logs sent to logstash are lost when the elasticsearch cirrus cluster is unavailable | |||
| Resolved | dcausse | T176430 api feature logs should be sent to both eqiad and codfw clusters |
Event Timeline
Comment Actions
Change 380507 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] logstash: api feature logs should be sent to both eqiad and codfw clusters
Comment Actions
Change 380507 merged by Gehel:
[operations/puppet@production] logstash: api feature logs should be sent to both eqiad and codfw clusters
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2017-09-25T14:13:55Z] <gehel> rolling restart of logstash for config change - T176430
Comment Actions
API Feature Usage logs are now sent to both codfw and eqiad. @dcausse could you make sure we have a mechanism in place to use the correct DC for reads as well?
Comment Actions
Change 380517 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] logstash: fix index name for apifeatureusage
Comment Actions
Change 380517 merged by Gehel:
[operations/puppet@production] logstash: fix index name for apifeatureusage
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2017-09-25T14:31:49Z] <gehel> rolling restart of logstash for config change - T176430
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2017-09-25T14:39:22Z] <gehel> dropping badly named apifeature index from cirrus elasticsearch codfw/eqiad - T176430
Comment Actions
@Gehel the indices are properly created and have data in them, switching to codfw for api-feature-usage will be handled automatically by the logic in mediawiki-config. Changing the local DC will force api-feature-usage to use codfw, we can also force api-feature-usage to use codfw by tweaking mediawiki-config.
I'll update mediawiki-config with some comments to make this more obvious. I'm confident that this should just work, but I'm fine to run a quick test on production to be 100% sure it'll work when needed.
Comment Actions
Change 380692 had a related patch set uploaded (by Gehel; owner: Guillaume Lederrey):
[operations/puppet@production] logstash: cleanup old api feature usage indices for all clusters
Comment Actions
Change 380692 merged by Gehel:
[operations/puppet@production] logstash: cleanup old api feature usage indices for all clusters