Page MenuHomePhabricator
Create Task
Maniphest T88280

/var/log full on deployment-elasticsearch* hosts
Closed, ResolvedPublic
Actions

Description

Is causing puppet failure, and probably also service degradation in search. Someone should clean it up and also investigate why it suddenly filled up.

Related Objects

Event Timeline

yuvipanda created this task.Feb 2 2015, 12:46 PM
yuvipanda raised the priority of this task from to Unbreak Now!.
yuvipanda updated the task description. (Show Details)
yuvipanda added a project: Beta-Cluster-Infrastructure.
yuvipanda subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 2 2015, 12:46 PM
greg added subscribers: greg, • demon.Feb 2 2015, 4:25 PM

@chad ? :)

greg added a comment.Feb 2 2015, 4:26 PM

See also: T87676: Searching for images in VisualEditor with Firefox is painfully slow in Beta Cluster

• demon added a comment.Feb 2 2015, 6:34 PM

apifeature usage started spamming the log, made it grow to almost 2G.

P254 apifeatureusage failures in beta
1[2015-01-27 20:58:45,951][WARN ][cluster.action.shard ] [deployment-elastic05] [apifeatureusage-2014.12.09][0] sending failed shard for [apifeatureusage-2014.12.09][0], node[bIp_JtFPQ5Wr2IsvksOYKA], [
2P], s[INITIALIZING], indexUUID [J3QlLykCTCq7XnLMbZgBgw], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[apifeatureusage-2014.12.09][0] failed to fetch index version after copy
3ing it over]; nested: IndexShardGatewayRecoveryException[[apifeatureusage-2014.12.09][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFound
4Exception[no segments* file found in store(least_used[rate_limited(default(mmapfs(/var/lib/elasticsearch/beta-search/nodes/0/indices/apifeatureusage-2014.12.09/0/index),niofs(/var/lib/elasticsearch/beta-s
5earch/nodes/0/indices/apifeatureusage-2014.12.09/0/index)), type=MERGE, rate=20.0)]): files: []]; ]]
6[2015-01-27 20:58:46,172][WARN ][indices.cluster ] [deployment-elastic05] [apifeatureusage-2014.11.23][0] failed to start shard
7org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [apifeatureusage-2014.11.23][0] failed to fetch index version after copying it over
8 at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:152)
9 at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
10 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
11 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
12 at java.lang.Thread.run(Thread.java:745)
13Caused by: org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [apifeatureusage-2014.11.23][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []
14 at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:131)
15 ... 4 more
16Caused by: org.apache.lucene.index.IndexNotFoundException: no segments* file found in store(least_used[rate_limited(default(mmapfs(/var/lib/elasticsearch/beta-search/nodes/0/indices/apifeatureusage-2014.11.23/0/index),niofs(/var/lib/elasticsearch/beta-search/nodes/0/indices/apifeatureusage-2014.11.23/0/index)), type=MERGE, rate=20.0)]): files: []
17 at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:870)
18 at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:758)
19 at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:453)
20 at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:98)
21 at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:122)
22 ... 4 more

• demon added a comment.Feb 2 2015, 6:37 PM

P255 Failed shards for apifeatureusage in beta
1root@deployment-elastic05:/var/log/elasticsearch# curl -s localhost:9200/_cat/shards | grep -v 'STARTED'
2apifeatureusage-2015.01.29 0 r UNASSIGNED
3apifeatureusage-2015.01.29 0 r UNASSIGNED
4apifeatureusage-2014.12.02 0 p UNASSIGNED
5apifeatureusage-2014.12.02 0 r UNASSIGNED
6apifeatureusage-2014.12.02 0 r UNASSIGNED
7apifeatureusage-2014.12.06 0 p INITIALIZING 10.68.16.38 deployment-elastic05
8apifeatureusage-2014.12.06 0 r UNASSIGNED
9apifeatureusage-2014.12.06 0 r UNASSIGNED
10apifeatureusage-2014.12.09 0 p UNASSIGNED
11apifeatureusage-2014.12.09 0 r UNASSIGNED
12apifeatureusage-2014.12.09 0 r UNASSIGNED
13apifeatureusage-2014.12.10 0 p INITIALIZING 10.68.17.82 deployment-elastic07
14apifeatureusage-2014.12.10 0 r UNASSIGNED
15apifeatureusage-2014.12.10 0 r UNASSIGNED
16apifeatureusage-2014.12.18 0 p INITIALIZING 10.68.17.82 deployment-elastic07
17apifeatureusage-2014.12.18 0 r UNASSIGNED
18apifeatureusage-2014.12.18 0 r UNASSIGNED
19apifeatureusage-2014.12.14 0 p UNASSIGNED
20apifeatureusage-2014.12.14 0 r UNASSIGNED
21apifeatureusage-2014.12.14 0 r UNASSIGNED
22apifeatureusage-2014.12.20 0 p UNASSIGNED
23apifeatureusage-2014.12.20 0 r UNASSIGNED
24apifeatureusage-2014.12.20 0 r UNASSIGNED
25apifeatureusage-2014.11.23 0 p INITIALIZING 10.68.17.82 deployment-elastic07
26apifeatureusage-2014.11.23 0 r UNASSIGNED
27apifeatureusage-2014.11.23 0 r UNASSIGNED
28apifeatureusage-2014.11.21 0 p UNASSIGNED
29apifeatureusage-2014.11.21 0 r UNASSIGNED
30apifeatureusage-2014.11.21 0 r UNASSIGNED
31apifeatureusage-2014.12.24 0 p INITIALIZING 10.68.16.38 deployment-elastic05
32apifeatureusage-2014.12.24 0 r UNASSIGNED
33apifeatureusage-2014.12.24 0 r UNASSIGNED
34apifeatureusage-2014.11.20 0 p UNASSIGNED
35apifeatureusage-2014.11.20 0 r UNASSIGNED
36apifeatureusage-2014.11.20 0 r UNASSIGNED

Which leads to red status, obvs.

hashar subscribed.Feb 3 2015, 1:54 PM

I have cleaned a bit /var/log/elasticsearch on deployment-elastic06 and deployment-elastic07 instances at 9:00am UTC

greg added a comment.Feb 4 2015, 10:22 PM

Is it worth reimaging these hosts with the new bigger var?

hashar added a project: Elasticsearch.Feb 5 2015, 9:10 AM
hashar added a subscriber: • Manybubbles.
In T88280#1016111, @greg wrote:

Is it worth reimaging these hosts with the new bigger var?

Most probably :-]

• demon closed this task as Resolved.Feb 11 2015, 9:22 PM
• demon claimed this task.

Reimaged.

greg moved this task from To Triage to Done on the Beta-Cluster-Infrastructure board.Mar 5 2015, 4:45 PM
Amire80 mentioned this in T88241: Wikipedia app user not aware of using Wikipedia Zero.Mar 10 2015, 2:12 PM
• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:55 PM
Restricted Application added projects: Discovery-ARCHIVED, Discovery-Search, Release-Engineering-Team (Kanban). · View Herald TranscriptJun 7 2017, 6:55 PM
Restricted Application added subscribers: Jay8g, TerraCodes. · View Herald Transcript
• Phabricator_maintenance edited projects, added RelEng-Archive-FY201718-Q1; removed Release-Engineering-Team (Kanban).Sep 26 2017, 11:48 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits