Page MenuHomePhabricator
Create Task
Maniphest T143226

Cluster-wide major compactions: parsoid.html table
Closed, ResolvedPublic
Actions

Description

The droppable tombstone ratio continues to be very high, and thus far, strategies for user-defined compactions have failed to move the needle much. Major compactions, while not a viable long-term strategy for this problem, do however result in a significant reduction to the droppable ratio, as well as reclaiming significant disk space (and it is something we can begin doing now).

  • eqiad
    • rack a
      • restbase1007.eqiad.wmnet
        • a (0.748) (0.082)
        • b (0.023)
        • c
      • restbase1010.eqiad.wmnet
        • a (0.840) (0.085)
        • b (0.024)
        • c
      • restbase1011.eqiad.wmnet
        • a (0.844) (0.079)
        • b (0.025)
        • c
    • rack b
      • restbase1008.eqiad.wmnet
        • a (0.837) (0.084)
        • b (0.025)
        • c
      • restbase1012.eqiad.wmnet
        • a (0.807) (0.082)
        • b (0.025)
        • c
      • restbase1013.eqiad.wmnet
        • a (0.806) (0.084)
        • b (0.025)
        • c
    • rack d
      • restbase1009.eqiad.wmnet
        • a (0.818) (0.028)
        • b
        • c (0.053)
      • restbase1014.eqiad.wmnet
        • a (0.833) (0.023)
        • b
        • c (0.056)
      • restbase1015.eqiad.wmnet
        • a (0.807) (0.026)
        • b
        • c
  • codfw
    • rack b
      • restbase2001.codfw.wmnet
        • a
        • b (0.023)
        • c
      • restbase2002.codfw.wmnet
        • a (0.055)
        • b (0.023)
        • c (0.019)
      • restbase2007.codfw.wmnet
        • a (0.156)
        • b (0.022)
        • c (0.018)
    • rack c
      • restbase2003.codfw.wmnet
        • a (0.091)
        • b (0.021)
        • c (0.017)
      • restbase2004.codfw.wmnet
        • a (0.081)
        • b (0.047)
        • c
      • restbase2008.codfw.wmnet
        • a (0.662)
        • b (0.021)
        • c (0.018)
    • rack d
      • restbase2005.codfw.wmnet
        • a (0.109)
        • b (0.021)
        • c (0.037)
      • restbase2006.codfw.wmnet
        • a (0.131)
        • b (0.026)
        • c (0.017)
      • restbase2009.codfw.wmnet
        • a (0.071)
        • b (0.021)
        • c
NOTE: These compactions will all be split output (nodetool compact -s), resulting in 4 SSTable files (one each of 50% and 25%, and two each of 12.5%)
NOTE: In an effort to limit compaction throughput, only one compaction will be run on a host at a time

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Eevans updated the task description. (Show Details)Aug 25 2016, 8:38 PM
Eevans updated the task description. (Show Details)Aug 25 2016, 8:42 PM
Eevans updated the task description. (Show Details)
Stashbot added a comment.Aug 29 2016, 9:11 PM

Mentioned in SAL [2016-08-29T21:11:41Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1007-b.eqiad.wmnet

Eevans updated the task description. (Show Details)Aug 29 2016, 9:12 PM
Eevans updated the task description. (Show Details)
Eevans updated the task description. (Show Details)Aug 29 2016, 9:15 PM
Eevans updated the task description. (Show Details)
Eevans updated the task description. (Show Details)
Eevans updated the task description. (Show Details)Aug 29 2016, 9:18 PM
Stashbot added a comment.Aug 29 2016, 9:29 PM

Mentioned in SAL [2016-08-29T21:29:02Z] <urandom> T143226: Stopping restbase1015-a.eqiad.wmnet to remove repairedAt attribute on a table

Stashbot added a comment.Aug 29 2016, 9:31 PM

Mentioned in SAL [2016-08-29T21:31:34Z] <urandom> T143226: Starting restbase1015-a.eqiad.wmnet

Stashbot added a comment.Aug 29 2016, 9:32 PM

Mentioned in SAL [2016-08-29T21:32:47Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1015-a.eqiad.wmnet

Eevans updated the task description. (Show Details)Aug 29 2016, 9:34 PM
Eevans updated the task description. (Show Details)Aug 31 2016, 3:12 PM
Eevans added a comment.Aug 31 2016, 9:25 PM

An experiment with 1015-a turned up a singular file containing a repairedAt field (the remnants of a long abandoned experiment w/ incremental repair). This was enough to split compaction into two pools, and even though the repaired pool only contained one table, enough overlap existed to prevent the GC'ing of most droppable tombstones. A major compaction of N-1 tables yielded a ratio of .81, a second major compaction of N tables (i.e. after removing repairedAt) yielded a droppable ratio of .03.

Before proceeding with these major compactions, we will need to similarly mark all tables unrepaired as necessary. Here is a list, current as of this writing:

P3956 Masterwork From Distant Lands
1restbase1007.eqiad.wmnet: a: /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-299361-big (1426826771017)
2restbase1007.eqiad.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-91574-big (1426826771017)
3restbase1007.eqiad.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-30026-big (1426826771017)
4restbase1007.eqiad.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-217 (1426826771017)
5restbase1010.eqiad.wmnet: a: /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-69714-big (1426826771017)
6restbase1010.eqiad.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-55032-big (1426826771017)
7restbase1010.eqiad.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-55034-big (1426826771017)
8restbase1010.eqiad.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-32168-big (1426826771017)
9restbase1010.eqiad.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-305 (1426826771017)
10restbase1010.eqiad.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-32171-big (1426826771017)
11restbase1011.eqiad.wmnet: a: /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-72980-big (1426826771017)
12restbase1011.eqiad.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-62554-big (1426826771017)
13restbase1011.eqiad.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-62548-big (1426826771017)
14restbase1011.eqiad.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-502 (1426826771017)
15restbase1008.eqiad.wmnet: a: /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-128999-big (1426826771017)
16restbase1008.eqiad.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-103081-big (1426826771017)
17restbase1008.eqiad.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-33078-big (1426826771017)
18restbase1008.eqiad.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-414 (1426826771017)
19restbase1012.eqiad.wmnet: a: /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-69017-big (1426826771017)
20restbase1012.eqiad.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-67115-big (1426826771017)
21restbase1012.eqiad.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-67091-big (1426826771017)
22restbase1012.eqiad.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-37413-big (1426826771017)
23restbase1012.eqiad.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-37402-big (1426826771017)
24restbase1013.eqiad.wmnet: a: /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-67845-big (1426826771017)
25restbase1013.eqiad.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-65741-big (1426826771017)
26restbase1013.eqiad.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-487-big (1426826771017)
27restbase1013.eqiad.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-489-big (1426826771017)
28restbase1009.eqiad.wmnet: a: /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-56614-big (1426826771017)
29restbase1009.eqiad.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-138525-big (1426826771017)
30restbase1009.eqiad.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-936-big (1426826771017)
31restbase1009.eqiad.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-9641-big (1426826771017)
32restbase1014.eqiad.wmnet: a: /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-62122-big (1426826771017)
33restbase1014.eqiad.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-66835-big (1426826771017)
34restbase1014.eqiad.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-66860-big (1426826771017)
35restbase1014.eqiad.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-751-big (1426826771017)
36restbase1014.eqiad.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-9620-big (1426826771017)
37restbase1015.eqiad.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-51271-big (1426826771017)
38restbase1015.eqiad.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-51328-big (1426826771017)
39restbase1015.eqiad.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-51272-big (1426826771017)
40restbase1015.eqiad.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-669-big (1426826771017)
41restbase1015.eqiad.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-674-big (1426826771017)
42restbase2003.codfw.wmnet: a: /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-497354-big (1426826771017)
43restbase2003.codfw.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-203 (1426826771017)
44restbase2003.codfw.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-460-big (1426826771017)
45restbase2004.codfw.wmnet: a: /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-90154-big (1426826771017)
46restbase2004.codfw.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-28321-big (1426826771017)
47restbase2004.codfw.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-23739-big (1426826771017)
48restbase2004.codfw.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-243-big (1426826771017)
49restbase2004.codfw.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-373-big (1426826771017)
50restbase2008.codfw.wmnet: a: /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-45226-big (1426826771017)
51restbase2008.codfw.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-38830-big (1426826771017)
52restbase2008.codfw.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-399-big (1426826771017)
53restbase2008.codfw.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-673-big (1426826771017)
54restbase2001.codfw.wmnet: a: /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-150896-big (1426826771017)
55restbase2001.codfw.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-125118 (1426826771017)
56restbase2001.codfw.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-125135 (1426826771017)
57restbase2001.codfw.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-89084 (1426826771017)
58restbase2001.codfw.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-82112 (1426826771017)
59restbase2002.codfw.wmnet: a: /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-107116-big (1426826771017)
60restbase2002.codfw.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-71816 (1426826771017)
61restbase2002.codfw.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-151959 (1426826771017)
62restbase2002.codfw.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-118293-big (1426826771017)
63restbase2002.codfw.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-48331 (1426826771017)
64restbase2007.codfw.wmnet: a: /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-27836-big (1426826771017)
65restbase2007.codfw.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-570 (1426826771017)
66restbase2007.codfw.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-224 (1426826771017)
67restbase2005.codfw.wmnet: a: /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-464472-big (1426826771017)
68restbase2005.codfw.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/local_group_wikipedia_T_parsoid_html-data-ka-732 (1426826771017)
69restbase2006.codfw.wmnet: a: /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-480385-big (1426826771017)
70restbase2006.codfw.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-31290-big (1426826771017)
71restbase2006.codfw.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-687-big (1426826771017)
72restbase2006.codfw.wmnet: c: /srv/cassandra-c/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-505-big (1426826771017)
73restbase2009.codfw.wmnet: a: /srv/cassandra-a/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-45116-big (1426826771017)
74restbase2009.codfw.wmnet: b: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-40804-big (1426826771017)

Stashbot added a comment.Aug 31 2016, 9:27 PM

Mentioned in SAL [2016-08-31T21:27:52Z] <urandom> T143226: Stopping restbase2001-a.codfw.wmnet to clear tables marked repaired

Stashbot added a comment.Aug 31 2016, 9:38 PM

Mentioned in SAL [2016-08-31T21:38:01Z] <urandom> T143226: Stopping restbase2001-{a,b} to clear tables marked repaired

Stashbot added a comment.Aug 31 2016, 9:39 PM

Mentioned in SAL [2016-08-31T21:39:53Z] <urandom> T143226: Starting restbase2001-{b,c}.codfw.wmnet

Stashbot added a comment.Aug 31 2016, 9:44 PM

Mentioned in SAL [2016-08-31T21:44:45Z] <urandom> T143226: Clearing repair status: restbase2002.codfw.wmnet

Stashbot added a comment.Aug 31 2016, 9:46 PM

Mentioned in SAL [2016-08-31T21:46:02Z] <urandom> T143226: Clearing repair status: restbase2007.codfw.wmnet

Stashbot added a comment.Aug 31 2016, 9:51 PM

Mentioned in SAL [2016-08-31T21:51:59Z] <urandom> T143226: Clearing repair status: restbase2003.codfw.wmnet

Stashbot added a comment.Aug 31 2016, 10:01 PM

Mentioned in SAL [2016-08-31T22:01:38Z] <urandom> T143226: Clearing repair status: restbase2004.codfw.wmnet

Stashbot added a comment.Aug 31 2016, 10:12 PM

Mentioned in SAL [2016-08-31T22:12:27Z] <urandom> T143226: Clearing repair status: restbase2008.codfw.wmnet

Stashbot added a comment.Aug 31 2016, 11:20 PM

Mentioned in SAL [2016-08-31T23:20:15Z] <urandom> T143226: Clearing repair status: codfw, rack 'd'

Stashbot added a comment.Aug 31 2016, 11:34 PM

Mentioned in SAL [2016-08-31T23:34:01Z] <urandom> T143226: Clearing repair status: eqiad, rack 'a' nodes

Stashbot added a comment.Sep 1 2016, 7:39 PM

Mentioned in SAL [2016-09-01T19:39:09Z] <urandom> T143226: Clearing repair status restbase1011-c.eqiad.wmnet

Stashbot added a comment.Sep 1 2016, 7:44 PM

Mentioned in SAL [2016-09-01T19:44:03Z] <urandom> T143226: Clearing repair status: eqiad, rack 'b' nodes

Stashbot added a comment.Sep 1 2016, 8:35 PM

Mentioned in SAL [2016-09-01T20:35:34Z] <urandom> T143226: Clearing repair status: eqiad, rack 'dd' nodes

Stashbot added a comment.Sep 1 2016, 8:35 PM

Mentioned in SAL [2016-09-01T20:35:38Z] <urandom> T143226: Clearing repair status: eqiad, rack 'd' nodes

Stashbot added a comment.Sep 1 2016, 9:21 PM

Mentioned in SAL [2016-09-01T21:21:26Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1007-a

Stashbot added a comment.Sep 1 2016, 9:21 PM

Mentioned in SAL [2016-09-01T21:21:58Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1010-a

Stashbot added a comment.Sep 1 2016, 9:22 PM

Mentioned in SAL [2016-09-01T21:22:23Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1011-a

Eevans updated the task description. (Show Details)Sep 1 2016, 9:23 PM
Stashbot added a comment.Sep 1 2016, 9:24 PM

Mentioned in SAL [2016-09-01T21:24:55Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1008-a

Stashbot added a comment.Sep 1 2016, 9:25 PM

Mentioned in SAL [2016-09-01T21:25:26Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1012-a

Stashbot added a comment.Sep 1 2016, 9:25 PM

Mentioned in SAL [2016-09-01T21:25:40Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1013-a

Eevans updated the task description. (Show Details)Sep 1 2016, 9:26 PM
• GWicke subscribed.Sep 1 2016, 9:33 PM

See related discussion at T144431.

Eevans updated the task description. (Show Details)Sep 6 2016, 4:06 PM
Eevans updated the task description. (Show Details)
Eevans updated the task description. (Show Details)Sep 6 2016, 4:08 PM
Eevans updated the task description. (Show Details)
Eevans updated the task description. (Show Details)Sep 6 2016, 4:10 PM
Eevans updated the task description. (Show Details)
Eevans renamed this task from Cluster-wide major compactions: parsoid-html to Cluster-wide major compactions: parsoid.html table.Sep 8 2016, 5:37 PM
Stashbot added a comment.Sep 8 2016, 5:41 PM

Mentioned in SAL [2016-09-08T17:41:13Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1009-a.eqiad.wmnet

Stashbot added a comment.Sep 8 2016, 5:41 PM

Mentioned in SAL [2016-09-08T17:41:23Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1014-a.eqiad.wmnet

Stashbot added a comment.Sep 8 2016, 5:41 PM

Mentioned in SAL [2016-09-08T17:41:32Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1015-a.eqiad.wmnet

Eevans updated the task description. (Show Details)Sep 8 2016, 5:43 PM
Stashbot added a comment.Sep 8 2016, 5:45 PM

Mentioned in SAL [2016-09-08T17:45:29Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1007-b.eqiad.wmnet

Stashbot added a comment.Sep 8 2016, 5:45 PM

Mentioned in SAL [2016-09-08T17:45:39Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1010-b.eqiad.wmnet

Stashbot added a comment.Sep 8 2016, 5:45 PM

Mentioned in SAL [2016-09-08T17:45:49Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1011-b.eqiad.wmnet

Eevans updated the task description. (Show Details)Sep 8 2016, 5:50 PM
Stashbot added a comment.Sep 8 2016, 5:53 PM

Mentioned in SAL [2016-09-08T17:53:12Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1008-b.eqiad.wmnet

Stashbot added a comment.Sep 8 2016, 5:53 PM

Mentioned in SAL [2016-09-08T17:53:24Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1012-b.eqiad.wmnet

Stashbot added a comment.Sep 8 2016, 5:53 PM

Mentioned in SAL [2016-09-08T17:53:35Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1013-b.eqiad.wmnet

Eevans updated the task description. (Show Details)Sep 8 2016, 5:55 PM
Stashbot added a comment.Sep 8 2016, 5:58 PM

Mentioned in SAL [2016-09-08T17:58:52Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase200[1-9]-b.codfw.wmnet

Eevans updated the task description. (Show Details)Sep 8 2016, 6:01 PM
Eevans updated the task description. (Show Details)Sep 12 2016, 4:10 PM
Eevans updated the task description. (Show Details)Sep 12 2016, 4:13 PM
Eevans updated the task description. (Show Details)
Eevans updated the task description. (Show Details)Sep 12 2016, 4:17 PM
Eevans updated the task description. (Show Details)
Eevans updated the task description. (Show Details)Sep 12 2016, 4:20 PM
Stashbot added a comment.Sep 12 2016, 7:42 PM

Mentioned in SAL [2016-09-12T19:42:34Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1007-c.eqiad.wmnet

Stashbot added a comment.Sep 12 2016, 7:42 PM

Mentioned in SAL [2016-09-12T19:42:46Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1010-c.eqiad.wmnet

Stashbot added a comment.Sep 12 2016, 7:42 PM

Mentioned in SAL [2016-09-12T19:42:57Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1011-c.eqiad.wmnet

Eevans updated the task description. (Show Details)Sep 12 2016, 7:43 PM
Stashbot added a comment.Sep 12 2016, 7:44 PM

Mentioned in SAL [2016-09-12T19:44:14Z] <urandom> !log T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1008-c.eqiad.wmnet

Stashbot added a comment.Sep 12 2016, 7:44 PM

Mentioned in SAL [2016-09-12T19:44:27Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1012-c.eqiad.wmnet

Stashbot added a comment.Sep 12 2016, 7:44 PM

Mentioned in SAL [2016-09-12T19:44:42Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1013-c.eqiad.wmnet

Eevans updated the task description. (Show Details)Sep 12 2016, 7:47 PM
Stashbot added a comment.Sep 12 2016, 8:28 PM

Mentioned in SAL [2016-09-12T20:28:18Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1009-b.eqiad.wmnet

Stashbot added a comment.Sep 12 2016, 8:28 PM

Mentioned in SAL [2016-09-12T20:28:31Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1014-b.eqiad.wmnet

Stashbot added a comment.Sep 12 2016, 8:28 PM

Mentioned in SAL [2016-09-12T20:28:42Z] <urandom> T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1015-b.eqiad.wmnet

Eevans updated the task description. (Show Details)Sep 12 2016, 8:30 PM
Eevans updated the task description. (Show Details)Sep 12 2016, 8:38 PM
Eevans updated the task description. (Show Details)Sep 14 2016, 3:25 PM
Eevans updated the task description. (Show Details)Sep 14 2016, 3:38 PM
Eevans updated the task description. (Show Details)
Eevans updated the task description. (Show Details)Sep 19 2016, 7:58 PM
Eevans updated the task description. (Show Details)
Eevans updated the task description. (Show Details)Sep 19 2016, 8:00 PM
Eevans updated the task description. (Show Details)
Eevans updated the task description. (Show Details)Sep 19 2016, 8:03 PM
Eevans updated the task description. (Show Details)
Eevans updated the task description. (Show Details)Sep 19 2016, 8:08 PM
Eevans updated the task description. (Show Details)Sep 22 2016, 8:34 PM
Eevans updated the task description. (Show Details)Sep 23 2016, 4:55 PM
Eevans updated the task description. (Show Details)Sep 26 2016, 4:25 PM
Eevans updated the task description. (Show Details)
Eevans closed this task as Resolved.Sep 26 2016, 4:36 PM

This round of compaction is complete.

Current droppable tombstone ratios for local_group_wikipedia_T_parsoid_html.data:

P4116 Masterwork From Distant Lands
1restbase1007.eqiad.wmnet: a: 0.30991881976778024
2restbase1007.eqiad.wmnet: b: 0.25877332921791363
3restbase1007.eqiad.wmnet: c: 0.2677108314033092
4restbase1010.eqiad.wmnet: a: 0.3042088722805095
5restbase1010.eqiad.wmnet: b: 0.25573878683248297
6restbase1010.eqiad.wmnet: c: 0.2342874102244836
7restbase1011.eqiad.wmnet: a: 0.30652365720812597
8restbase1011.eqiad.wmnet: b: 0.2782976199188483
9restbase1011.eqiad.wmnet: c: 0.27559170157574703
10restbase1008.eqiad.wmnet: a: 0.31471951874294884
11restbase1008.eqiad.wmnet: b: 0.2655710986209403
12restbase1008.eqiad.wmnet: c: 0.24817847781273525
13restbase1012.eqiad.wmnet: a: 0.3300407373520914
14restbase1012.eqiad.wmnet: b: 0.281872256519447
15restbase1012.eqiad.wmnet: c: 0.27665845574883724
16restbase1013.eqiad.wmnet: a: 0.33136872277697776
17restbase1013.eqiad.wmnet: b: 0.25943894805497125
18restbase1013.eqiad.wmnet: c: 0.2451301798208737
19restbase1009.eqiad.wmnet: a: 0.29164376796593594
20restbase1009.eqiad.wmnet: b: 0.24381887239116942
21restbase1009.eqiad.wmnet: c: 0.12227580043291576
22restbase1014.eqiad.wmnet: a: 0.24537943437798612
23restbase1014.eqiad.wmnet: b: 0.23346094825528113
24restbase1014.eqiad.wmnet: c: 0.13962206253774045
25restbase1015.eqiad.wmnet: a: 0.28903367169687794
26restbase1015.eqiad.wmnet: b: 0.24196675450904379
27restbase1015.eqiad.wmnet: c: 0.13941667589609882
28restbase2003.codfw.wmnet: a: 0.4860393212916763
29restbase2003.codfw.wmnet: b: 0.08636304553444514
30restbase2003.codfw.wmnet: c: 0.23405026797453424
31restbase2004.codfw.wmnet: a: 0.4220764781163524
32restbase2004.codfw.wmnet: b: 0.08509114315977814
33restbase2004.codfw.wmnet: c: 0.29639358622604484
34restbase2008.codfw.wmnet: a: 0.6926602605164239
35restbase2008.codfw.wmnet: b: 0.10880490389755908
36restbase2008.codfw.wmnet: c: 0.24027911324925588
37restbase2001.codfw.wmnet: a: 0.6167176356379133
38restbase2001.codfw.wmnet: b: 0.07819831516913914
39restbase2001.codfw.wmnet: c: 0.24383676002033464
40restbase2002.codfw.wmnet: a: 0.43285064988754357
41restbase2002.codfw.wmnet: b: 0.11277315119378999
42restbase2002.codfw.wmnet: c: 0.2632767114602838
43restbase2007.codfw.wmnet: a: 0.4668181771244282
44restbase2007.codfw.wmnet: b: 0.08681717740626473
45restbase2007.codfw.wmnet: c: 0.24564651602245716
46restbase2005.codfw.wmnet: a: 0.46631750386314585
47restbase2005.codfw.wmnet: b: 0.10824687809403657
48restbase2005.codfw.wmnet: c: 0.23084065112274027
49restbase2006.codfw.wmnet: a: 0.44700621046701094
50restbase2006.codfw.wmnet: b: 0.07773689723489159
51restbase2006.codfw.wmnet: c: 0.2553871054929549
52restbase2009.codfw.wmnet: a: 0.3839533037198856
53restbase2009.codfw.wmnet: b: 0.08148755329036476
54restbase2009.codfw.wmnet: c: 0.23717469207219236

Eevans mentioned this in T146211: Cluster-wide major compactions: parsoid.data-parsoid table .Sep 26 2016, 7:09 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL