combine_es_indices.php
ActivePublic
Actions

Authored by EBernhardson on Jan 21 2016, 8:35 PM.

Tags

Referenced Files

	F3264919: combine_es_indices.php
	Feb 24 2016, 5:27 AM

Subscribers

None

	<?php

	$languages= array();
	while ( $line = fgets( STDIN ) ) {
	$pieces = explode( ' ', preg_replace( '/ +/', ' ', $line ) );
	$language = preg_replace( '/wik(.*)$/', 'wik', $pieces[2] );
	if ( false !== strpos( $pieces[2], 'apifeatureusage' ) ) {
	continue;
	} elseif ( false !== strpos( $pieces[2], '_content_' ) ) {
	$indexType = 'content';
	} else {
	$indexType = 'general';
	}
	$languages[$language][$indexType][$pieces[2]] = array(
	'shards' => $pieces[3],
	'replicas' => $pieces[4],
	'size' => $pieces[8],
	);
	}

	$GB = pow(2, 30);
	$totalShards = 0;
	$reduction = 0;
	echo "Checking ", count( $language ), " languages\n";
	foreach ( $languages as $language => $indexTypes ) {
	foreach ( $indexTypes as $indexType => $indices ) {
	$langShards = 0;
	$langSize = 0;
	$wikis = array();
	foreach ( $indices as $index => $data ) {
	$totalShards += $data['shards'] * (1 + $data['replicas']);
	if ( false !== strpos( $index, 'titlesuggest' ) ) {
	// titlesuggest stays 1 index per wiki
	continue;
	}
	$langShards += $data['shards'] * (1 + $data['replicas']);
	$langSize += $data['size'];
	$wikis[] = $index;
	}
	$optimalPrimaries = ceil( $langSize / ( 2 * $GB ) );
	if ( $optimalPrimaries > 1 ) {
	// if more than 1 primary shards, not worth it
	continue;
	}
	// assumes 1 primary and 2 replicas
	$optimalShards = $optimalPrimaries * 3;
	if ( $optimalShards >= $langShards ) {
	continue;
	}
	$langSizeGB = number_Format( $langSize / $GB, 2);
	echo implode( ', ', $wikis ), " could be reduced from $langShards to $optimalShards shards for a size of $langSizeGB GB\n";
	$reduction += $langShards - $optimalShards;
	}
	}


	$percentDiff = number_format( 100 * $reduction / $totalShards, 1 );
	echo "\nCould reduce shard count across cluster by $reduction ({$percentDiff}%)\n";

Event Timeline

EBernhardson changed the title of this paste from untitled to combine_es_indices.php.Jan 21 2016, 8:35 PM

EBernhardson edited the content of this paste. (Show Details)

EBernhardson added a project: CirrusSearch.

EBernhardson edited the content of this paste. (Show Details)Jan 21 2016, 8:44 PM

EBernhardson mentioned this in T124332: Evaluate reducing shard counts for smaller wikis.

combine_es_indices.phpActivePublicActions

Event Timeline

combine_es_indices.php
ActivePublic
Actions