== Discrete Steps ==
[x] Craft set of commands
[] Actually run the commands
[] (once started) monitor tmux session `tmux session not created yet, check back later`
== Commands and how I got them ==
modified reindex function from https://wikitech.wikimedia.org/wiki/Search#In_place_reindex:
```
function reindex_single_es_index() {
cluster="$1"
wiki="$2"
es_index_suffix="$3"
mkdir -p "$HOME/cirrus_log/"
reindex_log="$HOME/cirrus_log/$wiki.$cluster.reindex.log"
if [ -z "$cluster" -o -z "$wiki" -z "es_index_suffix" ]; then
echo "Usage: reindex [cluster] [wiki] [es_index_suffix]"
return 1
fi
TZ=UTC export REINDEX_START=$(date +%Y-%m-%dT%H:%m:%SZ)
echo "Started at $REINDEX_START" > "$reindex_log"
mwscript extensions/CirrusSearch/maintenance/UpdateOneSearchIndexConfig.php --wiki $wiki --cluster $cluster --indexType $es_index_suffix --reindexAndRemoveOk --indexIdentifier now 2>&1 | tee -a "$reindex_log" && \
mwscript extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki $wiki --cluster $cluster --from $REINDEX_START --deletes | tee -a "$reindex_log" && \
mwscript extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki $wiki --cluster $cluster --from $REINDEX_START --queue | tee -a "$reindex_log"
mwscript extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki $wiki --cluster $cluster --from $REINDEX_START --archive| tee -a "$reindex_log"
}
```
quick ruby script to generate all `[cluster, wiki]` permutations to reduce the chance of human error:
```
# https://phabricator.wikimedia.org/T256928 -> https://phabricator.wikimedia.org/T257387
clusters = ['eqiad', 'codfw', 'cloudelastic']
indices = ['enwiki_content', 'dewiki_content', 'commonswiki_file', 'viwiki_general']
final_output = ""
clusters.each do |cluster|
indices.each_with_index do |es_index, idx|
es_index_base_name = es_index.split("_")[0]
es_index_type = es_index.split("_")[1]
unless (idx == indices.size - 1)
final_output += "reindex_single_es_index #{cluster} #{es_index_base_name} #{es_index_type} && \\\n"
else
final_output += "reindex_single_es_index #{cluster} #{es_index_base_name} #{es_index_type}"
end
end
final_output += "\n\n"
end
puts final_output.chomp # remove extra trailing newline
```
result (after some manual randomization of the order of indices within a cluster):
```
reindex_single_es_index eqiad enwiki content && \
reindex_single_es_index eqiad dewiki content && \
reindex_single_es_index eqiad commonswiki file && \
reindex_single_es_index eqiad viwiki general
reindex_single_es_index codfw dewiki content && \
reindex_single_es_index codfw viwiki general && \
reindex_single_es_index codfw commonswiki file && \
reindex_single_es_index codfw enwiki content
reindex_single_es_index cloudelastic viwiki general && \
reindex_single_es_index cloudelastic dewiki content && \
reindex_single_es_index cloudelastic enwiki content && \
reindex_single_es_index cloudelastic commonswiki file
```
So, we'll want a single tmux session divided into 3 panes, where each pane is running one of those 3 blocks.
Since we're not reindexing the whole world, this shouldn't take as long as normal reindex of everything, but we are doing the biggest wikis so I would still expect this to take quite some time.