Values tracked during the process (please fill them when making progress)
* LEXEME_DUMP=//URL of the lexeme dump used and pre-loaded//
* ENTITY_DUMP=//URL of the entity dump used and pre-loaded//
* DUMP_START_DATE=//date at which the dump started to be generated (extracted using a hql query while bootstrapping the flink job)//
* FLINK_EQIAD_JOB_START=//timestamp of the start date of the flink job in eqiad//
* FLINK_CODFW_JOB_START= //ditto for codfw//
* Week of sept. 20:
** [x] [[https://lists.wikimedia.org/hyperkitty/list/wikidata@lists.wikimedia.org/message/V57FNNLBN4KHVTKEVKSIZHR7YK2RAEGU/|Send com about the rollout]]
** [] increase retention to 1month on //codfw.rdf-streaming-updater.mutation// (topic name is about to change) in kafka-main@codfw
** [] make sure retention is 1month on //eqiad.rdf-streaming-updater.mutation// (topic name is about to change) in kafka-main@eqiad
* Week of sept. 27:
** [] (friday oct. 1): [[ https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Streaming_Updater#First_run_(bootstrap) | bootstrap flink ]], note `DUMP_START_DATE` and `FLINK_(EQIAD|CODFW)_JOB_START`
** [] (friday oct. 1): pre-fetch the dumps to wdqs1009 and wdqs2008 and note `LEXEME_DUMP` and `ENTITY_DUMP`
** [] (friday oct. 1): start the data-reload cookbook with `--reload-data wikidata --skolemize [TODO: new options to manage kafka offsets with FLINK_EQIAD_JOB_START]` on wdqs1009
** [] (friday oct. 1): start the data-reload cookbook with `--reload-data wikidata --skolemize [TODO: new options to manage kafka offsets with FLINK_CODFW_JOB_START]` on wdqs2008
** [] (friday oct. 1): merge the [[https://gerrit.wikimedia.org/r/c/operations/puppet/+/721281 | activation of the streaming updater profile]] on wdqs2008 while the reload is happening there
* Week of oct. 4:
** Monitor that the reload is progressing properly
* Week of oct. 11:
** [] Send a quick reminder com to users
** Start data transfer (use the new option to activate kafka offsets propagation and always activate the streaming_updater profile via puppet on the target machine)
** (internal cluster)
*** [] wdqs1009 -> wdqs1003
*** [] wdqs2008 -> wdqs2005
*** [] wdqs1009 -> wdqs1008
*** [] wdqs2008 -> wdqs2006
*** [] wdqs1009 -> wdqs1011
** (external cluster)
*** [] wdqs1009 -> wdqs1004
*** [] wdqs2008 -> wdqs2001
*** [] wdqs1009 -> wdqs1005
*** [] wdqs2008 -> wdqs2002
*** [] wdqs1009 -> wdqs1006
*** [] wdqs2008 -> wdqs2003
*** [] wdqs1009 -> wdqs1007
*** [] wdqs2008 -> wdqs2004
*** [] wdqs1009 -> wdqs1012
*** [] wdqs2008 -> wdqs2007
*** [] wdqs1009 -> wdqs1013
Note: wdqs1010 is kept with the old updater and the old journal.