Change Details

Values tracked during the process (please fill them when making progress) * LEXEME_DUMP=https://dumps.wikimedia.your.org/other/wikibase/wikidatawiki/20210924/wikidata-20210924-lexemes-BETA.ttl.bz2 * ENTITY_DUMP=https://dumps.wikimedia.your.org/other/wikibase/wikidatawiki/20210927/wikidata-20210927-all-BETA.ttl.bz2 * DUMP_START_DATE=`2021-09-24T23:00:01Z` * FLINK_EQIAD_JOB_START= `2021-10-01T10:22:25` * FLINK_CODFW_JOB_START= `2021-10-01T10:17:02` * BOOTSTRAP_STATE_EQIAD=swift://rdf-streaming-updater-eqiad.thanos-swift/wikidata/savepoints/bootstrap_20210927 * BOOTSTRAP_STATE_CODFW=swift://rdf-streaming-updater-codfw.thanos-swift/wikidata/savepoints/bootstrap_20210927 * Week of sept. 20: ** [x] [[https://lists.wikimedia.org/hyperkitty/list/wikidata@lists.wikimedia.org/message/V57FNNLBN4KHVTKEVKSIZHR7YK2RAEGU/|Send com about the rollout]] ** [] increase retention to 1month on //codfw.rdf-streaming-updater.mutation// (topic name is about to change) in kafka-main@codfw ** [] make sure retention is 1month on //eqiad.rdf-streaming-updater.mutation// (topic name is about to change) in kafka-main@eqiad * Week of sept. 27: ** [x] (friday oct. 1): [[ https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Streaming_Updater#First_run_(bootstrap) | bootstrap flink ]], note `DUMP_START_DATE` and `FLINK_(EQIAD|CODFW)_JOB_START` ** [x] (friday oct. 1): pre-fetch the dumps to wdqs1009 and wdqs2008 and note `LEXEME_DUMP` and `ENTITY_DUMP` ** [x] (friday oct. 1): start the data-reload cookbook with `--reload-data wikidata --skolemize [TODO: new options to manage kafka offsets with FLINK_EQIAD_JOB_START]` on wdqs1009 ** [x] (friday oct. 1): start the data-reload cookbook with `--reload-data wikidata --skolemize [TODO: new options to manage kafka offsets with FLINK_CODFW_JOB_START]` on wdqs2008 ** [x] (friday oct. 1): merge the [[https://gerrit.wikimedia.org/r/c/operations/puppet/+/721281 | activation of the streaming updater profile]] on wdqs2008 while the reload is happening there * Week of oct. 4: ** Monitor that the reload is progressing properly * Week of oct. 11: ** [x] Send a quick reminder com to users ** Start data transfer (use the new option to activate kafka offsets propagation and always activate the streaming_updater profile via puppet on the target machine) ** (internal cluster) *** [x] wdqs1009 -> wdqs1003 *** [x] wdqs2008 -> wdqs2005 *** [x] wdqs1003 -> wdqs1008 *** [x] wdqs2008 -> wdqs2006 *** [...] wdqs1003 -> wdqs1011 ** (external cluster) *** [] wdqs1003 -> wdqs1004 *** [x] wdqs2008 -> wdqs2001 *** [] wdqs1003 -> wdqs1005 *** [x] wdqs2008 -> wdqs2002 *** [] wdqs1003 -> wdqs1006 *** [x] wdqs2008 -> wdqs2003 *** [] wdqs1003 -> wdqs1007 *** [x] wdqs2008 -> wdqs2004 *** [] wdqs1003 -> wdqs1012 *** [x] wdqs2008 -> wdqs2007 *** [] wdqs1003 -> wdqs1013 Note: wdqs1010 is kept with the old updater and the old journal.

Values tracked during the process (please fill them when making progress) * LEXEME_DUMP=https://dumps.wikimedia.your.org/other/wikibase/wikidatawiki/20210924/wikidata-20210924-lexemes-BETA.ttl.bz2 * ENTITY_DUMP=https://dumps.wikimedia.your.org/other/wikibase/wikidatawiki/20210927/wikidata-20210927-all-BETA.ttl.bz2 * DUMP_START_DATE=`2021-09-24T23:00:01Z` * FLINK_EQIAD_JOB_START= `2021-10-01T10:22:25` * FLINK_CODFW_JOB_START= `2021-10-01T10:17:02` * BOOTSTRAP_STATE_EQIAD=swift://rdf-streaming-updater-eqiad.thanos-swift/wikidata/savepoints/bootstrap_20210927 * BOOTSTRAP_STATE_CODFW=swift://rdf-streaming-updater-codfw.thanos-swift/wikidata/savepoints/bootstrap_20210927 * Week of sept. 20: ** [x] [[https://lists.wikimedia.org/hyperkitty/list/wikidata@lists.wikimedia.org/message/V57FNNLBN4KHVTKEVKSIZHR7YK2RAEGU/|Send com about the rollout]] ** [] increase retention to 1month on //codfw.rdf-streaming-updater.mutation// (topic name is about to change) in kafka-main@codfw ** [] make sure retention is 1month on //eqiad.rdf-streaming-updater.mutation// (topic name is about to change) in kafka-main@eqiad * Week of sept. 27: ** [x] (friday oct. 1): [[ https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Streaming_Updater#First_run_(bootstrap) | bootstrap flink ]], note `DUMP_START_DATE` and `FLINK_(EQIAD|CODFW)_JOB_START` ** [x] (friday oct. 1): pre-fetch the dumps to wdqs1009 and wdqs2008 and note `LEXEME_DUMP` and `ENTITY_DUMP` ** [x] (friday oct. 1): start the data-reload cookbook with `--reload-data wikidata --skolemize [TODO: new options to manage kafka offsets with FLINK_EQIAD_JOB_START]` on wdqs1009 ** [x] (friday oct. 1): start the data-reload cookbook with `--reload-data wikidata --skolemize [TODO: new options to manage kafka offsets with FLINK_CODFW_JOB_START]` on wdqs2008 ** [x] (friday oct. 1): merge the [[https://gerrit.wikimedia.org/r/c/operations/puppet/+/721281 | activation of the streaming updater profile]] on wdqs2008 while the reload is happening there * Week of oct. 4: ** Monitor that the reload is progressing properly * Week of oct. 11: ** [x] Send a quick reminder com to users ** Start data transfer (use the new option to activate kafka offsets propagation and always activate the streaming_updater profile via puppet on the target machine) ** (internal cluster) *** [x] wdqs1009 -> wdqs1003 *** [x] wdqs2008 -> wdqs2005 *** [x] wdqs1003 -> wdqs1008 *** [x] wdqs2008 -> wdqs2006 *** [x] wdqs1003 -> wdqs1011 ** (external cluster) *** [] (IN PROGRESS) wdqs1003 -> wdqs1004 *** [x] wdqs2008 -> wdqs2001 *** [] wdqs1003 -> wdqs1005 *** [x] wdqs2008 -> wdqs2002 *** [] wdqs1003 -> wdqs1006 *** [x] wdqs2008 -> wdqs2003 *** [] wdqs1003 -> wdqs1007 *** [x] wdqs2008 -> wdqs2004 *** [] wdqs1003 -> wdqs1012 *** [x] wdqs2008 -> wdqs2007 *** [] wdqs1003 -> wdqs1013 Note: wdqs1010 is kept with the old updater and the old journal.

Values tracked during the process (please fill them when making progress) * LEXEME_DUMP=https://dumps.wikimedia.your.org/other/wikibase/wikidatawiki/20210924/wikidata-20210924-lexemes-BETA.ttl.bz2 * ENTITY_DUMP=https://dumps.wikimedia.your.org/other/wikibase/wikidatawiki/20210927/wikidata-20210927-all-BETA.ttl.bz2 * DUMP_START_DATE=`2021-09-24T23:00:01Z` * FLINK_EQIAD_JOB_START= `2021-10-01T10:22:25` * FLINK_CODFW_JOB_START= `2021-10-01T10:17:02` * BOOTSTRAP_STATE_EQIAD=swift://rdf-streaming-updater-eqiad.thanos-swift/wikidata/savepoints/bootstrap_20210927 * BOOTSTRAP_STATE_CODFW=swift://rdf-streaming-updater-codfw.thanos-swift/wikidata/savepoints/bootstrap_20210927 * Week of sept. 20: ** [x] [[https://lists.wikimedia.org/hyperkitty/list/wikidata@lists.wikimedia.org/message/V57FNNLBN4KHVTKEVKSIZHR7YK2RAEGU/|Send com about the rollout]] ** [] increase retention to 1month on //codfw.rdf-streaming-updater.mutation// (topic name is about to change) in kafka-main@codfw ** [] make sure retention is 1month on //eqiad.rdf-streaming-updater.mutation// (topic name is about to change) in kafka-main@eqiad * Week of sept. 27: ** [x] (friday oct. 1): [[ https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Streaming_Updater#First_run_(bootstrap) | bootstrap flink ]], note `DUMP_START_DATE` and `FLINK_(EQIAD|CODFW)_JOB_START` ** [x] (friday oct. 1): pre-fetch the dumps to wdqs1009 and wdqs2008 and note `LEXEME_DUMP` and `ENTITY_DUMP` ** [x] (friday oct. 1): start the data-reload cookbook with `--reload-data wikidata --skolemize [TODO: new options to manage kafka offsets with FLINK_EQIAD_JOB_START]` on wdqs1009 ** [x] (friday oct. 1): start the data-reload cookbook with `--reload-data wikidata --skolemize [TODO: new options to manage kafka offsets with FLINK_CODFW_JOB_START]` on wdqs2008 ** [x] (friday oct. 1): merge the [[https://gerrit.wikimedia.org/r/c/operations/puppet/+/721281 | activation of the streaming updater profile]] on wdqs2008 while the reload is happening there * Week of oct. 4: ** Monitor that the reload is progressing properly * Week of oct. 11: ** [x] Send a quick reminder com to users ** Start data transfer (use the new option to activate kafka offsets propagation and always activate the streaming_updater profile via puppet on the target machine) ** (internal cluster) *** [x] wdqs1009 -> wdqs1003 *** [x] wdqs2008 -> wdqs2005 *** [x] wdqs1003 -> wdqs1008 *** [x] wdqs2008 -> wdqs2006 *** [...x] wdqs1003 -> wdqs1011 ** (external cluster) *** [] (IN PROGRESS) wdqs1003 -> wdqs1004 *** [x] wdqs2008 -> wdqs2001 *** [] wdqs1003 -> wdqs1005 *** [x] wdqs2008 -> wdqs2002 *** [] wdqs1003 -> wdqs1006 *** [x] wdqs2008 -> wdqs2003 *** [] wdqs1003 -> wdqs1007 *** [x] wdqs2008 -> wdqs2004 *** [] wdqs1003 -> wdqs1012 *** [x] wdqs2008 -> wdqs2007 *** [] wdqs1003 -> wdqs1013 Note: wdqs1010 is kept with the old updater and the old journal.