To scale the execution of the user reconstruction algorithm we have to split the input.
The most optimal split is Joseph's connected graphs algorithm, that he tested with the page history.
We should adapt the user history code to follow the same structure.
For this, it's good that the code be transformed to purely functional.
And in the end also, we might want to factor out the part of the code that computed the connected components,
so that it can be reused in both page and user history.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T120037 Vital Signs: Please provide an "all languages" de-duplicated stream for the Community/Content groups of metrics | |||
| Resolved | None | T120036 Vital Signs: Please make the data for enwiki and other big wikis less sad, and not just be missing for most days | |||
| Resolved | • odimitrijevic | T130256 Wikistats 2.0. | |||
| Resolved | mforns | T141774 User history: Adapt the user history reconstruction to use scaling by clustering |
This task was about parallelization strategy, which worked, but code is not final.
Following up will be done on a different task.