Maniphest T203482

Investigate parallelisation of harvesting job
Open, HighPublic
Actions

Assigned To

None

Authored By

	JeanFred
	Sep 4 2018, 4:43 PM

Description

The last successful harvesting job (update_database.py) took close to 10 hours, which is mental.

2018-09-04_06:33:08 Starting full monument update.
2018-09-04_06:33:08 Load changes to monuments_config...
2018-09-04_06:33:23 Recreating the source tables...
2018-09-04_06:33:37 Full source database update...
2018-09-04_17:03:21 Update monuments_all table...
2018-09-04_17:58:27 Make statistics...

Each dataset harvest being independent, we should consider parallelizing the processing.

Related Objects

Mentioned In: T203500: Merging tables into mouments_all takes a crazy amout of time

Event Timeline

JeanFred created this task.Sep 4 2018, 4:43 PM

JeanFred triaged this task as High priority.Sep 4 2018, 7:48 PM

JeanFred mentioned this in T203500: Merging tables into mouments_all takes a crazy amout of time.

JeanFred updated the task description. (Show Details)

One more data point from today’s harvesting:

2018-09-05_03:01:13 Full source database update...
2018-09-05_13:20:31 Update monuments_all table...

So the harvesting took again 10.5 hours.

This is still roughly 10.5 hours.

2019-08-26_03:01:05 Full source database update...
2019-08-26_13:45:03 Update monuments_all table...

Investigate parallelisation of harvesting jobOpen, HighPublicActions

Description

Related Objects

Event Timeline

Investigate parallelisation of harvesting job
Open, HighPublic
Actions