Change Details

The main goal is to make sure that this script can run as a stand-alone cron job. We'll also refine it a little bit, details TBD. To start, this script should contain the whole ETL pipeline (**E**xtract, **T**ransform, **L**oad). We should also make sure it's idempotent, i.e. running it twice should produce the exact same outcome as running it once, provided that the data fetched from Toolhub and the state of our db haven't changed between the two runs.

The main goal is to make sure that this script can run as a stand-alone cron job. We'll also refine it a little bit, details TBD. To start, this script should contain the whole ETL pipeline (**E**xtract, **T**ransform, **L**oad). We should also make sure it's idempotent, i.e. running it twice should produce the exact same outcome as running it once, provided that the data fetched from Toolhub and the state of our db haven't changed between the two runs. The script being stand-alone also means it shouldn't depend on anything that requires having an up-and-running Flask app instance, including app.config and the db instance.

The main goal is to make sure that this script can run as a stand-alone cron job. We'll also refine it a little bit, details TBD. To start, this script should contain the whole ETL pipeline (**E**xtract, **T**ransform, **L**oad). We should also make sure it's idempotent, i.e. running it twice should produce the exact same outcome as running it once, provided that the data fetched from Toolhub and the state of our db haven't changed between the two runs. The script being stand-alone also means it shouldn't depend on anything that requires having an up-and-running Flask app instance, including app.config and the db instance.