What is the problem?
If yesterday's import was a fresh import and had failed, incomplete or missing batches, trying to run today's update returns:
$ docker-compose exec web ./main.sh --today 20240110 --yesterday 20240109 --debug true --batchsize 100 {"log.level":"info","@timestamp":"2024-01-30T13:34:32.948Z","process.pid":1594,"host.hostname":"1ea707481097","ecs.version":"8.10.0","message":"Update init already run, skipping..."} {"log.level":"info","@timestamp":"2024-01-30T13:34:32.949Z","process.pid":1594,"host.hostname":"1ea707481097","ecs.version":"8.10.0","message":"Update remove-tunnel-anonymous-property already run, skipping..."} {"log.level":"info","@timestamp":"2024-01-30T13:34:32.950Z","process.pid":1594,"host.hostname":"1ea707481097","ecs.version":"8.10.0","message":"Database updated!"} {"log.level":"debug","@timestamp":"2024-01-30T13:34:33.324Z","process.pid":1583,"host.hostname":"1ea707481097","ecs.version":"8.10.0","message":"Found incomplete import. Attempting to restart import from null and 20240109."} {"log.level":"info","@timestamp":"2024-01-30T13:34:33.870Z","process.pid":1665,"host.hostname":"1ea707481097","ecs.version":"8.10.0","message":"Feed for 20240109 already exists, skipping attempt to download."} /srv/service/get-feed.js:54 throw new Error( 'Invalid date passed. Date should be in yyyymmdd format.' ); ^ Error: Invalid date passed. Date should be in yyyymmdd format. at Object.init (/srv/service/get-feed.js:54:9) at [eval]:1:26 at Script.runInThisContext (node:vm:129:12) at Object.runInThisContext (node:vm:313:38) at node:internal/process/execution:76:19 at [eval]-wrapper:6:22 at evalScript (node:internal/process/execution:75:60) at node:internal/main/eval_string:27:3
This does not happen when yesterday's import was itself also an update (i.e. no --init true was passed to main.sh).
Steps to reproduce problem
- Start the ipoid docker container (e.g. docker compose up -d)
- docker compose exec web node -e "require('./create-users.js')();"
- docker compose exec web mkdir /tmp/ipoid
- Copy any (valid) test data to /tmp/ipoid/20240109.json.gz, e.g. docker compose exec web cp tmp/test_file.json.gz /tmp/ipoid/20240109.json.gz
- docker compose exec web ./main.sh --init true --today 20240109 --debug true --batchsize 1
- Update the import_status table to have failed, incomplete or missing batches (e.g. docker compose exec db mysql -u root -ppassword1 -e "UPDATE import_status SET batch_status=0 WHERE batchid=1;" ipoid)
- docker compose exec web ./main.sh --today 20240110 --yesterday 20240109 --debug true --batchsize 1
Expected behaviour: It reruns the failed/incomplete/missing batches for yesterday.
Observed behaviour: Exception in the description.
Environment
ipoid commit ef4c025096861a430933f5a40a3ed794191f7f88