What is the problem?
Currently, there are a few exceptions we throw when an update removes a particular field for an IP. (i.e. the field was there in a previous JSON feed but now is no longer.)
For example, if the location field is missing:
/srv/service/output-sql.js:184
location_country: actor.location.country || '',
^
TypeError: Cannot read properties of undefined (reading 'country')
at generateInsertActorQueries (/srv/service/output-sql.js:184:39)
at generateUpdateActorQueries (/srv/service/output-sql.js:214:15)
at Interface.diff (/srv/service/output-sql.js:337:46)
at Interface.emit (node:events:513:28)
at Interface._onLine (node:readline:491:10)
at Interface._normalWrite (node:readline:665:12)
at ReadStream.ondata (node:readline:272:10)
at ReadStream.emit (node:events:513:28)
at addChunk (node:internal/streams/readable:315:12)
at readableAddChunk (node:internal/streams/readable:289:9)Or the client field is missing:
/srv/service/output-sql.js:174
client_count: actor.client.count || 0,
^
TypeError: Cannot read properties of undefined (reading 'count')
at generateInsertActorQueries (/srv/service/output-sql.js:174:33)
at generateUpdateActorQueries (/srv/service/output-sql.js:214:15)
at Interface.diff (/srv/service/output-sql.js:337:46)
at Interface.emit (node:events:513:28)
at Interface._onLine (node:readline:491:10)
at Interface._normalWrite (node:readline:665:12)
at ReadStream.ondata (node:readline:272:10)
at ReadStream.emit (node:events:513:28)
at addChunk (node:internal/streams/readable:315:12)
at readableAddChunk (node:internal/streams/readable:289:9)And possibly others.
How do we want to handle this? Supposedly, only the ip field is required and All fields not explicitely marked as required may be omitted when the value is null
Steps to reproduce problem
- Download the two files below and place them in your tmp/ directory
- docker compose up -d
- docker compose exec web ./diff.sh --yesterday tmp/before1.json.gz --today tmp/after_missing_client.json.gz
Expected behaviour: Import runs fine.
Observed behaviour: See exception TypeError: Cannot read properties of undefined (reading 'count')
Alternative example: docker compose exec web ./diff.sh --yesterday tmp/before1.json.gz --today tmp/after_missing_location.json.gz
Environment
ipoid commit
Reproduction data
Before:
After, missing client:
After, missing location: