What is the problem?
I have noticed when updating that some IPs which are not in the updated feed are not being deleted. They are being updated based on data from yesterday.
Might this be an issue with how we are diffing?
Steps to reproduce problem
Assuming docker:
- Setup the docker environment (docker compose up -d; docker compose exec web mkdir /tmp/ipoid; docker compose exec web node -e "require('./create-users.js')();")
- Copy the two json snippets in Reproduction data and save them to tmp/20240101.json.gz and tmp/20240102.json.gz
- Copy them to the ipoid docker container (docker compose exec web cp tmp/20240101.json.gz /tmp/ipoid/20240101.json.gz; docker compose exec web cp tmp/20240102.json.gz /tmp/ipoid/20240102.json.gz)
- docker compose exec web ./main.sh --init true --today 20240101 --debug true
- (Optional, for comparison later) docker compose exec db mysql -u ipoid_ro -ppassword3 -e "SELECT * FROM actor_data" ipoid
- docker compose exec web ./main.sh --yesterday 20240101 --today 20240102 --debug true
- docker compose exec db mysql -u ipoid_ro -ppassword3 -e "SELECT * FROM actor_data" ipoid
Expected behaviour: Output of final step is:
+------+---------+------+--------------+-------+-----------+------------+--------------+-----------+------------------+-------+--------------+ | pkid | ip | org | client_count | types | conc_city | conc_state | conc_country | countries | location_country | risks | last_updated | +------+---------+------+--------------+-------+-----------+------------+--------------+-----------+------------------+-------+--------------+ | 3 | 0.0.0.0 | NULL | 0 | 1 | | | | 0 | | 1 | 1709733818 | +------+---------+------+--------------+-------+-----------+------------+--------------+-----------+------------------+-------+--------------+
Observed behaviour: Output of final step is. Note that it doesn't even match the output from step 5.
+------+----------------+------+--------------+-------+-----------+------------+--------------+-----------+------------------+-------+--------------+ | pkid | ip | org | client_count | types | conc_city | conc_state | conc_country | countries | location_country | risks | last_updated | +------+----------------+------+--------------+-------+-----------+------------+--------------+-----------+------------------+-------+--------------+ | 4 | 192.52.193.244 | NULL | 0 | 1 | | | | 0 | first line | 1 | 1709722710 | +------+----------------+------+--------------+-------+-----------+------------+--------------+-----------+------------------+-------+--------------+
Environment
ipoid commit 3fc04e78cb82ac031188446aed0aa1210d1200f0
Reproduction data
20240101.json.gz
{"client": {}, "ip": "192.52.193.244", "location": {"country": "first line", "state": "BCB", "city": "BdD"}, "infrastructure": [], "as": {"number": 402208}}
{"client": {}, "ip": "192.52.193.244", "location": {"country": "second line"}, "services": ["\u00ec\u00b3$\u001c\udbf7\udd67\u00f6p\u00f8\u00c3&M\u008a\u00b1\udb9b\ude6b\u009b"]}20240102.json.gz
{"client": {"behaviors": ["_123_proxy"]}, "ip": "0.0.0.0", "location": {}}
{"client": {"behaviors": ["_123_proxy"]}, "ip": "0.0.0.0", "location": {}}