What is the problem?
When an entry in the JSON feed has no types or risks or they are empty lists, it records them as UNKNOWN.
If they are values that the import script does not recognise, it records them as an empty string.
For consistency and to save future confusion, we should probably always record invalid values or values it does not recognise as UNKNOWN.
Steps to reproduce problem
- Save the JSON from "Reproduction data" below as a .gz file (e.g. reprod.json.gz) into the ipoid/tmp directory
- If necessary, start up docker in the ipoid directory (e.g. docker compose up -d)
- Initialise the database: docker compose exec web node init-db.js
- Run this command: docker compose exec web node import-data.js ./tmp/reprod.json.gz
- Access the database (e.g. docker compose exec db mysql test -u root -p)
- Run the query: SELECT ip,risks,types FROM actor_data;
Expected behavior: Each row and column has UNKNOWN
Observed behavior: Output is:
SELECT ip,risks,types FROM actor_data; +--------------------------------------+---------+---------+ | ip | risks | types | +--------------------------------------+---------+---------+ | 9ec540f8-c191-4c11-9d88-47a86c312324 | | | | 9ec540f8-c191-4c11-9d88-47a86c312325 | UNKNOWN | UNKNOWN | | 9ec540f8-c191-4c11-9d88-47a86c312326 | UNKNOWN | UNKNOWN | | 9ec540f8-c191-4c11-9d88-47a86c312327 | | | | 9ec540f8-c191-4c11-9d88-47a86c312328 | | | +--------------------------------------+---------+---------+
Environment
ipoid commit 8cb472e49f7b8297940a808a004a66a81ebbfaf4
Reproduction data
{"as": {"Organization": ""}, "client": {"behaviors": [], "concentration": {}, "count": false, "countries": false, "proxies": [], "spread": false, "types": [false]}, "infrastructure": false, "location": {}, "organization": false, "risks": [false], "services": [], "tunnels": [], "ip": "9ec540f8-c191-4c11-9d88-47a86c312324"}
{"as": {"Organization": ""}, "client": {"behaviors": [], "concentration": {}, "count": false, "countries": false, "proxies": [], "spread": false, "types": []}, "infrastructure": false, "location": {}, "organization": false, "risks": [], "services": [], "tunnels": [], "ip": "9ec540f8-c191-4c11-9d88-47a86c312325"}
{"as": {"Organization": ""}, "client": {"behaviors": [], "concentration": {}, "count": false, "countries": false, "proxies": [], "spread": false}, "infrastructure": false, "location": {}, "organization": false, "services": [], "tunnels": [], "ip": "9ec540f8-c191-4c11-9d88-47a86c312326"}
{"as": {"Organization": ""}, "client": {"behaviors": [], "concentration": {}, "count": false, "countries": false, "proxies": [], "spread": false, "types": [""]}, "infrastructure": false, "location": {}, "organization": false, "risks": [""], "services": [], "tunnels": [], "ip": "9ec540f8-c191-4c11-9d88-47a86c312327"}
{"as": {"Organization": ""}, "client": {"behaviors": [], "concentration": {}, "count": false, "countries": false, "proxies": [], "spread": false, "types": ["foo"]}, "infrastructure": false, "location": {}, "organization": false, "risks": ["foo"], "services": [], "tunnels": [], "ip": "9ec540f8-c191-4c11-9d88-47a86c312328"}QA Results - Ipoid SQL
| AC | Status | Details |
|---|---|---|
| 1 | ✅ | For consistency and to save future confusion, we should probably always record invalid values or values it does not recognise as UNKNOWN |
