In T338016#8927417, @dom_walden wrote:@STran How robust do we need/want this to be?
I find with input data:
{"as": {}, "client": {"behaviors": [], "concentration": {}, "count": true, "countries": true, "proxies": [0], "spread": false, "types": []}, "infrastructure": false, "location": {}, "organization": false, "risks": [], "services": [], "tunnels": [], "ip": "6dda8fa6-e044-498c-b66d-bcd90f3e99c6"} {"as": {}, "client": {"behaviors": [], "concentration": {"city": "", "country": 0}, "count": false, "countries": false, "proxies": [false], "spread": false, "types": []}, "infrastructure": false, "location": {}, "organization": "", "risks": [], "services": [], "tunnels": [], "ip": "039e2d19-8584-40cf-ad18-474f0f09c05b"}I get:
/srv/service/node_modules/mariadb/lib/misc/errors.js:61 return new SqlError(msg, sql, fatal, info, sqlState, errno, additionalStack, addHeader); ^ SqlError: (conn=1045, no: 1054, SQLState: 42S22) Unknown column 'undefined' in 'field list' sql: INSERT INTO actor_data_proxies (actor_data_id,proxy_id) VALUES (2, undefined) - parameters:[] at Object.module.exports.createError (/srv/service/node_modules/mariadb/lib/misc/errors.js:61:10) at PacketNodeEncoded.readError (/srv/service/node_modules/mariadb/lib/io/packet.js:511:19) at Query.readResponsePacket (/srv/service/node_modules/mariadb/lib/cmd/resultset.js:46:28) at PacketInputStream.receivePacketBasic (/srv/service/node_modules/mariadb/lib/io/packet-input-stream.js:104:9) at PacketInputStream.onData (/srv/service/node_modules/mariadb/lib/io/packet-input-stream.js:169:20) at Socket.emit (node:events:513:28) at addChunk (node:internal/streams/readable:315:12) at readableAddChunk (node:internal/streams/readable:289:9) at Socket.Readable.push (node:internal/streams/readable:228:10) at TCP.onStreamRead (node:internal/stream_base_commons:190:23) { text: "Unknown column 'undefined' in 'field list'", sql: 'INSERT INTO\n' + ' actor_data_proxies (actor_data_id,proxy_id)\n' + ' VALUES\n' + ' (2, undefined) - parameters:[]', fatal: false, errno: 1054, sqlState: '42S22', code: 'ER_BAD_FIELD_ERROR' }I have also seen similar errors reported with the actor_data_behaviors table.
I think something to do with the fact you have got both "proxies": [0] and "proxies": [false], which both get changed to 0 when attempting to insert into the database, so the second INSERT will fail. Notice that you can swap the order of those two lines and still get the same error.
FYI, I am using this script to generate the JSON for import: https://gitlab.wikimedia.org/dwalden/ipmasking-testing/-/blob/main/spur_random_data.py
@dom_walden do you know what data caused the error in actor_data_behaviors?