Page MenuHomePhabricator

'Parameter at position 2 is undefined\n' if tunnels.type is not set
Closed, ResolvedPublic2 Estimated Story PointsBUG REPORT

Description

What is the problem?

If the JSON entry does not contain a tunnels.type, the import-data.js reports the exception:

/srv/service/node_modules/mariadb/lib/misc/errors.js:61
  return new SqlError(msg, sql, fatal, info, sqlState, errno, additionalStack, addHeader);
         ^

SqlError: Parameter at position 2 is undefined
INSERT INTO
                        tunnels (operator, type, anonymous)
                    VALUES
                        (?, ?, ?) - parameters:['r',undefined,true]
sql: INSERT INTO
                        tunnels (operator, type, anonymous)
                    VALUES
                        (?, ?, ?) - parameters:['r',undefined,true]
    at Object.module.exports.createError (/srv/service/node_modules/mariadb/lib/misc/errors.js:61:10)
    at Query.throwNewError (/srv/service/node_modules/mariadb/lib/cmd/command.js:63:16)
    at Query.validateParameters (/srv/service/node_modules/mariadb/lib/cmd/query.js:179:14)
    at Query.start (/srv/service/node_modules/mariadb/lib/cmd/query.js:53:17)
    at _addCommandEnablePipeline (/srv/service/node_modules/mariadb/lib/connection.js:1245:11)
    at /srv/service/node_modules/mariadb/lib/connection.js:177:7
    at new Promise (<anonymous>)
    at Connection._queryPromise (/srv/service/node_modules/mariadb/lib/connection.js:174:12)
    at /srv/service/import-data.js:159:47
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  text: 'Parameter at position 2 is undefined\n' +
    'INSERT INTO\n' +
    '                        tunnels (operator, type, anonymous)\n' +
    '                    VALUES\n' +
    "                        (?, ?, ?) - parameters:['r',undefined,true]",
  sql: 'INSERT INTO\n' +
    '                        tunnels (operator, type, anonymous)\n' +
    '                    VALUES\n' +
    "                        (?, ?, ?) - parameters:['r',undefined,true]",
  fatal: false,
  errno: 45017,
  sqlState: 'HY000',
  code: 'ER_PARAMETER_UNDEFINED'
}
Steps to reproduce problem
  1. Save the JSON from "Reproduction data" below as a .gz file (e.g. reprod.json.gz) into the ipoid/tmp directory
  2. If necessary, start up docker in the ipoid directory (e.g. docker compose up -d)
  3. Initialise the database: docker compose exec web node init-db.js
  4. Run this command: docker compose exec -e FEED_PATH=./tmp/reprod.json.gz web node import-data.js
Environment

ipoid commit 9bac83bb26f9b808edc3756284a8183a4e21c5c2

Reproduction data
{"as": {}, "client": {"behaviors": [], "concentration": {}, "count": false, "countries": null, "proxies": [], "spread": null, "types": ["DESKTOP"]}, "infrastructure": false, "location": {}, "organization": false, "risks": ["TUNNEL"], "services": [], "tunnels": [{"operator": "r", "anonymous": true}], "ip": "633f9bba-6d4f-40d5-ba34-f15ac42743d3"}

Details

TitleReferenceAuthorSource BranchDest Branch
Ensure proper anonymous assignment, preserving value when tunnel.anonymous is falserepos/mediawiki/services/ipoid!31tsepothoabalaT339359-allow-falsemain
'Parameter at position 2 is undefined\n' if tunnels.type is not setrepos/mediawiki/services/ipoid!27tsepothoabalaT339359main
Customize query in GitLab

Event Timeline

I'm seeing this as well with the data feed from today.

Simliar bug if anonymous is not set. For example:

{"as": {"number": 12345, "Organization": "Foobar"}, "client": {"behaviors": ["FILE_SHARING"], "concentration": {"geohash": "srwcr5ugt", "city": "Foobar", "state": "Foobar", "country": "NL", "skew": 0.5, "density": 0.5, "countries": 10}, "count": 10, "countries": 10, "proxies": ["OXYLABS_PROXY"], "spread": 12345, "types": ["MOBILE"]}, "infrastructure": "MOBILE", "location": {"city": "Baku", "state": "Baku City", "country": "AZ"}, "organization": "Foobar", "risks": ["CALLBACK_PROXY"], "services": ["IPSEC"], "tunnels": [{"operator": "TUNNELBEAR_VPN", "type": "VPN"}], "ip": "6bcaac10-88d8-4e2c-8e98-ae27d8840e26"}

Output:

...
SqlError: Parameter at position 3 is undefined
...

As stated this happens when tunnels.type is not set, we could update the schema and make type and anonymous columns nullable.
Is it expected that some entries won't contain this data in production data? @STran

Sure that sounds like a reasonable solution. At the moment, we're not going to show either type or anonymous so it doesn't really matter if it exists or not. Niharika's asked us to keep it around just in case and I suppose if we eventually show it, we still don't want to negate any tunnels w/operators that don't have it set regardless.

tsepothoabala opened https://gitlab.wikimedia.org/repos/mediawiki/services/ipoid/-/merge_requests/27

'Parameter at position 2 is undefined\n' if tunnels.type is not set

Tested the patch on the main branch versus with this patch.
Confirming getting the first error and it being solved with Tsepo's patch.

@TThoabala A tunnel like {"operator":"GLOBAL_PROTECT_CLOUD_VPN","type":"VPN","anonymous":false} will be stored in the database as:

SELECT * FROM tunnels;
+------+--------------------------+------+-----------+
| pkid | operator                 | type | anonymous |
+------+--------------------------+------+-----------+
|    1 | GLOBAL_PROTECT_CLOUD_VPN | VPN  | NULL      |
+------+--------------------------+------+-----------+

I guess because tunnel.anonymous || null is null if tunnel.anonymous is false. But false is a legitimate value and should be stored in the database as 0 (I believe).

@dom_walden I didn't realise this can include false, I can move this back to in progress and create a separate patch. thank you.

Thanks @dom_walden and thanks @TThoabala for the updated patch.

Using reproduction data from T341605, and running the following commands

docker compose exec web node init-db.js
docker compose exec -e FEED_PATH=./tmp/reprod.json.gz web node import-data.js

I no longer get the error.
LGTM

AGueyte set the point value for this task to 2.Jul 17 2023, 3:57 PM

stran merged https://gitlab.wikimedia.org/repos/mediawiki/services/ipoid/-/merge_requests/31

Ensure proper anonymous assignment, preserving value when tunnel.anonymous is false

@TThoabala I no longer receive any errors as seen in the screenshot. The task will be moved to Done. Thanks for all your work!

Data Used

{"as": {}, "client": {"behaviors": [], "concentration": {}, "count": false, "countries": null, "proxies": [], "spread": null, "types": ["DESKTOP"]}, "infrastructure": false, "location": {}, "organization": false, "risks": ["TUNNEL"], "services": [], "tunnels": [{"operator": "testz", "type":"VPN", "anonymous": false}], "ip": "633f9bba-6d4f-40d5-ba34-f15ac42743d3"}

Commands ran:

docker-compose exec web node init-db.js 
gzip -c test_file.json > test1.json.gz 
docker-compose exec -e FEED_PATH=./tmp/test1.json.gz web node import-data.js

Sequel Pro app- Output

T339359_IPoid_AnonymousFalse.png (1×2 px, 281 KB)