Page MenuHomePhabricator

Density imported incorrectly
Closed, ResolvedPublicBUG REPORT

Description

What is the problem?

The init-db.js script only appears to save the values 0 or 1 in the database. But, the density values in the JSON can be numbers between 0 and 1, such as 0.3737244897959184.

Steps to reproduce problem
  1. Download {F37103087}
  2. Put it in the /tmp folder in your ipoid directory
  3. Start ipoid if it is not already running (e.g. docker-compose up -d), you might need to wait for it to start
  4. Run docker-compose exec -e FEED_PATH=./tmp/density_reprod.json.gz web node init-db.js
  5. Access the ipoid database (e.g. docker-compose exec db mysql -u root -p)
  6. Enter USE test and then SELECT ip, conc_density FROM actor_data;

Expected behavior: Value of conc_density is 0.3737244897959184.
Observed behavior: Value is 0.

Environment

ipoid: commit 8400f97f21ff3aa3bae82fb33dfdd4d5369075f4

Details

TitleReferenceAuthorSource BranchDest Branch
Update schema:conc_densityrepos/mediawiki/services/ipoid!21tsepothoabalaT339073main
Customize query in GitLab

Event Timeline

This script may help when testing. It compares the entries in the JSON with the entries in the database. https://gitlab.wikimedia.org/dwalden/ipmasking-testing/-/blob/main/ipoid_data_validation.py

It's because we set the schema as conc_density INT. This should probably be DECIMAL.

I imported a small amount of test data and checked that the density in the DB and JSON files matched.