Page MenuHomePhabricator

Allow Quarry to query its own database
Closed, ResolvedPublic

Description

One of the two requirements of the parent task T151158 is to support creating a query against Quarry's own database. I'm splitting that requirement into its own task.

This should be done with a read-only user. The database is currently in Trove, so we can easily create a quarry_readonly user there.

I checked the db schema (pasted below) and I have some minor privacy concerns if we were to expose all the tables and columns, as it would leak some user ids, and it would make it possible to check which queries have been starred by which users.

A more conservative approach could be to create one or more redacted views, depending on which data people are interested in querying.

Please leave a comment below if you have a use case for this feature, specifying which tables/columns need to be exposed.

MariaDB [quarry]> SHOW TABLES;
+------------------+
| Tables_in_quarry |
+------------------+
| query            |
| query_revision   |
| query_run        |
| star             |
| user             |
| user_group       |
+------------------+
6 rows in set (0.001 sec)

| query | CREATE TABLE `query` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `user_id` int(10) unsigned NOT NULL,
  `title` varchar(1024) DEFAULT NULL,
  `latest_rev_id` int(10) unsigned DEFAULT NULL,
  `last_touched` timestamp NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
  `parent_id` int(10) unsigned DEFAULT NULL,
  `published` smallint(6) NOT NULL DEFAULT 0,
  `description` text DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `query_parent_id_index` (`parent_id`)
) ENGINE=InnoDB AUTO_INCREMENT=83811 DEFAULT CHARSET=utf8 |

| query_revision | CREATE TABLE `query_revision` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `text` text DEFAULT NULL,
  `query_id` int(10) unsigned NOT NULL,
  `timestamp` timestamp NOT NULL DEFAULT current_timestamp(),
  `latest_run_id` int(10) unsigned DEFAULT NULL,
  `query_database` varchar(66) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `query_rev_query_id_index` (`query_id`),
  KEY `query_rev_query_database_index` (`query_database`)
) ENGINE=InnoDB AUTO_INCREMENT=900685 DEFAULT CHARSET=utf8 |

| query_run | CREATE TABLE `query_run` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `query_rev_id` int(10) unsigned NOT NULL,
  `status` tinyint(3) unsigned NOT NULL DEFAULT 0,
  `timestamp` timestamp NOT NULL DEFAULT current_timestamp(),
  `task_id` varchar(36) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
  `extra_info` text CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `query_run_status_index` (`status`)
) ENGINE=InnoDB AUTO_INCREMENT=874191 DEFAULT CHARSET=utf8 |

| star  | CREATE TABLE `star` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `user_id` int(10) unsigned NOT NULL,
  `query_id` int(10) unsigned NOT NULL,
  `timestamp` timestamp NOT NULL DEFAULT current_timestamp(),
  PRIMARY KEY (`id`),
  UNIQUE KEY `star_user_query_index` (`user_id`,`query_id`),
  KEY `star_user_id_index` (`user_id`),
  KEY `star_query_id_index` (`query_id`)
) ENGINE=InnoDB AUTO_INCREMENT=1876 DEFAULT CHARSET=latin1 |

| user  | CREATE TABLE `user` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `username` varchar(255) CHARACTER SET utf8 DEFAULT NULL,
  `wiki_uid` int(10) unsigned NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `wiki_uid` (`wiki_uid`),
  UNIQUE KEY `user_wiki_uid` (`wiki_uid`),
  UNIQUE KEY `username` (`username`),
  UNIQUE KEY `user_username_index` (`username`)
) ENGINE=InnoDB AUTO_INCREMENT=8925 DEFAULT CHARSET=latin1 |

| user_group | CREATE TABLE `user_group` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `user_id` int(10) unsigned NOT NULL,
  `group_name` varchar(255) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  PRIMARY KEY (`id`),
  KEY `user_group_user_group_index` (`user_id`,`group_name`),
  KEY `user_group_user_id_index` (`user_id`),
  KEY `user_group_group_name_index` (`group_name`)
) ENGINE=InnoDB AUTO_INCREMENT=12 DEFAULT CHARSET=utf8 |

Event Timeline

fnegri changed the task status from Open to In Progress.Jun 24 2024, 1:28 PM
fnegri claimed this task.

I checked the db schema (pasted below) and I have some minor privacy concerns if we were to expose all the tables and columns, as it would leak some user ids, and it would make it possible to check which queries have been starred by which users.

I checked again, and I don't think there are any privacy concerns: the user ids are just autoincrementing ids internal to Quarry, and it's already possible to see which queries have been starred by which users from the web interface.

I will create a quarry_readonly user in the Trove db, and see if I can figure out how to tell Quarry to use that user when "quarry" is inserted in the "db name" textbox.

Maybe wiki_uid in the user table is potentially sensitive? It seems to be the Mediawiki id for the user, I'm not sure if we should expose it.

Maybe wiki_uid in the user table is potentially sensitive? It seems to be the Mediawiki id for the user, I'm not sure if we should expose it.

MediaWiki user ids are public. They're exposed by the API and in the replicas as user_id field in the user table.

MediaWiki user ids are public. They're exposed by the API and in the replicas as user_id field in the user table.

Great, thanks!

OTOH exposing the list of users that have logged in to Quarry, even if they've not interacted with anything that leaves a public trace, feels a bit questionable.

OTOH exposing the list of users that have logged in to Quarry, even if they've not interacted with anything that leaves a public trace, feels a bit questionable.

Good point. Maybe we could exclude the user and user_group tables? The only use case I have in mind for this feature is to gather some stats on query execution times, and which tables are being queried, so having access to the query* tables would be enough. We could still group by user_id, as that is exposed in the query table.

fnegri triaged this task as Medium priority.Jun 24 2024, 2:25 PM

@bd808 I'm interested in your opinion on this one. I created a pull request, but I'm also wondering if anybody still wants it and maybe we should just ignore it until someone asks for it. The original request dates back to 2016 (see the parent task).

@bd808 I'm interested in your opinion on this one. I created a pull request, but I'm also wondering if anybody still wants it and maybe we should just ignore it until someone asks for it. The original request dates back to 2016 (see the parent task).

The main use case I have heard for this over the years is query discovery. Folks have hypothesized that it would be useful to search for queries that use particular tables or have been run against particular databases. I don't think this is possible today via the search at https://quarry.wmcloud.org/query/runs/all, but it may be that I am out of date on built-in features.

I don't have any objections to hiding the user and user_group tables. I am fairly sure all of the data they might contain is public via other systems, but I understand that aggregation in new locations can present novel exposure risks. Ease of bulk queries is generally the only "new" data exposure here. It is currently possible to view profile pages like https://quarry.wmcloud.org/BDavis_(WMF) that expose each user's query and star table data.

I created the quarry_readonly user manually because Trove doesn't let me create read-only users:

MariaDB [(none)]> CREATE USER quarry_readonly@'172.16.%' IDENTIFIED BY 'xxx';
Query OK, 0 rows affected (0.004 sec)

MariaDB [(none)]> GRANT SELECT, SHOW VIEW ON quarry_p.* TO quarry_readonly@'172.16.%';
Query OK, 0 rows affected (0.004 sec)

The password is stored in the Quarry repo at helm-quarry/prod-config.yaml (git-crypted).

The quarry_p database was also created manually, but I've also added the schema to schema.sql in https://github.com/toolforge/quarry/pull/61.

This is working and you can now type quarry or quarry_p in the "db name" input. quarry_p will always be used even if you type quarry, similarly to what happens for enwiki, etc.

The following tables can be queried:

  • query
  • query_revision
  • query_run
  • star