Page MenuHomePhabricator

Create an `mpic` MariaDB database
Closed, ResolvedPublic

Description

The MPIC application will require a MariaDB database. We decided to call it mpic, which is now the standard name for domains, database, chart, etc.

This database will need to have an owner, probably called mpic as well.

Event Timeline

We have the Analytics_Meta database service that we can use for this.

This is the MariaDB database service that currently hosts Hive, Druid, and Superset.
It is currently on MariaDB version 10.4 (which we have to upgrade to 10.6 soon) and the primary host is an-mariadb1001.
We have a standby replica on an-mariadb1002 and a backup replica on db1208.

I am still a bit confused by the way that this ticket crosses over with T331516: Design and get approval for database schema in which the database was approved for the misc cluster but was never created.
If the ultimate target for the mpic app is wikikube instead of dse-k8s, and the preferred database type is MariaDB, then I'm not sure why the analytics_meta database cluster is preferred over the misc cluster.

Gehel triaged this task as High priority.Apr 17 2024, 3:03 PM
+----------------------------------------------------------------------------------------------------------------+
| Grants for mpic_staging@10.%                                                                                   |
+----------------------------------------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO `mpic_staging`@`10.%` IDENTIFIED BY PASSWORD '*xxxx' |
| GRANT ALL PRIVILEGES ON `mpic_staging`.* TO `mpic_staging`@`10.%`                                              |
+----------------------------------------------------------------------------------------------------------------+
2 rows in set (0.000 sec)

MariaDB [mpic_production]> SHOW GRANTS FOR mpic_production@'10.%';
+-------------------------------------------------------------------------------------------------------------------+
| Grants for mpic_production@10.%                                                                                   |
+-------------------------------------------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO `mpic_production`@`10.%` IDENTIFIED BY PASSWORD '*xxxx' |
| GRANT ALL PRIVILEGES ON `mpic_production`.* TO `mpic_production`@`10.%`                                           |
+-------------------------------------------------------------------------------------------------------------------+
2 rows in set (0.000 sec)

Thank you @brouberol!
Are you ok if we provide you a SQL script to create the tables for the database?

Ok!
We are working on the schema and SQL script at T360748: [User Story] Create the MPIC database schema. As soon as it's available we'll provide it to you.

hi @brouberol -- here's an initial script to kick things off:

init.sql
CREATE TABLE IF NOT EXISTS instruments (
    id INT UNSIGNED NOT NULL AUTO_INCREMENT,
    name VARCHAR(255) NOT NULL,
    slug VARCHAR(255) NOT NULL,
    description TEXT,
    creator VARCHAR(255) NOT NULL,
    owner VARCHAR(255) NOT NULL,
    purpose VARCHAR(255),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
    start_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL,
    end_date TIMESTAMP,
    task VARCHAR(1000) NOT NULL,
    compliance_requirements SET('legal', 'gdpr') NOT NULL,
    sample_unit VARCHAR(255) NOT NULL,
    sample_rate FLOAT NOT NULL,
    environments SET('development', 'staging', 'production', 'external') NOT NULL,
    security_legal_review VARCHAR(1000) NOT NULL,
    status VARCHAR(255) NOT NULL,
    PRIMARY KEY (id)
);

CREATE TABLE IF NOT EXISTS instrument_sample_rates (
    id INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
    instrument_id INT UNSIGNED NOT NULL,
    dblist VARCHAR(255) NOT NULL,
    sample_rate FLOAT NOT NULL,
    constraint fk_de_type
    foreign key(instrument_id)
    references instruments (id)
);

CREATE TABLE IF NOT EXISTS contextual_attributes (
    id INT UNSIGNED NOT NULL AUTO_INCREMENT,
    contextual_attribute_name VARCHAR(255) NOT NULL,
    PRIMARY KEY (id)
);

CREATE TABLE IF NOT EXISTS instrument_contextual_attribute_lookup (
    id INT UNSIGNED NOT NULL AUTO_INCREMENT,
    instrument_id INT UNSIGNED NOT NULL,
    contextual_attribute_id INT UNSIGNED NOT NULL,
    PRIMARY KEY (id)
);

INSERT INTO contextual_attributes (contextual_attribute_name) VALUES
 ('agent_app_install_id'),
 ('agent_client_platform'),
 ('agent_client_platform_family'),
 ('page_id'),
 ('page_title'),
 ('page_namespace'),
 ('page_namespace_name'),
 ('page_revision_id'),
 ('page_wikidata_id'),
 ('page_wikidata_qid'),
 ('page_content_language'),
 ('page_is_redirect'),
 ('page_user_groups_allowed_to_move'),
 ('page_user_groups_allowed_to_edit'),
 ('mediawiki_skin'),
 ('mediawiki_version'),
 ('mediawiki_is_production'),
 ('mediawiki_is_debug_mode'),
 ('mediawiki_database'),
 ('mediawiki_site_content_language'),
 ('mediawiki_site_content_language_variant'),
 ('performer_is_logged_in'),
 ('performer_id'),
 ('performer_name'),
 ('performer_session_id'),
 ('performer_pageview_id'),
 ('performer_groups'),
 ('performer_is_bot'),
 ('performer_language'),
 ('performer_language_variant'),
 ('performer_can_probably_edit_page'),
 ('performer_edit_count'),
 ('performer_edit_count_bucket'),
 ('performer_registration_dt');

cc @Sfaci

@cjming @Sfaci I have applied the provided schema on an-mariadb1001.eqiad.wmnet, databases mpic_staging and mpic_production. Enjoy!

Change #1028486 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] global_config: Only expose the IP of the analytics meta master

https://gerrit.wikimedia.org/r/1028486

Change #1028486 merged by Brouberol:

[operations/puppet@production] global_config: Only expose the IP of the analytics meta master

https://gerrit.wikimedia.org/r/1028486