Page MenuHomePhabricator

Create MariaDB schema "image_suggestions", and grant permissions to user xcollazo
Closed, DeclinedPublic

Description

In T312799, we are doing a POC of connecting our Airflow infrastructure to a MariaDB instance on one of the misc servers.

Ask:
Create schema "image_suggestions". Create user xcollazo. Grant ALL to user xcollazo under this new schema. Provide us connection details.

Purpose:
We want to create some dummy tables under this "image_suggestions" schema to prove that we can access it from our Airflow infrastructure.

Sample DDL and data:

CREATE TABLE IF NOT EXISTS test (
    id INT AUTO_INCREMENT PRIMARY KEY,
    content VARCHAR(255) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

INSERT INTO test(content) VALUES ("hello"), ("world");

If the POC is successful, we would want to drop all the dummy tables and repurpose the schema to create a set of tables that would provide state for an app, but this is far in the future, and we don't know the DDL for the app yet. We would of course let you know before we move forward with that.

Event Timeline

misc clusters are considered production clusters and we do not normally create testing tables and testing users in production environments just like that.
What do you need to test the connection apart from that? Which misc cluster do you need to connect to?

We could create these set of things on our testing cluster (db1124/db1125) which are pretty much the same as misc clusters, apart that you do not connect to them thru a proxy as you'd normally do on the misc hosts.

Marostegui moved this task from Triage to In progress on the DBA board.

What do you need to test the connection apart from that?

We want to make sure there is no connectivity/firewall issues between our Airflow and Analytics cluster to the MariaDB instances.

We could create these set of things on our testing cluster (db1124/db1125) which are pretty much the same as misc clusters

And do these testing clusters have the same network/firewall settings as the misc cluster? If so, this does sound like a better fit for our POC.

What do you need to test the connection apart from that?

We want to make sure there is no connectivity/firewall issues between our Airflow and Analytics cluster to the MariaDB instances.

If that's all you'd need you can just try a telnet (or netcat) to one of the proxies to the port 3306.

If you let me know which would you'd connect from, I can try to test it myself.

More importantly the design of the tables are important (when you want to get the tables to production and start using it in your service). That's what's important.

If that's all you'd need you can just try a telnet (or netcat) to one of the proxies to the port 3306.

Sorry, I misspoke. I want my workflow system (Airflow) to connect to this MariaDB instance, as there may be helper functions we'd need to create to make this sort of connection easy for our customers, so it's not just making sure the connection is possible.

I like the idea of telneting for a quick connectivity check though. Can you please let me know the URL / point me to documentation where I can find this info?

More importantly the design of the tables are important (when you want to get the tables to production and start using it in your service). That's what's important.

Agreed. Other folks are working on defining the DDL and once that is ready we will let you all know.

If that's all you'd need you can just try a telnet (or netcat) to one of the proxies to the port 3306.

Sorry, I misspoke. I want my workflow system (Airflow) to connect to this MariaDB instance, as there may be helper functions we'd need to create to make this sort of connection easy for our customers, so it's not just making sure the connection is possible.

I like the idea of telneting for a quick connectivity check though. Can you please let me know the URL / point me to documentation where I can find this info?

Basically from the host you'd need to access:
telnet m1-master.eqiad.wmnet 3306

Maybe you won't have privileges to run that, that's why I as asking for the hostnames to see if I can access them myself and try.

More importantly the design of the tables are important (when you want to get the tables to production and start using it in your service). That's what's important.

Agreed. Other folks are working on defining the DDL and once that is ready we will let you all know.

I would suggest you provide data beforehand about all this, as this is probably more important than the connection itself. This is pretty much the first news we have about a new service that requires hosting in our misc servers.
Please read https://wikitech.wikimedia.org/wiki/MariaDB#Database_creation_template and provide some of the stats (even if they are approximate at this stage) here.

Connection is successful:

$ hostname -f
an-airflow1003.eqiad.wmnet
$ telnet m1-master.eqiad.wmnet 3306
Trying 10.64.16.18...
Connected to dbproxy1014.eqiad.wmnet.
Escape character is '^]'.
]
...

Please read https://wikitech.wikimedia.org/wiki/MariaDB#Database_creation_template and provide some of the stats (even if they are approximate at this stage) here.

@Marostegui : will discuss this with my team and get back to you.

Connection is successful:

$ hostname -f
an-airflow1003.eqiad.wmnet
$ telnet m1-master.eqiad.wmnet 3306
Trying 10.64.16.18...
Connected to dbproxy1014.eqiad.wmnet.
Escape character is '^]'.
]
...

Good, so there's no connection issue.

Please read https://wikitech.wikimedia.org/wiki/MariaDB#Database_creation_template and provide some of the stats (even if they are approximate at this stage) here.

@Marostegui : will discuss this with my team and get back to you.

Thanks, this is a lot more important than the connection itself.
I am going to close this task for now as we've seen the connection works fine but please open a new one once you've discussed more with your team and are ready to provide info on what you'd expect us to host.