wmfdata.mariadb relies on analytics-mysql being available
Open, LowPublic
Actions

Assigned To

None

Authored By

	nettrom_WMF
	Oct 4 2021, 8:11 PM

Description

mariadb.connect relies on analytics-mysql to identify the database host and port number to connect to, ref mariadb.py lines 30 through 45.

By default, cron jobs on stat100* servers do not have the PATH set to include the location of analytics-mysql, and thus throw an error when trying to query the MariaDB servers. Setting the path of course solves this problem, but there might be preferable long-term solutions that take of this seamlessly.

In addition, analytics-mysql is not available on an-test-client1001, which complicates the process of testing Wmfdata.

Related Objects

Mentioned In: T293700: Decide whether and how to consolidate Wmfdata-Python and Refinery's Python modules

Event Timeline

nettrom_WMF created this task.Oct 4 2021, 8:11 PM

ldelench_wmf assigned this task to nshahquinn-wmf.Oct 5 2021, 5:09 PM

ldelench_wmf edited projects, added Product-Analytics (Kanban); removed Product-Analytics.

ldelench_wmf moved this task from Next 2 weeks to Needs Investigation on the Product-Analytics (Kanban) board.

odimitrijevic triaged this task as High priority.Oct 18 2021, 4:48 PM

odimitrijevic added projects: SRE, Analytics-Kanban.

odimitrijevic moved this task from Incoming to Operational Excellence on the Analytics board.

I think I can handle this just by using an absolute reference to the file in refinery in wmfdata-python.

Otherwise, it's standard for cron jobs to run with a very limited environment that doesn't even include the defaults specified in /etc/profile (which in our case puts analytics-mysql on the PATH). I'm not sure if that can or should be changed. I suppose that's a question for Analytics 😊

It seems like the priority isn't that high since there's a pretty easy workaround (setting the PATH in the cron job, which is a pretty normal thing to do). @odimitrijevic feel free to change it back if you disagree.

nshahquinn-wmf moved this task from Triage to Backlog on the Product-Analytics board.Oct 18 2021, 8:33 PM

odimitrijevic added projects: Data-Engineering-Kanban, Data-Engineering.Oct 28 2021, 5:25 AM

odimitrijevic moved this task from Incoming (new tickets) to Ops Week on the Data-Engineering board.Nov 4 2021, 4:51 PM

One idea is to move the logic that finds the host/port to wmfdata, and then to make refinery depend on wmfdata. Pro: it makes sense for this to belong in wmfdata, as the central place that deals with abstracting connection information from users. Con: adds dependencies to refinery.

nshahquinn-wmf mentioned this in T293700: Decide whether and how to consolidate Wmfdata-Python and Refinery's Python modules.Dec 22 2021, 2:59 AM

odimitrijevic removed a project: Analytics.Jan 12 2022, 12:29 AM

Unclear whether or not we want this logic to live in Wmfdata-Python; it makes sense there, but that might cause problems making it widely available as a command-line tool.

JArguello-WMF removed a project: Data-Engineering-Kanban.Apr 5 2022, 8:05 PM

Updated the description to note:

In addition, analytics-mysql is not available on an-test-client1001, which complicates the process of testing Wmfdata.

This shouldn't be assigned to me; I've never had a concrete plan to work on it.

I don't believe there are any specific actions for SRE here, untagging.

JArguello-WMF moved this task from Ops Week to Data Products & Metrics on the Data-Engineering board.Jun 29 2023, 10:42 PM

lbowmaker moved this task from Data Products & Metrics to Icebox (not considered in current quarter) on the Data-Engineering board.Nov 10 2023, 2:42 PM

wmfdata.mariadb relies on analytics-mysql being availableOpen, LowPublicActions

Description

Related Objects

Event Timeline

wmfdata.mariadb relies on analytics-mysql being available
Open, LowPublic
Actions