Page MenuHomePhabricator
Create Task
Maniphest T330154

Check if new airflow package is usable on both buster and bullseye
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

The airflow packages are now created using a conda environment
https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/main/debian/Dockerfile

They are uploaded to the package repository for the airflow-dags
https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/packages

From there it is manually copied to the active apt server (apt2001 at the time of writing) and added to apt.wikimedia.org using reprepro.

This ticket is about finding out whether the same version of the package is usable on both buster and bullseye.

Note that the conda environment itself is still built using a buster image:
https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/main/debian/Dockerfile#L38
FROM docker-registry.wikimedia.org/buster:latest as conda_dist

Therefore we will need to plan to upgrade this image at some point as well, even if it works.

If the package does not work on bullseye, we will need to modify the build process to build both bullseye and buster packages.

Related Objects
Search...

Event Timeline

Stevemunene created this task.Feb 21 2023, 1:36 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 21 2023, 1:36 PM
JArguello-WMF moved this task from Backlog to To be discussed /To be estimated on the Data Pipelines board.Feb 21 2023, 4:23 PM
JArguello-WMF moved this task from To be discussed /To be estimated to Sprint 11 on the Data Pipelines board.Feb 21 2023, 5:20 PM
JArguello-WMF edited projects, added Data Pipelines (Sprint 11); removed Data Pipelines.
JArguello-WMF set the point value for this task to 3.Feb 21 2023, 5:23 PM
JArguello-WMF moved this task from Next Up to Next Up on the Data Pipelines (Sprint 11) board.Feb 21 2023, 5:45 PM
Antoine_Quhen added a parent task: T326193: Airflow upgrade (refactor deb creation + version bump + switch to PostgreSQL).Feb 21 2023, 8:14 PM
lbowmaker removed a project: Data-Engineering.Feb 21 2023, 10:59 PM
Antoine_Quhen assigned this task to BTullis.Feb 23 2023, 3:40 PM
Antoine_Quhen updated Other Assignee, added: BTullis.
Antoine_Quhen updated Other Assignee, removed: BTullis.
JArguello-WMF edited projects, added Data Pipelines (sprint 10); removed Data Pipelines (Sprint 11).Mar 13 2023, 1:59 PM
JArguello-WMF moved this task from Ready to Next Up on the Data Pipelines (sprint 10) board.
JArguello-WMF reassigned this task from BTullis to Stevemunene.Mar 29 2023, 4:08 PM
JArguello-WMF added a subscriber: BTullis.
BTullis updated the task description. (Show Details)Mar 30 2023, 8:53 AM
BTullis added a comment.Mar 30 2023, 8:56 AM

For reference, here is an-test-client1001, running buster, which is where the analytics_dev instance of airflow is installed.

btullis@an-test-client1001:~$ cat /etc/debian_version 
10.13
btullis@an-test-client1001:~$ apt-cache policy airflow
airflow:
  Installed: 2.5.1-py3.10-20230228
  Candidate: 2.5.1-py3.10-20230228
  Version table:
 *** 2.5.1-py3.10-20230228 1001
       1001 http://apt.wikimedia.org/wikimedia buster-wikimedia/main amd64 Packages
        100 /var/lib/dpkg/status

an-test-client1002 is a bullseye server with the same role.

btullis@an-test-client1002:~$ apt-cache policy airflow
N: Unable to locate package airflow
BTullis added a comment.Mar 30 2023, 8:59 AM

Following the guidance here: https://wikitech.wikimedia.org/wiki/Reprepro#Copying_between_distributions

Check the contents of the repository

btullis@apt2001:~$ sudo -i reprepro ls airflow
airflow | 2.5.1-py3.10-20230228 | buster-wikimedia | amd64

Copy the airflow package from the buster to the bullseye distribution. Note the unusual order of the targets in the copy statement.

btullis@apt2001:~$ sudo -i reprepro copy bullseye-wikimedia buster-wikimedia airflow
Exporting indices...

Check the contents again.

btullis@apt2001:~$ sudo -i reprepro ls airflow
airflow | 2.5.1-py3.10-20230228 |   buster-wikimedia | amd64
airflow | 2.5.1-py3.10-20230228 | bullseye-wikimedia | amd64
BTullis added a comment.Mar 30 2023, 9:00 AM

Run an apt update on the client and then check for its availability.

btullis@an-test-client1002:~$ sudo apt update
Hit:1 http://mirrors.wikimedia.org/debian bullseye InRelease
Get:2 http://mirrors.wikimedia.org/debian bullseye-updates InRelease [44.1 kB]
Get:3 http://mirrors.wikimedia.org/debian bullseye-backports InRelease [49.0 kB]
Hit:4 http://deb.debian.org/debian-debug bullseye-debug InRelease
Get:5 http://apt.wikimedia.org/wikimedia bullseye-wikimedia InRelease [108 kB]
Hit:6 http://security.debian.org/debian-security bullseye-security InRelease
Get:7 http://apt.wikimedia.org/wikimedia bullseye-wikimedia/main amd64 Packages [57.1 kB]
Fetched 258 kB in 1s (223 kB/s)     
su: warning: cannot change directory to /nonexistent: No such file or directory

You do not have a valid Kerberos ticket in the credential cache, remember to kinit.
INFO:debmonitor:Found 918 installed binary packages
INFO:debmonitor:Found 0 upgradable binary packages (including new dependencies)
INFO:debmonitor:Successfully sent the upgradable update to the DebMonitor server
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
btullis@an-test-client1002:~$ apt-cache policy airflow
airflow:
  Installed: (none)
  Candidate: 2.5.1-py3.10-20230228
  Version table:
     2.5.1-py3.10-20230228 1001
       1001 http://apt.wikimedia.org/wikimedia bullseye-wikimedia/main amd64 Packages

@Stevemunene - over to you now, can you please check to see whether we feel that this airflow package is usable on bullseye? Thanks.

JArguello-WMF edited projects, added Data Pipelines (Sprint 11); removed Data Pipelines (sprint 10).Mar 31 2023, 1:59 PM
JArguello-WMF moved this task from Next Up to Next Up on the Data Pipelines (Sprint 11) board.Mar 31 2023, 2:40 PM
JArguello-WMF moved this task from Next Up to In Progress on the Data Pipelines (Sprint 11) board.Apr 3 2023, 4:07 PM
Stevemunene added a comment.Apr 13 2023, 12:59 PM

Package successfully installs on Buster machines.

stevemunene@trino-test-coord-1:~$ apt-cache policy airflow
airflow:
  Installed: 2.5.1-py3.10-20230228
  Candidate: 2.5.1-py3.10-20230228
  Version table:
 *** 2.5.1-py3.10-20230228 1001
       1001 http://apt.wikimedia.org/wikimedia bullseye-wikimedia/main amd64 Packages
        100 /var/lib/dpkg/status

Looking to get the role on the new test client 'an-test-client1002.eqiad.wmnet' for a much more thorough test on functionality.

JArguello-WMF moved this task from In Progress to Done on the Data Pipelines (Sprint 11) board.Apr 13 2023, 4:07 PM
JArguello-WMF closed this task as Resolved.Apr 13 2023, 4:10 PM
Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL