Airflow target version:
We could try to upgrade to the very last one. From the upgrade to 2.3.4, I think we mainly risk deprecation warnings.
Target environments:
Target instances
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Antoine_Quhen | T326193 Airflow upgrade (refactor deb creation + version bump + switch to PostgreSQL) | |||
| Resolved | Antoine_Quhen | T326194 Generate Airflow deb package with Gitlab CI | |||
| Resolved | Antoine_Quhen | T309552 Update Airflow DAGs code to make it compatible with version V2.3.4 of Airflow | |||
| Resolved | Antoine_Quhen | T325204 Add missing tests in airflow-dags | |||
| Resolved | Stevemunene | T315580 Upgrade Puppet code to make Airflow configuration files compatible with version 2.5.0 | |||
| Resolved | Stevemunene | T326195 Edit puppet code to provide Airflow the PostgreSQL connection | |||
| Resolved | Stevemunene | T330154 Check if new airflow package is usable on both buster and bullseye | |||
| Restricted Task |
The merge requests on Gitlab:
Gerrit changes:
Change 889572 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Add a postgresql database and user for airflow_search_platform
Change 889572 merged by Btullis:
[operations/puppet@production] Add a postgresql database and user for airflow_search_platform
Note: we should create a new branch, main_airflow_2_5_1, in airflow-dags to deploy the code only one the instances which are migrated. Hopefully, it shouldn't take long. Then when the migration process is finished we could merge to main and deploy from main.
Change 891804 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Ensure that the airflow database names match existing conventions
Change 891804 merged by Btullis:
[operations/puppet@production] Ensure that the airflow database names match existing conventions
We have:
btullis@apt1001:~$ sudo -i reprepro ls airflow airflow | 2.1.4-py3.7-5 | buster-wikimedia | amd64
btullis@apt1001:~$ sudo -i reprepro includedeb buster-wikimedia `pwd`/airflow-2.5.1-py3.10-20230221_amd64.deb Exporting indices... Deleting files no longer referenced...
btullis@apt1001:~$ sudo -i reprepro ls airflow airflow | 2.5.1-py3.10-20230221 | buster-wikimedia | amd64
btullis@an-test-client1001:~$ apt-cache policy airflow
airflow:
Installed: 2.1.4-py3.7-5
Candidate: 2.1.4-py3.7-5
Version table:
*** 2.1.4-py3.7-5 1001
1001 http://apt.wikimedia.org/wikimedia buster-wikimedia/main amd64 Packages
100 /var/lib/dpkg/status
btullis@an-test-client1001:~$ sudo apt update --quiet
Get:1 http://apt.wikimedia.org/wikimedia buster-wikimedia InRelease [173 kB]
Hit:2 http://mirrors.wikimedia.org/debian buster InRelease
Hit:3 http://mirrors.wikimedia.org/debian buster-updates InRelease
Hit:4 http://mirrors.wikimedia.org/debian buster-backports InRelease
Hit:5 http://security.debian.org/debian-security buster/updates InRelease
Hit:6 http://deb.debian.org/debian-debug buster-debug InRelease
Get:7 http://apt.wikimedia.org/wikimedia buster-wikimedia/main amd64 Packages [110 kB]
Fetched 282 kB in 1s (249 kB/s)
su: warning: cannot change directory to /nonexistent: No such file or directory
You do not have a valid Kerberos ticket in the credential cache, remember to kinit.
INFO:debmonitor:Found 1108 installed binary packages
INFO:debmonitor:Found 40 upgradable binary packages (including new dependencies)
INFO:debmonitor:Successfully sent the upgradable update to the DebMonitor server
Reading package lists...
Building dependency tree...
Reading state information...
38 packages can be upgraded. Run 'apt list --upgradable' to see them.
btullis@an-test-client1001:~$ apt-cache policy airflow
airflow:
Installed: 2.1.4-py3.7-5
Candidate: 2.5.1-py3.10-20230221
Version table:
2.5.1-py3.10-20230221 1001
1001 http://apt.wikimedia.org/wikimedia buster-wikimedia/main amd64 Packages
*** 2.1.4-py3.7-5 100
100 /var/lib/dpkg/statusChange 896098 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Upgrade Airflon on an-launcher1002 to version 2.5.1
Change 896098 merged by Btullis:
[operations/puppet@production] Upgrade Airflow on an-launcher1002 to version 2.5.1
Change 897845 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Upgrade the airflow package in stat hosts
Change 897845 abandoned by Btullis:
[operations/puppet@production] Upgrade the airflow package in stat hosts
Reason:
Unnecessary
For the Research team we will wait until Fabian is back to participate in the upgrade, requested by Miriam
Change 900366 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Upgrade Airflow on the platform_eng instance and switch to PostgreSQL
Change 900366 merged by Btullis:
[operations/puppet@production] Upgrade Airflow on the platform_eng instance and switch to PostgreSQL
Change 900403 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Remove the overridden configuration for airflow-platform_eng
Change 900403 merged by Btullis:
[operations/puppet@production] Remove the overridden configuration for airflow-platform_eng
Change 903242 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Upgrade the research airflow instance
Change 903242 merged by Btullis:
[operations/puppet@production] Upgrade the research airflow instance
Change 903254 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/puppet@production] Remove stray referece to ariflow db from research instance
Change 903254 merged by Btullis:
[operations/puppet@production] Remove stray referece to ariflow db from research instance
Can we say that this epic is closed now @Antoine_Quhen or is there more work to do on it?
Oh, I suppose that there are still some puppet bit to clean up, now that we no longer have any Airflow 2.1 instances deployed.