Page MenuHomePhabricator

Upgrade Airflow to version 2.6.3
Closed, ResolvedPublic

Description

We have decided that we would like to upgrade our Airflow instances across the board to version 2.6.3

The current version is 2.5.1 and this upgrade was completed recently.
One of the purposes of this ticket is to ensure that we have a standard procedure for upgrading Airflow versions, whether they be major versions, minor versions, or point releases.

This task is intended to be the tracking task for the whole upgrade and subtasks will be generated as required. The following tasks will be required at a minimum:

  • Ensure that we have the Airflow upgrade procedure documented on Wikitech
  • Create a Debian package of Airflow 2.6.3
  • Deploy the new version of Airflow to the analytics_test instance
  • Test the analytics DAGs with the new version
  • Upgrade all production instances

Details

Related Changes in Gerrit:
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Update airflow to version 2.6.3repos/data-engineering/airflow-dags!450btullisbump_airflow_hive_connectormain
Update airflow to version 2.6.1repos/data-engineering/airflow-dags!385btullisupdate_airflow_2_6main
Customize query in GitLab

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Checking database:

btullis@an-test-client1001:~$ sudo -u analytics airflow-analytics_test db check
/usr/lib/airflow/lib/python3.10/site-packages/airflow/configuration.py:423 FutureWarning: Bad scheme in Airflow configuration core > sql_alchemy_conn: `postgres`. As of SQLAlchemy 1.4 (adopted in Airflow 2.3) this is no longer supported.  You must change to `postgresql` before the next Airflow release.
/usr/lib/airflow/lib/python3.10/site-packages/airflow/utils/db.py:1737 RemovedIn20Warning: [31mDeprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. [32mTo prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". [36mSet environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings.  Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message.[0m (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
[2023-05-17T13:50:24.947+0000] {db.py:1738} INFO - Connection successful.

Upgrading database

btullis@an-test-client1001:~$ sudo -u analytics airflow-analytics_test db upgrade
/usr/lib/airflow/lib/python3.10/site-packages/airflow/configuration.py:423 FutureWarning: Bad scheme in Airflow configuration core > sql_alchemy_conn: `postgres`. As of SQLAlchemy 1.4 (adopted in Airflow 2.3) this is no longer supported.  You must change to `postgresql` before the next Airflow release.
DB: postgresql://airflow_analytics_test:***@an-db1001.eqiad.wmnet/airflow_analytics_test?sslmode=require&sslrootcert=%2Fetc%2Fssl%2Fcerts%2FPuppet_Internal_CA.pem
Performing upgrade with database postgresql://airflow_analytics_test:***@an-db1001.eqiad.wmnet/airflow_analytics_test?sslmode=require&sslrootcert=%2Fetc%2Fssl%2Fcerts%2FPuppet_Internal_CA.pem
[2023-05-17T13:50:40.783+0000] {migration.py:213} INFO - Context impl PostgresqlImpl.
[2023-05-17T13:50:40.783+0000] {migration.py:216} INFO - Will assume transactional DDL.
[2023-05-17T13:50:40.826+0000] {db.py:1588} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 290244fb8b83 -> 6abdffdd4815, add dttm index on log table
INFO  [alembic.runtime.migration] Running upgrade 6abdffdd4815 -> 98ae134e6fff, Increase length of user identifier columns in ``ab_user`` and ``ab_register_user`` tables
Upgrades done

Restarted all airflow services on an-test-client1001 and they start cleanly.

The UI looks OK to me.

The @hourly jobs have now triggered, so I'll see if they complete.
Maybe someone like @mforns or @Antoine_Quhen or @Milimetric could have another check to see if things look OK as far as the DAGs go.

BTullis moved this task from In Progress to Needs Review on the Data-Platform-SRE board.

Moving to in-review whilst I work on double-checking that the DAGs are working as expected.

Hmm. The aqs_hourly job seems to have been running for longer than usual, waiting for webrequest.

image.png (1,541×726 px, 78 KB)

We had a failure email from the aqs_hourly test job.

Try 6 out of 6
Exception:
'SparkSkeinSubmitHook' object has no attribute '_spark_binary'
Log: Link
Host: an-test-client1001.eqiad.wmnet
Mark success: Link

This is similar to what we saw from the failing tests, so I'll see if I can find a reference to it in the DAGs.

We had a failure email from the aqs_hourly test job.

Try 6 out of 6
Exception:
'SparkSkeinSubmitHook' object has no attribute '_spark_binary'
Log: Link
Host: an-test-client1001.eqiad.wmnet
Mark success: Link

This is similar to what we saw from the failing tests, so I'll see if I can find a reference to it in the DAGs.

Submitted a patch on top of @BTullis MR that should solve this issue.

TLDR: Upstream Spark provider changed internals and since our code inherits from them, we had to change the way we fetch the spark_binary.

Ah! Yes, it seems we didn't deploy the fix to the analytics-test instance, doing now.

We had a failure email from the aqs_hourly test job.

Try 6 out of 6
Exception:
'SparkSkeinSubmitHook' object has no attribute '_spark_binary'
Log: Link
Host: an-test-client1001.eqiad.wmnet
Mark success: Link

This is similar to what we saw from the failing tests, so I'll see if I can find a reference to it in the DAGs.

Submitted a patch on top of @BTullis MR that should solve this issue.

TLDR: Upstream Spark provider changed internals and since our code inherits from them, we had to change the way we fetch the spark_binary.

@xcollazo Oh, wait... There was a prior fix that worked in airflow analytics instance. I just forgot to deploy it to analytics_test.
I think that is the reason it was failing. Do we still need to apply this last fix?

I just deployed to analytics_test and all seems to work.
I can not see the failed task that you mention in previous comments.

We had a failure email from the aqs_hourly test job.

Try 6 out of 6
Exception:
'SparkSkeinSubmitHook' object has no attribute '_spark_binary'
Log: Link
Host: an-test-client1001.eqiad.wmnet
Mark success: Link

This is similar to what we saw from the failing tests, so I'll see if I can find a reference to it in the DAGs.

Submitted a patch on top of @BTullis MR that should solve this issue.

TLDR: Upstream Spark provider changed internals and since our code inherits from them, we had to change the way we fetch the spark_binary.

@xcollazo Oh, wait... There was a prior fix that worked in airflow analytics instance. I just forgot to deploy it to analytics_test.
I think that is the reason it was failing. Do we still need to apply this last fix?

Either fix should be fine. If the other one is tested already, feel free to revert mine (I had applied it as a commit to your MR).

That's both.

FYI We're getting the 'SparkSkeinSubmitHook' object has no attribute '_spark_binary emails again at the moment.

I didn't have much of a chance to look at it today, but the hotfix that @xcollazo had put in place on an-test-client1001 did seem to work.

I'm happy with either approach though.

I added my fix as a 2 commits to the initial MR from @BTullis, see: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/385
It made all tests pass, but I didn't test it in any real instance.

That said, I think I know what happened yesterday.
I thought that Ben's MR was merged, but not yet deployed to the analytics_test Airflow instance.
So, I just deployed the HEAD of the repo's main branch to analytics_test. And that obviously didn't make any sense.
I should have deployed Ben's airflow-dags code there, which is the 2.6-compatible one.
Will do that now.

I think that we need a plan for how to move forward with this Airflow 2.6 upgrade. I have a few questions.

  • Will we need to use a feature branch to deploy to instances that have been upgraded to version 2.6, owing to the change in the SparkSubmitHook or can we somehow backport that to main?
  • Should we upgrade the existing feature branch to version 2.6.1, which has just been released? The changelog is here: https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html#airflow-2-6-1-2023-05-16
  • Do we think that other teams' DAGs will be likely to work on version 2.6 without modification, or will we need to coordinate changes to their subdirectories in the airflow_dags repo?

I think that we need a plan for how to move forward with this Airflow 2.6 upgrade. I have a few questions.

  • Will we need to use a feature branch to deploy to instances that have been upgraded to version 2.6, owing to the change in the SparkSubmitHook or can we somehow backport that to main?

The fix goes into wmf_airflow_common, so every instance will pick it up automatically. So I don't think we need a separate branch for this reason. Right @mforns?

+1 to bump to 2.6.1. Free bug fixes!

  • Do we think that other teams' DAGs will be likely to work on version 2.6 without modification, or will we need to coordinate changes to their subdirectories in the airflow_dags repo?

According to my investigation on T336306, we should be good:

Apache Airflow uses strict SemVer versioning policy, which means that we strive for any release of a given MAJOR Version (version "2" currently) to be backwards compatible.

The fix goes into wmf_airflow_common, so every instance will pick it up automatically. So I don't think we need a separate branch for this reason. Right @mforns?

I think that fix will only work if the instance is running on 2.6. In 2.6 they changed the name of the spark binary field from self._spark_binary to self.spark_binary (no leading underscore). So we should only deploy the fix to instances that are running 2.6. I guess we were accessing a private field in SparkSubmitHook, so this doesn't count as breaking backwards incompatibility?
Hmmm. is there a way of knowing which version airflow is running from python code? If so, we could improve the fix and add an if which will use self._spark_binary if we're running 2.5 and will use self.spark_binary if we're using 2.6, no?
Once the upgrade is finished, we can remove the extra code.

The fix goes into wmf_airflow_common, so every instance will pick it up automatically. So I don't think we need a separate branch for this reason. Right @mforns?

I think that fix will only work if the instance is running on 2.6. In 2.6 they changed the name of the spark binary field from self._spark_binary to self.spark_binary (no leading underscore). So we should only deploy the fix to instances that are running 2.6. I guess we were accessing a private field in SparkSubmitHook, so this doesn't count as breaking backwards incompatibility?

Ah, yes, yes, I don't know why I thought all instances would magically get upgraded to 2.6 once we merged into main 😃. I agree self._spark_binary seems internal.

Hmmm. is there a way of knowing which version airflow is running from python code? If so, we could improve the fix and add an if which will use self._spark_binary if we're running 2.5 and will use self.spark_binary if we're using 2.6, no?
Once the upgrade is finished, we can remove the extra code.

Yes there is:

xcollazo@stat1007:~$ source conda-analytics-activate xcollazo_airflow_base
(xcollazo_airflow_base) xcollazo@stat1007:~$ conda list | grep airflow
# packages in environment at /home/xcollazo/.conda/envs/xcollazo_airflow_base:
airflow-exporter          1.4.1                    pypi_0    pypi
...
(xcollazo_airflow_base) xcollazo@stat1007:~$ python
Python 3.7.15 (default, Nov 24 2022, 21:12:53) 
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import airflow
/home/xcollazo/.conda/envs/xcollazo_airflow_base/lib/python3.7/site-packages/airflow/configuration.py:240: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if StrictVersion(sqlite3.sqlite_version) < StrictVersion(min_sqlite_version):   
>>> airflow.__version__
'2.1.4'
>>> type(airflow.__version__)
<class 'str'>

Just stumbled on this task, I created the missing patch.
It should work regardless of the version of Airflow that is running it.
https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/434

Thanks so much @mforns - That looks excellent.

BTullis renamed this task from Upgrade Airflow to version 2.6.0 to Upgrade Airflow to version 2.6.1.Jun 20 2023, 9:35 AM
BTullis updated the task description. (Show Details)

I've updated the branch to build Airflow 2.6.1 instead and I'm now creating a package from it. I'll test this new package on an-test-client1001 when it's complete.

I have successfully built the airflow 2.6.1 package and added it to reprepro.

btullis@apt1001:~$ wget https://gitlab.wikimedia.org/api/v4/projects/93/packages/generic/airflow/2.6.1-py3.10-20230619/airflow-2.6.1-py3.10-20230619_amd64.deb
--2023-06-20 15:52:16--  https://gitlab.wikimedia.org/api/v4/projects/93/packages/generic/airflow/2.6.1-py3.10-20230619/airflow-2.6.1-py3.10-20230619_amd64.deb
Resolving gitlab.wikimedia.org (gitlab.wikimedia.org)... 2620:0:861:2:208:80:154:145, 208.80.154.145
Connecting to gitlab.wikimedia.org (gitlab.wikimedia.org)|2620:0:861:2:208:80:154:145|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 625911260 (597M) [application/octet-stream]
Saving to: ‘airflow-2.6.1-py3.10-20230619_amd64.deb’

airflow-2.6.1-py3.10-20230619_amd64.deb              100%[=====================================================================================================================>] 596.92M   109MB/s    in 5.5s    

2023-06-20 15:52:22 (109 MB/s) - ‘airflow-2.6.1-py3.10-20230619_amd64.deb’ saved [625911260/625911260]

btullis@apt1001:~$ sudo -i reprepro ls airflow
airflow | 2.5.1-py3.10-20230228 |   buster-wikimedia | amd64
airflow | 2.5.1-py3.10-20230228 | bullseye-wikimedia | amd64
btullis@apt1001:~$ sudo -i reprepro includedeb buster-wikimedia `pwd`/airflow-2.6.1-py3.10-20230619_amd64.deb
Exporting indices...
btullis@apt1001:~$ sudo -i reprepro includedeb bullseye-wikimedia `pwd`/airflow-2.6.1-py3.10-20230619_amd64.deb
Exporting indices...
Deleting files no longer referenced...
btullis@apt1001:~$ sudo -i reprepro ls airflow
airflow | 2.6.1-py3.10-20230619 |   buster-wikimedia | amd64
airflow | 2.6.1-py3.10-20230619 | bullseye-wikimedia | amd64

Change 931637 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Bump the version of airflow installed on the analytics_test instance

https://gerrit.wikimedia.org/r/931637

Change 931637 merged by Btullis:

[operations/puppet@production] Bump the version of airflow installed on the analytics_test instance

https://gerrit.wikimedia.org/r/931637

Change 932241 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Remove the specific airflow version override for an-test-client1001

https://gerrit.wikimedia.org/r/932241

Change 932241 merged by Btullis:

[operations/puppet@production] Remove the specific airflow version override for an-test-client1001

https://gerrit.wikimedia.org/r/932241

I've now installed this on an-test-client1001.

btullis@an-test-client1001:~$ sudo run-puppet-agent
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for an-test-client1001.eqiad.wmnet
Info: Unable to serialize catalog to json, retrying with pson
Info: Applying configuration version '(75d84a2b02) Btullis - Remove the specific airflow version override for an-test-client1001'
Notice: /Stage[main]/Airflow/Package[airflow]/ensure: ensure changed '2.6.0-py3.10-20230517' to '2.6.1-py3.10-20230619'
Notice: Applied catalog in 177.03 seconds

I've run the database migrations:

btullis@an-test-client1001:~$ sudo -u analytics airflow-analytics_test db check
/usr/lib/airflow/lib/python3.10/site-packages/airflow/configuration.py:424 FutureWarning: Bad scheme in Airflow configuration core > sql_alchemy_conn: `postgres`. As of SQLAlchemy 1.4 (adopted in Airflow 2.3) this is no longer supported.  You must change to `postgresql` before the next Airflow release.
/usr/lib/airflow/lib/python3.10/site-packages/airflow/utils/db.py:1738 RemovedIn20Warning: [31mDeprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. [32mTo prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". [36mSet environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings.  Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message.[0m (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
[2023-06-22T14:33:44.515+0000] {db.py:1739} INFO - Connection successful.
btullis@an-test-client1001:~$ sudo -u analytics airflow-analytics_test db upgrade
/usr/lib/airflow/lib/python3.10/site-packages/airflow/configuration.py:424 FutureWarning: Bad scheme in Airflow configuration core > sql_alchemy_conn: `postgres`. As of SQLAlchemy 1.4 (adopted in Airflow 2.3) this is no longer supported.  You must change to `postgresql` before the next Airflow release.
DB: postgresql://airflow_analytics_test:***@an-db1001.eqiad.wmnet/airflow_analytics_test?sslmode=require&sslrootcert=%2Fetc%2Fssl%2Fcerts%2FPuppet_Internal_CA.pem
Performing upgrade with database postgresql://airflow_analytics_test:***@an-db1001.eqiad.wmnet/airflow_analytics_test?sslmode=require&sslrootcert=%2Fetc%2Fssl%2Fcerts%2FPuppet_Internal_CA.pem
[2023-06-22T14:33:54.501+0000] {migration.py:213} INFO - Context impl PostgresqlImpl.
[2023-06-22T14:33:54.502+0000] {migration.py:216} INFO - Will assume transactional DDL.
[2023-06-22T14:33:54.550+0000] {db.py:1589} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
Upgrades done

I've restarted the airflow services on an-test-client1001.

btullis@an-test-client1001:~$ sudo systemctl restart airflow-scheduler@analytics_test.service airflow-webserver@analytics_test.service

Initial results from the web interface seem good. I will monitor for successful DAG runs.

Change 933087 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Upgrade the analytics airflow instance to 2.6.1

https://gerrit.wikimedia.org/r/933087

Change 933088 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Upgrade the search instance of airflow to version 2.6.1

https://gerrit.wikimedia.org/r/933088

Change 933089 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Upgrade the research instance of airflow to version 2.6.1

https://gerrit.wikimedia.org/r/933089

Change 933090 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Update the platform_eng airflow instance to version 2.6.1

https://gerrit.wikimedia.org/r/933090

Change 933091 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Upgrade the analytics_product airflow instance to version 2.6.1

https://gerrit.wikimedia.org/r/933091

BTullis triaged this task as Medium priority.Jun 28 2023, 10:29 AM

Change 934286 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] datahub: Retain the setup jobs after execution to help with debugging

https://gerrit.wikimedia.org/r/934286

Change 934286 merged by jenkins-bot:

[operations/deployment-charts@master] datahub: Retain the setup jobs after execution to help with debugging

https://gerrit.wikimedia.org/r/934286

Change 933087 merged by Btullis:

[operations/puppet@production] Upgrade the analytics airflow instance to 2.6.1

https://gerrit.wikimedia.org/r/933087

We've paused all of the DAGS on the nalytics instance manually, then merged the puppet patch and is has applied cleanly.

btullis@an-launcher1002:~$ sudo run-puppet-agent
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for an-launcher1002.eqiad.wmnet
Info: Unable to serialize catalog to json, retrying with pson
Info: Applying configuration version '(321091b785) Btullis - Upgrade the analytics airflow instance to 2.6.1'
Notice: /Stage[main]/Airflow/Package[airflow]/ensure: ensure changed '2.5.1-py3.10-20230228' to '2.6.1-py3.10-20230619'
Notice: Applied catalog in 155.80 seconds

We applied the migrations by running sudo -u analytics airflow-analytics db upgrade

btullis@an-launcher1002:~$ sudo -u analytics airflow-analytics db check
/usr/lib/airflow/lib/python3.10/site-packages/airflow/configuration.py:424 FutureWarning: Bad scheme in Airflow configuration core > sql_alchemy_conn: `postgres`. As of SQLAlchemy 1.4 (adopted in Airflow 2.3) this is no longer supported.  You must change to `postgresql` before the next Airflow release.
/usr/lib/airflow/lib/python3.10/site-packages/airflow/utils/db.py:1738 RemovedIn20Warning: [31mDeprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. [32mTo prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". [36mSet environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings.  Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message.[0m (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
[2023-06-29T13:50:13.304+0000] {db.py:1739} INFO - Connection successful.
btullis@an-launcher1002:~$ sudo -u analytics airflow-analytics db upgrade
/usr/lib/airflow/lib/python3.10/site-packages/airflow/configuration.py:424 FutureWarning: Bad scheme in Airflow configuration core > sql_alchemy_conn: `postgres`. As of SQLAlchemy 1.4 (adopted in Airflow 2.3) this is no longer supported.  You must change to `postgresql` before the next Airflow release.
DB: postgresql://airflow_analytics:***@an-db1001.eqiad.wmnet/airflow_analytics?sslmode=require&sslrootcert=%2Fetc%2Fssl%2Fcerts%2FPuppet_Internal_CA.pem
Performing upgrade with database postgresql://airflow_analytics:***@an-db1001.eqiad.wmnet/airflow_analytics?sslmode=require&sslrootcert=%2Fetc%2Fssl%2Fcerts%2FPuppet_Internal_CA.pem
[2023-06-29T13:50:20.950+0000] {migration.py:213} INFO - Context impl PostgresqlImpl.
[2023-06-29T13:50:20.951+0000] {migration.py:216} INFO - Will assume transactional DDL.
[2023-06-29T13:50:20.994+0000] {db.py:1589} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 290244fb8b83 -> 6abdffdd4815, add dttm index on log table
INFO  [alembic.runtime.migration] Running upgrade 6abdffdd4815 -> 98ae134e6fff, Increase length of user identifier columns in ``ab_user`` and ``ab_register_user`` tables
Upgrades done

We unpaused all DAGs. So far it looks good, but we will keep an eye out for any failures.

The next step is to schedule the upgrades for other teams' instances. I will look to do this early next week, unless there are any requests to defer them.

The next step is to schedule the upgrades for other teams' instances. I will look to do this early next week, unless there are any requests to defer them.

CCing platform_eng Airflow instance folks for awareness: @mfossati @Cparle @Htriedman .

Unfortunately, we have to update again because of a new vulnerability announced in the hive-connector. Mentioned in: T336244

We can either stick at 2.6.1 and just regenerate the conda environment with the updated packages, or we could upgrade to version 2.6.2
https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html#airflow-2-6-2-2023-06-17

There don't appear to be any breaking changes, only bugfixes and other miscellaneous fixes.
I suspect I know the answer, but should I upgrade to version 2.6.2 or just update the packages in 2.6.1 @xcollazo ?

I made apatch to try the upgrade to version 2.6.2 but I get an error from pytest that I don't understand: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/jobs/115493

Unfortunately, we have to update again because of a new vulnerability announced in the hive-connector. Mentioned in: T336244

We can either stick at 2.6.1 and just regenerate the conda environment with the updated packages, or we could upgrade to version 2.6.2
https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html#airflow-2-6-2-2023-06-17

There don't appear to be any breaking changes, only bugfixes and other miscellaneous fixes.
I suspect I know the answer, but should I upgrade to version 2.6.2 or just update the packages in 2.6.1 @xcollazo ?

+1 to move forward with 2.6.2. Given the security concern, we have to pay the testing penalty anyways, and we get a point release of bug fixes. Win win!

I made apatch to try the upgrade to version 2.6.2 but I get an error from pytest that I don't understand: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/jobs/115493

Looks like the pydantic dependency has to be pinned: https://www.reddit.com/r/apache_airflow/comments/14oji77/airflow_262_and_pydantic_warningserrors/

I made apatch to try the upgrade to version 2.6.2 but I get an error from pytest that I don't understand: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/jobs/115493

Looks like the pydantic dependency has to be pinned: https://www.reddit.com/r/apache_airflow/comments/14oji77/airflow_262_and_pydantic_warningserrors/

Great! Thanks for the pointer. I'll give it another go.

I made apatch to try the upgrade to version 2.6.2 but I get an error from pytest that I don't understand: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/jobs/115493

Looks like the pydantic dependency has to be pinned: https://www.reddit.com/r/apache_airflow/comments/14oji77/airflow_262_and_pydantic_warningserrors/

Great! Thanks for the pointer. I'll give it another go.

There was a 2.6.3 release, which fixes five additional issues (all marked "low" by upstream, though):

CVE-2023-22888: Apache Airflow: Scheduler remote DoS
https://www.openwall.com/lists/oss-security/2023/07/11/5

CVE-2023-36543: Apache Airflow: ReDoS via dags function
https://www.openwall.com/lists/oss-security/2023/07/11/4

CVE-2022-46651: Apache Airflow: Security vulnerability on AirFlow Connections
https://www.openwall.com/lists/oss-security/2023/07/11/3

CVE-2023-22887: Apache Airflow path traversal by authenticated user
https://www.openwall.com/lists/oss-security/2023/07/11/2

CVE-2023-35908: Apache Airflow: Access to DAGs without relevant permission
https://www.openwall.com/lists/oss-security/2023/07/11/1

I'm running the publish-debian-package pipeline manually on https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/pipelines/21904

When it's complete I will upload this package to apt.wikimedia.org and deploy it to analytics_test for testing.

There was a 2.6.3 release, which fixes five additional issues (all marked "low" by upstream, though):

Oh, these Airflow releases come thick and fast.

I'll have a look, but probably build a 2.6.3 release. At least we're getting experience of doing the upgrades.

MoritzMuehlenhoff renamed this task from Upgrade Airflow to version 2.6.1 to Upgrade Airflow to version 2.6.3.Jul 11 2023, 5:22 PM

Change 938803 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Deploy airflow version 2.6.3 to analytics_test

https://gerrit.wikimedia.org/r/938803

Change 938803 merged by Btullis:

[operations/puppet@production] Deploy airflow version 2.6.3 to analytics_test

https://gerrit.wikimedia.org/r/938803

I have upgraded the analytics_test instance and evexuted the migrations.

btullis@an-test-client1002:~$ sudo -u analytics airflow-analytics_test db check
/usr/lib/airflow/lib/python3.10/site-packages/airflow/configuration.py:421 FutureWarning: Bad scheme in Airflow configuration core > sql_alchemy_conn: `postgres`. As of SQLAlchemy 1.4 (adopted in Airflow 2.3) this is no longer supported.  You must change to `postgresql` before the next Airflow release.
/usr/lib/airflow/lib/python3.10/site-packages/airflow/utils/db.py:1740 RemovedIn20Warning: [31mDeprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. [32mTo prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". [36mSet environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings.  Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message.[0m (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
[2023-07-17T09:27:22.024+0000] {db.py:1741} INFO - Connection successful.
btullis@an-test-client1002:~$ sudo -u analytics airflow-analytics_test db upgrade
/usr/lib/airflow/lib/python3.10/site-packages/airflow/configuration.py:421 FutureWarning: Bad scheme in Airflow configuration core > sql_alchemy_conn: `postgres`. As of SQLAlchemy 1.4 (adopted in Airflow 2.3) this is no longer supported.  You must change to `postgresql` before the next Airflow release.
DB: postgresql://airflow_analytics_test:***@an-db1001.eqiad.wmnet/airflow_analytics_test?sslmode=require&sslrootcert=%2Fetc%2Fssl%2Fcerts%2FPuppet_Internal_CA.pem
Performing upgrade with database postgresql://airflow_analytics_test:***@an-db1001.eqiad.wmnet/airflow_analytics_test?sslmode=require&sslrootcert=%2Fetc%2Fssl%2Fcerts%2FPuppet_Internal_CA.pem
[2023-07-17T09:27:32.067+0000] {migration.py:213} INFO - Context impl PostgresqlImpl.
[2023-07-17T09:27:32.068+0000] {migration.py:216} INFO - Will assume transactional DDL.
[2023-07-17T09:27:32.121+0000] {db.py:1591} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 98ae134e6fff -> c804e5c76e3e, Add ``onupdate`` cascade to ``task_map`` table
Upgrades done

Change 939347 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Upgrade the analytics instance of airflow to version 2.6.3

https://gerrit.wikimedia.org/r/939347

Change 939347 merged by Btullis:

[operations/puppet@production] Upgrade the analytics instance of airflow to version 2.6.3

https://gerrit.wikimedia.org/r/939347

I have upgraded the analytics instance.

All migrations executed.

btullis@an-launcher1002:~$ sudo su - analytics airflow-analytics db upgrade
su: warning: cannot change directory to /nonexistent: No such file or directory

Found a valid Kerberos ticket in the credential cache:
Ticket cache: FILE:/tmp/krb5cc_906
Default principal: analytics/an-launcher1002.eqiad.wmnet@WIKIMEDIA

Valid starting       Expires              Service principal
07/31/2023 12:25:00  08/02/2023 12:25:00  krbtgt/WIKIMEDIA@WIKIMEDIA
	renew until 08/07/2023 12:25:00
/usr/lib/airflow/lib/python3.10/site-packages/airflow/configuration.py:421 FutureWarning: Bad scheme in Airflow configuration core > sql_alchemy_conn: `postgres`. As of SQLAlchemy 1.4 (adopted in Airflow 2.3) this is no longer supported.  You must change to `postgresql` before the next Airflow release.
DB: postgresql://airflow_analytics:***@an-db1001.eqiad.wmnet/airflow_analytics?sslmode=require&sslrootcert=%2Fetc%2Fssl%2Fcerts%2FPuppet_Internal_CA.pem
Performing upgrade with database postgresql://airflow_analytics:***@an-db1001.eqiad.wmnet/airflow_analytics?sslmode=require&sslrootcert=%2Fetc%2Fssl%2Fcerts%2FPuppet_Internal_CA.pem
[2023-07-31T12:29:14.286+0000] {migration.py:213} INFO - Context impl PostgresqlImpl.
[2023-07-31T12:29:14.287+0000] {migration.py:216} INFO - Will assume transactional DDL.
[2023-07-31T12:29:14.333+0000] {db.py:1591} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 98ae134e6fff -> c804e5c76e3e, Add ``onupdate`` cascade to ``task_map`` table
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
Upgrades done

Change 943557 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Update airflow sqlalchemy URI for all airflow instances

https://gerrit.wikimedia.org/r/943557

Change 943557 merged by Btullis:

[operations/puppet@production] Update airflow sqlalchemy URI for all airflow instances

https://gerrit.wikimedia.org/r/943557

Change 933088 merged by Btullis:

[operations/puppet@production] Upgrade the search instance of airflow to version 2.6.3

https://gerrit.wikimedia.org/r/933088

Upgraded the search instance:

btullis@an-airflow1005:~$ sudo run-puppet-agent
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for an-airflow1005.eqiad.wmnet
Info: Unable to serialize catalog to json, retrying with pson
Info: Applying configuration version '(9cb22f9754) Btullis - Upgrade the search instance of airflow to version 2.6.3'
Notice: /Stage[main]/Airflow/Package[airflow]/ensure: ensure changed '2.5.1-py3.10-20230228' to '2.6.3-py3.10-20230711'
Notice: Applied catalog in 159.86 seconds

Applied the migrations:

btullis@an-airflow1005:~$ sudo -u analytics-search airflow-search db upgrade
DB: postgresql://airflow_search:***@an-db1001.eqiad.wmnet/airflow_search?sslmode=require&sslrootcert=%2Fetc%2Fssl%2Fcerts%2FPuppet_Internal_CA.pem
Performing upgrade with database postgresql://airflow_search:***@an-db1001.eqiad.wmnet/airflow_search?sslmode=require&sslrootcert=%2Fetc%2Fssl%2Fcerts%2FPuppet_Internal_CA.pem
[2023-07-31T13:09:15.594+0000] {migration.py:213} INFO - Context impl PostgresqlImpl.
[2023-07-31T13:09:15.596+0000] {migration.py:216} INFO - Will assume transactional DDL.
[2023-07-31T13:09:15.655+0000] {db.py:1591} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 290244fb8b83 -> 6abdffdd4815, add dttm index on log table
INFO  [alembic.runtime.migration] Running upgrade 6abdffdd4815 -> 98ae134e6fff, Increase length of user identifier columns in ``ab_user`` and ``ab_register_user`` tables
INFO  [alembic.runtime.migration] Running upgrade 98ae134e6fff -> c804e5c76e3e, Add ``onupdate`` cascade to ``task_map`` table
Upgrades done

Restarted the services:

btullis@an-airflow1005:~$ sudo systemctl restart airflow-scheduler@search.service airflow-webserver@search.service

Change 933089 merged by Btullis:

[operations/puppet@production] Upgrade the research instance of airflow to version 2.6.3

https://gerrit.wikimedia.org/r/933089

Upgraded the research instance:

btullis@an-airflow1002:~$ sudo run-puppet-agent
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for an-airflow1002.eqiad.wmnet
Info: Unable to serialize catalog to json, retrying with pson
Info: Applying configuration version '(1c66bf7a78) Btullis - Upgrade the research instance of airflow to version 2.6.3'
Notice: /Stage[main]/Airflow/Package[airflow]/ensure: ensure changed '2.5.1-py3.10-20230228' to '2.6.3-py3.10-20230711'
Notice: Applied catalog in 149.07 seconds
btullis@an-airflow1002:~$ sudo -u analytics-research airflow-research db upgrade
DB: postgresql://airflow_research:***@an-db1001.eqiad.wmnet/airflow_research?sslmode=require&sslrootcert=%2Fetc%2Fssl%2Fcerts%2FPuppet_Internal_CA.pem
Performing upgrade with database postgresql://airflow_research:***@an-db1001.eqiad.wmnet/airflow_research?sslmode=require&sslrootcert=%2Fetc%2Fssl%2Fcerts%2FPuppet_Internal_CA.pem
[2023-07-31T13:36:04.610+0000] {migration.py:213} INFO - Context impl PostgresqlImpl.
[2023-07-31T13:36:04.610+0000] {migration.py:216} INFO - Will assume transactional DDL.
[2023-07-31T13:36:04.664+0000] {db.py:1591} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 290244fb8b83 -> 6abdffdd4815, add dttm index on log table
INFO  [alembic.runtime.migration] Running upgrade 6abdffdd4815 -> 98ae134e6fff, Increase length of user identifier columns in ``ab_user`` and ``ab_register_user`` tables
INFO  [alembic.runtime.migration] Running upgrade 98ae134e6fff -> c804e5c76e3e, Add ``onupdate`` cascade to ``task_map`` table
Upgrades done
btullis@an-airflow1002:~$ sudo systemctl restart airflow-webserver@research.service airflow-scheduler@research.service

Change 933090 merged by Btullis:

[operations/puppet@production] Update the platform_eng airflow instance to version 2.6.3

https://gerrit.wikimedia.org/r/933090

I have upgraded the platform_eng instance.

btullis@an-airflow1004:~$ sudo run-puppet-agent
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for an-airflow1004.eqiad.wmnet
Info: Unable to serialize catalog to json, retrying with pson
Info: Applying configuration version '(3670182459) Btullis - Update the platform_eng airflow instance to version 2.6.3'
Notice: /Stage[main]/Airflow/Package[airflow]/ensure: ensure changed '2.5.1-py3.10-20230228' to '2.6.3-py3.10-20230711'
Notice: Applied catalog in 133.11 seconds
btullis@an-airflow1004:~$ sudo -u analytics-platform-eng airflow-platform_eng db upgrade
DB: postgresql://airflow_platform_eng:***@an-db1001.eqiad.wmnet/airflow_platform_eng?sslmode=require&sslrootcert=%2Fetc%2Fssl%2Fcerts%2FPuppet_Internal_CA.pem
Performing upgrade with database postgresql://airflow_platform_eng:***@an-db1001.eqiad.wmnet/airflow_platform_eng?sslmode=require&sslrootcert=%2Fetc%2Fssl%2Fcerts%2FPuppet_Internal_CA.pem
[2023-07-31T14:07:34.687+0000] {migration.py:213} INFO - Context impl PostgresqlImpl.
[2023-07-31T14:07:34.688+0000] {migration.py:216} INFO - Will assume transactional DDL.
[2023-07-31T14:07:34.741+0000] {db.py:1591} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 290244fb8b83 -> 6abdffdd4815, add dttm index on log table
INFO  [alembic.runtime.migration] Running upgrade 6abdffdd4815 -> 98ae134e6fff, Increase length of user identifier columns in ``ab_user`` and ``ab_register_user`` tables
INFO  [alembic.runtime.migration] Running upgrade 98ae134e6fff -> c804e5c76e3e, Add ``onupdate`` cascade to ``task_map`` table
Upgrades done
btullis@an-airflow1004:~$ sudo systemctl restart airflow-webserver@platform_eng.service airflow-scheduler@platform_eng.service 
btullis@an-airflow1004:~$

Change 933091 merged by Btullis:

[operations/puppet@production] Upgrade the analytics_product airflow instance to version 2.6.3

https://gerrit.wikimedia.org/r/933091

I have upgraded the analyics-product instance of Airflow, which is the last of the production instances.

btullis@an-airflow1006:~$ sudo run-puppet-agent
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for an-airflow1006.eqiad.wmnet
Info: Unable to serialize catalog to json, retrying with pson
Info: Applying configuration version '(ce2d740267) Ben Tullis - Upgrade the analytics_product airflow instance to version 2.6.3'
Notice: /Stage[main]/Airflow/Package[airflow]/ensure: ensure changed '2.5.1-py3.10-20230228' to '2.6.3-py3.10-20230711'
Notice: Applied catalog in 166.53 seconds
btullis@an-airflow1006:~$ sudo -u analytics-product airflow-analytics_product db upgrade
DB: postgresql://airflow_analytics_product:***@an-db1001.eqiad.wmnet/airflow_analytics_product?sslmode=require&sslrootcert=%2Fetc%2Fssl%2Fcerts%2FPuppet_Internal_CA.pem
Performing upgrade with database postgresql://airflow_analytics_product:***@an-db1001.eqiad.wmnet/airflow_analytics_product?sslmode=require&sslrootcert=%2Fetc%2Fssl%2Fcerts%2FPuppet_Internal_CA.pem
[2023-07-31T15:12:07.619+0000] {migration.py:213} INFO - Context impl PostgresqlImpl.
[2023-07-31T15:12:07.620+0000] {migration.py:216} INFO - Will assume transactional DDL.
[2023-07-31T15:12:07.671+0000] {db.py:1591} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 290244fb8b83 -> 6abdffdd4815, add dttm index on log table
INFO  [alembic.runtime.migration] Running upgrade 6abdffdd4815 -> 98ae134e6fff, Increase length of user identifier columns in ``ab_user`` and ``ab_register_user`` tables
INFO  [alembic.runtime.migration] Running upgrade 98ae134e6fff -> c804e5c76e3e, Add ``onupdate`` cascade to ``task_map`` table
Upgrades done
btullis@an-airflow1006:~$ sudo systemctl restart airflow-scheduler@analytics_product.service airflow-webserver@analytics_product.service