I don't think going through an HTTP proxy is required to reach the outside world from the Beta Cluster, but just to check, I configured the service to proxy HTTP requests through deployment-urldownloader02, but the result was the same.

Have you tried using the same device token locally with the same credentials? First thing that comes in mind is that all requests to APNS are HTTP/2 so if we go through a proxy, it should support it.

Yes, the same request succeeds when running the service locally with the same credential.

I got some debug logging output. Looks like openssl is having an issue with the cert. I'm not sure why it's happening in the Beta Cluster but not locally.

Jun 12 12:41:39 deployment-push-notifications01 docker[8495]: 2020-06-12T12:41:39.389Z apn Request ended with status null and responseData:
Jun 12 12:41:39 deployment-push-notifications01 docker[8495]: 2020-06-12T12:41:39.393Z apn Request error: Error [ERR_HTTP2_STREAM_CANCEL]: The pending stream has been canceled (caused by: unable to get local issuer certificate)
Jun 12 12:41:39 deployment-push-notifications01 docker[8495]: 2020-06-12T12:41:39.394Z apn Session error: Error: unable to get local issuer certificate
Jun 12 12:41:39 deployment-push-notifications01 docker[8495]: 2020-06-12T12:41:39.395Z apn Session closed

For reference, my local openssl version is OpenSSL 1.1.1 11 Sep 2018, and on deployment-push-notifications01 it's OpenSSL 1.1.1d 10 Sep 2019.

In T255169#6219014, @Mholloway wrote:

I got some debug logging output. Looks like openssl is having an issue with the cert. I'm not sure why it's happening in the Beta Cluster but not locally.

Jun 12 12:41:39 deployment-push-notifications01 docker[8495]: 2020-06-12T12:41:39.389Z apn Request ended with status null and responseData:
Jun 12 12:41:39 deployment-push-notifications01 docker[8495]: 2020-06-12T12:41:39.393Z apn Request error: Error [ERR_HTTP2_STREAM_CANCEL]: The pending stream has been canceled (caused by: unable to get local issuer certificate)
Jun 12 12:41:39 deployment-push-notifications01 docker[8495]: 2020-06-12T12:41:39.394Z apn Session error: Error: unable to get local issuer certificate
Jun 12 12:41:39 deployment-push-notifications01 docker[8495]: 2020-06-12T12:41:39.395Z apn Session closed

For reference, my local openssl version is OpenSSL 1.1.1 11 Sep 2018, and on deployment-push-notifications01 it's OpenSSL 1.1.1d 10 Sep 2019.

Doesn't look like APNS cert error, see T250493#6193114 with a similar error, fixed by installing ca-certificates in the pipeline image.

Dryrun mode works fine for me.

The container has ca-certificates, though, as a result of your patch.

runuser@7a7b6642c4b0:/srv/service$ apt list ca-certificates
Listing... Done
ca-certificates/now 20200601~deb10u1 all [installed,local]

I run the following on the local docker env and on beta:

openssl s_client -verify 2 -connect api.sandbox.push.apple.com:443

On local env I got:

....
Verification: OK
...

and on beta:

...
Verification error: unable to get local issuer certificate
...

so its not an app specific issue

In T255169#6219390, @MSantos wrote:

In T255169#6219014, @Mholloway wrote:

I got some debug logging output. Looks like openssl is having an issue with the cert. I'm not sure why it's happening in the Beta Cluster but not locally.

Jun 12 12:41:39 deployment-push-notifications01 docker[8495]: 2020-06-12T12:41:39.389Z apn Request ended with status null and responseData:
Jun 12 12:41:39 deployment-push-notifications01 docker[8495]: 2020-06-12T12:41:39.393Z apn Request error: Error [ERR_HTTP2_STREAM_CANCEL]: The pending stream has been canceled (caused by: unable to get local issuer certificate)
Jun 12 12:41:39 deployment-push-notifications01 docker[8495]: 2020-06-12T12:41:39.394Z apn Session error: Error: unable to get local issuer certificate
Jun 12 12:41:39 deployment-push-notifications01 docker[8495]: 2020-06-12T12:41:39.395Z apn Session closed

For reference, my local openssl version is OpenSSL 1.1.1 11 Sep 2018, and on deployment-push-notifications01 it's OpenSSL 1.1.1d 10 Sep 2019.

Doesn't look like APNS cert error, see T250493#6193114 with a similar error, fixed by installing ca-certificates in the pipeline image.

That said, does it make sense to change the base image for our local env so we avoid this type of issues in the future ?

OK, I think I've found the issue: it appears that ca-certificates version 20190110 contains the GeoTrust Global CA certificate we need to connect to APNS, but version 20200601~deb10u1 (installed by default) does not. The command openssl s_client -verify 2 -connect api.sandbox.push.apple.com:443 finds the full certificate chain with the former, but fails to do so with the latter. We can fix this for now by specifically requesting ca-certificates=20190110 in the Blubberfile. (I'll add akosiaris as a sanity check on that solution.)

Change 605277 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[mediawiki/services/push-notifications@master] Blubber: Pin ca-certificates to version 20190110

https://gerrit.wikimedia.org/r/605277

Debian bug about this issue: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=962596
And the initial bug leading to the removal of the certificate from ca-certificates: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=911289

As it happens, the change has been reverted and it appears a new version will be released sometime soon: https://salsa.debian.org/debian/ca-certificates/-/commit/679daf6e9bf6fcdcb574b8029297d24836fafde0

All this being the case, I think we're safe pinning the version for the time being as I've done in the patch.

See also: https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1749843.html

And finally, for the big-picture background: https://blog.mozilla.org/security/2018/03/12/distrust-symantec-tls-certificates/

Change 605277 merged by jenkins-bot:
[mediawiki/services/push-notifications@master] Blubber: Pin ca-certificates to version 20190110

https://gerrit.wikimedia.org/r/605277

OK, APNS requests now succeed on the Beta Cluster. I'll keep this task open for tracking the upstream issue, since akosiaris commented that this is fine for beta but we shouldn't go to production with the package version pinned in this way.

If the ca-certificates update isn't published in the next week or so then maybe we're better off directly grabbing and installing the needed certificate as described at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=962596#43.

Apple's docs recommend directly installing the GeoTrust certificate.

IMPORTANT

To establish HTTP/2-based TLS sessions with APNs, you must ensure that a GeoTrust Global CA root certificate is installed on each of your providers. If a provider is running macOS, this root certificate is in the keychain by default. On other systems, this certificate might require explicit installation. You can download this certificate from the GeoTrust Root Certificates website. Here is a direct link to the certificate.

https://developer.apple.com/library/archive/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/APNSOverview.html

In T255169#6241557, @Mholloway wrote:

Apple's docs recommend directly installing the GeoTrust certificate.

Discussed with Alexandros; we're not doing this. We'll wait for the release of the fixed package.

@Mholloway, without us knowing when the fixed package will be released, will this hold up the project indefinitely. There seems to be a lack of guidance on this matter amongst the broader community besides following Apple's recommendation of installing it direction from the GeoTrust website. I remember Alexandros was not in favor of this (nor of doing these types of installations from generic sources). But what is the workaround?

Hi @akosiaris, following up on this issue with ca-certificates which we discussed in our meeting about push notifications a few weeks ago. To recap, the most recent release of ca-certificates (20200601~deb10u1) removed the GeoTrust Global CA root certificate which is required to connect to APNS. A new release candidate (20200611) reverting the removal of this certificate was prepared but never released, and the package maintainer has been unresponsive since June 12 (last messages here and here).

At this point it doesn't look like we can safely assume that the fix will be released by the time we want to launch push notifications to production. I believe we have two workaround options:

1. Pin ca-certificates to the last working version (20190110) (this is what we're currently doing)
2. Directly download and install the certificate as described at T255169#6241557.

Is either of these acceptable for launching to production? Thank you.

In T255169#6344778, @Mholloway wrote:

Hi @akosiaris, following up on this issue with ca-certificates which we discussed in our meeting about push notifications a few weeks ago. To recap, the most recent release of ca-certificates (20200601~deb10u1) removed the GeoTrust Global CA root certificate which is required to connect to APNS. A new release candidate (20200611) reverting the removal of this certificate was prepared but never released, and the package maintainer has been unresponsive since June 12 (last messages here and here).

Adding @Muehlenhoff in case he is able to push this a bit.

At this point it doesn't look like we can safely assume that the fix will be released by the time we want to launch push notifications to production. I believe we have two workaround options:

1. Pin ca-certificates to the last working version (20190110) (this is what we're currently doing)
2. Directly download and install the certificate as described at T255169#6241557.

Is either of these acceptable for launching to production? Thank you.

I don't like the latter much as handling individual CAs manually is bound to have a problematic update process. Let's see what @Muehlenhoff can add for the upstream bug, but we can keep carrying the version pin listed in 1. for now as a fix (it's also easy to revert when it's fixed).

In T255169#6344837, @akosiaris wrote:

I don't like the latter much as handling individual CAs manually is bound to have a problematic update process. Let's see what @Muehlenhoff can add for the upstream bug, but we can keep carrying the version pin listed in 1. for now as a fix (it's also easy to revert when it's fixed).

I have no idea what's going on there, the ca-certificates maintainer hasn't followed up further, there was already a ping on that task two weeks ago, so not sure pinging it again would make any difference...

Let's keep the version pin for now and revert when it's finally fixed.

This is now pending for Buster and will reach the next point update on the next point update to be released 2021-02-06:
https://tracker.debian.org/news/1225178/accepted-ca-certificates-20200601deb10u2-source-into-proposed-updates-stable-new-proposed-updates/

This was released via stable-updates, so is already available ahead of the point release: https://lists.debian.org/debian-stable-announce/2021/01/msg00000.html

Change 661089 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[mediawiki/services/push-notifications@master] Revert "Blubber: Pin ca-certificates to version 20190110"

https://gerrit.wikimedia.org/r/661089

Change 661089 merged by jenkins-bot:
[mediawiki/services/push-notifications@master] Revert "Blubber: Pin ca-certificates to version 20190110"

https://gerrit.wikimedia.org/r/661089