Thanks @jcrespo for helping with the ticket. I will also consult my manager and @MSantos who helped me with onboarding about the specific groups.

@RhinosF1 I updated the username.

Regarding Requested group membership:

Adding my manager @dcipoletti in the loop

A couple of questions about the metrics

• Except of the metrics provided out of the box by service-runner, are we going to use the app.metrics infrastructure provided by the package?
  • As far as I understand, our infrastructure is based on prometheus but service-runner sends statsd metrics. Is this exported to prometheus using statsd_exporter?
• For metrics that refer both to the Echo extension and the node service, do we want to send metrics from both projects?
  • Eg: # of FCM notifications on Echo and # of FCM notifications on push-notifications service

From the node debugger:

> this._socket
TLSSocket {_tlsOptions: {…}, _secureEstablished: true, _securePending: false, _newSessionPending: false, _controlReleased: true}
> this._socket.alpnProtocol
'h2'

I agree with @Mholloway also I checked the codebase and it looks like its using HTTP/2. I will try to investigate that I bit more, but I think that the HTTP/1.1 requests in the proxy logs are only the first hop to the proxy.

Thanks for the context @bearND

I submitted a patch that adds the caching in mobile-html-offline-resources. I have a question though about the following:

From firebase-admin documentation:
https://firebase.google.com/docs/reference/admin/node/admin.app.AppOptions#httpAgent

From the node-apn module documentation:
https://github.com/node-apn/node-apn#connecting-through-an-http-proxy

I tried to reproduce locally to figure out what causes this issue. It looks like even calling Parsoid directly, the citation in response also shows number [4].

In T255169#6219390, @MSantos wrote:

In T255169#6219014, @Mholloway wrote:

I got some debug logging output. Looks like openssl is having an issue with the cert. I'm not sure why it's happening in the Beta Cluster but not locally.

Jun 12 12:41:39 deployment-push-notifications01 docker[8495]: 2020-06-12T12:41:39.389Z apn Request ended with status null and responseData:
Jun 12 12:41:39 deployment-push-notifications01 docker[8495]: 2020-06-12T12:41:39.393Z apn Request error: Error [ERR_HTTP2_STREAM_CANCEL]: The pending stream has been canceled (caused by: unable to get local issuer certificate)
Jun 12 12:41:39 deployment-push-notifications01 docker[8495]: 2020-06-12T12:41:39.394Z apn Session error: Error: unable to get local issuer certificate
Jun 12 12:41:39 deployment-push-notifications01 docker[8495]: 2020-06-12T12:41:39.395Z apn Session closed

For reference, my local openssl version is OpenSSL 1.1.1 11 Sep 2018, and on deployment-push-notifications01 it's OpenSSL 1.1.1d 10 Sep 2019.

Doesn't look like APNS cert error, see T250493#6193114 with a similar error, fixed by installing ca-certificates in the pipeline image.

I run the following on the local docker env and on beta:

openssl s_client -verify 2 -connect api.sandbox.push.apple.com:443

Since most of the things we identified were metadata around the notification messages one solution could be to add a new column to store the metadata as JSON blobs, since we are not planning to query based on metadata.

Have you tried using the same device token locally with the same credentials? First thing that comes in mind is that all requests to APNS are HTTP/2 so if we go through a proxy, it should support it.

I managed to issue the certificate. Thanks @JoeWalsh for helping sorting this out!

I can see the new Push IDs but I am still not allowed to generate certificates (clicking the button directly is disabled, and web push cert doesn't show up as an option under create new certificate)

I built a minimal website to work on push notifications on Safari (so we don't need a device for the API service development).
Currently its deployed on toolforge as a static page [1] backed by [2] to register the push package as documented in the Apple documentation [3]

Sounds good, thanks for the context @JoeWalsh

@JoeWalsh Should we consider fixing it in the user level by adding notheme as a solution for all similar issues or is it just a workaround for this wiki article specifically?

CC'ing also my manager @dr0ptp4kt

Hi @colewhite I updated the purpose field in the task.

Hi @Aklapper I fixed the linked mediawiki account on my phabricator account. Now it should point to the one created by WMF Office IT.