Page MenuHomePhabricator

Find a way for the push service to authenticate to MediaWiki in beta and production
Closed, ResolvedPublic

Description

The push notifications service needs to be able to authenticate to MediaWiki in order to clean up subscriptions that are reported by push provider APIs as expired or otherwise invalid.

As currently implemented, the service is authenticating using the action=clientlogin API module and successfully deleting subscriptions when running locally.

Problem

This login strategy currently does not work in the Beta Cluster. $wgForceHTTPS is configured to true for all wikis in InitialiseSettings-labs.php, and as a result, all cookies that MediaWiki provides to clients include the Secure attribute. The Secure attribute prohibits clients from sending a cookie back over insecure (non-https) connections. In the Beta Cluster, we are interacting with a deployment-mediawiki instance over a secure connection, and TLS is unavailable. Since the clientlogin flow relies on cookies when generating and evaluating CSRF tokens, requests from the service are failing with badtoken errors.

It is likely that this problem will also prevent authenticating via action=clientlogin in production, since production is configured similarly and the service will likely be interacting with MediaWiki through insecure connections to a service proxy.

Possible solutions

  1. Update the wiki configurations to account for internal clients using cookies

$wgForceHTTPS pertains to how external clients may connect to MediaWiki. Forcing the Secure attribute on all cookies sent by MediaWiki if $wgForceHTTPS is true does not account for the possibility of internal clients using cookies in their interactions with MediaWiki. We could work with Platform Engineering and others to reevaluate this setting and explore alternative options that better support internal clients.

  1. Update the authentication strategy used by the push service

According to the docs on mediawiki.org, action=clientlogin is primarily intended to support interactive authentication flows. There may be better authentication options available to clients running internally to the cluster. OAuth is one option worth exploring in particular. The OAuth extension is running on all wikis in production and on beta and should be able to support OAuth authentication by the push service.

  1. Don't authenticate at all; use an internal-only rest.php endpoint instead

When a job comes to the head of the job queue, cpjobqueue makes a request to a RunSingleJob endpoint of the MediaWiki REST API (rest.php). MediaWiki is configured such that this endpoint only appears internally to the cluster and is not publicly exposed. Jobs are also signed by MediaWiki upon submission to the job queue, and this signature is verified at execution time. This could be an alternative approach to explore. See the implementation in the JobQueueEventBus class in the EventBus extension.

Event Timeline

@apaskulin is currently working on OAuth documentation that can help us evaluate and potentially implement an OAuth login strategy. In the meantime she provided the following links:

Mholloway updated the task description. (Show Details)
Mholloway updated the task description. (Show Details)

I'm in favor of the recently added option 3. Do we need to coordinate with the Platform Engineering Team for guidance and reviews?

When a job comes to the head of the job queue, cpjobqueue makes a request to a RunSingleJob endpoint of the MediaWiki REST API (rest.php). MediaWiki is configured such that this endpoint only appears internally to the cluster and is not publicly exposed. Jobs are also signed by MediaWiki upon submission to the job queue, and this signature is verified at execution time. This could be an alternative approach to explore. See the implementation in the JobQueueEventBus class in the EventBus extension.

@Mholloway Does it mean that token cleanup is going to be handled by a task or it is more of an example on how unauthenticated internal API calls work?

@Jgiannelos That was just an example of an unauthenticated, internal API call.

I have a question for @eprodromou about the internal rest endpoint approach:

  • Is it possible that this internal network layer is not accessible by the services?

Change 632480 had a related patch set uploaded (by MSantos; owner: MSantos):
[mediawiki/extensions/Echo@master] WIP: REST API invalid token deletion

https://gerrit.wikimedia.org/r/632480

So, I have a POC up with patches for Echo and the push service to implement approach number 3, I would like some input regarding it from Platform Engineering, can you take a look at it? cc/ @BPirkle, @Pchelolo and @Clarakosi

The patches are:

I'm not actually looking for code-review but rather a confirmation if this approach makes sense and if there are any questions or concerns regarding it.

Still in consideration with approach number 3, authentication concerns between push-notifications and the internal mw rest endpoint has risen, maybe this could have an input from serviceops (cc/
@Joe) and maybe Security (cc/ @sbassett).

What kind of authentication should we use between the push service and the Echo extension rest endpoint?

  1. No authentication necessary because the communication only happens internally
  2. pre-shared secret between both platforms to make sure every request comes from the push-service? Should we use public-private key approach or even use OpenSSL?

@MSantos:

  1. Are there major technical reasons why the deployment-mediawiki instance can't support TLS connections? Given T235411, it will likely have to at some point, no? Unless no other services use beta in this way and never will.
  2. I'd agree that implementing authentication for option 3 is likely a good idea as an extra layer of security, despite internal-only rest requests hopefully being pretty safe. PKI is the most standard way of doing this over potentially-unencrypted channels, even if it seems a bit heavy of a solution in this case. I suppose a shared secret could also work, perhaps as an hmac transaction, but that would be less secure and involve more consideration of replay attacks, time-based expirations, etc.

Change 636466 had a related patch set uploaded (by MSantos; owner: MSantos):
[mediawiki/services/push-notifications@master] Pass X-Forwarded-Proto header for beta cluster auth

https://gerrit.wikimedia.org/r/636466

Change 636466 merged by jenkins-bot:
[mediawiki/services/push-notifications@master] Pass X-Forwarded-Proto header for beta cluster auth

https://gerrit.wikimedia.org/r/636466

Change 637551 had a related patch set uploaded (by MSantos; owner: MSantos):
[mediawiki/services/push-notifications@master] manually set cookies received from mwapi

https://gerrit.wikimedia.org/r/637551

Change 637551 merged by jenkins-bot:
[mediawiki/services/push-notifications@master] manually set cookies received from mwapi

https://gerrit.wikimedia.org/r/637551

Change 638606 had a related patch set uploaded (by MSantos; owner: MSantos):
[mediawiki/services/push-notifications@master] set cookies domain properly

https://gerrit.wikimedia.org/r/638606

Change 638606 merged by jenkins-bot:
[mediawiki/services/push-notifications@master] set cookies domain properly

https://gerrit.wikimedia.org/r/638606

Change 643113 had a related patch set uploaded (by MSantos; owner: MSantos):
[mediawiki/services/push-notifications@master] token expiration: enforce cookies in the header

https://gerrit.wikimedia.org/r/643113

With the latest patch, the authentication is fixed for internal requests.

The reason for the authentication to fail is that the requests don't persist the cookies using the default cookie jar, because MW calls returns set-cookie headers with secure property and because it's an internal request with a host header parameter being sent, the Cookie is omitted by security reasons.

The solution:

  • enforce the cookies received by MW in the nodejs request cookie jar
  • assign the correct hostname to the cookies in the jar (the host header)
  • enforce the cookies in the request headers as well
  • pass x-forwarded-proto: https to make MW understand it's a secure request
  • persist cookies through all chained requests to achieve MW Login

Change 643113 merged by jenkins-bot:
[mediawiki/services/push-notifications@master] token expiration: enforce cookies in the header

https://gerrit.wikimedia.org/r/643113