There's a script in the internal frack "packages" repository that fetches the package and reports the portion of changelog associated with the latest package. The deb package is then added to the frack internal repository using "reprepro includedeb".

The basic install of dagster and its dependencies has been puppetized. Puppet also configures a pip-audit script to check for updates in the virtualenv and emails with success (clean audit) or an alert (patches are available). There's no project configuration in puppet yet, this part will be developed with BDC.

This needs some thought and investigation. Maybe we can just modify djangobannerstats to drop the additional logs into pgehres.landingpageimpressions_raw even though the new rows will be missing a couple of fields. This works with the existing schema, but we'd need to make sure anything querying that table will handle/ignore rows missing those fields.

More on vulnerability tracking. This isn't awesome but:

https://github.com/minio/minio/security executive summary: watch the blog :-|

In T360779#9849539, @MoritzMuehlenhoff wrote:

In T360779#9842162, @Jgreen wrote:

In T360779#9841199, @MoritzMuehlenhoff wrote:

@Jgreen: Just to doublecheck, the certificate expiry is tracked via monitoring internal to fr-tech (or some manual equialent like a gcal entry)? Just want to make sure nothing else is needed before I clean out the old cergen definitions for the previously used Kafka cert.

I added a local nagios/icinga check, once the above-linked commit is merged we should be good to go.

With the patch merged, can you please doublecheck the cert monitoring is now working fine for you? Then as the last cleanup step, I'd go ahead and remove the legacy cert from cergen.

In T360779#9841199, @MoritzMuehlenhoff wrote:

@Jgreen: Just to doublecheck, the certificate expiry is tracked via monitoring internal to fr-tech (or some manual equialent like a gcal entry)? Just want to make sure nothing else is needed before I clean out the old cergen definitions for the previously used Kafka cert.

Looks like coincidental rolling kafka restarts.

This isn't limited to the two hosts I mentioned above. It seems like at any given time one of the kafka-jumbo is refusing connections.

Hmmmm. I'm seeing this occasionally:

In T360779#9789862, @MoritzMuehlenhoff wrote:

@Dwisehaupt @Jgreen The kafka cert issued by the PKI is now getting deployed to /etc/fr-tech-kafka-client on cumin1002/cumin2002. Could you please sync it to fr-tech and test/deploy instead of the old cergen-issued cert? When this has been confirmed to work fine, we can add a systemd timer which sends a notification if the key renewal is forthcoming.

I ended up using nginx "access_by_lua_file" to use a lua function to cycle through a list of regexes (roughly one per package) and return a 403 if there's no match.

(Puppetized) progress so far: I have an nginx https vhost running on tcp/444, and have configured pip to route everything to that host/port as a mirror. That vhost does a proxy_pass to either pypi or the package mirror depending on the request, and it restricts requests to GETs. It also modifies the response content on indexes to rewrite host in package URLs, so that traffic also goes through the proxy.

Bumping this task back to "Up Next" because there is renewed interest from FR Analytics on whether this is feasible.

Now on to pip-audit, which has the https://pypi.org/... address hardcoded in lib/python3.11/site-packages/pip_audit/_service/pypi.py.

Fixed by rewriting the URL in the response in nginx.

Scratch that. Pip is just getting the index files from pypi. Back to the drawing board.

I looked at using pip with a proxy or mirror. The proxy behavior was not what I expected. With "proxy = http://..." pip attempts to tunnel through the proxy to remote port 443. This doesn't fit our existing nginx filtering+caching proxy scheme which is mainly for apt. I had better luck treating the proxy as a mirror. I also ran into an issue around SNI, since our existing configuration didn't employ this when connecting to the upstream server.

In T360779#9688343, @elukey wrote:

Hi Jeff!

After a chat with Moritz we agreed that the simplest solution would be to create the cert via puppet in production on some host (we need to figure out which one) and then you can copy the necessary files over to Fundraising when needed.

Caveat: the TLS certs issued by the Kafka intermediate CA last for 180 days, so puppet will create a new cert twice a year.

The code to generate the client cert should be the same (or very similar) to what we do for varnishkafka in profile::cache::kafka::certificate.

In T360907#9664537, @jcrespo wrote:

@DBu-WMF Hi, we are discussing how to proceed, as handling postmaster access is a new process for us.

The first question is what domain you want email from?

Switched to the new cert and we're still getting an A+ from Qualy SSL Labs, so this is done!

We've received the new certificate and deployed it to the backup servers at codfw. So far things look good, it's showing up here https://crt.sh/?id=12150254625 . We'll give it a couple more days before deploying it to the live servers.

Done!

@jhathaway I am fairly confident what we're seeing in the postmaster tools is misleading reporting based on no data. Looking at all the dmarc feedback we collected via the feedback address, since 6/23 there are only a handful of reports which are all dkim/spf failures from non-wmf IPs. So I think there is no legitimate mail being sent with this domain, other than its use in the envelope sender for the passing mail above.

@jhathaway looking at examples of normal TY emails from civicrm, the ARC-Authentication-Results header shows dkim=pass and spf=pass but, as expected, it is signed for the @wikimedia.org domain of the message header. So I suspect these messages are being counted in the postmaster tools success rate for domain wikimedia.org.

@jhathaway interesting, I see what you mean re. dkim and dmarc authentication success rate for the domain. I will investigate tomorrow!

In T356221#9519268, @jhathaway wrote:

@Dwisehaupt postmaster tools shows dmarc failures for this domain. Do you know where it is being actively used in emails being sent out?

I got the software install, schema fixes, and basic database upgrade to work in testing. Next step is to try upgrading the live instance and make sure various charts and reports work properly.

Switched project tags because the documentation referenced on this task is for FR Tech not FR Tech Operations.

In T354575#9447346, @Tsevener wrote:

@Jgreen Probably a bug. Is this for the payment methods endpoint? If so, it's possible we are fetching multiple times for each app language they have set up in the app (although I would have expected most users to be at 1). This was the same placement as our previous wikifeeds announcement endpoint. We can look at optimizing this.

We scrape some metrics from logs that could be informative and/or useful for comparison re. this task: https://frmon.wikimedia.org/d/xbcwsjY4k/payment-transaction-outcomes?orgId=1&refresh=1m

In T348533#9239869, @Ejegg wrote:

@Elliott Eggleston what was the reason we set up this alert again?

The main reason for the queue alerts is to remind us if we turn the consumers off and forget to turn them on again.