Basically, we need:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • Tfinc | T105394 EPIC: Deploy Maps service to production (Q1) | |||
| Resolved | BBlack | T105076 Set up standard HTTPS Termination -> 2layer caching for maps service | |||
| Resolved | MaxSem | T108376 Implement HTCP purging npm library |
Questions from the data end:
Change 230243 had a related patch set uploaded (by BBlack):
add maps.wikimedia.org (-> maps-lb.eqiad)
Change 230242 had a related patch set uploaded (by BBlack):
create maps-lb.eqiad, repurposing osm-lb.eqiad ipv4
Change 230241 had a related patch set uploaded (by BBlack):
remove unused revdns for osm-lb.esams
Change 230246 had a related patch set uploaded (by BBlack):
Add cache_maps and related LVS config, using cp104[34]
Change 230246 merged by BBlack:
Add cache_maps and related LVS config, using cp104[34]
This is now basically working at https://maps.wikimedia.org/static/ . Don't link that anywhere or use it on wikis anywhere yet, there are probably lots of ancillary details to sort out, review, and/or fix first.
Note also that due to taking whatever available spares we have here and there, the cluster config isn't really ideal or standard. The cache cluster for this is a pair of non-SSD older nodes in eqiad, while the service hosts are over in codfw.
@BBlack, thank you for doing this on such a short notice! The results so far are amazing! We will be doing performance eval, and regen the tiles to fix the most outstanding issues. Once again, it is awesome!
@BBlack, re non-SSD -- I think there are a few smaller unused SSDs that might be used to upgrade these servers. @mark mentioned them when we were considering the storage for the codfw servers.
Also, I will close this issue and move the cache invalidation to a separate task - max is working on the htcp component, and once done, we should figure it out. Thanks!
Not done; to repeat my questions above:
It will be a new "maps" one, and yes there's probably still work to do there on my side and otto's to hook up the ends of the pipes. But it's using our same basic puppetization infrastructure as e.g. upload, text, etc.
Gotcha; awesome! Long as the pipes get hooked up I'm all happy :).
(We need a new task for that, or..?)
No idea, I'm going to ping @Ottomata Monday and talk it out with him first and then see what we need to do.
Change 230535 had a related patch set uploaded (by Ottomata):
Make camus import webrequest_maps from new maps varnish cluster
Yup.
I just created the webrequest_maps kafka topic:
kafka topic --create --topic webrequest_maps --partitions 12 --replication-factor 3
And we'll also need:
https://gerrit.wikimedia.org/r/#/c/230535/
As long as the new varnish maps cluster has varnishkafka configured in the same way as elsewhere (with topic => 'webrequest_maps'), then this should just work.
@BBlack, just a note for this in the future. We have recently started running Kafka with auto.create.topics.enable set, which means if anything starts producing to a Kafka topic that doesn't exist, that topic will be created. Unfortunately, we cannot delete topics in Kafka. It doesn't really hurt to have unused topics in Kafka, but it is just ugly and I have to look at them. I just wanted you to know in case you fire up varnishkafka instances in places with a new topic name set.
Change 230539 had a related patch set uploaded (by BBlack):
Add webrequest_maps kafka topic output for cache_maps
Change 230546 had a related patch set uploaded (by BBlack):
fix description for maps LVS
Change 230547 had a related patch set uploaded (by BBlack):
remove icinga monitoring for maps.wm.o for now (not production)
Change 230547 merged by BBlack:
remove icinga monitoring for maps.wm.o for now (not production)
Change 230539 merged by Ottomata:
Add webrequest_maps kafka topic output for cache_maps
Change 231726 had a related patch set uploaded (by BBlack):
maps.wm.o: turn back on, but only for beta self referer
Change 231726 merged by BBlack:
maps.wm.o: turn back on, but only for beta self referer
per IRC with @BBlack, closing this task as complete.
<bblack> there are outstanding spinoff issues like purging, or like actually deploying a production-ready variant of this better
So we will need more tasks to resolve them, but separate from this one.
Change 230535 abandoned by Ottomata:
Make camus import webrequest_maps from new maps varnish cluster
Reason:
This was done by joseph in another commit.