Page MenuHomePhabricator

Create vhtcpd replacement
Closed, ResolvedPublic

Description

We need to write a replacement for vhtcpd with the following capabilities:

  • read multicast HTCP cache purge requests and quickly process those
  • support kafka as an alternative input source to be ready for a future when all purges are made available on a kafka topic
  • establish multiple TCP connections to cache frontends and backends, evenly spread PURGE HTTP traffic on them to avoid issues like T241232

Details

ProjectBranchLines +/-Subject
operations/software/purgedmaster+2 -2
operations/software/purgedmaster+48 -1
operations/puppetproduction+1 -1
operations/puppetproduction+1 -0
operations/software/purgedmaster+51 -17
operations/puppetproduction+1 -0
operations/puppetproduction+4 -2
operations/software/purgedmaster+1 -1
operations/puppetproduction+1 -1
operations/software/purgedmaster+55 -13
operations/puppetproduction+28 -9
operations/puppetproduction+1 -3
operations/puppetproduction+1 -0
operations/software/purgedmaster+9 -0
operations/software/purgedmaster+22 -3
operations/software/purgedmaster+23 -12
operations/software/purgedmaster+11 -4
operations/puppetproduction+1 -1
operations/software/purgedmaster+24 -14
operations/puppetproduction+1 -0
operations/software/purgedmaster+33 -11
operations/puppetproduction+1 -0
operations/puppetproduction+13 -0
operations/puppetproduction+0 -1
operations/puppetproduction+1 -0
operations/puppetproduction+11 -9
operations/puppetproduction+3 -0
operations/software/purgedmaster+75 -27
integration/configmaster+1 -0
operations/puppetproduction+0 -1
operations/software/purgedmaster+8 -1
operations/puppetproduction+37 -18
operations/puppetproduction+21 -2
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+1 -0
operations/puppetproduction+17 -6
operations/puppetproduction+26 -0
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 588655 merged by Ema:
[operations/puppet@production] cache: ensure vhtcpd is stopped if using purged and vice versa

https://gerrit.wikimedia.org/r/588655

Change 588658 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] Release version 0.3

https://gerrit.wikimedia.org/r/588658

Change 588658 merged by Ema:
[operations/software/purged@master] Release version 0.3

https://gerrit.wikimedia.org/r/588658

Change 588659 had a related patch set uploaded (by Ema; owner: Ema):
[integration/config@master] purged: test with debian-glue-backports

https://gerrit.wikimedia.org/r/588659

Change 588660 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: test purged on cp3050

https://gerrit.wikimedia.org/r/588660

Change 588683 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] Refactor frontendWorker, add worker test

https://gerrit.wikimedia.org/r/588683

Change 588659 merged by jenkins-bot:
[integration/config@master] purged: test with debian-glue-backports

https://gerrit.wikimedia.org/r/588659

Change 588683 merged by Ema:
[operations/software/purged@master] 0.4: refactor frontendWorker, add worker test

https://gerrit.wikimedia.org/r/588683

Change 588689 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] purged: run as nobody/nogroup

https://gerrit.wikimedia.org/r/588689

Change 588698 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] purged: run one frontend and multiple backend workers

https://gerrit.wikimedia.org/r/588698

Mentioned in SAL (#wikimedia-operations) [2020-04-14T13:55:15Z] <ema> upload purged 0.4 to buster-wikimedia T249583

Change 588689 merged by Ema:
[operations/puppet@production] purged: run with DynamicUser

https://gerrit.wikimedia.org/r/588689

Change 588698 merged by Ema:
[operations/puppet@production] purged: run one frontend and multiple backend workers

https://gerrit.wikimedia.org/r/588698

Change 588660 merged by Ema:
[operations/puppet@production] cache: test purged on cp3050

https://gerrit.wikimedia.org/r/588660

Change 588722 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] Revert "cache: test purged on cp3050"

https://gerrit.wikimedia.org/r/588722

Change 588722 merged by Ema:
[operations/puppet@production] Revert "cache: test purged on cp3050"

https://gerrit.wikimedia.org/r/588722

Mentioned in SAL (#wikimedia-operations) [2020-04-14T15:52:36Z] <ema> cp3050: suspend purged testing, varnish-frontend-restart to clear mailbox lag T249583

Change 588940 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: check that the purged process is running

https://gerrit.wikimedia.org/r/588940

Change 588942 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: test purged on cp2029

https://gerrit.wikimedia.org/r/588942

Change 588940 merged by Ema:
[operations/puppet@production] cache: check that the purged process is running

https://gerrit.wikimedia.org/r/588940

Change 588942 merged by Ema:
[operations/puppet@production] cache: test purged on cp2029

https://gerrit.wikimedia.org/r/588942

Change 588959 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] 0.5: purge full request URI, not just the path

https://gerrit.wikimedia.org/r/588959

Change 588959 merged by Ema:
[operations/software/purged@master] 0.5: purge full request URI, not just the path

https://gerrit.wikimedia.org/r/588959

Mentioned in SAL (#wikimedia-operations) [2020-04-15T10:02:27Z] <ema> upload purged 0.5 to buster-wikimedia T249583

Change 588971 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: test purged on cp3050

https://gerrit.wikimedia.org/r/588971

Change 588971 merged by Ema:
[operations/puppet@production] cache: test purged on cp3050

https://gerrit.wikimedia.org/r/588971

Mentioned in SAL (#wikimedia-operations) [2020-04-15T10:25:06Z] <ema> cp3050: varnish-frontend-restart to clear mbox lag and see how long it takes to show up T249583

Change 589037 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] 0.6: use golang.org/x/net/ipv4 for the multicast reader

https://gerrit.wikimedia.org/r/589037

Change 589040 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] purged: stop passing udp port

https://gerrit.wikimedia.org/r/589040

Change 589037 merged by Ema:
[operations/software/purged@master] 0.6: use golang.org/x/net/ipv4 for the multicast reader

https://gerrit.wikimedia.org/r/589037

Mentioned in SAL (#wikimedia-operations) [2020-04-15T15:30:07Z] <ema> upload purged 0.6 to buster-wikimedia T249583

Change 589040 merged by Ema:
[operations/puppet@production] purged: stop passing udp port

https://gerrit.wikimedia.org/r/589040

Mentioned in SAL (#wikimedia-operations) [2020-04-15T15:36:28Z] <ema> cp2029,cp3050: upgrade purged to 0.6, restart varnish-fe T249583

Change 589049 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] multicast: set read buffer

https://gerrit.wikimedia.org/r/589049

Change 589218 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] Add metric 'purged_htcp_packets_total'

https://gerrit.wikimedia.org/r/589218

Change 589217 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] Add metric 'purged_frontend_backlog'

https://gerrit.wikimedia.org/r/589217

Change 589049 merged by Ema:
[operations/software/purged@master] multicast: set read buffer

https://gerrit.wikimedia.org/r/589049

Change 589217 merged by Ema:
[operations/software/purged@master] Add layer to 'purged_backlog'

https://gerrit.wikimedia.org/r/589217

Change 589218 merged by Ema:
[operations/software/purged@master] Add metric 'purged_htcp_packets_total'

https://gerrit.wikimedia.org/r/589218

Change 589264 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] 0.7: add -mcast_bufsize and new metrics

https://gerrit.wikimedia.org/r/589264

Change 589264 merged by Ema:
[operations/software/purged@master] 0.7: add -mcast_bufsize and new metrics

https://gerrit.wikimedia.org/r/589264

Mentioned in SAL (#wikimedia-operations) [2020-04-16T08:22:04Z] <ema> upload purged 0.7 to buster-wikimedia T249583

Change 589270 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: move text@esams to purged

https://gerrit.wikimedia.org/r/589270

Change 589270 merged by Ema:
[operations/puppet@production] cache: move text@esams to purged

https://gerrit.wikimedia.org/r/589270

Change 589540 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: use purged on cache_text

https://gerrit.wikimedia.org/r/589540

Change 589540 merged by Ema:
[operations/puppet@production] cache: use purged on cache_text

https://gerrit.wikimedia.org/r/589540

Change 591010 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] prometheus::node_vhtcpd: add 'ensure' attribute

https://gerrit.wikimedia.org/r/591010

Change 591010 merged by Ema:
[operations/puppet@production] prometheus::node_vhtcpd: add 'ensure' attribute

https://gerrit.wikimedia.org/r/591010

Change 591024 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] Add Host regex filtering

https://gerrit.wikimedia.org/r/591024

Change 591108 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] purged: raise 'frontend_workers'

https://gerrit.wikimedia.org/r/591108

Change 591024 merged by Ema:
[operations/software/purged@master] Add Host regex filtering

https://gerrit.wikimedia.org/r/591024

Change 591300 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] Release 0.8

https://gerrit.wikimedia.org/r/591300

Change 591108 merged by Ema:
[operations/puppet@production] purged: raise 'frontend_workers'

https://gerrit.wikimedia.org/r/591108

Change 591300 merged by Ema:
[operations/software/purged@master] Release 0.8

https://gerrit.wikimedia.org/r/591300

Change 592646 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] purged: pass host_regex

https://gerrit.wikimedia.org/r/592646

Change 592648 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: test purged on cp2030, part of cache_upload

https://gerrit.wikimedia.org/r/592648

Change 592646 merged by Ema:
[operations/puppet@production] purged: pass host_regex

https://gerrit.wikimedia.org/r/592646

Change 592648 merged by Ema:
[operations/puppet@production] cache: test purged on cp2030, part of cache_upload

https://gerrit.wikimedia.org/r/592648

Change 592891 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] Retry requests in case of errors up to sendAttempts times

https://gerrit.wikimedia.org/r/592891

Change 592891 merged by Ema:
[operations/software/purged@master] Retry requests in case of errors up to sendAttempts times

https://gerrit.wikimedia.org/r/592891

Change 592911 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: test purged on cp2030, part of cache_upload

https://gerrit.wikimedia.org/r/592911

Change 592911 merged by Ema:
[operations/puppet@production] cache: test purged on cp2030, part of cache_upload

https://gerrit.wikimedia.org/r/592911

Change 592928 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: move to purged

https://gerrit.wikimedia.org/r/592928

Change 592928 merged by Ema:
[operations/puppet@production] cache: move to purged

https://gerrit.wikimedia.org/r/592928

Change 594454 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] Add flag -frontend_delay

https://gerrit.wikimedia.org/r/594454

Change 594454 merged by Ema:
[operations/software/purged@master] Add flag -frontend_delay

https://gerrit.wikimedia.org/r/594454

Change 594684 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] Release 0.10: new flag -frontend_delay

https://gerrit.wikimedia.org/r/594684

Change 594684 merged by Ema:
[operations/software/purged@master] Release 0.10: new flag -frontend_delay

https://gerrit.wikimedia.org/r/594684