Page MenuHomePhabricator
Create Task
Maniphest T249583

Create vhtcpd replacement
Closed, ResolvedPublic
Actions

Description

We need to write a replacement for vhtcpd with the following capabilities:

  • read multicast HTCP cache purge requests and quickly process those
  • support kafka as an alternative input source to be ready for a future when all purges are made available on a kafka topic
  • establish multiple TCP connections to cache frontends and backends, evenly spread PURGE HTTP traffic on them to avoid issues like T241232

Details

SubjectRepoBranchLines +/-
Release 0.10: new flag -frontend_delayoperations/software/purgedmaster+2 -2
Add flag -frontend_delayoperations/software/purgedmaster+48 -1
cache: move to purgedoperations/puppetproduction+1 -1
cache: test purged on cp2030, part of cache_uploadoperations/puppetproduction+1 -0
Retry requests in case of errors up to sendAttempts timesoperations/software/purgedmaster+51 -17
cache: test purged on cp2030, part of cache_uploadoperations/puppetproduction+1 -0
purged: pass host_regexoperations/puppetproduction+4 -2
Release 0.8operations/software/purgedmaster+1 -1
purged: raise 'frontend_workers'operations/puppetproduction+1 -1
Add Host regex filteringoperations/software/purgedmaster+55 -13
prometheus::node_vhtcpd: add 'ensure' attributeoperations/puppetproduction+28 -9
cache: use purged on cache_textoperations/puppetproduction+1 -3
cache: move text@esams to purgedoperations/puppetproduction+1 -0
0.7: add -mcast_bufsize and new metricsoperations/software/purgedmaster+9 -0
Add metric 'purged_htcp_packets_total'operations/software/purgedmaster+22 -3
Add layer to 'purged_backlog'operations/software/purgedmaster+23 -12
multicast: set read bufferoperations/software/purgedmaster+11 -4
purged: stop passing udp portoperations/puppetproduction+1 -1
0.6: use golang.org/x/net/ipv4 for the multicast readeroperations/software/purgedmaster+24 -14
cache: test purged on cp3050operations/puppetproduction+1 -0
0.5: purge full request URI, not just the pathoperations/software/purgedmaster+33 -11
cache: test purged on cp2029operations/puppetproduction+1 -0
cache: check that the purged process is runningoperations/puppetproduction+13 -0
Revert "cache: test purged on cp3050"operations/puppetproduction+0 -1
cache: test purged on cp3050operations/puppetproduction+1 -0
purged: run one frontend and multiple backend workersoperations/puppetproduction+11 -9
purged: run with DynamicUseroperations/puppetproduction+3 -0
0.4: refactor frontendWorker, add worker testoperations/software/purgedmaster+75 -27
purged: test with debian-glue-backportsintegration/configmaster+1 -0
Revert "cache: test purged on cp3050"operations/puppetproduction+0 -1
Release version 0.3operations/software/purgedmaster+8 -1
cache: ensure vhtcpd is stopped if using purged and vice versaoperations/puppetproduction+37 -18
prometheus: job definition for purgedoperations/puppetproduction+21 -2
purged: use processorcount with explicit namespaceoperations/puppetproduction+1 -1
purged: set concurrency to 'processorcount'operations/puppetproduction+1 -1
cache: test purged on cp3050operations/puppetproduction+1 -0
cache: add support for purged to profile::cache::baseoperations/puppetproduction+17 -6
purged: add puppet moduleoperations/puppetproduction+26 -0
Show related patches Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a project: Patch-For-Review.Apr 14 2020, 9:05 AM
gerritbot added a comment.Apr 14 2020, 9:36 AM

Change 588655 merged by Ema:
[operations/puppet@production] cache: ensure vhtcpd is stopped if using purged and vice versa

https://gerrit.wikimedia.org/r/588655

• ema mentioned this in rOSPUa77d09ef5233: Distinguish between frontend and backend workers.Apr 14 2020, 9:48 AM
gerritbot added a comment.Apr 14 2020, 9:49 AM

Change 588658 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] Release version 0.3

https://gerrit.wikimedia.org/r/588658

gerritbot added a comment.Apr 14 2020, 9:51 AM

Change 588658 merged by Ema:
[operations/software/purged@master] Release version 0.3

https://gerrit.wikimedia.org/r/588658

gerritbot added a comment.Apr 14 2020, 9:54 AM

Change 588659 had a related patch set uploaded (by Ema; owner: Ema):
[integration/config@master] purged: test with debian-glue-backports

https://gerrit.wikimedia.org/r/588659

gerritbot added a comment.Apr 14 2020, 9:57 AM

Change 588660 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: test purged on cp3050

https://gerrit.wikimedia.org/r/588660

gerritbot added a comment.Apr 14 2020, 12:38 PM

Change 588683 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] Refactor frontendWorker, add worker test

https://gerrit.wikimedia.org/r/588683

gerritbot added a comment.Apr 14 2020, 12:42 PM

Change 588659 merged by jenkins-bot:
[integration/config@master] purged: test with debian-glue-backports

https://gerrit.wikimedia.org/r/588659

gerritbot added a comment.Apr 14 2020, 1:10 PM

Change 588683 merged by Ema:
[operations/software/purged@master] 0.4: refactor frontendWorker, add worker test

https://gerrit.wikimedia.org/r/588683

gerritbot added a comment.Apr 14 2020, 1:15 PM

Change 588689 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] purged: run as nobody/nogroup

https://gerrit.wikimedia.org/r/588689

gerritbot added a comment.Apr 14 2020, 1:48 PM

Change 588698 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] purged: run one frontend and multiple backend workers

https://gerrit.wikimedia.org/r/588698

Stashbot added a comment.Apr 14 2020, 1:55 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-14T13:55:15Z] <ema> upload purged 0.4 to buster-wikimedia T249583

gerritbot added a comment.Apr 14 2020, 2:00 PM

Change 588689 merged by Ema:
[operations/puppet@production] purged: run with DynamicUser

https://gerrit.wikimedia.org/r/588689

gerritbot added a comment.Apr 14 2020, 2:06 PM

Change 588698 merged by Ema:
[operations/puppet@production] purged: run one frontend and multiple backend workers

https://gerrit.wikimedia.org/r/588698

gerritbot added a comment.Apr 14 2020, 2:07 PM

Change 588660 merged by Ema:
[operations/puppet@production] cache: test purged on cp3050

https://gerrit.wikimedia.org/r/588660

Maintenance_bot removed a project: Patch-For-Review.Apr 14 2020, 2:10 PM
Stashbot added a comment.Apr 14 2020, 2:12 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-14T14:12:26Z] <ema> cp3050: resume purged testing T249583

gerritbot added a comment.Apr 14 2020, 3:48 PM

Change 588722 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] Revert "cache: test purged on cp3050"

https://gerrit.wikimedia.org/r/588722

gerritbot added a project: Patch-For-Review.Apr 14 2020, 3:48 PM

Change 588722 merged by Ema:
[operations/puppet@production] Revert "cache: test purged on cp3050"

https://gerrit.wikimedia.org/r/588722

Stashbot added a comment.Apr 14 2020, 3:52 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-14T15:52:36Z] <ema> cp3050: suspend purged testing, varnish-frontend-restart to clear mailbox lag T249583

Maintenance_bot removed a project: Patch-For-Review.Apr 14 2020, 4:10 PM
gerritbot added a comment.Apr 15 2020, 6:57 AM

Change 588940 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: check that the purged process is running

https://gerrit.wikimedia.org/r/588940

gerritbot added a project: Patch-For-Review.Apr 15 2020, 6:57 AM
gerritbot added a comment.Apr 15 2020, 7:01 AM

Change 588942 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: test purged on cp2029

https://gerrit.wikimedia.org/r/588942

gerritbot added a comment.Apr 15 2020, 8:27 AM

Change 588940 merged by Ema:
[operations/puppet@production] cache: check that the purged process is running

https://gerrit.wikimedia.org/r/588940

gerritbot added a comment.Apr 15 2020, 8:36 AM

Change 588942 merged by Ema:
[operations/puppet@production] cache: test purged on cp2029

https://gerrit.wikimedia.org/r/588942

Maintenance_bot removed a project: Patch-For-Review.Apr 15 2020, 9:10 AM
gerritbot added a comment.Apr 15 2020, 9:28 AM

Change 588959 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] 0.5: purge full request URI, not just the path

https://gerrit.wikimedia.org/r/588959

gerritbot added a project: Patch-For-Review.Apr 15 2020, 9:28 AM
gerritbot added a comment.Apr 15 2020, 9:33 AM

Change 588959 merged by Ema:
[operations/software/purged@master] 0.5: purge full request URI, not just the path

https://gerrit.wikimedia.org/r/588959

Stashbot added a comment.Apr 15 2020, 10:02 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-15T10:02:27Z] <ema> upload purged 0.5 to buster-wikimedia T249583

gerritbot added a comment.Apr 15 2020, 10:08 AM

Change 588971 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: test purged on cp3050

https://gerrit.wikimedia.org/r/588971

gerritbot added a comment.Apr 15 2020, 10:09 AM

Change 588971 merged by Ema:
[operations/puppet@production] cache: test purged on cp3050

https://gerrit.wikimedia.org/r/588971

Maintenance_bot removed a project: Patch-For-Review.Apr 15 2020, 10:10 AM
Stashbot added a comment.Apr 15 2020, 10:25 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-15T10:25:06Z] <ema> cp3050: varnish-frontend-restart to clear mbox lag and see how long it takes to show up T249583

gerritbot added a comment.Apr 15 2020, 3:09 PM

Change 589037 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] 0.6: use golang.org/x/net/ipv4 for the multicast reader

https://gerrit.wikimedia.org/r/589037

gerritbot added a project: Patch-For-Review.Apr 15 2020, 3:09 PM
gerritbot added a comment.Apr 15 2020, 3:15 PM

Change 589040 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] purged: stop passing udp port

https://gerrit.wikimedia.org/r/589040

gerritbot added a comment.Apr 15 2020, 3:26 PM

Change 589037 merged by Ema:
[operations/software/purged@master] 0.6: use golang.org/x/net/ipv4 for the multicast reader

https://gerrit.wikimedia.org/r/589037

Stashbot added a comment.Apr 15 2020, 3:30 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-15T15:30:07Z] <ema> upload purged 0.6 to buster-wikimedia T249583

gerritbot added a comment.Apr 15 2020, 3:33 PM

Change 589040 merged by Ema:
[operations/puppet@production] purged: stop passing udp port

https://gerrit.wikimedia.org/r/589040

Stashbot added a comment.Apr 15 2020, 3:36 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-15T15:36:28Z] <ema> cp2029,cp3050: upgrade purged to 0.6, restart varnish-fe T249583

Maintenance_bot removed a project: Patch-For-Review.Apr 15 2020, 4:11 PM
gerritbot added a comment.Apr 15 2020, 4:11 PM

Change 589049 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] multicast: set read buffer

https://gerrit.wikimedia.org/r/589049

gerritbot added a project: Patch-For-Review.Apr 15 2020, 4:11 PM
gerritbot added a comment.Apr 16 2020, 7:16 AM

Change 589218 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] Add metric 'purged_htcp_packets_total'

https://gerrit.wikimedia.org/r/589218

gerritbot added a comment.Apr 16 2020, 7:27 AM

Change 589217 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] Add metric 'purged_frontend_backlog'

https://gerrit.wikimedia.org/r/589217

gerritbot added a comment.Apr 16 2020, 7:31 AM

Change 589049 merged by Ema:
[operations/software/purged@master] multicast: set read buffer

https://gerrit.wikimedia.org/r/589049

gerritbot added a comment.Apr 16 2020, 8:07 AM

Change 589217 merged by Ema:
[operations/software/purged@master] Add layer to 'purged_backlog'

https://gerrit.wikimedia.org/r/589217

gerritbot added a comment.Apr 16 2020, 8:08 AM

Change 589218 merged by Ema:
[operations/software/purged@master] Add metric 'purged_htcp_packets_total'

https://gerrit.wikimedia.org/r/589218

Maintenance_bot removed a project: Patch-For-Review.Apr 16 2020, 8:10 AM
gerritbot added a comment.Apr 16 2020, 8:10 AM

Change 589264 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] 0.7: add -mcast_bufsize and new metrics

https://gerrit.wikimedia.org/r/589264

gerritbot added a project: Patch-For-Review.Apr 16 2020, 8:10 AM
gerritbot added a comment.Apr 16 2020, 8:15 AM

Change 589264 merged by Ema:
[operations/software/purged@master] 0.7: add -mcast_bufsize and new metrics

https://gerrit.wikimedia.org/r/589264

Stashbot added a comment.Apr 16 2020, 8:22 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-16T08:22:04Z] <ema> upload purged 0.7 to buster-wikimedia T249583

Stashbot added a comment.Apr 16 2020, 8:22 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-16T08:22:30Z] <ema> cp3050: upgrade purged to 0.7 T249583

gerritbot added a comment.Apr 16 2020, 8:47 AM

Change 589270 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: move text@esams to purged

https://gerrit.wikimedia.org/r/589270

gerritbot added a comment.Apr 16 2020, 8:59 AM

Change 589270 merged by Ema:
[operations/puppet@production] cache: move text@esams to purged

https://gerrit.wikimedia.org/r/589270

Maintenance_bot removed a project: Patch-For-Review.Apr 16 2020, 9:10 AM
gerritbot added a comment.Apr 17 2020, 7:38 AM

Change 589540 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: use purged on cache_text

https://gerrit.wikimedia.org/r/589540

gerritbot added a project: Patch-For-Review.Apr 17 2020, 7:38 AM
gerritbot added a comment.Apr 17 2020, 7:50 AM

Change 589540 merged by Ema:
[operations/puppet@production] cache: use purged on cache_text

https://gerrit.wikimedia.org/r/589540

Maintenance_bot removed a project: Patch-For-Review.Apr 17 2020, 8:10 AM
gerritbot added a comment.Apr 20 2020, 10:08 AM

Change 591010 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] prometheus::node_vhtcpd: add 'ensure' attribute

https://gerrit.wikimedia.org/r/591010

gerritbot added a project: Patch-For-Review.Apr 20 2020, 10:08 AM
gerritbot added a comment.Apr 20 2020, 10:23 AM

Change 591010 merged by Ema:
[operations/puppet@production] prometheus::node_vhtcpd: add 'ensure' attribute

https://gerrit.wikimedia.org/r/591010

gerritbot added a comment.Apr 20 2020, 11:10 AM

Change 591024 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] Add Host regex filtering

https://gerrit.wikimedia.org/r/591024

gerritbot added a comment.Apr 20 2020, 5:12 PM

Change 591108 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] purged: raise 'frontend_workers'

https://gerrit.wikimedia.org/r/591108

gerritbot added a comment.Apr 21 2020, 7:00 AM

Change 591024 merged by Ema:
[operations/software/purged@master] Add Host regex filtering

https://gerrit.wikimedia.org/r/591024

gerritbot added a comment.Apr 21 2020, 7:40 AM

Change 591300 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] Release 0.8

https://gerrit.wikimedia.org/r/591300

gerritbot added a comment.Apr 21 2020, 7:46 AM

Change 591108 merged by Ema:
[operations/puppet@production] purged: raise 'frontend_workers'

https://gerrit.wikimedia.org/r/591108

gerritbot added a comment.Apr 27 2020, 9:45 AM

Change 591300 merged by Ema:
[operations/software/purged@master] Release 0.8

https://gerrit.wikimedia.org/r/591300

Maintenance_bot removed a project: Patch-For-Review.Apr 27 2020, 10:11 AM
Stashbot added a comment.Apr 27 2020, 12:06 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-27T12:06:32Z] <ema> cp: upgrade purged to 0.8 T249583

gerritbot added a comment.Apr 27 2020, 12:23 PM

Change 592646 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] purged: pass host_regex

https://gerrit.wikimedia.org/r/592646

gerritbot added a project: Patch-For-Review.Apr 27 2020, 12:23 PM
gerritbot added a comment.Apr 27 2020, 12:32 PM

Change 592648 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: test purged on cp2030, part of cache_upload

https://gerrit.wikimedia.org/r/592648

gerritbot added a comment.Apr 27 2020, 1:22 PM

Change 592646 merged by Ema:
[operations/puppet@production] purged: pass host_regex

https://gerrit.wikimedia.org/r/592646

gerritbot added a comment.Apr 27 2020, 1:28 PM

Change 592648 merged by Ema:
[operations/puppet@production] cache: test purged on cp2030, part of cache_upload

https://gerrit.wikimedia.org/r/592648

gerritbot added a comment.Apr 28 2020, 9:30 AM

Change 592891 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] Retry requests in case of errors up to sendAttempts times

https://gerrit.wikimedia.org/r/592891

gerritbot added a comment.Apr 28 2020, 9:53 AM

Change 592891 merged by Ema:
[operations/software/purged@master] Retry requests in case of errors up to sendAttempts times

https://gerrit.wikimedia.org/r/592891

Maintenance_bot removed a project: Patch-For-Review.Apr 28 2020, 10:10 AM
gerritbot added a comment.Apr 28 2020, 10:41 AM

Change 592911 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: test purged on cp2030, part of cache_upload

https://gerrit.wikimedia.org/r/592911

gerritbot added a project: Patch-For-Review.Apr 28 2020, 10:41 AM
gerritbot added a comment.Apr 28 2020, 10:48 AM

Change 592911 merged by Ema:
[operations/puppet@production] cache: test purged on cp2030, part of cache_upload

https://gerrit.wikimedia.org/r/592911

gerritbot added a comment.Apr 28 2020, 12:23 PM

Change 592928 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cache: move to purged

https://gerrit.wikimedia.org/r/592928

gerritbot added a comment.Apr 29 2020, 8:14 AM

Change 592928 merged by Ema:
[operations/puppet@production] cache: move to purged

https://gerrit.wikimedia.org/r/592928

• ema closed this task as Resolved.Apr 29 2020, 8:35 AM
• ema mentioned this in T251374: Puppet cleanup after purged transition.

purged deployed fleet-wide.

Maintenance_bot removed a project: Patch-For-Review.Apr 29 2020, 9:10 AM
gerritbot added a comment.May 5 2020, 10:43 AM

Change 594454 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] Add flag -frontend_delay

https://gerrit.wikimedia.org/r/594454

gerritbot added a project: Patch-For-Review.May 5 2020, 10:43 AM
gerritbot added a comment.May 6 2020, 9:18 AM

Change 594454 merged by Ema:
[operations/software/purged@master] Add flag -frontend_delay

https://gerrit.wikimedia.org/r/594454

Maintenance_bot removed a project: Patch-For-Review.May 6 2020, 10:10 AM
gerritbot added a comment.May 6 2020, 10:21 AM

Change 594684 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/purged@master] Release 0.10: new flag -frontend_delay

https://gerrit.wikimedia.org/r/594684

gerritbot added a project: Patch-For-Review.May 6 2020, 10:21 AM
gerritbot added a comment.May 6 2020, 10:24 AM

Change 594684 merged by Ema:
[operations/software/purged@master] Release 0.10: new flag -frontend_delay

https://gerrit.wikimedia.org/r/594684

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits