Page MenuHomePhabricator
Create Task
Maniphest T340983

provide haproxy silent-drop support for port 80 as well
Closed, ResolvedPublic
Actions

Description

Given the issues spotted while debugging T339898 haproxy would benefit from porting the silent-drop approach used on port 443 for port 80 as well

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
hiera: apply silent-drop on port 80 to all cp hostsoperations/puppetproduction+43 -287
hiera: apply silent-drop on port 80 to eqiad cp hostsoperations/puppetproduction+46 -0
hiera: apply silent-drop on port 80 to drmrs cp hostsoperations/puppetproduction+46 -0
hiera: apply silent-drop on port 80 to ulsfo cp hostsoperations/puppetproduction+47 -0
hiera: apply silent-drop on port 80 to all eqsin cp hostsoperations/puppetproduction+11 -52
hiera: add silent-drop directives for http frontendoperations/puppetproduction+186 -35
haproxy: fix variable type and better namingoperations/puppetproduction+11 -11
haproxy: support different actions for tls and http frontendoperations/puppetproduction+119 -71
haproxy: support different actions for tls and http frontendoperations/puppetproduction+117 -71
Customize query in gerrit

Related Objects
Search...

Event Timeline

Vgutierrez created this task.Jul 3 2023, 1:02 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 3 2023, 1:02 PM
Vgutierrez triaged this task as Medium priority.Jul 3 2023, 1:02 PM
Fabfur added a comment.Jul 3 2023, 2:23 PM

Just as reminder:

As agreed with @Vgutierrez we decided to split the current haproxy acls/other actions per frontend in hieradata, eg.:

profile::cache::haproxy::acls:
  tls:
    - name: 'too_many_concurrent_queries'
      criterion: 'sc0_trackers(httpreqrate)'
      operator: 'ge'
      value: '2000'
      [...]
  http:
    - name: 'too_much_recent_concurrency'
      # Add hysteresis.
      criterion: 'sc0_gpc0_rate(httpreqrate)'
      operator: 'gt'
      value: '0'
      [...]

This will require templates and hiera files modifications and extensive tests.

gerritbot added a comment.Jul 3 2023, 3:23 PM

Change 935095 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] haproxy: support different actions for tls and http frontend

https://gerrit.wikimedia.org/r/935095

gerritbot added a project: Patch-For-Review.Jul 3 2023, 3:23 PM
gerritbot added a comment.Jul 5 2023, 2:03 PM

Change 935095 merged by Fabfur:

[operations/puppet@production] haproxy: support different actions for tls and http frontend

https://gerrit.wikimedia.org/r/935095

Maintenance_bot removed a project: Patch-For-Review.Jul 5 2023, 2:12 PM
gerritbot added a comment.Jul 5 2023, 2:48 PM

Change 935760 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] haproxy: support different actions for tls and http frontend

https://gerrit.wikimedia.org/r/935760

gerritbot added a project: Patch-For-Review.Jul 5 2023, 2:48 PM
Stashbot added a comment.Jul 6 2023, 8:17 AM

Mentioned in SAL (#wikimedia-operations) [2023-07-06T08:17:42Z] <fabfur> disabling puppet temporary on cp1075.eqiad.wmnet, cp2027.codfw.wmnet, cp3050.esams.wmnet to apply 935760 (T340983)

gerritbot added a comment.Jul 6 2023, 8:19 AM

Change 935760 merged by Fabfur:

[operations/puppet@production] haproxy: support different actions for tls and http frontend

https://gerrit.wikimedia.org/r/935760

Maintenance_bot removed a project: Patch-For-Review.Jul 6 2023, 8:29 AM
gerritbot added a comment.Jul 6 2023, 9:33 AM

Change 935988 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] haproxy: fix variable type and better naming

https://gerrit.wikimedia.org/r/935988

gerritbot added a project: Patch-For-Review.Jul 6 2023, 9:34 AM
gerritbot added a comment.Jul 6 2023, 10:33 AM

Change 935988 merged by Fabfur:

[operations/puppet@production] haproxy: fix variable type and better naming

https://gerrit.wikimedia.org/r/935988

Maintenance_bot removed a project: Patch-For-Review.Jul 6 2023, 11:10 AM
gerritbot added a comment.Jul 10 2023, 1:00 PM

Change 936701 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: add silent-drop directives for http frontend

https://gerrit.wikimedia.org/r/936701

gerritbot added a project: Patch-For-Review.Jul 10 2023, 1:00 PM
gerritbot added a comment.Jul 13 2023, 10:08 AM

Change 936701 merged by Fabfur:

[operations/puppet@production] hiera: add silent-drop directives for http frontend

https://gerrit.wikimedia.org/r/936701

Maintenance_bot removed a project: Patch-For-Review.Jul 13 2023, 10:11 AM
gerritbot added a comment.Jul 13 2023, 7:14 PM

Change 938002 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: apply silent-drop on port 80 to all eqsin cp hosts

https://gerrit.wikimedia.org/r/938002

gerritbot added a project: Patch-For-Review.Jul 13 2023, 7:14 PM
Stashbot added a comment.Jul 17 2023, 8:30 AM

Mentioned in SAL (#wikimedia-operations) [2023-07-17T08:30:12Z] <fabfur> disable puppet on all cp* hosts in eqsin to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/938002 (T340983)

gerritbot added a comment.Jul 17 2023, 8:32 AM

Change 938002 merged by Fabfur:

[operations/puppet@production] hiera: apply silent-drop on port 80 to all eqsin cp hosts

https://gerrit.wikimedia.org/r/938002

Stashbot added a comment.Jul 17 2023, 8:51 AM

Mentioned in SAL (#wikimedia-operations) [2023-07-17T08:51:50Z] <fabfur> enable puppet on A:cp-eqsin to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/938002 (T340983)

Maintenance_bot removed a project: Patch-For-Review.Jul 17 2023, 9:10 AM
gerritbot added a comment.Jul 17 2023, 9:26 AM

Change 938807 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: apply silent-drop on port 80 to ulsfo cp hosts

https://gerrit.wikimedia.org/r/938807

gerritbot added a project: Patch-For-Review.Jul 17 2023, 9:26 AM
Stashbot added a comment.Jul 17 2023, 9:44 AM

Mentioned in SAL (#wikimedia-operations) [2023-07-17T09:44:45Z] <fabfur> disabled puppet on A:cp hosts in ulsfo to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/938807 (T340983)

gerritbot added a comment.Jul 17 2023, 9:45 AM

Change 938807 merged by Fabfur:

[operations/puppet@production] hiera: apply silent-drop on port 80 to ulsfo cp hosts

https://gerrit.wikimedia.org/r/938807

Stashbot added a comment.Jul 17 2023, 9:48 AM

Mentioned in SAL (#wikimedia-operations) [2023-07-17T09:48:35Z] <fabfur> enabled puppet on A:cp hosts in ulsfo to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/938807 (T340983) (hosts will run puppet with the usual schedule)

Maintenance_bot removed a project: Patch-For-Review.Jul 17 2023, 10:11 AM
Stashbot added a comment.Jul 17 2023, 12:58 PM

Mentioned in SAL (#wikimedia-operations) [2023-07-17T12:58:05Z] <fabfur> disabled puppet on A:cp-codfw to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/938840 (T340983)

Stashbot added a comment.Jul 17 2023, 1:04 PM

Mentioned in SAL (#wikimedia-operations) [2023-07-17T13:04:40Z] <fabfur> run puppet on cp2027 to deploy https://gerrit.wikimedia.org/r/c/operations/puppet/+/938840 (T340983)

Stashbot added a comment.Jul 17 2023, 1:07 PM

Mentioned in SAL (#wikimedia-operations) [2023-07-17T13:07:16Z] <fabfur> enabled puppet on A:cp-codfw to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/938840 (T340983) (hosts will run puppet with the usual schedule)

gerritbot added a comment.Jul 18 2023, 7:47 AM

Change 938902 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: apply silent-drop on port 80 to drmrs cp hosts

https://gerrit.wikimedia.org/r/938902

gerritbot added a project: Patch-For-Review.Jul 18 2023, 7:47 AM
Stashbot added a comment.Jul 18 2023, 8:13 AM

Mentioned in SAL (#wikimedia-operations) [2023-07-18T08:13:22Z] <fabfur> disable puppet on A:cp-drmrs to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/938902/ (T340983)

gerritbot added a comment.Jul 18 2023, 8:15 AM

Change 938902 merged by Fabfur:

[operations/puppet@production] hiera: apply silent-drop on port 80 to drmrs cp hosts

https://gerrit.wikimedia.org/r/938902

Stashbot added a comment.Jul 18 2023, 8:17 AM

Mentioned in SAL (#wikimedia-operations) [2023-07-18T08:17:47Z] <fabfur> enable puppet on A:cp-drmrs for https://gerrit.wikimedia.org/r/c/operations/puppet/+/938902/ (T340983) (hosts will run puppet with the usual schedule)

Maintenance_bot removed a project: Patch-For-Review.Jul 18 2023, 8:30 AM
gerritbot added a comment.Jul 18 2023, 8:34 AM

Change 939235 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: apply silent-drop on port 80 to eqiad cp hosts

https://gerrit.wikimedia.org/r/939235

gerritbot added a project: Patch-For-Review.Jul 18 2023, 8:34 AM
Stashbot added a comment.Jul 18 2023, 8:55 AM

Mentioned in SAL (#wikimedia-operations) [2023-07-18T08:55:28Z] <fabfur> disable puppet on A:cp-eqiad to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/939235 (T340983)

gerritbot added a comment.Jul 18 2023, 8:56 AM

Change 939235 merged by Fabfur:

[operations/puppet@production] hiera: apply silent-drop on port 80 to eqiad cp hosts

https://gerrit.wikimedia.org/r/939235

Stashbot added a comment.Jul 18 2023, 8:58 AM

Mentioned in SAL (#wikimedia-operations) [2023-07-18T08:58:37Z] <fabfur> enable puppet on A:cp-eqiad for https://gerrit.wikimedia.org/r/939235 (T340983) (hosts will run puppet with the usual schedule)

Maintenance_bot removed a project: Patch-For-Review.Jul 18 2023, 9:11 AM
gerritbot added a comment.Jul 18 2023, 9:16 AM

Change 939242 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: apply silent-drop on port 80 to all cp hosts

https://gerrit.wikimedia.org/r/939242

gerritbot added a project: Patch-For-Review.Jul 18 2023, 9:16 AM
Stashbot added a comment.Jul 18 2023, 9:52 AM

Mentioned in SAL (#wikimedia-operations) [2023-07-18T09:52:39Z] <fabfur> disable puppet on A:cp-esams to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/939242 (T340983)

gerritbot added a comment.Jul 18 2023, 10:01 AM

Change 939242 merged by Fabfur:

[operations/puppet@production] hiera: apply silent-drop on port 80 to all cp hosts

https://gerrit.wikimedia.org/r/939242

Stashbot added a comment.Jul 18 2023, 10:02 AM

Mentioned in SAL (#wikimedia-operations) [2023-07-18T10:02:25Z] <fabfur> enable puppet on A:cp-esams for https://gerrit.wikimedia.org/r/939235 (T340983) (hosts will run puppet with the usual schedule)

Maintenance_bot removed a project: Patch-For-Review.Jul 18 2023, 10:10 AM
Fabfur closed this task as Resolved.Jul 18 2023, 10:15 AM
Fabfur claimed this task.

The HAProxy configuration on all DCs has been updated to apply silent-drop to abusive clients hitting port 80, as been already done for port 443.

To check (eg. from cumin) if HAProxy is "silent-dropping" connections:

For port 443:
sudo cumin --ignore-exit-codes A:cp 'journalctl -u haproxy --since=-1h | grep silent-drop_for'

For port 80:
sudo cumin --ignore-exit-codes A:cp 'journalctl -u haproxy --since=-1h | grep silent-drop_port80_for'

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits