Page MenuHomePhabricator
Create Task
Maniphest T378578

Rollout haproxykafka on all hosts
Closed, ResolvedPublic
Actions

Description

Currently haproxykafka is running on 2 hosts for test (cp4037, cp3066) in production. As puppet configuration and Debian package are now ready, we can define a roadmap for a more structured rollout on production, since should be fairly easy to deploy it along with configuration and HAProxy modifications using hiera keys.

Timeline for deploying

  • Deployed on single hosts in ulsfo, eqsin and esams
  • Deploy on whole DCs
    • ulsfo
    • eqsin
    • drmrs
    • magru
    • esams (after December break)
    • codfw (after December break)
    • eqiad (after December break)
  • Cleanup hiera data

Useful dashboards:

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
hiera: add haproxykafka to codfwoperations/puppetproduction+4 -1
hiera: consolidate haproxykafka into common profileoperations/puppetproduction+4 -22
hiera: enable haproxykafka on eqiadoperations/puppetproduction+3 -0
hiera: enable haproxykafka on codfwoperations/puppetproduction+4 -1
hiera: enable haproxykafka on all datacentersoperations/puppetproduction+4 -16
hiera: add haproxykafka to esamsoperations/puppetproduction+1 -1
hiera: enable haproxykafka on drmrs and magruoperations/puppetproduction+6 -0
hiera: removed cp5017 host override for haproxykafkaoperations/puppetproduction+0 -3
hiera: enable haproxykafka on eqsin (round two)operations/puppetproduction+3 -0
hiera: enable haproxykafka on cp5017 for debuggingoperations/puppetproduction+3 -0
hiera: enable haproxykafka on eqsinoperations/puppetproduction+3 -0
hiera: moving haproxykafka common keys to profileoperations/puppetproduction+1 -5
hiera: split haproxykafka topics based on roleoperations/puppetproduction+2 -2
hiera: split haproxykafka topics based on roleoperations/puppetproduction+3 -7
haproxykafka: restart service on config file changesoperations/puppetproduction+5 -4
hiera: enable haproxykafka on whole ulsfo dcoperations/puppetproduction+4 -0
hiera: hpk batch_deadline on socket set to 1soperations/puppetproduction+1 -1
hiera: fix haproxykafka workers numberoperations/puppetproduction+1 -1
hiera: haproxykafka defaults to 2 workersoperations/puppetproduction+1 -1
haproxykafka: fixed default permissions for socketoperations/puppetproduction+2 -1
hiera: enable haproxykafka on cp3066 and cp4037 (again)operations/puppetproduction+6 -6
haproxykafka: use user resource due to bug in systemd::sysusersoperations/puppetproduction+18 -8
hiera: enable haproxykafka on cp3066 and cp4037operations/puppetproduction+6 -6
haproxykafka: create user with puppet (and not with deb package)operations/puppetproduction+8 -2
hiera: fix path for haproxykafka socketoperations/puppetproduction+1 -1
hiera: enable haproxykafka on cp3066 and cp4037operations/puppetproduction+6 -6
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 30 2024, 9:53 AM
JAllemandou subscribed.Oct 30 2024, 2:01 PM
gerritbot added a comment.Oct 31 2024, 7:26 AM

Change #1085308 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: enable haproxykafka on cp3066 and cp4037

https://gerrit.wikimedia.org/r/1085308

gerritbot added a project: Patch-For-Review.Oct 31 2024, 7:26 AM
gerritbot added a comment.Oct 31 2024, 11:17 AM

Change #1085308 merged by Fabfur:

[operations/puppet@production] hiera: enable haproxykafka on cp3066 and cp4037

https://gerrit.wikimedia.org/r/1085308

Stashbot added a comment.Oct 31 2024, 11:17 AM

Mentioned in SAL (#wikimedia-operations) [2024-10-31T11:17:50Z] <fabfur> install haproxykafka on cp4037 and cp3066 (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1085308) (T378578)

Stashbot added a comment.Oct 31 2024, 11:26 AM

Mentioned in SAL (#wikimedia-operations) [2024-10-31T11:26:06Z] <fabfur> reverted previous action (T378578)

Maintenance_bot removed a project: Patch-For-Review.Oct 31 2024, 11:30 AM
gerritbot added a comment.Oct 31 2024, 12:10 PM

Change #1085363 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] haproxykafka: create user with puppet (and not with deb package)

https://gerrit.wikimedia.org/r/1085363

gerritbot added a project: Patch-For-Review.Oct 31 2024, 12:10 PM
gerritbot added a comment.Oct 31 2024, 2:12 PM

Change #1085395 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: fix path for haproxykafka socket

https://gerrit.wikimedia.org/r/1085395

gerritbot added a comment.Oct 31 2024, 2:17 PM

Change #1085395 merged by Fabfur:

[operations/puppet@production] hiera: fix path for haproxykafka socket

https://gerrit.wikimedia.org/r/1085395

gerritbot added a comment.Oct 31 2024, 2:38 PM

Change #1085363 merged by Fabfur:

[operations/puppet@production] haproxykafka: create user with puppet (and not with deb package)

https://gerrit.wikimedia.org/r/1085363

gerritbot added a comment.Oct 31 2024, 2:43 PM

Change #1085399 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: enable haproxykafka on cp3066 and cp4037

https://gerrit.wikimedia.org/r/1085399

gerritbot added a comment.Oct 31 2024, 2:54 PM

Change #1085399 merged by Fabfur:

[operations/puppet@production] hiera: enable haproxykafka on cp3066 and cp4037

https://gerrit.wikimedia.org/r/1085399

Maintenance_bot removed a project: Patch-For-Review.Oct 31 2024, 3:30 PM
gerritbot added a comment.Oct 31 2024, 3:32 PM

Change #1085424 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] haproxykafka: use user function due to bug in systemd::sysusers

https://gerrit.wikimedia.org/r/1085424

gerritbot added a project: Patch-For-Review.Oct 31 2024, 3:32 PM
gerritbot added a comment.Oct 31 2024, 4:35 PM

Change #1085433 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: enable haproxykafka on cp3066 and cp4037 (again)

https://gerrit.wikimedia.org/r/1085433

gerritbot added a comment.Oct 31 2024, 4:37 PM

Change #1085424 merged by Fabfur:

[operations/puppet@production] haproxykafka: use user resource due to bug in systemd::sysusers

https://gerrit.wikimedia.org/r/1085424

gerritbot added a comment.Oct 31 2024, 4:37 PM

Change #1085433 merged by Fabfur:

[operations/puppet@production] hiera: enable haproxykafka on cp3066 and cp4037 (again)

https://gerrit.wikimedia.org/r/1085433

gerritbot added a comment.Oct 31 2024, 4:53 PM

Change #1085436 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] haproxykafka: fixed default permissions for socket

https://gerrit.wikimedia.org/r/1085436

gerritbot added a comment.Oct 31 2024, 5:29 PM

Change #1085436 merged by Fabfur:

[operations/puppet@production] haproxykafka: fixed default permissions for socket

https://gerrit.wikimedia.org/r/1085436

Maintenance_bot removed a project: Patch-For-Review.Oct 31 2024, 5:30 PM
gerritbot added a comment.Oct 31 2024, 6:58 PM

Change #1085465 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: fix haproxykafka workers number

https://gerrit.wikimedia.org/r/1085465

gerritbot added a project: Patch-For-Review.Oct 31 2024, 6:58 PM
gerritbot added a comment.Nov 5 2024, 7:56 AM

Change #1087359 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: haproxykafka defaults to 2 workers

https://gerrit.wikimedia.org/r/1087359

gerritbot added a comment.Nov 5 2024, 7:59 AM

Change #1087359 merged by Fabfur:

[operations/puppet@production] hiera: haproxykafka defaults to 2 workers

https://gerrit.wikimedia.org/r/1087359

gerritbot added a comment.Nov 5 2024, 8:36 AM

Change #1087365 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: hpk batch_deadline on socket set to 1s

https://gerrit.wikimedia.org/r/1087365

gerritbot added a comment.Nov 5 2024, 8:40 AM

Change #1087365 merged by Fabfur:

[operations/puppet@production] hiera: hpk batch_deadline on socket set to 1s

https://gerrit.wikimedia.org/r/1087365

gerritbot added a comment.Nov 5 2024, 8:46 AM

Change #1087371 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] haproxykafka: restart service on config file changes

https://gerrit.wikimedia.org/r/1087371

gerritbot added a comment.Nov 5 2024, 10:59 AM

Change #1085465 abandoned by Fabfur:

[operations/puppet@production] hiera: fix haproxykafka workers number

Reason:

superseded by 1087359

https://gerrit.wikimedia.org/r/1085465

gerritbot added a comment.Nov 6 2024, 8:46 AM

Change #1087862 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: enable haproxykafka on whole ulsfo dc

https://gerrit.wikimedia.org/r/1087862

Fabfur updated the task description. (Show Details)Nov 6 2024, 8:47 AM

Ticket description has been updated to reflect the outcome of latest meeting with @gmodena and @Ahoelzl .

gerritbot added a comment.Nov 6 2024, 10:42 AM

Change #1087862 merged by Fabfur:

[operations/puppet@production] hiera: enable haproxykafka on whole ulsfo dc

https://gerrit.wikimedia.org/r/1087862

Stashbot added a comment.Nov 6 2024, 10:43 AM

Mentioned in SAL (#wikimedia-operations) [2024-11-06T10:43:33Z] <fabfur> rolling out haproxykafka on all ULSFO cp hosts (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1087862) (T378578)

Fabfur updated the task description. (Show Details)Nov 6 2024, 10:58 AM
gerritbot added a comment.Nov 6 2024, 11:50 AM

Change #1087371 abandoned by Fabfur:

[operations/puppet@production] haproxykafka: restart service on config file changes

Reason:

not really needed, and can cause message loss

https://gerrit.wikimedia.org/r/1087371

Maintenance_bot removed a project: Patch-For-Review.Nov 6 2024, 12:30 PM
gerritbot added a comment.Nov 6 2024, 2:14 PM

Change #1087917 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: split haproxykafka topics based on role

https://gerrit.wikimedia.org/r/1087917

gerritbot added a project: Patch-For-Review.Nov 6 2024, 2:14 PM
Ahoelzl edited projects, added Data-Engineering (Q2 2024 October 1st - December 31th); removed Data-Engineering.Nov 6 2024, 3:50 PM
Ahoelzl moved this task from Next Up to Radar (External Teams) on the Data-Engineering (Q2 2024 October 1st - December 31th) board.
Ahoelzl added a subscriber: Antoine_Quhen.
gerritbot added a comment.Nov 6 2024, 4:13 PM

Change #1087917 abandoned by Fabfur:

[operations/puppet@production] hiera: split haproxykafka topics based on role

Reason:

splitting the patch

https://gerrit.wikimedia.org/r/1087917

gerritbot added a comment.Nov 6 2024, 4:16 PM

Change #1087940 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: split haproxykafka topics based on role

https://gerrit.wikimedia.org/r/1087940

gerritbot added a comment.Nov 6 2024, 4:21 PM

Change #1087940 merged by Fabfur:

[operations/puppet@production] hiera: split haproxykafka topics based on role

https://gerrit.wikimedia.org/r/1087940

Maintenance_bot removed a project: Patch-For-Review.Nov 6 2024, 4:30 PM
gerritbot added a comment.Nov 6 2024, 4:31 PM

Change #1087943 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: moving haproxykafka common keys to profile

https://gerrit.wikimedia.org/r/1087943

gerritbot added a project: Patch-For-Review.Nov 6 2024, 4:31 PM
gerritbot added a comment.Nov 7 2024, 2:17 PM

Change #1087943 merged by Fabfur:

[operations/puppet@production] hiera: moving haproxykafka common keys to profile

https://gerrit.wikimedia.org/r/1087943

Maintenance_bot removed a project: Patch-For-Review.Nov 7 2024, 2:30 PM
gerritbot added a comment.Nov 11 2024, 7:39 AM

Change #1089605 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: enable haproxykafka on eqsin

https://gerrit.wikimedia.org/r/1089605

gerritbot added a project: Patch-For-Review.Nov 11 2024, 7:39 AM
gerritbot added a comment.Nov 12 2024, 7:56 AM

Change #1089605 merged by Fabfur:

[operations/puppet@production] hiera: enable haproxykafka on eqsin

https://gerrit.wikimedia.org/r/1089605

Maintenance_bot removed a project: Patch-For-Review.Nov 12 2024, 8:30 AM
gerritbot added a comment.Nov 12 2024, 8:36 AM

Change #1090424 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: enable haproxykafka on cp5017 for debugging

https://gerrit.wikimedia.org/r/1090424

gerritbot added a project: Patch-For-Review.Nov 12 2024, 8:36 AM
gerritbot added a comment.Nov 12 2024, 8:47 AM

Change #1090424 merged by Fabfur:

[operations/puppet@production] hiera: enable haproxykafka on cp5017 for debugging

https://gerrit.wikimedia.org/r/1090424

Maintenance_bot removed a project: Patch-For-Review.Nov 12 2024, 9:30 AM
gerritbot added a comment.Nov 13 2024, 10:43 AM

Change #1090814 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: enable haproxykafka on cp6001

https://gerrit.wikimedia.org/r/1090814

gerritbot added a project: Patch-For-Review.Nov 13 2024, 10:43 AM
gerritbot added a comment.Nov 13 2024, 2:37 PM

Change #1090865 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: enable haproxykafka on eqsin (round two)

https://gerrit.wikimedia.org/r/1090865

gerritbot added a comment.Nov 13 2024, 3:11 PM

Change #1090865 merged by Fabfur:

[operations/puppet@production] hiera: enable haproxykafka on eqsin (round two)

https://gerrit.wikimedia.org/r/1090865

gerritbot added a comment.Nov 13 2024, 3:13 PM

Change #1090873 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: removed cp5017 host override for haproxykafka

https://gerrit.wikimedia.org/r/1090873

gerritbot added a comment.Nov 13 2024, 3:17 PM

Change #1090873 merged by Fabfur:

[operations/puppet@production] hiera: removed cp5017 host override for haproxykafka

https://gerrit.wikimedia.org/r/1090873

Fabfur closed subtask T379611: Investigate transport errors in eqsin as Resolved.Nov 13 2024, 3:59 PM
Fabfur updated the task description. (Show Details)
Fabfur updated the task description. (Show Details)Dec 3 2024, 4:57 PM
gerritbot added a comment.Dec 5 2024, 9:14 AM

Change #1090814 merged by Fabfur:

[operations/puppet@production] hiera: enable haproxykafka on drmrs and magru

https://gerrit.wikimedia.org/r/1090814

Stashbot added a comment.Dec 5 2024, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2024-12-05T09:15:53Z] <fabfur> deploying haproxykafka also on magru and drmrs (T378578)

Fabfur updated the task description. (Show Details)Dec 5 2024, 9:16 AM
Maintenance_bot removed a project: Patch-For-Review.Dec 5 2024, 9:30 AM
Fabfur changed the task status from Open to In Progress.Dec 6 2024, 1:24 PM
gerritbot added a comment.Jan 14 2025, 10:57 AM

Change #1111193 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: add haproxykafka to codfw

https://gerrit.wikimedia.org/r/1111193

gerritbot added a project: Patch-For-Review.Jan 14 2025, 10:57 AM
Ahoelzl edited projects, added Data-Engineering; removed Data-Engineering (Q2 2024 October 1st - December 31th).Jan 18 2025, 12:32 AM
Ahoelzl moved this task from Q2 2024 October 1st - December 31th to Tag with Radar on the Data-Engineering board.
gerritbot added a comment.Jan 27 2025, 8:30 AM

Change #1114329 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: add haproxykafka to esams

https://gerrit.wikimedia.org/r/1114329

gerritbot added a comment.Jan 27 2025, 10:31 AM

Change #1114329 merged by Fabfur:

[operations/puppet@production] hiera: add haproxykafka to esams

https://gerrit.wikimedia.org/r/1114329

Stashbot added a comment.Jan 27 2025, 10:34 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-27T10:34:05Z] <fabfur> installing haproxykafka on esams (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1114329) (T378578)

Fabfur updated the task description. (Show Details)Jan 27 2025, 10:34 AM
gerritbot added a comment.Jan 27 2025, 2:37 PM

Change #1114392 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: enable haproxykafka on all datacenters

https://gerrit.wikimedia.org/r/1114392

gerritbot added a comment.Jan 27 2025, 4:20 PM

Change #1114415 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: enable haproxykafka on codfw

https://gerrit.wikimedia.org/r/1114415

gerritbot added a comment.Jan 27 2025, 4:23 PM

Change #1114417 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: enable haproxykafka on eqiad

https://gerrit.wikimedia.org/r/1114417

gerritbot added a comment.Jan 27 2025, 4:26 PM

Change #1114392 abandoned by Fabfur:

[operations/puppet@production] hiera: enable haproxykafka on all datacenters

Reason:

splitted into more specific changes

https://gerrit.wikimedia.org/r/1114392

gerritbot added a comment.Jan 28 2025, 9:26 AM

Change #1114415 merged by Fabfur:

[operations/puppet@production] hiera: enable haproxykafka on codfw

https://gerrit.wikimedia.org/r/1114415

Stashbot added a comment.Jan 28 2025, 9:27 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-28T09:27:52Z] <fabfur> installing/enabling haproxykafka on codfw (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1114415) (T378578)

Fabfur updated the task description. (Show Details)Jan 28 2025, 9:28 AM
gerritbot added a comment.Jan 28 2025, 1:30 PM

Change #1114417 merged by Fabfur:

[operations/puppet@production] hiera: enable haproxykafka on eqiad

https://gerrit.wikimedia.org/r/1114417

Stashbot added a comment.Jan 28 2025, 1:33 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-28T13:33:30Z] <fabfur> installing/enabling haproxykafka on eqiad (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1114417) (T378578)

Fabfur updated the task description. (Show Details)Jan 28 2025, 1:34 PM
gerritbot added a comment.Jan 28 2025, 2:22 PM

Change #1114728 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: consolidate haproxykafka into common profile

https://gerrit.wikimedia.org/r/1114728

gerritbot added a comment.Jan 29 2025, 8:43 AM

Change #1114728 merged by Fabfur:

[operations/puppet@production] hiera: consolidate haproxykafka into common profile

https://gerrit.wikimedia.org/r/1114728

Fabfur closed this task as Resolved.Jan 29 2025, 8:45 AM
Fabfur updated the task description. (Show Details)
gerritbot added a comment.Feb 2 2025, 9:08 PM

Change #1111193 abandoned by Fabfur:

[operations/puppet@production] hiera: add haproxykafka to codfw

Reason:

Not useful anymore

https://gerrit.wikimedia.org/r/1111193

Maintenance_bot removed a project: Patch-For-Review.Feb 2 2025, 9:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits