Page MenuHomePhabricator
Create Task
Maniphest T393772

Shutdown varnishkafka webrequest instances
Closed, ResolvedPublic
Actions

Description

As the webrequest dataset source has been migrated to haproxykafka (on 1st of April), we're now ready to switch off varnishkafka instances that are still producing and sending data to the "old" kafka topics.

If everyone involved agrees, I would define the following roadmap:

  • On Monday (12 May 2025) I'll disable and stop the varnishkafka service on both text and upload cluster in MAGRU DC, to test eventual and unforeseen issues with monitoring, configuration or performances.
  • On the next week (19-23 May 2025) we can switch off varnishkafka instances on all cache hosts, on all datacenters. This is a good time to check that everything on the DE side works as expected and (eventually) do some fixes.
  • On June the 19th we will remove entirely varnishkafka, and all related configurations, from production hosts.

Let me know if this sounds like a feasible roadmap for all involved people

  • Shutdown varnishkafka (webrequest) in MAGRU DC
  • Shutdown varnishkafka (webrequest) instances on all cp hosts
  • Remove varnishkafka (webrequest) instances and related configuration from all cp hosts

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
profile:cache: remove varnishkafka (webrequest) from cp hostsoperations/puppetproduction+2 -6
hiera: disable vk (webrequest) on A:cp hostsoperations/puppetproduction+3 -2
hiera: enable vk monitoring in magru to actually remove itoperations/puppetproduction+0 -2
varnishkafka: disable webrequest monitoring if ensure => absentoperations/puppetproduction+1 -0
hiera: disable varnishkafka (webrequest) in magru text|uploadoperations/puppetproduction+4 -0
cache: add option to enable or disable varnishkafka instanceoperations/puppetproduction+8 -5
Customize query in gerrit

Related Objects
Search...

Event Timeline

Fabfur created this task.May 9 2025, 11:23 AM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptMay 9 2025, 11:23 AM
Vgutierrez added a comment.May 9 2025, 11:27 AM

Please note that you should only shutdown the webrequest instance

Ahoelzl added a comment.May 9 2025, 6:02 PM

@JAllemandou please comment on proposed timeline, looks to me its aligned with our plan / data retention.

JAllemandou added a comment.May 13 2025, 10:03 AM

Hey folks,
I'm sorry I was OoO last week and missed the timeline. I'm happy for VK to be shut-down when you wish. we don't gather data anymore on the cluster and everything has been moved, so you can clean up on your side. I've created T394011 about cleaning up on our side too.
Thank you!

Ahoelzl removed a project: Data-Engineering.May 13 2025, 5:44 PM
Fabfur added a comment.May 13 2025, 5:53 PM

Sorry the first date has been postponed to tomorrow (May 13th)

gerritbot added a comment.May 13 2025, 6:12 PM

Change #1145282 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] cache: add option to enable or disable varnishkafka instance

https://gerrit.wikimedia.org/r/1145282

gerritbot added a project: Patch-For-Review.May 13 2025, 6:12 PM
gerritbot added a comment.May 14 2025, 11:27 AM

Change #1145282 merged by Fabfur:

[operations/puppet@production] cache: add option to enable or disable varnishkafka instance

https://gerrit.wikimedia.org/r/1145282

gerritbot added a comment.May 14 2025, 1:40 PM

Change #1145948 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: disable varnishkafka in magru text|upload

https://gerrit.wikimedia.org/r/1145948

gerritbot added a comment.May 14 2025, 3:13 PM

Change #1145948 merged by Fabfur:

[operations/puppet@production] hiera: disable varnishkafka (webrequest) in magru text|upload

https://gerrit.wikimedia.org/r/1145948

Stashbot added a comment.May 14 2025, 3:25 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-14T15:25:07Z] <fabfur> removing varnishkafka from magru (T393772)

gerritbot added a comment.May 14 2025, 3:38 PM

Change #1146008 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] varnishkafka: disable webrequest monitoring if ensure => absent

https://gerrit.wikimedia.org/r/1146008

gerritbot added a comment.May 14 2025, 4:36 PM

Change #1146021 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: enable vk monitoring in magru to actually remove it

https://gerrit.wikimedia.org/r/1146021

gerritbot added a comment.May 14 2025, 5:27 PM

Change #1146008 merged by Fabfur:

[operations/puppet@production] varnishkafka: disable webrequest monitoring if ensure => absent

https://gerrit.wikimedia.org/r/1146008

gerritbot added a comment.May 15 2025, 6:55 AM

Change #1146021 merged by Fabfur:

[operations/puppet@production] hiera: enable vk monitoring in magru to actually remove it

https://gerrit.wikimedia.org/r/1146021

Fabfur updated the task description. (Show Details)May 15 2025, 7:13 AM
Milimetric awarded a token.May 16 2025, 2:15 PM
gerritbot added a comment.May 19 2025, 2:41 PM

Change #1147783 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: disable vk on A:cp hosts

https://gerrit.wikimedia.org/r/1147783

Fabfur updated the task description. (Show Details)May 19 2025, 7:13 PM
Stashbot added a comment.May 20 2025, 7:25 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-20T07:25:58Z] <fabfur> disabling varnishkafka (webrequest) on A:cp (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1147783) (T393772)

Fabfur renamed this task from Shutdown varnishkafka instances to Shutdown varnishkafka webrequest instances.May 20 2025, 7:26 AM
gerritbot added a comment.May 20 2025, 7:31 AM

Change #1147783 merged by Fabfur:

[operations/puppet@production] hiera: disable vk (webrequest) on A:cp hosts

https://gerrit.wikimedia.org/r/1147783

Fabfur changed the task status from Open to In Progress.May 20 2025, 7:39 AM
Fabfur updated the task description. (Show Details)

varnishkafka webrequest has been shut down on all cache hosts

Stashbot added a comment.May 20 2025, 7:54 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-20T07:54:38Z] <fabfur> removing varnishkafka related alerts from prometheus (https://gerrit.wikimedia.org/r/c/operations/alerts/+/1146516) (T393772)

Fabfur added a project: HaproxyKafka.May 20 2025, 11:10 AM
Jgreen mentioned this in T394777: Adjust Fundraising kafkatee topic configuration for production proxy changes..May 20 2025, 1:17 PM
Jgreen mentioned this in T395089: Low volume in new webrequest feed.May 22 2025, 9:47 PM
ssingh subscribed.Aug 28 2025, 6:59 PM

@Fabfur: I think this all done and the alerts have been removed as well. Confirming: can we close this ticket as resolved? If yes, please do so. Thanks!

Fabfur added a comment.Aug 29 2025, 9:07 AM
In T393772#11129887, @ssingh wrote:

@Fabfur: I think this all done and the alerts have been removed as well. Confirming: can we close this ticket as resolved? If yes, please do so. Thanks!

The very last task (removing varnishkafka from puppet) is still pending, I'll work on that before closing this

gerritbot added a comment.Aug 29 2025, 9:27 AM

Change #1183081 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] profile:cache: remove varnishkafka (webrequest) from cp hosts

https://gerrit.wikimedia.org/r/1183081

gerritbot added a comment.Sep 1 2025, 7:37 AM

Change #1183081 merged by Fabfur:

[operations/puppet@production] profile:cache: remove varnishkafka (webrequest) from cp hosts

https://gerrit.wikimedia.org/r/1183081

Fabfur closed this task as Resolved.Sep 1 2025, 7:49 AM

With https://gerrit.wikimedia.org/r/c/operations/puppet/+/1183081 I think we can consider this as closed. New cache hosts reimaged won't have varnishkafka references (except for statsv)

Fabfur updated the task description. (Show Details)Sep 1 2025, 7:49 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits