Page MenuHomePhabricator

Fabfur (Fabrizio Furnari)
User

Projects

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Tuesday

  • Clear sailing ahead.

User Details

User Since
May 15 2023, 9:41 AM (47 w, 6 d)
Availability
Available
IRC Nick
fabfur
LDAP User
Fabfur
MediaWiki User
FFurnari-WMF [ Global Accounts ]

Recent Activity

Thu, Apr 11

Fabfur closed T360430: esams text cp nvme upgrade as Resolved.
Thu, Apr 11, 2:46 PM · Patch-For-Review, SRE, Traffic, ops-esams, DC-Ops
Fabfur updated the task description for T360430: esams text cp nvme upgrade.
Thu, Apr 11, 9:38 AM · Patch-For-Review, SRE, Traffic, ops-esams, DC-Ops

Wed, Apr 10

Fabfur updated the task description for T360430: esams text cp nvme upgrade.
Wed, Apr 10, 8:58 AM · Patch-For-Review, SRE, Traffic, ops-esams, DC-Ops

Tue, Apr 9

Fabfur added a comment to P60144 Bash function: open puppet repo path in Gerrit web UI.

on Linux I usually use xdg-open

Tue, Apr 9, 8:27 PM · SRE

Thu, Apr 4

Fabfur updated the task description for T360430: esams text cp nvme upgrade.
Thu, Apr 4, 5:11 PM · Patch-For-Review, SRE, Traffic, ops-esams, DC-Ops
Fabfur created T361845: Add metrics to Benthos.
Thu, Apr 4, 2:36 PM · Patch-For-Review, Observability-Logging, Traffic

Wed, Apr 3

Fabfur updated the task description for T360430: esams text cp nvme upgrade.
Wed, Apr 3, 9:24 AM · Patch-For-Review, SRE, Traffic, ops-esams, DC-Ops

Tue, Apr 2

Fabfur updated the task description for T360430: esams text cp nvme upgrade.
Tue, Apr 2, 2:42 PM · Patch-For-Review, SRE, Traffic, ops-esams, DC-Ops
Fabfur added a comment to T360430: esams text cp nvme upgrade.

cp3066 has been reimaged successfully, no evidence of errors

Tue, Apr 2, 2:42 PM · Patch-For-Review, SRE, Traffic, ops-esams, DC-Ops

Wed, Mar 27

Fabfur updated the task description for T360430: esams text cp nvme upgrade.
Wed, Mar 27, 2:56 PM · Patch-For-Review, SRE, Traffic, ops-esams, DC-Ops
Fabfur added a comment to T360430: esams text cp nvme upgrade.

esams has been repooled at 12:15UTC

Wed, Mar 27, 2:55 PM · Patch-For-Review, SRE, Traffic, ops-esams, DC-Ops
Fabfur updated the task description for T360430: esams text cp nvme upgrade.
Wed, Mar 27, 12:12 PM · Patch-For-Review, SRE, Traffic, ops-esams, DC-Ops
Fabfur updated the task description for T360430: esams text cp nvme upgrade.
Wed, Mar 27, 12:00 PM · Patch-For-Review, SRE, Traffic, ops-esams, DC-Ops
Fabfur updated the task description for T360430: esams text cp nvme upgrade.
Wed, Mar 27, 11:50 AM · Patch-For-Review, SRE, Traffic, ops-esams, DC-Ops
Fabfur updated the task description for T360430: esams text cp nvme upgrade.
Wed, Mar 27, 10:02 AM · Patch-For-Review, SRE, Traffic, ops-esams, DC-Ops
Fabfur updated the task description for T360430: esams text cp nvme upgrade.
Wed, Mar 27, 7:24 AM · Patch-For-Review, SRE, Traffic, ops-esams, DC-Ops
Fabfur added a comment to T360430: esams text cp nvme upgrade.

ESAMS DC started depooling @05:58UTC

Wed, Mar 27, 6:00 AM · Patch-For-Review, SRE, Traffic, ops-esams, DC-Ops

Mon, Mar 25

Fabfur updated the task description for T358109: Install new Benthos instance on cp hosts.
Mon, Mar 25, 11:34 AM · Patch-For-Review, Data-Engineering, Observability-Logging, Traffic
Fabfur added a comment to T360642: Remove extra fields currently sent to Kafka.

meta.id and meta.request_id

meta.id is used to uniquely identify an event, and it is usually used for deduplication.

Mon, Mar 25, 10:05 AM · Event-Platform, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic

Fri, Mar 22

Fabfur closed T352098: Requesting access to "researchers" and "analytics-privatedata-users" for Xiao Xiao as Resolved.
Fri, Mar 22, 4:05 PM · Patch-For-Review, SRE, SRE-Access-Requests
Fabfur closed T360639: Grant Access to ldap/WMF for zoe as Resolved.

User added to the ldap/wmf group, please let me know if you can use these services.

Fri, Mar 22, 2:40 PM · Patch-For-Review, SRE, LDAP-Access-Requests
Fabfur closed T360367: Grant Access to ldap/wmf for Katie Coleman as Resolved.
Fri, Mar 22, 2:39 PM · Patch-For-Review, SRE, LDAP-Access-Requests
Fabfur added a comment to T360367: Grant Access to ldap/wmf for Katie Coleman.

User added to the ldap/wmf group

Fri, Mar 22, 2:36 PM · Patch-For-Review, SRE, LDAP-Access-Requests
Fabfur added a member for WMF-NDA: KColeman-WMF.
Fri, Mar 22, 2:32 PM
Fabfur added a comment to T360367: Grant Access to ldap/wmf for Katie Coleman.

Thanks, I'll notice you soon with the confirmation!

Fri, Mar 22, 2:19 PM · Patch-For-Review, SRE, LDAP-Access-Requests
Fabfur added a member for WMF-NDA: zoe.
Fri, Mar 22, 1:20 PM
Fabfur claimed T360639: Grant Access to ldap/WMF for zoe.
Fri, Mar 22, 1:10 PM · Patch-For-Review, SRE, LDAP-Access-Requests
Fabfur claimed T360367: Grant Access to ldap/wmf for Katie Coleman.
Fri, Mar 22, 1:09 PM · Patch-For-Review, SRE, LDAP-Access-Requests
Fabfur added a comment to T360367: Grant Access to ldap/wmf for Katie Coleman.

Hello, thanks for this request, could your direct manager please confirm this (it's sufficient to respond to this ticket).

Fri, Mar 22, 1:09 PM · Patch-For-Review, SRE, LDAP-Access-Requests
Fabfur created T360766: Return 403 to non HEAD|GET requests in HAProxy tls frontend.
Fri, Mar 22, 11:52 AM · Traffic
Fabfur added a comment to T360642: Remove extra fields currently sent to Kafka.

These are the fields that are sent from Benthos that aren't present in the current webrequest stream:

FWIW meta and $schema are not part of webrequest, but a requirements for EP integration: https://wikitech.wikimedia.org/wiki/Event_Platform/Schemas#Required_event_data. Usually they are managed by our node/java producers.

We don't need for sure two uuid (generate by different parts of the processing pipeline), that are expensive to generate under heavy load and could result in a potential waste of bandwidth/space on Kafka.

I need to investigate the historical reason behind both meta.id and meta.request_id, but if performance is a concern I think we can live without meta.id. meta itself is a required field in the webrequest schema, but payload should validate with missing/empty id.

kafka-jumbo and hadoop should be fine (storage wise), but I do appreciate that at webrequest scale every byte sent over the wire counts (and adds up quickly).
For my own education, do you have some datapoints that show how much overhead uuidv4 in benthos produces (cpu-wise)?

Fri, Mar 22, 8:56 AM · Event-Platform, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic

Thu, Mar 21

Fabfur created T360642: Remove extra fields currently sent to Kafka.
Thu, Mar 21, 2:53 PM · Event-Platform, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic

Wed, Mar 20

Fabfur closed T360462: Access to rua-dmarc@wikimedia.org as Resolved.

Sent information about dmarc address privately (mail) to the ticket author

Wed, Mar 20, 12:01 PM · SRE, SRE-Access-Requests, Fundraising-Backlog
Fabfur claimed T360462: Access to rua-dmarc@wikimedia.org.
Wed, Mar 20, 11:19 AM · SRE, SRE-Access-Requests, Fundraising-Backlog

Tue, Mar 19

Fabfur closed T359627: Benthos: better management for unparsable logs, a subtask of T351117: Move analytics log from Varnish to HAProxy, as Resolved.
Tue, Mar 19, 11:26 PM · Data Products, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic
Fabfur closed T359627: Benthos: better management for unparsable logs as Resolved.
Tue, Mar 19, 11:26 PM · Patch-For-Review, Data-Engineering, Observability-Logging, Traffic
Fabfur closed T360415: Review haproxy captured header lengths, a subtask of T351117: Move analytics log from Varnish to HAProxy, as Resolved.
Tue, Mar 19, 11:26 PM · Data Products, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic
Fabfur closed T360415: Review haproxy captured header lengths as Resolved.
Tue, Mar 19, 11:26 PM · Traffic
Fabfur created T360454: Better Benthos performances.
Tue, Mar 19, 5:09 PM · Patch-For-Review, Data-Engineering, Observability-Logging, Traffic
Fabfur closed T360450: Add $schema key to Benthos payload as Resolved.
Tue, Mar 19, 5:07 PM · Event-Platform, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic
Fabfur closed T360450: Add $schema key to Benthos payload, a subtask of T358109: Install new Benthos instance on cp hosts, as Resolved.
Tue, Mar 19, 5:06 PM · Patch-For-Review, Data-Engineering, Observability-Logging, Traffic
Fabfur created T360450: Add $schema key to Benthos payload.
Tue, Mar 19, 3:50 PM · Event-Platform, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic
Fabfur triaged T359627: Benthos: better management for unparsable logs as Low priority.

Even without metrics generation, this has been fixed with a small processing on the input side.
Leave the ticket in progress to add code for metric generation.

Tue, Mar 19, 10:58 AM · Patch-For-Review, Data-Engineering, Observability-Logging, Traffic
Fabfur changed the status of T358109: Install new Benthos instance on cp hosts from Open to In Progress.
Tue, Mar 19, 10:56 AM · Patch-For-Review, Data-Engineering, Observability-Logging, Traffic
Fabfur changed the status of T358109: Install new Benthos instance on cp hosts, a subtask of T351117: Move analytics log from Varnish to HAProxy, from Open to In Progress.
Tue, Mar 19, 10:55 AM · Data Products, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic
Fabfur closed T359626: Benthos: add specific unit tests for different logs, a subtask of T351117: Move analytics log from Varnish to HAProxy, as Resolved.
Tue, Mar 19, 10:54 AM · Data Products, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic
Fabfur closed T359626: Benthos: add specific unit tests for different logs as Resolved.

This is fixed

Tue, Mar 19, 10:54 AM · Patch-For-Review, Traffic
Fabfur created T360415: Review haproxy captured header lengths.
Tue, Mar 19, 10:52 AM · Traffic
Fabfur added a comment to T352098: Requesting access to "researchers" and "analytics-privatedata-users" for Xiao Xiao.

Contacted privately on separate channel to inform her about this issue

Tue, Mar 19, 9:20 AM · Patch-For-Review, SRE, SRE-Access-Requests

Mon, Mar 18

Fabfur changed the status of T359627: Benthos: better management for unparsable logs, a subtask of T351117: Move analytics log from Varnish to HAProxy, from Open to In Progress.
Mon, Mar 18, 6:21 PM · Data Products, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic
Fabfur changed the status of T359627: Benthos: better management for unparsable logs from Open to In Progress.

After some second of live logging I can confirm that the SSL handshake failure messages are correctly dropped by Benthos and not sent to Kafka topic. Thanks to @ssingh for the help!

Mon, Mar 18, 6:21 PM · Patch-For-Review, Data-Engineering, Observability-Logging, Traffic
Fabfur added a comment to T351117: Move analytics log from Varnish to HAProxy.

I've fixed some errors (now metadata field name should be correct) in the Benthos configuration. I've repooled cp4037 for about one minute (https://wikitech.wikimedia.org/wiki/Server_Admin_Log#2024-03-18) to collect some logs.

Mon, Mar 18, 4:59 PM · Data Products, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic
Fabfur closed T358105: Change HAProxy log-format to support missing information, a subtask of T351117: Move analytics log from Varnish to HAProxy, as Resolved.
Mon, Mar 18, 11:13 AM · Data Products, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic
Fabfur closed T358105: Change HAProxy log-format to support missing information as Resolved.

This can be closed as the new log format is tested and ready to be enabled fleet-wide with a hiera switch along with the Benthos installation.

Mon, Mar 18, 11:13 AM · Patch-For-Review, Data Products, Data-Engineering, Observability-Logging, Traffic

Mar 14 2024

Fabfur created T360146: Proposal: implement `dcl ssh` subcommand.
Mar 14 2024, 6:45 PM · Infrastructure-Foundations

Mar 8 2024

Fabfur added a comment to T351117: Move analytics log from Varnish to HAProxy.

Note: I've repooled cp4037 for the next days as I'll be busy on the SRE Summit to work on it.

Mar 8 2024, 3:34 PM · Data Products, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic
Fabfur created T359627: Benthos: better management for unparsable logs.
Mar 8 2024, 12:45 PM · Patch-For-Review, Data-Engineering, Observability-Logging, Traffic
Fabfur created T359626: Benthos: add specific unit tests for different logs.
Mar 8 2024, 12:40 PM · Patch-For-Review, Traffic
Fabfur added a comment to T351117: Move analytics log from Varnish to HAProxy.

Update: yesterday we modified the HAProxy log destination to send them into Benthos and repooled cp4037 for a very short time to capture "real" logs. Benthos configuration sent them to two separate Kafka topics: webrequest_text_test for "regular" logs and webrequest_text_test_error for errored (unparsable) logs.

Mar 8 2024, 12:23 PM · Data Products, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic

Mar 6 2024

Fabfur added a comment to T351117: Move analytics log from Varnish to HAProxy.

Update: Benthos is installed on cp4037 and after some minor fixes, is finally ready to ingest, process and send HAProxy logs to the temporary Kafka topic (webrequest_text_test) on the jumbo cluster. Considering that, when we'll switch on the new log destination in HAProxy and repool the host, a very large amount of data will be processed and sent, I prefer starting "manually" sending single log lines to the Benthos socket and checking that all is correct on the Kafka side.
This means that the first messages on the topic will be "artificial" and can be safely discarded.

Mar 6 2024, 11:10 AM · Data Products, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic

Mar 5 2024

Fabfur created T359178: Check statsv and eventlogging VarnishKafka instances.
Mar 5 2024, 2:16 PM · Data-Engineering, Observability-Logging, Traffic
Fabfur created T359172: Create a new temporary topic on Kafka jumbo cluster to test Benthos.
Mar 5 2024, 1:52 PM · Data Products, Data-Engineering, Observability-Logging, Traffic

Mar 4 2024

Fabfur claimed T359053: Reimage one of each Traffic hosts before magru.
Mar 4 2024, 2:12 PM · Traffic

Feb 28 2024

Fabfur closed T355158: Generate netmapper file for abuse requestctl ipblocks as Declined.
Feb 28 2024, 11:53 AM · Traffic
Fabfur updated the task description for T358109: Install new Benthos instance on cp hosts.
Feb 28 2024, 11:51 AM · Patch-For-Review, Data-Engineering, Observability-Logging, Traffic
Fabfur created T358647: hieradata for syslog/centralserver should use hash instead of array for env_variables.
Feb 28 2024, 10:08 AM · SRE Observability (FY2023/2024-Q3), observability

Feb 26 2024

Fabfur added a comment to T358105: Change HAProxy log-format to support missing information.

Applied to cp4037 the new log format to check eventual errors

Feb 26 2024, 2:52 PM · Patch-For-Review, Data Products, Data-Engineering, Observability-Logging, Traffic

Feb 22 2024

Fabfur added a comment to T355868: Migrate servers in codfw rack B2 from asw-b2-codfw to lsw1-b2-codfw.

cp2031 and cp2032 are ok and repooled

Feb 22 2024, 4:33 PM · SRE-swift-storage, ops-codfw, netops, Infrastructure-Foundations, SRE

Feb 21 2024

Fabfur created T358109: Install new Benthos instance on cp hosts.
Feb 21 2024, 2:41 PM · Patch-For-Review, Data-Engineering, Observability-Logging, Traffic
Fabfur updated the task description for T358105: Change HAProxy log-format to support missing information.
Feb 21 2024, 2:38 PM · Patch-For-Review, Data Products, Data-Engineering, Observability-Logging, Traffic
Fabfur created T358107: Change mtail configuration to ignore new fields in HAProxy logs.
Feb 21 2024, 2:29 PM · Patch-For-Review, Data Products, Data-Engineering, Observability-Logging, Traffic
Fabfur created T358105: Change HAProxy log-format to support missing information.
Feb 21 2024, 2:26 PM · Patch-For-Review, Data Products, Data-Engineering, Observability-Logging, Traffic

Feb 15 2024

Fabfur added a comment to T351117: Move analytics log from Varnish to HAProxy.

the currently suggested one is webrequest.frontend. @gmodena, the idea there is to group all webrequest topics into the same stream, by setting topics manually in stream config. Gobblin will ingest the topics configured in stream config.

We discussed this OTR in slack, and this should be feasible with the current gobblin config.

We might want to a new field to indicate the cache cluster / webrequest source the request is from. The webrequest refine job will pull the Hive webrequest_source partition from the topic name; but I think it might be best to have this info in the event data too. (I suppose the Refine job could do that, but it's probably more future proof to let the haproxy producer set the value explicitly).

Good point @Ottomata
I'd also prefer if this was set upstream by the haproxy producer. Would be useful if / when accessing the kafka topics (instead of refined data in HDFS).

cc / @Fabfur

Feb 15 2024, 11:53 AM · Data Products, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic

Feb 14 2024

Fabfur added a comment to T357479: Stop sending X-Webkit-CSP and X-Webkit-CSP-Report-Only headers.

Thanks @TheDJ for your contribution! The patch has been applied to our hosts!

Feb 14 2024, 5:15 PM · SecTeam-Processed, Security-Team, ContentSecurityPolicy, Traffic

Feb 9 2024

Fabfur added a comment to T351117: Move analytics log from Varnish to HAProxy.
Feb 9 2024, 11:46 AM · Data Products, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic
Fabfur added a comment to T351117: Move analytics log from Varnish to HAProxy.

With Benthos we have the ability to set actual metadata attached to the message (eg. with meta keyword) or simply define another meta key under the root one and set all needed fields under that.

Feb 9 2024, 10:59 AM · Data Products, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic

Feb 8 2024

Fabfur added a comment to T351117: Move analytics log from Varnish to HAProxy.

I've adapted the Benthos configuration to produce an output similar to the current (webrequest) data:

Feb 8 2024, 5:53 PM · Data Products, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic
Fabfur added a comment to T351117: Move analytics log from Varnish to HAProxy.

@xcollazo Unfortunately the timestamp is referred to the request timestamp. I can check if I can somehow use the last restart timestamp...

Feb 8 2024, 5:27 PM · Data Products, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic
Fabfur added a comment to T351117: Move analytics log from Varnish to HAProxy.

Some updates:

Feb 8 2024, 2:52 PM · Data Products, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic

Feb 7 2024

Fabfur added a comment to T351117: Move analytics log from Varnish to HAProxy.

Some updates about the ongoing work:

Feb 7 2024, 2:28 PM · Data Products, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic

Jan 17 2024

Fabfur created T355256: requestctl should fail with error if fails parsing yaml file.
Jan 17 2024, 6:26 PM · conftool, serviceops

Jan 16 2024

Fabfur added a comment to T355158: Generate netmapper file for abuse requestctl ipblocks.

Possibly related to T353910

Jan 16 2024, 9:01 PM · Traffic
Fabfur added a comment to T351117: Move analytics log from Varnish to HAProxy.

To add some information about having sequences in logs, I'd like to elaborate a bit more about our current options:

Jan 16 2024, 2:39 PM · Data Products, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic

Jan 12 2024

Fabfur added a comment to T351117: Move analytics log from Varnish to HAProxy.
  • The sequence number is also set by HAProxy (and eventually benthos if we need another one).

Oh really? I thought it was varnish that controls that, so I'd need to learn more here. Like, we have a bunch of data quality code that checks for false positives based on what we know about varnish restarts and how those affect sequence numbers. When this work goes forward, it'd be good to zoom in on these sequence numbers.

@Fabfur could you please point me to the HAProxy (and/or Benthos) documentation about the mechanism that we intend to use to generate these sequence numbers? From our side (Data Products) we'd like to start studying them to see whether we need to modify our data quality checks or not.

Jan 12 2024, 3:51 PM · Data Products, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic

Jan 9 2024

Fabfur created T354712: purged package cannot be built due to failing test.
Jan 9 2024, 11:11 PM · Patch-For-Review, Traffic

Dec 21 2023

Fabfur added a comment to T351117: Move analytics log from Varnish to HAProxy.

Do you have an estimate of the duration for which we'd be dual-writing?

I think we hope to get this done in Q3. But YMMV ¯\_(ツ)_/¯

Hm, @Fabfur does the same cache host that serves the haproxy request serve the varnish request? I.e. does haproxy only route to localhost? If so, we can probably do a bulk of the comparison analysis by enabling the haproxy producer on only a few cache hosts, and then compare the varnishkafka vs haproxy logs generated by that host.

Dec 21 2023, 6:08 PM · Data Products, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic
Fabfur added a comment to T314956: [Event Platform] Declare webrequest as an Event Platform stream.

How should we layout and name the new stream(s)?

Currently, we have webrequest_text and webrequest_upload topics. Which topic is produced to is determine by which cache_cluster the server is in. IIUC, haproxy will be colocated in the same way as varnish, so we can continue to use this layout if we want to. (cc @Fabfur to confirm).

Dec 21 2023, 4:11 PM · Patch-For-Review, Data-Engineering, Event-Platform

Dec 20 2023

Fabfur updated the task description for T353779: sre.dns.roll-restart-reboot-wikimedia-dns cookbook sometimes cannot remove downtime.
Dec 20 2023, 10:00 AM · Patch-For-Review, Traffic
Fabfur created T353779: sre.dns.roll-restart-reboot-wikimedia-dns cookbook sometimes cannot remove downtime.
Dec 20 2023, 9:58 AM · Patch-For-Review, Traffic

Dec 18 2023

Fabfur added a comment to T352876: cp4037 reimage for cookbook getting stuck at PXE boot.

cp4037 repooled

Dec 18 2023, 3:18 PM · Traffic, DC-Ops
Fabfur added a comment to T352876: cp4037 reimage for cookbook getting stuck at PXE boot.

I'm reimaging cp4037 today

Dec 18 2023, 10:09 AM · Traffic, DC-Ops

Dec 13 2023

Fabfur updated Other Assignee for T352876: cp4037 reimage for cookbook getting stuck at PXE boot, added: Fabfur.
Dec 13 2023, 12:09 PM · Traffic, DC-Ops

Dec 4 2023

Fabfur added a comment to T351117: Move analytics log from Varnish to HAProxy.

Hi @Milimetric sorry for the late reply, I'll try to answer to your question but consider we're still investigating about all pro and cons of this "migration", and for sure we'll share our thought and our action plan before moving on with this...

Dec 4 2023, 2:05 PM · Data Products, Patch-For-Review, Data-Engineering, Observability-Logging, Traffic

Nov 29 2023

Fabfur created T352291: Provide better error pages for HAProxy.
Nov 29 2023, 2:11 PM · Traffic
Fabfur closed T349244: Q1:Install cp11[00-15] and rotate into production, a subtask of T342159: Q1:rack/setup/install cp11[00-15], as Resolved.
Nov 29 2023, 12:10 PM · SRE, ops-eqiad, Traffic, DC-Ops
Fabfur closed T349244: Q1:Install cp11[00-15] and rotate into production as Resolved.

All activities for this task have been completed, refer to the other linked tasks for more details on decommissioning old hosts

Nov 29 2023, 12:10 PM · ops-eqiad, DC-Ops, Traffic, SRE
Fabfur updated the task description for T349244: Q1:Install cp11[00-15] and rotate into production.
Nov 29 2023, 12:08 PM · ops-eqiad, DC-Ops, Traffic, SRE
Fabfur added projects to T352253: Decommission task for old cp hosts (cp1075-1090): DC-Ops, ops-eqiad.
Nov 29 2023, 11:21 AM · SRE, ops-eqiad, DC-Ops, Traffic
Fabfur closed T352078: Consolidate hieradata for new eqiad cp hosts as Resolved.

This is complete

Nov 29 2023, 11:08 AM · Traffic
Fabfur created T352253: Decommission task for old cp hosts (cp1075-1090).
Nov 29 2023, 10:16 AM · SRE, ops-eqiad, DC-Ops, Traffic