Page MenuHomePhabricator
Create Task
Maniphest T341488

Split Thanos components from thanos-fe hosts into titan hosts
Closed, ResolvedPublic
Actions

Description

This task tracks the implementation of splitting Thanos components from its current home (thanos-fe* hosts). In other words, leaving thanos-fe / thanos-be to purely run Swift for storage. The new hosts and role will be named titan

The following steps are @fgiunchedi's proposal on how to proceed:

  • Decide on a name for the hosts/project (This is by far the hardest part. Filippo' suggestion are sth related to MCU's Thanos, for example anything short will do: vision, tony, glove, etc)
  • Procure hardware: T341237 T341236
  • Create new puppet roles to run Thanos only and apply the role to the new hosts (T341999)
  • Pool the new hosts for thanos-query LVS service only, verify queries can be served as expected (T341999)
  • Depool the thanos-fe hosts from thanos-query LVS service
  • Clean up the existing puppet role thanos::frontend T346143

Details

SubjectRepoBranchLines +/-
Update role contact for thanos frontendoperations/puppetproduction+0 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

fgiunchedi created this task.Jul 10 2023, 3:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 10 2023, 3:17 PM
fgiunchedi mentioned this in Unknown Object (Task).Jul 10 2023, 3:26 PM
fgiunchedi mentioned this in Unknown Object (Task).
fgiunchedi added subscribers: MatthewVernon, Eevans.Jul 10 2023, 3:32 PM

@MatthewVernon @Eevans please let me know what you think of the above proposal. I was imagining the final state to be thanos-fe / thanos-be running only Swift, and the new hardware to be running Thanos only.

fgiunchedi added a project: User-fgiunchedi.Jul 10 2023, 3:42 PM
Eevans added a comment.Jul 10 2023, 8:00 PM
In T341488#9001995, @fgiunchedi wrote:

@MatthewVernon @Eevans please let me know what you think of the above proposal. I was imagining the final state to be thanos-fe / thanos-be running only Swift, and the new hardware to be running Thanos only.

All seems reasonable to me!

P.S. Isn't Titan the home of Thanos? ;)

fgiunchedi added a comment.Jul 11 2023, 7:44 AM
In T341488#9003223, @Eevans wrote:
In T341488#9001995, @fgiunchedi wrote:

@MatthewVernon @Eevans please let me know what you think of the above proposal. I was imagining the final state to be thanos-fe / thanos-be running only Swift, and the new hardware to be running Thanos only.

All seems reasonable to me!

P.S. Isn't Titan the home of Thanos? ;)

very good point! I'd be for titan for sure

MatthewVernon added a comment.EditedJul 13 2023, 3:01 PM

It might not be possible, but if we could end up with the thanos-fe* nodes running swift::*classes that would be nice. It feels like it ought to be doable once they're not also running thanos?

That notwithstanding, this seems like a good approach, thank you. I have no further opinions to offer on naming things :)

fgiunchedi added a comment.Jul 14 2023, 10:21 AM
In T341488#9012959, @MatthewVernon wrote:

It might not be possible, but if we could end up with the thanos-fe* nodes running swift::*classes that would be nice. It feels like it ought to be doable once they're not also running thanos?

Yes that's definitely doable and desirable too!

That notwithstanding, this seems like a good approach, thank you. I have no further opinions to offer on naming things :)

Thank you, unless I hear differently by beginning of next week I'm going with titan as the hostname and role naming

fgiunchedi renamed this task from Split Thanos components from thanos-fe hosts to Split Thanos components from thanos-fe hosts into titan hosts.Jul 17 2023, 2:26 PM
fgiunchedi updated the task description. (Show Details)
fgiunchedi updated the task description. (Show Details)
fgiunchedi moved this task from Backlog to Up next on the User-fgiunchedi board.Jul 18 2023, 9:03 AM
lmata triaged this task as Medium priority.Jul 18 2023, 5:08 PM
lmata added a project: SRE Observability (FY2023/2024-Q1).
lmata moved this task from Inbox to Prioritized on the Observability-Metrics board.
lmata moved this task from Inbox to Up next on the SRE Observability (FY2023/2024-Q1) board.
KOfori subscribed.Aug 2 2023, 4:38 PM
fgiunchedi moved this task from Up next to Doing on the User-fgiunchedi board.Sep 12 2023, 8:07 AM
fgiunchedi updated the task description. (Show Details)Sep 12 2023, 8:12 AM
fgiunchedi updated the task description. (Show Details)Sep 12 2023, 12:12 PM
fgiunchedi added a comment.EditedSep 12 2023, 12:24 PM

re: the last point, namely cleaning up thanos components off thanos-fe (therefore leaving only swift) I initially thought of going the ensure => absent route, though that seems more trouble than removing the thanos profiles from thanos::frontend role and roll-reimage the thanos-fe hosts. What do you think @MatthewVernon ?

MatthewVernon added a comment.Sep 12 2023, 1:16 PM

Yeah, if it's easier to just reimage them (especially if that can be done with swift::* profiles), I'm entirely happy for you to do that - just let me know when so I don't start rebooting the others at the same time :-)

fgiunchedi closed subtask T341999: Create 'titan' role and put new hosts in service as Resolved.Sep 12 2023, 1:24 PM
fgiunchedi reopened subtask T341999: Create 'titan' role and put new hosts in service as Open.Sep 12 2023, 1:42 PM
Stashbot added a comment.Sep 13 2023, 12:17 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-13T12:17:39Z] <godog> pool only titan hosts for thanos-web and thanos-query services - T341488

fgiunchedi closed subtask T341999: Create 'titan' role and put new hosts in service as Resolved.Sep 13 2023, 12:17 PM
fgiunchedi updated the task description. (Show Details)
lmata awarded a token.Sep 13 2023, 2:48 PM
fgiunchedi updated the task description. (Show Details)Sep 14 2023, 9:52 AM
fgiunchedi closed this task as Resolved.Oct 4 2023, 11:43 AM
fgiunchedi claimed this task.
fgiunchedi closed subtask T346143: Remove thanos components from thanos-fe role and reimage thanos-fe hosts as Resolved.
fgiunchedi updated the task description. (Show Details)

This is done, we have split titan* hosts and thanos-fe*. The former running thanos components and the latter running swift-proxy to serve thanos-swift object storage

lmata moved this task from Up next to Done on the SRE Observability (FY2023/2024-Q1) board.Oct 9 2023, 4:28 PM
gerritbot added a comment.Oct 27 2023, 9:49 AM

Change 969305 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Update role contact for thanos frontend

https://gerrit.wikimedia.org/r/969305

gerritbot added a project: Patch-For-Review.Oct 27 2023, 9:49 AM
gerritbot added a comment.Oct 27 2023, 10:06 AM

Change 969305 merged by Muehlenhoff:

[operations/puppet@production] Update role contact for thanos frontend

https://gerrit.wikimedia.org/r/969305

Maintenance_bot removed a project: Patch-For-Review.Oct 27 2023, 10:10 AM
ayounsi reopened this task as Open.Dec 15 2023, 9:41 AM
ayounsi subscribed.

Could someone update https://wikitech.wikimedia.org/wiki/SRE/Infrastructure_naming_conventions to mention the new hostname ?

MatthewVernon closed this task as Resolved.Dec 15 2023, 11:42 AM

Done, thanks for the reminder.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL