rack/setup/install codfw logstash elasticsearch storage servers
Closed, ResolvedPublic

Description

This task will track the racking, setup, and installation of 3 new logstash elasticsearch storage servers

logstash2001

  • - receive in system on procurement task T207889
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, internal vlan)
    • end on-site specific steps
  • - production dns entries added (internal subnets for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

logstash2002

  • - receive in system on procurement task T207889
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, internal vlan)
    • end on-site specific steps
  • - production dns entries added (internal subnets for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

logstash2003

  • - receive in system on procurement task T207889
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, internal vlan)
    • end on-site specific steps
  • - production dns entries added (internal subnets for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation
Papaul created this task.Mon, Dec 3, 10:45 PM
Restricted Application added a project: Operations. · View Herald TranscriptMon, Dec 3, 10:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Papaul triaged this task as High priority.Mon, Dec 3, 10:46 PM
Papaul updated the task description. (Show Details)Mon, Dec 3, 10:52 PM
fgiunchedi updated the task description. (Show Details)Tue, Dec 4, 9:57 AM

@Papaul names replaced! thanks

Also please rack these systems across different rows, any combination of rows will do. The rest of the task LGTM

Papaul added a comment.Tue, Dec 4, 1:20 PM

@fgiunchedi Please provide partman recipe to use. I have 4x4TB disks

Change 477536 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] install_server: update logstash partman recipe

https://gerrit.wikimedia.org/r/477536

Change 477536 merged by Filippo Giunchedi:
[operations/puppet@production] install_server: update logstash partman recipe

https://gerrit.wikimedia.org/r/477536

@fgiunchedi Please provide partman recipe to use. I have 4x4TB disks

You can use logstash.cfg similarly to our eqiad hardware, should work as-is.

herron added a subscriber: herron.Tue, Dec 4, 5:30 PM
Papaul updated the task description. (Show Details)Tue, Dec 4, 10:14 PM

Change 477868 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Add production and mgmt DNS entries for logstash200[1-3]

https://gerrit.wikimedia.org/r/477868

Papaul updated the task description. (Show Details)Wed, Dec 5, 9:30 PM

Change 477911 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/puppet@production] DHCP: Add MAC address entries for logstash200[1-3]

https://gerrit.wikimedia.org/r/477911

Change 477923 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/puppet@production] Partman: Add logstash200[1-3]

https://gerrit.wikimedia.org/r/477923

Papaul updated the task description. (Show Details)Wed, Dec 5, 10:30 PM

Change 477923 merged by Dzahn:
[operations/puppet@production] Partman: Add logstash200[1-3]

https://gerrit.wikimedia.org/r/477923

Change 477911 merged by Dzahn:
[operations/puppet@production] DHCP: Add MAC address entries for logstash200[1-3]

https://gerrit.wikimedia.org/r/477911

Change 477868 merged by Dzahn:
[operations/dns@master] DNS: Add production and mgmt DNS entries for logstash200[1-3]

https://gerrit.wikimedia.org/r/477868

Papaul added a comment.Fri, Dec 7, 2:20 AM

@fgiunchedi the installation is complaining about not finding any swap partition.

──────────────────────┤ [!!] Partition disks ├────────────────────────┐

│                                                                       │    
│ You have not selected any partitions for use as swap space. Enabling  │    
│ swap space is recommended so that the system can make better use of   │    
│ the available physical memory, and so that it behaves better when     │    
│ if you do not have enough physical memory.                            │    
│                                                                       │    
│ If you do not go back to the partitioning menu and assign a swap      │    
│ partition, the installation will continue without swap space.         │    
│                                                                       │    
│ Do you want to return to the partitioning menu?

I checked logstash.cfg, there is no line that disable swap. if you don't want to use swap can you please add
partman-basicfilesystems partman-basicfilesystems/no_swap boolean false to logstash.cfg,

Thanks.

Papaul updated the task description. (Show Details)Fri, Dec 7, 2:21 AM

Change 478132 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] change IP for logstash2001, 10.192.0.104 is already in use

https://gerrit.wikimedia.org/r/478132

Change 478132 merged by Papaul:
[operations/dns@master] change IP for logstash2001, 10.192.0.104 is already in use

https://gerrit.wikimedia.org/r/478132

Change 478149 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] install_server: fix logstash partman recipe

https://gerrit.wikimedia.org/r/478149

Change 478149 merged by Filippo Giunchedi:
[operations/puppet@production] install_server: fix logstash partman recipe

https://gerrit.wikimedia.org/r/478149

@fgiunchedi the installation is complaining about not finding any swap partition.

──────────────────────┤ [!!] Partition disks ├────────────────────────┐

│                                                                       │    
│ You have not selected any partitions for use as swap space. Enabling  │    
│ swap space is recommended so that the system can make better use of   │    
│ the available physical memory, and so that it behaves better when     │    
│ if you do not have enough physical memory.                            │    
│                                                                       │    
│ If you do not go back to the partitioning menu and assign a swap      │    
│ partition, the installation will continue without swap space.         │    
│                                                                       │    
│ Do you want to return to the partitioning menu?

I checked logstash.cfg, there is no line that disable swap. if you don't want to use swap can you please add
partman-basicfilesystems partman-basicfilesystems/no_swap boolean false to logstash.cfg,

Fixed now, partman should be happier!

Papaul updated the task description. (Show Details)Fri, Dec 7, 3:09 PM
Papaul updated the task description. (Show Details)Fri, Dec 7, 3:41 PM
Papaul reassigned this task from Papaul to fgiunchedi.

@fgiunchedi all yours

herron claimed this task.Fri, Dec 7, 3:57 PM

Awesome! Thanks much @Papaul!

I'll be working on these along with @fgiunchedi and will get started on service config

Change 478973 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] assign codfw logstash hardware elasticsearch and kafka roles

https://gerrit.wikimedia.org/r/478973

Change 478973 merged by Herron:
[operations/puppet@production] assign codfw logstash hardware elasticsearch and kafka roles

https://gerrit.wikimedia.org/r/478973

Change 479023 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] add logstash_codfw icinga group

https://gerrit.wikimedia.org/r/479023

Change 479023 merged by Herron:
[operations/puppet@production] add logstash_codfw icinga group

https://gerrit.wikimedia.org/r/479023

Change 479031 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] add ip6 mapped interfaces to logstash codfw metal hosts

https://gerrit.wikimedia.org/r/479031

Change 479031 merged by Herron:
[operations/puppet@production] add ip6 mapped interfaces to logstash codfw metal hosts

https://gerrit.wikimedia.org/r/479031

Papaul removed a subscriber: Papaul.Tue, Dec 11, 8:37 PM

Change 479048 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] network::constants: add codfw logstash kafka brokers

https://gerrit.wikimedia.org/r/479048

Change 479048 merged by Herron:
[operations/puppet@production] network::constants: add codfw logstash kafka brokers

https://gerrit.wikimedia.org/r/479048

herron updated the task description. (Show Details)EditedTue, Dec 11, 10:25 PM
herron closed this task as Resolved.

New codfw logging storage hosts are online and puppetized with elasticsearch/kafka roles