User Details
User Details
- User Since
- Dec 18 2014, 3:39 PM (270 w, 3 d)
- Availability
- Available
- LDAP User
- Papaul
- MediaWiki User
- Unknown
Sat, Feb 22
Sat, Feb 22
Papaul added a comment to T241852: rack/setup/install new codfw mw systems.
| Servers Rack D3 | Ready for service |
| mw2366 | Yes |
| mw2367 | Yes |
| mw2368 | Yes |
| mw2369 | Yes |
| mw2370 | Yes |
| mw2371 | Yes |
| mw2372 | Yes |
| mw2373 | Yes |
| mw2374 | Yes |
| mw2375 | Yes |
| mw2376 | Yes |
Papaul updated the task description for T243319: decommission labstore2003.codfw.wmnet and labstore2004.codfw.wmnet.
Fri, Feb 21
Fri, Feb 21
Papaul updated the task description for T241852: rack/setup/install new codfw mw systems.
Papaul added a comment to T245192: Investigate Juniper storm control.
Reading from librenms on ms1-eqiad
Thu, Feb 20
Thu, Feb 20
Papaul updated the task description for T244783: (no date provided) rack/setup/install ganeti20[19-24].
@Jgreen All yours
A reboot fixed the problem
Papaul closed T245714: es2022 power supply issues, a subtask of T243052: Productionize es1020-es1025, es2020-es2025, as Resolved.
Papaul closed T245164: Alert for device ps1-a8-codfw.mgmt.codfw.wmnet - Device rebooted as Resolved.
We replaced the PDU, all good
Papaul updated the task description for T245566: (ASAP) rack/setup/install frdb2001.
Papaul added a comment to T245566: (ASAP) rack/setup/install frdb2001.
papaul@fasw-c-codfw# show | compare
[edit interfaces interface-range vlan-fundraising]
member "ge-[0-1]/0/20" { ... }
+ member "ge-[0-1]/0/23";
[edit interfaces interface-range disabled]
- member ge-0/0/23;
- member ge-1/0/23;
[edit interfaces]
+ ge-0/0/23 {
+ description frdb2001:eth0;
+ }
+ ge-1/0/23 {
+ description frdb2001:eth1;
+ }Papaul updated the task description for T244169: codfw: rack/setup/install 3 new payments server for frack.
Papaul added a comment to T244169: codfw: rack/setup/install 3 new payments server for frack.
papaul@fasw-c-codfw# show | compare
[edit interfaces interface-range disabled]
- member "ge-[0-1]/0/8";
- member "ge-[0-1]/0/9";
- member "ge-[0-1]/0/10";
[edit interfaces interface-range vlan-payments]
member "ge-[0-1]/0/4" { ... }
+ member "ge-[0-1]/0/8";
+ member "ge-[0-1]/0/9";
+ member "ge-[0-1]/0/10";
[edit interfaces]
+ ge-0/0/8 {
+ description payments2001-a:eth0;
+ }
+ ge-0/0/9 {
+ description payments2002-a:eth0;
+ }
+ ge-0/0/10 {
+ description payments2003-a:eth0;
+ }
+ ge-1/0/8 {
+ description payments2001-a:eth1;
+ }
+ ge-1/0/9 {
+ description payments2002-a:eth1;
+ }
+ ge-1/0/10 {
+ description payments2003-a:eth1;
+ }Wed, Feb 19
Wed, Feb 19
Papaul updated the task description for T243319: decommission labstore2003.codfw.wmnet and labstore2004.codfw.wmnet.
Papaul updated the task description for T243329: decommission labstore2001.codfw.wmnet and labstore2002.codfw.wmnet.
Complete
Papaul updated the task description for T244950: codfw:fundraising single-cpu misc servers.
Papaul updated the task description for T245566: (ASAP) rack/setup/install frdb2001.
Papaul updated the task description for T245566: (ASAP) rack/setup/install frdb2001.
Papaul closed Unknown Object (Task), a subtask of T238395: Fundraising Analytics Infrastructure and Setup, as Resolved.
Papaul closed Unknown Object (Task), a subtask of T242257: 2019-2020 Q3 fundraising hardware refresh and capex, as Resolved.
Papaul added a comment to T244196: codfw: Delete cloud interface-range.
@ayounsi since i deleted the interface range do you want me to delete also the VLAN cloud-support1-b-codfw
Papaul added a comment to T244196: codfw: Delete cloud interface-range.
papaul@asw-b-codfw# show | compare
[edit interfaces]
- interface-range vlan-cloud-support1-b-codfw {
- member ge-8/0/9;
- mtu 9192;
- unit 0 {
- family ethernet-switching {
- interface-mode access;
- vlan {
- members cloud-support1-b-codfw;
- }
- }
- }
- }complete
Papaul updated the task description for T244743: decommission bellatrix.frack.codfw.wmnet.
Papaul moved T245164: Alert for device ps1-a8-codfw.mgmt.codfw.wmnet - Device rebooted from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
Papaul moved T245566: (ASAP) rack/setup/install frdb2001 from Backlog to Racking Tasks on the ops-codfw board.
Papaul reassigned T241337: (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.codfw.wmnet from Papaul to Gehel.
@Gehel All yours let me know if you have any questions.
Papaul updated the task description for T241337: (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.codfw.wmnet.
Papaul updated the task description for T241337: (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.codfw.wmnet.
Papaul updated the task description for T241337: (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.codfw.wmnet.
Tue, Feb 18
Tue, Feb 18
Papaul added a comment to T245164: Alert for device ps1-a8-codfw.mgmt.codfw.wmnet - Device rebooted.
I open a request ticket (TICKET NO.1578279) with CY1 to assistance me on unplugging the old PDU and plugging the new one tomorrow the 19th at 10:30 Dallas time
Papaul assigned T245164: Alert for device ps1-a8-codfw.mgmt.codfw.wmnet - Device rebooted to wiki_willy.
@wiki_willy ps1-a8 is not stable to still in production. I notice that there is a clicking noise coming from the PDU and the readings are not stable it keeps flapping.
Papaul updated the task description for T241337: (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.codfw.wmnet.
PEM 0 has no problem the problem is from PS1-a8 where PEM 0 is plug into. maybe problem from last week upgrade i have no readings on PS1-a8 right now.
closing this task
Fri, Feb 14
Fri, Feb 14
Papaul updated the task description for T241337: (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.codfw.wmnet.
Thu, Feb 13
Thu, Feb 13
Papaul renamed T241337: (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.codfw.wmnet from (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.wikimedia.org to (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.codfw.wmnet.
Papaul added a comment to T243715: elastic2043 has hardware errors that trigger reboots.
BIOS Version 1.5.6
iDRAC Firmware Version 3.21.21.21
Papaul added a comment to T243715: elastic2043 has hardware errors that trigger reboots.
"i have a case open with Dell. See below for case reference. According to Dell like always they will not give you a solution to the problem like they don't know also what the issue is keep trying possible solutions like upgrade BIOS and IDRAC.
Papaul added a comment to T241852: rack/setup/install new codfw mw systems.
| servers Rack A3 | OS | ready for service | servers Rack A6 | OS | ready for service | |
| mw2291 | Stretch | yes | mw2301 | Stretch | yes | |
| mw2292 | Stretch | yes | mw2302 | Stretch | yes | |
| mw2293 | Stretch | yes | mw2303 | Stretch | yes | |
| mw2294 | Stretch | yes | mw2304 | Stretch | yes | |
| mw2295 | Stretch | yes | mw2305 | Stretch | yes | |
| mw2296 | Stretch | yes | mw2306 | Stretch | yes | |
| mw2297 | Stretch | yes | mw2307 | Stretch | yes | |
| mw2298 | Stretch | yes | mw2308 | Stretch | yes | |
| mw2299 | Stretch | yes | mw2309 | Stretch | yes | |
| mw2300 | Stretch | yes |
Papaul updated the task description for T241852: rack/setup/install new codfw mw systems.
Wed, Feb 12
Wed, Feb 12
Papaul updated the task description for T243112: codfw: rack/setup/install parse200[1-20].codfw.wmnet.
Papaul moved T244196: codfw: Delete cloud interface-range from Backlog to Non-Urgent on the ops-codfw board.
Papaul moved T244950: codfw:fundraising single-cpu misc servers from Backlog to Racking Tasks on the ops-codfw board.
@Jgreen Please rename the name of each server and assign back to me.
Thanks.
Papaul updated the task description for T244743: decommission bellatrix.frack.codfw.wmnet.
Tue, Feb 11
Tue, Feb 11
Papaul updated the task description for T241852: rack/setup/install new codfw mw systems.
Papaul updated the task description for T244169: codfw: rack/setup/install 3 new payments server for frack.
Papaul updated the task description for T244169: codfw: rack/setup/install 3 new payments server for frack.
Papaul renamed T244783: (no date provided) rack/setup/install ganeti20[19-24] from (no date provided) rack/setup/install ganeti20[16-21] to (no date provided) rack/setup/install ganeti20[19-24].
Mon, Feb 10
Mon, Feb 10
Papaul updated the task description for T241852: rack/setup/install new codfw mw systems.
Papaul reassigned T244783: (no date provided) rack/setup/install ganeti20[19-24] from Papaul to RobH.
@RobH we have already ganeti2009-ganeti2014 in codfw
Papaul updated the task description for T244743: decommission bellatrix.frack.codfw.wmnet.
Papaul added a comment to T244743: decommission bellatrix.frack.codfw.wmnet.
papaul@fasw-c-codfw# show | compare
[edit interfaces interface-range disabled]
member "ge-[0-1]/0/12" { ... }
+ member "ge-[0-1]/0/7";
[edit interfaces interface-range vlan-administration]
- member "ge-[0-1]/0/7";
[edit interfaces]
- ge-0/0/7 {
- description bellatrix;
- }
- ge-1/0/7 {
- description bellatrix;
- }Papaul added a comment to T244743: decommission bellatrix.frack.codfw.wmnet.
@Jgreen haha i was waiting for the task thanks
Thu, Feb 6
Thu, Feb 6
@Gehel All yours
Papaul updated the task description for T242301: codfw: rack/setup/install wdqs200[7-8].codfw.wmnet.
Papaul updated the task description for T244169: codfw: rack/setup/install 3 new payments server for frack.
Papaul added a comment to T244438: codfw: new mw servers not getting an IP when default to Stretch.
Did the re-install on mw2311 it works . Thanks
Papaul triaged T244438: codfw: new mw servers not getting an IP when default to Stretch as Medium priority.
Wed, Feb 5
Wed, Feb 5
Papaul updated subscribers of T241852: rack/setup/install new codfw mw systems.
All the servers on the table above are running Buster I had a chat with @MoritzMuehlenhoff and he mentioned that we need to install Stretch on those servers so I have to update the dhcp file and re-image those servers
Papaul added a comment to T241337: (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.codfw.wmnet.
Talked to @Gehel on IRC those servers will be in the private VLAN and not in the public VLAN with Stretch as OS.
Papaul added a comment to T241852: rack/setup/install new codfw mw systems.
| servers | ready for service |
| mw2310 | yes |
| mw2311 | yes |
| mw2312 | yes |
| mw2313 | yes |
| mw2314 | yes |
| mw2315 | yes |
| mw2316 | yes |
| mw2317 | yes |
| mw2318 | yes |
| mw2319 | yes |
| mw2320 | yes |
| mw2321 | yes |
| mw2322 | yes |
| mw2323 | yes |
| mw2324 | yes |
Papaul updated the task description for T241852: rack/setup/install new codfw mw systems.
Tue, Feb 4
Tue, Feb 4
Papaul updated the task description for T241852: rack/setup/install new codfw mw systems.
Papaul added a comment to T244196: codfw: Delete cloud interface-range.
okay thanks will do that
Upgrade CPLD to version 1.0.10 no more errors. system is back up running
Papaul moved T243715: elastic2043 has hardware errors that trigger reboots from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.
This is has been fixed
Closing this task since we haven;t seen any errors since November
Papaul moved T244169: codfw: rack/setup/install 3 new payments server for frack from Backlog to Racking Tasks on the ops-codfw board.
complete
Papaul closed T242990: decommission alnitak.frack.codfw.wmnet, a subtask of T242257: 2019-2020 Q3 fundraising hardware refresh and capex, as Resolved.
Papaul updated the task description for T242990: decommission alnitak.frack.codfw.wmnet.
complete
Papaul closed T242983: decommission frdb2001.frack.codfw.wmnet, a subtask of T242257: 2019-2020 Q3 fundraising hardware refresh and capex, as Resolved.
Papaul updated the task description for T242983: decommission frdb2001.frack.codfw.wmnet.
Papaul updated the task description for T243319: decommission labstore2003.codfw.wmnet and labstore2004.codfw.wmnet.
Papaul added a comment to T243319: decommission labstore2003.codfw.wmnet and labstore2004.codfw.wmnet.
I did not remove ge-8/0/9 from interface-range vlan-cloud-support1-b-codfw see T244196
Papaul added a comment to T243319: decommission labstore2003.codfw.wmnet and labstore2004.codfw.wmnet.
papaul@asw-b-codfw# show | compare
[edit interfaces interface-range vlan-cloud-support1-b-codfw]
- member ge-8/0/1;
- member ge-8/0/9;
[edit interfaces interface-range disabled]
member ge-1/0/1 { ... }
+ member ge-8/0/1;
+ member ge-8/0/9;
[edit interfaces]
- ge-8/0/1 {
- description labstore2004;
- enable;
- }
- ge-8/0/9 {
- description labstore2003;
- enable;
- }Papaul updated the task description for T243329: decommission labstore2001.codfw.wmnet and labstore2002.codfw.wmnet.
Papaul added a comment to T243329: decommission labstore2001.codfw.wmnet and labstore2002.codfw.wmnet.
apaul@asw-b-codfw# show | compare
[edit interfaces interface-range vlan-cloud-support1-b-codfw]
- member-range ge-1/0/0 to ge-1/0/1;
[edit interfaces interface-range disabled]
member ge-5/0/40 { ... }
+ member ge-1/0/0;
+ member ge-1/0/1;
[edit interfaces]
- ge-1/0/0 {
- description labstore2001;
- enable;
- }
- ge-1/0/1 {
- description labstore2002;
- enable;
- }Mon, Feb 3
Mon, Feb 3
Papaul updated the task description for T244169: codfw: rack/setup/install 3 new payments server for frack.
Papaul triaged T244169: codfw: rack/setup/install 3 new payments server for frack as Medium priority.
Thu, Jan 30
Thu, Jan 30
Papaul closed T240876: hw troubleshooting: hardware RAID predictive failure for bellatrix.frack.codfw.wmnet as Declined.
No need for this task since we have a replacing server setup on T237440
I think we don't need alerting when the readings are low since some racks in codfw are not fully populated. For example rack d8 has only 7 servers.
Let us only alert when the values are high. Thanks
Resolving this task
Done
Wed, Jan 29
Wed, Jan 29
Papaul updated subscribers of T243927: Export Netbox Stats for DCops to a visualization tool.
Jan 25 2020
Jan 25 2020
Papaul updated the task description for T241852: rack/setup/install new codfw mw systems.
Papaul added a comment to T241852: rack/setup/install new codfw mw systems.
11 servers in row D racked and Netnox updated
Papaul updated the task description for T241852: rack/setup/install new codfw mw systems.
Papaul added a comment to T241852: rack/setup/install new codfw mw systems.
@jijiki I think we are off with the server distribution