Page MenuHomePhabricator

Expand storage for WMCH demo server
Closed, ResolvedPublic

Description

To proceed with T268292: Migrate Wikimini into WMCH infrastructure, Wikimedia CH needs additional storage for their demo webserver.

Their demo webserver is the one reachable via WAN and LAN IP 89.207.237.59 and 192.168.128.40.

This task covers the upstream ticket sent to our service provider visible here for WMCH Infrastructure internals:

{P13455}

Event Timeline

valerio.bozzolan renamed this task from Request additional storage for WMCH demo server to Expand storage for WMCH demo server.Nov 27 2020, 3:25 PM
valerio.bozzolan triaged this task as Medium priority.

ho aumentato (lato hypervisor) la capacità del disco da 32 a 50 GB. Provvede lei ad allargare il disco all'interno della VM?
― service provider

So let's check the current situation and expand the volume:

$ fdisk -l
Disk /dev/sda: 32 GiB, 34359738368 bytes, 67108864 sectors
Disk model: Virtual disk    
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xd40c071f

Device     Boot  Start      End  Sectors  Size Id Type
/dev/sda1  *      2048   499711   497664  243M 83 Linux
/dev/sda2       501758 67106815 66605058 31.8G  5 Extended
/dev/sda5       501760 67106815 66605056 31.8G 8e Linux LVM


Disk /dev/mapper/demo--vg-root: 27.8 GiB, 29804724224 bytes, 58212352 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/demo--vg-swap_1: 4 GiB, 4294967296 bytes, 8388608 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

$ # no additional space recognized. Try a rescan
$ echo 1 > /sys/block/sda/device/rescan

$ fdisk -l
(same output)

$ # still no additional space recognized, try another rescan
$ apt install scsitools
$ rescan-scsi-bus
/usr/sbin/rescan-scsi-bus: line 592: [: 1.98: integer expression expected
Host adapter 0 (vmw_pvscsi) found.
Host adapter 1 (ata_piix) found.
Host adapter 2 (ata_piix) found.
Scanning SCSI subsystem for new devices
Scanning host 0 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning for device 0 0 0 0 ... 
OLD: Host: scsi0 Channel: 00 Id: 00 Lun: 00
      Vendor: VMware   Model: Virtual disk     Rev: 1.0 
      Type:   Direct-Access                    ANSI SCSI revision: 02
Scanning host 1 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning for device 1 0 0 0 ... 
OLD: Host: scsi1 Channel: 00 Id: 00 Lun: 00
      Vendor: NECVMWar Model: VMware IDE CDR00 Rev: 1.00
      Type:   CD-ROM                           ANSI SCSI revision: 05
Scanning host 2 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
0 new device(s) found.                     
0 device(s) removed.

$ fdisk -l
(same output)

$ # still no additional space recognized, try a reboot
$ reboot
...
$ fdisk -l
(same outut)

$ # still no additional space recognized also after reboot :(

$ parted
GNU Parted 3.2
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted)

$ p                                                              
Model: VMware Virtual disk (scsi)
Disk /dev/sda: 34.4GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type      File system  Flags
 1      1049kB  256MB   255MB   primary   ext2         boot
 2      257MB   34.4GB  34.1GB  extended
 5      257MB   34.4GB  34.1GB  logical                lvm

Actually inside this VPS we are unable to detect the storage change applied from its hypervisor and we are so unable to expand our LVM. Contacting the service provider.

Moreover the machine is not reachable anymore similarly to what happened the last week to another server of Wikimedia CH:

T268371: Incident report for WMCH server "members2" (2020-11-20)

This means now we are blocked by:

T269342: Incident in WMCH server "demo" (2020-12-03)

Now the server is operative.

Let's increase the LVM partition without rebooting.

challenge-accepted

Available space:

$ ssh wmch-demo
$ parted /dev/sda unit GB print free
Model: VMware Virtual disk (scsi)
Disk /dev/sda: 53.7GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type      File system  Flags
        0.00GB  0.00GB  0.00GB            Free Space
 1      0.00GB  0.26GB  0.25GB  primary   ext2         boot
        0.26GB  0.26GB  0.00GB            Free Space
 2      0.26GB  34.4GB  34.1GB  extended
 5      0.26GB  34.4GB  34.1GB  logical                lvm
        34.4GB  53.7GB  19.3GB            Free Space

Now the server is aware of the additional 19.3G.

Precisely the start and stop sectors are 67106816 and 104857599:

$ parted /dev/sda unit s print free
Model: VMware Virtual disk (scsi)
Disk /dev/sda: 104857600s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start      End         Size       Type      File system  Flags
        63s        2047s       1985s                Free Space
 1      2048s      499711s     497664s    primary   ext2         boot
        499712s    501757s     2046s                Free Space
 2      501758s    67106815s   66605058s  extended
 5      501760s    67106815s   66605056s  logical                lvm
        67106816s  104857599s  37750784s            Free Space

Creating a new LVM partition:

$ fdisk /dev/sda
Welcome to fdisk (util-linux 2.33.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

fdisk: cannot open /dev/vda: No such file or directory
root@wmch-demo:~# fdisk /dev/sda

Welcome to fdisk (util-linux 2.33.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

$ n
Partition type
   p   primary (1 primary, 1 extended, 2 free)
   l   logical (numbered from 5)

$ p
Partition number (3,4, default 3): 

$ 3
First sector (499712-104857599, default 499712): 

$ 67106816
Last sector, +/-sectors or +/-size{K,M,G,T,P} (67106816-104857599, default 104857599):

$ 104857599
Created a new partition 3 of type 'Linux' and of size 18 GiB.

$ t

Partition number (1-3,5, default 5): 
$ 3

Hex code (type L to list all codes):

$ L
 0  Empty           24  NEC DOS         81  Minix / old Lin bf  Solaris        
 1  FAT12           27  Hidden NTFS Win 82  Linux swap / So c1  DRDOS/sec (FAT-
 2  XENIX root      39  Plan 9          83  Linux           c4  DRDOS/sec (FAT-
 3  XENIX usr       3c  PartitionMagic  84  OS/2 hidden or  c6  DRDOS/sec (FAT-
 4  FAT16 <32M      40  Venix 80286     85  Linux extended  c7  Syrinx         
 5  Extended        41  PPC PReP Boot   86  NTFS volume set da  Non-FS data    
 6  FAT16           42  SFS             87  NTFS volume set db  CP/M / CTOS / .
 7  HPFS/NTFS/exFAT 4d  QNX4.x          88  Linux plaintext de  Dell Utility   
 8  AIX             4e  QNX4.x 2nd part 8e  Linux LVM       df  BootIt         
 9  AIX bootable    4f  QNX4.x 3rd part 93  Amoeba          e1  DOS access     
 a  OS/2 Boot Manag 50  OnTrack DM      94  Amoeba BBT      e3  DOS R/O        
 b  W95 FAT32       51  OnTrack DM6 Aux 9f  BSD/OS          e4  SpeedStor      
 c  W95 FAT32 (LBA) 52  CP/M            a0  IBM Thinkpad hi ea  Rufus alignment
 e  W95 FAT16 (LBA) 53  OnTrack DM6 Aux a5  FreeBSD         eb  BeOS fs        
 f  W95 Ext'd (LBA) 54  OnTrackDM6      a6  OpenBSD         ee  GPT            
10  OPUS            55  EZ-Drive        a7  NeXTSTEP        ef  EFI (FAT-12/16/
11  Hidden FAT12    56  Golden Bow      a8  Darwin UFS      f0  Linux/PA-RISC b
12  Compaq diagnost 5c  Priam Edisk     a9  NetBSD          f1  SpeedStor      
14  Hidden FAT16 <3 61  SpeedStor       ab  Darwin boot     f4  SpeedStor      
16  Hidden FAT16    63  GNU HURD or Sys af  HFS / HFS+      f2  DOS secondary  
17  Hidden HPFS/NTF 64  Novell Netware  b7  BSDI fs         fb  VMware VMFS    
18  AST SmartSleep  65  Novell Netware  b8  BSDI swap       fc  VMware VMKCORE 
1b  Hidden W95 FAT3 70  DiskSecure Mult bb  Boot Wizard hid fd  Linux raid auto
1c  Hidden W95 FAT3 75  PC/IX           bc  Acronis FAT32 L fe  LANstep        
1e  Hidden W95 FAT1 80  Old Minix       be  Solaris boot    ff  BBT

$ 8e
Changed type of partition 'Linux' to 'Linux LVM'.

$ p
Disk /dev/sda: 50 GiB, 53687091200 bytes, 104857600 sectors
Disk model: Virtual disk    
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xd40c071f

Device     Boot    Start       End  Sectors  Size Id Type
/dev/sda1  *        2048    499711   497664  243M 83 Linux
/dev/sda2         501758  67106815 66605058 31.8G  5 Extended
/dev/sda3       67106816 104857599 37750784   18G 8e Linux LVM
/dev/sda5         501760  67106815 66605056 31.8G 8e Linux LVM

Partition table entries are not in disk order.

$ w
The partition table has been altered.
Syncing disks.

Now the new 18G partition is available:

$ fdisk -l
Disk /dev/sda: 50 GiB, 53687091200 bytes, 104857600 sectors
Disk model: Virtual disk    
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xd40c071f

Device     Boot    Start       End  Sectors  Size Id Type
/dev/sda1  *        2048    499711   497664  243M 83 Linux
/dev/sda2         501758  67106815 66605058 31.8G  5 Extended
/dev/sda3       67106816 104857599 37750784   18G 8e Linux LVM
/dev/sda5         501760  67106815 66605056 31.8G 8e Linux LVM

Partition table entries are not in disk order.


Disk /dev/mapper/demo--vg-root: 27.8 GiB, 29804724224 bytes, 58212352 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/demo--vg-swap_1: 4 GiB, 4294967296 bytes, 8388608 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Let's create a new physical LVM partition for that.

First, collect some information:

$ vgdisplay
  --- Volume group ---
  VG Name               demo-vg
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <31.76 GiB
  PE Size               4.00 MiB
  Total PE              8130
  Alloc PE / Size       8130 / <31.76 GiB
  Free  PE / Size       0 / 0   
  VG UUID               fY0ARf-yFC3-T2zr-K0OW-6gtu-NdDh-hcx9dB

$ lvdisplay
  --- Logical volume ---
  LV Path                /dev/demo-vg/root
  LV Name                root
  VG Name                demo-vg
  LV UUID                TqRiLl-DjdL-vD15-4dyp-ENsz-aC9G-apZ0rK
  LV Write Access        read/write
  LV Creation host, time demo, 2020-06-03 12:15:10 +0200
  LV Status              available
  # open                 1
  LV Size                <27.76 GiB
  Current LE             7106
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:0
   
  --- Logical volume ---
  LV Path                /dev/demo-vg/swap_1
  LV Name                swap_1
  VG Name                demo-vg
  LV UUID                Q3vrdu-d9aP-J0qq-YrlA-bs81-7W3x-xEJS2d
  LV Write Access        read/write
  LV Creation host, time demo, 2020-06-03 12:15:11 +0200
  LV Status              available
  # open                 2
  LV Size                4.00 GiB
  Current LE             1024
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:1

So we have

  • VG name demo-vg
  • LV path /dev/demo-vg/root

Let's initialize a new physical volume and attach it to the virtual group:

$ pvcreate /dev/sda3
  Physical volume "/dev/sda3" successfully created.

$ vgextend demo-vg /dev/sda3
  Volume group "demo-vg" successfully extended

$ pvscan
  PV /dev/sda5   VG demo-vg         lvm2 [<31.76 GiB / 0    free]
  PV /dev/sda3   VG demo-vg         lvm2 [18.00 GiB / 18.00 GiB free]
  Total: 2 [<49.76 GiB] / in use: 2 [<49.76 GiB] / in no VG: 0 [0   ]

$ lvextend /dev/demo-vg/root /dev/sda3
  Size of logical volume demo-vg/root changed from <27.76 GiB (7106 extents) to <45.76 GiB (11714 extents).
  Logical volume demo-vg/root successfully resized.

$ resize2fs /dev/demo-vg/root
resize2fs 1.44.5 (15-Dec-2018)
Filesystem at /dev/demo-vg/root is mounted on /; on-line resizing required
old_desc_blocks = 4, new_desc_blocks = 6
The filesystem on /dev/demo-vg/root is now 11995136 (4k) blocks long.

Now we have 40G of free space in the main partition:

$ df -lh
Filesystem                 Size  Used Avail Use% Mounted on
udev                       2.0G     0  2.0G   0% /dev
tmpfs                      395M  5.5M  390M   2% /run
/dev/mapper/demo--vg-root   45G  3.0G   40G   7% /
tmpfs                      2.0G     0  2.0G   0% /dev/shm
tmpfs                      5.0M     0  5.0M   0% /run/lock
tmpfs                      2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/sda1                  236M   59M  166M  26% /boot
tmpfs                      395M     0  395M   0% /run/user/0

Ya-hi!

Macro seal-of-approval: LVM volume unexploded seal of approval

Done this again. We need additional storage again.

$ ssh wmch-demo
$ parted /dev/sda unit s print free
Model: VMware Virtual disk (scsi)
Disk /dev/sda: 146800640s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start       End         Size       Type      File system  Flags
        63s         2047s       1985s                Free Space
 1      2048s       499711s     497664s    primary   ext2         boot
        499712s     501757s     2046s                Free Space
 2      501758s     67106815s   66605058s  extended
 5      501760s     67106815s   66605056s  logical                lvm
 3      67106816s   104857599s  37750784s  primary                lvm
        104857600s  146800639s  41943040s            Free Space

Now the server is aware of the additional 20G.

Precisely the start and stop sectors are 104857600 and 146800639.

Creating a new LVM partition:

$ fdisk /dev/sda

Welcome to fdisk (util-linux 2.33.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

$ n
Partition type
   p   primary (1 primary, 1 extended, 2 free)
   l   logical (numbered from 5)

$ p
Partition number (3,4, default 3): 

$ 4
First sector (499712-146800639, default 499712):

$ 104857600                      
                        
Last sector, +/-sectors or +/-size{K,M,G,T,P} (104857600-146800639, default 146800639):
$ 146800639

$ t

Partition number (1-5, default 5):
$ 4

Hex code (type L to list all codes):

$ 8e
Changed type of partition 'Linux' to 'Linux LVM'.

$ p
Disk /dev/sda: 70 GiB, 75161927680 bytes, 146800640 sectors                                             
Disk model: Virtual disk                                                                                
Units: sectors of 1 * 512 = 512 bytes                                                                   
Sector size (logical/physical): 512 bytes / 512 bytes                                                   
I/O size (minimum/optimal): 512 bytes / 512 bytes                                                       
Disklabel type: dos
Disk identifier: 0xd40c071f

Device     Boot     Start       End  Sectors  Size Id Type
/dev/sda1  *         2048    499711   497664  243M 83 Linux
/dev/sda2          501758  67106815 66605058 31.8G  5 Extended
/dev/sda3        67106816 104857599 37750784   18G 8e Linux LVM
/dev/sda4       104857600 146800639 41943040   20G 8e Linux LVM
/dev/sda5          501760  67106815 66605056 31.8G 8e Linux LVM

Partition table entries are not in disk order.

$ w
The partition table has been altered.
Syncing disks.

Now the new 20G partition is available:

$ fdisk -l                                                                              
Disk /dev/sda: 70 GiB, 75161927680 bytes, 146800640 sectors                                             
Disk model: Virtual disk                                                                                
Units: sectors of 1 * 512 = 512 bytes                                                                   
Sector size (logical/physical): 512 bytes / 512 bytes                                                   
I/O size (minimum/optimal): 512 bytes / 512 bytes                                                       
Disklabel type: dos                                                                                     
Disk identifier: 0xd40c071f                                                                             
                                                                                                        
Device     Boot     Start       End  Sectors  Size Id Type                                              
/dev/sda1  *         2048    499711   497664  243M 83 Linux                                             
/dev/sda2          501758  67106815 66605058 31.8G  5 Extended                                          
/dev/sda3        67106816 104857599 37750784   18G 8e Linux LVM                                         
/dev/sda4       104857600 146800639 41943040   20G 8e Linux LVM                                         
/dev/sda5          501760  67106815 66605056 31.8G 8e Linux LVM                                         
                                                                                                        
Partition table entries are not in disk order.                                                          
                                                                                                        
                                                                                                        
Disk /dev/mapper/demo--vg-root: 45.8 GiB, 49132077056 bytes, 95961088 sectors                           
Units: sectors of 1 * 512 = 512 bytes                                                                   
Sector size (logical/physical): 512 bytes / 512 bytes                                                   
I/O size (minimum/optimal): 512 bytes / 512 bytes                                                       
                                                                                                        
                                                                                                        
Disk /dev/mapper/demo--vg-swap_1: 4 GiB, 4294967296 bytes, 8388608 sectors                              
Units: sectors of 1 * 512 = 512 bytes                                                                   
Sector size (logical/physical): 512 bytes / 512 bytes                                                   
I/O size (minimum/optimal): 512 bytes / 512 bytes

Let's initialize a new physical volume and attach it to the virtual group:

$ pvcreate /dev/sda4
  Physical volume "/dev/sda4" successfully created.

$ vgextend demo-vg /dev/sda4
  Volume group "demo-vg" successfully extended

$ pvscan
  PV /dev/sda5   VG demo-vg         lvm2 [<31.76 GiB / 0    free]
  PV /dev/sda3   VG demo-vg         lvm2 [18.00 GiB / 0    free]
  PV /dev/sda4   VG demo-vg         lvm2 [<20.00 GiB / <20.00 GiB free]
  Total: 3 [69.75 GiB] / in use: 3 [69.75 GiB] / in no VG: 0 [0   ]

$ lvextend /dev/demo-vg/root /dev/sda4
  Size of logical volume demo-vg/root changed from <45.76 GiB (11714 extents) to 65.75 GiB (16833 extents).
  Logical volume demo-vg/root successfully resized.

$ resize2fs /dev/demo-vg/root
resize2fs 1.44.5 (15-Dec-2018)
Filesystem at /dev/demo-vg/root is mounted on /; on-line resizing required
old_desc_blocks = 6, new_desc_blocks = 9
The filesystem on /dev/demo-vg/root is now 17236992 (4k) blocks long.

Now we have 28G of free space again in the main partition:

$ df -lh
Filesystem                 Size  Used Avail Use% Mounted on
udev                       2.0G     0  2.0G   0% /dev
tmpfs                      395M  5.5M  390M   2% /run
/dev/mapper/demo--vg-root   65G   35G   28G  56% /
tmpfs                      2.0G     0  2.0G   0% /dev/shm
tmpfs                      5.0M     0  5.0M   0% /run/lock
tmpfs                      2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/sda1                  236M   59M  166M  26% /boot
tmpfs                      395M     0  395M   0% /run/user/0