-- Logs begin at Tue 2020-11-10 03:39:48 UTC, end at Fri 2021-02-19 08:21:06 UTC. --
Feb 19 04:43:57 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:43:57,872] Cleaning up expired backups for VM webperf.edit-save-timings(691723d0-f5aa-4d89-86df-97c30f1a9fa7)
Feb 19 04:44:00 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:44:00,973] Cleaned up 1 expired backups for VM webperf.edit-save-timings(691723d0-f5aa-4d89-86df-97c30f1a9fa7)
Feb 19 04:44:00 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:44:00,973] Creating backup for vm webperf.wpt-runner(98e078f9-a810-475a-adb9-ffbcdd9aaefa)
Feb 19 04:44:01 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:44:01,120] Creating differential backup of pool:eqiad1-compute image_name:98e078f9-a810-475a-adb9-ffbcdd9aaefa_disk from rbd snapshot eqiad1-compute/98e078f9-a810-475a-adb9-ffbcdd9aaefa_disk@2021-02-18T04:44:05_cloudvirt1024 and backy2 version eed66ce4-71a3-11eb-8dbf-b02628295df0
Feb 19 04:44:26 cloudvirt1024 wmcs-backup[56558]: [37B blob data]
Feb 19 04:44:26 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:44:26,884] Cleaning up expired backups for VM webperf.wpt-runner(98e078f9-a810-475a-adb9-ffbcdd9aaefa)
Feb 19 04:44:28 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:44:28,319] Cleaned up 1 expired backups for VM webperf.wpt-runner(98e078f9-a810-475a-adb9-ffbcdd9aaefa)
Feb 19 04:44:28 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:44:28,320] Creating backup for vm webperf.GPSI(d04689dd-f70f-42bc-a4b3-43e664e29420)
Feb 19 04:44:28 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:44:28,473] Creating differential backup of pool:eqiad1-compute image_name:d04689dd-f70f-42bc-a4b3-43e664e29420_disk from rbd snapshot eqiad1-compute/d04689dd-f70f-42bc-a4b3-43e664e29420_disk@2021-02-18T04:44:21_cloudvirt1024 and backy2 version f8e4f822-71a3-11eb-bddc-b02628295df0
Feb 19 04:44:40 cloudvirt1024 wmcs-backup[56558]: [37B blob data]
Feb 19 04:44:41 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:44:41,351] Cleaning up expired backups for VM webperf.GPSI(d04689dd-f70f-42bc-a4b3-43e664e29420)
Feb 19 04:44:42 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:44:42,812] Cleaned up 1 expired backups for VM webperf.GPSI(d04689dd-f70f-42bc-a4b3-43e664e29420)
Feb 19 04:44:42 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:44:42,813] Creating backup for vm wikiapiary.pub2(035e1323-825a-486e-b355-2a296ae7f540)
Feb 19 04:44:42 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:44:42,967] Creating differential backup of pool:eqiad1-compute image_name:035e1323-825a-486e-b355-2a296ae7f540_disk from rbd snapshot eqiad1-compute/035e1323-825a-486e-b355-2a296ae7f540_disk@2021-02-18T04:44:38_cloudvirt1024 and backy2 version 029b1572-71a4-11eb-b4f4-b02628295df0
Feb 19 04:48:56 cloudvirt1024 wmcs-backup[56558]: [37B blob data]
Feb 19 04:48:57 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:48:57,464] Cleaning up expired backups for VM wikiapiary.pub2(035e1323-825a-486e-b355-2a296ae7f540)
Feb 19 04:49:06 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:49:06,561] Cleaned up 1 expired backups for VM wikiapiary.pub2(035e1323-825a-486e-b355-2a296ae7f540)
Feb 19 04:49:06 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:49:06,562] Creating backup for vm wikibase-registry.misc-01(ac928b98-a468-422c-9352-8b2438f5056f)
Feb 19 04:49:06 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:49:06,714] Creating differential backup of pool:eqiad1-compute image_name:ac928b98-a468-422c-9352-8b2438f5056f_disk from rbd snapshot eqiad1-compute/ac928b98-a468-422c-9352-8b2438f5056f_disk@2021-02-18T04:49:14_cloudvirt1024 and backy2 version a700af14-71a4-11eb-aa64-b02628295df0
Feb 19 04:49:34 cloudvirt1024 wmcs-backup[56558]: [37B blob data]
Feb 19 04:49:34 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:49:34,957] Cleaning up expired backups for VM wikibase-registry.misc-01(ac928b98-a468-422c-9352-8b2438f5056f)
Feb 19 04:49:36 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:49:36,593] Cleaned up 1 expired backups for VM wikibase-registry.misc-01(ac928b98-a468-422c-9352-8b2438f5056f)
Feb 19 04:49:36 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:49:36,594] Creating backup for vm wikibase-registry.wbregistry-01(fdb069da-4c57-4703-bc63-b0f1a7640bb0)
Feb 19 04:49:36 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:49:36,754] Creating differential backup of pool:eqiad1-compute image_name:fdb069da-4c57-4703-bc63-b0f1a7640bb0_disk from rbd snapshot eqiad1-compute/fdb069da-4c57-4703-bc63-b0f1a7640bb0_disk@2021-02-18T04:49:44_cloudvirt1024 and backy2 version b91aeda4-71a4-11eb-b2b6-b02628295df0
Feb 19 04:50:02 cloudvirt1024 wmcs-backup[56558]: [37B blob data]
Feb 19 04:50:03 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:50:03,295] Cleaning up expired backups for VM wikibase-registry.wbregistry-01(fdb069da-4c57-4703-bc63-b0f1a7640bb0)
Feb 19 04:50:05 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:50:05,256] Cleaned up 1 expired backups for VM wikibase-registry.wbregistry-01(fdb069da-4c57-4703-bc63-b0f1a7640bb0)
Feb 19 04:50:05 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:50:05,257] Creating backup for vm wikibase-registry.orig-01(e90bba42-4092-4cf1-a80a-bb379ac5888f)
Feb 19 04:50:05 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:50:05,403] Creating differential backup of pool:eqiad1-compute image_name:e90bba42-4092-4cf1-a80a-bb379ac5888f_disk from rbd snapshot eqiad1-compute/e90bba42-4092-4cf1-a80a-bb379ac5888f_disk@2021-02-18T04:50:14_cloudvirt1024 and backy2 version cab319d8-71a4-11eb-89a6-b02628295df0
Feb 19 04:50:29 cloudvirt1024 wmcs-backup[56558]: [37B blob data]
Feb 19 04:50:30 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:50:30,197] Cleaning up expired backups for VM wikibase-registry.orig-01(e90bba42-4092-4cf1-a80a-bb379ac5888f)
Feb 19 04:50:31 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:50:31,956] Cleaned up 1 expired backups for VM wikibase-registry.orig-01(e90bba42-4092-4cf1-a80a-bb379ac5888f)
Feb 19 04:50:31 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:50:31,957] Creating backup for vm wikibase-registry.libcanada-01(596da987-3e2d-4e1a-8574-a2a5b232104d)
Feb 19 04:50:32 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:50:32,117] Creating differential backup of pool:eqiad1-compute image_name:596da987-3e2d-4e1a-8574-a2a5b232104d_disk from rbd snapshot eqiad1-compute/596da987-3e2d-4e1a-8574-a2a5b232104d_disk@2021-02-18T04:50:42_cloudvirt1024 and backy2 version dbe176aa-71a4-11eb-9cd6-b02628295df0
Feb 19 04:50:58 cloudvirt1024 wmcs-backup[56558]: [37B blob data]
Feb 19 04:50:59 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:50:59,353] Cleaning up expired backups for VM wikibase-registry.libcanada-01(596da987-3e2d-4e1a-8574-a2a5b232104d)
Feb 19 04:51:01 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:51:01,193] Cleaned up 1 expired backups for VM wikibase-registry.libcanada-01(596da987-3e2d-4e1a-8574-a2a5b232104d)
Feb 19 04:51:01 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:51:01,194] Creating backup for vm wikicommunityhealth.wchm01(9392cf0f-524f-40a1-920b-3a6ab8b1ca81)
Feb 19 04:51:01 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:51:01,337] Creating differential backup of pool:eqiad1-compute image_name:9392cf0f-524f-40a1-920b-3a6ab8b1ca81_disk from rbd snapshot eqiad1-compute/9392cf0f-524f-40a1-920b-3a6ab8b1ca81_disk@2021-02-18T04:51:11_cloudvirt1024 and backy2 version ed1c2866-71a4-11eb-a30e-b02628295df0
Feb 19 04:54:00 cloudvirt1024 wmcs-backup[56558]: [37B blob data]
Feb 19 04:54:01 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:54:01,330] Cleaning up expired backups for VM wikicommunityhealth.wchm01(9392cf0f-524f-40a1-920b-3a6ab8b1ca81)
Feb 19 04:54:03 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:54:03,356] Cleaned up 1 expired backups for VM wikicommunityhealth.wchm01(9392cf0f-524f-40a1-920b-3a6ab8b1ca81)
Feb 19 04:54:03 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:54:03,357] Creating backup for vm wikidata-autodesc.wikidata-autodesc(384734b3-4094-4587-8908-70998c56b940)
Feb 19 04:54:03 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:54:03,506] Creating differential backup of pool:eqiad1-compute image_name:384734b3-4094-4587-8908-70998c56b940_disk from rbd snapshot eqiad1-compute/384734b3-4094-4587-8908-70998c56b940_disk@2021-02-18T04:54:26_cloudvirt1024 and backy2 version 60b832e2-71a5-11eb-9195-b02628295df0
Feb 19 04:57:01 cloudvirt1024 wmcs-backup[56558]: [37B blob data]
Feb 19 04:57:02 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:57:02,177] Cleaning up expired backups for VM wikidata-autodesc.wikidata-autodesc(384734b3-4094-4587-8908-70998c56b940)
Feb 19 04:57:04 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:57:04,606] Cleaned up 1 expired backups for VM wikidata-autodesc.wikidata-autodesc(384734b3-4094-4587-8908-70998c56b940)
Feb 19 04:57:04 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:57:04,606] Creating backup for vm wikidata-history-query-service.wdhqs-1(59e95aa5-ae70-47b7-9af8-f5c3f913a992)
Feb 19 04:57:04 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 04:57:04,746] Creating differential backup of pool:eqiad1-compute image_name:59e95aa5-ae70-47b7-9af8-f5c3f913a992_disk from rbd snapshot eqiad1-compute/59e95aa5-ae70-47b7-9af8-f5c3f913a992_disk@2021-02-18T04:57:41_cloudvirt1024 and backy2 version d539cc3e-71a5-11eb-9add-b02628295df0
Feb 19 05:01:58 cloudvirt1024 wmcs-backup[56558]: [37B blob data]
Feb 19 05:01:58 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 05:01:58,785] Cleaning up expired backups for VM wikidata-history-query-service.wdhqs-1(59e95aa5-ae70-47b7-9af8-f5c3f913a992)
Feb 19 05:02:15 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 05:02:15,782] Cleaned up 1 expired backups for VM wikidata-history-query-service.wdhqs-1(59e95aa5-ae70-47b7-9af8-f5c3f913a992)
Feb 19 05:02:15 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 05:02:15,783] Creating backup for vm wikidata-realtime-dumps.wikidata-realtime-dumps(06179518-b478-4fdc-95a5-4b4e18b55208)
Feb 19 05:02:15 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 05:02:15,917] Creating differential backup of pool:eqiad1-compute image_name:06179518-b478-4fdc-95a5-4b4e18b55208_disk from rbd snapshot eqiad1-compute/06179518-b478-4fdc-95a5-4b4e18b55208_disk@2021-02-18T05:03:18_cloudvirt1024 and backy2 version 9e4c85ee-71a6-11eb-8679-b02628295df0
Feb 19 05:05:19 cloudvirt1024 wmcs-backup[56558]: [37B blob data]
Feb 19 05:05:19 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 05:05:19,905] Cleaning up expired backups for VM wikidata-realtime-dumps.wikidata-realtime-dumps(06179518-b478-4fdc-95a5-4b4e18b55208)
Feb 19 05:05:28 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 05:05:28,458] Cleaned up 1 expired backups for VM wikidata-realtime-dumps.wikidata-realtime-dumps(06179518-b478-4fdc-95a5-4b4e18b55208)
Feb 19 05:05:28 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 05:05:28,459] Creating backup for vm wikidocumentaries.roope(9dd91776-e93e-4f56-88cb-ce355f918076)
Feb 19 05:05:28 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 05:05:28,621] Creating differential backup of pool:eqiad1-compute image_name:9dd91776-e93e-4f56-88cb-ce355f918076_disk from rbd snapshot eqiad1-compute/9dd91776-e93e-4f56-88cb-ce355f918076_disk@2021-02-18T05:06:31_cloudvirt1024 and backy2 version 1102c4b8-71a7-11eb-bda1-b02628295df0
Feb 19 05:05:53 cloudvirt1024 wmcs-backup[56558]: [37B blob data]
Feb 19 05:05:53 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 05:05:53,497] Cleaning up expired backups for VM wikidocumentaries.roope(9dd91776-e93e-4f56-88cb-ce355f918076)
Feb 19 05:05:56 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 05:05:56,527] Cleaned up 1 expired backups for VM wikidocumentaries.roope(9dd91776-e93e-4f56-88cb-ce355f918076)
Feb 19 05:05:56 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 05:05:56,527] Creating backup for vm wikidocumentaries.hupu(b6305f09-d892-4122-a08e-bb00d0cd925f)
Feb 19 05:05:56 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 05:05:56,693] Creating differential backup of pool:eqiad1-compute image_name:b6305f09-d892-4122-a08e-bb00d0cd925f_disk from rbd snapshot eqiad1-compute/b6305f09-d892-4122-a08e-bb00d0cd925f_disk@2021-02-18T05:06:59_cloudvirt1024 and backy2 version 21d6b06a-71a7-11eb-bc24-b02628295df0
Feb 19 05:06:58 cloudvirt1024 wmcs-backup[56558]: [37B blob data]
Feb 19 05:06:59 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 05:06:59,342] Cleaning up expired backups for VM wikidocumentaries.hupu(b6305f09-d892-4122-a08e-bb00d0cd925f)
Feb 19 05:07:01 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 05:07:01,442] Cleaned up 1 expired backups for VM wikidocumentaries.hupu(b6305f09-d892-4122-a08e-bb00d0cd925f)
Feb 19 05:07:01 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 05:07:01,443] Creating backup for vm wikilabels.wikilabels-backups-01(4a85bfa1-f920-489c-badb-e8cc9f8d5692)
Feb 19 05:07:01 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 05:07:01,578] Creating differential backup of pool:eqiad1-compute image_name:4a85bfa1-f920-489c-badb-e8cc9f8d5692_disk from rbd snapshot eqiad1-compute/4a85bfa1-f920-489c-badb-e8cc9f8d5692_disk@2021-02-18T05:08:06_cloudvirt1024 and backy2 version 49a827b8-71a7-11eb-9cd6-b02628295df0
Feb 19 05:07:14 cloudvirt1024 wmcs-backup[56558]: [37B blob data]
Feb 19 05:07:15 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 05:07:15,139] Cleaning up expired backups for VM wikilabels.wikilabels-backups-01(4a85bfa1-f920-489c-badb-e8cc9f8d5692)
Feb 19 05:07:16 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 05:07:16,624] Cleaned up 1 expired backups for VM wikilabels.wikilabels-backups-01(4a85bfa1-f920-489c-badb-e8cc9f8d5692)
Feb 19 05:07:16 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 05:07:16,625] Creating backup for vm wikilabels.wikilabels-staging-02(65343972-d77a-4c93-a69f-6a0ff15d082c)
Feb 19 05:07:16 cloudvirt1024 wmcs-backup[56558]: INFO:[2021-02-19 05:07:16,773] Creating differential backup of pool:eqiad1-compute image_name:65343972-d77a-4c93-a69f-6a0ff15d082c_disk from rbd snapshot eqiad1-compute/65343972-d77a-4c93-a69f-6a0ff15d082c_disk@2021-02-18T05:08:24_cloudvirt1024 and backy2 version 547f81fe-71a7-11eb-bee1-b02628295df0
Feb 19 05:07:31 cloudvirt1024 wmcs-backup[56558]: Traceback (most recent call last):
Feb 19 05:07:31 cloudvirt1024 wmcs-backup[56558]: File "/usr/local/sbin/wmcs-backup", line 2012, in <module>
Feb 19 05:07:31 cloudvirt1024 wmcs-backup[56558]: args.func()
Feb 19 05:07:31 cloudvirt1024 wmcs-backup[56558]: File "/usr/local/sbin/wmcs-backup", line 1847, in <lambda>
Feb 19 05:07:31 cloudvirt1024 wmcs-backup[56558]: ).backup_assigned_vms(from_cache=args.from_cache, noop=args.noop)
Feb 19 05:07:31 cloudvirt1024 wmcs-backup[56558]: File "/usr/local/sbin/wmcs-backup", line 1312, in backup_assigned_vms
Feb 19 05:07:31 cloudvirt1024 wmcs-backup[56558]: noop=noop,
Feb 19 05:07:31 cloudvirt1024 wmcs-backup[56558]: File "/usr/local/sbin/wmcs-backup", line 1121, in create_vm_backup
Feb 19 05:07:31 cloudvirt1024 wmcs-backup[56558]: vm_info=vm_info, noop=noop
Feb 19 05:07:31 cloudvirt1024 wmcs-backup[56558]: File "/usr/local/sbin/wmcs-backup", line 1030, in create_vm_backup
Feb 19 05:07:31 cloudvirt1024 wmcs-backup[56558]: new_backup = self.vms_backups[vm_id].create_next_backup(noop=noop)
Feb 19 05:07:31 cloudvirt1024 wmcs-backup[56558]: File "/usr/local/sbin/wmcs-backup", line 720, in create_next_backup
Feb 19 05:07:31 cloudvirt1024 wmcs-backup[56558]: noop=noop,
Feb 19 05:07:31 cloudvirt1024 wmcs-backup[56558]: File "/usr/local/sbin/wmcs-backup", line 551, in create_diff_backup
Feb 19 05:07:31 cloudvirt1024 wmcs-backup[56558]: noop=noop,
Feb 19 05:07:31 cloudvirt1024 wmcs-backup[56558]: File "/usr/lib/python3/dist-packages/rbd2backy2.py", line 332, in create_diff_backup
Feb 19 05:07:31 cloudvirt1024 wmcs-backup[56558]: noop=noop,
Feb 19 05:07:31 cloudvirt1024 wmcs-backup[56558]: File "/usr/lib/python3/dist-packages/rbd2backy2.py", line 31, in run_command
Feb 19 05:07:31 cloudvirt1024 wmcs-backup[56558]: output = subprocess.check_output(args).decode("utf8")
Feb 19 05:07:31 cloudvirt1024 wmcs-backup[56558]: File "/usr/lib/python3.7/subprocess.py", line 395, in check_output
Feb 19 05:07:31 cloudvirt1024 wmcs-backup[56558]: **kwargs).stdout
Feb 19 05:07:31 cloudvirt1024 wmcs-backup[56558]: File "/usr/lib/python3.7/subprocess.py", line 487, in run
Feb 19 05:07:31 cloudvirt1024 wmcs-backup[56558]: output=stdout, stderr=stderr)
Feb 19 05:07:31 cloudvirt1024 wmcs-backup[56558]: subprocess.CalledProcessError: Command '['/usr/bin/backy2', 'backup', '--snapshot-name', '2021-02-19T05:07:16_cloudvirt1024', '--rbd', '/tmp/tmpbgykl78e', '--from-version', '547f81fe-71a7-11eb-bee1-b02628295df0', 'rbd://eqiad1-compute/65343972-d77a-4c93-a69f-6a0ff15d082c_disk@2021-02-19T05:07:16_cloudvirt1024', '65343972-d77a-4c93-a69f-6a0ff15d082c_disk', '--expire', '2021-02-24 05:07:16', '--tag', 'differential_backup']' died with <Signals.SIGSEGV: 11>.
Feb 19 05:07:31 cloudvirt1024 systemd[1]: backup_vms.service: Main process exited, code=exited, status=1/FAILURE
Feb 19 05:07:31 cloudvirt1024 systemd[1]: backup_vms.service: Failed with result 'exit-code'.