Page MenuHomePhabricator
Create Task
Maniphest T313287

paws-k8s-worker-1 file system errors
Closed, ResolvedPublic
Actions

Description

[12715.299666] EXT4-fs error (device dm-0): ext4_lookup:1646: inode #3311670: comm dockerd: deleted inode referenced: 3348076
[12715.305081] EXT4-fs error (device dm-0): ext4_lookup:1646: inode #3311670: comm dockerd: deleted inode referenced: 3348248
[12715.310565] EXT4-fs error (device dm-0): ext4_lookup:1646: inode #3311670: comm dockerd: deleted inode referenced: 3291260
[12715.315492] EXT4-fs error (device dm-0): ext4_lookup:1646: inode #3311670: comm dockerd: deleted inode referenced: 3348074
[12775.563858] EXT4-fs error (device dm-0): ext4_lookup:1646: inode #3311670: comm dockerd: deleted inode referenced: 3348077
[12775.569745] EXT4-fs error (device dm-0): ext4_lookup:1646: inode #3311670: comm dockerd: deleted inode referenced: 3348081
[12775.574791] EXT4-fs error (device dm-0): ext4_lookup:1646: inode #3311670: comm dockerd: deleted inode referenced: 3348078
[12775.579989] EXT4-fs error (device dm-0): ext4_lookup:1646: inode #3311670: comm dockerd: deleted inode referenced: 3348082
[12775.585203] EXT4-fs error (device dm-0): ext4_lookup:1646: inode #3311670: comm dockerd: deleted inode referenced: 3348076
[12775.590148] EXT4-fs error (device dm-0): ext4_lookup:1646: inode #3311670: comm dockerd: deleted inode referenced: 3348248
[12775.595866] EXT4-fs error (device dm-0): ext4_lookup:1646: inode #3311670: comm dockerd: deleted inode referenced: 3291260
[12775.599968] EXT4-fs error (device dm-0): ext4_lookup:1646: inode #3311670: comm dockerd: deleted inode referenced: 3348074
[12775.605008] EXT4-fs error (device dm-0): ext4_lookup:1646: inode #3311670: comm dockerd: deleted inode referenced: 3348080
[12775.609523] EXT4-fs error (device dm-0): ext4_lookup:1646: inode #3311670: comm dockerd: deleted inode referenced: 3348075
[12835.911086] EXT4-fs error (device dm-0): ext4_lookup:1646: inode #3311670: comm dockerd: deleted inode referenced: 3348077
[12835.916729] EXT4-fs error (device dm-0): ext4_lookup:1646: inode #3311670: comm dockerd: deleted inode referenced: 3348081
[12835.922611] EXT4-fs error (device dm-0): ext4_lookup:1646: inode #3311670: comm dockerd: deleted inode referenced: 3348078
[12835.928748] EXT4-fs error (device dm-0): ext4_lookup:1646: inode #3311670: comm dockerd: deleted inode referenced: 3348082
[12835.934232] EXT4-fs error (device dm-0): ext4_lookup:1646: inode #3311670: comm dockerd: deleted inode referenced: 3348076
[12835.939611] EXT4-fs error (device dm-0): ext4_lookup:1646: inode #3311670: comm dockerd: deleted inode referenced: 3348248

Event Timeline

taavi created this task.Jul 19 2022, 8:11 AM
Stashbot added a comment.Jul 19 2022, 8:12 AM

Mentioned in SAL (#wikimedia-cloud) [2022-07-19T08:12:12Z] <taavi> drain paws-k8s-worker-1 T313287

rook subscribed.Jul 20 2022, 3:16 PM

Getting kicked out after a short time on ssh. Setting fstab to run fsck and rebooting...

RhinosF1 subscribed.Jul 20 2022, 3:25 PM
rook added a comment.Jul 20 2022, 6:51 PM

Deployed paws-k8s-worker-8 to replace paws-k8s-worker-1

rook added a comment.Aug 2 2022, 12:22 PM

Deleting worker-1 from cluster and shutting down.

Stashbot added a comment.Aug 2 2022, 12:22 PM

Mentioned in SAL (#wikimedia-cloud) [2022-08-02T12:22:56Z] <Rook> deleting worker-1 from cluster and shutting down T313287

rook added a comment.Aug 10 2022, 3:20 PM

paws-k8s-worker-1 deleted.

rook closed this task as Resolved.Aug 10 2022, 3:20 PM
rook claimed this task.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL