之前写过一篇PVE ZFS更换故障盘的文章“Proxmox VE的ZFS如何更换故障盘”，有个大佬点评：其实你这样替换会有些问题，因为是用一个新盘的整盘替换了一个旧盘的分区。随着不停替换，以后会影响启动的。目前看没问题。我挺在意这个问题，所以去查了一下，在Reddit和一些论坛上确实有人分享这个经验，和大佬点评的一样：如果像我之前那样直接把整盘当作新设备加入，而 ZFS 最初 pool 是以 “分区 (partition)” 作为 member（例如p3）的话，可能造成分区结构和 GUID / 分区表不一致，长期累积多次替换，可能会影响启动 (boot)，尤其是当 pool 包含系统盘(boot pool / rpool) 的情况下。也有人说如果新盘没有明确 copy 分区表 (partition layout) + 重建 boot 分区 (EFI/BIOS) + 安装boot loader (如 grub 或 proxmox-boot-tool)，系统可能无法从新盘启动。可以看到，这些问题都是基于将ZFS作为系统盘启动会遇到的，如果你不是将ZFS作为系统盘，而是作为数据盘则我原文是没问题的。但如果将ZFS作为系统盘，则建议使用下文方法更换故障盘，避免潜在风险。在此谢谢大佬指正！本文以/dev/nvme2n1故障为例，其实也不能说是故障，这个设备不支持3个PCIE 4.0的U.2盘，装上容易掉盘，当前虽然lsscsi显示了盘，实际上是不可用的，可以看到zfs status中有个盘状态是REMOVED。root@usa-zfs-amd-23:~# lsscsi [16:0:0:0] cd/dvd AMI Virtual CDROM0 1.00 /dev/sr0 [N:0:0:1] disk INTEL SSDPF2KX038XZ__1 /dev/nvme0n1 [N:1:0:1] disk INTEL SSDPF2KX038XZ__1 /dev/nvme1n1 [N:2:0:1] disk INTEL SSDPF2KX038XZ__1 /dev/nvme2n1 root@usa-zfs-amd-23:~# root@usa-zfs-amd-23:~# ls -l /dev/disk/by-path/ total 0 lrwxrwxrwx 1 root root 9 Jun 12 11:05 pci-0000:04:00.3-usb-0:2.1:1.0-scsi-0:0:0:0 -> ../../sr0 lrwxrwxrwx 1 root root 13 Jun 12 11:05 pci-0000:41:00.0-nvme-1 -> ../../nvme2n1 lrwxrwxrwx 1 root root 15 Jun 12 11:05 pci-0000:41:00.0-nvme-1-part1 -> ../../nvme2n1p1 lrwxrwxrwx 1 root root 15 Jun 12 11:05 pci-0000:41:00.0-nvme-1-part2 -> ../../nvme2n1p2 lrwxrwxrwx 1 root root 15 Jun 12 11:05 pci-0000:41:00.0-nvme-1-part3 -> ../../nvme2n1p3 lrwxrwxrwx 1 root root 13 Jun 12 11:05 pci-0000:82:00.0-nvme-1 -> ../../nvme0n1 lrwxrwxrwx 1 root root 15 Jun 12 11:05 pci-0000:82:00.0-nvme-1-part1 -> ../../nvme0n1p1 lrwxrwxrwx 1 root root 15 Jun 12 11:05 pci-0000:82:00.0-nvme-1-part2 -> ../../nvme0n1p2 lrwxrwxrwx 1 root root 15 Jun 12 11:05 pci-0000:82:00.0-nvme-1-part3 -> ../../nvme0n1p3 lrwxrwxrwx 1 root root 13 Jun 12 11:05 pci-0000:83:00.0-nvme-1 -> ../../nvme1n1 lrwxrwxrwx 1 root root 15 Jun 12 11:05 pci-0000:83:00.0-nvme-1-part1 -> ../../nvme1n1p1 lrwxrwxrwx 1 root root 15 Jun 12 11:05 pci-0000:83:00.0-nvme-1-part2 -> ../../nvme1n1p2 lrwxrwxrwx 1 root root 15 Jun 12 11:05 pci-0000:83:00.0-nvme-1-part3 -> ../../nvme1n1p3 root@usa-zfs-amd-23:~# root@usa-zfs-amd-23:~# root@usa-zfs-amd-23:~# zpool status rpool pool: rpool state: DEGRADED status: One or more devices has been removed by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Online the device using zpool online' or replace the device with 'zpool replace'. scan: resilvered 495G in 00:29:49 with 0 errors on Thu Jun 12 11:05:33 2025 config: NAME STATE READ WRITE CKSUM rpool DEGRADED 0 0 0 raidz1-0 DEGRADED 0 0 0 nvme-eui.01000000000000005cd2e462c4645651-part3 ONLINE 0 0 0 nvme-eui.01000000000000005cd2e4e4c3645651-part3 ONLINE 0 0 0 nvme-eui.01000000000000005cd2e47dd4695651-part3 REMOVED 0 0 0 errors: No known data errors root@usa-zfs-amd-23:~# 更换新盘后，查看确认一下root@usa-zfs-amd-23:~# ls -la /dev/disk/by-id total 0 drwxr-xr-x 2 root root 600 Dec 6 07:41 . drwxr-xr-x 9 root root 180 Jun 12 10:41 .. lrwxrwxrwx 1 root root 13 Dec 6 07:41 nvme-eui.000000000000000100a0750127b4e0b2 -> ../../nvme2n1 lrwxrwxrwx 1 root root 13 Jun 12 11:05 nvme-eui.01000000000000005cd2e462c4645651 -> ../../nvme0n1 lrwxrwxrwx 1 root root 15 Jun 12 11:05 nvme-eui.01000000000000005cd2e462c4645651-part1 -> ../../nvme0n1p1 lrwxrwxrwx 1 root root 15 Jun 12 11:05 nvme-eui.01000000000000005cd2e462c4645651-part2 -> ../../nvme0n1p2 lrwxrwxrwx 1 root root 15 Jun 12 11:05 nvme-eui.01000000000000005cd2e462c4645651-part3 -> ../../nvme0n1p3 lrwxrwxrwx 1 root root 13 Jun 12 11:05 nvme-eui.01000000000000005cd2e4e4c3645651 -> ../../nvme1n1 lrwxrwxrwx 1 root root 15 Jun 12 11:05 nvme-eui.01000000000000005cd2e4e4c3645651-part1 -> ../../nvme1n1p1 lrwxrwxrwx 1 root root 15 Jun 12 11:05 nvme-eui.01000000000000005cd2e4e4c3645651-part2 -> ../../nvme1n1p2 lrwxrwxrwx 1 root root 15 Jun 12 11:05 nvme-eui.01000000000000005cd2e4e4c3645651-part3 -> ../../nvme1n1p3 lrwxrwxrwx 1 root root 13 Jun 12 11:05 nvme-INTEL_SSDPF2KX038XZ_PHAO3482049K3P8UGN -> ../../nvme1n1 lrwxrwxrwx 1 root root 13 Jun 12 11:05 nvme-INTEL_SSDPF2KX038XZ_PHAO3482049K3P8UGN_1 -> ../../nvme1n1 lrwxrwxrwx 1 root root 15 Jun 12 11:05 nvme-INTEL_SSDPF2KX038XZ_PHAO3482049K3P8UGN_1-part1 -> ../../nvme1n1p1 lrwxrwxrwx 1 root root 15 Jun 12 11:05 nvme-INTEL_SSDPF2KX038XZ_PHAO3482049K3P8UGN_1-part2 -> ../../nvme1n1p2 lrwxrwxrwx 1 root root 15 Jun 12 11:05 nvme-INTEL_SSDPF2KX038XZ_PHAO3482049K3P8UGN_1-part3 -> ../../nvme1n1p3 lrwxrwxrwx 1 root root 15 Jun 12 11:05 nvme-INTEL_SSDPF2KX038XZ_PHAO3482049K3P8UGN-part1 -> ../../nvme1n1p1 lrwxrwxrwx 1 root root 15 Jun 12 11:05 nvme-INTEL_SSDPF2KX038XZ_PHAO3482049K3P8UGN-part2 -> ../../nvme1n1p2 lrwxrwxrwx 1 root root 15 Jun 12 11:05 nvme-INTEL_SSDPF2KX038XZ_PHAO3482049K3P8UGN-part3 -> ../../nvme1n1p3 lrwxrwxrwx 1 root root 13 Jun 12 11:05 nvme-INTEL_SSDPF2KX038XZ_PHAO348204D93P8UGN -> ../../nvme0n1 lrwxrwxrwx 1 root root 13 Jun 12 11:05 nvme-INTEL_SSDPF2KX038XZ_PHAO348204D93P8UGN_1 -> ../../nvme0n1 lrwxrwxrwx 1 root root 15 Jun 12 11:05 nvme-INTEL_SSDPF2KX038XZ_PHAO348204D93P8UGN_1-part1 -> ../../nvme0n1p1 lrwxrwxrwx 1 root root 15 Jun 12 11:05 nvme-INTEL_SSDPF2KX038XZ_PHAO348204D93P8UGN_1-part2 -> ../../nvme0n1p2 lrwxrwxrwx 1 root root 15 Jun 12 11:05 nvme-INTEL_SSDPF2KX038XZ_PHAO348204D93P8UGN_1-part3 -> ../../nvme0n1p3 lrwxrwxrwx 1 root root 15 Jun 12 11:05 nvme-INTEL_SSDPF2KX038XZ_PHAO348204D93P8UGN-part1 -> ../../nvme0n1p1 lrwxrwxrwx 1 root root 15 Jun 12 11:05 nvme-INTEL_SSDPF2KX038XZ_PHAO348204D93P8UGN-part2 -> ../../nvme0n1p2 lrwxrwxrwx 1 root root 15 Jun 12 11:05 nvme-INTEL_SSDPF2KX038XZ_PHAO348204D93P8UGN-part3 -> ../../nvme0n1p3 lrwxrwxrwx 1 root root 13 Dec 6 07:41 nvme-Micron_9300_MTFDHAL3T8TDP_201627B4E0B2 -> ../../nvme2n1 lrwxrwxrwx 1 root root 13 Dec 6 07:41 nvme-Micron_9300_MTFDHAL3T8TDP_201627B4E0B2_1 -> ../../nvme2n1 lrwxrwxrwx 1 root root 9 Jun 12 11:05 usb-AMI_Virtual_CDROM0_AAAABBBBCCCC1-0:0 -> ../../sr0 root@usa-zfs-amd-23:~# root@usa-zfs-amd-23:~# lsscsi [16:0:0:0] cd/dvd AMI Virtual CDROM0 1.00 /dev/sr0 [N:0:0:1] disk INTEL SSDPF2KX038XZ__1 /dev/nvme0n1 [N:1:0:1] disk INTEL SSDPF2KX038XZ__1 /dev/nvme1n1 [N:2:1:1] disk Micron_9300_MTFDHAL3T8TDP__1 /dev/nvme2n1 已经正常识别了，接下来查看下分区，可以看到新盘/dev/nvme2n1是没有分区的，但其他正常盘例如/dev/nvme1n1是有分区的root@usa-zfs-amd-23:~# lsblk /dev/nvme2n1 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS nvme2n1 259:0 0 3.5T 0 disk root@usa-zfs-amd-23:~# root@usa-zfs-amd-23:~# lsblk /dev/nvme1n1 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS nvme1n1 259:5 0 3.5T 0 disk ├─nvme1n1p1 259:6 0 1007K 0 part ├─nvme1n1p2 259:7 0 1G 0 part └─nvme1n1p3 259:8 0 3.5T 0 part 现在我们基于Proxmox官方推荐的方式开始替换硬盘① 复制旧盘分区表到新盘先从正常可用的盘（例如 /dev/nvme0n1）复制分区表到新盘root@usa-zfs-amd-23:~# sgdisk --replicate=/dev/nvme2n1 /dev/nvme0n1 The operation has completed successfully. 随机化 GUIDroot@usa-zfs-amd-23:~# sgdisk --randomize-guids /dev/nvme2n1 The operation has completed successfully. 检查确认，其结构应该与 /dev/nvme0n1 一致root@usa-zfs-amd-23:~# lsblk /dev/nvme2n1 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS nvme2n1 259:0 0 3.5T 0 disk ├─nvme2n1p1 259:1 0 1007K 0 part ├─nvme2n1p2 259:2 0 1G 0 part └─nvme2n1p3 259:3 0 3.5T 0 part ② 初始化 boot 分区（p1/p2）以便可引导Proxmox 使用 proxmox-boot-tool 来管理 ESP。格式化 ESP（p2分区通常是EFI，从容量也可以看出来）root@usa-zfs-amd-23:~# proxmox-boot-tool format /dev/nvme2n1p2 UUID="" SIZE="1073741824" FSTYPE="" PARTTYPE="c12a7328-f81f-11d2-ba4b-00a0c93ec93b" PKNAME="nvme2n1" MOUNTPOINT="" Formatting '/dev/nvme2n1p2' as vfat.. mkfs.fat 4.2 (2021-01-31) Done. 初始化 bootloaderroot@usa-zfs-amd-23:~# proxmox-boot-tool init /dev/nvme2n1p2 Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace.. UUID="096E-9317" SIZE="1073741824" FSTYPE="vfat" PARTTYPE="c12a7328-f81f-11d2-ba4b-00a0c93ec93b" PKNAME="nvme2n1" MOUNTPOINT="" Mounting '/dev/nvme2n1p2' on '/var/tmp/espmounts/096E-9317'. Installing systemd-boot.. Created "/var/tmp/espmounts/096E-9317/EFI/systemd". Created "/var/tmp/espmounts/096E-9317/EFI/BOOT". Created "/var/tmp/espmounts/096E-9317/loader". Created "/var/tmp/espmounts/096E-9317/loader/entries". Created "/var/tmp/espmounts/096E-9317/EFI/Linux". Copied "/usr/lib/systemd/boot/efi/systemd-bootx64.efi" to "/var/tmp/espmounts/096E-9317/EFI/systemd/systemd-bootx64.efi". Copied "/usr/lib/systemd/boot/efi/systemd-bootx64.efi" to "/var/tmp/espmounts/096E-9317/EFI/BOOT/BOOTX64.EFI". Random seed file /var/tmp/espmounts/096E-9317/loader/random-seed successfully written (32 bytes). Created EFI boot entry "Linux Boot Manager". Configuring systemd-boot.. Unmounting '/dev/nvme2n1p2'. Adding '/dev/nvme2n1p2' to list of synced ESPs.. Refreshing kernels and initrds.. Running hook script 'proxmox-auto-removal'.. Running hook script 'zz-proxmox-boot'.. Copying and configuring kernels on /dev/disk/by-uuid/096E-9317 Copying kernel and creating boot-entry for 6.8.12-4-pve Copying kernel and creating boot-entry for 6.8.12-9-pve Copying and configuring kernels on /dev/disk/by-uuid/59AD-8721 Copying kernel and creating boot-entry for 6.8.12-4-pve Copying kernel and creating boot-entry for 6.8.12-9-pve Copying and configuring kernels on /dev/disk/by-uuid/59F7-7114 Copying kernel and creating boot-entry for 6.8.12-4-pve Copying kernel and creating boot-entry for 6.8.12-9-pve WARN: /dev/disk/by-uuid/59F7-DB10 does not exist - clean '/etc/kernel/proxmox-boot-uuids'! - skipping 确认root@usa-zfs-amd-23:~# proxmox-boot-tool status Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace.. System currently booted with uefi 096E-9317 is configured with: uefi (versions: 6.8.12-4-pve, 6.8.12-9-pve) 59AD-8721 is configured with: uefi (versions: 6.8.12-4-pve, 6.8.12-9-pve) 59F7-7114 is configured with: uefi (versions: 6.8.12-4-pve, 6.8.12-9-pve) WARN: /dev/disk/by-uuid/59F7-DB10 does not exist - clean '/etc/kernel/proxmox-boot-uuids'! - skipping 这是正确的，出现1条 WARN 只是说明有一个旧 UUID（59F7-DB10）在 /etc/kernel/proxmox-boot-uuids 中，但现在找不到对应设备，可以清理它（可选，但建议）：root@usa-zfs-amd-23:~# cat /etc/kernel/proxmox-boot-uuids 096E-9317 59AD-8721 59F7-7114 59F7-DB10 正常应该一个盘一个uuid，可以看到多出一个。通过以下命令删除及刷新配置：root@usa-zfs-amd-23:~# grep -v '59F7-DB10' /etc/kernel/proxmox-boot-uuids > /tmp/uuids && mv /tmp/uuids /etc/kernel/proxmox-boot-uuids root@usa-zfs-amd-23:~# proxmox-boot-tool refresh Running hook script 'proxmox-auto-removal'.. Running hook script 'zz-proxmox-boot'.. Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace.. Copying and configuring kernels on /dev/disk/by-uuid/096E-9317 Copying kernel and creating boot-entry for 6.8.12-4-pve Copying kernel and creating boot-entry for 6.8.12-9-pve Copying and configuring kernels on /dev/disk/by-uuid/59AD-8721 Copying kernel and creating boot-entry for 6.8.12-4-pve Copying kernel and creating boot-entry for 6.8.12-9-pve Copying and configuring kernels on /dev/disk/by-uuid/59F7-7114 Copying kernel and creating boot-entry for 6.8.12-4-pve Copying kernel and creating boot-entry for 6.8.12-9-pve 现在重新查看status，已经没有警告了：root@usa-zfs-amd-23:~# proxmox-boot-tool status Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace.. System currently booted with uefi 096E-9317 is configured with: uefi (versions: 6.8.12-4-pve, 6.8.12-9-pve) 59AD-8721 is configured with: uefi (versions: 6.8.12-4-pve, 6.8.12-9-pve) 59F7-7114 is configured with: uefi (versions: 6.8.12-4-pve, 6.8.12-9-pve) ③ ZFS 用新盘替换旧盘（使用分区，不是整盘）找到新旧盘p3分区信息，留意其中新盘的p3分区信息：root@usa-zfs-amd-23:~# ls -la /dev/disk/by-id | grep nvme2n1 lrwxrwxrwx 1 root root 13 Dec 6 13:34 nvme-eui.000000000000000100a0750127b4e0b2 -> ../../nvme2n1 lrwxrwxrwx 1 root root 15 Dec 6 13:34 nvme-eui.000000000000000100a0750127b4e0b2-part1 -> ../../nvme2n1p1 lrwxrwxrwx 1 root root 15 Dec 6 13:35 nvme-eui.000000000000000100a0750127b4e0b2-part2 -> ../../nvme2n1p2 lrwxrwxrwx 1 root root 15 Dec 6 13:34 nvme-eui.000000000000000100a0750127b4e0b2-part3 -> ../../nvme2n1p3 lrwxrwxrwx 1 root root 13 Dec 6 13:34 nvme-Micron_9300_MTFDHAL3T8TDP_201627B4E0B2 -> ../../nvme2n1 lrwxrwxrwx 1 root root 13 Dec 6 13:34 nvme-Micron_9300_MTFDHAL3T8TDP_201627B4E0B2_1 -> ../../nvme2n1 lrwxrwxrwx 1 root root 15 Dec 6 13:34 nvme-Micron_9300_MTFDHAL3T8TDP_201627B4E0B2_1-part1 -> ../../nvme2n1p1 lrwxrwxrwx 1 root root 15 Dec 6 13:35 nvme-Micron_9300_MTFDHAL3T8TDP_201627B4E0B2_1-part2 -> ../../nvme2n1p2 lrwxrwxrwx 1 root root 15 Dec 6 13:34 nvme-Micron_9300_MTFDHAL3T8TDP_201627B4E0B2_1-part3 -> ../../nvme2n1p3 lrwxrwxrwx 1 root root 15 Dec 6 13:34 nvme-Micron_9300_MTFDHAL3T8TDP_201627B4E0B2-part1 -> ../../nvme2n1p1 lrwxrwxrwx 1 root root 15 Dec 6 13:35 nvme-Micron_9300_MTFDHAL3T8TDP_201627B4E0B2-part2 -> ../../nvme2n1p2 lrwxrwxrwx 1 root root 15 Dec 6 13:34 nvme-Micron_9300_MTFDHAL3T8TDP_201627B4E0B2-part3 -> ../../nvme2n1p3 root@usa-zfs-amd-23:~# root@usa-zfs-amd-23:~# zpool status rpool pool: rpool state: DEGRADED status: One or more devices has been removed by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Online the device using zpool online' or replace the device with 'zpool replace'. scan: resilvered 495G in 00:29:49 with 0 errors on Thu Jun 12 11:05:33 2025 config: NAME STATE READ WRITE CKSUM rpool DEGRADED 0 0 0 raidz1-0 DEGRADED 0 0 0 nvme-eui.01000000000000005cd2e462c4645651-part3 ONLINE 0 0 0 nvme-eui.01000000000000005cd2e4e4c3645651-part3 ONLINE 0 0 0 nvme-eui.01000000000000005cd2e47dd4695651-part3 REMOVED 0 0 0 errors: No known data errors 开始用新盘p3分区替换旧盘p3分区root@usa-zfs-amd-23:~# zpool replace rpool nvme-eui.01000000000000005cd2e47dd4695651-part3 /dev/disk/by-id/nvme-eui.000000000000000100a0750127b4e0b2-part3 查看替换状态root@usa-zfs-amd-23:~# zpool status rpool pool: rpool state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Sat Dec 6 13:46:10 2025 186G / 3.56T scanned at 5.18G/s, 0B / 3.55T issued 0B resilvered, 0.00% done, no estimated completion time config: NAME STATE READ WRITE CKSUM rpool DEGRADED 0 0 0 raidz1-0 DEGRADED 0 0 0 nvme-eui.01000000000000005cd2e462c4645651-part3 ONLINE 0 0 0 nvme-eui.01000000000000005cd2e4e4c3645651-part3 ONLINE 0 0 0 replacing-2 DEGRADED 0 0 0 nvme-eui.01000000000000005cd2e47dd4695651-part3 REMOVED 0 0 0 nvme-eui.000000000000000100a0750127b4e0b2-part3 ONLINE 0 0 0 errors: No known data errors 现在只要等待就可以，完成后就是正常ONLINE状态。
PBS备份PVE很好用，对于PVE以外的系统，PBS给了proxmox-backup-client这个客户端，在Debian/Ubuntu下有官方维护的包，但CentOS下没有。github上有个大神封装了RHEL8和RHEL9的客户端，也支持RockyLinux8 / RockyLinux9 / AlmaLinux8 / AlmaLinux 9。魔方云有提供一个冷备份的功能，但是我不太喜欢。魔方云的这个冷备份功能是通过libvirt的blockcopy功能导出备份，然后scp去备份服务器。例如：virsh -c qemu+tcp:///system blockcopy kvm100 vda --xml /home/kvm/kvm100/backup-disk.xml --wait --transient-job --verbose 这样能安全的克隆出正在运行的虚拟机副本，但这也相当于将虚拟机全盘在宿主服务器硬盘重写了一遍，然后再全盘scp复制去备份服务器硬盘也重写一边，每次备份都会重复这一循环。对硬盘寿命的损耗、和每次scp出去对内网宽带的占用是很大的，另外我测试单节点3TB+数据、200虚拟机、内网到备份服务器10Gbps端口的情况下，魔方云冷备份需要24小时左右耗时也很长。PBS支持权限控制、备份压缩等，最重要的是支持增量备份，这点很好！完成首次备份之后，第二次备份它会校对首次备份数据和本次备份数据，只会备份和传输修改、新增部分，这意味着之后的备份不用再次“全盘打包”以“劳民伤财”的方式进行。如果你直接用proxmox-backup-client客户端备份魔方云的/home目录，那么对正在运行的虚拟机来说，这个备份数据是不完整的，将丢失还未存储的临时数据，比如内存中待写入硬盘的数据、缓存数据。对备份高要求的人来说，应该和魔方云官方做法一样，写个脚本读取所有kvmid，然后通过libvirt的blockcopy功能逐个导出备份，然后再通过proxmox-backup-client客户端上传，然后删除本地的备份，循环这步骤。但对我这种不承诺提供备份、也无备份服务，仅做内部安全风险控制的，直接默认整个/home打包就行，简单、方便。这种备份也不是不能用，因为现代系统是具备纠错功能的，你恢复后其实就相当于断电关机后再启动。对要求不高的普通用户来说比起极端情况下数据丢失，断电关机这点影响算不得什么。当然，如果是收了客户备份的钱，还是建议开放魔方云自己的冷备份功能给客户或者基于blockcopy进行备份。魔方云宿主服务器系统是CentOS Stream 8，需要安装大佬封装的客户端：https://github.com/TomGem/proxmox-backup-client安装依赖：dnf install systemd-libs libgcc libzstd libacl fuse3-libs libuuid openssl-libs 下载并安装rpm：wget https://github.com/TomGem/proxmox-backup-client/releases/download/3.0.4/proxmox-backup-3.0.4-1.rhel8.x86_64.rpm dnf install proxmox-backup-3.0.4-1.x86_64.rpm 完成后查看版本能正常输出就安装好了：proxmox-backup-client version PBS配置好以后在魔方云宿主服务器登入测试：[root@node127 ~]# proxmox-backup-client login --repository mfy-hk-platinum-33@pbs@192.168.2.253:8007:mfy-hk-platinum-33 Password for "mfy-hk-platinum-33@pbs": ****************** fingerprint: 40:6d:39:a2:32:45:ee:ac:28:d4:24:eb:89:79:ac:54:11:a7:33:f9:29:d1:ef:21:d0:cb:83:73:bf:f1:d6:5e Are you sure you want to continue connecting? (y/n): y fingerprint: 40:6d:39:a2:32:45:ee:ac:28:d4:24:eb:89:79:ac:54:11:a7:33:f9:29:d1:ef:21:d0:cb:83:73:bf:f1:d6:5e Are you sure you want to continue connecting? (y/n): y 备份整个/home目录proxmox-backup-client backup home.pxar:/home --repository mfy-hk-platinum-33@pbs@192.168.2.253:8007:mfy-hk-platinum-33 完成后可查看快照[root@node127 ~]# proxmox-backup-client snapshot list --repository mfy-hk-platinum-33@pbs@192.168.2.253:8007:mfy-hk-platinum-33 Password for "mfy-hk-nvmenode-33@pbs": ****************** ┌───────────────────────────────────┬────────────┬────────────────────────────────────┐ │ snapshot │ size │ files │ ╞═══════════════════════════════════╪════════════╪════════════════════════════════════╡ │ host/node127/2025-10-31T17:55:44Z │ 56.961 GiB │ catalog.pcat1 home.pxar index.json │ ├───────────────────────────────────┼────────────┼────────────────────────────────────┤ │ host/node127/2025-10-31T21:17:33Z │ 64.693 GiB │ catalog.pcat1 home.pxar index.json │ ├───────────────────────────────────┼────────────┼────────────────────────────────────┤ │ host/node127/2025-10-31T21:37:15Z │ 66.867 GiB │ catalog.pcat1 home.pxar index.json │ └───────────────────────────────────┴────────────┴────────────────────────────────────┘ 交互式恢复（下演示将备份中local/下的所有文件恢复到指定目录/home/kvm/local/）[root@node127 ~]# proxmox-backup-client catalog shell --repository mfy-hk-platinum-33@pbs@192.168.2.253:8007:mfy-hk-platinum-33 host/node58/2025-10-31T21:37:15Z home.pxar Starting interactive shell pxar:/ > cd kvm/ pxar:/kvm > ls config.json images kvm43806 kvm43849 local pxar:/kvm > restore /home/kvm/local/ --pattern "local/**" pxar:/kvm > exit 定时备份脚本：#!/bin/bash source /etc/profile # Verify the PBS password is set if [ -z "$PBS_PASSWORD" ]; then echo "错误：PBS_PASSWORD 环境变量未设置！" >&2 exit 1 fi # Run PBS proxmox-backup-client backup home.pxar:/home --repository mfy-hk-platinum-33@pbs@192.168.2.253:8007:mfy-hk-platinum-33 赋予权限chmod 700 /path/pbs_backups.sh Crontab定时任务（每周日凌晨1点运行）0 1 * * 7 PBS_PASSWORD="pbsUserPassword" /path/pbs_backups.sh >> /path/pbs_backups.log 2>&1 更多命令参考官方文档：https://pbs.proxmox.com/docs/command-syntax.html#proxmox-backup-client
Proxmox VE pve-firewall : status update error: iptables_restore_cmdlist : Try `iptables-restore -h'
我们在近期运营中，PVE节点开通的VM客户反馈了一些异常情况：1、PVE VM启用firewall/ macfilter/ ipfilter，ipset rules正确，但没有网络，重装系统/ 防火墙关闭也无法恢复；但在PVE VM网卡停用firewall网络即恢复。2、PVE VM网络正常，但firewall rules不生效。3、pve-firewall出现错误: status update error: iptables_restore_cmdlist: Try `iptables-restore -h' or 'iptables-restore --help' for more information.对于第1个问题，通过简单的排错可以很明确知道是firewall异常，但在PVE GUI发现防火墙处于运行状态。登入SSH运行:iptables-save -c | grep 2804（This is vmid） 找到异常VM的iptables规则:[0:0] -A tap2804i0-OUT -m mac ! --mac-source bc:24:11:b8:97:88 -j DROP tap2804i0中，2804是vmid，i0是第0个网卡；根据这个信息，对比iptables规则与vm的mac确认了mac是不匹配的。由此可以确认firewall已经异常，无法正常更新iptables rules。（tip：pve-firewall并不是独立组件，它最终会生成命令载入iptables。）如果我们简单pve-firewall restart，此时VM网络即恢复正常。看似一切正常，但第2个问题即出现，所有防火墙规则失效。当我再次执行iptables-save -c，返回已经只剩下：~# iptables-save -c # Generated by iptables-save v1.8.9 on Wed Aug 21 19:00:45 2024 *raw :PREROUTING ACCEPT [348735666585:213325671547273] :OUTPUT ACCEPT [4591886294:3380972084311] COMMIT # Completed on Wed Aug 21 19:00:45 2024 # Generated by iptables-save v1.8.9 on Wed Aug 21 19:00:45 2024 *filter :INPUT ACCEPT [6813060:3638198461] :FORWARD ACCEPT [215129855:125354233180] :OUTPUT ACCEPT [5768979:4211817661] COMMIT 这是因为pve-firewall生成的iptables rules存在错误，无法被载入到iptables。执行systemctl status pvefw-logger pve-firewall可以看到类似的错误日志：root@testnode:~# systemctl status pvefw-logger pve-firewall ● pvefw-logger.service - Proxmox VE firewall logger Loaded: loaded (/lib/systemd/system/pvefw-logger.service; enabled; preset: enabled) Active: active (running) since Wed 2024-08-21 00:00:09 HKT; 19h ago Main PID: 1649446 (pvefw-logger) Tasks: 2 (limit: 629145) Memory: 444.0K CPU: 8.328s CGroup: /system.slice/pvefw-logger.service └─1649446 /usr/sbin/pvefw-logger Aug 21 00:00:09 testnode systemd[1]: Starting pvefw-logger.service - Proxmox VE firewall logger... Aug 21 00:00:09 testnode pvefw-logger[1649446]: starting pvefw logger Aug 21 00:00:09 testnode systemd[1]: Started pvefw-logger.service - Proxmox VE firewall logger. ● pve-firewall.service - Proxmox VE firewall Loaded: loaded (/lib/systemd/system/pve-firewall.service; enabled; preset: enabled) Active: active (running) since Wed 2024-08-21 19:07:43 HKT; 16min ago Process: 4058179 ExecStartPre=/usr/bin/update-alternatives --set ebtables /usr/sbin/ebtables-legacy (code=exited, status=0/SUCCESS) Process: 4058181 ExecStartPre=/usr/bin/update-alternatives --set iptables /usr/sbin/iptables-legacy (code=exited, status=0/SUCCESS) Process: 4058182 ExecStartPre=/usr/bin/update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy (code=exited, status=0/SUCCESS) Process: 4058183 ExecStart=/usr/sbin/pve-firewall start (code=exited, status=0/SUCCESS) Process: 4088640 ExecReload=/usr/sbin/pve-firewall restart (code=exited, status=0/SUCCESS) Main PID: 4058255 (pve-firewall) Tasks: 1 (limit: 629145) Memory: 117.2M CPU: 1min 52.846s CGroup: /system.slice/pve-firewall.service └─4058255 pve-firewall Aug 21 19:21:32 testnode pve-firewall[4058255]: status update error: iptables_restore_cmdlist: Try `iptables-restore -h' or 'iptables-restore --help' for more information. Aug 21 19:21:42 testnode pve-firewall[4058255]: status update error: iptables_restore_cmdlist: Try `iptables-restore -h' or 'iptables-restore --help' for more information. Aug 21 19:21:52 testnode pve-firewall[4058255]: status update error: iptables_restore_cmdlist: Try `iptables-restore -h' or 'iptables-restore --help' for more information. 除此外，我们还可以debug启动：pve-firewall stop; pve-firewall start -debug这样我们会知道具体的错误类型（如--dport）、行号，这个信息非常的不清晰，我查询了很多文档好像无法知道这行的内容；只能运行：pve-firewall compile 来检查每个客户的firewal rules看有没错误。为了快速锁定错误内容，既然-debug可以提示错误行，那么它就有完整的iptables rules。我查看了pve-firewall源码（https://github.com/proxmox/pve-firewall/blob/master/src/PVE/Firewall.pm），我们可以直接修改Firewall.pm源码。vi编辑/usr/share/perl5/PVE/Firewall.pm找到这个sub：sub iptables_restore_cmdlist { my ($cmdlist, $table) = @_; $table = 'filter' if !$table; run_command(['iptables-restore', '-T', $table, '-n'], input => $cmdlist, errmsg => "iptables_restore_cmdlist"); } 在$table增加一个行打印所有cmdlist（iptables rules）：sub iptables_restore_cmdlist { my ($cmdlist, $table) = @_; $table = 'filter' if !$table; # 打印 cmdlist warn "Restoring iptables rules: $cmdlist\n"; # 使用 warn 打印到标准错误 run_command(['iptables-restore', '-T', $table, '-n'], input => $cmdlist, errmsg => "iptables_restore_cmdlist"); } 现在我们再次执行：pve-firewall stop; pve-firewall start -debug现在会输出所有它要执行的iptables rules、错误类型、错误行号，根据完整的iptables rules，我们就可以很轻松找到错误行。最终发现是客户增加的 udplite 协议rules导致了iptables错误。接下来就好办了：~# pve-firewall compile | grep udplite -A tap2744i0-IN -p udplite --dport 19885 -j ACCEPT -A tap2744i0-IN -p udplite --dport 20715 -j ACCEPT -A tap2744i0-IN -p udplite --dport 19885 -j ACCEPT -A tap2744i0-IN -p udplite --dport 20715 -j ACCEPT 我们已经找到了错误rules，现在去客户VM2744 Firewall rules删除udplite相关记录。然后 pve-firewall restart。现在不再提示Try `iptables-restore -h' or 'iptables-restore --help' for more information，启动正常，问题解决。记得在/usr/share/perl5/PVE/Firewall.pm注释或删除warn "Restoring iptables rules: $cmdlist\n"。如果希望完全不受其困扰，可以在客户管理平台不再允许添加udplite rules；如果因为某些原因无法进行，也可以编辑 /usr/share/perl5/PVE/Firewall.pm（不建议，因为每次pve-firewall更新后修改可能被覆盖），找到verify_rule方法，添加检查，修改后的完整方法：if ($rule->{proto}) { eval { pve_fw_verify_protocol_spec($rule->{proto}); }; &$add_error('proto', $@) if $@; &$set_ip_version(4) if $rule->{proto} eq 'icmp'; &$set_ip_version(6) if $rule->{proto} eq 'icmpv6'; &$set_ip_version(6) if $rule->{proto} eq 'ipv6-icmp'; $is_icmp = $proto_is_icmp->($rule->{proto}); # 插入对 udplite 协议的检查 if ($rule->{proto} eq 'udplite') { &$add_error('proto', "'udplite' protocol is not supported for firewall rules."); } } 重启pve-firewall后客户再添加udplite rules不会再造成iptables错误，但会返回错误，如下所示：~# systemctl status pvefw-logger pve-firewall ● pvefw-logger.service - Proxmox VE firewall logger Loaded: loaded (/lib/systemd/system/pvefw-logger.service; enabled; preset: enabled) Active: active (running) since Thu 2024-08-22 13:13:18 HKT; 17s ago Process: 4165930 ExecStart=/usr/sbin/pvefw-logger (code=exited, status=0/SUCCESS) Main PID: 4165933 (pvefw-logger) Tasks: 2 (limit: 629145) Memory: 480.0K CPU: 88ms CGroup: /system.slice/pvefw-logger.service └─4165933 /usr/sbin/pvefw-logger Aug 22 13:13:18 testnode systemd[1]: Starting pvefw-logger.service - Proxmox VE firewall logger... Aug 22 13:13:18 testnode pvefw-logger[4165933]: starting pvefw logger Aug 22 13:13:18 testnode systemd[1]: Started pvefw-logger.service - Proxmox VE firewall logger. ● pve-firewall.service - Proxmox VE firewall Loaded: loaded (/lib/systemd/system/pve-firewall.service; enabled; preset: enabled) Active: active (running) since Fri 2024-07-05 06:35:42 HKT; 1 month 17 days ago Process: 3723 ExecStartPre=/usr/bin/update-alternatives --set ebtables /usr/sbin/ebtables-legacy (code=exited, status=0/SUCCESS) Process: 3725 ExecStartPre=/usr/bin/update-alternatives --set iptables /usr/sbin/iptables-legacy (code=exited, status=0/SUCCESS) Process: 3727 ExecStartPre=/usr/bin/update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy (code=exited, status=0/SUCCESS) Process: 3729 ExecStart=/usr/sbin/pve-firewall start (code=exited, status=0/SUCCESS) Process: 4165880 ExecReload=/usr/sbin/pve-firewall restart (code=exited, status=0/SUCCESS) Main PID: 3744 (pve-firewall) Tasks: 1 (limit: 629145) Memory: 123.0M CPU: 4d 18h 11min 46.192s CGroup: /system.slice/pve-firewall.service └─3744 pve-firewall Aug 22 13:13:15 testnode pve-firewall[3744]: received signal HUP Aug 22 13:13:15 testnode pve-firewall[3744]: server shutdown (restart) Aug 22 13:13:15 testnode systemd[1]: Reloaded pve-firewall.service - Proxmox VE firewall. Aug 22 13:13:15 testnode pve-firewall[3744]: restarting server Aug 22 13:13:16 testnode pve-firewall[3744]: /etc/pve/firewall/4076.fw (line 37) - errors in rule parameters: IN ACCEPT -i net4 -p udpl> Aug 22 13:13:16 testnode pve-firewall[3744]: proto: 'udplite' protocol is not supported for firewall rules.
巡检中发现PVE的zfs-raidz2坏了一个盘，查询显示：~# zpool status rpool pool: rpool state: DEGRADED status: One or more devices has been removed by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Online the device using zpool online' or replace the device with 'zpool replace'. scan: scrub repaired 0B in 00:43:00 with 0 errors on Sun May 12 01:07:02 2024 config: NAME STATE READ WRITE CKSUM rpool DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 ata-INTEL_SSDSC2KB019TZ_PHYI334000GD1P9DGN-part3 ONLINE 0 0 0 ata-INTEL_SSDSC2KB019TZ_PHYI334000NS1P9DGN-part3 ONLINE 0 0 0 ata-INTEL_SSDSC2KB019TZ_PHYI334000Q21P9DGN-part3 ONLINE 0 0 0 ata-INTEL_SSDSC2KB019TZ_PHYI334000Y91P9DGN-part3 ONLINE 0 0 0 ata-INTEL_SSDSC2KB019TZ_PHYI334000GE1P9DGN-part3 REMOVED 0 0 0 请留意，如果你将ZFS作为数据盘则本文适用；如果作为系统盘（系统安装在ZFS）请通过替换分区方式更换故障盘，避免潜在风险：https://www.cnweed.com/archives/4420/故障盘的状态已经是 REMOVED。我现在把坏的拔出来，插入了一个新的盘，再次执行 zpool status rpool 回显信息是不变的。但ls -la /dev/disk/by-id 可以看到没有配置的 sdd 盘：~# ls -la /dev/disk/by-id total 0 drwxr-xr-x 2 root root 720 Jun 11 00:30 . drwxr-xr-x 8 root root 160 Apr 29 11:33 .. lrwxrwxrwx 1 root root 9 Apr 29 11:33 ata-INTEL_SSDSC2KB019TZ_PHYI334000GD1P9DGN -> ../../sda lrwxrwxrwx 1 root root 10 Apr 29 11:33 ata-INTEL_SSDSC2KB019TZ_PHYI334000GD1P9DGN-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 Apr 29 11:33 ata-INTEL_SSDSC2KB019TZ_PHYI334000GD1P9DGN-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 Apr 29 11:33 ata-INTEL_SSDSC2KB019TZ_PHYI334000GD1P9DGN-part3 -> ../../sda3 lrwxrwxrwx 1 root root 9 May 12 00:38 ata-INTEL_SSDSC2KB019TZ_PHYI334000NS1P9DGN -> ../../sdb lrwxrwxrwx 1 root root 10 Apr 29 11:33 ata-INTEL_SSDSC2KB019TZ_PHYI334000NS1P9DGN-part1 -> ../../sdb1 lrwxrwxrwx 1 root root 10 Apr 29 11:33 ata-INTEL_SSDSC2KB019TZ_PHYI334000NS1P9DGN-part2 -> ../../sdb2 lrwxrwxrwx 1 root root 10 Apr 29 11:33 ata-INTEL_SSDSC2KB019TZ_PHYI334000NS1P9DGN-part3 -> ../../sdb3 lrwxrwxrwx 1 root root 9 Apr 29 11:33 ata-INTEL_SSDSC2KB019TZ_PHYI334000Q21P9DGN -> ../../sdc lrwxrwxrwx 1 root root 10 Apr 29 11:33 ata-INTEL_SSDSC2KB019TZ_PHYI334000Q21P9DGN-part1 -> ../../sdc1 lrwxrwxrwx 1 root root 10 Apr 29 11:33 ata-INTEL_SSDSC2KB019TZ_PHYI334000Q21P9DGN-part2 -> ../../sdc2 lrwxrwxrwx 1 root root 10 Apr 29 11:33 ata-INTEL_SSDSC2KB019TZ_PHYI334000Q21P9DGN-part3 -> ../../sdc3 lrwxrwxrwx 1 root root 9 May 25 17:43 ata-INTEL_SSDSC2KB019TZ_PHYI334000Y91P9DGN -> ../../sde lrwxrwxrwx 1 root root 10 Apr 29 11:33 ata-INTEL_SSDSC2KB019TZ_PHYI334000Y91P9DGN-part1 -> ../../sde1 lrwxrwxrwx 1 root root 10 Apr 29 11:33 ata-INTEL_SSDSC2KB019TZ_PHYI334000Y91P9DGN-part2 -> ../../sde2 lrwxrwxrwx 1 root root 10 Apr 29 11:33 ata-INTEL_SSDSC2KB019TZ_PHYI334000Y91P9DGN-part3 -> ../../sde3 lrwxrwxrwx 1 root root 9 Jun 11 00:30 ata-INTEL_SSDSC2KB019TZ_PHYI33840KVB1P9DGN -> ../../sdd lrwxrwxrwx 1 root root 9 Apr 29 11:33 wwn-0x55cd2e415651c850 -> ../../sda lrwxrwxrwx 1 root root 10 Apr 29 11:33 wwn-0x55cd2e415651c850-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 Apr 29 11:33 wwn-0x55cd2e415651c850-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 Apr 29 11:33 wwn-0x55cd2e415651c850-part3 -> ../../sda3 lrwxrwxrwx 1 root root 9 May 12 00:38 wwn-0x55cd2e415651c929 -> ../../sdb lrwxrwxrwx 1 root root 10 Apr 29 11:33 wwn-0x55cd2e415651c929-part1 -> ../../sdb1 lrwxrwxrwx 1 root root 10 Apr 29 11:33 wwn-0x55cd2e415651c929-part2 -> ../../sdb2 lrwxrwxrwx 1 root root 10 Apr 29 11:33 wwn-0x55cd2e415651c929-part3 -> ../../sdb3 lrwxrwxrwx 1 root root 9 Apr 29 11:33 wwn-0x55cd2e415651c955 -> ../../sdc lrwxrwxrwx 1 root root 10 Apr 29 11:33 wwn-0x55cd2e415651c955-part1 -> ../../sdc1 lrwxrwxrwx 1 root root 10 Apr 29 11:33 wwn-0x55cd2e415651c955-part2 -> ../../sdc2 lrwxrwxrwx 1 root root 10 Apr 29 11:33 wwn-0x55cd2e415651c955-part3 -> ../../sdc3 lrwxrwxrwx 1 root root 9 May 25 17:43 wwn-0x55cd2e415651ca6c -> ../../sde lrwxrwxrwx 1 root root 10 Apr 29 11:33 wwn-0x55cd2e415651ca6c-part1 -> ../../sde1 lrwxrwxrwx 1 root root 10 Apr 29 11:33 wwn-0x55cd2e415651ca6c-part2 -> ../../sde2 lrwxrwxrwx 1 root root 10 Apr 29 11:33 wwn-0x55cd2e415651ca6c-part3 -> ../../sde3 lrwxrwxrwx 1 root root 9 Jun 11 00:30 wwn-0x55cd2e4156563eaf -> ../../sdd 根据以上信息让我们确认sdd的by-id，执行：~# ls -l /dev/disk/by-id/wwn-0x55cd2e4156563eaf lrwxrwxrwx 1 root root 9 Jun 11 00:30 /dev/disk/by-id/wwn-0x55cd2e4156563eaf -> ../../sdd 接下来我们登入PVE WEB管理界面，点击节点磁盘，可以看到未配置的sdd磁盘，然后点击“使用GPT初始化磁盘”。接着我们继续在CLI替换故障盘：~# zpool replace rpool ata-INTEL_SSDSC2KB019TZ_PHYI334000GE1P9DGN-part3 /dev/disk/by-id/wwn-0x55cd2e4156563eaf 其中ata-INTEL_SSDSC2KB019TZ_PHYI334000GE1P9DGN-part3是旧盘，/dev/disk/by-id/wwn-0x55cd2e4156563eaf是新盘。我们再次查询zpool状态已经开始替换。~# zpool status rpool pool: rpool state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Tue Jun 11 00:46:36 2024 197G / 3.45T scanned at 12.3G/s, 0B / 3.44T issued 0B resilvered, 0.00% done, no estimated completion time config: NAME STATE READ WRITE CKSUM rpool DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 ata-INTEL_SSDSC2KB019TZ_PHYI334000GD1P9DGN-part3 ONLINE 0 0 0 ata-INTEL_SSDSC2KB019TZ_PHYI334000NS1P9DGN-part3 ONLINE 0 0 0 ata-INTEL_SSDSC2KB019TZ_PHYI334000Q21P9DGN-part3 ONLINE 0 0 0 ata-INTEL_SSDSC2KB019TZ_PHYI334000Y91P9DGN-part3 ONLINE 0 0 0 replacing-4 DEGRADED 0 0 0 ata-INTEL_SSDSC2KB019TZ_PHYI334000GE1P9DGN-part3 REMOVED 0 0 0 wwn-0x55cd2e4156563eaf ONLINE 0 0 0 可以看到整个过程还是很简单、快捷的。
Proxmox VE的ZFS-RAID-Z2与ZFS-RAID-Z3，以及硬卡RAID10差异
Proxmox+VE用于公有云、VPS的可靠性得到了很多同行长期的验证认可，整个生态环境也满足我的需求，所需功能也基本都能满足，所以我开始用PVE了。我考虑用本地存储，如果是用智简魔方的公有云平台，一般情况只有硬卡RAID10是最好方案；但PVE出现了多个方案，我挑了几个，但很犹豫哪个最合适我：ZFS-RAID-Z2、ZFS-RAID-Z3、ZFS-RAID10，以及硬卡RAID10。首先硬卡RAID10就被淘汰了，因为我计划使用NVMe+U.2的盘，常见的硬卡无法对U.2做列阵。列阵原理：ZFS-RAID-Z2、ZFS-RAID-Z3以下官方的解释：A raidz group can have single, double, or triple parity, meaning that the raidzgroup can sustain one, two, or three failures, respectively, without losing anydata.The raidz1 vdev type specifies a single-parity raidz group; the raidz2 vdev type specifies a double-parity raidz group; and the raidz3 vdev typespecifies a triple-parity raidz group.The raidz vdev type is an alias for raidz1.通俗理解就是如果我有6个NVMe+U.2盘组建ZFS-RAID-Z2，则最大可以故障任意2个盘；ZFS-RAID-Z3+则可以最大故障任意3个盘；而可用容量则是6个盘-2个盘（Z2），或3个盘（Z3）。ZFS-RAID-Z10与硬卡RAID10是相同的原理。多个RAID1组建为RAID0，则为RAID10，只能大于等于4的偶数盘位数量组建，可用容量为一半。性能区别：6个盘的情况下，我测试了ZFS-RAID-Z10、ZFS-RAID-Z2性能差别不大，但可用容量Z2多出了1个盘。安全性：有关安全性的讨论很激烈，仁者见仁。RAID10我用了很多年，是非常可靠的，但并不意味着万无一失，正如上面提到的，其是多个RAID1(镜像)组建的RAID0。RAID10可以在故障多个盘的时候保持数据可用，可如果坏的2个盘是同一组镜像内的，RAID10列阵将损坏；即便你有12、24个盘的RAID10，同一组镜像内的盘坏了，也会面临同样的结局。我已经遇到了多次……但如果选用好的硬盘可以很大程度上减少这个情况，例如Intel+S4520+SSD，目前还没一例这个情况。ZFS-RAID-Z2和ZFS-RAID-Z3则面临着另外一个问题，当一个盘发生故障后，剩余磁盘的IO压力将增加，因此如果同步恢复速度慢（或没有即时去更换故障盘），剩下磁盘接连故障的可能性就会增加；而RAID10同一组镜像内故障了一个盘，另一个盘不会因此而IO压力增加，同一组镜像内接连故障的可能性相对较低。不同安全级别的硬盘磨损（重写）问题RAID10几乎没有写入放大问题，ZFS-RAIDZ-2/3则存在写入放大的问题，即写入1T数据实际可能会对硬盘写入2T，这会造成硬盘的寿命比预期更短，我们需要实际测试观察写入放大的程度，决定是否使用。如果是SATA+SSD用硬卡还是ZFS-RAID10呢？虽然可能会占用一些内存、CPU性能，但我个人仍然会更倾向ZFS-RAID10，因为其具备在线扩展的能力，初始硬盘数量并不需要特别大，如果不足时候可以后面再补硬盘扩大列阵，这些都可以通过PVE在线完成。虽然硬卡RAID10也可以后期扩展，但其操作相对复杂还有数据损坏的风险，且需要停机重启。结论我最后决定6盘的话使用+ZFS-RAIDZ-2，7-8盘位使用+ZFS-RAIDZ-3；我们使用了企业级的Intel+NVMe+U.2或者Intel+S4520，以及有人员可以即时响应更换，可以忽略写入放大的问题；但长期来看ZFS-RAID10仍然是安全、性能的最佳平衡。还有个情况就是ZFS-RAIDZ-2/3官方建议集群盘数量要小于等于16个，但我认为应该更小；大于10个盘应该选择RAID10。另外日常维护中，可以偶尔拔掉一两个盘和不同批次的硬盘互换、或者在组建之前逐一写入一些测试数据，让列阵中的盘写入寿命不相同，避免同时故障的可能性。本文参考：https://www.reddit.com/r/homelab/comments/15fzf44/zfs_raid10_or_raidz2_which_one_to_chose_for_6x1tb/https://www.reddit.com/r/Proxmox/comments/uukjgm/copy_template_between_different_clusters/https://www.liujason.com/article/679.htmlhttps://community.spiceworks.com/topic/549079-raid-10-vs-raid-70-striped-raidz3-vdevshttps://forum.proxmox.com/threads/zfs-best-practices-36x-6tb-hdd-enterprise-nvme.135079/
今天遇到一个很狗血的问题…有1台华为S6720-32X-LI-32S-AC万兆交换机，设备只跑了100Mbps左右，但交换机的统计却显示跑了7-8%，也就是800Mbps左右。输入 dis mac-address 检查发现没有学习到任何mac地址，判断可能是流量泛洪。MAC地址转发时，采用相同的HASH算法去查找对应的VLAN+MAC表项，如果无法找到对应的表项，则产生流量泛洪。尝试绑定mac-端口-vlan，后流量泛洪消失。mac-address static 0024-XXXX-424f GigabitEthernet 1/7/0/44 vlan 199 参考引用：https://support.huawei.com/enterprise/zh/knowledge/EKB1000601298检查了交换机里的配置，没有mac、arp相关的额外设置，也就是都是默认使能、最宽松的设置。然后检查了发生的时间，找了当日的其他配置，trunk上联interface Vlanif多加了一个/24。超出交换机限制？应该不至于，三十多个/24而已。S5720都不会出现这个情况，一脸懵逼。。。
Python对mdadm组建的RAID软列阵定时检测异常自动短信/电话通知
基于mysubmail接口发送短信、电话语音通知。创建脚本后可以通过crontab -e每小时或半小时运行一次，例如：/30 * python /etc/weed/checkRAID.py可以在raid出现异常状态时快速获知及时处理，避免故障扩大。# -*- coding: utf-8 -*- #!/usr/bin/python import os import requests node = '香港vps-nvmenode-1' error = 0 def get_status(value): try: status = value.split(": ") return status[1].strip() except: return False def send_warning(): global node # 语音通知 voice_url = 'http://api.mysubmail.com/voice/send.json' voice_params = { 'appid': '*****', 'to': '13200000000', 'content': '紧急事态：'+node+'硬盘状态异常，请立即检查', 'signature': '**************' } #voice_res = requests.post(voice_url, data=voice_params) # print voice_res.text # 短信通知 message_url = 'http://api.mysubmail.com/message/send.json' message_params = { 'appid': '*****', 'to': '13200000000', 'content': '【XXX】紧急事态：'+node+'硬盘状态异常，请立即检查', 'signature': '**************' } message_res = requests.post(message_url, data=message_params) # print(message_res.text) # 检查RAID状态，注意/dev/md10是变动参数，自行fdisk -l查看你的软列阵磁盘名称，如raid1则为/dev/md1 raidinfos = os.popen('mdadm -D /dev/md10').readlines() for raidinfo in raidinfos: raidinfo = raidinfo.strip('\n') print(raidinfo) if "State : " in raidinfo: status = get_status(raidinfo) if status != 'active' and status != 'active, checking': error = 1 if "Failed Devices : " in raidinfo: status = get_status(raidinfo) if status != '0': error = 1 if "Active Devices : " in raidinfo: status = get_status(raidinfo) if status != '2': error = 1 # 发送通知 if error == 1: send_warning()
DELL630 H730P RAID10不同Strip Size设置的IOPS差异
如果RAID0用作CEPH OSD，则建议禁用磁盘级的缓存，也就是磁盘标签上写的那个256MB缓存：# /opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp -DisDskCache -Immediate -Lall -aAll Set Disk Cache Policy to Disabled on Adapter 0, VD 0 (target id: 0) success Set Disk Cache Policy to Disabled on Adapter 0, VD 1 (target id: 1) success Set Disk Cache Policy to Disabled on Adapter 0, VD 2 (target id: 2) success Set Disk Cache Policy to Disabled on Adapter 0, VD 4 (target id: 4) success Set Disk Cache Policy to Disabled on Adapter 0, VD 5 (target id: 5) success Set Disk Cache Policy to Disabled on Adapter 0, VD 7 (target id: 7) success Set Disk Cache Policy to Disabled on Adapter 0, VD 8 (target id: 8) success 如果要启用target id 0 （系统盘）缓存则运行：/opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp -EnDskCache -Immediate -L0 -a0 查看结果：# /opt/MegaRAID/MegaCli/MegaCli64 -LDGetProp -DskCache -Lall -a0 Adapter 0-VD 0(target id: 0): Disk Write Cache : Disabled Adapter 0-VD 1(target id: 1): Disk Write Cache : Disabled Adapter 0-VD 2(target id: 2): Disk Write Cache : Disabled Adapter 0-VD 4(target id: 4): Disk Write Cache : Disabled Adapter 0-VD 5(target id: 5): Disk Write Cache : Disabled Adapter 0-VD 7(target id: 7): Disk Write Cache : Disabled Adapter 0-VD 8(target id: 8): Disk Write Cache : Disabled 如果计划对RAID0使用RAID的缓存，则务必确保机器电池有电要定期检查，否则突然断电可能造成RAID缓存中的数据丢失，这种情况很少，但还是要留意；最好是不启用RAID的缓存，也就是直写硬盘，或者writethrough模式。如果还没买设备，或者后面还有计划增加设备，最好买有直通的raid卡！如果因为一些原因没办法用直通卡，只能raid0来创建osd则建议禁用磁盘缓存、禁用raid卡缓存或使用writethrough模式。学习参考自：https://bean-li.github.io/disk-cache-policy/https://serverfault.com/questions/505275/megaraid-changing-cache-policy-onlinehttps://www.dell.com/community/PowerEdge-HDD-SCSI-RAID/How-do-I-view-change-physical-disk-cache-policy-on-PowerEDGE/td-p/2581739
我的CEPH OSD因为列阵卡不支持直通是基于RAID0的，最近ceph性能不足，排查过程iostat观察发现有一些盘r_await或w_await持续1000多，这就需要换盘了。ceph存储一份数据如果是默认3副本，那么就会3份副本写完才会完成写入，如果有1个osd延迟很高，就会影响整体写入速度。首先可以运行 ceph-volume lvm list 查看osd对应的盘符，如果是bcache，则需要再运行lsblk查看在哪个盘符下。然后查看是哪个盘符，以及盘位，进行更换重建。Python基于MegaCli获取raid卡状态的代码：https://raw.githubusercontent.com/eLvErDe/hwraid/master/wrapper-scripts/megaclisas-status这个大佬的python脚本可以快速帮忙定位，运行完后再执行下：lsscsi 查看盘符在系统中的编号就可以定位到盘了，记得MegaCli得先安装。代码贴上：#!/usr/bin/python # $Id: megaclisas-status,v 1.78 2018/10/01 03:52:57 root Exp root $ # # Written by Adam Cecile <gandalf@NOSPAM.le-vert.net> # Modified by Vincent S. Cojot <vincent@NOSPAM.cojot.name> # import os import re import sys import pdb import inspect import argparse os.system('lsscsi') if sys.platform == "win32": import ctypes def_megaclipath = "/opt/MegaRAID/MegaCli/MegaCli64" # Non-Nagios Mode defaults nagiosmode = False nagiosoutput = "" nagiosgoodarray = 0 nagiosbadarray = 0 nagiosgooddisk = 0 nagiosbaddisk = 0 # Sane defaults printarray = True printcontroller = True debugmode = False notempmode = False totaldrivenumber = 0 totalconfdrivenumber = 0 totalunconfdrivenumber = 0 # Hardcode a max of 16 HBA and 128 LDs for now. LDTable must be initialized to accept populating list of LD's into each ctlr's list. MaxNumHBA = 16 MaxNumLD = 128 LDTable = [[] * MaxNumHBA for i in range(MaxNumLD)] NestedLDTable = [[False for i in range(MaxNumLD)] for j in range(MaxNumHBA)] # Outputs is a 'dict' of all MegaCLI outputs so we can re-use them during loops.. Outputs = {} ConfDisks = {} NagiosBadDisks = {} NagiosGoodDisks = {} # We need root access to query if __name__ == "__main__": # deal with command line options parser = argparse.ArgumentParser() parser.add_argument("--nagios", help="enable nagios support", action="store_true") parser.add_argument("--debug", help="enable debugging output", action="store_true") parser.add_argument("--notemp", help="disable temperature reporting", action="store_true") args = parser.parse_args() nagiosmode = args.nagios debugmode = args.debug notempmode = args.notemp try: root_or_admin = os.geteuid() == 0 except AttributeError: root_or_admin = ctypes.windll.shell32.IsUserAnAdmin() != 0 if not root_or_admin: print("# This script requires Administrator privileges") sys.exit(5) # Functions def dbgprint(msg): if debugmode: sys.stderr.write(str("# DEBUG (" + str(inspect.currentframe().f_back.f_lineno) + ") : " + msg + "\n")) def is_exe(fpath): return os.path.isfile(fpath) and os.access(fpath, os.X_OK) def which(program): import os fpath, fname = os.path.split(program) if fpath: if is_exe(program): return program else: # Add some defaults os.environ["PATH"] += os.pathsep + "/opt/MegaRAID/MegaCli" os.environ["PATH"] += os.pathsep + "/ms/dist/hwmgmt/bin" os.environ["PATH"] += os.pathsep + "/opt/MegaRAID/perccli" os.environ["PATH"] += os.pathsep + "/opt/MegaRAID/storcli" os.environ["PATH"] += os.pathsep + "/opt/lsi/storcli" os.environ["PATH"] += os.pathsep + os.path.dirname(os.path.realpath(sys.argv[0])) for path in os.environ["PATH"].split(os.pathsep): dbgprint("Looking in PATH " + str(path)) path = path.strip('"') exe_file = os.path.join(path, program) if is_exe(exe_file): dbgprint('Found "' + program + '" at ' + exe_file) return exe_file return None # Find MegaCli for megabin in "MegaCli64", "MegaCli", "megacli", "MegaCli.exe", "perccli64", "perccli", "storcli64", "storcli": dbgprint("Looking for " + str(megabin) + " in PATH...") megaclipath = which(megabin) if megaclipath != None: dbgprint("Will use this executable: " + str(megaclipath)) break # Check binary exists (and +x), if not print an error message if megaclipath != None: if os.path.exists(megaclipath) and os.access(megaclipath, os.X_OK): pass else: if nagiosmode: print("UNKNOWN - Cannot find " + megaclipath) else: print("Cannot find " + megaclipath + "in your PATH. Please install it.") sys.exit(3) else: print('Cannot find "MegaCli{64,}", "megacli{64,}", "perccli{64,}" or "storcli{64,}" in your PATH. Please install one of them.') sys.exit(3) #### pdb.set_trace() def returnWdthFromArrayCol(glarray, idx): maxwdth = 0 for glrow in glarray: if len(glrow[idx]) > maxwdth: maxwdth = len(glrow[idx]) return maxwdth # Get and cache command output def getOutput(cmd): lines = [] if cmd in Outputs: dbgprint("Got Cached value: " + str(cmd)) lines = Outputs[cmd] else: dbgprint("Not a Cached value: " + str(cmd)) output = os.popen(cmd) for line in output: if not re.match(r"^$", line.strip()): lines.append(line.strip()) Outputs[cmd] = lines return lines # Get and cache disks, make sure we don't count the same disk twice def AddDisk(mytable, disk): lines = [] if disk in mytable: dbgprint("Disk: " + str(disk) + " Already present in Disk Table") return False else: dbgprint("Confed " + str(nagiosgooddisk) + "/" + str(nagiosbaddisk) + "Disk: " + str(disk) + " Not already present in Disk Table, adding") mytable[disk] = True return True def returnControllerNumber(output): for line in output: if re.match(r"^Controller Count.*$", line.strip()): return int(line.split(":")[1].strip().strip(".")) def returnTotalDriveNumber(output): for line in output: if re.match(r"Number of Physical Drives on Adapter.*$", line.strip()): return int(line.split(":")[1].strip()) def returnRebuildProgress(output): percent = 0 tmpstr = "" for line in output: if re.match(r"^Rebuild Progress on Device at Enclosure.*, Slot .* Completed ", line.strip()): tmpstr = line.split("Completed")[1].strip() percent = int(tmpstr.split("%")[0].strip()) return percent def returnConfDriveNumber(controllerid, output): # Count the configured drives confdrives = 0 enclid = "N/A" slotid = "N/A" for line in output: if re.match(r"Enclosure Device ID: .*$", line.strip()): # We match here early in the analysis so reset the vars if this is a new disk we're reading.. enclid = line.split(":")[1].strip() elif re.match(r"Slot Number: .*$", line.strip()): slotid = line.split(":")[1].strip() if AddDisk(ConfDisks, str(controllerid) + enclid + slotid): confdrives += 1 return int(confdrives) def returnUnConfDriveNumber(output): # Count the un-configured/Hotspare drives unconfdrives = 0 for line in output: if re.match(r"^Firmware state: Unconfigured.*$", line.strip()): unconfdrives += 1 elif re.match(r"^Firmware state: Hotspare.*$", line.strip()): unconfdrives += 1 return int(unconfdrives) def returnControllerModel(output): for line in output: if re.match(r"^Product Name.*$", line.strip()): return line.split(":")[1].strip() def returnMemorySize(output): for line in output: if re.match(r"^Memory Size.*$", line.strip()): return line.split(":")[1].strip() def returnFirmwareVersion(output): for line in output: if re.match(r"^FW Package Build.*$", line.strip()): return line.split(":")[1].strip() def returnROCTemp(output): ROCtemp = "" tmpstr = "" if notempmode: return str("N/A") else: for line in output: if re.match(r"^ROC temperature :.*$", line.strip()): tmpstr = line.split(":")[1].strip() ROCtemp = re.sub(" +.*$", "", tmpstr) if ROCtemp != "": return str(str(ROCtemp) + "C") else: return str("N/A") def returnBBUPresence(output): BBU = "" tmpstr = "" for line in output: if re.match(r"^BBU +:.*$", line.strip()): tmpstr = line.split(":")[1].strip() BBU = re.sub(" +.*$", "", tmpstr) break if BBU != "": return str(BBU) else: return str("N/A") def returnBBUStatus(output): BBUStatus = "" tmpstr = "" for line in output: if re.match(r"^ *Battery Replacement required +:.*$", line.strip()): tmpstr = line.split(":")[1].strip() BBUStatus = re.sub(" +.*$", "", tmpstr) break if BBUStatus == "Yes": return str("REPL") else: return str("Good") def returnArrayNumber(output): i = 0 for line in output: if re.match(r"^(CacheCade )?Virtual Drive:.*$", line.strip()): i += 1 return i def returnHBAPCIInfo(output): busprefix = "0000" busid = "" devid = "" functionid = "" pcipath = "" for line in output: if re.match(r"^Bus Number.*:.*$", line.strip()): busid = str(line.strip().split(":")[1].strip()).zfill(2) if re.match(r"^Device Number.*:.*$", line.strip()): devid = str(line.strip().split(":")[1].strip()).zfill(2) if re.match(r"^Function Number.*:.*$", line.strip()): functionid = str(line.strip().split(":")[1].strip()).zfill(1) if busid: pcipath = str(busprefix + ":" + busid + ":" + devid + "." + functionid) dbgprint("Array PCI path : " + pcipath) return str(pcipath) else: return None def returnHBAInfo(table, output, controllerid): controllermodel = "Unknown" controllerram = "Unknown" controllerrev = "Unknown" controllertemp = "" controllermodel = returnControllerModel(output) controllerram = returnMemorySize(output) controllerrev = returnFirmwareVersion(output) controllertemp = returnROCTemp(output) controllerbbu = returnBBUPresence(output) if controllerbbu == "Present": cmd = "%s -AdpBbuCmd -GetBbuStatus -a%d -NoLog" % (megaclipath, controllerid) output = getOutput(cmd) controllerbbu = returnBBUStatus(output) if controllermodel != "Unknown": table.append(["c" + str(controllerid), controllermodel, controllerram, str(controllertemp), str(controllerbbu), str("FW: " + controllerrev)]) def returnArrayInfo(output, controllerid, arrayid, arrayindex): id = "c" + str(controllerid) + "u" + str(arrayid) operationlinennumber = False linenumber = 0 targetid = "" raidtype = "" raidlvl = "" size = "" state = "N/A" strpsz = "" dskcache = "N/A" properties = "" spandepth = 0 diskperspan = 0 cachecade_info = "None" for line in output: if re.match(r"^(CacheCade )?Virtual Drive:.*(Target Id: [0-9]+).*$", line.strip()): # Extract the SCSI Target ID targetid = line.strip().split(":")[2].split(")")[0].strip() elif re.match(r"^RAID Level.*?:.*$", line.strip()): # Extract the primary raid type, decide on X0 RAID level later when we hit Span Depth raidlvl = int(line.strip().split(":")[1].split(",")[0].split("-")[1].strip()) elif re.match(r"^Size.*?:.*$", line.strip()): # Size reported in MB if re.match(r"^.*MB$", line.strip().split(":")[1]): size = line.strip().split(":")[1].strip("MB").strip() if float(size) > 1000: size = str(int(round((float(size) / 1000)))) + "G" else: size = str(int(round(float(size)))) + "M" # Size reported in TB elif re.match(r"^.*TB$", line.strip().split(":")[1]): size = line.strip().split(":")[1].strip("TB").strip() size = str(int(round((float(size) * 1000)))) + "G" # Size reported in GB (default) else: size = line.strip().split(":")[1].strip("GB").strip() size = str(int(round((float(size))))) + "G" elif re.match(r"^Span Depth.*?:.*$", line.strip()): # If Span Depth is greater than 1 chances are we have a RAID 10, 50 or 60 spandepth = line.strip().split(":")[1].strip() elif re.match(r"^State.*?:.*$", line.strip()): state = line.strip().split(":")[1].strip() elif re.match(r"^Strip Size.*?:.*$", line.strip()): strpsz = line.strip().split(":")[1].strip() elif re.match(r"^Number Of Drives per span.*:.*$", line.strip()): diskperspan = int(line.strip().split(":")[1].strip()) elif re.match(r"^Current Cache Policy.*?:.*$", line.strip()): props = line.strip().split(":")[1].strip() if re.search("ReadAdaptive", props): properties += "ADRA" if re.search("ReadAhead", props): properties += "RA" if re.match("ReadAheadNone", props): properties += "NORA" if re.search("WriteBack", props): properties += ",WB" if re.match("WriteThrough", props): properties += ",WT" elif re.match(r"^Disk Cache Policy.*?:.*$", line.strip()): props = line.strip().split(":")[1].strip() if re.search("Disabled", props): dskcache = "Disabled" if re.search("Disk.s Default", props): dskcache = "Default" if re.search("Enabled", props): dskcache = "Enabled" elif re.match(r"^Ongoing Progresses.*?:.*$", line.strip()): operationlinennumber = linenumber elif re.match(r"Cache Cade Type\s*:.*$", line): cachecade_info = "Type : " + line.strip().split(":")[1].strip() elif re.match(r"^Target Id of the Associated LDs\s*:.*$", line): associated = [] for array in line.split(":")[1].strip().split(","): if array.isdigit(): associated.append("c%du%d" % (controllerid, int(array))) if len(associated) >= 1: cachecade_info = "Associated : %s" % (", ".join(associated)) linenumber += 1 # If there was an ongoing operation, find the relevant line in the previous output if operationlinennumber: inprogress = str(output[operationlinennumber + 1]) # some ugly output fix.. str1 = inprogress.split(":")[0].strip() str2 = inprogress.split(":")[1].strip() inprogress = str1 + " : " + str2 else: inprogress = "None" # Compute the RAID level NestedLDTable[int(controllerid)][int(arrayindex)] = False if raidlvl == "": raidtype = str("N/A") else: if int(spandepth) >= 2: raidtype = str("RAID-" + str(raidlvl) + "0") NestedLDTable[controllerid][int(arrayindex)] = True else: if raidlvl == 1: if diskperspan > 2: raidtype = str("RAID-10") NestedLDTable[controllerid][int(arrayindex)] = True else: raidtype = str("RAID-" + str(raidlvl)) else: raidtype = str("RAID-" + str(raidlvl)) dbgprint("RAID Level: " + str(raidlvl) + " Span Depth: " + str(spandepth) + " Disk Per Span: " + str(diskperspan) + " Raid Type: " + str(raidtype)) return [id, raidtype, size, strpsz, properties, dskcache, state, targetid, cachecade_info, inprogress] def returnDiskInfo(output, controllerid): arrayid = False arrayindex = -1 sarrayid = "Unknown" diskid = False oldenclid = False enclid = False spanid = False slotid = False lsidid = "Unknown" table = [] fstate = "Offline" substate = "Unknown" model = "Unknown" speed = "Unknown" dsize = "Unknown" temp = "Unk0C" percent = 0 for line in output: if re.match(r"^Span: [0-9]+ - Number of PDs:", line.strip()): spanid = line.split(":")[1].strip() spanid = re.sub(" - Number of PDs.*", "", spanid) elif re.match(r"Enclosure Device ID: .*$", line.strip()): # We match here early in the analysis so reset the vars if this is a new disk we're reading.. oldenclid = enclid enclid = line.split(":")[1].strip().replace("N/A", "") if oldenclid != False: fstate = "Offline" model = "Unknown" speed = "Unknown" temp = "Unk0C" slotid = False lsidid = "Unknown" elif re.match(r"^Coerced Size: ", line.strip()): dsize = line.split(":")[1].strip() dsize = re.sub(" \[.*\.*$", "", dsize) dsize = re.sub("[0-9][0-9] GB", " Gb", dsize) elif re.match(r"^(CacheCade )?Virtual (Disk|Drive): [0-9]+.*$", line.strip()): arrayindex += 1 arrayid = line.split("(")[0].split(":")[1].strip() elif re.match(r"^Drive.s posi*tion: DiskGroup: [0-9]+,.*$", line.strip()): notarrayid = line.split(",")[1].split(":")[1].strip() elif re.match(r"PD: [0-9]+ Information.*$", line.strip()): diskid = line.split()[1].strip() elif re.match(r"^Device Id: .*$", line.strip()): lsidid = line.split(":")[1].strip() elif re.match(r"Slot Number: .*$", line.strip()): slotid = line.split(":")[1].strip() elif re.match(r"Firmware state: .*$", line.strip()): fstate = line.split(":")[1].strip() subfstate = re.sub("\(.*", "", fstate) dbgprint("Firmware State: " + str(fstate) + " " + str(subfstate)) elif re.match(r"Inquiry Data: .*$", line.strip()): model = line.split(":")[1].strip() model = re.sub(" +", " ", model) # re-define our "sub-code" # our seagate drives have an ID string of # 'Z1E19S2QST2000DM001-1CH164 CC43' # or # '6XW02738ST32000542AS CC32' m = re.match(r"(\w{8})(ST\w+)(?:-(\w{6}))?(?:\s+(\w+))", model) if m: if m.group(3): model = "{0}-{1} {2} {3}".format(m.group(2), m.group(3), m.group(4), m.group(1)) else: model = "{0} {1:>10} {2}".format(m.group(2), m.group(4), m.group(1)) continue # Sub code manuf = re.sub(" .*", "", model) dtype = re.sub(manuf + " ", "", model) dtype = re.sub(" .*", "", dtype) hwserial = re.sub(".*" + dtype + " *", "", model) elif re.match(r"^Media Type: .*$", line.strip()): mtype = line.split(":")[1].strip() if mtype == "Hard Disk Device": mtype = "HDD" else: if mtype == "Solid State Device": mtype = "SSD" else: mtype = "N/A" elif re.match(r"Device Speed: .*$", line.strip()): speed = line.split(":")[1].strip() elif re.match(r"Drive Temperature :.*$", line.strip()): if notempmode: temp = "N/A" else: # Drive temp is amongst the last few lines matched, decide here if we add information to the table.. temp = line.split(":")[1].strip() temp = re.sub(" \(.*\)", "", temp) if model != "Unknown": dbgprint("Disk Info: " + str(arrayid) + " " + str(diskid) + " " + str(oldenclid)) if subfstate == "Rebuild": cmd = "%s pdrbld -showprog -physdrv\[%s:%s\] -a%d -NoLog" % (megaclipath, enclid, slotid, controllerid) output = getOutput(cmd) percent = returnRebuildProgress(output) fstate = str("Rebuilding (%d%%)" % (percent)) if (NestedLDTable[controllerid][int(arrayindex)] == True) and (spanid != False): sarrayid = str(arrayid) + "s" + spanid else: sarrayid = str(arrayid) table.append([sarrayid, str(diskid), mtype, model, dsize, fstate, speed, temp, enclid, slotid, lsidid]) return table def returnUnconfDiskInfo(output, controllerid): arrayid = False diskid = False olddiskid = False enclid = False slotid = False lsidid = "Unknown" table = [] fstate = "Offline" substate = "Unknown" model = "Unknown" speed = "Unknown" mtype = "Unknown" dsize = "Unknown" temp = "Unk0C" ospath = "N/A" for line in output: if re.match(r"Enclosure Device ID: .*$", line.strip()): # We match here early in the analysis so reset the vars if this is a new disk we're reading.. oldenclid = enclid enclid = line.split(":")[1].strip().replace("N/A", "") if oldenclid != False: arrayid = False fstate = "Offline" model = "Unknown" speed = "Unknown" temp = "Unk0C" slotid = False lsidid = "Unknown" elif re.match(r"^Coerced Size: ", line.strip()): dsize = line.split(":")[1].strip() dsize = re.sub(" \[.*\.*$", "", dsize) dsize = re.sub("[0-9][0-9] GB", " Gb", dsize) elif re.match(r"^Drive.s posi*tion: DiskGroup: [0-9]+,.*$", line.strip()): arrayid = line.split(",")[1].split(":")[1].strip() elif re.match(r"^Device Id: [0-9]+.*$", line.strip()): diskid = line.split(":")[1].strip() elif re.match(r"Slot Number: .*$", line.strip()): slotid = line.split(":")[1].strip() elif re.match(r"Firmware state: .*$", line.strip()): fstate = line.split(":")[1].strip() subfstate = re.sub("\(.*", "", fstate) dbgprint("Firmware State: " + str(fstate) + " " + str(subfstate)) elif re.match(r"Inquiry Data: .*$", line.strip()): model = line.split(":")[1].strip() model = re.sub(" +", " ", model) # re-define our "sub-code" # our seagate drives have an ID string of # 'Z1E19S2QST2000DM001-1CH164 CC43' # or # '6XW02738ST32000542AS CC32' m = re.match(r"(\w{8})(ST\w+)(?:-(\w{6}))?(?:\s+(\w+))", model) if m: if m.group(3): model = "{0}-{1} {2} {3}".format(m.group(2), m.group(3), m.group(4), m.group(1)) else: model = "{0} {1:>10} {2}".format(m.group(2), m.group(4), m.group(1)) continue manuf = re.sub(" .*", "", model) dtype = re.sub(manuf + " ", "", model) dtype = re.sub(" .*", "", dtype) hwserial = re.sub(".*" + dtype + " *", "", model) elif re.match(r"^Media Type: .*$", line.strip()): mtype = line.split(":")[1].strip() if mtype == "Hard Disk Device": mtype = "HDD" else: if mtype == "Solid State Device": mtype = "SSD" else: mtype = "N/A" elif re.match(r"Device Speed: .*$", line.strip()): speed = line.split(":")[1].strip() elif re.match(r"Drive Temperature :.*$", line.strip()): # Drive temp is amongst the last few lines matched, decide here if we add information to the table.. if notempmode: temp = "N/A" else: temp = line.split(":")[1].strip() temp = re.sub("\(.*\)", "", temp) if arrayid == False: if subfstate == "Unconfigured": dbgprint("Unconfigured Disk: Arrayid: " + str(arrayid) + " DiskId: " + str(diskid) + " " + str(olddiskid) + " " + str(fstate)) elif subfstate == "Online, Spun Up": dbgprint("Online Unconfed Disk: Arrayid: " + str(arrayid) + " DiskId: " + str(diskid) + " " + str(olddiskid) + " " + str(fstate)) table.append([mtype, model, dsize, fstate, speed, temp, enclid, slotid, diskid, ospath]) return table cmd = "%s -adpCount -NoLog" % (megaclipath) output = getOutput(cmd) controllernumber = returnControllerNumber(output) bad = False # List available controller if printcontroller: if controllernumber: if not nagiosmode: print("-- Controller information --") i = 0 controllerid = 0 mlen = 0 hbainfo = [] while controllerid < controllernumber: cmd = "%s -AdpAllInfo -a%d -NoLog" % (megaclipath, controllerid) output = getOutput(cmd) returnHBAInfo(hbainfo, output, controllerid) controllerid += 1 mlen = returnWdthFromArrayCol(hbainfo, 1) controllerid = 0 for hba in hbainfo: hbafmt = str("%-5s | %-" + str(mlen) + "s | %-6s | %-4s | %-6s | %-12s ") # Header if i == 0: if not nagiosmode: print(hbafmt % ("-- ID", "H/W Model", "RAM", "Temp", "BBU", "Firmware")) if not nagiosmode: print(hbafmt % (hba[0], hba[1], hba[2], hba[3], hba[4], hba[5])) i += 1 if not nagiosmode: print("") else: print("No MegaRAID or PERC adapter detected on your system!") exit(1) if printarray: if not nagiosmode: print("-- Array information --") controllerid = 0 pcipath = "" diskpath = "" i = 0 j = 0 mlen = 0 rlen = 0 clen = 0 while controllerid < controllernumber: arrayindex = 0 cmd = "%s -LDInfo -lall -a%d -NoLog" % (megaclipath, controllerid) output = getOutput(cmd) arraynumber = returnArrayNumber(output) # We need to explore each HBA to look for gaps in LD's ldid = 0 ldcount = 0 while ldcount < arraynumber: cmd = "%s -LDInfo -l%d -a%d -NoLog" % (megaclipath, ldid, controllerid) output = getOutput(cmd) for line in output: if re.match(r"^Adapter.*Virtual Drive .* Does not Exist", line.strip()): ldid += 1 elif re.match(r"^(CacheCade )?Virtual Drive:", line.strip()): LDTable[controllerid].append(ldid) # NestedLDTable[controllerid][int(arrayindex)] = False ldcount += 1 ldid += 1 while arrayindex < arraynumber: ldid = LDTable[controllerid][arrayindex] cmd = "%s -LDInfo -l%d -a%d -NoLog" % (megaclipath, ldid, controllerid) output = getOutput(cmd) arrayinfo = returnArrayInfo(output, controllerid, ldid, arrayindex) if len(arrayinfo[1]) > rlen: rlen = len(arrayinfo[1]) if len(arrayinfo[4]) > mlen: mlen = len(arrayinfo[4]) if len(arrayinfo[8]) > clen: clen = len(arrayinfo[8]) arrayindex += 1 controllerid += 1 controllerid = 0 while controllerid < controllernumber: arrayindex = 0 cmd = "%s -AdpGetPciInfo -a%d -NoLog" % (megaclipath, controllerid) output = getOutput(cmd) pcipath = returnHBAPCIInfo(output) cmd = "%s -LDInfo -lall -a%d -NoLog" % (megaclipath, controllerid) output = getOutput(cmd) arraynumber = returnArrayNumber(output) while arrayindex < arraynumber: ldid = LDTable[controllerid][arrayindex] cmd = "%s -LDInfo -l%d -a%d -NoLog" % (megaclipath, ldid, controllerid) output = getOutput(cmd) arrayinfo = returnArrayInfo(output, controllerid, ldid, arrayindex) if pcipath: diskprefix = str("/dev/disk/by-path/pci-" + pcipath + "-scsi-0:") dbgprint("Will look for DISKprefix : " + diskprefix) # RAID disks are usually with a channel of '2', JBOD disks with a channel of '0' for j in range(1, 8): diskpath = diskprefix + str(j) + ":" + str(arrayinfo[7]) + ":0" dbgprint("Looking for DISKpath : " + diskpath) if os.path.exists(diskpath): arrayinfo[7] = os.path.realpath(diskpath) dbgprint("Found DISK match: " + diskpath + " -> " + arrayinfo[7]) break else: arrayinfo[7] = "N/A" # Pad the string length, just to make sure it's aligned with the headers... if rlen < len("Type"): rlen = len("Type") if mlen < len("Flags"): mlen = len("Flags") if clen < len("CacheCade"): clen = len("CacheCade") ldfmt = str("%-5s | %-" + str(rlen) + "s | %7s | %7s | %" + str(mlen) + "s | %8s | %8s | %8s | %-" + str(clen) + "s |%-12s ") # Header if i == 0: if not nagiosmode: print(ldfmt % ("-- ID", "Type", "Size", "Strpsz", "Flags", "DskCache", "Status", "OS Path", "CacheCade", "InProgress")) if not nagiosmode: print( ldfmt % ( arrayinfo[0], arrayinfo[1], arrayinfo[2], arrayinfo[3], arrayinfo[4], arrayinfo[5], arrayinfo[6], arrayinfo[7], arrayinfo[8], arrayinfo[9], ) ) dbgprint("Array state : LD " + arrayinfo[0] + ", status : " + arrayinfo[6]) if arrayinfo[6] not in ["Optimal", "N/A"]: bad = True nagiosbadarray += 1 else: nagiosgoodarray += 1 arrayindex += 1 i += 1 controllerid += 1 if not nagiosmode: print("") controllerid = 0 while controllerid < controllernumber: cmd = "%s -PDGetNum -a%d -NoLog" % (megaclipath, controllerid) output = getOutput(cmd) totaldrivenumber += returnTotalDriveNumber(output) controllerid += 1 if totaldrivenumber: if not nagiosmode: print("-- Disk information --") i = 0 dlen = 0 mlen = 0 flen = 0 controllerid = 0 while controllerid < controllernumber: arrayid = 0 cmd = "%s -LDInfo -lall -a%d -NoLog" % (megaclipath, controllerid) output = getOutput(cmd) arraynumber = returnArrayNumber(output) #### BUG: -LdPdInfo shows all PD on the adapter, not just for the LD we wanted.. #### while arrayid <= arraynumber: cmd = "%s -LdPdInfo -a%d -NoLog" % (megaclipath, controllerid) output = getOutput(cmd) arraydisk = returnDiskInfo(output, controllerid) for array in arraydisk: diskname = str(controllerid) + array[8] + array[9] dbgprint("Disk c" + diskname + " status : " + array[5]) if re.match("|".join(["^Online$", "^Online, Spun Up$", "^Rebuilding \(.*"]), array[5]): if AddDisk(NagiosGoodDisks, diskname): nagiosgooddisk += 1 else: bad = True if AddDisk(NagiosBadDisks, diskname): nagiosbaddisk += 1 if returnWdthFromArrayCol(arraydisk, 0) > dlen: dlen = returnWdthFromArrayCol(arraydisk, 0) if returnWdthFromArrayCol(arraydisk, 3) > mlen: mlen = returnWdthFromArrayCol(arraydisk, 3) if returnWdthFromArrayCol(arraydisk, 5) > flen: flen = returnWdthFromArrayCol(arraydisk, 5) controllerid += 1 controllerid = 0 while controllerid < controllernumber: arrayid = 0 cmd = "%s -LDInfo -lall -a%d -NoLog" % (megaclipath, controllerid) output = getOutput(cmd) arraynumber = returnArrayNumber(output) #### BUG: -LdPdInfo shows all PD on the adapter, not just for said LD.. #### while arrayid <= arraynumber: cmd = "%s -LdPdInfo -a%d -NoLog" % (megaclipath, controllerid) output = getOutput(cmd) arraydisk = returnDiskInfo(output, controllerid) # Adjust print format with width computed above drvfmt = "%-" + str(dlen + 6) + "s | %-4s | %-" + str(mlen) + "s | %-8s | %-" + str(flen) + "s | %-8s | %-4s | %-8s | %-8s" for array in arraydisk: # Header if i == 0: if not nagiosmode: print(drvfmt % ("-- ID", "Type", "Drive Model", "Size", "Status", "Speed", "Temp", "Slot ID", "LSI ID")) # Drive information if not nagiosmode: print( drvfmt % ( str("c" + str(controllerid) + "u" + array[0] + "p" + array[1]), # c0p0 array[2], # HDD/SDD array[3], # Model Information (Variable len) array[4], # Size array[5], # Status (Variable len) array[6], # Speed array[7], # Temp str("[" + array[8] + ":" + array[9] + "]"), # Slot ID array[10], ) ) # LSI ID i = i + 1 controllerid += 1 if not nagiosmode: print("") controllerid = 0 totalconfdrivenumber = 0 totalunconfdrivenumber = 0 totaldrivenumber = 0 while controllerid < controllernumber: cmd = "%s -LdPdInfo -a%d -NoLog" % (megaclipath, controllerid) output = getOutput(cmd) totalconfdrivenumber += returnConfDriveNumber(controllerid, output) cmd = "%s -PDGetNum -a%d -NoLog" % (megaclipath, controllerid) output = getOutput(cmd) totaldrivenumber += returnTotalDriveNumber(output) cmd = "%s -PDList -a%d -NoLog" % (megaclipath, controllerid) output = getOutput(cmd) # Sometimes a drive will be reconfiguring without any info on that it is going through a rebuild process. # This happens when expanding an R{5,6,50,60} array, for example. In that case, totaldrivenumber will still be # greater than totalconfdrivenumber while returnUnConfDriveNumber(output) will be zero. The math below attempts to solve this. totalunconfdrivenumber += max(returnUnConfDriveNumber(output), totaldrivenumber - totalconfdrivenumber) controllerid += 1 dbgprint("Total Drives in system : " + str(totaldrivenumber)) dbgprint("Total Configured Drives : " + str(totalconfdrivenumber)) dbgprint("Total Unconfigured Drives : " + str(totalunconfdrivenumber)) if totalunconfdrivenumber: if not nagiosmode: print("-- Unconfigured Disk information --") controllerid = 0 pcipath = "" while controllerid < controllernumber: arrayid = 0 cmd = "%s -LDInfo -lall -a%d -NoLog" % (megaclipath, controllerid) output = getOutput(cmd) arraynumber = returnArrayNumber(output) cmd = "%s -AdpGetPciInfo -a%d -NoLog" % (megaclipath, controllerid) output = getOutput(cmd) pcipath = returnHBAPCIInfo(output) #### BUG: -LdPdInfo shows all PD on the adapter, not just for given LD.. #### while arrayid <= arraynumber: cmd = "%s -PDList -a%d -NoLog" % (megaclipath, controllerid) output = getOutput(cmd) arraydisk = returnUnconfDiskInfo(output, controllerid) for array in arraydisk: dbgprint("Unconfed " + str(nagiosgooddisk) + "/" + str(nagiosbaddisk) + " Disk c" + str(controllerid) + "uXpY status : " + array[3]) if array[3] in [ "Online", "Unconfigured(good), Spun Up", "Unconfigured(good), Spun down", "JBOD", "Hotspare, Spun Up", "Hotspare, Spun down", "Online, Spun Up", ]: nagiosgooddisk += 1 else: bad = True nagiosbaddisk += 1 # JBOD disks has a real device path and are not masked. Try to find a device name here, if possible. if pcipath: if array[3] in ["JBOD"]: diskprefix = str("/dev/disk/by-path/pci-" + pcipath + "-scsi-0:0:") dbgprint("Will look for DISKprefix : " + diskprefix) # RAID disks are usually with a channel of '2', JBOD disks with a channel of '0' diskpath = diskprefix + str(array[8]) + ":0" dbgprint("Looking for DISKpath : " + diskpath) if os.path.exists(diskpath): dbgprint("Found DISK match: " + diskpath + " -> " + array[9]) array[9] = os.path.realpath(diskpath) else: dbgprint("DISK NOT present: " + diskpath) array[9] = "N/A" mlen = returnWdthFromArrayCol(arraydisk, 1) flen = returnWdthFromArrayCol(arraydisk, 3) # Adjust print format with widths computed above drvfmt = "%-7s | %-4s | %-" + str(mlen) + "s | %-8s | %-" + str(flen + 2) + "s | %-8s | %-4s | %-8s | %-6s | %-8s" i = 0 for array in arraydisk: # Header if i == 0: if not nagiosmode: print(drvfmt % ("-- ID", "Type", "Drive Model", "Size", "Status", "Speed", "Temp", "Slot ID", "LSI ID", "Path")) # Drive information if not nagiosmode: print( drvfmt % ( str("c" + str(controllerid) + "uXpY"), # cXpY array[0], # HDD/SDD array[1], # Model Information (Variable len) array[2], # Size array[3], # Status (Variable len) array[4], # Speed array[5], # Temp str("[" + array[6] + ":" + array[7] + "]"), # Slot ID array[8], # LSI ID array[9], ) ) # OS path, if any i += 1 controllerid += 1 if not nagiosmode: print("") if debugmode: dbgprint("Printing Outputs[][]") for myl in Outputs: dbgprint(myl + "\n") sys.stderr.write("\n".join("".join(map(str, myd)) for myd in Outputs[myl]) + "\n") dbgprint("Printing arraydisk[]") sys.stderr.write("\n".join(" | ".join(map(str, myd)) for myd in arraydisk) + "\n") dbgprint("Printing ConfDisks[]") sys.stderr.write("\n".join("".join(map(str, myd)) for myd in ConfDisks) + "\n") dbgprint("Printing NagiosGoodDisks[]") sys.stderr.write("\n".join("".join(map(str, myd)) for myd in NagiosGoodDisks) + "\n") dbgprint("Printing NagiosBadDisks[]") sys.stderr.write("\n".join("".join(map(str, myd)) for myd in NagiosBadDisks) + "\n") if nagiosmode: if bad: print( "RAID ERROR - Arrays: OK:" + str(nagiosgoodarray) + " Bad:" + str(nagiosbadarray) + " - Disks: OK:" + str(nagiosgooddisk) + " Bad:" + str(nagiosbaddisk) ) sys.exit(2) else: print( "RAID OK - Arrays: OK:" + str(nagiosgoodarray) + " Bad:" + str(nagiosbadarray) + " - Disks: OK:" + str(nagiosgooddisk) + " Bad:" + str(nagiosbaddisk) ) else: if bad: # DO NOT MODIFY OUTPUT BELOW # Scripts may relies on it # https://github.com/eLvErDe/hwraid/issues/99 print("\nThere is at least one disk/array in a NOT OPTIMAL state.") print( "RAID ERROR - Arrays: OK:" + str(nagiosgoodarray) + " Bad:" + str(nagiosbadarray) + " - Disks: OK:" + str(nagiosgooddisk) + " Bad:" + str(nagiosbaddisk) ) sys.exit(1) 保存为 megaclisas-status.py ，执行 python megaclisas-status.py 运行结果，可能页面显示看的比较遭，暂时没时间鼓捣，可以复制内容去文本全屏看的比较直观：# python megaclisas-status.py [0:2:0:0] disk DELL PERC H710P 3.13 /dev/sda [0:2:1:0] disk DELL PERC H710P 3.13 /dev/sdb [0:2:2:0] disk DELL PERC H710P 3.13 /dev/sdc [0:2:3:0] disk DELL PERC H710P 3.13 /dev/sdd [0:2:4:0] disk DELL PERC H710P 3.13 /dev/sde [0:2:5:0] disk DELL PERC H710P 3.13 /dev/sdf [0:2:6:0] disk DELL PERC H710P 3.13 /dev/sdg [0:2:7:0] disk DELL PERC H710P 3.13 /dev/sdh -- Controller information -- -- ID | H/W Model | RAM | Temp | BBU | Firmware c0 | PERC H710P Mini | 1024MB | 45C | Good | FW: 21.3.1-0004 -- Array information -- -- ID | Type | Size | Strpsz | Flags | DskCache | Status | OS Path | CacheCade |InProgress c0u0 | RAID-0 | 232G | 64 KB | ADRA,WB | Disabled | Optimal | 0 | None |None c0u1 | RAID-0 | 3637G | 64 KB | ADRA,WB | Disabled | Optimal | 1 | Type : Read Only |None c0u2 | RAID-0 | 3637G | 64 KB | ADRA,WB | Disabled | Optimal | 2 | Type : Read Only |None c0u3 | RAID-0 | 3637G | 64 KB | ADRA,WB | Disabled | Optimal | 3 | Type : Read Only |None c0u4 | RAID-0 | 3637G | 64 KB | ADRA,WB | Disabled | Optimal | 4 | Type : Read Only |None c0u5 | RAID-0 | 3637G | 64 KB | ADRA,WB | Disabled | Optimal | 5 | Type : Read Only |None c0u6 | RAID-0 | 3637G | 64 KB | ADRA,WB | Disabled | Optimal | 6 | Type : Read Only |None c0u7 | RAID-0 | 12732G | 64 KB | ADRA,WB | Disabled | Optimal | 7 | Type : Read Only |None -- Disk information -- -- ID | Type | Drive Model | Size | Status | Speed | Temp | Slot ID | LSI ID c0u0p0 | SSD | S4CKNF0NC22284X Samsung SSD 860 EVO 250GB RVT04B6Q | 232.3 Gb | Online, Spun Up | 6.0Gb/s | 30C | [32:0] | 0 c0u1p0 | HDD | ST4000NM0035 SN07 WE26UXL9 | 3.637 TB | Online, Spun Up | 6.0Gb/s | 32C | [32:1] | 1 c0u2p0 | HDD | ST4000NM0035 SN07 WE26UXLP | 3.637 TB | Online, Spun Up | 6.0Gb/s | 33C | [32:2] | 2 c0u3p0 | HDD | ST4000NM0035 SN07 WE28V2ST | 3.637 TB | Online, Spun Up | 6.0Gb/s | 32C | [32:4] | 4 c0u4p0 | HDD | ST4000NM0035 SN07 WE28V2UL | 3.637 TB | Online, Spun Up | 6.0Gb/s | 30C | [32:5] | 5 c0u5p0 | HDD | ST4000NM000A SN07 WS21NA19 | 3.637 TB | Online, Spun Up | 6.0Gb/s | 31C | [32:3] | 3 c0u6p0 | HDD | VBH8A06F HGST HUS726T4TALE6L4 VKGNW9G0 | 3.637 TB | Online, Spun Up | 6.0Gb/s | 27C | [32:6] | 6 c0u7p0 | HDD | Y6GD3X2C WDC WUH721414ALE6L4 LDGNW240 | 12.732 TB | Online, Spun Up | 6.0Gb/s | 25C | [32:7] | 7 例如要找/dev/sdd，看lsscsi的输出可以知道在系统中的sdd的信息“0:2:3:0”，然后看Array information 中 OS Path 3的ID是“c0u3”，接着看Disk information中“c0u3p0”中磁盘盘位是4。这批有问题的盘是以前国内淘宝买的，真是被骗的好惨……说是全新，刚买回来测验也没问题，码也没问题，刚一年不到就这样了，基本可以确定是贴标的盘，造假技术太强了，现在硬盘带不回国只能扔掉了。湾仔电脑城买的延迟、速度就很正常，价钱高一些但可以确保是正品，坏了也可以马上拿去保修换新的。硬盘千万不要贪便宜淘宝买，要找专门经销商，检查好他们的资质，一定小心被骗。