PVE 硬件直通

配置 GRUB 引导参数

nano /etc/default/grub

修改这一行(intel cpu)

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

amd cpu 默认开启了 iommu,改为 quiet iommu=pt 就行

保存后执行

update-grub

加载必要内核模块

nano /etc/modules

增加

 vfio
 vfio_iommu_type1
 vfio_pci

执行如下命令刷新内核模块

update-initramfs -u -k all

验证已开启设备直通

  1. 重启系统,让 initramfs 和 grub 参数生效
reboot
  1. 检查模块是否正确加载,输入
lsmod | grep vfio
 
# 输出示范
vfio_pci               16384  0
vfio_pci_core          86016  1 vfio_pci
irqbypass              12288  2 vfio_pci_core,kvm
vfio_iommu_type1       49152  0
vfio                   65536  3 vfio_pci_core,vfio_iommu_type1,vfio_pci
iommufd                94208  1 vfio
  1. 验证 IOMMU 状态
dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
 
# 输出示范
[    0.017876] ACPI: DMAR 0x0000000074FA0000 000088 (v02 INTEL  EDK2     00000002      01000013)
[    0.017910] ACPI: Reserving DMAR table memory at [mem 0x74fa0000-0x74fa0087]
[    0.047773] DMAR: IOMMU enabled
[    0.107725] DMAR: Host address width 39
[    0.107725] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.107733] DMAR: dmar0: reg_base_addr fed90000 ver 4:0 cap 1c0000c40660462 ecap 29a00f0505e
[    0.107735] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.107739] DMAR: dmar1: reg_base_addr fed91000 ver 5:0 cap d2008c40660462 ecap f050da
[    0.107741] DMAR: RMRR base: 0x0000007c000000 end: 0x000000803fffff
[    0.107744] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.107745] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.107746] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.109397] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.353593] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
[    0.449214] DMAR: No ATSR found
[    0.449215] DMAR: No SATC found
[    0.449216] DMAR: IOMMU feature fl1gp_support inconsistent
[    0.449218] DMAR: IOMMU feature pgsel_inv inconsistent
[    0.449218] DMAR: IOMMU feature nwfs inconsistent
[    0.449219] DMAR: IOMMU feature dit inconsistent
[    0.449220] DMAR: IOMMU feature sc_support inconsistent
[    0.449221] DMAR: IOMMU feature dev_iotlb_support inconsistent
[    0.449222] DMAR: dmar0: Using Queued invalidation
[    0.449225] DMAR: dmar1: Using Queued invalidation
[    0.449688] DMAR: Intel(R) Virtualization Technology for Directed I/O

应该看到类似 IOMMU enabledAMD-Vi: EnabledInterrupt Remapping enabled 之类的输出。

  1. 确认 IOMMU 分组情况(确保设备不跟不相关的设备在一个 group)
pvesh get /nodes/$(hostname)/hardware/pci --pci-class-blacklist ""

比如下面我有两个 ssd,它们的 iommugroup 不一样

Image