Overview

What Is SR-IOV

SR-IOV (Single Root I/O Virtualization) is a hardware-based virtualization solution. This technology allows a single PCIe device, such as a physical NIC, to create multiple independent Virtual Functions (VFs). These VFs can be directly assigned to multiple instances, including VM instances, VPC vRouters, and load balancers. SR-IOV delivers near-native physical device I/O performance and low latency, significantly improving data processing performance and scalability in virtualized environments.

ZStack Cloud supports the SR-IOV specification. Administrators can virtualize a single physical NIC into multiple VF NICs. After the virtualization, you can directly assign VF NICs to instances.

Compared to traditional vNICs, VF NICs offer the following advantages:

Bypass the virtualization layer and shorten the data transmission path, providing VM instances, VPC vRouters, and load balancers with I/O performance close to that of physical devices.
Significantly reduce consumption of host CPUs. Even under high CPU load, this effectively reduces network packet loss and improve transmission efficiency.

Applications

Network Functions Virtualization (NFV)

As technologies like cloud computing and virtualization mature, the telecommunications industry is also transforming the architecture of traditional networking through virtualization. NFV devices are accelerating the shift to software-based solutions, enabling large-scale virtualized deployment of applications within network equipment. This approach helps reduce costs, increase flexibility, and effectively enhance competitiveness.

However, these applications have extremely high demands on network capabilities such as throughput, forwarding, and packet processing, requiring high-performance virtual networks to function effectively. Consequently, NICs with SR-IOV capabilities, valued for their maturity and virtualizability, are widely adopted in NFV devices. This allows data centers to obtain high-performance, easily scalable, and elastic network management capabilities at a relatively low cost.

Cloud Gaming

With the development of broadband networks and the widespread adoption of mobile devices, the cloud gaming model—where game computation is offloaded to the cloud, and the client is only responsible for display and control—has quietly gained popularity. For games with high real-time performance requirements, network capabilities such as packet forwarding, instantaneous throughput, and latency stability are essential. Without these, issues like frame drops and input lag may occur, severely impacting the gaming experience.

In this cloud gaming model, you can leverage the stable network performance of physical NICs by using SR-IOV to directly assign VFs to VM instances. The traffic from the VM instance's virtual NIC is sent directly to the VF, bypassing intermediate software switches like bridge NICs or Open vSwitch. This significantly reduces packet processing overhead and provides users with a better gaming experience.

Video Streaming

With the evolution of the Internet, the transmission of multimedia information online has become increasingly critical. To ensure transmission speed, video streaming based on the UDP protocol is widely used. For scenarios requiring real-time video transmission (such as live streaming and video conferencing), network performance in terms of packet forwarding, instantaneous throughput, and latency stability is indispensable. Otherwise, issues like video stuttering and reduced resolution may occur, severely degrading the viewing experience.

It is well-known that while the UDP protocol can increase transmission speed and significantly reduce transmission time, it cannot guarantee data reliability. Network fluctuations and packet loss directly impact video quality—haste makes waste. NICs with SR-IOV capabilities, leveraging their maturity and virtualizability, offer unique advantages in real-time video scenarios. They not only guarantee network performance but also reduce CPU load, effectively minimize packet loss, and improve transmission efficiency. This perfectly compensates for the reliability shortcomings of the UDP protocol, making them an excellent fit for such applications.

Before You Begin

Known Limitations

The SR-IOV feature has the following constraints with NIC bonding:

If a physical NIC is in the bond and the bond contains more than two NICs, you cannot virtualize the physical NIC.
If a physical NIC is in the bond, when you virtualize this physical NIC, all physical NICs in that bond will be virtualized.
If a bond contains a virtualized physical NIC, you cannot add or remove physical NICs from that bond.
If a physical NIC has been virtualized, it cannot be added to a bond that contains non-virtualized NICs.

The SR-IOV feature has the following constraints with hot migration:

Before you hot migrate a VM instance with VF NICs attached, you must install GuestTools (QGA) in the VM instance and make sure the GuestTools is running normally.
VPC vRouters and load balancers come with the GuestTools pre-installed, you can hot migrate them without requiring additional installation.
Hot migration with VF NICs may fail if the internal driver version of the VM instance, VPC vRouter, or load balancers is too low. Check and update the driver version as needed.
Depending on the actual network environment, short network interruptions may occur during the hot migration of VM instance with VF NICs. Use with caution in business-critical scenarios.

Other limitations or constraints:

Security group network services are not available for VF NICs.
For a VPC vRouter with SR-IOV enabled, if any of its attached networks have VIP QoS or Network Bandwidth services configured, the VPC vRouter performance may be affected.
For users upgrading from a version prior to ZStack Cloud 5.5.0, if a VPC vRouter was already SR-IOV enabled before the upgrade and did not have VIP QoS or Network Bandwidth configured, it is recommended to manually reboot that VPC vRouter after the upgrade to ensure optimal performance.
For VM instances, VPC vRouters, or load balancers already using VF NICs:
- Do not uninstall the PF driver of the physical NIC, as this will cause the system to forcibly reclaim the VF NICs.
- Stopping an instance will automatically release its VF NICs. Starting the instance will re-acquire VF NICs. If no VF NICs are available, the instance will fail to start.

Prerequisites

To use SR-IOV in ZStack Cloud, your environment must meet the following configuration requirements:

Ensure the physical NIC supports SR-IOV and that SR-IOV is enabled.
Ensure the Intel VT-d or AMD IOMMU and SR-IOV are enabled in the host's BIOS.
BIOS configuration varies across different servers. For specific procedures, consult your hardware vendor's documentation.
Ensure the host's CPU supports Interrupt Remapping.
1. Create a script named interrupt_remapping_check.sh with the following content.
```
#!/bin/sh
if [ $(dmesg | grep ecap | wc -l) -eq 0 ]; then
  echo "No interrupt remapping support found"
  exit 1
fi

for i in $(dmesg | grep ecap | awk '{print $NF}'); do
  if [ $(( (0x$i & 0xf) >> 3 )) -ne 1 ]; then
    echo "Interrupt remapping not supported"
    exit 1
  fi
done
```
2. Grant the script executable permissions.
3. Execute the interrupt_remapping_check.sh script on the host to check if the CPU supports Interrupt Remapping.
  - If the output is No interrupt remapping support found or Interrupt remapping not supported, the CPU does not support Interrupt Remapping.
  - If there is no output, the CPU supports Interrupt Remapping.
4. If the CPU does not support Interrupt Remapping, you can configure it by executing the following command.
```
[root@localhost ~]# echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
```
Ensure the relevant drivers are installed.
- The host requires the PF NIC driver to be installed, ensuring the physical NIC is recognized by the host and can be virtualized using SR-IOV.
- VM instances, VPC vRouters, or load balancers require the VF NIC driver to be installed, ensuring the VF NIC is recognized and functions correctly.
Note: The operating system release contains a default driver for certain NICs, while for others you must download and install it from a location provided by the vendor of the NIC or the host.

Using SR-IOV

Configure VF NICs for a VM instance

Prerequisites

To use SR-IOV, ensure your environment meets the software and hardware compatibility requirements described in Prerequisites.

Procedure

Log in to ZStack Cloud.
Enable IOMMU on the host.
For a new host to be added:
1. On the main menu of ZStack Cloud, choose Resource Center > Hardware > Host.
2. On the Host page, click Add Host.
3. On the Add Host page, complete the host configuration and enable the Scan Host IOMMU Setting option.
Figure 1. Enable Scan Host IOMMU When Adding Hosts
For an existing host:
1. On the main menu of ZStack Cloud, choose Resource Center > Hardware > Host.
2. On the Host page, click the name of the target host to enter its details page.
3. On the Overview tab of the target host's details page, enable IOMMU State.
4. After enabling it, restart the host for the configuration to take effect.
Figure 2. Enable IOMMU State for Existing Hosts
Virtualize the physical NIC using SR-IOV.
1. On the details page of the target host, click Associated Resource > Physical NIC.
  Administrators can check the Virtualization Status in the Physical NIC list to determine if a physical NIC supports SR-IOV.
2. In the Physical NIC list, select the target physical NIC, then click Actions > SR-IOV.
3. In the SR-IOV dialog, enter the number of VFs to create.
4. Click OK.
Figure 3. SR-IOV Physical NICs
Create an SR-IOV enabled network.
1. On the main menu of ZStack Cloud, choose Resource Center > Network Resource > L2 Network.
2. On the L2 Network page, click Create L2 Network.
3. On the Create L2 Network page, complete the relevant configuration. For Cluster, select the cluster that contains the hosts with IOMMU enabled. For NIC Name, select the name of the physical NIC that has been virtualized.
4. Click OK.
5. On the Network Resource menu, click Flat Network.
6. On the Flat Network page, click Create Flat Network.
7. On the Create Flat Network page, complete the relevant network configurations. For L2 Network, select the L2 network you just created.
8. Click OK.
Create a VM instance with VF NIC attached.
1. On the main menu of ZStack Cloud, choose Resource Center > Resource Pool > VM Instance.
2. On the VM Instance page, click Create VM Instance.
3. On the Create VM Instance page, complete the basic configuration. In the resource configuration, for Network, select the flat network created in step 4, and select the Enabled SR-IOV option.
4. Continue to complete the system configuration for the VM instance, then click OK.
Install the VF NIC driver for the VM instance.
VM instances require the VF NIC driver to be installed to ensure the VF NIC is recognized and functions correctly. For more information on obtaining and installing VF NIC drivers, contact your NIC vendor for assistance.

Configure SR-IOV for a VPC vRouter with Bonded NICs

Prerequisites

To use SR-IOV, ensure your environment meets the software and hardware compatibility requirements described in Prerequisites.
Ensure that the bond containing the physical NIC you intend to virtualize has no more than two NICs. Otherwise, SR-IOV will fail. For more information about usage limitations, see Known Limitations.
Obtain the openEuler 22.03 image for the VPC vRouter in advance.
Plan the public network and the management network for the VPC vRouter in advance. These two networks must be deployed separately.
(Optional) Based on your actual resource and business requirements, you can plan to create a new cluster using two independent servers dedicated to hosting VPC vRouters and providing high performance networking. These two servers can be configured into a VPC vRouter HA group for redundancy. For more information, see (Optional) Create a VPC vRouter HA group.

Procedure

Log in to ZStack Cloud.
Enable IOMMU on the host.
For a new host to be added:
1. On the main menu of ZStack Cloud, choose Resource Center > Hardware > Host.
2. On the Host page, click Add Host.
3. On the Add Host page, complete the host configuration and enable the Scan Host IOMMU Setting option.
Figure 4. Enable Scan Host IOMMU When Adding Hosts
For an existing host:
1. On the main menu of ZStack Cloud, choose Resource Center > Hardware > Host.
2. On the Host page, click the name of the target host to enter its details page.
3. On the Overview tab of the target host's details page, enable IOMMU State.
4. After enabling it, restart the host for the configuration to take effect.
Figure 5. Enable IOMMU State for Existing Hosts
Add a bond.
1. On the details page of the target host, click Associated Resource > Physical NIC.
2. On the Physical NIC page, select Bond, then click Add Bond.
  
  Figure 6. Add Bond
3. In the Add Bond drawer, select two physical NICs, choose the bond mode, and specify the IPv4 and netmask.
  - It is recommended to select physical NICs with the same type, vendor, and model.
  - After selecting the bond mode, configure the switch side to the corresponding mode accordingly.
  Figure 7. Configure Bond Settings
4. Click OK.
Virtualize a bonded NIC using SR-IOV.
1. On the Physical NIC page, select the Physical NIC tab.
2. In the Physical NIC list, select the target physical NIC, then click Actions > SR-IOV.
3. In the SR-IOV dialog, enter the number of VFs to create.
  Because the selected physical NIC is in a bond, virtualizing this physical NIC will simultaneously virtualize all NICs in the bond.
4. Click OK.
Figure 8. SR-IOV Bonded NIC
Create an L2 network.
1. On the main menu of ZStack Cloud, choose Resource Center > Network Resource > L2 Network.
2. On the L2 Network page, click Create L2 Network.
3. On the Create L2 Network page, complete the relevant configuration as needed. For Cluster, select the cluster that contains the hosts with IOMMU enabled. For NIC Name, select the name of the physical NIC that has been virtualized.
4. Click OK.
Figure 9. Create L2 Network
Create a public network.
1. On the Network Resource menu, click Public Network.
2. On the Public Network page, click Create Public Network.
3. On the Create Public Network page, complete the public network configuration.
4. Click OK.
Figure 10. Create Public Network
Create a management network.
1. On the Network Resource menu, click Management Network.
2. On the Management Network page, click Create Management Network.
3. On the Create Management Network page, complete the management network configurations.
4. Click OK.
Figure 11. Create Management Network
Add a vRouter image.
1. On the Network Resource menu, click vRouter Image.
2. On the vRouter Image page, click Add vRouter Image.
3. On the Add vRouter Image page, upload the obtained openEuler image via URL or as a local file.
4. Click OK.
Figure 12. Add vRouter Image
Create a vRouter offering.
1. On the Network Resource menu, click vRouter Offering.
2. On the vRouter Offering page, click Create vRouter Offering.
3. On the Create vRouter Offering page, complete the specification settings.
  - Set the vRouter offering name, CPU, and memory.
  - For image, select the VPC vRouter image added in Step 8.
  - For management network and public network, select two separate networks: select the public network created in Step 6 and the management network created in Step 7.
4. Click OK.
Figure 13. Create vRouter Offering
Create a VPC vRouter and enable SR-IOV.
1. On the Network Resource menu, click VPC vRouter.
2. On the VPC vRouter page, click Create VPC vRouter.
3. On the Create VPC vRouter page, select Enable SR-IOV, then complete the other vRouter configurations.
Figure 14. Create VPC vRouter
Create a VPC network.
1. On the Network Resource menu, click VPC Network.
2. On the VPC Network page, click Create VPC Network.
3. On the Create VPC Network page, complete the VPC network configurations.
4. Click OK.
Figure 15. Create VPC Network
Attach a VPC vRouter to the VPC network and enable SR-IOV.
1. On the Network Resource menu, click VPC Network.
2. On the VPC Network page, locate the VPC network you created in Step 11.
3. Click Actions > Attach VPC vRouter.
4. In the Select VPC vRouter drawer, select the VPC vRouter created in Step 10.
5. Select Enable SR-IOV.
6. Click OK.
Figure 16. Attach VPC vRouter and Enable SR-IOV
(Optional) Create a VPC vRouter HA group.
1. On the main menu of ZStack Cloud, choose Resource Center > Network Resource > VPC vRouter HA Group.
2. On the VPC vRouter HA Group page, click Create VPC vRouter HA Group.
3. On the Create VPC vRouter HA Group page, select importing an existing VPC vRouter or creating a new VPC vRouter.
4. Click OK.
Figure 17. Create VPC vRouter HA Group

FAQ

Host IOMMU is Enabled, But IOMMU Status is Unavailable

After enabling host IOMMU on a host, you must also ensure its IOMMU state is available. Otherwise, you cannot use the SR-IOV feature normally.

Possible Cause 1

IOMMU was enabled but the host was not rebooted.

Solution

Procedure

Locate the target host and click Actions > Power Control > Reboot.
After acknowledging the risks, click OK.

Possible Cause 2

Incorrect host configuration.

Solution

Procedure

Locate the target host and enter its BIOS.
Enable the Intel VT-d or AMD IOMMU setting to make the IOMMU configuration effective in the kernel.
BIOS configuration varies across different servers. For specific procedures, consult your hardware vendor's documentation.

Cannot Select Enable SR-IOV When Adding L3 Network to a VM Instance

When creating a VM instance and selecting a L3 network in the network configuration, the Enable SR-IOV checkbox for that L3 network is grayed and unavailable for selection.

Possible Cause

The physical NIC associated with this L3 network has no available VF NICs.

Solution

Check if the physical NIC associated with this L3 network has been virtualized.
Change to a different L3 network.

PreviousSDN Tutorial NextGPU Passthrough Tutorial