Procedure

1. Enable IOMMU in the host BIOS.
   Make sure that Intel VT-d or AMD IOMMU is enabled in the host BIOS before you enable the IOMMU option on ZStack Cloud.

   • For adding host: Choose Resource Center > Hardware > Computing Facility > Host > Add Host. Then set Scan Host IOMMU Setting to true to enable IOMMU.

     
     
     

   • For added host: Select one added host and set IOMMU State to true on its details page. Reboot the host and the IOMMU setting will take effect.

   Note: After enabling IOMMU in the host, you have to make sure that IOMMU Status on the same page is available. Otherwise, the GPU passthrough feature cannot work as expected. If IOMMU State is enabled, yet IOMMU Status is unavailable, the reasons could be as follows:

   • The IOMMU setting is enabled but the host is not rebooted. Just reboot the host.
   • If a host configuration error occurs, please enter the host BIOS and enable Intel VT-d or AMD IOMMU.
2. Set ROM (Optional).

   ROM is a configuration file used for passing through pGPUs. The ROM file you upload is updated to the pGPU that the specification specifies.

   ZStack Cloud provides built-in basic ROM files, which can satisfy the majority of passthrough needs. Additionally, you can obtain other ROM files you need on the official site of the GPU supplier and then upload them.

   On the main menu of ZStack Cloud, choose Resource Center > Resource Pool > Compute Configuration > GPU Specification. Select the pGPU that you need and click Actions > Set ROM. A pop-up menu appears and you can upload ROM file herein.

   
   
   

   Note: When uploading ROM file, note the following cases:

   • The upoaded ROM file must match the target pGPU's specifications and versions. Otherwise, the passed-through pGPU cannot work properly.
   • The latest ROM file that you upload overwrites the previous ROM file.
3. Attach pGPUs to the VM instance.
   This step enables the pGPU to be directly passed through to the VM instance. On the ZStack Cloud, you can use the following methods to attach pGPUs to the VM instance:

   • Method One: Create a VM Instance and Attach pGPUs to It
     To create a VM instance, you need to choose Resource Center > Resource Pool > Virtual Resource > VM Instance > Create VM Instance. After you complete Basic Configuration, you come to next stage, that is, Resource Configurations. We support two GPU attachment policies including attaching GPU specification and attaching GPU device. Set the following parameters as you need:

     • Attach GPU Specification: Select a GPU specification and the system allocates GPU device(s) to the VM instance according to this specification. You can choose whether to make these GPU device(s) automatically detached when the VM is stopped. If you set Auto Detach to true, these GPU device(s) would be automatically detached when the VM is stopped. When the VM restarts, the system re-allocates GPU device(s) to it according to the GPU specification. If you set Auto Detach to false, the VM would keep these GPU devices attached and continue using them when it restarts.

       
       
       

     • Attach GPU Device: Select a GPU device and attach it directly to the VM instance.

       
       
       

     After completing the configurations, click OK. Then you'll get a VM instance attached to a pGPU.
   • Method Two: Attach pGPUs to an Existing VM Instance

     In the management interface of VM Instance, select the name of one existing VM instance and enter its details page. Choose Configuration info on the top row. Find pGPU Device on this page and click Attach.

     
     
     

     • A single VM instance can attach multiple pGPUs simultaneously but does not support attaching both pGPUs and vGPUs at the same time.
     • If you want to detach a GPU device, select it and click Actions > Detach.
       Note: If you detach a pGPU from a running VM instance, a blue screen or suspension may occur. We recommend stopping the VM instance before performing detach operation.
   • Method Three: Attach pGPUs to Existing VM Instances

     Select one or more stopped VM instances in the management interface of VM Instance, and click Bulk Action > System Configurations > Set GPU Policy. Then you have two options to choose, that is, attach GPU specification or attach GPU device.

     
     
     

4. Install GPU drivers on the VM instance.
   After attaching a GPU device to the VM instance, you need to install corresponding GPU drivers. Download paths for AMD or NVIDIA drivers are as follows:

   • Linux OS supports AMD GPU drivers (computing, gaming or professional series included). Linux has built-in community driver, providing you with services such as compute acceleration, displaying acceleration, and checking GPU monitoring. Click here to install the official driver.
   • Linux OS supports NVIDA GPU drivers (computing, gaming or professional series included). Linux has built-in community driver, providing you with services such as compute acceleration, displaying acceleration, and checking GPU monitoring. Click here to install the official driver.
   • Windows OS supports AMD GPU drivers (computing, gaming or professional series included). Click here to download the proper driver that matches the type of GPU and the version of OS.
   • Window OS only supports the computing series of NVIDA GPU drivers. Click here to download the proper driver that matches the type of GPU and the version of OS.

   Note: If you use AMD Firepro S7150 X2 that features two physical GPUs (pGPUs), and want the two pGPUs to be passed through to different VMs, please install the drivers of the same versions on your VMs, so that GPU monitoring data can be obtained normally.

   The procedures of installing the GPU driver vary depending on the version of your GPU device. For more information, you can contact GPU suppliers for assistance. This chapter takes installing a NVIDA GPU on the Linux VM instance as an example. You can refer to the operational procedures below:

   1. Obtain the required driver installation packages:

      Obtain the driver and CUDA toolkit compatible with the GPU device.

   2. Disable the Nouveau kernel driver:
      If NVIDIA drivers conflict with the Nouveau kernel driver, you can run the command lsmod | grep nouveau to check whether the Nouveau driver has been installed. If the output data suggests the Nouveau driver has been installed, you can perform the following operations to disable it. If no output is displayed, just skip this procedure.

      # touch  /etc/modprobe.d/nvidia-installer-disable-nouveau.conf  # Create a file and save the two lines below into it
      blacklist nouveau
      options nouveau modeset=0

   3. Install the gcc, kernel-devel, and kernel-headers files:
      Run the following commands to install the gcc, kernel-devel, and kernel-headers files and ensure that these kernel source files are of the same version. We recommend using the same version of ISO to configure local installations.

      # yum install gcc kernel-devel-$(uname -r)  kernel-headers-$(uname -r)     # Reconstruct initramfs image
      # cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak
      # dracut /boot/initramfs-$(uname -r).img $(uname -r) --force       # Only reboot the VM in the text mode
      # systemctl set-default multi-user.target
      # init 3
      # reboot
      # lsmod | grep nouveau    # After the VM instance is rebooted, check whether the nouveau driver is used or not

   4. Install an NVIDIA GPU driver:
      Upload the downloaded package to the VM instance and run the following commands to install the driver.

      # chmod +x NVIDIA-Linux-x86_64-346.47.run    # Configure executable permissions
      # ./NVIDIA-Linux-x86_64-346.47.run      # Execute the driver script 

      After you run the commands, the driver package will begin to unpack and you can follow the installation instructions. During the installation, some warnings may appear. Confirm these warnings in sequence as they do not have any real impact. If some errors occur, please refer to the table below to check the environment.

      Error Message	Solution
      ERROR: Unable to find the kernel source tree for the currently running kernel. Please make sure you have installed the kernel source files for your kernel and that they are properly configured; on Red Hat Linux systems, for example, be sure you have the 'kernel-source' or 'kernel-devel' RPM installed. If you know the correct kernel source files are installed, you may specify the kernel source path with the '--kernel-source-path' command line option.	You need to have all of the kernel source files (including kernel, kernel-headers, and kernel-devel) installed and ensure that they are of the same version
      ERROR: The Nouveau kernel driver is currently in use by your system. This driver is incompatible with the NVIDIA driver, and must be disabled before proceeding. Please consult the ow to correctly disable the Nouveau kernel driver.	You have to disable the Nouveau kernel driver
      ERROR: Failed to find dkms on the system! ERROR: Failed to install the kernel module through DKMS. No kernel module was installed; please try installing again without DKMS, or check the DKMS logs for more information.	You need to install DKMS, which helps maintain out-of-tree drivers by automatically regenerating new modules when the kernel version changes
      ERROR: Unable to load the kernel module 'nvidia.ko'. This happens most frequently when this kernel module was built against the wrong or improperly configured kernel sources, with a version of gcc that differs from the one used to build the target kernel, or if a driver such as rivafb, nvidiafb, or nouveau is present and prevents the NVIDIA kernel module from obtaining ownership of the NVIDIA graphics device(s), or no NVIDIA GPU installed in this system is supported by this NVIDIA Linux graphics driver release.	Just run the commands ./NVIDIA-Linux-x86_64-384.98.run --kernel-source-path=/usr/src/kernels/3.10.0-XXX.x86_64/ -k $(uname -r)

   5. Check whether the installation is successful:
      Respectively run the following two commands to check whether the installation is successful. If GPU information such as model is displayed in the command output, the driver has been installed successfully.

      # lspci |grep NVIDIA
      # nvidia-smi

   6. Install the CUDA Toolkit:
      Download CUDA Toolkit installation package and upload this package to the VM system. Run the following commands to execute the driver script:

      # chmod +x cuda_8.0.61_375.26_linux.run      # Configure executable permissions
      # ./cuda_8.0.61_375.26_linux.run     # Execute the driver script 

      During the installation, please set the following parameters:

      
      
      

   7. Configure the environment variables:
      Run the vim /root/.bashrc command and save the content below to the same file:

      #gpu driver
      export CUDA_HOME=/usr/local/cuda-8.0
      export PATH=/usr/local/cuda-8.0/bin:$PATH
      export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64:$LD_LIBRARY_PATH
      export LD_LIBRARY_PATH="/usr/local/cuda-8.0/lib:${LD_LIBRARY_PATH}"

      Environment variables will take effect once added. To verify the effect, you can run the following commands:

      # source ~/.bashrc
      # cd /usr/local/cuda-8.0/samples/1_Utilities/deviceQuery
      # make
      # ./deviceQuery

5. Install GuestTools.
   To capture real-time data on GPU load monitor, GuestTools are required for the VM instance. The procedures of installing GuestTools vary depending on the OS of your VM instance.

   • For Linux VM Instance
     1. Enter the VM instance details page and find GuestTools on the top row.
     2. Attach ISO.
     3. Launch VM console and run the following commands:

        # Create a mount point.
        mkdir /mnt/cdrom
        # Mount the CD-ROM image.
        mount /dev/cdrom /mnt/cdrom
        # Install GuestTools.
        cd /mnt/cdrom/
        bash ./zs-tools-install.sh
        # Unmount the CD-ROM image(Optional)
        cd ~
        umount /mnt/cdrom

        Note:

        • The commands above can be directly copied to VM console.
        • Before you install GuestTools, ensure that you have installed Linux command-line tools, for example, tar, wget, curl.
        • If you install GuestTools for OpenEuler VM, you need to disable selinux. Otherwise, the QGA feature may be affected.

        
        
        
        
        

   • For Windows VM Instance
     1. Enter the VM instance detail page and find GuestTools on the top row.
     2. Install ISO.
     3. Launch VM console and follow the steps to install GuestTools.