Appendix III: Installing GPU Drivers

GPU Driver Installation

This section describes how to install an NVIDIA GPU driver.

1. Disable the nouveau driver.

   echo 'blacklist nouveau
   options nouveau modeset=0' > /etc/modprobe.d/nvidia-installer-disable-nouveau.conf

2. Install kernel dependencies.

   yum install -y kernel-devel-$(uname -r) kernel-headers-$(uname -r)
   cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak
   dracut /boot/initramfs-$(uname -r).img $(uname -r) --force
   systemctl set-default multi-user.target
   init 3

3. Restart the server.

   reboot

4. Verify that the nouveau driver is not in use.

   No output indicates that the driver is not in use.

   lsmod | grep nouveau

5. Install dependencies.

   Select the appropriate installation method for the CPU architecture.

   # x86 architecture
   yum install -y elfutils-libelf-devel
   
   # ARM architecture
   yum install -y gcc-c++ make elfutils-libelf-devel

6. Install the driver.

   The following example installs a driver for an NVIDIA T4 GPU on an x86 server.

   x86 driver file (or use a later officially supported driver version): http://oss.zstack.io/solutions/edge/zstack-zaku/gpu-driver/NVIDIA-Linux-x86_64-550.90.07.run

   ARM driver file (or use a later officially supported driver version): http://oss.zstack.io/solutions/edge/zstack-zaku/gpu-driver/Ascend-hdk-310p-npu-driver_24.1.rc2_linux-aarch64.run

   bash NVIDIA-Linux-x86_64-550.90.07.run

7. After installing the driver, view the GPU information.

   nvidia-smi

Converting a Compute Node to a GPU Node

Skip this section if GPU node was selected when the business cluster was created or the node was added.

This section applies only when GPU node was not selected as the compute node joined the business cluster:

• The GPU node role was omitted when creating the cluster or adding the node.

• A GPU was added to an existing compute node after it had been running for some time.

1. Install the GPU driver as described in NVIDIA GPU > GPU Driver Installation in this appendix.

   Details omitted.

2. Install the toolkit.

   yum install -y nvidia-container-toolkit

3. Change the default runtime for containerd.

   Edit /etc/containerd/config.toml, find default_runtime_name, and change it to the following value:

   default_runtime_name = "nvidia"

4. Restart the container service.

   systemctl daemon-reload
   systemctl restart containerd

5. Add node labels.

   kubectl label node #NODE_NAME# node-role.kubernetes.io/gpu= --overwrite=true
   kubectl label node #NODE_NAME# nvidia.com/gpu=true --overwrite=true
   kubectl label node #NODE_NAME# gpu=on --overwrite=true

NPU Driver Installation

This section describes how to install a huawei NPU driver.

1. Create runtime users.

   huawei NPUs use user IDs and group IDs 1000 and 9000. If these IDs are already in use, adjust them as described in this document: https://www.hiascend.com/document/caselibrary/detail/topic_0000001591705097

   groupadd -g 1000 HwHiAiUser
   useradd -g HwHiAiUser -d /home/HwHiAiUser -u 1000 -m HwHiAiUser -s /bin/bash
   groupadd -g 9000 hwMindX
   useradd -g hwMindX -d /home/hwMindX -u 9000 -m -s /sbin/nologin hwMindX
   usermod -a -G HwHiAiUser hwMindX

2. Install dependencies.

   yum install -y net-tools make dkms gcc kernel-headers-$(uname -r) kernel-devel-$(uname -r)

3. Install the driver.

   The following example installs a driver for a 310P-series accelerator on an x86 server.

   x86 driver file (or use a later officially supported driver version): http://oss.zstack.io/solutions/edge/zstack-zaku/gpu-driver/Ascend-hdk-310p-npu-driver_24.1.rc2_linux-x86-64.run

   ARM driver file (or use a later officially supported driver version): http://oss.zstack.io/solutions/edge/zstack-zaku/gpu-driver/Ascend-hdk-310p-npu-driver_24.1.rc2_linux-aarch64.run

   chmod u+x Ascend-hdk-310p-npu-driver_24.1.rc2_linux-x86-64.run
   ./Ascend-hdk-310p-npu-driver_24.1.rc2_linux-x86-64.run --check
   ./Ascend-hdk-310p-npu-driver_24.1.rc2_linux-x86-64.run --full

   If driver installation reports a permission error for /usr/local/Ascend, run: chmod +x /usr/local/Ascend.

4. Verify driver installation.

   npu-smi info

5. Install the firmware.

   The following example uses a 310P-series accelerator.

   Firmware file (or use a later officially supported version): http://oss.zstack.io/solutions/edge/zstack-zaku/gpu-driver/Ascend-hdk-310p-npu-firmware_7.3.0.1.231.run

   chmod u+x Ascend-hdk-310p-npu-firmware_7.3.0.1.231.run
   ./Ascend-hdk-310p-npu-firmware_7.3.0.1.231.run --check
   ./Ascend-hdk-310p-npu-firmware_7.3.0.1.231.run --full

6. Restart the server.

   reboot

Converting a Compute Node to a GPU Node

Skip this section if GPU node was selected when the business cluster was created or the node was added.

This section applies only when GPU node was not selected as the compute node joined the business cluster:

• The GPU node role was omitted when creating the cluster.

• A GPU was added to an existing compute node after it had been running for some time.

1. Install the NPU driver as described in huawei NPU > NPU Driver Installation in this appendix.

   Details omitted.

2. Disable vNPU configuration recovery.

   npu-smi set -t vnpu-cfg-recover -d 0

3. Install Ascend-docker-runtime.

   Select the appropriate installation method for the CPU architecture.

   # x86 environment
   mkdir -p /opt/zstack-edge/copy
   cd /opt/zstack-edge/copy
   wget http://#BASIC_SERVICE_VIP#:7700/kubespray/release-2.20/v1.24.6/amd64/Ascend-docker-runtime_6.0.RC2_linux-x86_64.run
   chmod +x Ascend-docker-runtime_6.0.RC2_linux-x86_64.run
   ./Ascend-docker-runtime_6.0.RC2_linux-x86_64.run --install
   
   # ARM environment
   mkdir -p /opt/zstack-edge/copy
   cd /opt/zstack-edge/copy
   wget http://#BASIC_SERVICE_VIP#:7700/kubespray/release-2.20/v1.24.6/arm64/Ascend-docker-runtime_6.0.RC2_linux-aarch64.run
   chmod +x Ascend-docker-runtime_6.0.RC2_linux-aarch64.run
   ./Ascend-docker-runtime_6.0.RC2_linux-aarch64.run --install

4. Change the default runtime for containerd.

   Edit /etc/containerd/config.toml, find default_runtime_name, and change it to the following value:

   default_runtime_name = "ascend"

5. Restart containerd.

   systemctl daemon-reload
   systemctl restart containerd

6. Add node labels.

   kubectl label node #NODE_NAME# node-role.kubernetes.io/gpu= --overwrite=true
   kubectl label node #NODE_NAME# workerselector=dls-worker-node --overwrite=true
   kubectl label node #NODE_NAME# ascend=on --overwrite=true

DCU Driver Installation

1. Identify the DCU device.

   lspci -nn | grep "Display" |grep "Haiguang"

2. Install dependencies.

   yum -y install cmake3 automake gcc-c++ rpm-build kernel-devel-`uname -r` kernel-headers-`uname -r` kernel-modules-extra-`uname -r` mlocate

3. Install the driver.

   Driver file (or use a later officially supported driver version): http://oss.zstack.io/solutions/edge/zstack-zaku/gpu-driver/rock-5.7.1-6.2.26-V1.5.aio.run

   bash rock-5.7.1-6.2.26-V1.5.aio.run

4. Restart the server.

   reboot

5. Install DTK.

   DTK file (or use a later officially supported version): http://oss.zstack.io/solutions/edge/zstack-zaku/gpu-driver/DTK-24.04.3-Kylinv10SP3-x86_64.tar.gz

   tar -zxvf DTK-24.04.3-Kylinv10SP3-x86_64.tar.gz -C /opt
   ln -s /opt/dtk-24.04.3 /opt/dtk

6. Verify that installation succeeded.

   source /opt/dtk/env.sh
   hy-smi

Converting a Compute Node to a GPU Node

Skip this section if GPU node was selected when the business cluster was created or the node was added.

This section applies only when GPU node was not selected as the compute node joined the business cluster:

• The GPU node role was omitted when creating the cluster.

• A GPU was added to an existing compute node after it had been running for some time.

1. Install the DCU driver as described in HYGON DCU > DCU Driver Installation in this appendix.

   Details omitted.

2. Add node labels.

   kubectl label node #NODE_NAME# node-role.kubernetes.io/gpu= --overwrite=true
   kubectl label node #NODE_NAME# dcu=on --overwrite=true

GPU Driver Installation

1. Identify the GPU device.

   0001 identifies a Tiangai accelerator, and 0002 identifies a Zhikai accelerator. These accelerators use different drivers; select the appropriate one.

   The following steps use the Tiangai 100 as an example.

   lspci -vv | grep 1e3e

2. Install the firmware.

   Firmware file (or use a later officially supported version): http://oss.zstack.io/solutions/edge/zstack-zaku/gpu-driver/ixfw-bi_v100-5.0.8_x86_64.run

   bash ixfw-bi_v100-5.0.8_x86_64.run

3. Restart the server.

   reboot

4. Install dependencies.

   yum install -y kernel-devel-$(uname -r) kernel-headers-$(uname -r) ncurses-libs elfutils-libelf-devel

5. Install CUDA.

   CUDA file (or use a later officially supported version): http://oss.zstack.io/solutions/edge/zstack-zaku/gpu-driver/cuda_10.2.89_440.33.01_linux.run

   Minimum required version: cuda_10.2.89_440.33.01_linux.run

   During installation, select only Toolkit.

   bash cuda_10.2.89_440.33.01_linux.run

6. Install the driver.

   Driver file (or use a later officially supported driver version): http://oss.zstack.io/solutions/edge/zstack-zaku/gpu-driver/corex-installer-linux64-3.2.1_x86_64_10.2.run

   # Example: Iluvatar Tiangai 100
   bash corex-installer-linux64-3.2.1_x86_64_10.2.run --silent --driver --toolkit

7. Set environment variables.

   Edit /root/.bash_profile and append the following lines:

   PATH=$PATH:$HOME/bin:/usr/local/corex-3.2.1/bin
   LD_LIBRARY_PATH=/usr/local/corex-3.2.1/libexport LD_LIBRARY_PATH

8. Apply the environment variable settings.

   source /root/.bash_profile

9. Verify that installation succeeded.

   ixsmi

Converting a Compute Node to a GPU Node

Skip this section if GPU node was selected when the business cluster was created or the node was added.

This section applies only when GPU node was not selected as the compute node joined the business cluster:

• The GPU node role was omitted when creating the cluster.

• A GPU was added to an existing compute node after it had been running for some time.

1. Install the GPU driver as described in Iluvatar GPU > GPU Driver Installation in this appendix.

   Details omitted.

2. Add node labels.

   kubectl label node #NODE_NAME# node-role.kubernetes.io/gpu= --overwrite=true
   kubectl label node #NODE_NAME# iluvatar=on --overwrite=true