CUDA is not just one thing you install. It is a layered software stack that starts with the NVIDIA driver in the Linux kernel and climbs toward the tools you use to build and run GPU programs. This note goes from the bottom of that stack to the top, showing what each layer does and where it fits.

Create SSH key

Create key:

ssh-keygen

View public key:

cat ~/.ssh/id_ed25519.pub

Update ~/.ssh/config like:

Host l4
    HostName 161.104.48.116
    User root

Create VM

Install docker:

# From https://docs.docker.com/engine/install/ubuntu remove old
sudo apt remove $(dpkg --get-selections docker.io docker-compose docker-compose-v2 docker-doc podman-docker containerd runc | cut -f1)
# Update registry
sudo apt update
sudo apt install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")
Components: stable
Signed-By: /etc/apt/keyrings/docker.asc
EOF
sudo apt update
# Install latest version
sudo apt install --yes docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
# Check status
sudo systemctl status docker
# If not started starts with
sudo systemctl start docker
# Check that works
sudo docker run hello-world

You might have an error current user does not have permission to run docker. Solve this by adding current user to docker group:

sudo usermod -aG docker $USER

Reboot and check access again:

sudo reboot

NVIDIA Kernel Module

Part of the NVIDIA driver that runs inside the Linux kernel. It translates requests from user space CUDA libraries into actual GPU operations. Normal Linux process can not access PCIe device registers or program DMA engines for example. Driver talks to the GPU via PCIe. You can find drivers here:

sudo apt update
sudo apt install -y git linux-headers-$(uname -r) dkms build-essential
sudo apt install -y nvidia-driver-580

Reboot system to make driver work:

sudo reboot

Make sure that works:

nvidia-smi

Install docker support:

# To make it work with GPUs from https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
sudo apt-get update && sudo apt-get install -y --no-install-recommends \
   curl \
   gnupg2
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.18.1-1
sudo apt-get install -y \
    nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}
sudo systemctl restart docker

Validate:

sudo docker run --gpus all ubuntu:24.04 nvidia-smi

CUDA Driver API

The CUDA Driver API is the low-level user-space interface to the NVIDIA driver. Its main shared library is libcuda.so, which is installed by the NVIDIA driver package, not by the CUDA Toolkit. The same driver package also installs tools such as nvidia-smi, which reports the installed driver version and the maximum CUDA version supported by that driver.

The driver is responsible for GPU context management, memory management, scheduling, module loading, and PTX JIT compilation. Applications can call the Driver API directly, but most CUDA programs use the higher-level CUDA Runtime instead.

The minimum driver version depends on the GPU and the CUDA features you need. An old driver may not be able to JIT compile PTX for a newer GPU architecture, because that architecture did not exist when the driver was released.

Newer drivers can generally run applications built with older CUDA versions, but older drivers cannot necessarily run applications built against newer CUDA versions. In practice, production systems usually pin a recent stable driver instead of upgrading constantly, because driver changes can affect CUDA libraries such as NCCL.

CUDA Runtime

The CUDA Runtime is the higher-level user-space CUDA interface that most applications use. It is provided by the libcudart.so shared library and exposes familiar APIs such as cudaMalloc, cudaMemcpy, stream management, events, and CUDA kernel launches. Internally, it uses the CUDA Driver API, but hides much of the context and module-management boilerplate.

An application built with CUDA 12.4 can usually run with the CUDA 12.8 runtime, because CUDA generally preserves compatibility within the same major version. Across major versions, for example from CUDA 12 to CUDA 13, you should not assume compatibility. The reverse direction is not guaranteed either: an application built with CUDA 12.8 may fail with the CUDA 12.4 runtime if it uses symbols, APIs, or behavior that did not exist yet in 12.4.

The runtime can be installed from the CUDA Toolkit, system packages, or provided by a container image such as nvidia/cuda:12.8.0-runtime-ubuntu24.04.

NVCC

Compiler driver for CUDA code that can be installed from CUDA Toolkit. Allows to compile CUDA C++ files such as .cu files. Version determines what CUDA language features, PTX instructions, and GPU architectures you can target during compilation. NVCC 12.4 compiles for CUDA 12.4 runtime. Best to compile using version you will be using in runtime.

You can compile for newer runtime with older nvcc as long as the CUDA Toolkit version is compatible. The GPU you compile on does not have to match the GPU you run on.

You compile for a GPU architecture, not a driver version. The runtime machine needs a driver new enough to understand and run the code produced by your nvcc.

SASS is the final machine code that actually runs on a specific GPU architecture. If you have SASS for the exact GPU architecture, startup is faster because the driver can load it directly. The downside is that SASS is not portable. SASS for H100 will not run on T4.

PTX is intermediate representation or virtual ISA. It is not tied to one exact GPU model. You can think of it as virtual assembly language for GPUs. PTX is a portable intermediate form. It is not the final code the GPU executes. The driver can JIT compile PTX into SASS for the GPU that is actually present at runtime. This is useful when you want the same binary to survive newer GPUs, because a newer driver may be able to compile the PTX for a GPU that did not exist when you built the program. PTX gives you forward compatibility, but not magic access to future hardware features. PTX only contains operations and assumptions known to the CUDA Toolkit you used at build time. So it cannot express brand new features that did not exist yet.

When you compile with nvcc it can put both into the binary.

Validate:

docker run --gpus all nvidia/cuda:12.8.1-devel-ubuntu24.04 nvcc --version

With driver installed on OS level and different docker containers running different CUDA Toolkit versions you can play around enough.

Next, create .devcontainer/l4/devcontainer.json with:

{
  "name": "l4",
  "image": "nvidia/cuda:12.8.1-devel-ubuntu24.04",
  "workspaceFolder": "/mnt",
  "workspaceMount": "source=${localEnv:HOME},target=/mnt,type=bind,consistency=cached",
  "customizations": {
    "vscode": {
      "extensions": [
        "ms-python.python"
      ]
    }
  },
  "runArgs": [
    "--gpus", "all",
    "--mount", "type=bind,source=${localEnv:HOME}/.bashrc,target=/root/.bashrc,readonly",
    "--cap-add", "SYS_ADMIN"
  ],
  "postCreateCommand": "git config --global --add safe.directory '*'"
}

cuBLAS

cuBLAS is NVIDIA's CUDA implementation of BLAS: a library of optimized GPU kernels for linear algebra. It includes vector operations, matrix-vector operations, and matrix multiplication, with GEMM being the routine most people encounter through ML frameworks. In practice, you often use cuBLAS indirectly through PyTorch, TensorFlow, JAX, or other libraries, but you can also call it directly from CUDA code.

cuda-gdb

NVIDIA provides cuda-gdb, an official debugger for CUDA programs. It is based on GDB, but understands CUDA kernels, GPU threads, blocks, and device memory, so you can set breakpoints and inspect values inside GPU code. Use it for correctness bugs, not performance profiling: illegal memory accesses, wrong intermediate values, or control-flow behavior that only appears on the GPU.

Compile with debug information before running it:

nvcc -g -G

-g emits host debug information, while -G emits device debug information and disables most device optimizations. Do not use -G for profiling, because it changes the generated GPU code.

NVIDIA Nsight Systems

Nsight Systems is NVIDIA's system-wide performance profiler. It shows a timeline of the whole application: CPU threads, CUDA API calls, kernel launches, memory copies, GPU work, synchronization, and other runtime events. You use it to see where time goes across the program and whether the GPU is waiting on the CPU, data transfers, or synchronization. In CUDA programs, it is especially useful for finding CPU launch overhead, gaps between kernels, synchronization stalls, host-to-device and device-to-host copies, and poor overlap between CPU and GPU work or between copy and compute.

NVIDIA Nsight Compute

Nsight Compute is NVIDIA's kernel-level profiler for CUDA programs. It helps inspect occupancy, memory throughput, instruction mix, warp stalls, launch configuration, and source-level metrics when they are available.

Nsight Compute CLI is included in the CUDA Toolkit:

docker run --gpus all nvidia/cuda:12.8.1-devel-ubuntu24.04 ncu -v

Suppose:

import torch


n = 1024
steps = 8
device = 0

for _ in range(steps):
    a = torch.rand(n, n, dtype=torch.float16, device=device)
    b = torch.rand(n, n, dtype=torch.float16, device=device)
    a @ b

Performance counters are enabled. The command below writes a report to /mnt/profile.ncu-rep, overwrites an existing report with -f, filters kernels with -k, skips the first 4 matching launches with -s 4, profiles the next 2 with -c 2, and collects the full metric set with --set full.

docker run \
  --gpus all \
  --cap-add SYS_ADMIN \
  -v /mnt:/mnt \
  nvidia/cuda:12.8.1-devel-ubuntu24.04 \
  bash -lc '
    apt-get update &&
    apt-get install -y python3 python3-pip &&
    pip install torch --break-system-packages &&
    ncu \
      -o /mnt/profile \
      -f \
      -k "regex:.*fp16_.*" \
      -s 4 \
      -c 2 \
      --set full \
      python3 /mnt/example.py
  '

Copy the report from the remote machine to your local machine:

scp root@l4:/mnt/profile.ncu-rep ~/Documents/ncu/profile.ncu-rep

Open the report in NVIDIA Nsight Compute:

open -a "NVIDIA Nsight Compute" ~/Documents/ncu/profile.ncu-rep