The first time this happened to me, I assumed the provider had sold me a broken box.
I’d spun up an instance, waited the ninety seconds, SSH’d in, and typed nvidia-smi the way you do. Muscle memory. Back came:
| nvidia-smi: command not found |
I opened a support ticket about it. I actually did. Halfway through composing a polite paragraph about how the GPU I was paying for by the hour did not appear to exist, I deleted the draft, because a former colleague’s voice at the back of my skull asked whether I had installed the driver.
I had not installed the driver.
That’s most people’s story. nvidia-smi command not found on a fresh cloud server almost never means broken hardware. It means the image shipped clean. Providers hand you plain Ubuntu with no NVIDIA driver baked in, because the right driver depends on the card you rented and the framework you’re about to run. The decision stays yours. The price is that you make it.
So let’s make it. Below is how I install cuda nvidia drivers cloud gpu workloads depend on, in the order that prevents the silent failures. Fifteen minutes if nothing fights back.
Also Read: How to Right-Size GPU Instances and Stop Overpaying in 2026
If you’ve done this before
Ubuntu 24.04, CUDA 13.2, PyTorch. Run these in order, reboot where marked, skip the rest. I keep a version in a gist and paste it maybe twice a month.
| # 1. Update, install the driver, reboot (your SSH session will drop) sudo apt update && sudo apt upgrade -y sudo ubuntu-drivers autoinstall sudo reboot # 2. Reconnect, confirm the driver loaded nvidia-smi # 3. CUDA Toolkit 13.2 wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb sudo dpkg -i cuda-keyring_1.1-1_all.deb sudo apt update && sudo apt install cuda-toolkit-13-2 -y # 4. PATH echo ‘export PATH=/usr/local/cuda/bin:$PATH’ >> ~/.bashrc echo ‘export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH’ >> ~/.bashrc source ~/.bashrc && nvcc –version # 5. cuDNN sudo apt install cudnn9-cuda-13 -y # 6. PyTorch, matched to the toolkit above pip install torch –index-url https://download.pytorch.org/whl/cu132 python -c “import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))” |
People are bitten by two things. Any cu132 toolkit requires the driver branch to have 580 or newer “The version of cu132 has to be compatible with the toolkit you installed, not the one you intended to install”
Everyone else, from the top.
Why this one step trips up so many people
Before a tensor can be processed in the GPU, three different software programs must agree.
The driver helps the kernel communicate with the card. CUDA Toolkit, it gives your code a compiler and a runtime. The PyTorch or TensorFlow version an individual created as a wheel against a particular CUDA release months ago and shipped and the CUDA build of your framework.
Install CUDA 13.2 while your PyTorch wheel expects 12.6, and nothing complains. No warning. No traceback. You call torch..cuda. is_available () returns False, and that’s the entire diagnostic.
The silence is what gets you.
Most Linux failures are loud. A missing package throws a dependency error you can read. A bad path throws “command not found.” A cuda version mismatch error hands you a boolean and walks off. I once watched a good engineer lose most of a Thursday to it, reinstalling the toolkit in three different orders, certain the problem lived in CUDA, when the fix was five characters in a pip URL.
Which is why order matters here more than in most setup guides. Driver, toolkit, cuDNN, framework, with a check after each one so a break announces itself immediately rather than forty minutes later, three layers from the cause, blaming the hardware again.
Also Read: Hidden Cloud GPU Costs That Are Secretly Draining Your Budget in 2026
Before you start, check the GPU and the OS
A working nvidia driver cloud server setup starts before you install anything. Two commands, ten seconds. Skipping them has cost me far more time than running them.
First, can the OS see the card:
| lspci | grep -i nvidia |
One line per GPU, naming the card. H100, A100, L40S, RTX 4090, whatever you provisioned. If nothing prints, stop. No driver will fix it, because the card isn’t reaching your instance. Open a ticket. This is the one case where blaming the provider is correct, and it took me a while to learn to check first.
Second, what are you running:
| lsb_release -a |
Everything below assumes Ubuntu, since Hostrunway and most GPU providers default to 22.04 or 24.04. Debian is near enough identical. On RHEL or Rocky, swap apt for dnf and take the repository path from NVIDIA’s docs.
Also Read: Cloud GPU Availability in 2026: Which GPUs Are Easy to Get Right Now?
Optional: clear out old driver conflicts
Skip this if the server is fresh.
If you’re re-imaging, or an earlier attempt died partway, old NVIDIA packages block the new install in ways that produce baffling errors:
| sudo apt-get purge nvidia* -ysudo apt-get autoremove -y |
The first removes every nvidia package, the second sweeps up orphans. Insurance, not a step.
Also Read: Cloud GPU for Beginners: Complete Step-by-Step Guide 2026
Step 1: update the system first
| sudo apt update && sudo apt upgrade -y |
apt update refreshes the catalogue, apt upgrade -y installs newer versions of what’s already there, and the && means the upgrade only fires if the update succeeded.
Two to five minutes. Let it finish. I once got impatient and hit Ctrl+C halfway, which left a lock file I then spent ten minutes clearing by hand. Net saving: negative eight minutes.
Also Read: Cloud GPU for AI Inference vs Training: Different Needs Explained
Step 2: install the NVIDIA driver
Two routes. Take the first unless you have a reason not to.
Automatic:
| sudo ubuntu-drivers autoinstall |
This looks at the card and pulls the stable driver Ubuntu considers correct for it. Watch the output; it should settle on something like nvidia-driver-580. That one command is how to install nvidia drivers ubuntu recommends for its own users, and it handles the overwhelming majority of GPU cloud cases without argument.
Manual, when something pins you to a branch:
| sudo apt install nvidia-driver-580 |
One number to keep in your head: any CUDA 13.x toolkit needs a driver of release 580 or newer. Older 570 and 550 drivers install perfectly happily, then refuse to run CUDA 13 code.
Reboot. You have to reboot. The kernel loads the driver module at boot and only at boot. Skipping this is the most common reason people finish an install and still get “driver not found.” I’m putting it in a box because I have personally ignored it.
| sudo reboot |
Your session drops. Wait thirty seconds, reconnect. On PuTTY the window reports a connection loss and looks alarming; just reopen the saved session.
Also Read:
Step 3: verify the driver actually loaded
Run the command that failed at the top of this article:
| nvidia-smi |
This time you get a table: GPU name (NVIDIA H100 80GB HBM3), driver version (580.65.06), CUDA version top right (CUDA Version: 13.2), memory (0MiB / 81920MiB on an idle card), utilisation at 0%. Copy it somewhere; you’ll want that driver number shortly.
The confusing thing for almost everyone (including me for an embarrassingly long while): the CUDA version there isn’t the toolkit you have installed. It’s the highest release this driver can support. You can see CUDA Version: 13.2 there with no toolkit installed at all. A ceiling, not an inventory. People declare victory at this screen and then wonder why nvcc doesn’t exist.
GPU not detected Linux fix: read the kernel log
If nvidia-smi still fails, stop guessing and read what the kernel says:
| dmesg | grep -i nvidia |
The right gpu not detected linux fix depends on which line comes back:
- Nouveau blocking the NVIDIA module. The open source driver grabbed the card first. Blacklist Nouveau, rebuild the initramfs, reboot.
- No NVIDIA GPUs detected. Hardware, not software. Back to your provider.
- Driver initialization failed. Reboot once more, and if it survives that, try a different branch.
Once nvidia-smi prints your GPU, the hard part is done.
Step 4: install the CUDA Toolkit
Stop before you type anything. Find out which CUDA version your framework wants.
For PyTorch, read the CUDA options on the Get Started page. At the time of writing, PyTorch 2.13 ships wheels for CUDA 12.6, 13.0 and 13.2. TensorFlow publishes a tested build configurations table.
Match it. Don’t reach for the newest toolkit out of habit; I’ve done it, it feels productive, and it’s how you end up back at False. New CUDA releases routinely land weeks ahead of the wheels supporting them.
The cuda toolkit installation linux users need is four commands, here targeting CUDA 13.2 on Ubuntu 24.04:
| wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.debsudo dpkg -i cuda-keyring_1.1-1_all.debsudo apt updatesudo apt install cuda-toolkit-13-2 |
wget fetches NVIDIA’s repository keyring, dpkg -i registers it so apt trusts the packages, then you refresh and pull the toolkit. On Ubuntu 22.04, change ubuntu2404 to ubuntu2204.
The toolkit lands in /usr/local/cuda-13.2 with a /usr/local/cuda symlink, and your shell knows nothing about it:
| nano ~/.bashrc |
Scroll to the bottom, add two lines:
| export PATH=/usr/local/cuda/bin:$PATH export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH |
Ctrl+O to save, Ctrl+X to exit, then source ~/.bashrc.
The first puts nvcc and friends on your path. The second tells the linker where the shared libraries live, which prevents “cannot open shared object file” a week from now. Point both at the symlink, not the versioned folder, and future you won’t have to remember to edit them.
Step 5: verify the CUDA installation
| nvcc –version |
A working install prints a short block ending in something like Cuda compilation tools, release 13.2. That number needs to match what you installed and what your framework expects. Three-way agreement in one line, which is about as good as this process gets.
If you get command not found, the PATH export didn’t take. Check both lines really landed at the bottom of ~/.bashrc; running nano from the wrong directory and editing a different file is a classic, and I say that as someone who has done it. Run source ~/.bashrc again and retry. Still nothing? Scroll back through the apt install output, because a half-finished toolkit install looks exactly like this.
Step 6: install cuDNN
Quick definition, because the naming does nobody any favours. CUDA is the general layer: how any program talks to the GPU. cuDNN sits on top with hand-tuned routines for the operations neural networks repeat millions of times. Without it you get an outright error, or a model that trains far slower than the hardware you’re renting can manage.
The package name changed with cuDNN 9 and now tracks your CUDA major version:
| sudo apt install cudnn9-cuda-13 |
That brings in the runtime library and the development headers. On a CUDA 12.x toolkit, use cudnn9-cuda-12. Older guides, and plenty still rank, tell you to install libcudnn8. It won’t resolve against a CUDA 13 repository. If you’re pasting libcudnn8 in 2026, the tutorial you’re reading is stale.
No verification command here. It installs cleanly or apt says why not.
Step 7: test the whole stack together
Build a clean environment, so you’re testing the stack and not whatever an earlier project left behind:
| conda create -n gpu-test python=3.12conda activate gpu-testpip install torch –index-url https://download.pytorch.org/whl/cu132 |
That cu132 is the whole ballgame. It tells pip to fetch the build compiled against CUDA 13.2. Installed 13.0? cu130. Still on 12.6? cu126. Get this tag wrong and everything installs beautifully and nothing works, which is the most annoying failure mode in software.
Open Python, run four lines:
| import torchprint(torch.cuda.is_available())print(torch.cuda.get_device_name(0))print(torch.backends.cudnn.version()) |
You want True, your GPU name, and a cuDNN build number. Keep that third line. Plenty of people confirm CUDA works, celebrate, then hit cuDNN errors twenty minutes into a run because they never checked the layer above it.
If line two returns False, work in order rather than thrashing. Run nvcc –version to see which CUDA is genuinely installed, reinstall torch with the index URL matching that number, then run nvidia-smi again, because a kernel upgrade since your reboot can quietly unload the driver.
Also Read: Cloud GPU for AI Video Generation in 2026: VRAM Requirements Your Desktop Can’t Meet
The Docker alternative, and when it beats all of the above
Everything so far installs CUDA onto the host. Plenty of teams never do that, and for some workloads they’re right not to.
The other route is the NVIDIA Container Toolkit. You install the driver on the host and stop. Framework images arrive with their own CUDA and cuDNN inside them, and the container runtime hands them the GPU:
| sudo apt install -y nvidia-container-toolkitsudo nvidia-ctk runtime configure –runtime=dockersudo systemctl restart dockerdocker run –rm –gpus all nvidia/cuda:13.0.0-base-ubuntu24.04 nvidia-smi |
If that last line prints the same table you saw earlier, containers can reach the card. And if an old guide tells you to use the nvidia-docker wrapper, don’t. That project was archived and replaced by Container Toolkit; it survives in posts nobody has updated.
Containers win when projects with conflicting CUDA requirements share one server, when instances get rebuilt constantly, or when the environment has to reproduce exactly elsewhere. Host install wins when one long-lived project owns the box, or when you’re compiling custom kernels and want nvcc right there. I’ve gone back and forth and I don’t think there’s a universal answer.
One thing containers won’t save you from: the host driver. An image carrying CUDA 13.2 still needs a host driver that supports it. Docker moves the toolkit and cuDNN out of your hair. It doesn’t move the driver.
Common errors and what actually fixes them
| Error message | What caused it | The fix |
| nvidia-smi command not found | Driver never installed, or installed without a reboot | Reboot. If it survives that, run dmesg | grep -i nvidia, read the error, try a different driver branch |
| CUDA driver version is insufficient for CUDA runtime version | Toolkit newer than the driver supports. Classically CUDA 13.x on a 570 driver | Upgrade the driver to 580 or later, or drop to a toolkit the driver covers |
| torch.cuda.is_available() returns False | PyTorch wheel compiled against a different CUDA than you installed | Reinstall with the matching index URL. CUDA 13.2 means –index-url https://download.pytorch.org/whl/cu132 |
| NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver | Kernel module not loaded | Reboot. If it repeats, run lsmod | grep nouveau. Any output means Nouveau has the card: blacklist it, rebuild initramfs, reinstall |
| cuDNN not found during training | cuDNN missing, or built for the wrong CUDA major version | Run dpkg -l | grep cudnn and check the package matches your toolkit. Reinstall if not |
A pattern runs through all five, and noticing it saves more time than memorising the table. Each is either a version relationship or a load-order problem, and each is diagnosable in one command. When something breaks, resist the urge to reinstall everything. Find the failing layer first.
Also Read: Cloud GPU for Running LLMs in 2026: When Your Local GPU Isn’t Enough
Getting more out of the setup
Watch the GPU while you train. Second terminal, alongside the job:
| watch -n 1 nvidia-smi |
Healthy training pushes utilisation past 80% and memory climbs as the model and batches load. If utilisation sits near zero while your script cheerfully prints loss values, your code is on CPU and you haven’t noticed. This habit has caught more wasted GPU hours for me than everything else on this page combined.
Clear cached memory between runs. PyTorch caches allocations hard, which is usually right and occasionally why run three of an identical script dies out of memory:
| torch.cuda.empty_cache() |
Be careful with future upgrades. A routine apt upgrade can pull a new kernel, and the NVIDIA module is compiled against the running one. Driver and kernel move together or not at all. I learned this the way everyone does.
Use every card you’re paying for. With more than one GPU, model = torch.nn.DataParallel(model) spreads a job across all of them. Fine to start with. For serious multi-GPU work DistributedDataParallel is much better, at the cost of more setup.
Where Hostrunway fits
Every step above assumed you could install whatever you wanted. That assumption does a lot of quiet work.
Hostrunway GPU cloud servers come with full root access, so you install the exact driver branch, toolkit version and cuDNN build your project needs instead of inheriting an image somebody else chose. If your code is pinned to CUDA 12.6 because a dependency hasn’t caught up, you install CUDA 12.6 and nobody argues.
The network spans 160+ global locations, which matters when inference response times affect your users, or when data residency rules keep a workload in one region. Support is staffed 24/7 by people rather than a chatbot, and having been on the wrong end of a driver install at an unreasonable hour, I’d call that the difference between a ten minute problem and a lost night. New cards join the fleet as they ship, so you rent current hardware rather than buying silicon that loses value each quarter.
Once torch.cuda.is_available() prints True, the setup is over and the real work starts. View Hostrunway GPU Cloud Plans to pick the card and region that suit your workload.
Frequently asked questions
What CUDA version should I install for my GPU server?
Your framework decides, not your GPU. Read which CUDA builds ship with the current PyTorch release, or TensorFlow’s tested build configurations table, and install the toolkit that matches. Reaching for the newest CUDA causes most mismatches I’ve seen, because wheels lag toolkits by weeks.
Do I need to install cuDNN separately from CUDA?
Yes. CUDA is the general communication layer between your code and the GPU. cuDNN is a specialised library of deep learning routines on top. PyTorch and TensorFlow need both. The toolkit alone leaves you with a GPU that computes but trains badly.
Why does nvidia-smi work but torch.cuda.is_available() still returns False?
They check different layers. nvidia-smi confirms the driver loaded and the card responds; it knows nothing about your toolkit or Python environment. torch.cuda.is_available() asks whether PyTorch’s CUDA build can find a matching runtime. False alongside a working nvidia-smi is almost always the wrong wheel.
Do I need to reboot after installing the NVIDIA driver?
Always, and this is the least negotiable line in the article. The kernel loads the driver module during boot and there’s no way around it on a fresh install. Every “I installed the driver and nvidia-smi still fails” thread online ends the same way: somebody reboots and feels silly.
Can I upgrade CUDA later without rebuilding the server?
Yes, with conditions. Remove the old toolkit, install the new one, and the driver can usually stay put provided it supports the version you’re moving to. CUDA 13.x needs driver 580 or newer. Then reinstall your framework with the matching build; the old wheel points at the old runtime.
How do I know if my CUDA and driver versions are compatible?
NVIDIA publishes a compatibility table listing minimum driver versions per toolkit. The rule underneath is easy to memorise: newer drivers support older toolkits, never the reverse. A 580 driver runs CUDA 12.x and 13.x. A 550 driver won’t touch CUDA 13. Unsure? Newer driver, older toolkit.
What’s the difference between the driver, CUDA Toolkit, and cuDNN?
Think of a road network. The driver is the road connecting your code to the hardware; without it nothing moves. CUDA is the highway system on top. cuDNN is a set of express lanes for deep learning traffic. All three are required.
Can I install multiple CUDA versions on the same server?
Technically yes. Toolkits install into versioned directories like /usr/local/cuda-12.6 and /usr/local/cuda-13.2, so they coexist. What can’t is your PATH, which points at one at a time, and it fails quietly rather than loudly. Unless you maintain several codebases with hard pins, run one toolkit.
Why does nvidia-smi say “NVIDIA-SMI has failed” after installation?
The kernel module didn’t load. Reboot first, which fixes it most of the time. If the message outlives a reboot, run dmesg | grep -i nvidia and read what the kernel says. Usual suspects: Nouveau holding the card, a kernel update the driver wasn’t rebuilt against, or a half-finished install.
Do I need CUDA if I’m only running inference, not training?
Yes. Inference runs the same matrix operations as training minus the backward pass, and needs CUDA to reach the GPU. Without it PyTorch falls back to CPU without saying much and speed collapses. cuDNN is closer to optional here, but its convolution kernels matter for most vision models.
