{"id":1368,"date":"2026-10-12T06:30:53","date_gmt":"2026-10-12T06:30:53","guid":{"rendered":"https:\/\/www.hostrunway.com\/blog\/?p=1368"},"modified":"2026-09-18T07:34:16","modified_gmt":"2026-09-18T07:34:16","slug":"how-to-install-docker-and-run-ai-containers-on-a-cloud-gpu-server","status":"publish","type":"post","link":"https:\/\/www.hostrunway.com\/blog\/how-to-install-docker-and-run-ai-containers-on-a-cloud-gpu-server\/","title":{"rendered":"How to install Docker and run AI containers on a cloud GPU server"},"content":{"rendered":"\n<p>Anyone who has moved a model off a laptop and onto a <a href=\"https:\/\/www.hostrunway.com\/gpu-dedicated-server.php\" title=\"\">GPU<\/a> box knows the failure. The script runs fine locally, then dies on the server with an import error, because the machine carries a slightly different PyTorch build. Patch that version and a second project breaks, because it wanted the old one. Setting up a <strong>Docker GPU server<\/strong> the right way ends this, since a container seals your model, its libraries, and the exact runtime into one image that behaves the same on any host.<\/p>\n\n\n\n<p>That is the whole idea. PyTorch 2.1, CUDA 12.1, Python 3.10, pinned together and shipped as a single unit. Move it to any Linux machine with a GPU, and it runs the way it ran on yours, not almost the same way.<\/p>\n\n\n\n<p>The GPU is where people get stuck. A fresh container cannot see the card. NVIDIA&#8217;s Container Toolkit is what changes that, and once it is installed, <strong>AI containers&#8217; GPU<\/strong> access is a single flag at runtime. Your framework finds the hardware with nothing else to wire up.<\/p>\n\n\n\n<p>This walkthrough starts on a bare Ubuntu GPU server and finishes with a container running your model on the GPU. You will <strong>install Docker, NVIDIA GPU<\/strong> support, confirm the card is visible from inside a container, and run both PyTorch and TensorFlow.<\/p>\n\n\n\n<p>Also Read: <a href=\"https:\/\/www.hostrunway.com\/blog\/how-to-run-kubernetes-on-cloud-gpu-complete-beginner-guide-2026\/\">How to Run Kubernetes on Cloud GPU \u2013 Complete Beginner Guide 2026<\/a><\/p>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_85 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.hostrunway.com\/blog\/how-to-install-docker-and-run-ai-containers-on-a-cloud-gpu-server\/#Why_Docker_for_AI_on_GPU_servers\" >Why Docker for AI on GPU servers<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.hostrunway.com\/blog\/how-to-install-docker-and-run-ai-containers-on-a-cloud-gpu-server\/#Prerequisites_and_verification\" >Prerequisites and verification<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.hostrunway.com\/blog\/how-to-install-docker-and-run-ai-containers-on-a-cloud-gpu-server\/#Step_1_Install_Docker\" >Step 1: Install Docker<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.hostrunway.com\/blog\/how-to-install-docker-and-run-ai-containers-on-a-cloud-gpu-server\/#Step_2_Install_the_NVIDIA_Container_Toolkit\" >Step 2: Install the NVIDIA Container Toolkit<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.hostrunway.com\/blog\/how-to-install-docker-and-run-ai-containers-on-a-cloud-gpu-server\/#Step_3_Pull_and_run_a_PyTorch_container\" >Step 3: Pull and run a PyTorch container<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/www.hostrunway.com\/blog\/how-to-install-docker-and-run-ai-containers-on-a-cloud-gpu-server\/#Step_4_Run_your_own_AI_model_in_a_container\" >Step 4: Run your own AI model in a container<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/www.hostrunway.com\/blog\/how-to-install-docker-and-run-ai-containers-on-a-cloud-gpu-server\/#Step_5_Docker_Compose_for_multi-container_setups\" >Step 5: Docker Compose for multi-container setups<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/www.hostrunway.com\/blog\/how-to-install-docker-and-run-ai-containers-on-a-cloud-gpu-server\/#Getting_containers_production-ready\" >Getting containers production-ready<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/www.hostrunway.com\/blog\/how-to-install-docker-and-run-ai-containers-on-a-cloud-gpu-server\/#GPU_memory\" >GPU memory<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/www.hostrunway.com\/blog\/how-to-install-docker-and-run-ai-containers-on-a-cloud-gpu-server\/#Storage_and_limits\" >Storage and limits<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/www.hostrunway.com\/blog\/how-to-install-docker-and-run-ai-containers-on-a-cloud-gpu-server\/#Security_and_isolation\" >Security and isolation<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-12\" href=\"https:\/\/www.hostrunway.com\/blog\/how-to-install-docker-and-run-ai-containers-on-a-cloud-gpu-server\/#Housekeeping\" >Housekeeping<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-13\" href=\"https:\/\/www.hostrunway.com\/blog\/how-to-install-docker-and-run-ai-containers-on-a-cloud-gpu-server\/#Troubleshooting_common_errors\" >Troubleshooting common errors<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-14\" href=\"https:\/\/www.hostrunway.com\/blog\/how-to-install-docker-and-run-ai-containers-on-a-cloud-gpu-server\/#Running_TensorFlow_containers_on_the_GPU\" >Running TensorFlow containers on the GPU<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-15\" href=\"https:\/\/www.hostrunway.com\/blog\/how-to-install-docker-and-run-ai-containers-on-a-cloud-gpu-server\/#Running_AI_at_scale_on_Hostrunway\" >Running AI at scale on Hostrunway<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-16\" href=\"https:\/\/www.hostrunway.com\/blog\/how-to-install-docker-and-run-ai-containers-on-a-cloud-gpu-server\/#Frequently_asked_questions\" >Frequently asked questions<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\" style=\"font-size:22px\"><span class=\"ez-toc-section\" id=\"Why_Docker_for_AI_on_GPU_servers\"><\/span><strong>Why Docker for AI on GPU servers<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Four things make Docker worth the setup for AI work, and the fourth is the one that actually matters on a GPU box.<\/p>\n\n\n\n<p>Start with reproducibility, since it is why most teams reach for containers at all. A model that trains cleanly on your laptop can fall over on a server running an older framework build, and chasing that mismatch burns an afternoon you will not get back. The container carries its own versions, so the environment stops being a moving part.<\/p>\n\n\n\n<p>Isolation comes next. One project needs TensorFlow 2.12, another needs 2.14, and on a shared host those two quietly overwrite each other in the same Python environment. Give each its own container and the conflict is gone. Nothing leaks across.<\/p>\n\n\n\n<p>Scaling falls out of the first two. Once a job runs in a container, running it on ten servers is the same image ten times, with no per-machine drift to debug. What passed in testing is what runs in production, down to the byte.<\/p>\n\n\n\n<p>The fourth reason is the hard one, and it is specific to <a href=\"https:\/\/www.hostrunway.com\/powerful-gpus.php\" title=\"\">GPUs<\/a>: getting the card into the container. NVIDIA&#8217;s answer, shipped in the early days as <strong>docker nvidia-docker<\/strong> tooling and now folded into the NVIDIA Container Toolkit, makes the card visible from inside. Without it, your framework reports no CUDA device and quietly drops to the CPU while you wonder why training crawls. With it, the GPU just appears.<\/p>\n\n\n\n<p>One thing this guide leaves out on purpose is writing your own Dockerfile. Here you run images that already exist, the official PyTorch and TensorFlow builds. Rolling your own is worth doing later. It is a different job.<\/p>\n\n\n\n<p>Also Read: <a href=\"https:\/\/www.hostrunway.com\/blog\/how-to-host-server-in-china-without-an-icp-license-in-2026-practical-guide-for-foreign-businesses\/\">How to Host Server in China Without an ICP License in 2026 \u2013 Practical Guide for Foreign Businesses<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" style=\"font-size:22px\"><span class=\"ez-toc-section\" id=\"Prerequisites_and_verification\"><\/span><strong>Prerequisites and verification<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Five checks before you install anything. Each takes seconds, and each heads off a specific, confusing failure down the line.<\/p>\n\n\n\n<p>The host driver comes first. A container ships CUDA but never the driver, so the driver has to already be there:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>nvidia-smi<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>A table of your GPU and its driver version means you are set. command not found means the driver is missing, and that is the thing to fix before anything else.<\/p>\n\n\n\n<p>The CUDA Toolkit on the host is optional. Containers bring their own, so you only need this for compiling or debugging outside a container:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>nvcc &#8211;version<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>If it fails, shrug and move on. Nothing here depends on it.<\/p>\n\n\n\n<p>GPU containers want Linux. Ubuntu 20.04, 22.04, and 24.04 all work well; Windows and macOS run Docker but handle GPU passthrough poorly.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>lsb_release -a<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Keep an eye on disk. AI images are heavy, roughly 5 to 8 GB for PyTorch and 4 to 6 GB for TensorFlow, so leave at least 30 GB free:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>df -h \/ | grep -v Filesystem<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Last, the server needs the internet for the first pull. A large image can take 5 to 10 minutes to download; after that it is cached and starts instantly. Clear all five and Docker is next.<\/p>\n\n\n\n<p>Also Read: <a href=\"https:\/\/www.hostrunway.com\/blog\/llama-405b-deepseek-best-gpus-compared\/\">Llama 405B vs DeepSeek: Which Cloud GPU Actually Handles Both in 2026?<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" style=\"font-size:22px\"><span class=\"ez-toc-section\" id=\"Step_1_Install_Docker\"><\/span><strong>Step 1: Install Docker<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Docker installs from its own apt repository. The steps below use Docker&#8217;s current signed-keyring method, not the apt-key approach that newer Ubuntu releases now warn about.<\/p>\n\n\n\n<p>Clear out any distro-packaged Docker first. On a clean server this does nothing, which is fine:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt-get remove -y $pkgdone<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Now add Docker&#8217;s repository and install the engine:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>sudo apt-get updatesudo apt-get install -y ca-certificates curlsudo install -m 0755 -d \/etc\/apt\/keyringssudo curl -fsSL https:\/\/download.docker.com\/linux\/ubuntu\/gpg -o \/etc\/apt\/keyrings\/docker.ascsudo chmod a+r \/etc\/apt\/keyrings\/docker.ascecho \\&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#8220;deb [arch=$(dpkg &#8211;print-architecture) signed-by=\/etc\/apt\/keyrings\/docker.asc] https:\/\/download.docker.com\/linux\/ubuntu \\ $(. \/etc\/os-release &amp;&amp; echo &#8220;$VERSION_CODENAME&#8221;) stable&#8221; | \\&nbsp;sudo tee \/etc\/apt\/sources.list.d\/docker.list &gt; \/dev\/nullsudo apt-get updatesudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Most of that block is plumbing: pull in dependencies, register Docker&#8217;s GPG key so apt trusts the packages, point apt at Docker&#8217;s repository for your Ubuntu release, then install the engine along with the CLI, containerd, and the Buildx and Compose plugins. The only line worth memorizing is the last one.<\/p>\n\n\n\n<p>Check the version:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>docker &#8211;version<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Something like Docker version 27.3.1 confirms it landed. Set the daemon to run now and on every boot:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>sudo systemctl enable &#8211;now docker<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>If you would rather not type sudo before every command, add your user to the docker group:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>sudo usermod -aG docker $USERnewgrp docker&nbsp;<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Worth knowing: group membership is close to root access, so security-minded teams skip it and keep using sudo docker. Neither choice is wrong. It is a trade between typing less and shrinking the blast radius.<\/p>\n\n\n\n<p>Finish with the smoke test everyone runs:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>docker run hello-world<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>A &#8220;<strong>Hello from Docker!<\/strong>&#8221; line means the engine works. The card is next.<\/p>\n\n\n\n<p>Also Read: <a href=\"https:\/\/www.hostrunway.com\/blog\/docker-or-bare-metal-on-cloud-gpu-how-to-choose-the-right-one-in-2026\/\">Docker or Bare Metal on Cloud GPU? How to Choose the Right One in 2026<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" style=\"font-size:22px\"><span class=\"ez-toc-section\" id=\"Step_2_Install_the_NVIDIA_Container_Toolkit\"><\/span><strong>Step 2: Install the NVIDIA Container Toolkit<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>By default, a container is blind to the GPU. Your code asks for a CUDA device, the container has nothing to hand back, and the request fails. The NVIDIA Container Toolkit is the piece that gives the container a window onto the host card. It is the modern replacement for the old nvidia-docker wrappers you still see in stale tutorials.<\/p>\n\n\n\n<p>Add NVIDIA&#8217;s repository and install the toolkit. This matches NVIDIA&#8217;s current guide:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>curl -fsSL https:\/\/nvidia.github.io\/libnvidia-container\/gpgkey |&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\\ sudo gpg &#8211;dearmor -o \/usr\/share\/keyrings\/nvidia-container-toolkit-keyring.gpgcurl -s -L https:\/\/nvidia.github.io\/libnvidia-container\/stable\/deb\/nvidia-container-toolkit.list | \\sed &#8216;s#deb https:\/\/#deb [signed-by=\/usr\/share\/keyrings\/nvidia-container-toolkit-keyring.gpg] https:\/\/#g&#8217; | \\&nbsp;sudo tee \/etc\/apt\/sources.list.d\/nvidia-container-toolkit.listsudo apt-get updatesudo apt-get install -y nvidia-container-toolkit&nbsp;<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Then hook it into Docker and restart the daemon so the change takes effect:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>sudo nvidia-ctk runtime configure &#8211;runtime=dockersudo systemctl restart docker&nbsp;<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>The nvidia-ctk command edits \/etc\/docker\/daemon.json to register the NVIDIA runtime. Skip the restart, and Docker keeps running on its old config, which is a common reason the next test fails for people who did everything else right.<\/p>\n\n\n\n<p>Now prove it. This launches a <strong>Docker CUDA container<\/strong> from NVIDIA&#8217;s official CUDA image and runs the GPU report from inside it:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>docker run &#8211;rm &#8211;gpus all nvidia\/cuda:12.1.1-runtime-ubuntu22.04 nvidia-smi<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>The &#8211;gpus all part is what hands every card to the container. If the same GPU table you saw on the host now prints from inside the container, the toolkit works and you can <strong>run GPU container<\/strong> jobs for real.<\/p>\n\n\n\n<p>The error you are most likely to meet here is could not select device driver &#8220;&#8221; with capabilities: [[gpu]]. Nine times out of ten it means the daemon was never restarted after nvidia-ctk. Run the restart again and retry. If it sticks around, confirm the package is even installed with nvidia-ctk &#8211;version.<\/p>\n\n\n\n<p>Also Read: <a href=\"https:\/\/www.hostrunway.com\/blog\/cloud-gpu-for-ai-inference-vs-training-different-needs-explained\/\">Cloud GPU for AI Inference vs Training: Different Needs Explained<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" style=\"font-size:22px\"><span class=\"ez-toc-section\" id=\"Step_3_Pull_and_run_a_PyTorch_container\"><\/span><strong>Step 3: Pull and run a PyTorch container<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>An image is the blueprint; a container is a running copy of it. For AI work, that blueprint is bulky, because a <strong>GPU Docker image<\/strong> stacks a full CUDA and cuDNN runtime on top of the framework and the base OS layers.<\/p>\n\n\n\n<p>Pull the official <strong>PyTorch Docker container<\/strong>:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>docker pull pytorch\/pytorch:2.1.0-cuda12.1-cudnn8-runtime<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>The first pull runs a few minutes on a normal connection. After that it is cached and starts at once. The tag spells out what is inside: 2.1.0 is the PyTorch version, cuda12.1 and cudnn8 are the GPU libraries it was compiled against, and runtime is the slim build. There is a devel tag too, far larger, that adds compilers and headers for building extensions. To run models, runtime is the one you want.<\/p>\n\n\n\n<p>Confirm it downloaded:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>docker images | grep pytorch<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Roughly 3 to 5 GB should show up. Open an interactive shell with the GPU attached:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>docker run &#8211;rm &#8211;gpus all -it pytorch\/pytorch:2.1.0-cuda12.1-cudnn8-runtime bash<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Two flags carry the weight here. &#8211;rm throws the container away on exit so nothing piles up, and &#8211;gpus all is what actually grants the card. The -it pair just gives you a terminal you can type into.<\/p>\n\n\n\n<p>From inside, ask PyTorch what it can see:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>python3 &#8211; &lt;&lt;&#8216;PYEOF&#8217;import torchprint(f&#8221;GPU available: {torch.cuda.is_available()}&#8221;)print(f&#8221;GPU device: {torch.cuda.get_device_name(0)}&#8221;)print(f&#8221;CUDA version: {torch.version.cuda}&#8221;)PYEOF&nbsp;<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>You want this:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>GPU available: True<br>GPU device: NVIDIA H100 80GB HBM3<br>CUDA version: 12.1<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>A False on the first line means the passthrough did not happen. Run nvidia-smi inside the container to check whether the card is visible at all, then go back to Step 2.<\/p>\n\n\n\n<p>Type exit to leave. For real work you never copy data into a container. You mount it from the host:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>docker run &#8211;rm &#8211;gpus all -v \/path\/to\/your\/data:\/data -it \\<br>&nbsp; pytorch\/pytorch:2.1.0-cuda12.1-cudnn8-runtime bash<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>The -v flag maps a host folder onto a path inside the container. Your data and weights stay on the host at \/data, the container reads and writes them there, and then the container disappears. Treat it as disposable and keep everything you care about outside it.<\/p>\n\n\n\n<p>Also Read: <a href=\"https:\/\/www.hostrunway.com\/blog\/best-gpu-for-running-local-llms-and-private-ai-in-2026-complete-buyers-guide-ollama-lm-studio-llama-cpp\/\">Best GPU for Running Local LLMs and Private AI in 2026: Complete Buyer\u2019s Guide (Ollama, LM Studio &amp; llama.cpp)<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" style=\"font-size:22px\"><span class=\"ez-toc-section\" id=\"Step_4_Run_your_own_AI_model_in_a_container\"><\/span><strong>Step 4: Run your own AI model in a container<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Interactive shells are for poking around. Real jobs run a script, unattended, so the pattern shifts: drop the script on the host, mount it, and let the container run it.<\/p>\n\n\n\n<p>Save this as train.py. It is deliberately trivial, a stand-in that proves the GPU is doing the work:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>import torch<br>import torch.nn as nn<br><br>print(f&#8221;GPU available: {torch.cuda.is_available()}&#8221;) device = torch.device(&#8220;cuda&#8221; if torch.cuda.is_available() else &#8220;cpu&#8221;) print(f&#8221;Using device: {device}&#8221;)<br><br>model = nn.Sequential(<br>&nbsp; &nbsp; nn.Linear(10, 64),<br>&nbsp; &nbsp; nn.ReLU(),<br>&nbsp; &nbsp; nn.Linear(64, 1),<br>).to(device)<br><br>X = torch.randn(100, 10).to(device)<br>y = torch.randn(100, 1).to(device)<br><br>optimizer = torch.optim.Adam(model.parameters())<br>criterion = nn.MSELoss()<br><br>for epoch in range(10):<br>&nbsp; &nbsp; optimizer.zero_grad()<br>&nbsp; &nbsp; output = model(X)<br>&nbsp; &nbsp; loss = criterion(output, y)<br>&nbsp; &nbsp; loss.backward()<br>&nbsp; &nbsp; optimizer.step()<br>&nbsp; &nbsp; if (epoch + 1) % 2 == 0:<br>&nbsp; &nbsp; &nbsp; &nbsp; print(f&#8221;Epoch {epoch + 1}: Loss {loss.item():.4f}&#8221;)<br><br>print(&#8220;Training complete!&#8221;)<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Swap in your real model and data and the run command does not change. That is the payoff of this whole workflow: the <strong>container ai models<\/strong> you build run on the GPU with zero host-side installation, here or on the next server you spin up.<\/p>\n\n\n\n<p>Run it:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>docker run &#8211;rm &#8211;gpus all -v \/path\/to\/your\/scripts:\/workspace \\<br>&nbsp; pytorch\/pytorch:2.1.0-cuda12.1-cudnn8-runtime python \/workspace\/train.py<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Mount the scripts folder, point Python at the file, and it runs. The output ends with &#8220;Training complete!&#8221;.<\/p>\n\n\n\n<p>While it runs, watch the card from a second terminal on the host:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>watch -n 1 nvidia-smi<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Usage climbing and VRAM filling is your proof the container is on the GPU and not quietly grinding away on the CPU.<\/p>\n\n\n\n<p>Two small jobs can share one card. Point both at the same device and let the driver interleave them:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>docker run &#8211;rm &#8211;gpus &#8216;&#8221;device=0&#8243;&#8216; -v \/data:\/data \\<br>&nbsp; pytorch\/pytorch:2.1.0-cuda12.1-cudnn8-runtime python \/workspace\/train_model_1.py &amp; docker run &#8211;rm &#8211;gpus &#8216;&#8221;device=0&#8243;&#8216; -v \/data:\/data \\<br>&nbsp; pytorch\/pytorch:2.1.0-cuda12.1-cudnn8-runtime python \/workspace\/train_model_2.py &amp;<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Fine for inference and light experiments. A serious training run should own its card, though, because two memory-hungry jobs on one GPU will collide in VRAM and one of them will crash. Picking the right card for the job is its own decision, and the breaks down which workloads suit which.<\/p>\n\n\n\n<p>Also Read: <a href=\"https:\/\/www.hostrunway.com\/blog\/cloud-vs-dedicated-servers-the-decision-framework-every-cto-should-know\/\">Cloud vs. Dedicated Servers: The Decision Framework Every CTO Should Know<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" style=\"font-size:22px\"><span class=\"ez-toc-section\" id=\"Step_5_Docker_Compose_for_multi-container_setups\"><\/span><strong>Step 5: Docker Compose for multi-container setups<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>A single docker run holds up fine until you have two or three containers that need to start together, share volumes, and come up in the right order. Docker Compose puts all of that in one file.<\/p>\n\n\n\n<p>The GPU block below is the part to read closely. Compose v2 reserves cards through a deploy section, which replaced the old runtime: nvidia shortcut, and the capabilities: [gpu] line is mandatory. Leave it out and the service refuses to start.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>services:<br>&nbsp; ai_training:<br>&nbsp; &nbsp; image: pytorch\/pytorch:2.1.0-cuda12.1-cudnn8-runtime<br>&nbsp; &nbsp; container_name: ai_train_job<br>&nbsp; &nbsp; deploy:<br>&nbsp; &nbsp; &nbsp; resources:<br>&nbsp; &nbsp; &nbsp; &nbsp; reservations:<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; devices:<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &#8211; driver: nvidia<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; count: all<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; capabilities: [gpu]<br>&nbsp; &nbsp; environment:<br>&nbsp; &nbsp; &nbsp; &#8211; PYTHONUNBUFFERED=1<br>&nbsp; &nbsp; volumes:<br>&nbsp; &nbsp; &nbsp; &#8211; \/path\/to\/data:\/data<br>&nbsp; &nbsp; &nbsp; &#8211; \/path\/to\/models:\/models<br>&nbsp; &nbsp; &nbsp; &#8211; \/path\/to\/scripts:\/workspace<br>&nbsp; &nbsp; working_dir: \/workspace<br>&nbsp; &nbsp; command: python train.py<br>&nbsp; &nbsp; restart: unless-stopped<br><br>&nbsp; inference_server:<br>&nbsp; &nbsp; image: pytorch\/pytorch:2.1.0-cuda12.1-cudnn8-runtime<br>&nbsp; &nbsp; container_name: ai_inference<br>&nbsp; &nbsp; deploy:<br>&nbsp; &nbsp; &nbsp; resources:<br>&nbsp; &nbsp; &nbsp; &nbsp; reservations:<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; devices:<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &#8211; driver: nvidia<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; count: all<br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; capabilities: [gpu]<br>&nbsp; &nbsp; environment:<br>&nbsp; &nbsp; &nbsp; &#8211; PORT=8000<br>&nbsp; &nbsp; volumes:<br>&nbsp; &nbsp; &nbsp; &#8211; \/path\/to\/models:\/models<br>&nbsp; &nbsp; &nbsp; &#8211; \/path\/to\/inference_code:\/app<br>&nbsp; &nbsp; working_dir: \/app<br>&nbsp; &nbsp; ports:<br>&nbsp; &nbsp; &nbsp;&#8211; &#8220;8000:8000&#8221;<br>&nbsp; &nbsp; command: python serve.py<br>&nbsp; &nbsp; depends_on:<br>&nbsp; &nbsp; &nbsp; &#8211; ai_training<br>&nbsp; &nbsp; restart: unless-stopped<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Top to bottom, that defines two containers: a training job, and an inference service that waits for it through depends_on, exposes port 8000, and restarts itself if it dies. Older Compose files opened with a version: key. Drop it. Compose v2 ignores it.<\/p>\n\n\n\n<p>Bring the whole thing up in the background:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>docker compose up -d<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Mind the space. docker compose is the plugin you installed in Step 1. docker-compose with a hyphen is the retired standalone tool, and confusing the two is a frequent source of &#8220;command not found.&#8221;<\/p>\n\n\n\n<p>Watch what is happening:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>docker compose ps<br>docker compose logs -f ai_training<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Tear it all down with docker compose down. Anything written to \/path\/to\/models survives, because it lives on the host and not in the containers. For heavier multi-service stacks, the goes further.<\/p>\n\n\n\n<p>Also Read: <a href=\"https:\/\/www.hostrunway.com\/blog\/llm-training-in-2026-what-nobody-tells-you-about-infrastructure-costs\/\">LLM Training in 2026: What Nobody Tells You About Infrastructure Costs<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" style=\"font-size:22px\"><span class=\"ez-toc-section\" id=\"Getting_containers_production-ready\"><\/span><strong>Getting containers production-ready<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>A container that runs once on your desk is not the same thing as one you trust to run overnight, restart cleanly, and not drag the whole server down with it. Four areas close that gap.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" style=\"font-size:20px\"><span class=\"ez-toc-section\" id=\"GPU_memory\"><\/span><strong>GPU memory<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>A myth worth killing first: &#8211;gpus cannot cap VRAM. The flag chooses which cards a container sees, and that is all it does. There is no memory= option on it, whatever older guides show. To stop one container from eating a whole card, you have three real levers. Pin it to a single device:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>docker run &#8211;rm &#8211;gpus &#8216;&#8221;device=0&#8243;&#8216; \\<br>&nbsp; pytorch\/pytorch:2.1.0-cuda12.1-cudnn8-runtime python train.py<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Cap the fraction from inside the framework:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>import torchtorch.cuda.set_per_process_memory_fraction(0.5, device=0)&nbsp;<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Or, on <a href=\"https:\/\/www.hostrunway.com\/gpu-server\/nvidia-a100.php\" title=\"\">A100<\/a> and <a href=\"https:\/\/www.hostrunway.com\/gpu-server\/nvidia-h100.php\" title=\"\">H100<\/a> cards, use MIG to split the GPU into hardware-isolated slices and give each container its own. MIG is the only one of the three that is a hard wall. The other two are cooperative and depend on the jobs&#8217; behaviour.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" style=\"font-size:20px\"><span class=\"ez-toc-section\" id=\"Storage_and_limits\"><\/span><strong>Storage and limits<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>Trained weights should outlive the container that produced them. A named volume takes care of that:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>docker volume create ai_models<br>docker run &#8211;rm &#8211;gpus all -v ai_models:\/models \\<br>&nbsp; pytorch\/pytorch:2.1.0-cuda12.1-cudnn8-runtime python train.py<br>docker volume ls<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Cap CPU and host memory so a runaway process cannot take the box down with it:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>docker run &#8211;rm &#8211;gpus all &#8211;cpus 4 &#8211;memory 32g \\<br>&nbsp; pytorch\/pytorch:2.1.0-cuda12.1-cudnn8-runtime python train.py<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>And when something dies at 3 a.m., you want it back without you. In Compose:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>deploy:<br>&nbsp; restart_policy:<br>&nbsp; &nbsp; condition: on-failure<br>&nbsp; &nbsp; max_attempts: 3<br>&nbsp; &nbsp; delay: 10s<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" style=\"font-size:20px\"><span class=\"ez-toc-section\" id=\"Security_and_isolation\"><\/span><strong>Security and isolation<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>The defaults lean permissive, so tighten them. Run as a normal user instead of root, which keeps a container breakout from landing on the host with root in hand:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>docker run &#8211;user 1000:1000 &#8211;rm &#8211;gpus all \\<br>&nbsp; pytorch\/pytorch:2.1.0-cuda12.1-cudnn8-runtime python train.py<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>For offline training on sensitive data, cut the network entirely with &#8211;network none. The container computes, writes to its volume, and talks to nothing else. There is more to lock down at the host level, and it covers it.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" style=\"font-size:20px\"><span class=\"ez-toc-section\" id=\"Housekeeping\"><\/span><strong>Housekeeping<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>Check GPU use in a live container without opening a shell:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>docker exec &lt;container_id&gt; nvidia-smi<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Read a stopped container&#8217;s output when you are hunting a crash:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>docker logs &#8211;tail 100 &lt;container_id&gt;<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>And reclaim disk when images stack up. Read the confirmation before you agree, because -a removes every unused image, not just dangling layers:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>docker system prune -a<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Also Read: <a href=\"https:\/\/www.hostrunway.com\/blog\/ai-video-generation-2026-best-gpus-vram-guide-and-smart-setups-that-work\/\">AI Video Generation 2026: Best GPUs, VRAM Guide, and Smart Setups That Work<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" style=\"font-size:22px\"><span class=\"ez-toc-section\" id=\"Troubleshooting_common_errors\"><\/span><strong>Troubleshooting common errors<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Most GPU and Docker trouble traces back to a short list of causes. Here are the ones you will actually run into.<\/p>\n\n\n\n<p>docker: command not found means the install did not take, or the shell cannot find the binary. Confirm, then reinstall if it is genuinely missing:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>which dockersudo apt-get install -y docker-ce&nbsp;<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>permission denied while trying to connect to the Docker daemon is the docker-group issue from Step 1. Add yourself and reload the group, or fall back to sudo:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>sudo usermod -aG docker $USERnewgrp docker&nbsp;<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>could not select device driver &#8220;&#8221; with capabilities: [[gpu]] is the classic GPU failure. Either the toolkit is missing, or the daemon was never restarted after you configured it:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>nvidia-ctk &#8211;version<br>sudo nvidia-ctk runtime configure &#8211;runtime=docker<br>sudo systemctl restart docker<br>docker run &#8211;rm &#8211;gpus all nvidia\/cuda:12.1.1-runtime-ubuntu22.04 nvidia-smi<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>no space left on device turns up mid-pull, when images have filled the disk. See what Docker is holding, then clear it out:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>df -h<br>docker system df<br>docker image prune -a<br>docker container prune<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>torch.cuda.is_available() returning False has two usual causes: the &#8211;gpus flag got left off, or the host driver is older than the CUDA version baked into the image. Check both, working from the outside in:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>nvidia-smi<br>docker run &#8211;rm &#8211;gpus all nvidia\/cuda:12.1.1-runtime-ubuntu22.04 nvidia-smi<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>If the host nvidia-smi works but the container&#8217;s does not, the toolkit is your suspect. If the host driver is simply too old for the image, update it.<\/p>\n\n\n\n<p>CUDA out of memory is a model-too-big problem, not a Docker one. Watch the VRAM as it climbs:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>docker exec &lt;container_id&gt; nvidia-smi<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Then shrink the footprint: smaller batch size, gradient checkpointing, or 8-bit or 4-bit quantization. Since &#8211;gpus will not cap memory for you, control it in code or give the job a card of its own.<\/p>\n\n\n\n<p>A container that exits the instant it starts almost always ran a command that failed or pointed at the wrong directory. Read the logs first, then drop into a shell and look around:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>docker logs &lt;container_id&gt;<br>docker run &#8211;rm &#8211;gpus all -it pytorch\/pytorch:2.1.0-cuda12.1-cudnn8-runtime bash<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>When a GPU container will not cooperate, three suspects cover most cases: a missing NVIDIA toolkit, a driver that lags the image&#8217;s CUDA version, and a forgotten &#8211;gpus flag. Start there before you look anywhere else. If the host itself seems off, the team can check it from their side.<\/p>\n\n\n\n<p>Also Read: <a href=\"https:\/\/www.hostrunway.com\/blog\/what-is-ddos-protection-on-a-dedicated-server-and-why-your-business-needs-it-in-2026\/\">What Is DDoS Protection on a Dedicated Server \u2014 And Why Your Business Needs It in 2026?<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" style=\"font-size:22px\"><span class=\"ez-toc-section\" id=\"Running_TensorFlow_containers_on_the_GPU\"><\/span><strong>Running TensorFlow containers on the GPU<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>PyTorch got the long version. TensorFlow is the same shape, so this part is short. Everything you already set up, the toolkit and the &#8211;gpus flag, carries straight into <strong>TensorFlow Docker GPU<\/strong> work.<\/p>\n\n\n\n<p>Pull the official image:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>docker pull tensorflow\/tensorflow:latest-gpu<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>Run it and check what it sees:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>docker run &#8211;rm &#8211;gpus all -it tensorflow\/tensorflow:latest-gpu \\<br>python3 -c &#8220;import tensorflow as tf; print(&#8216;GPUs:&#8217;, tf.config.list_physical_devices(&#8216;GPU&#8217;))&#8221;<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>A non-empty list means the card is available. Running a script looks identical to the PyTorch case, just with a different image name:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><tbody><tr><td>docker run &#8211;rm &#8211;gpus all -v \/path\/to\/scripts:\/workspace \\<br>&nbsp; tensorflow\/tensorflow:latest-gpu python \/workspace\/train.py<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p>What differs between the two frameworks is small. The image is tensorflow\/tensorflow, and its GPU tag is a plain latest-gpu instead of the explicit CUDA and cuDNN numbers PyTorch pins into its tag. Both honor &#8211;gpus all and &#8211;gpus &#8216;&#8221;device=0&#8243;&#8216; the same way, and both land in the 4 to 7 GB range. Past that, nothing changes. Volumes, Compose, resource limits, monitoring, all of it behaves the same. Use whichever framework your team already writes in. The container workflow does not care which one you picked.<\/p>\n\n\n\n<p>Also Read: <a href=\"https:\/\/www.hostrunway.com\/blog\/how-to-right-size-gpu-instances-and-stop-overpaying-in-2026\/\">How to Right-Size GPU Instances and Stop Overpaying in 2026<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" style=\"font-size:22px\"><span class=\"ez-toc-section\" id=\"Running_AI_at_scale_on_Hostrunway\"><\/span><strong>Running AI at scale on Hostrunway<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>That is the full path. Docker installed, the NVIDIA runtime wired in, and both PyTorch and TensorFlow reaching the GPU from inside a container. The same commands that ran one job here run a hundred across a fleet, unchanged, which is the real reason to bother with any of it.<\/p>\n\n\n\n<p>The value shows up when a model moves from experiment to service. The container that trained on one card today deploys tomorrow with its environment intact, and rolling back a bad release is nothing more than running an older tag. The host stays clean, because every dependency lives in the image instead of scattered across the machine.<\/p>\n\n\n\n<p>If you are still sizing the hardware, <a href=\"https:\/\/www.hostrunway.com\/\" title=\"\">Hostrunway<\/a> runs <a href=\"https:\/\/www.hostrunway.com\/gpu-cloud-server.php\" title=\"\">cloud GPU servers<\/a> built for exactly this, with current NVIDIA cards and Ubuntu images that show up ready for the driver and toolkit setup above. You can provision one, follow this guide end to end, and have a container on the GPU the same afternoon. The line the cards up against workloads.<\/p>\n\n\n\n<p>One last honest note before you scale it. A bare docker run is not a serving platform. Real request volume needs a model server such as Triton or TorchServe and an orchestrator such as Kubernetes to place and scale replicas across nodes. What Docker hands you is the reliable, identical unit sitting underneath all of that. Deciding what to build around it is the next problem, and a good one to have.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" style=\"font-size:22px\"><span class=\"ez-toc-section\" id=\"Frequently_asked_questions\"><\/span><strong>Frequently asked questions<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p><strong>Do I need CUDA installed on the host if I&#8217;m using Docker containers?<\/strong><\/p>\n\n\n\n<p>No, the container brings its own CUDA and cuDNN. The host only needs the NVIDIA driver plus the Container Toolkit. The one real constraint is that the host driver has to be recent enough for the CUDA version inside your image, since the driver is never bundled in the container. A full host CUDA Toolkit is optional and mostly handy for compiling or debugging outside containers.<\/p>\n\n\n\n<p><strong>Can I run multiple AI containers on one GPU simultaneously?<\/strong><\/p>\n\n\n\n<p>Yes. Point them at the same card with &#8211;gpus &#8216;&#8221;device=0&#8243;&#8216; and the driver time-slices between them. That works well for inference and small jobs, less well for training, where they fight over VRAM. On A100 and H100 cards, MIG gives each container an isolated slice and sidesteps the contention.<\/p>\n\n\n\n<p><strong>How do I version control my trained models inside Docker containers?<\/strong><\/p>\n\n\n\n<p>Keep the model out of the image entirely. Write checkpoints to a mounted volume so they sit on the host, then track them with a large-file tool like Git LFS or DVC, or push them to object storage with a version tag. The container stays a throwaway runtime; the weights and their history live on storage you control and can roll back.<\/p>\n\n\n\n<p><strong>What&#8217;s the difference between nvidia-docker and the &#8211;gpus flag?<\/strong><\/p>\n\n\n\n<p>nvidia-docker was the wrapper you needed before Docker 19.03 to get a GPU into a container. Since then that support moved into Docker itself as the &#8211;gpus flag, backed by the NVIDIA Container Toolkit. You no longer install a separate nvidia-docker binary. The toolkit supplies the runtime, and &#8211;gpus all or &#8211;gpus &#8216;&#8221;device=0&#8243;&#8216; is how you ask for cards.<\/p>\n\n\n\n<p><strong>Can I push a trained model to a Docker registry like Docker Hub?<\/strong><\/p>\n\n\n\n<p>You technically can, by baking the weights into an image and pushing that, but it is usually the wrong call. Images balloon in size, pulls turn slow, and every retrain spawns a fresh image to store and later clean up. Registries were built for code and environments, not gigabytes of data. Put the weights in object storage or a model registry such as MLflow or the Hugging Face Hub, and have the container pull them at startup from a mounted path or a short download step. Your images stay small and your model history stays somewhere tools can reason about it.<\/p>\n\n\n\n<p><strong>How do I monitor GPU memory usage inside a running container?<\/strong><\/p>\n\n\n\n<p>From the host, docker exec &lt;container_id&gt; nvidia-smi shows live use without opening a shell, and wrapping it in watch -n 1 gives a rolling view. Inside your code, torch.cuda.memory_allocated() reports what the framework is holding right now.<\/p>\n\n\n\n<p><strong>Is Docker secure enough for production AI workloads?<\/strong><\/p>\n\n\n\n<p>It runs in production everywhere, but the defaults lean permissive, so you harden it. Run as a non-root user, drop capabilities you do not use, mount read-only where you can, and never expose the Docker socket to a container. Keep base images patched. GPU work adds little risk on its own, though the NVIDIA runtime asks for some host trust, so limit who is allowed to launch privileged containers.<\/p>\n\n\n\n<p><strong>Can I use Docker to run AI inference at scale, thousands of requests per second?<\/strong><\/p>\n\n\n\n<p>Yes, though not from a single docker run. Wrap the model in an inference server such as Triton or TorchServe, then run many replicas behind a load balancer, usually with Kubernetes scheduling them across GPU nodes. Docker gives you one identical replica; the orchestration layer handles the volume, batching, and failover. Triton in particular is built for high-throughput GPU serving.<\/p>\n\n\n\n<p><strong>How much overhead does Docker add to GPU training performance?<\/strong><\/p>\n\n\n\n<p>Almost none. The GPU runs the math directly and the NVIDIA runtime passes the device through without a virtualization layer, so throughput lands within a point or two of bare metal. The measurable costs sit elsewhere: first-run image pulls, some storage-driver I\/O, and startup time, all of which fade next to a long training job.<\/p>\n\n\n\n<p><strong>Can I build my own AI container images instead of using official ones?<\/strong><\/p>\n\n\n\n<p>Yes, and most teams get there eventually. Start a Dockerfile from an official base like pytorch\/pytorch or nvidia\/cuda, then layer your dependencies and code on top, which locks your exact versions into a reproducible image you own. Building up from bare Linux is rarely worth the effort, since the official CUDA and framework bases already solve the painful driver-compatibility part. That full build workflow is a guide of its own.<\/p>\n\n\n\n<p><strong>If a container crashes during training, does my model get saved?<\/strong><\/p>\n\n\n\n<p>Only what already reached disk. Anything held in the container&#8217;s memory or its writable layer is gone the moment it exits. Save checkpoints on a schedule to a mounted volume, so a crash costs you minutes instead of the whole run. Pair that with a restart policy and resume-from-checkpoint logic, and the job comes back close to where it stopped.<\/p>\n\n\n\n<p><strong>How do I pass environment variables to a containerized AI model?<\/strong><\/p>\n\n\n\n<p>Use -e on the command line, like docker run -e MODEL_PATH=\/models\/latest, or list them under environment: in a Compose file. When there are many, or when some are secrets, load them from a file with &#8211;env-file rather than typing them out. GPU jobs often set NVIDIA_VISIBLE_DEVICES to choose cards and read CUDA_VISIBLE_DEVICES inside the framework. Keep genuine secrets out of the image and out of version control, and inject them at run time.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Anyone who has moved a model off a laptop and onto a GPU box knows the failure. The script runs fine locally, then dies on the server with an import&hellip;<\/p>\n","protected":false},"author":2,"featured_media":1369,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[28,102],"tags":[1349,1353,1355,1347,1352,1351,1354,1348,1350],"class_list":["post-1368","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-ml","category-gpu-server","tag-ai-containers-gpu","tag-docker-compose-gpu-setup","tag-docker-container","tag-docker-gpu-server","tag-docker-gpu-server-setup","tag-gpu-docker-image","tag-how-to-install-docker-on-gpu-server","tag-install-docker-nvidia-gpu","tag-pytorch-docker-container"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 4.9.9 - aioseo.com -->\n\t<meta name=\"description\" content=\"Step-by-step guide to installing Docker on a cloud GPU server and running AI containers with NVIDIA GPU support. Covers PyTorch, TensorFlow, and fixes.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Mike Jonshan\"\/>\n\t<meta name=\"keywords\" content=\"ai containers gpu,docker compose gpu setup,docker container,docker gpu server,docker gpu server setup,gpu docker image,how to install docker on gpu server,install docker nvidia gpu,pytorch docker container\" \/>\n\t<link rel=\"canonical\" href=\"https:\/\/www.hostrunway.com\/blog\/how-to-install-docker-and-run-ai-containers-on-a-cloud-gpu-server\/\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 4.9.9\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"Hostrunway Blog -\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"How to Install Docker and Run AI Containers on GPU Server\" \/>\n\t\t<meta property=\"og:description\" content=\"Step-by-step guide to installing Docker on a cloud GPU server and running AI containers with NVIDIA GPU support. Covers PyTorch, TensorFlow, and fixes.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/www.hostrunway.com\/blog\/how-to-install-docker-and-run-ai-containers-on-a-cloud-gpu-server\/\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/www.hostrunway.com\/blog\/wp-content\/uploads\/2026\/09\/Untitled-September-11-2026-at-10.34.08-6.jpeg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/www.hostrunway.com\/blog\/wp-content\/uploads\/2026\/09\/Untitled-September-11-2026-at-10.34.08-6.jpeg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"1920\" \/>\n\t\t<meta property=\"og:image:height\" content=\"1080\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2026-10-12T06:30:53+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2026-09-18T07:34:16+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/hostrunway\/\" \/>\n\t\t<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n\t\t<meta name=\"twitter:site\" content=\"@hostrunway\" \/>\n\t\t<meta name=\"twitter:title\" content=\"How to Install Docker and Run AI Containers on GPU Server\" \/>\n\t\t<meta name=\"twitter:description\" content=\"Step-by-step guide to installing Docker on a cloud GPU server and running AI containers with NVIDIA GPU support. Covers PyTorch, TensorFlow, and fixes.\" \/>\n\t\t<meta name=\"twitter:creator\" content=\"@hostrunway\" \/>\n\t\t<meta name=\"twitter:image\" content=\"https:\/\/www.hostrunway.com\/blog\/wp-content\/uploads\/2026\/09\/Untitled-September-11-2026-at-10.34.08-6.jpeg\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"How to Install Docker and Run AI Containers on GPU Server","description":"Step-by-step guide to installing Docker on a cloud GPU server and running AI containers with NVIDIA GPU support. Covers PyTorch, TensorFlow, and fixes.","canonical_url":"https:\/\/www.hostrunway.com\/blog\/how-to-install-docker-and-run-ai-containers-on-a-cloud-gpu-server\/","robots":"max-image-preview:large","keywords":"ai containers gpu,docker compose gpu setup,docker container,docker gpu server,docker gpu server setup,gpu docker image,how to install docker on gpu server,install docker nvidia gpu,pytorch docker container","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"Hostrunway Blog -","og:type":"article","og:title":"How to Install Docker and Run AI Containers on GPU Server","og:description":"Step-by-step guide to installing Docker on a cloud GPU server and running AI containers with NVIDIA GPU support. Covers PyTorch, TensorFlow, and fixes.","og:url":"https:\/\/www.hostrunway.com\/blog\/how-to-install-docker-and-run-ai-containers-on-a-cloud-gpu-server\/","og:image":"https:\/\/www.hostrunway.com\/blog\/wp-content\/uploads\/2026\/09\/Untitled-September-11-2026-at-10.34.08-6.jpeg","og:image:secure_url":"https:\/\/www.hostrunway.com\/blog\/wp-content\/uploads\/2026\/09\/Untitled-September-11-2026-at-10.34.08-6.jpeg","og:image:width":"1920","og:image:height":"1080","article:published_time":"2026-10-12T06:30:53+00:00","article:modified_time":"2026-09-18T07:34:16+00:00","article:publisher":"https:\/\/www.facebook.com\/hostrunway\/","twitter:card":"summary_large_image","twitter:site":"@hostrunway","twitter:title":"How to Install Docker and Run AI Containers on GPU Server","twitter:description":"Step-by-step guide to installing Docker on a cloud GPU server and running AI containers with NVIDIA GPU support. Covers PyTorch, TensorFlow, and fixes.","twitter:creator":"@hostrunway","twitter:image":"https:\/\/www.hostrunway.com\/blog\/wp-content\/uploads\/2026\/09\/Untitled-September-11-2026-at-10.34.08-6.jpeg"},"aioseo_meta_data":{"post_id":"1368","title":"How to Install Docker and Run AI Containers on GPU Server","description":"Step-by-step guide to installing Docker on a cloud GPU server and running AI containers with NVIDIA GPU support. Covers PyTorch, TensorFlow, and fixes.","keywords":null,"keyphrases":{"focus":{"keyphrase":"","score":0,"analysis":{"keyphraseInTitle":{"score":0,"maxScore":9,"error":1}}},"additional":[]},"primary_term":null,"canonical_url":null,"og_title":"How to Install Docker and Run AI Containers on GPU Server","og_description":"Step-by-step guide to installing Docker on a cloud GPU server and running AI containers with NVIDIA GPU support. Covers PyTorch, TensorFlow, and fixes.","og_object_type":"article","og_image_type":"featured","og_image_url":"https:\/\/www.hostrunway.com\/blog\/wp-content\/uploads\/2026\/09\/Untitled-September-11-2026-at-10.34.08-6.jpeg","og_image_width":"1920","og_image_height":"1080","og_image_custom_url":null,"og_image_custom_fields":null,"og_video":"","og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"BlogPosting","isEnabled":true},"graphs":[]},"schema_type":"default","schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":"-1","robots_max_videopreview":"-1","robots_max_imagepreview":"large","priority":null,"frequency":"default","local_seo":null,"breadcrumb_settings":null,"limit_modified_date":false,"ai":{"faqs":[],"keyPoints":[],"schemas":[],"titles":[],"descriptions":[],"socialPosts":{"email":{"subject":"","preview":"","content":""},"linkedin":[],"twitter":[],"facebook":[],"instagram":[]}},"created":"2026-09-18 06:30:53","updated":"2026-10-12 06:31:17","seo_analyzer_scan_date":null},"aioseo_breadcrumb":"<div class=\"aioseo-breadcrumbs\"><span class=\"aioseo-breadcrumb\">\n\t\t\t<a href=\"https:\/\/www.hostrunway.com\/blog\" title=\"Home\">Home<\/a>\n\t\t<\/span><span class=\"aioseo-breadcrumb-separator\">&raquo;<\/span><span class=\"aioseo-breadcrumb\">\n\t\t\t<a href=\"https:\/\/www.hostrunway.com\/blog\/category\/ai-ml\/\" title=\"AL\/ML\">AL\/ML<\/a>\n\t\t<\/span><span class=\"aioseo-breadcrumb-separator\">&raquo;<\/span><span class=\"aioseo-breadcrumb\">\n\t\t\tHow to install Docker and run AI containers on a cloud GPU server\n\t\t<\/span><\/div>","aioseo_breadcrumb_json":[{"label":"Home","link":"https:\/\/www.hostrunway.com\/blog"},{"label":"AL\/ML","link":"https:\/\/www.hostrunway.com\/blog\/category\/ai-ml\/"},{"label":"How to install Docker and run AI containers on a cloud GPU server","link":"https:\/\/www.hostrunway.com\/blog\/how-to-install-docker-and-run-ai-containers-on-a-cloud-gpu-server\/"}],"_links":{"self":[{"href":"https:\/\/www.hostrunway.com\/blog\/wp-json\/wp\/v2\/posts\/1368","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.hostrunway.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.hostrunway.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.hostrunway.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.hostrunway.com\/blog\/wp-json\/wp\/v2\/comments?post=1368"}],"version-history":[{"count":1,"href":"https:\/\/www.hostrunway.com\/blog\/wp-json\/wp\/v2\/posts\/1368\/revisions"}],"predecessor-version":[{"id":1370,"href":"https:\/\/www.hostrunway.com\/blog\/wp-json\/wp\/v2\/posts\/1368\/revisions\/1370"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.hostrunway.com\/blog\/wp-json\/wp\/v2\/media\/1369"}],"wp:attachment":[{"href":"https:\/\/www.hostrunway.com\/blog\/wp-json\/wp\/v2\/media?parent=1368"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.hostrunway.com\/blog\/wp-json\/wp\/v2\/categories?post=1368"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.hostrunway.com\/blog\/wp-json\/wp\/v2\/tags?post=1368"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}