How to Set Up and Run Ollama on a GPU-Powered VM (vast.ai)

<p>

In this tutorial, we'll walk you through the process of setting up and using Ollama for private model inference on a VM with GPU, either on your local machine or a rented VM from <a href="https://cloud.vast.ai/?ref_id=145250&amp;ref=airabbit.blog" rel="nofollow">Vast.ai</a>or <a href="https://runpod.io/?ref=7su8gs12" rel="nofollow">Runpod</a>.io. Ollama allows you to run models privately, ensuring data security and faster inference times thanks to the power of GPUs. By leveraging a GPU-powered VM, you can significantly improve the performance and efficiency of your model inference tasks.</p>
<h1 class="relative group flex items-center">
<a class="block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full" href="#outline" id="outline" rel="nofollow">
<span class="header-link"><svg aria-hidden="true" class="text-gray-500 hover:text-black dark:hover:text-gray-200 w-4" height="1em" preserveaspectratio="xMidYMid meet" role="img" viewbox="0 0 256 256" width="1em" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span>
</a>
<span>
		Outline
	</span>
</h1>
<ol>
<li>Set up a VM with GPU on Vast.ai</li>
<li>Start Jupyter Terminal</li>
<li>Install Ollama</li>
<li>Run Ollama Serve</li>
<li>Test Ollama with a model</li>
<li>(Optional) using your own model</li>
</ol>
<h1 class="relative group flex items-center">
<a class="block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full" href="#setting-up-a-vm-with-gpu-on-vastai" id="setting-up-a-vm-with-gpu-on-vastai" rel="nofollow">
<span class="header-link"><svg aria-hidden="true" class="text-gray-500 hover:text-black dark:hover:text-gray-200 w-4" height="1em" preserveaspectratio="xMidYMid meet" role="img" viewbox="0 0 256 256" width="1em" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span>
</a>
<span>
		Setting Up a VM with GPU on Vast.ai
	</span>
</h1>
<p><strong>1. Create a VM with GPU:</strong> --- Visit <a href="https://cloud.vast.ai/create/?ref=airabbit.blog" rel="nofollow">Vast.ai</a> to create your VM. --- Choose a VM with at least 30 GB of storage to accommodate the models. This ensures you have enough space for installation and model storage. --- Select a VM that costs less than $0.30 per hour to keep the setup cost-effective.</p>
<p><a href="https://miro.medium.com/v2/resize:fit:1400/1*BmfKGSHXTwv552eWCYyzSA.png" rel="nofollow"><img alt="" src="https://miro.medium.com/v2/resize:fit:1400/1*BmfKGSHXTwv552eWCYyzSA.png"/></a></p>
<p><strong>2. Start Jupyter Terminal:</strong> --- Once your VM is up and running, start Jupyter and open a terminal within it.</p>
<p><a href="https://miro.medium.com/v2/resize:fit:1400/1*5V3zvvO8WMCFR9kr6WXdrg.png" rel="nofollow"><img alt="" src="https://miro.medium.com/v2/resize:fit:1400/1*5V3zvvO8WMCFR9kr6WXdrg.png"/></a></p>
<h1 class="relative group flex items-center">
<a class="block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full" href="#downloading-and-running-ollama" id="downloading-and-running-ollama" rel="nofollow">
<span class="header-link"><svg aria-hidden="true" class="text-gray-500 hover:text-black dark:hover:text-gray-200 w-4" height="1em" preserveaspectratio="xMidYMid meet" role="img" viewbox="0 0 256 256" width="1em" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span>
</a>
<span>
		Downloading and Running Ollama
	</span>
</h1>
<ol>
<li><strong>Start Jupyter Terminal:</strong> --- Once your VM is up and running, start Jupyter and open a terminal within it. This is the easiest method to get started. --- Alternatively, you can use SSH on your local VM, for example with VSCode, but you will need to create an SSH key to use it.</li>
</ol>
<p><a href="https://miro.medium.com/v2/resize:fit:1400/1*5FThAEYy7bY-zrOqdexX5g.png" rel="nofollow"><img alt="" src="https://miro.medium.com/v2/resize:fit:1400/1*5FThAEYy7bY-zrOqdexX5g.png"/></a></p>
<ol>
<li><strong>Install Ollama:</strong> --- Open the terminal in Jupyter and run the following command to install Ollama:</li>
</ol>
<pre><code>bash curl -fsSL https://ollama.com/install.sh | sh
</code></pre>
<p><strong>2. Run Ollama Serve:</strong> --- After installation, start the Ollama service by running:</p>
<pre><code>bash ollama serve &amp;
</code></pre>
<p>Ensure there are no GPU errors. If there are issues, the response will be slow when interacting with the model.</p>
<p><strong>3. Test Ollama with a Model:</strong> --- Test the setup by running a sample model like Mistral:</p>
<pre><code>bash ollama run mistral
</code></pre>
<p>You can now start chatting with the model to ensure everything is working correctly.</p>
<p><strong>Optional (Check GPU usage)</strong></p>
<p><strong>Check GPU Utilization:</strong> --- During the inference (last step), check if the GPU is being utilized by running the following command:<code>bash nvidia-smi </code>- Ensure that the memory utilization is greater than 0%. This indicates that the GPU is being used for the inference process.</p>
<p><a href="https://miro.medium.com/v2/resize:fit:1400/1*9bZnZptF5Wj4eXdo7Y7haw.png" rel="nofollow"><img alt="" src="https://miro.medium.com/v2/resize:fit:1400/1*9bZnZptF5Wj4eXdo7Y7haw.png"/></a></p>
<h1 class="relative group flex items-center">
<a class="block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full" href="#using-your-own-hugging-face-model-with-ollama" id="using-your-own-hugging-face-model-with-ollama" rel="nofollow">
<span class="header-link"><svg aria-hidden="true" class="text-gray-500 hover:text-black dark:hover:text-gray-200 w-4" height="1em" preserveaspectratio="xMidYMid meet" role="img" viewbox="0 0 256 256" width="1em" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span>
</a>
<span>
		Using Your Own Hugging Face Model with Ollama
	</span>
</h1>
<p><strong>1. Install Hugging Face CLI:</strong> --- If you want to use your own model from Hugging Face, first install the Hugging Face CLI. Here we will use an example of a fine tuned Mistral model <strong>TheBloke/em_german_mistral_v01-GGUF em_german_mistral_v01.Q4_K_M.gguf</strong></p>
<p><strong>2. Download Your Model:</strong> --- Download your desired model from Hugging Face. For example, to download a fine-tuned Mistral model:</p>
<pre><code>pip3 install huggingface-hub# Try with my custom model for fine tuned Mistral
huggingface-cli download TheBloke/em_german_mistral_v01-GGUF em_german_mistral_v01.Q4_K_M.gguf --local-dir . --local-dir-use-symlinks False
</code></pre>
<p><strong>3. Create a Model File:</strong> --- Create a model config file ***Modelfile ***with the following content:</p>
<pre><code>FROM em_german_mistral_v01.Q4_K_M.gguf

# set the temperature to 1 [higher is more creative, lower is more coherent]
PARAMETER temperature 0

# # set the system message
# SYSTEM """
# You are Mario from Super Mario Bros. Answer as Mario, the assistant, only.
# """
</code></pre>
<p><strong>4. Instruct Ollama to Create the Model:</strong> --- Create the custom model using Ollama with the command:</p>
<pre><code>ollama create -f mymodel Modelfile
</code></pre>
<p><strong>5. Run Your Custom Model:</strong> --- Run your custom model using:</p>
<pre><code>ollama run mymodel
</code></pre>
<p>By following these steps, you can effectively utilize Ollama for private model inference on a VM with GPU, ensuring secure and efficient operations for your machine learning projects.</p>
<p>Happy prompting!</p>
<p><a href="https://airabbit.blog/how-to-provide-inference-pay-per-use/" rel="nofollow"></a></p>
