Ollama - ArchWiki

Jump to content

# Ollama

From ArchWiki

Related articles

- General-purpose computing on graphics processing units
- llama.cpp
- Vulkan

Ollama is an application which lets you run offline large language models locally.

## Installation

Install the ollama package, which provides a daemon, command line tool, and CPU inference.

For GPU inference:

- Install ollama-cuda for inference with CUDA.
- Install ollama-rocm for inference with ROCm.
- Install ollama-vulkan for inference with Vulkan (experimental).

Next, enable/start `ollama.service`. Then, verify Ollama's status:

    $ ollama --version

If it says `Warning: could not connect to a running Ollama instance`, then the Ollama service has not been run; otherwise, the Ollama service is running and is ready to accept user requests.

Next, verify that you can run models. The following command downloads the latest 270M parameter model of Gemma 3 and returns an Ollama prompt that allows you to talk to the model:

    $ ollama run gemma3:270m

    >>> Send a message (/? for help)

## Usage

The Ollama executable does not provide a search interface. There is no such command as `ollama search`. To search for a model, you need to visit their search page.

To run a model:

    $ ollama run model

To stop a model:

    $ ollama stop model

To update a model:

    $ ollama pull model

To remove a model:

    $ ollama rm model

To view locally available models:

    $ ollama list

## Troubleshooting

### ROCm is not utilizing my AMD GPU

You may have used utilities like amdgpu_top to monitor the utilization of your GPU during an Ollama session, but only to notice that your GPU has not been used at all.

Without configuration, ROCm simply ignores unsupported GPUs, causing everything to be computed on CPU.

Note Verify supported GPUs by consulting ROCm System Requirements.

To work this around, create a drop-in file for `ollama.service`:

    /etc/systemd/system/ollama.service.d/override_gfx_version.conf

    [Service]
    Environment="HSA_OVERRIDE_GFX_VERSION=X.Y.Z"

Where `X.Y.Z` is dependent to the GFX version that is shipped with your system.

To determine which GFX version to use, first make sure rocminfo has already been installed. It should be pulled in to your system as a dependency of rocblas, which is itself a dependency of ollama-rocm.

Next, query the actual GFX version of your system:

    $ /opt/rocm/bin/rocminfo | grep amdhsa

You need to remember the digits printed after the word `gfx`, because this is the actual GFX version of your system. The digits are interpreted as follows:

- If the digits are 4-digit, they are interpreted as `XX.Y.Z`, where the first two digits are interpreted as the `X` part.
- If the digits are 3-digit, they are interpreted as `X.Y.Z`.

Then, find all installed rocblas kernels:

    $ find /opt/rocm/lib/rocblas/library -name 'Kernels.so-*'

You need to set `X.Y.Z` to one of the available versions listed there. The rules are summarized as follows:

1. For the `X` part, it must be strictly equal to the actual version.
2. For the `Y` part, mismatch is allowed, but it must be no greater than the actual version.
3. For the `Z` part, mismatch is allowed, but it must be no greater than the actual version.

After setting the correct `X.Y.Z`, perform a daemon-reload and restart `ollama.service`.

Then, run your model as usual. You may wish to monitor GPU utilization with amdgpu_top again.

### Models are not removed after uninstalling Ollama

You can manually remove the model files. They are stored in `/var/lib/ollama/blobs`.

## See also

- Ollama Blog
- Ollama Docs
- What is rocBLAS

Categories:

- Development
- Graphics

Ollama
Add topic