AI Servers

This guide describes how to register an AI server in AI Knowledge Fabric, start and stop it, and look at its models. You find the screens under Admin → AI Servers.

1. What an AI server is

An AI server is the place where the AI models run. AI Knowledge Fabric does not contain any model itself. Every model a topic or an agent uses comes from a registered AI server.

You need AI servers for:

  • Topics — a language model that writes the answers, an embedding model that turns your documents into searchable vectors, and optionally a vision model that describes pictures.

  • GraphRAG Topics — the same models, plus optional models for single steps like extraction or routing.

  • Pipelines — the modules that call a model (for example Agent, Guardrails, Structured Output, Embedding) pick their model from an AI server with Select AI Server & Model.

One AI server can serve many topics and pipelines. One topic can use models from several AI servers, for example a cloud model for answers and a local model for embeddings. No AI server exists after installation. You add one yourself.

2. Providers

The Provider says what kind of server it is. It decides which fields the form shows.

Provider What it is Address API key Embedding models

Ollama

An Ollama server. The only provider this app can also run itself (managed)

You enter host and port

No field

Yes

OpenAI API (self-hosted)

Any server that speaks the OpenAI API at an address you choose: for example vLLM, TGI, SGLang, LocalAI, llama-server or LM Studio, a gateway like LiteLLM or OpenRouter, or a provider like Groq, Together or Azure OpenAI

You enter host, port and TLS

Optional

If the server offers them

Anthropic (Cloud)

Anthropic’s Claude models

Filled in: api.anthropic.com

Yes

No. Anthropic offers no embedding models. Pick the embedding model from another server

OpenAI (Cloud)

OpenAI’s own service

Filled in: api.openai.com

Yes

Yes

Google Gemini (Cloud)

Google’s Gemini models

Filled in: generativelanguage.googleapis.com

Yes

Yes

The three cloud providers always use HTTPS. They need no port.

A vision model, which describes pictures in documents, can come from any provider. The model itself must be able to read pictures.

Step-by-step examples:

3. Managed or external

When you add a server, you choose its Server Type:

Managed (deployed by this app) External (existing instance)

Who runs it

This app starts and stops an Ollama container for you

It runs somewhere else, or it is a cloud provider

Provider

Always Ollama

Any provider

Start and stop in this app

Yes

No. Only a reachability check

Download models in this app

Yes

Only for an Ollama server

Good for

A quick local start, test setups

A server with a strong graphics card, a shared central server, cloud models

A managed server needs Docker on the host of this app (or Kubernetes, see section 9). The first start downloads the Ollama image. This can take several minutes. A new managed server has no models. Download them on the Models tab after the start, see section 7.

4. The AI Servers list

Navigate to Admin → AI Servers. The list shows every registered server:

Column Shows

Name

The name of the server

Endpoint

The host, and the port if there is one. Cloud providers have no port

Type

Managed or External

Provider

The provider, see section 2

Status

The current status, see Status values

The Search…​ field filters the list by name or host.

The buttons at the end of each row depend on the server type:

Button Tooltip Shown for Does

▶

Start

Managed

Starts the server. Disabled while it runs or while a start is in progress (tooltip Start already running)

■

Stop

Managed

Stops the server. Enabled while it is running or starting

⟳

Test if the server can be reached

External

Checks now whether the server answers, and updates the status

⚙

Settings

Both

Opens the detail page of the server

Add AI Server opens the form for a new server. Add AI Server, ▶, ■ and ⚙ are only shown to users who may manage servers, see section 10.

5. Add or edit an AI server

Click Add AI Server. To change an existing server, open its detail page (⚙) and click Edit on the Configuration tab. The page title is then Edit AI Server.

A new server starts as Managed, Ollama, port 11434. With Docker the host is host.docker.internal.

Which fields appear depends on the server type and the provider:

Field Managed Ollama External Ollama OpenAI API (self-hosted) Cloud providers

Server Type

✔

✔

✔

✔

Provider

✔ (fixed to Ollama)

✔

✔

✔

Name

✔

✔

✔

✔

Host

✔

✔

✔

✔ (filled in)

Port

✔

✔

✔

–

Use TLS (HTTPS)

–

✔

✔

– (always HTTPS)

Default Model ID

–

✔

✔

✔

API Key

–

–

✔

✔

Management API Key Env Var

–

–

✔

✔

Enable WSL2 GPU passthrough

✔

–

–

–

Click Save to store the server. You see AI server saved. A new server then appears in the list. After an edit you return to the detail page. Cancel goes back to the list without saving.

Name

The name you see when a topic or a pipeline picks its model.

  • Only letters and are allowed. A space becomes while you type. Numbers and other special characters are refused.

  • At most 50 characters.

  • Every AI server needs its own name. Upper and lower case count as the same name.

Host

The address topics and agents use to reach the server.

  • A server on the same machine (Docker): enter host.docker.internal. localhost does not work, because each topic runs in its own container.

  • A server on another machine: enter its host name or IP address.

  • Cloud provider: the host is filled in for you. Keep it.

  • Kubernetes: see section 9.

You may paste a full address as the vendor shows it, for example 10.0.0.5:8000/v1. On Save the form splits it for you:

  • https:// switches Use TLS (HTTPS) on. http:// switches it off.

  • :8000 moves into the Port field.

  • A /v1 at the end is removed, because this app adds it by itself.

  • Any other path stays, for example /openai in api.groq.com/openai/v1.

Port

The port the server listens on. Usual values: Ollama 11434, vLLM 8000, LocalAI 8080, TGI 8080, LM Studio 1234.

For a managed server with Docker, this is the port on this machine where topics reach the server. Keep 11434 unless another program already uses it. Then the server cannot start. The form does not check whether two managed servers use the same port. Give each managed server its own port.

A changed port of a managed server takes effect the next time the server starts.

Use TLS (HTTPS)

Only for an external server that is not a cloud provider. Switch it on when the server is reached over https://. A gateway protected by an API key usually is.

  • On: this app and the topics connect with https://. Without a port, port 443 is used.

  • Off: they connect with plain http://. Without a port, port 80 is used.

If this does not match the server, every connection fails.

API Key

The key that topics and agents send to log in to the server. Not shown for Ollama, because Ollama uses no key.

  • Cloud providers: required by the provider. Without a key, the connection test and every model call fail.

  • OpenAI API (self-hosted): optional. Enter it only if the server asks for one, for example vLLM started with --api-key, or a LiteLLM gateway.

The key is stored in this app. When a topic or a GraphRAG Topic starts, it receives the key. This is the same with Docker and Kubernetes. A changed key reaches a topic only after you stop and start it, see section 8.

This app also uses this key for its own calls (connection test, model list), unless you set the next field.

Kubernetes, advanced: if the key must not be stored in this app, keep it in a Kubernetes Secret. Enter the key as ${NAME}, where NAME is the name of the key in the Secret. Topics then read the key from the Secret when they start. See Keeping API keys out of the app.

Management API Key Env Var

Optional. The name of an environment variable of this app, which holds a key, for example ANTHROPIC_API_KEY.

  • Only this app uses it: to test the connection and to load the list of models for the pickers. Topics and agents always use the API Key above.

  • Empty, or the variable is not set: this app uses the API Key above as well.

  • This app reads its environment when it starts. A variable you add later needs a restart of this app.

You need this field mainly with the ${NAME} option on Kubernetes. Then the API Key field holds no real key, so this app needs the key from its own environment.

Default Model ID

Only for an external server. A model name, for example claude-sonnet-4-6 or gpt-4o.

  • When the model list of this server cannot be loaded, the topic pickers fill in this name for you. You can change it.

  • On a topic’s Configuration tab, it is pre-selected when you switch the language model to this server.

Empty: nothing is filled in. The field never limits which model someone picks.

Enable WSL2 GPU passthrough

Only for a managed Ollama server. It lets Ollama use the graphics card. Models then answer much faster.

  • Docker: only for a Windows machine with WSL2. Leave it off everywhere else.

  • Kubernetes: the server asks the cluster for one NVIDIA graphics card. Switch it on only if the cluster has one free. Otherwise the server cannot start.

A change takes effect the next time the server starts.

Validation messages

Message Meaning

Server type is required

Choose Managed or External

Provider is required

Choose a provider

Name is required

The name is empty

Name may only contain letters; numbers and special characters are not allowed

Remove numbers and special characters

Name must not exceed 50 characters

Shorten the name

An AI server with this name already exists

Choose another name

Host is required

Enter a host

6. Starting and stopping a managed server

You start and stop a managed server in the list, with ▶ and ■. You need a valid license to start a server, see License.

What you see during a start

After you click ▶, a message says:

Starting "name". If the Ollama image is not on this host yet it is downloaded first, which can take several minutes.

While the start runs, the Status cell shows Starting…​, a moving progress bar, the current step with the elapsed time (for example Creating the container · 0:12), and the hint "First start downloads the Ollama image and can take several minutes." The list updates itself every two seconds. You can leave the page and come back, or open it in another browser tab. The start goes on and the step is still shown.

The steps with Docker are:

  1. Preparing…

  2. Downloading the Ollama image (several GB) — only if the image is not on the host yet

  3. Creating the container

  4. Starting the container

  5. Waiting for the AI server to answer

With Docker, the server is only Running when it really answers. Starting the container is not enough. The app waits up to 90 seconds for that answer after the container started.

The steps with Kubernetes are:

  1. Preparing…

  2. Applying Kubernetes resources

  3. Waiting for the pod (pulling the image if the node lacks it)

At the end you see one of these messages:

Message Meaning

"name" is running.

The server answers. You can download models and use it

"name" did not answer after starting. It may still be downloading, or it could not start. The application log shows the details. Then try again.

The status is now Error

Failed to start "name": …

The start was refused or failed, for example without a license. The reason follows the colon

A second click on ▶ does not start a second copy. The button is disabled while a start runs.

Stopping

■ stops the server. The status becomes Stopped. Topics that use it cannot answer until it runs again.

  • Docker: the container is removed. The downloaded models stay, so you do not download them again.

  • Kubernetes: the server’s workload is removed. Its address and its models stay.

If the stop fails, the status becomes Error. With Kubernetes you also see Failed to stop "name": … with the reason.

Status values

Status Managed server External server

Created

Saved, never started

Saved, never checked

Starting…​

A start is in progress

–

Running

The server answered at the last check

The server answered at the last check

Stopped

Stopped in this app

–

Error

The start failed, the stop failed, or the server did not answer at the last check

The server did not answer at the last check

For an external server, Error means "could not be reached or refused the login". It does not mean that the settings are broken.

Status check when you open the list

Each time you open the list, the app checks every server that shows Running or Starting…​:

  • It answers → Running.

  • It does not answer → Error.

A server that this app is starting right now is not checked. So a long start is not turned into Error by this check.

The check does not cover servers in Error or Created. They keep their status until you act:

  • External server: click ⟳ (Test if the server can be reached).

  • Managed server: click ▶ again. With Docker, a container that already runs is kept, and the start only waits for its answer before it shows Running.

7. The detail page

Click ⚙ in the list to open the detail page. Back returns to the list. The page has three tabs.

Configuration

Shows the stored settings read-only: Name, Server Type, Provider, Host, Port and Base URL. The Base URL is the full address topics and agents use. It is built from host and port. For a managed Ollama server, the tab also shows whether WSL2 GPU passthrough enabled is on. Edit opens the form from section 5.

Models

Shows the models the server offers right now. Opening the tab loads the list live from the server. You see Connected — n model(s) available, or Failed to load models: … with the reason. Refresh loads the list again.

Column Shown for

Name

All providers. This is the name you pick in topics and pipelines

Size

Ollama

Modified

All providers, if the server reports it

🗑 (Delete from server)

Ollama

A managed server must run for this tab. Otherwise the tab shows Server Not Running. Start the server first.

Pull New Model (Ollama only, managed or external) downloads a model onto the server:

  1. Enter the model name from the Ollama library, for example llama3.2:3b or nomic-embed-text. The part after the colon picks the size. Browse the Ollama model library ↗ opens the library.

  2. Click Pull. A progress bar and the current step appear.

  3. When the download is done, you see Model 'name' pulled. The model then appears in the list and in the pickers of topics and pipelines.

Larger models give better answers, but need more memory and answer more slowly. A failed download shows Pull failed: … with the reason.

🗑 removes a model from an Ollama server and frees its disk space. A dialog asks you to confirm with Delete. Topics and pipelines that use the model stop working until you download it again.

For the other providers, the list is read-only. You manage their models on the server or in the provider’s own console.

Actions

Docker Compose or Kubernetes YAML (managed servers only). Download docker-compose.yml or Download Kubernetes YAML gives you the file this app uses to run the server, with the current settings. Use it to check the settings, or to run the server by hand.

Danger Zone removes the server:

Server type Button What happens

Managed, Docker

Stop & Delete AI Server

Stops the server, deletes its storage (all downloaded models are lost) and removes it from this app

Managed, Kubernetes

Stop & Delete AI Server

Stops the server, deletes everything it uses in the cluster, including all downloaded models, and removes it from this app

External

Remove from Catalog

Removes the server from this app. The server itself keeps running where it is

A dialog asks you to confirm with Delete or Remove. This cannot be undone.

A server that topics still use cannot be deleted. This counts every use: answers, embeddings, picture descriptions, and every other model a topic or GraphRAG Topic picks from this server. Instead of the button, the Danger Zone then says:

Still used by n topic(s): names. These topics use a model of this server, for example for answers, embeddings or picture descriptions. Choose another AI server in these topics first, then you can delete this one.

GraphRAG Topics appear in that list with (GraphRAG) after their name. Nothing is stopped or deleted while a topic still uses the server, so the downloaded models of a managed server stay safe.

Pipelines are not checked. A pipeline module keeps its own copy of the server’s address and key. If you delete a server that a pipeline still uses, pick another server in that module.

8. What a change reaches

Model lists in the pickers

The model pickers in topics, GraphRAG Topics and pipelines do not ask the AI server each time you open them. This app keeps the last model list of every server and shows it at once:

  • The app loads all lists again in the background every 10 minutes, and about 15 seconds after this app starts.

  • Each picker shows how old its list is, for example "Models on MyOllama · 4 min ago". The Refresh button in the picker loads the list now. Use it right after you downloaded a model.

  • When a server does not answer, the picker keeps showing the last list it got, with the reason ("Could not refresh …: … Showing the list from …").

  • When there is no list at all, the picker asks you to type the model name by hand: "The model list could not be retrieved. Enter the model details manually." Write the name exactly as the server expects it.

  • Save on the AI server form drops the stored list of that server. The next picker loads it fresh from the new address.

  • Opening the Models tab of a server always loads its list fresh, and the pickers then use it too.

The administrator of this app can change the interval with AI_SERVER_MODEL_LIST_REFRESH_INTERVAL (in seconds, default 600). A shorter interval shows new models sooner, but asks every server more often. The time limit for one list request is AI_SERVER_MODEL_LIST_TIMEOUT (default 10 seconds).

Topics and GraphRAG Topics

A topic learns the address, the provider and the API key of its AI servers when it starts. After you change Host, Port, Use TLS (HTTPS) or API Key of a server, stop and start every running topic and GraphRAG Topic that uses it. Until then they keep the old values.

Pipelines

Select AI Server & Model in the Pipeline Designer copies the address, the model name and the API key of the server into the module. A later change on the AI server does not reach these modules. Open each module that uses the server, pick the server again, and save the pipeline. See the Pipeline Editor.

9. Kubernetes

When this app runs in Kubernetes, a managed Ollama server works a little differently. See also the Kubernetes installation guide.

  • A start creates a Deployment and a Service named ollama-server-<id>, and a volume ollama-server-data-<id> for the downloaded models. See What the app creates in the cluster.

  • For a managed server, Host is not required. On start, the app sets the address inside the cluster by itself, with port 11434. Start a new managed server once before topics use it, because the address exists only after the first start.

  • Stop removes the Deployment. The Service and the volume stay.

  • Stop & Delete AI Server also removes the Service and the volume.

  • CPU and memory of a managed Ollama server are set by the administrator of this app, see CPU and memory. Larger models need more memory.

  • An external server must have an address that pods can reach. Never use localhost or host.docker.internal, see Reaching AI servers and Docling.

10. Permissions

Permission Allows

SERVER_VIEW (View Servers)

See Admin → AI Servers and the list. Check external servers with ⟳

SERVER_MANAGE (Manage Servers)

Add, edit, start, stop and delete AI servers. Shows Add AI Server, ▶, ■, ⚙ and the delete button

The form for adding and editing needs SERVER_MANAGE. Starting a server also needs a valid license. See Roles and Permissions.

11. Troubleshooting

Symptom Cause and fix

The first start stays in Downloading the Ollama image (several GB) for minutes

Normal on a new host. Wait, the step changes when the download is done

Start ends with "name" did not answer after starting

Read the application log. If the server was only slow to answer, click ▶ again. A port that another program uses also stops the start: choose another Port

Failed to start "name": There is no valid license.

Enter a license key under Admin → License

External server shows Error

The app cannot reach it, or the login was refused. Check host, port and Use TLS (HTTPS), and the API key. Then click ⟳

External server stays Created

It was never checked. Click ⟳

A server shows Error, but you fixed the problem

Errors are not re-checked automatically. Click ⟳ (external) or ▶ (managed)

Cannot resolve hostname '…'. Check the Host field.

The host name is wrong or unknown on this machine

Connection refused at …. Is the server running?

Nothing answers at that address and port. Start the server, or check the port

Authentication failed (HTTP 401). Check the API key configured for this AI server.

The key is wrong or missing. With Management API Key Env Var set, check that variable too

Server returned HTTP 404 — …

Often a wrong path. Enter the host without /v1, or paste the full address and let the form split it

A managed server works, but topics cannot reach it (Docker)

Use host.docker.internal as Host, not localhost. Then stop and start the topics

A model you just downloaded is missing in a picker

The picker shows its stored list. Click Refresh in the picker

A picker asks you to type the model name

The model list could not be loaded. Check that the server runs, or type the exact model name

The Models tab shows Server Not Running

The managed server is not running. Start it in the list

Topics still use an old key or address

Topics learn them when they start. Stop and start the topics

A pipeline still uses an old key or address

Pick the server again with Select AI Server & Model in each module, then save

Kubernetes: the status shows Running, but the model list stays empty

After two minutes the app shows Running even if the pod is not ready yet, for example while the image still downloads. Wait, then click Refresh on the Models tab

Kubernetes: topics cannot reach a managed server that was never started

The address is set on the first start. Start the AI server, then restart the topic

No delete button, only "Still used by …"

Topics still use the server. Choose another AI server in them first

No buttons in the list, no Add AI Server

Your role has only SERVER_VIEW. Ask an administrator for SERVER_MANAGE

Related