An on-premises LLM server: what do you need?

The measure that matters most: GPU memory

A language model has to fit entirely in GPU memory to be fast. The rule of thumb: the number of parameters times the number of bytes per parameter. With quantisation you store the weights in fewer bits, with little loss of quality for large models.

That is only the model itself. On top comes the KV cache: working memory per conversation, which grows with the length of the text and the number of users active at the same time. A 70B model in 4-bit runs on one 48 GB GPU, but with little room for long documents or many simultaneous questions.

How many users can one server handle?

Not every employee asks a question at the same moment. What counts is the number of concurrent requests and the length of the texts. An employee who has a forty-page contract summarised uses far more memory than someone who has a short email rewritten.

Modern inference software processes requests in batches, so one GPU serves several users at once. In practice:

• A pilot with a small team can run on one 48 GB datacentre GPU.
• An organisation of a few hundred employees, with peaks of dozens of concurrent questions and long documents, needs several GPUs in one server.
• Larger organisations scale with more GPUs per server or with several servers.

Always test with your own work: the length of your documents and the usage pattern of your teams determine the size, not the headcount alone.

Which models can you run?

On-premises works with open-weight models, whose weights are published: including Llama (Meta), Mistral and Qwen (Alibaba). When choosing, watch three things:

• The licence. Not every open-weight model may be used commercially without limits. Read the terms, especially as a large organisation.
• The language. Test the quality in the languages of your staff and customers, with your own kind of texts.
• The task. A smaller model is faster and cheaper for classifying and summarising; a larger one is better at reasoning and writing. Many organisations run more than one, per role or task.

Software, power and cooling

Inference software. For one user on a workstation, tools such as Ollama or llama.cpp are enough. For an organisation you need a server that handles many requests efficiently at the same time, such as vLLM, with an API your applications can call.

Power and cooling. A datacentre GPU easily draws 300 to 400 watts under load; a server with four GPUs reaches a few kilowatts together with the rest of the hardware. Check that your server room or rack can handle the power, the cooling and the weight. Many office server rooms are not set up for GPU servers.

Network. The server belongs in an isolated segment, reachable by your users and applications, without outbound connections you have not allowed yourself.

The part that is often forgotten: operations

Running a model is a weekend’s work for a handy engineer. A defensible AI environment for a whole organisation is something else. It needs:

• integration with your identity provider and rights per role;
• safeguards that stop sensitive patterns before they reach a model;
• logging and an audit trail, with export to your SIEM;
• model management: testing, rolling out and rolling back new versions;
• monitoring of usage, waiting times and hardware;
• a policy on how much insight the organisation gets into use.

That is exactly what werqly delivers: a dedicated AI node in your server room with werqly AI Control as the management console. Your IT department runs it, or werqly does with werqly Pro management. If you have no suitable server room, the node sits in a locked rack in our European datacentre, for you alone. Why organisations make that choice is in What is private AI?