Hardware for your own AI server: GPUs, memory and model size
Short answer
GPU memory decides which language model you can run. As a rule of thumb, each billion parameters takes about 2 GB at 16-bit, 1 GB at 8-bit and half a GB at 4-bit, plus room for ongoing conversations. A model with 70 billion parameters therefore fits on two datacentre GPUs with 48 GB each. On top of that, plan for a rack, a UPS, cooling and several kilowatts of power.
GPU memory is the limiting factor
A language model is loaded entirely into the GPU’s memory while in use. How much space that takes depends on the number of parameters and the precision they are stored in:
| Model | 16-bit | 8-bit | 4-bit |
|---|---|---|---|
| 8 billion parameters | approx. 16 GB | approx. 8 GB | approx. 5 GB |
| 32 billion parameters | approx. 64 GB | approx. 32 GB | approx. 18 GB |
| 70 billion parameters | approx. 140 GB | approx. 70 GB | approx. 40 GB |
Fewer bits (quantisation) save a lot of memory, with a limited loss of quality that varies by model and task. Which model suits which task is covered in Comparing open-weight models.
How many users can one server handle?
Besides the model itself, memory is needed for the conversations running at that moment: the context of each question, including attached documents. The longer the documents and the more people working at once, the more memory that takes.
Modern inference software such as vLLM processes requests from many users at once in batches. That lets one server serve hundreds of employees: not everyone asks a question at the same moment. What matters is the peak, for example the morning a whole department has case files summarised.
What it asks of your server room
- Rack space: a GPU server is usually a 19-inch server of two to four rack units.
- Power: expect several kilowatts under load, on its own circuit and behind a UPS.
- Cooling: that power becomes heat. A server cabinet in an office room without climate control is not built for it.
- Noise: GPU servers are loud; they do not belong next to a desk.
- Network: a segmented zone, and only the outbound connections needed for management and updates.
If that space is not available, the node can sit in a locked rack in a datacentre, still yours alone. See Running your own LLM server on-premises.
How werqly chooses the hardware
werqly supplies the node as a whole: GPU server, models, software and werqly AI Control, sized to the number of employees and the kind of work. You do not have to choose GPUs or install models yourself. What the hardware choice means for cost is covered in What does private AI cost? See also What is private AI?
Frequently asked questions
Questions on this topic
How much GPU memory does a 70B model need?
About 140 GB at 16-bit precision, 70 GB at 8-bit and around 40 GB at 4-bit, plus extra memory for the conversations running at the same time. In practice a quantised 70B model runs on two datacentre GPUs with 48 GB each.
Can your own AI server run on an ordinary office PC?
Small models can, for a single user. For an organisation with dozens or hundreds of users, larger models and long documents, you need a datacentre GPU server, with suitable power, cooling and security.
How much power does an AI server use?
A server with several datacentre GPUs draws several kilowatts under load. That energy is released as heat, so cooling and a dedicated power circuit with a UPS belong in the plan.
What if we have no suitable server room?
Then the node can sit in a locked rack in a datacentre. It stays dedicated to your organisation: no shared environment and no external AI vendor receiving the text.
Sources
This article is general information, not legal advice. Rules and guidance change; check the sources listed or ask your lawyer or DPO if in doubt.