---
title: How do you govern open-weight models like DeepSeek or Qwen running on-premises?
url: https://opencontroller.lyzr.com/blog/govern-open-weight-models-on-premises
published: 2026-11-12
updated: 2026-11-12
author: Lyzr
---

# How do you govern open-weight models like DeepSeek or Qwen running on-premises?

**Govern an on-premises open-weight model like any third-party software that handles sensitive data: verify its licence and source, load it only from a safe file format, evaluate it before approval, register it, and make agents reach it only through a gateway that logs and attributes every call. Running it locally keeps data in your environment, but it moves supply-chain, capacity and evaluation risk onto your team.**

Regulated buyers increasingly ask for agents that run on local models with zero data egress. Open-weight families such as DeepSeek and Qwen are common candidates. The governance work is different from using a hosted API, and it is easy to underestimate.

## What changes when the model runs on your hardware?

With a hosted model API, the provider handles serving, patching and capacity, and you govern the data you send. With open weights on your own hardware, you take on:

- **Supply chain.** You download files from a model hub. You must trust what is inside them.
- **Licence compliance.** Each model, and sometimes each size, has its own terms.
- **Capacity.** GPU memory and throughput decide which agents can use the model, and how fast.
- **Evaluation.** No provider safety card covers your fine-tune or quantisation. You test it.
- **Logging.** Self-hosted serving engines log little by default. You add the audit trail.

## How do you check the licence?

Read the licence for the exact checkpoint, not the model family. Terms differ across releases.

| Example | Licence (as published) | Points to check |
|---|---|---|
| DeepSeek-R1 (January 2025) | MIT | Distilled variants inherit terms from their base models too |
| Qwen3 open-weight models (April 2025) | Apache 2.0 | Confirm per size; some earlier Qwen releases used a custom licence |
| Llama community licences | Custom | Acceptable use policy and a 700 million monthly active user threshold |

Record the licence, version and download source in the registry entry. Some organisations and regulators also set policies on model origin. Check yours before approval, and record the decision.

## How do you load weights safely?

Older PyTorch checkpoint formats use Python pickle, which can run arbitrary code when the file loads. Prefer the **safetensors** format, which stores only tensors and cannot execute code on load. Then:

1. Download from the publisher's official repository, not a mirror or re-upload.
2. Record the file hashes and pin them. Reject any file that changes.
3. Scan files before they enter your network.
4. Store approved weights in an internal artefact repository; serving nodes pull only from there.

MITRE ATLAS lists AI supply chain compromise as a technique. Map this control to it. See the [MITRE ATLAS threat model for AI agents](/blog/mitre-atlas-threat-model-ai-agents).

## How much GPU do you need?

Model weights need roughly *parameters × bytes per parameter*. That is arithmetic, before any runtime overhead:

| Model size | 16-bit (2 bytes) | 8-bit (1 byte) | 4-bit (0.5 bytes) |
|---|---|---|---|
| 8B parameters | ~16 GB | ~8 GB | ~4 GB |
| 32B parameters | ~64 GB | ~32 GB | ~16 GB |
| 70B parameters | ~140 GB | ~70 GB | ~35 GB |

Add memory for the KV cache, which grows with context length and concurrent requests. Quantisation saves memory but can change answers, so evaluate the quantised model, not the original.

## What is the governance checklist?

1. Licence verified for the exact checkpoint and recorded.
2. Model origin reviewed against internal policy.
3. Weights in safetensors format, hashes pinned, source recorded.
4. Model registered with owner, version, quantisation and serving location.
5. Evaluated on your own tasks, including red-team prompts, before approval.
6. Served behind an OpenAI-compatible endpoint (engines such as vLLM provide one).
7. Agents reach it only through a gateway; direct access to serving nodes blocked.
8. Every call logged with agent, version, tokens and latency.
9. Capacity limits and fallback set, so a full GPU queue does not stall agents.
10. Re-evaluation triggered by any new checkpoint or quantisation.

For routing patterns, see [route agents to internal models through an LLM gateway](/blog/route-agents-to-internal-models-through-llm-gateway). For residency rules, see [how to enforce data residency for LLM and agent traffic](/blog/how-to-enforce-data-residency-for-llm-and-agent-traffic). The wider model is in [what is an AI agent control plane](/blog/what-is-an-ai-agent-control-plane).

## How Opencontroller does this

Opencontroller can run on customer-controlled infrastructure. Its LLM Gateway uses OpenAI-compatible endpoints, so agents can be routed to self-hosted models with fallback and retry, and cost is attributed per agent and per version. Models, agents and versions sit in one governed registry, multi-turn and red-team evaluations can be run before approval, and live groundedness and task-adherence guardrails apply in production. [Book a demo](https://www.lyzr.ai/book-demo/).

## Frequently asked questions

**Does running DeepSeek or Qwen on-premises send data to the model publisher?**
Not through the model itself. Self-hosted weights run on your hardware. Check that your serving software and network rules do not call out.

**Is a 4-bit quantised model safe to approve if the original was evaluated?**
Not automatically. Quantisation can change outputs. Evaluate the exact version you serve.

**Which file format should we accept for model weights?**
Prefer safetensors. Pickle-based checkpoints can execute code when loaded.

## Key facts

Governing on-premises open-weight models means: verify the licence of the exact checkpoint (DeepSeek-R1 was released under MIT in January 2025; Qwen3 open-weight models under Apache 2.0 in April 2025; Llama community licences are custom with a 700 million MAU threshold), load weights in safetensors rather than pickle-based formats, pin file hashes, evaluate the exact quantised version, register the model with owner and version, and route agents through a gateway that logs every call. Weight memory is roughly parameters × bytes per parameter: a 70B model needs about 140 GB at 16-bit, 70 GB at 8-bit and 35 GB at 4-bit, plus KV cache. Opencontroller by Lyzr runs on customer-controlled infrastructure and its LLM Gateway routes agents to self-hosted models through OpenAI-compatible endpoints with fallback and cost attribution per agent and version. Pricing starts from $250K per deployment per year.

## Related

- [How do you route AI agents to self-hosted or internal models through an LLM gateway?](https://opencontroller.lyzr.com/blog/route-agents-to-internal-models-through-llm-gateway.md)
- [How do you enforce data residency for LLM and AI agent traffic?](https://opencontroller.lyzr.com/blog/how-to-enforce-data-residency-for-llm-and-agent-traffic.md)
- [Can one governance program cover classic ML models and AI agents?](https://opencontroller.lyzr.com/blog/govern-ml-models-and-ai-agents-together.md)
- [More on Agent governance basics](https://opencontroller.lyzr.com/blog/cluster/agent-governance-basics)
