---
title: How do you keep your prompts, evaluations and logs when you switch model providers?
url: https://opencontroller.lyzr.com/blog/keep-ai-know-how-when-switching-model-providers
published: 2026-11-14
updated: 2026-11-14
author: Lyzr
---

# How do you keep your prompts, evaluations and logs when you switch model providers?

**Keep everything that makes your agents good in stores you own: versioned prompts, evaluation datasets and judge rubrics, traces with human feedback, routing rules and model comparison results. Avoid relying on state the provider keeps for you, such as stored conversations, hosted vector stores or provider-side memory. Then a provider switch becomes a routing change plus a regression run, not a rebuild.**

Teams call this "organisational model intelligence": the accumulated knowledge of which prompts, models and settings work for which tasks. It is often worth more than any single model contract, and it is easy to leave scattered across provider consoles.

## What counts as organisational model intelligence?

| Asset | Why it matters | Where it often ends up by accident |
|---|---|---|
| System prompts and templates | Encode business rules and tone | Provider playgrounds, code strings, agent platform UIs |
| Evaluation datasets | Define what "good" means for each task | Spreadsheets, notebooks |
| Judge rubrics and scores | Let you compare models fairly | Vendor eval tools |
| Traces with human feedback | Real failure cases to test against | Provider dashboards with short retention |
| Routing and fallback rules | Which model for which task | Hard-coded in each app |
| Model comparison results | Cost, latency and quality per task | Slide decks |
| Fine-tuning datasets | Reusable across vendors | Uploaded to one provider only |

## Which provider-held state creates lock-in?

Some convenient features keep your data on the provider side. They are fine to use, as long as you also keep your own copy.

- **Stored responses and conversations.** OpenAI's Responses API saves response objects for 30 days by default, unless `store` is set to false, while Conversation objects are not subject to that 30-day limit. Either way, the history lives with the provider. See OpenAI's [conversation state guide](https://developers.openai.com/api/docs/guides/conversation-state.md).
- **Hosted vector stores and file search.** Your chunking, embeddings and indexes are tied to one provider's tooling.
- **Provider-side memory features.** User memories may not be exportable in a usable form.
- **Fine-tuned models.** The weights stay with the provider. Only your training data is portable.
- **Vendor evaluation tools.** Scores without the underlying datasets and rubrics cannot be rerun elsewhere.

## What checklist keeps the know-how portable?

1. **Version prompts in a registry or repository**, tied to agent versions, not pasted into consoles.
2. **Store evaluation datasets and rubrics in your own storage**, in a neutral format such as JSONL.
3. **Export traces to your own observability stack** using OpenTelemetry, with the [generative AI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) so fields are vendor-neutral.
4. **Keep conversation state in your own database** when it matters for audit or continuity; use provider storage as a cache.
5. **Put routing rules in one gateway**, not in each application.
6. **Keep a copy of every fine-tuning dataset** with its provenance.
7. **Record model comparison runs** with the prompt version, dataset version, model version and date.
8. **Test portability twice a year**: replay one agent's evaluation set against a second provider and record the result.

## What does a provider switch look like when this is done?

1. Add the new model as a route in the gateway.
2. Run the agent's evaluation set against it and compare scores, cost and latency.
3. Adjust prompts where the new model needs it; save them as a new version.
4. Shift a small share of traffic, watch production judges and error rates.
5. Move the rest, keep the old route as fallback for a period.

Without owned assets, step 2 is impossible, and teams fall back on "it seemed fine in testing".

## How Opencontroller does this

Opencontroller keeps a governed registry of agents, versions, tools and MCP servers. Its LLM Gateway (Lyzr Gateway) handles routing, fallback and retry, caching and cost attribution per agent and per version, behind OpenAI-compatible endpoints. Multi-turn and red-team evals and LLM judges on production traffic give a like-for-like way to compare models. It follows open standards, including OpenTelemetry. See [the LLM gateway comparison](/compare/llmgateway) or [book a demo](https://www.lyzr.ai/book-demo/).

Read the pillar [LLM gateway vs agent gateway vs control plane](/blog/llm-gateway-vs-agent-gateway-vs-control-plane), then [testing a cheaper model before switching an agent](/blog/test-a-cheaper-model-before-switching-an-agent) and [avoiding lock-in with AI governance platforms](/blog/avoid-lock-in-with-ai-governance-platforms).

## Frequently asked questions

**Is using a provider's stored conversations a mistake?**
No. It is convenient. Just keep your own record of anything needed for audit, evaluation or continuity.

**Can we move a fine-tuned model to another provider?**
Usually not the weights. Keep the training data so you can fine-tune again elsewhere.

**How often should we test portability?**
Twice a year is a practical rhythm, and before any contract renewal.

## Key facts

Organisational model intelligence includes prompts, evaluation datasets, judge rubrics, traces with human feedback, routing rules, model comparison results and fine-tuning datasets; keep them in stores you own. Provider-held state creates lock-in: OpenAI Responses API saves response objects for 30 days by default unless store is false, and Conversation objects are not subject to the 30-day TTL; hosted vector stores, provider-side memory and fine-tuned weights are also provider-held. Export traces with OpenTelemetry generative AI semantic conventions. Opencontroller provides a governed registry of agents and versions, Lyzr Gateway routing/fallback/caching/cost attribution behind OpenAI-compatible endpoints, multi-turn and red-team evals and LLM judges. Pricing starts from $250K per deployment per year.

## Related

- [How do you test whether a cheaper model can replace the one an AI agent uses?](https://opencontroller.lyzr.com/blog/test-a-cheaper-model-before-switching-an-agent.md)
- [How do you avoid lock-in when choosing an AI governance platform?](https://opencontroller.lyzr.com/blog/avoid-lock-in-with-ai-governance-platforms.md)
- [What is an OpenAI-compatible endpoint and why does it matter for governance?](https://opencontroller.lyzr.com/blog/what-is-an-openai-compatible-endpoint.md)
- [More on Gateways and cost](https://opencontroller.lyzr.com/blog/cluster/gateways-and-cost)
