Self-Hosted LLM for Enterprise: A Decision Test

A self-hosted LLM for enterprise use is the right call for a few workloads. For most, it is an expensive way to solve a problem you can solve with a contract.

The real question is not “cloud or on-prem.” It is “which data class am I sending to which model, and who has to be able to see it?”

Answer that first. Then run the four-part test below. If you fail two parts, do not self-host yet.

Why the Question Is Really About Data Class

Teams ask me whether they should self-host. I ask what data the model will touch.

A support bot that reads public help articles is a different problem than an agent that reads patient charts, trade secrets, or source code you cannot show anyone. The model is the same. The risk is not.

The pull toward on-prem is real. In a survey of 100 enterprise CIOs published June 10, 2025, a16z found open source model adoption was higher at larger enterprises, “where on-prem is still a major consideration.” They tied it to “data security and compliance considerations” and the ability to fine-tune. Note what they did not say. They did not say most enterprises had moved on-prem.

Sort your data into three rough classes:

• Public or low risk. Marketing copy, public docs, generic summaries. A managed API is fine.
• Sensitive but contractable. Customer records, financial data, health data. A managed service with the right contract and controls usually works. More on this below.
• Cannot leave your boundary. Data under a hard legal, contractual, or classified restriction. No vendor, no matter what the contract says. This is the real case for self-hosting.

Most of what enterprises call “too sensitive” lands in the middle class. That is where the decision gets interesting. For the sovereignty side of this, see my posts on sovereign AI for the enterprise and data sovereignty for AI agents.

Four Options on a Spectrum

Self-hosting is not one thing. It is a spectrum, and each step trades control for effort.

1. Public API. You send prompts to the model maker. Least effort. Least control over where data goes.
2. Managed model in your cloud account. Services like Amazon Bedrock or Azure OpenAI run the model inside a cloud you already trust. AWS states that model providers do not have access to the accounts that run Bedrock models, so they cannot see logs, prompts, or completions. Microsoft commits that customer prompts and completions are not used to train models without your permission.
3. Open-weight model on rented GPUs in your own cloud. You run the model on your own virtual machines, in your own network. You own the serving layer, the scaling, and the patching.
4. Open-weight model on hardware you own. On-prem, possibly air-gapped. Maximum control. Maximum effort. You also own the hardware risk.

Options 3 and 4 are what people mean by “self-hosted.” I would not jump to 4 unless a rule forces it. In my view, option 2 closes most of the gap for most buyers. If you are weighing open models against commercial ones, see open-source versus commercial LLMs.

The Decision Test for a Self-Hosted LLM for Enterprise

Score each part as pass or fail. A pass means self-hosting is justified on that axis.

1. Data

Pass if the data truly cannot leave your boundary, or if a regulator or customer contract says so in writing.

Fail if the objection is “it feels risky.” Feelings are not a requirement. Ask legal for the actual clause.

Here is a common trap. Teams assume regulated data forces on-prem. Often it does not. The HHS guidance on HIPAA and cloud computing says covered entities can use cloud services if they sign a business associate agreement (BAA) with the provider. The same page says HIPAA does not certify technology products. So “we handle health data” does not by itself mean “we must self-host.” It means “we need the right paper and the right controls.” For a deeper look at healthcare rules, see AI agents and healthcare compliance.

2. Volume

Pass if you send steady, high token volume that keeps GPUs busy most of the day.

Fail if your traffic is bursty, small, or still unproven.

This is the part that surprises people. A GPU costs the same whether it works or sits idle. An API charges only for what you use. So your cost per token on your own hardware depends almost entirely on how full you keep it.

Here is a simple way to see it. These numbers are made up to show the shape, not to price anything:

• Say a GPU node costs $10,000 a month, all in.
• If it serves 500 million tokens a month, that is $20 per million tokens.
• If it serves 100 million tokens, that is $100 per million tokens.

Same hardware. Five times the unit cost. Run this math with your own numbers before you buy anything.

GPU pricing also moves. In June 2025, AWS cut on-demand prices by 44 percent for P5 instances (H100 GPUs) and 33 percent for P4d instances (A100 GPUs). If you buy hardware, you carry that price risk. If you rent, you pick it up for free.

3. Team

Pass if you already run production infrastructure for GPU workloads, or you can hire for it and keep them.

Fail if the plan is “our platform engineer will figure it out.” That plan tends to stall.

A model server is a production service. It needs on-call coverage, upgrades, capacity planning, and security review. One widely used open serving engine is vLLM. Its original paper reported 2 to 4 times the throughput of earlier systems at the same latency. But a fast engine is still a thing you operate.

My rough view on staff, and this is my opinion, not a survey result: plan on at least two people who can own it, so one vacation does not take you down. One is not a team.

4. Latency and Control

Pass if you need response times, fixed model versions, or offline operation that a shared service cannot promise.

Fail if your only reason is “we want to pick the model.” You can usually pin a model version on a managed service too.

Self-hosting does give you one thing nothing else does: the model never changes unless you change it. For regulated workflows that need a frozen, tested model, that matters.

Scoring

Passes	What I’d do
0 or 1	Use a managed service. Revisit in a year.
2	Run a small pilot in your own cloud (option 3). Prove the math.
3 or 4	Self-hosting is justified. Plan the team before you buy hardware.

Data is a gate, not just a point. If the data fails and nothing else forces your hand, stop there.

Hidden Costs of Self-Hosting

The GPU bill is the visible part. These are the parts that show up later.

• Idle time. Nights and weekends, when no one is using the model, still cost money.
• Redundancy. One node is one failure. Real uptime needs a second one.
• People. The biggest line item, and the one budgets leave out. This is also why building in-house costs more than expected.
• Security work. The vLLM security guidance tells you to deploy on a dedicated, isolated network, and says traffic between nodes in a multi-node setup is insecure by default. That is a normal engineering task. It is still your task.
• Model upgrades and evaluation. New models arrive. Every swap needs testing against your own workload.
• License terms. Open weights are not terms-free. The Llama 3 license, for example, requires a separate license from Meta if you pass 700 million monthly active users. Most companies are nowhere near that, but read the license anyway.
• Quality gap. Open models may trail the best commercial ones on your task. Test it. Do not assume.

On quality, the a16z survey noted that as capabilities improve, most enterprises were not seeing as much return on fine-tuning as the year before, and mainly chose open source for “highly cost-sensitive use cases.” Cost-sensitive and high-volume go together. That fits the volume test above.

Hybrid Patterns

You do not have to pick one lane. Here are patterns I see working.

Route by data class. Low-risk prompts go to a managed API. Sensitive prompts go to a model in your own cloud. A small gateway decides based on a tag on the request.

Redact first. Strip names, IDs, and account numbers before text leaves your boundary. Send the cleaned text out. Put the real values back after.

Self-host the narrow, call out for the hard. Run a small model on-prem for high-volume, simple tasks like classification. Send rare, hard reasoning tasks to a managed service.

Check the retention terms. Before you call a managed API sensitive-data-safe, read how long it keeps your prompts. OpenAI’s enterprise privacy page says API data is not used for training by default and may be kept up to 30 days to detect abuse, with zero data retention available on request for qualifying cases. Azure runs abuse monitoring with human review by default, and approved customers can apply to modify it. Those settings can close the gap without a single GPU. For more, see the glossary entries on zero data retention and data sovereignty.

What the Pilot Should Prove

A self-hosting pilot is not “can we get a model running.” Anyone can. It should prove five things:

1. Quality. On 50 to 100 real examples from your workflow, does the open model meet the bar your current model meets?
2. Unit cost. What is the real cost per million tokens at your actual traffic, with staff time included?
3. Utilization. Can you keep the hardware busy enough to beat the managed price?
4. Operations. Can your team deploy, roll back, and recover a failed node without help?
5. Security review. Does the setup pass your security team, including network isolation and access logging?

Set the pass line before you start. Write it down. If you cannot name a number for each of the five, you are not ready to start.

Keep the pilot small. Rent GPUs by the month. Do not buy hardware for a pilot.

Exit Paths

Plan the way out on day one.

• Keep an API-compatible interface. vLLM ships an OpenAI-compatible server, so your applications can swap the backend with a config change. Build against that, not against the engine.
• Keep your evaluation set. Your test cases are what let you compare any new model or vendor fast.
• Avoid long hardware commitments early. Rent first. Buy only after a year of stable usage.
• Document the runbook. If the one engineer who built it leaves, the service should still run.

If the pilot fails the cost test, move to option 2 and you lose a few weeks. If you bought racks first, you lose a budget year.

Bottom Line

Start with the data class, not the infrastructure.

Self-hosting wins when the data cannot leave, the volume is steady, and you already have people who run GPU services. It loses when any of those is missing.

My recommendation is to default to a managed model in your own cloud account, and self-host only the workloads that pass at least three of the four tests.

Next step: list your top five AI workloads, tag each with a data class, and score it against the table above.

FAQ

Should we self-host an LLM? Only if the data cannot leave your boundary, your volume keeps GPUs busy, and you have a team that can run them. If you pass fewer than three of the four tests, use a managed service in your own cloud account and revisit later.

Which data classes require self-hosting? Data under a hard legal, contractual, or classified restriction that forbids any outside processor. Regulated data is not automatically in this class. HHS guidance allows HIPAA-covered data in the cloud with a signed BAA, so check the actual clause before you assume.

What does it cost to run a self-hosted LLM? The GPUs are only part of it. Add idle time, a second node for uptime, security work, upgrades, and staff time. Your unit cost depends mostly on utilization, so a node that is busy 20 percent of the day costs far more per token than one busy 80 percent of the day.

What staff do we need to self-host? At minimum, two people who can deploy, monitor, and recover a GPU-backed service, so coverage survives time off. That is my view as an operator, not a survey figure. Add security review time on top.

What are the alternatives to full self-hosting? Run a managed model inside your cloud account, route sensitive prompts to a private model while sending low-risk ones to an API, redact data before it leaves your network, or negotiate zero data retention with your provider. Each one reduces exposure without owning GPUs.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect