// Glossary · technical

Local LLM

A large language model deployed on private hardware (Llama 3, Qwen, Mistral) rather than accessed via a hosted API like OpenAI or Anthropic.

A Local LLM is a large language model whose weights live on hardware the buyer owns and whose inference calls execute on that same hardware. The model can be an open-weight release (Llama 3.1 8B/70B/405B, Mistral Large, Mixtral 8x22B, Qwen 2.5, Phi-3, DeepSeek), or a private fine-tune of any of those against buyer-specific data. The defining property is that the weights, the data, and the compute all sit inside one network perimeter. EOI deploys this pattern as part of the local agent setup engagement.

Where Local LLMs sit in the stack matters. A hosted API call goes: buyer app sends request over the internet to vendor endpoint, vendor runs model on vendor hardware, response returns over the internet. A Local LLM call goes: buyer app sends request to internal IP, GPU server runs model on local hardware, response returns inside the LAN. The end-user experience is the same. The compliance and custody story is completely different, which is why regulated teams in fintech and healthcare ask for it specifically.

Model selection per workload matters more than picking 'the best one.' Llama 3.1 70B is the workhorse for general English reasoning. Mistral Large performs better on European languages. Qwen 2.5 is the right pick for multilingual workloads including Cantonese and Mandarin. Phi-3 mini runs CPU-only for lightweight workloads. DeepSeek competes hard on cost-to-quality for technical tasks. The right model is the smallest one that hits the quality bar for the specific workload, not the biggest available. See On-Device AI Agent for the full runtime pattern.

// Examples
  • A Series B fintech runs Llama 3.1 70B on dual L40S to draft KYC review notes. The model never sees a customer name outside the production network. Compliance signed off without a DPA conversation because no vendor exists in the data path.
  • A European insurer picks Mistral Large for German and French policy reading. Quality on language-specific tasks beats the 70B Llama equivalent by 12 to 18% on internal evaluation, at lower cost.
  • A Hong Kong investment firm runs Qwen 2.5 32B for Cantonese-language meeting summaries. The model handles the Cantonese-Mandarin code-switching that breaks most English-first models, and stays inside the firm network for client confidentiality.
// Common questions
What models are supported on a local install?
Llama 3 and 3.1 (8B, 70B, 405B), Mistral Large and Mixtral, Qwen 2.5, Phi-3, DeepSeek, and any model on Hugging Face that fits the runtime. EOI defaults to open-weight models for license clarity. Private fine-tunes of any of those run locally without the fine-tune leaving the network.
How does a Local LLM compare to GPT-4 or Claude on quality?
For specific workloads, a tuned 70B local model matches or beats the hosted frontier models. For broad general-purpose reasoning, the frontier hosted models still lead. The right comparison is not "is Llama as smart as GPT-4" but "does the local model handle this workload at the quality bar I need." For most departmental workflows the answer is yes.
How much does the hardware cost?
A single H100 server lands around $30K to $40K depending on configuration. Dual L40S is roughly $20K. CPU-only deployments running quantized smaller models can run on existing servers with no new hardware spend. The breakeven against hosted API costs typically lands between 5 and 15 million tokens a month.
Who maintains the model after install?
Two options. EOI runs the optional monthly retainer that handles model upgrades, security patches, prompt tuning, and new connector requests. Or the customer IT team takes the handoff at week eight and runs it with the EOI runbook. Most regulated clients pick the retainer because staying current on model releases is real work.
// Related terms
// Ready to ship?

EOI runs fractional AI departments for funded teams under 50. Sales, Content, Ops, Support. Live in 14 days on a monthly retainer.

Apply for a sprint →See all departments →