Skip to content
ML ResearchLocal LLM inference & benchmarking

offline-llm-engineering

/Case Study

Offline LLM inference pipeline benchmarking Llama 3.2 3B, Phi-4 Mini, and Mistral 7B on a consumer Intel Mac — zero cloud dependency. Covers CLI streaming, automated benchmarks, Pydantic structured outputs, and a 180-run model comparison study.

Category
ML Research
Type
Local LLM inference & benchmarking
Core stack
Python · Ollama · Pydantic
Status
Code on GitHub
/06 · ML ResearchGitHub

300+ inference calls. Real numbers. No cherry-picking.

/What this is

Everything runs offline. No API keys. No data leaves the machine.

  1. CLI tool: stream responses from any of three models, measure TTFT and latency in real time.
  2. Inference benchmarking: automated 60-run benchmark across models and temperatures, exported to CSV.
  3. Structured outputs: JSON schema enforcement with Pydantic validation and retry logic.
  4. Model comparison study: 180 runs across 30 standardized prompts in 6 categories with quality analysis.

/Why local inference matters

  1. Privacy: HIPAA, GDPR, and SOC 2 often prohibit sending sensitive data to third-party APIs.
  2. Latency: cloud round-trips add 200–600ms before the first token; local inference removes the network.
  3. Cost at scale: per-token billing vs fixed GPU capital; breakeven often at 3–6 months at high volume.
  4. Edge deployment: factory, medical, and field systems need self-contained on-device models.

/Phase 5 — Full comparison study (180 runs)

30 prompts × 3 models × 2 temperatures · Intel MacBook Pro · 6-Core i7 · CPU-only

ModelTempAvg TTFTAvg latencyWords/secVerdict
Llama 3.2 3B0.011.76s38.23s1.3Best for CPU
Llama 3.2 3B0.712.45s38.42s1.1Consistent
Phi-4 Mini0.012.65s52.07s1.2Good at T=0
Phi-4 Mini0.724.83s120.70s0.7Degrades at T=0.7
Mistral 7B0.023.79s71.69s0.6Slow, needs GPU
Mistral 7B0.734.26s106.10s0.4Unusable on CPU

/Cold vs warm inference (Llama 3.2 3B)

StateTTFTTotal latency
Cold (loading from disk)8.66s12.37s
Warm (resident in RAM)0.41s4.08s
Speedup21×3×

/Key findings

05 points
  1. Llama 3.2 3B is the only model suitable for interactive CPU deployment — ~38s average latency, best instruction compliance.

  2. Phi-4 Mini has a temperature cliff

    At T=0.7 latency triples and training data leakage can appear. Use T=0.0 with conciseness instructions.

  3. Mistral 7B needs a GPU

    0.4 words/sec at T=0.7 on CPU due to memory bandwidth limits.

  4. Cold start costs 21× more than warm inference

    Production systems must pre-warm models on startup.

  5. Quality and speed are orthogonal

    Speed benchmarks alone cannot substitute for quality evaluation.

/Models

ModelParametersCreatorDiskNotes
llama3.2:3b3BMeta2.0 GBBest overall for CPU
phi4-mini3.8BMicrosoft2.5 GBUse T=0.0 only
mistral:7b7BMistral AI4.4 GBNeeds GPU for interactive use

/Concepts demonstrated

ConceptWhere
Local vs cloud inference tradeoffsArchitecture, Phase 1
GGUF quantizationModel setup, technical report
Time to first token (TTFT)CLI, Phases 2–3
Cold start vs warm inferencePhase 2, 21× speedup
Pydantic schema validationPhase 4
Training data leakagePhase 5 Phi-4 finding
Prompt injection defensecomparison_study.py CONCISE_SUFFIX