Open Source Models

Why Open Source Matters

Closed-API models (Claude, GPT-4o) are powerful but come with trade-offs: your data leaves your machine, you pay per token, and you depend on a vendor. Open source models give you control.

┌─────────────────────────────────────────────────────────────┐
│ Reasons to use open source:                                  │
│                                                              │
│ 1. PRIVACY     — Data never leaves your machine              │
│ 2. COST        — Free inference after hardware investment     │
│ 3. CUSTOMIZATION — Fine-tune on your own data                │
│ 4. NO VENDOR LOCK-IN — Switch models freely                  │
│ 5. OFFLINE     — Works without internet                      │
│ 6. COMPLIANCE  — Meet data residency requirements            │
└─────────────────────────────────────────────────────────────┘

---

Major Open Source Models

Llama (Meta)

The most popular open source model family. Meta releases weights freely for research and commercial use.

Why it's popular: Best balance of size, capability, and license terms. Huge community with fine-tuned variants for every use case.

Mistral

European AI lab. Known for efficient, high-performing models.

What's MoE (Mixture of Experts)? The model has 176B total parameters but only activates 44B for each token. This means near-large-model quality at medium-model speed. Think of it as 8 specialist sub-models that hand off to whichever is most relevant.

DeepSeek

Chinese AI lab. Known for extremely cost-efficient training and strong reasoning.

Why notable: DeepSeek-R1 was trained at a fraction of the cost of comparable models and performs competitively with o1 on reasoning benchmarks. Open weights.

Gemma (Google)

Google's open models. Small but well-optimized.

Best for: Developers who want a small, capable model from a trusted source.

Phi (Microsoft)

Microsoft's small language models. Surprisingly capable for their size.

Why notable: Phi models demonstrate that training data quality matters more than raw size. They consistently outperform larger models on benchmarks.

Qwen (Alibaba)

Alibaba's model family. Strong multilingual support and competitive performance.

Best for: Multilingual applications, especially Chinese-English. Qwen-Coder is one of the best open code models.

---

Running Models Locally

Ollama (Recommended for Beginners)

The easiest way to run open models on your machine:

bash
# Install
curl -fsSL https://ollama.ai/install.sh | sh

# Run a model (downloads automatically on first use)
ollama run llama3.2        # 8B model, ~5GB download
ollama run codestral       # 22B code model
ollama run deepseek-r1:8b  # 8B reasoning model

# Use as an API (OpenAI-compatible)
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.2",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Why Ollama: One command to download and run. Exposes an OpenAI-compatible API, so tools like Cline and Continue work with it out of the box.

LM Studio

Desktop app with a GUI. Good for non-terminal users:

• Browse and download models from a catalog
• Chat interface built in
• Local API server (OpenAI-compatible)
• Adjust parameters (temperature, context length) visually

llama.cpp

The underlying engine that Ollama and many other tools use. For maximum control:

bash
# Build from source
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && make

# Run a model
./main -m models/llama-3.2-8b-q4_K_M.gguf \
       -p "Write a Python function to sort a list" \
       -n 256

When to use: Custom deployments, embedding in applications, maximum performance tuning.

vLLM

High-performance serving for production deployments:

bash
pip install vllm

# Serve a model with OpenAI-compatible API
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.2-8B-Instruct \
  --port 8000

When to use: Serving models to multiple users, production APIs, when you need high throughput.

---

Hardware Requirements

Practical guidelines:

• MacBook with 16GB RAM: Run 7-8B models comfortably
• MacBook with 32GB RAM: Run 13B models, try 34B quantized
• MacBook with 64GB+ RAM: Run 70B models quantized
• Gaming PC with RTX 4090 (24GB): Run up to 34B models with GPU acceleration
• No GPU? CPU inference works for 7-8B models, just slower (~5-15 tokens/sec)

---

Quantization

Quantization reduces model size by using lower precision numbers. It trades a small amount of quality for significant memory and speed gains.

Full precision (FP16):  Each parameter = 16 bits
                        70B model = ~140 GB

INT8 quantization:      Each parameter = 8 bits
                        70B model = ~70 GB
                        Quality: ~98% of original

INT4 quantization:      Each parameter = 4 bits
                        70B model = ~35 GB
                        Quality: ~93-95% of original

Common Quantization Formats

Rule of thumb: Q4_K_M is the sweet spot. Below Q3, quality degrades noticeably. Prefer running a smaller model at higher quantization over a larger model at extreme quantization.

---

When to Use Open Source vs API

Decision Framework

Is privacy critical (healthcare, legal, defense)?
  YES → Open source (local deployment)

Is the task narrow and high-volume (classification, extraction)?
  YES → Fine-tuned open source model (cheaper at scale)

Do you need maximum quality on complex reasoning?
  YES → API (Claude Opus, o3)

Are you prototyping or low volume?
  YES → API (faster to start, pay per use)

Do you need to work offline?
  YES → Open source

Budget for 1M+ API calls/month?
  NO → Open source
  YES → API (if quality matters more than cost)

---

The Practical Starting Point

If you're new to open source models:

1. Install Ollama — one command, works on Mac/Linux/Windows
2. Run ollama run llama3.2 — download and chat with an 8B model
3. Connect to your tools — Cline, Continue, or any OpenAI-compatible client can point to localhost:11434
4. Try different models — ollama run codestral for code, ollama run deepseek-r1:8b for reasoning
5. Scale up if needed — bigger models, GPU acceleration, vLLM for production

---

Resources

• 🔗 Ollama
• 🔗 LM Studio
• 🔗 llama.cpp
• 🔗 vLLM
• 🔗 Hugging Face Open LLM Leaderboard
• 🔗 Meta Llama
• 🔗 Mistral AI
• 🔗 DeepSeek
• 📄 GGUF Format Explanation

---

Previous: 12 - AI Coding Assistants | Next: 14 - AI APIs & SDKs