23 - Service Mesh & Advanced Patterns

What Is a Service Mesh?

A dedicated infrastructure layer for handling service-to-service communication in microservices. Adds a sidecar proxy to every pod that handles:

• Mutual TLS (mTLS) -- encrypted communication between services
• Traffic management -- retries, timeouts, circuit breaking
• Observability -- metrics, traces, logs for all traffic
• Traffic splitting -- canary, blue-green, A/B testing
• Rate limiting and access control

Without vs With Service Mesh

Without mesh:
App A ──HTTP──► App B
(app code handles retries, TLS, tracing)

With mesh:
App A → Proxy A ──mTLS──► Proxy B → App B
        (handles retries, TLS, tracing, circuit breaking)

Popular Service Meshes

Istio Architecture

┌─── Control Plane (istiod) ─────────────────────┐
│  Pilot: service discovery, traffic rules       │
│  Citadel: certificate management (mTLS)        │
│  Galley: config validation                     │
└────────────────────┬───────────────────────────┘
                     │ pushes config
                     ▼
┌─── Data Plane ──────────────────────────────────┐
│                                                 │
│  ┌─── Pod ─────┐    ┌─── Pod ─────┐             │
│  │ App   Envoy │◄──►│ App   Envoy │             │ 
│  │       Proxy │    │       Proxy │             │
│  └─────────────┘    └─────────────┘             │
│                                                 │
│  Every pod gets an Envoy sidecar automatically  │
└─────────────────────────────────────────────────┘

Install Istio

bash
istioctl install --set profile=default
kubectl label namespace default istio-injection=enabled
# Now all new pods in this namespace get Envoy sidecars

Traffic Management with Istio

yaml
# VirtualService: route traffic
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: api
spec:
  hosts:
    - api
  http:
    - match:
        - headers:
            x-canary:
              exact: "true"
      route:
        - destination:
            host: api
            subset: v2
    - route:
        - destination:
            host: api
            subset: v1
          weight: 90
        - destination:
            host: api
            subset: v2
          weight: 10    # 10% canary

---
# DestinationRule: define subsets
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: api
spec:
  host: api
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 30s    # Circuit breaker
  subsets:
    - name: v1
      labels:
        version: v1
    - name: v2
      labels:
        version: v2

Advanced Kubernetes Patterns

Operator Pattern

An Operator is a custom controller that manages complex applications:

┌─── Operator ─────────────────────────────┐
│  Custom Controller + Custom Resource     │
│                                          │
│  Watch: PostgresCluster CRD              │
│     ↓                                    │
│  Reconcile: Create StatefulSet,          │
│    Services, ConfigMaps, backups,        │
│    replication, failover                 │
└──────────────────────────────────────────┘

yaml
# Custom Resource (what users create)
apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: production-db
spec:
  image: registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-16.1
  postgresVersion: 16
  instances:
    - replicas: 3
      dataVolumeClaimSpec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 100Gi
  backups:
    pgbackrest:
      repos:
        - name: repo1
          schedules:
            full: "0 1 * * 0"
            incremental: "0 1 * * 1-6"

The operator handles all the complexity (replication, failover, backups) behind a simple YAML API.

Popular Operators

Custom Resource Definitions (CRDs)

Extend the Kubernetes API with your own resources:

yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: backups.myapp.example.com
spec:
  group: myapp.example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                schedule:
                  type: string
                retention:
                  type: integer
  scope: Namespaced
  names:
    plural: backups
    singular: backup
    kind: Backup

Multi-Cluster Patterns

Federation

┌─── Cluster 1 (US-East) ───┐    ┌─── Cluster 2 (EU-West) ───┐
│  api-deployment (3 pods)  │    │  api-deployment (3 pods)  │
│  db-statefulset           │    │  db-statefulset (replica) │
└───────────┬───────────────┘    └───────────┬───────────────┘
            │                                │
            └────── Global Load Balancer ────┘
                        │
                    DNS-based routing
                  (latency or geo)

Tools for Multi-Cluster

FinOps / Cost Optimization

yaml
# Spot/Preemptible nodes for non-critical workloads
affinity:
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 80
        preference:
          matchExpressions:
            - key: karpenter.sh/capacity-type
              operator: In
              values: ["spot"]

# Right-size with VPA recommendations
# kubectl get vpa api-vpa -o yaml

Cost optimization strategies:

• Right-size pods (VPA recommendations)
• Use Spot/Preemptible instances (70-90% savings)
• Cluster Autoscaler/Karpenter to scale down
• KEDA to scale to zero
• Resource quotas to prevent waste
• Tools: Kubecost, OpenCost, AWS Cost Explorer

FAANG Interview Angle

Common questions:

1. "What is a service mesh and when would you use one?"
2. "Explain the Operator pattern"
3. "How would you manage a multi-cluster deployment?"
4. "How do you handle circuit breaking in microservices?"
5. "How would you optimize Kubernetes costs?"

Key answers:

• Service mesh adds mTLS, observability, traffic management via sidecar proxies; use with many microservices
• Operator = custom controller that encodes operational knowledge; automates complex app lifecycle
• Multi-cluster: GitOps per cluster, global LB for traffic, federation for resource sync
• Istio/Linkerd circuit breaking: eject unhealthy pods from load balancing after consecutive failures
• Right-size (VPA), Spot instances, autoscaling, scale-to-zero, resource quotas, cost monitoring

Official Links

• Istio
• Linkerd
• Cilium Service Mesh
• Operator Framework
• OperatorHub
• Kubernetes Patterns (book)

---