Docker and Kubernetes: Containerizing Python Applications

Introduction

What is Docker?

• Consistency: Same environment everywhere
• Isolation: Applications run in isolated environments
• Portability: Easy to move between different systems
• Scalability: Easy to scale applications up or down
• Efficiency: Better resource utilization than virtual machines
• Version Control: Track changes to your application environment

What is Kubernetes?

• Container Orchestration: Manages container lifecycle
• Auto-scaling: Automatically scales based on demand
• Load Balancing: Distributes traffic across containers
• Service Discovery: Automatically discovers and connects services
• Rolling Updates: Zero-downtime deployments
• Self-healing: Automatically restarts failed containers
• Resource Management: Efficiently manages CPU and memory

Containerizing Python Applications with Docker

dockerfile
# Use Python 3.9 as base image
FROM python:3.9-slim

# Set working directory
WORKDIR /app

# Copy requirements file
COPY requirements.txt .

# Install dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY . .

# Expose port
EXPOSE 8000

# Run the application
CMD ["python", "app.py"]

Flask==2.0.1
Gunicorn==20.1.0
Redis==3.5.3

bash
docker build -t my-python-app .

bash
docker run -p 8000:8000 my-python-app

bash
curl http://localhost:8000

Docker Best Practices

dockerfile
# Build stage
FROM python:3.9-slim as builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --user -r requirements.txt

# Production stage
FROM python:3.9-slim
WORKDIR /app
COPY --from=builder /root/.local /root/.local
COPY . .
CMD ["python", "app.py"]

• Copy requirements.txt first
• Install dependencies before copying code
• Use .dockerignore to exclude unnecessary files

• Use specific Python versions
• Choose slim images for smaller size
• Consider Alpine Linux for minimal images

• Don't run as root user
• Use non-root user in container
• Keep base images updated
• Scan images for vulnerabilities

• Set memory and CPU limits
• Use health checks
• Implement graceful shutdowns

Docker Compose for Development

yaml
version: '3.8'
services:
  web:
    build: .
    ports:
      - "8000:8000"
    environment:
      - FLASK_ENV=development
    depends_on:
      - redis
      - postgres
    volumes:
      - .:/app

  redis:
    image: redis:alpine
    ports:
      - "6379:6379"

  postgres:
    image: postgres:13
    environment:
      POSTGRES_DB: myapp
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:

bash
# Start all services
docker-compose up

# Start in background
docker-compose up -d

# Stop services
docker-compose down

# View logs
docker-compose logs

Introduction to Kubernetes

• Pods: Smallest deployable units
• Services: Network access to pods
• Deployments: Manage pod replicas
• ConfigMaps: Configuration data
• Secrets: Sensitive data
• Namespaces: Resource isolation

• Master Node: Control plane components
• Worker Nodes: Run application containers
• API Server: Central management point
• etcd: Distributed key-value store
• Scheduler: Assigns pods to nodes
• Controller Manager: Manages cluster state

• Use minikube for local development
• Deploy to managed Kubernetes services
• Learn kubectl command-line tool

Deploying Python Applications to Kubernetes

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: python-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: python-app
  template:
    metadata:
      labels:
        app: python-app
    spec:
      containers:
      - name: python-app
        image: my-python-app:latest
        ports:
        - containerPort: 8000
        env:
        - name: FLASK_ENV
          value: "production"

yaml
apiVersion: v1
kind: Service
metadata:
  name: python-app-service
spec:
  selector:
    app: python-app
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8000
  type: LoadBalancer

bash
# Apply the deployment
kubectl apply -f deployment.yaml

# Apply the service
kubectl apply -f service.yaml

# Check status
kubectl get pods
kubectl get services

Kubernetes Configuration Management

yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  DATABASE_URL: "postgresql://user:password@postgres:5432/myapp"
  REDIS_URL: "redis://redis:6379"
  LOG_LEVEL: "INFO"

yaml
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
data:
  database-password: cGFzc3dvcmQ=
  api-key: YWJjZGVmZ2g=

yaml
spec:
  containers:
  - name: python-app
    image: my-python-app:latest
    env:
    - name: DATABASE_URL
      valueFrom:
        configMapKeyRef:
          name: app-config
          key: DATABASE_URL
    - name: DATABASE_PASSWORD
      valueFrom:
        secretKeyRef:
          name: app-secrets
          key: database-password

Scaling and Auto-scaling

bash
# Scale deployment to 5 replicas
kubectl scale deployment python-app --replicas=5

# Check replica status
kubectl get pods -l app=python-app

yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: python-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: python-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: python-app-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: python-app
  updatePolicy:
    updateMode: "Auto"

Monitoring and Logging

• Web-based UI for cluster management
• View pods, services, and deployments
• Monitor resource usage
• Debug application issues

• Prometheus for metrics collection
• Grafana for visualization
• Custom dashboards for applications
• Alerting for critical issues

• Centralized logging
• Log aggregation and analysis
• Search and visualization
• Real-time monitoring

python
# Add monitoring to your Python app
from prometheus_client import Counter, Histogram, start_http_server

# Define metrics
REQUEST_COUNT = Counter('requests_total', 'Total requests')
REQUEST_DURATION = Histogram('request_duration_seconds', 'Request duration')

# Start metrics server
start_http_server(8001)

# Use metrics in your app
@REQUEST_DURATION.time()
def handle_request():
    REQUEST_COUNT.inc()
    # Your application logic

Security Best Practices

• Use minimal base images
• Keep images updated
• Scan for vulnerabilities
• Don't run as root
• Use read-only filesystems

• Use RBAC for access control
• Network policies for traffic control
• Pod security policies
• Secrets management
• Regular security updates

dockerfile
# Use specific image tags
FROM python:3.9-slim

# Create non-root user
RUN groupadd -r appuser && useradd -r -g appuser appuser
USER appuser

# Use read-only filesystem
# Add security scanning

yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: python-app-netpol
spec:
  podSelector:
    matchLabels:
      app: python-app
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: nginx
    ports:
    - protocol: TCP
      port: 8000

CI/CD with Docker and Kubernetes

yaml
name: Build and Deploy
on:
  push:
    branches: [main]

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    
    - name: Build Docker image
      run: docker build -t my-python-app:${{ github.sha }} .
    
    - name: Push to registry
      run: docker push my-python-app:${{ github.sha }}
    
    - name: Deploy to Kubernetes
      run: |
        kubectl set image deployment/python-app python-app=my-python-app:${{ github.sha }}
        kubectl rollout status deployment/python-app

yaml
stages:
  - build
  - deploy

build:
  stage: build
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

deploy:
  stage: deploy
  script:
    - kubectl set image deployment/python-app python-app=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - kubectl rollout status deployment/python-app

Troubleshooting Common Issues

• Build Failures: Check Dockerfile syntax and dependencies
• Image Size: Use multi-stage builds and minimal base images
• Performance: Optimize layer caching and resource usage
• Networking: Check port mappings and network configuration

• Pod Not Starting: Check resource limits and image availability
• Service Not Accessible: Verify service configuration and selectors
• Scaling Issues: Check HPA configuration and metrics
• Storage Issues: Verify persistent volume claims

bash
# Check pod status
kubectl get pods
kubectl describe pod <pod-name>

# View logs
kubectl logs <pod-name>
kubectl logs -f <pod-name>

# Execute commands in pod
kubectl exec -it <pod-name> -- /bin/bash

# Check events
kubectl get events

# Check resource usage
kubectl top pods
kubectl top nodes

Conclusion

Share this article

Comments