DevOps Best Practices for Python Applications

Introduction

Continuous Integration and Deployment (CI/CD)

• Automated Testing: Run tests on every code commit
• Code Quality Checks: Static analysis and linting
• Security Scanning: Vulnerability detection
• Build Automation: Automated build and packaging

• Automated Deployment: Deploy to staging and production
• Environment Management: Consistent environments
• Rollback Capabilities: Quick rollback on failures
• Blue-Green Deployments: Zero-downtime deployments

yaml
# GitHub Actions workflow
name: CI/CD Pipeline
on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Set up Python
      uses: actions/setup-python@v2
      with:
        python-version: 3.9
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
        pip install -r requirements-dev.txt
    - name: Run tests
      run: |
        pytest --cov=src --cov-report=xml
    - name: Run linting
      run: |
        flake8 src/
        black --check src/
    - name: Security scan
      run: |
        bandit -r src/

Testing Strategies

• Unit Tests: Test individual functions and classes
• Integration Tests: Test component interactions
• End-to-End Tests: Test complete user workflows
• Performance Tests: Test under load

• pytest: Modern testing framework
• unittest: Built-in testing framework
• coverage.py: Code coverage analysis
• factory_boy: Test data generation
• responses: Mock HTTP requests

python
# pytest.ini
[tool:pytest]
testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*
addopts = --cov=src --cov-report=html --cov-report=term

# conftest.py
import pytest
from app import create_app

@pytest.fixture
def app():
    app = create_app('testing')
    return app

@pytest.fixture
def client(app):
    return app.test_client()

Monitoring and Observability

• Metrics: Quantitative data about system performance
• Logs: Detailed records of events
• Traces: Request flow through distributed systems

• Prometheus: Metrics collection and alerting
• Grafana: Metrics visualization
• ELK Stack: Log aggregation and analysis
• Jaeger/Zipkin: Distributed tracing
• Sentry: Error tracking and performance monitoring

python
# Prometheus metrics
from prometheus_client import Counter, Histogram, start_http_server

REQUEST_COUNT = Counter('requests_total', 'Total requests')
REQUEST_DURATION = Histogram('request_duration_seconds', 'Request duration')

# Start metrics server
start_http_server(8000)

# Application monitoring
import logging
from sentry_sdk import init as sentry_init

# Configure logging
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)

# Initialize Sentry
sentry_init(dsn='YOUR_SENTRY_DSN')

Infrastructure as Code (IaC)

• Version Control: Track infrastructure changes
• Reproducibility: Consistent environments
• Automation: Reduce manual errors
• Documentation: Infrastructure as documentation

• Terraform: Multi-cloud infrastructure
• Ansible: Configuration management
• CloudFormation: AWS-specific
• Pulumi: General-purpose with Python

hcl
# main.tf
provider "aws" {
  region = "us-west-2"
}

resource "aws_instance" "web" {
  ami           = "ami-0c02fb55956c7d316"
  instance_type = "t3.micro"
  
  tags = {
    Name = "Python App Server"
  }
}

resource "aws_security_group" "web" {
  name_prefix = "web-"
  
  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Configuration Management

• Environment-specific: Different configs for dev/staging/prod
• Secure: Never commit secrets to version control
• Centralized: Single source of truth
• Validated: Validate configuration at startup

• Environment Variables: Simple and portable
• Config Files: YAML, JSON, or INI formats
• Secret Management: AWS Secrets Manager, HashiCorp Vault
• Configuration Services: Consul, etcd

python
# config.py
import os
from dataclasses import dataclass
from typing import Optional

@dataclass
class Config:
    database_url: str
    redis_url: str
    secret_key: str
    debug: bool = False
    
    @classmethod
    def from_env(cls) -> 'Config':
        return cls(
            database_url=os.getenv('DATABASE_URL', 'sqlite:///app.db'),
            redis_url=os.getenv('REDIS_URL', 'redis://localhost:6379'),
            secret_key=os.getenv('SECRET_KEY', 'dev-secret-key'),
            debug=os.getenv('DEBUG', 'False').lower() == 'true'
        )

# Usage
config = Config.from_env()

Deployment Strategies

• Blue-Green: Zero-downtime deployments
• Rolling: Gradual replacement of instances
• Canary: Gradual rollout to subset of users
• Feature Flags: Control feature availability

• Docker: Containerization
• Kubernetes: Container orchestration
• Docker Compose: Multi-container applications
• AWS ECS/EKS: Managed container services

yaml
# .github/workflows/deploy.yml
name: Deploy
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Build Docker image
      run: docker build -t myapp:${{ github.sha }} .
    - name: Deploy to staging
      run: |
        kubectl set image deployment/myapp myapp=myapp:${{ github.sha }}
        kubectl rollout status deployment/myapp
    - name: Run smoke tests
      run: pytest tests/smoke/
    - name: Deploy to production
      if: success()
      run: |
        kubectl set image deployment/myapp-prod myapp=myapp:${{ github.sha }}
        kubectl rollout status deployment/myapp-prod

Security Best Practices

• Least Privilege: Minimum required permissions
• Defense in Depth: Multiple security layers
• Regular Updates: Keep dependencies updated
• Security Scanning: Automated vulnerability detection

• bandit: Python security linter
• safety: Check for known vulnerabilities
• docker-bench: Docker security best practices
• kube-bench: Kubernetes security assessment

yaml
# Security checks in CI/CD
- name: Security scan
  run: |
    bandit -r src/
    safety check
    docker run --rm -v $(pwd):/app securecodewarrior/docker-bench-security

python
# Using AWS Secrets Manager
import boto3
from botocore.exceptions import ClientError

def get_secret(secret_name):
    session = boto3.session.Session()
    client = session.client(
        service_name='secretsmanager',
        region_name='us-west-2'
    )
    
    try:
        response = client.get_secret_value(SecretId=secret_name)
        return response['SecretString']
    except ClientError as e:
        raise e

Performance Optimization

• APM Tools: New Relic, DataDog, AppDynamics
• Profiling: cProfile, py-spy, memory_profiler
• Load Testing: Locust, JMeter, Artillery

python
# Use appropriate data structures
from collections import defaultdict, Counter

# Optimize database queries
from django.db import connection
from sqlalchemy.orm import joinedload

# Caching
from functools import lru_cache
import redis

@lru_cache(maxsize=128)
def expensive_function(arg):
    return arg * 2

# Async programming
import asyncio
import aiohttp

async def fetch_data(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
            return await response.json()

• Use connection pooling
• Implement query caching
• Optimize database indexes
• Use read replicas for read-heavy workloads

Disaster Recovery and Backup

• Database Backups: Regular automated backups
• Application State: Backup critical application data
• Configuration: Version control for configuration
• Infrastructure: Infrastructure as code

• RTO (Recovery Time Objective): How quickly to recover
• RPO (Recovery Point Objective): How much data loss is acceptable
• Multi-region: Deploy across multiple regions
• Automated Failover: Automatic failover mechanisms

python
# Database backup script
import subprocess
import boto3
from datetime import datetime

def backup_database():
    timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
    backup_file = f'db_backup_{timestamp}.sql'
    
    # Create database backup
    subprocess.run([
        'pg_dump',
        '--host=localhost',
        '--port=5432',
        '--username=user',
        '--dbname=mydb',
        '--file', backup_file
    ])
    
    # Upload to S3
    s3 = boto3.client('s3')
    s3.upload_file(backup_file, 'my-backup-bucket', backup_file)
    
    print(f'Backup completed: {backup_file}')

Conclusion

Share this article

Comments