# Python Best Practices for Production Systems

## Project Structure

A well-organized Python project follows a consistent structure:
- Source code in a dedicated package directory
- Configuration files at project root (config.yaml, pyproject.toml)
- Tests in a parallel tests/ directory mirroring source structure
- Virtual environment isolation (venv or conda)

## Error Handling

### Use Typed Exceptions
Instead of returning error strings, define a hierarchy of exceptions:
```python
class AppError(Exception):
    def __init__(self, message, code, details=None):
        self.message = message
        self.code = code
        self.details = details or {}
```

This enables callers to catch specific error types and map them to appropriate responses (e.g., HTTP 503 for service errors, 400 for validation errors).

### Fail Fast at Startup
Validate configuration and dependencies at application startup rather than at first use. Check database connections, API endpoints, and required environment variables before accepting traffic.

## Configuration Management

### YAML for Application Config
Use YAML files for structured configuration. Separate runtime config from secrets:
- config.yaml: Application settings, thresholds, feature flags
- Environment variables: API keys, passwords, connection strings

### Never Hardcode Secrets
Use environment variables for sensitive values. Validate required env vars at startup with clear error messages.

## Testing

### Mock External Services
Unit and integration tests should mock external HTTP services (APIs, databases) to ensure:
- Tests are fast and deterministic
- Tests work offline
- Tests don't hit rate limits or cost money

### Use Fixtures for Shared State
pytest fixtures provide clean setup/teardown for test dependencies:
```python
@pytest.fixture
def mock_llm():
    return MockLLMClient(responses={"default": "Test answer"})
```

## Logging

### Structured Logging (JSONL)
Use JSON-lines format for production logs:
- Machine-parseable for log aggregation tools
- Include timestamps, event types, and correlation IDs
- Never log raw user input or secrets

### Audit vs Application Logs
Separate audit trails (security events, access logs) from application logs (debug info, performance metrics). Audit logs should be append-only and tamper-evident.

## Performance

### Lazy Loading
Load expensive resources (ML models, database connections) on first use, not at import time. Cache them for subsequent calls.

### Batch Operations
When processing multiple items (embeddings, database inserts), batch operations are significantly faster than individual calls due to reduced overhead per item.