CalejoControl/DEPLOYMENT_TESTING.md

# Deployment Testing Strategy

This document outlines the strategy for testing deployments to ensure successful and reliable deployments to production and staging environments.

## Current Deployment Process

### Deployment Scripts
- **Primary Script**: `deploy/ssh/deploy-remote.sh`
- **Python Version**: `deploy/ssh/deploy-remote.py`
- **Target Server**: 95.111.206.155 (root user)
- **Configuration**: Git-ignored deployment configuration

### Current Capabilities
- SSH-based deployment
- Environment-specific configurations (production, staging)
- Dry-run mode for testing
- Key management system
- Configuration validation

## Deployment Testing Strategy

### 1. Pre-Deployment Testing

#### Local Validation
```bash
# Run all tests before deployment
./scripts/run-reliable-e2e-tests.py
pytest tests/unit/
pytest tests/integration/
```

#### Configuration Validation
```bash
# Validate deployment configuration
deploy/ssh/deploy-remote.sh -e production --dry-run --verbose
```

### 2. Staging Environment Testing

#### Recommended Enhancement
Create a staging environment for pre-production testing:

1. **Staging Server**: Separate server for testing deployments
2. **Smoke Tests**: Automated tests that verify deployment success
3. **Integration Tests**: Test with staging SCADA/optimizer services
4. **Rollback Testing**: Verify rollback procedures work

### 3. Post-Deployment Testing

#### Current Manual Process
After deployment, manually verify:
- Services are running
- Health endpoints respond
- Basic functionality works

#### Recommended Automated Process
Create automated smoke tests:

```bash
# Post-deployment smoke tests
./scripts/deployment-smoke-tests.sh
```

## Proposed Deployment Test Structure

### Directory Structure
```
tests/
├── deployment/              # Deployment-specific tests
│   ├── smoke_tests.py      # Post-deployment smoke tests
│   ├── staging_tests.py    # Staging environment tests
│   └── rollback_tests.py   # Rollback procedure tests
└── e2e/                    # Existing e2e tests (mock-dependent)
```

### Deployment Test Categories

#### 1. Smoke Tests (`tests/deployment/smoke_tests.py`)
- **Purpose**: Verify basic functionality after deployment
- **Execution**: Run on deployed environment
- **Tests**:
  - Service health checks
  - API endpoint availability
  - Database connectivity
  - Basic workflow validation

#### 2. Staging Tests (`tests/deployment/staging_tests.py`)
- **Purpose**: Full test suite on staging environment
- **Execution**: Run on staging server
- **Tests**:
  - Complete e2e workflows
  - Integration with staging services
  - Performance validation
  - Security compliance

#### 3. Rollback Tests (`tests/deployment/rollback_tests.py`)
- **Purpose**: Verify rollback procedures work
- **Execution**: Test rollback scenarios
- **Tests**:
  - Database rollback
  - Configuration rollback
  - Service restart procedures

## Implementation Plan

### Phase 1: Smoke Tests
1. Create `tests/deployment/smoke_tests.py`
2. Add basic health and connectivity tests
3. Integrate with deployment script
4. Run automatically after deployment

### Phase 2: Staging Environment
1. Set up staging server
2. Configure staging services
3. Create staging-specific tests
4. Run full test suite on staging

### Phase 3: Automated Deployment Pipeline
1. Integrate deployment tests with CI/CD
2. Add automated rollback triggers
3. Implement deployment metrics
4. Create deployment dashboards

## Current Deployment Script Usage

### Dry Run (Safe Testing)
```bash
# Test deployment without actually deploying
deploy/ssh/deploy-remote.sh -e production --dry-run --verbose
```

### Actual Deployment
```bash
# Deploy to production
deploy/ssh/deploy-remote.sh -e production
```

### With Custom Configuration
```bash
# Use custom configuration
deploy/ssh/deploy-remote.sh -e production -c deploy/config/custom.yaml
```

## Integration with Existing Tests

### Mock Services vs Real Deployment
- **Mock Services**: Use for development and local testing
- **Staging Services**: Use for pre-production testing
- **Production Services**: Use for post-deployment verification

### Test Execution Flow
```
Local Development → Mock Services → Unit/Integration Tests
    ↓
Staging Deployment → Staging Services → Deployment Tests
    ↓
Production Deployment → Production Services → Smoke Tests
```

## Security Considerations

### Deployment Security
- SSH key management
- Configuration encryption
- Access control
- Audit logging

### Test Data Security
- Use test data in staging
- Never use production data in tests
- Secure test credentials
- Clean up test data

## Monitoring and Metrics

### Deployment Metrics
- Deployment success rate
- Rollback frequency
- Test coverage percentage
- Performance impact

### Health Monitoring
- Service uptime
- Response times
- Error rates
- Resource utilization

## Next Steps

### Immediate Actions
1. Create basic smoke tests in `tests/deployment/`
2. Update deployment script to run smoke tests
3. Document deployment verification procedures

### Medium Term
1. Set up staging environment
2. Create comprehensive deployment test suite
3. Integrate with CI/CD pipeline

### Long Term
1. Implement automated rollback
2. Create deployment dashboards
3. Add performance benchmarking
4. Implement canary deployments