CalejoControl/docs/alert_system_setup.md

Alert System Setup Guide

Overview

The Calejo Control Adapter includes a comprehensive multi-channel alert system that can notify operators of safety events, system failures, and operational issues through multiple channels:

• Email Alerts - For all safety events
• SMS Alerts - For critical events only
• Webhook Integration - For external monitoring systems
• SCADA Alarms - For HMI integration (Phase 4)

Current Implementation Status

✅ Fully Implemented

• Alert routing and management framework
• Email sending logic (requires SMTP configuration)
• Webhook integration (requires endpoint configuration)
• Alert history and statistics
• Multi-channel coordination

⚠️ Requires External Configuration

• SMS Integration - Needs Twilio account and billing setup
• Email Integration - Needs real SMTP server credentials
• Webhook Integration - Needs real webhook endpoints
• SCADA Integration - Planned for Phase 4

Configuration Guide

1. Email Alert Setup

Prerequisites

• SMTP server access (Gmail, Office 365, or company SMTP)
• Valid email account credentials

Configuration Steps

1. Update environment variables in config/.env:

# Email Configuration
SMTP_SERVER=smtp.gmail.com
SMTP_PORT=587
SMTP_USERNAME=your-email@gmail.com
SMTP_PASSWORD=your-app-password  # Use app password for Gmail
SMTP_USE_TLS=true

2. Update settings in config/settings.py:

alert_email_enabled: bool = True
alert_email_from: str = "calejo-control@your-company.com"
alert_email_recipients: List[str] = ["operator1@company.com", "operator2@company.com"]

3. For Gmail users:
   • Enable 2-factor authentication
   • Generate an "App Password" for the application
   • Use the app password instead of your regular password

Testing Email Configuration

# Test email configuration
from src.monitoring.alerts import AlertManager
from config.settings import Settings

settings = Settings()
alert_manager = AlertManager(settings)

# Send test alert
await alert_manager.send_alert(
    alert_type="TEST_ALERT",
    severity="INFO", 
    message="Test email configuration",
    context={"test": "email"}
)

2. SMS Alert Setup (Twilio)

Prerequisites

• Twilio account (https://www.twilio.com)
• Verified phone numbers for recipients
• Billing information (SMS costs money)

Configuration Steps

1. Get Twilio credentials:

   • Sign up for Twilio account
   • Get Account SID and Auth Token from dashboard
   • Get your Twilio phone number

2. Update environment variables in config/.env:

# Twilio Configuration
TWILIO_ACCOUNT_SID=ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
TWILIO_AUTH_TOKEN=your_auth_token
TWILIO_PHONE_NUMBER=+15551234567

3. Update settings in config/settings.py:

alert_sms_enabled: bool = True
alert_sms_recipients: List[str] = ["+393401234567", "+393407654321"]

4. Implement SMS sending (currently only logs):
   • The current implementation only logs SMS alerts
   • To enable actual SMS sending, uncomment and implement the Twilio integration in src/monitoring/alerts.py:

# In _send_sms_alert method, replace the TODO section with:
import twilio
from twilio.rest import Client

client = Client(self.settings.twilio_account_sid, self.settings.twilio_auth_token)

for phone_number in self.settings.alert_sms_recipients:
    message = client.messages.create(
        body=f"[{alert_data['severity']}] {alert_data['message']}",
        from_=self.settings.twilio_phone_number,
        to=phone_number
    )

3. Webhook Integration

Prerequisites

• Webhook endpoint URL
• Authentication token (if required)

Configuration Steps

1. Update environment variables in config/.env:

# Webhook Configuration
ALERT_WEBHOOK_URL=https://your-monitoring-system.com/webhook
ALERT_WEBHOOK_TOKEN=your_bearer_token

2. Update settings in config/settings.py:

alert_webhook_enabled: bool = True
alert_webhook_url: str = "https://your-monitoring-system.com/webhook"
alert_webhook_token: str = "your_bearer_token"

Webhook Payload Format

When an alert is triggered, the webhook receives a JSON payload:

{
  "alert_type": "SAFETY_VIOLATION",
  "severity": "ERROR",
  "message": "Speed limit exceeded",
  "context": {
    "requested_speed": 55.0,
    "max_speed": 50.0
  },
  "station_id": "STATION_001",
  "pump_id": "PUMP_001",
  "timestamp": 1234567890.0,
  "app_name": "Calejo Control Adapter",
  "app_version": "2.0.0"
}

4. SCADA Alarm Integration (Phase 4)

Status: Planned for Phase 4 implementation

When implemented, SCADA alarms will:

• Trigger alarms in SCADA HMI systems via OPC UA
• Provide visual and audible alerts in control rooms
• Integrate with existing alarm management systems

Alert Types and Severity

Alert Types

• SAFETY_VIOLATION - Safety limit exceeded
• FAILSAFE_ACTIVATED - Failsafe mode activated
• EMERGENCY_STOP - Emergency stop activated
• SYSTEM_ERROR - System or communication error
• WATCHDOG_TIMEOUT - Database update timeout

Severity Levels

• INFO - Informational messages
• WARNING - Non-critical warnings
• ERROR - Errors requiring attention
• CRITICAL - Critical failures requiring immediate action

Testing the Alert System

Unit Tests

Run the comprehensive alert system tests:

pytest tests/unit/test_alerts.py -v

Manual Testing

# Test all alert channels
from src.monitoring.alerts import AlertManager
from config.settings import Settings

settings = Settings()
alert_manager = AlertManager(settings)

# Test different alert types and severities
test_alerts = [
    ("SAFETY_VIOLATION", "ERROR", "Speed limit exceeded"),
    ("FAILSAFE_ACTIVATED", "CRITICAL", "Failsafe mode activated"),
    ("SYSTEM_ERROR", "WARNING", "Communication timeout"),
]

for alert_type, severity, message in test_alerts:
    result = await alert_manager.send_alert(
        alert_type=alert_type,
        severity=severity,
        message=message,
        context={"test": True}
    )
    print(f"{alert_type}: {result}")

Troubleshooting

Common Issues

1. Email not sending:

   • Check SMTP server credentials
   • Verify TLS/SSL settings
   • Check firewall rules for outbound SMTP

2. SMS not working:

   • Verify Twilio account is active and funded
   • Check phone numbers are verified in Twilio
   • Ensure SMS integration is implemented (currently only logs)

3. Webhook failures:

   • Verify webhook URL is accessible
   • Check authentication tokens
   • Monitor webhook server logs

4. No alerts being sent:

   • Check alert channels are enabled in settings
   • Verify alert system is initialized in main application
   • Check application logs for alert-related errors

Logging

Alert system activities are logged with the following events:

• alert_sent - Alert successfully sent
• email_alert_failed - Email delivery failed
• sms_alert_failed - SMS delivery failed
• webhook_alert_failed - Webhook delivery failed
• scada_alert_failed - SCADA alarm failed

Security Considerations

• Store SMTP and Twilio credentials securely (environment variables)
• Use app passwords instead of regular passwords for email
• Rotate authentication tokens regularly
• Monitor alert system for abuse or excessive alerts
• Implement rate limiting if needed

Monitoring and Maintenance

Alert Statistics

Use the built-in statistics to monitor alert patterns:

alert_manager = AlertManager(settings)
stats = alert_manager.get_alert_stats()
print(f"Total alerts: {stats['total_alerts']}")
print(f"Severity counts: {stats['severity_counts']}")
print(f"Type counts: {stats['type_counts']}")

Alert History

Review recent alerts:

recent_alerts = alert_manager.get_alert_history(limit=50)
for alert in recent_alerts:
    print(f"{alert['timestamp']} - {alert['alert_type']}: {alert['message']}")

Next Steps

1. Immediate: Configure email and webhook for basic alerting
2. Short-term: Implement Twilio SMS integration if needed
3. Long-term: Implement SCADA OPC UA alarm integration in Phase 4

For questions or issues with alert system setup, refer to the application logs or contact the development team.