Add comprehensive alert system setup documentation
- Create detailed alert system setup guide with configuration examples - Document current implementation status and external service requirements - Include step-by-step setup for email, SMS, webhook, and SCADA alerts - Update README with alert system documentation reference - Enhance environment example file with all alert configuration options - Add troubleshooting guide and testing procedures Co-authored-by: openhands <openhands@all-hands.dev>
This commit is contained in:
openhands
parent
d89d65f03d
commit
fe72175a04
|
|
@ -77,7 +77,9 @@ calejo-control-adapter/
|
|||
│ ├── settings.py # Application settings
|
||||
│ └── docker-compose.yml # Docker configuration
|
||||
├── docs/
|
||||
│ └── specification.txt # Full implementation specification
|
||||
│ ├── specification.txt # Full implementation specification
|
||||
│ ├── optimization_plan_management.md # Optimization system documentation
|
||||
│ └── alert_system_setup.md # Alert system configuration guide
|
||||
├── requirements.txt # Python dependencies
|
||||
├── Dockerfile # Docker container definition
|
||||
└── README.md # This file
|
||||
|
|
@ -135,6 +137,11 @@ Key configuration options:
|
|||
- `REST_API_PORT`: REST API port (default: 8080)
|
||||
- `SAFETY_TIMEOUT_SECONDS`: Database watchdog timeout (default: 1200)
|
||||
|
||||
### Alert System Configuration
|
||||
|
||||
For detailed alert system setup (email, SMS, webhook integration), see:
|
||||
[Alert System Setup Guide](docs/alert_system_setup.md)
|
||||
|
||||
## Safety Framework
|
||||
|
||||
The adapter implements a comprehensive three-layer safety architecture:
|
||||
|
|
|
|||
|
|
@ -14,13 +14,21 @@ JWT_SECRET_KEY=your-secret-key-change-in-production
|
|||
JWT_ALGORITHM=HS256
|
||||
|
||||
# Alert Settings (Optional)
|
||||
# Email Configuration
|
||||
SMTP_SERVER=smtp.example.com
|
||||
SMTP_PORT=587
|
||||
SMTP_USERNAME=your-email@example.com
|
||||
SMTP_PASSWORD=your-email-password
|
||||
SMTP_USE_TLS=true
|
||||
|
||||
# SMS Configuration (Twilio)
|
||||
TWILIO_ACCOUNT_SID=your-twilio-account-sid
|
||||
TWILIO_AUTH_TOKEN=your-twilio-auth-token
|
||||
TWILIO_PHONE_NUMBER=+1234567890
|
||||
|
||||
# Webhook Configuration
|
||||
ALERT_WEBHOOK_URL=https://your-monitoring-system.com/webhook
|
||||
ALERT_WEBHOOK_TOKEN=your_bearer_token
|
||||
|
||||
# Logging
|
||||
LOG_LEVEL=INFO
|
||||
|
|
@ -0,0 +1,291 @@
|
|||
# Alert System Setup Guide
|
||||
|
||||
## Overview
|
||||
|
||||
The Calejo Control Adapter includes a comprehensive multi-channel alert system that can notify operators of safety events, system failures, and operational issues through multiple channels:
|
||||
|
||||
- **Email Alerts** - For all safety events
|
||||
- **SMS Alerts** - For critical events only
|
||||
- **Webhook Integration** - For external monitoring systems
|
||||
- **SCADA Alarms** - For HMI integration (Phase 4)
|
||||
|
||||
## Current Implementation Status
|
||||
|
||||
### ✅ Fully Implemented
|
||||
- **Alert routing and management framework**
|
||||
- **Email sending logic** (requires SMTP configuration)
|
||||
- **Webhook integration** (requires endpoint configuration)
|
||||
- **Alert history and statistics**
|
||||
- **Multi-channel coordination**
|
||||
|
||||
### ⚠️ Requires External Configuration
|
||||
- **SMS Integration** - Needs Twilio account and billing setup
|
||||
- **Email Integration** - Needs real SMTP server credentials
|
||||
- **Webhook Integration** - Needs real webhook endpoints
|
||||
- **SCADA Integration** - Planned for Phase 4
|
||||
|
||||
## Configuration Guide
|
||||
|
||||
### 1. Email Alert Setup
|
||||
|
||||
#### Prerequisites
|
||||
- SMTP server access (Gmail, Office 365, or company SMTP)
|
||||
- Valid email account credentials
|
||||
|
||||
#### Configuration Steps
|
||||
|
||||
1. **Update environment variables** in `config/.env`:
|
||||
```bash
|
||||
# Email Configuration
|
||||
SMTP_SERVER=smtp.gmail.com
|
||||
SMTP_PORT=587
|
||||
SMTP_USERNAME=your-email@gmail.com
|
||||
SMTP_PASSWORD=your-app-password # Use app password for Gmail
|
||||
SMTP_USE_TLS=true
|
||||
```
|
||||
|
||||
2. **Update settings** in `config/settings.py`:
|
||||
```python
|
||||
alert_email_enabled: bool = True
|
||||
alert_email_from: str = "calejo-control@your-company.com"
|
||||
alert_email_recipients: List[str] = ["operator1@company.com", "operator2@company.com"]
|
||||
```
|
||||
|
||||
3. **For Gmail users**:
|
||||
- Enable 2-factor authentication
|
||||
- Generate an "App Password" for the application
|
||||
- Use the app password instead of your regular password
|
||||
|
||||
#### Testing Email Configuration
|
||||
```python
|
||||
# Test email configuration
|
||||
from src.monitoring.alerts import AlertManager
|
||||
from config.settings import Settings
|
||||
|
||||
settings = Settings()
|
||||
alert_manager = AlertManager(settings)
|
||||
|
||||
# Send test alert
|
||||
await alert_manager.send_alert(
|
||||
alert_type="TEST_ALERT",
|
||||
severity="INFO",
|
||||
message="Test email configuration",
|
||||
context={"test": "email"}
|
||||
)
|
||||
```
|
||||
|
||||
### 2. SMS Alert Setup (Twilio)
|
||||
|
||||
#### Prerequisites
|
||||
- Twilio account (https://www.twilio.com)
|
||||
- Verified phone numbers for recipients
|
||||
- Billing information (SMS costs money)
|
||||
|
||||
#### Configuration Steps
|
||||
|
||||
1. **Get Twilio credentials**:
|
||||
- Sign up for Twilio account
|
||||
- Get Account SID and Auth Token from dashboard
|
||||
- Get your Twilio phone number
|
||||
|
||||
2. **Update environment variables** in `config/.env`:
|
||||
```bash
|
||||
# Twilio Configuration
|
||||
TWILIO_ACCOUNT_SID=ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
|
||||
TWILIO_AUTH_TOKEN=your_auth_token
|
||||
TWILIO_PHONE_NUMBER=+15551234567
|
||||
```
|
||||
|
||||
3. **Update settings** in `config/settings.py`:
|
||||
```python
|
||||
alert_sms_enabled: bool = True
|
||||
alert_sms_recipients: List[str] = ["+393401234567", "+393407654321"]
|
||||
```
|
||||
|
||||
4. **Implement SMS sending** (currently only logs):
|
||||
- The current implementation only logs SMS alerts
|
||||
- To enable actual SMS sending, uncomment and implement the Twilio integration in `src/monitoring/alerts.py`:
|
||||
|
||||
```python
|
||||
# In _send_sms_alert method, replace the TODO section with:
|
||||
import twilio
|
||||
from twilio.rest import Client
|
||||
|
||||
client = Client(self.settings.twilio_account_sid, self.settings.twilio_auth_token)
|
||||
|
||||
for phone_number in self.settings.alert_sms_recipients:
|
||||
message = client.messages.create(
|
||||
body=f"[{alert_data['severity']}] {alert_data['message']}",
|
||||
from_=self.settings.twilio_phone_number,
|
||||
to=phone_number
|
||||
)
|
||||
```
|
||||
|
||||
### 3. Webhook Integration
|
||||
|
||||
#### Prerequisites
|
||||
- Webhook endpoint URL
|
||||
- Authentication token (if required)
|
||||
|
||||
#### Configuration Steps
|
||||
|
||||
1. **Update environment variables** in `config/.env`:
|
||||
```bash
|
||||
# Webhook Configuration
|
||||
ALERT_WEBHOOK_URL=https://your-monitoring-system.com/webhook
|
||||
ALERT_WEBHOOK_TOKEN=your_bearer_token
|
||||
```
|
||||
|
||||
2. **Update settings** in `config/settings.py`:
|
||||
```python
|
||||
alert_webhook_enabled: bool = True
|
||||
alert_webhook_url: str = "https://your-monitoring-system.com/webhook"
|
||||
alert_webhook_token: str = "your_bearer_token"
|
||||
```
|
||||
|
||||
#### Webhook Payload Format
|
||||
When an alert is triggered, the webhook receives a JSON payload:
|
||||
|
||||
```json
|
||||
{
|
||||
"alert_type": "SAFETY_VIOLATION",
|
||||
"severity": "ERROR",
|
||||
"message": "Speed limit exceeded",
|
||||
"context": {
|
||||
"requested_speed": 55.0,
|
||||
"max_speed": 50.0
|
||||
},
|
||||
"station_id": "STATION_001",
|
||||
"pump_id": "PUMP_001",
|
||||
"timestamp": 1234567890.0,
|
||||
"app_name": "Calejo Control Adapter",
|
||||
"app_version": "2.0.0"
|
||||
}
|
||||
```
|
||||
|
||||
### 4. SCADA Alarm Integration (Phase 4)
|
||||
|
||||
**Status**: Planned for Phase 4 implementation
|
||||
|
||||
When implemented, SCADA alarms will:
|
||||
- Trigger alarms in SCADA HMI systems via OPC UA
|
||||
- Provide visual and audible alerts in control rooms
|
||||
- Integrate with existing alarm management systems
|
||||
|
||||
## Alert Types and Severity
|
||||
|
||||
### Alert Types
|
||||
- `SAFETY_VIOLATION` - Safety limit exceeded
|
||||
- `FAILSAFE_ACTIVATED` - Failsafe mode activated
|
||||
- `EMERGENCY_STOP` - Emergency stop activated
|
||||
- `SYSTEM_ERROR` - System or communication error
|
||||
- `WATCHDOG_TIMEOUT` - Database update timeout
|
||||
|
||||
### Severity Levels
|
||||
- `INFO` - Informational messages
|
||||
- `WARNING` - Non-critical warnings
|
||||
- `ERROR` - Errors requiring attention
|
||||
- `CRITICAL` - Critical failures requiring immediate action
|
||||
|
||||
## Testing the Alert System
|
||||
|
||||
### Unit Tests
|
||||
Run the comprehensive alert system tests:
|
||||
```bash
|
||||
pytest tests/unit/test_alerts.py -v
|
||||
```
|
||||
|
||||
### Manual Testing
|
||||
```python
|
||||
# Test all alert channels
|
||||
from src.monitoring.alerts import AlertManager
|
||||
from config.settings import Settings
|
||||
|
||||
settings = Settings()
|
||||
alert_manager = AlertManager(settings)
|
||||
|
||||
# Test different alert types and severities
|
||||
test_alerts = [
|
||||
("SAFETY_VIOLATION", "ERROR", "Speed limit exceeded"),
|
||||
("FAILSAFE_ACTIVATED", "CRITICAL", "Failsafe mode activated"),
|
||||
("SYSTEM_ERROR", "WARNING", "Communication timeout"),
|
||||
]
|
||||
|
||||
for alert_type, severity, message in test_alerts:
|
||||
result = await alert_manager.send_alert(
|
||||
alert_type=alert_type,
|
||||
severity=severity,
|
||||
message=message,
|
||||
context={"test": True}
|
||||
)
|
||||
print(f"{alert_type}: {result}")
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Email not sending**:
|
||||
- Check SMTP server credentials
|
||||
- Verify TLS/SSL settings
|
||||
- Check firewall rules for outbound SMTP
|
||||
|
||||
2. **SMS not working**:
|
||||
- Verify Twilio account is active and funded
|
||||
- Check phone numbers are verified in Twilio
|
||||
- Ensure SMS integration is implemented (currently only logs)
|
||||
|
||||
3. **Webhook failures**:
|
||||
- Verify webhook URL is accessible
|
||||
- Check authentication tokens
|
||||
- Monitor webhook server logs
|
||||
|
||||
4. **No alerts being sent**:
|
||||
- Check alert channels are enabled in settings
|
||||
- Verify alert system is initialized in main application
|
||||
- Check application logs for alert-related errors
|
||||
|
||||
### Logging
|
||||
Alert system activities are logged with the following events:
|
||||
- `alert_sent` - Alert successfully sent
|
||||
- `email_alert_failed` - Email delivery failed
|
||||
- `sms_alert_failed` - SMS delivery failed
|
||||
- `webhook_alert_failed` - Webhook delivery failed
|
||||
- `scada_alert_failed` - SCADA alarm failed
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- Store SMTP and Twilio credentials securely (environment variables)
|
||||
- Use app passwords instead of regular passwords for email
|
||||
- Rotate authentication tokens regularly
|
||||
- Monitor alert system for abuse or excessive alerts
|
||||
- Implement rate limiting if needed
|
||||
|
||||
## Monitoring and Maintenance
|
||||
|
||||
### Alert Statistics
|
||||
Use the built-in statistics to monitor alert patterns:
|
||||
|
||||
```python
|
||||
alert_manager = AlertManager(settings)
|
||||
stats = alert_manager.get_alert_stats()
|
||||
print(f"Total alerts: {stats['total_alerts']}")
|
||||
print(f"Severity counts: {stats['severity_counts']}")
|
||||
print(f"Type counts: {stats['type_counts']}")
|
||||
```
|
||||
|
||||
### Alert History
|
||||
Review recent alerts:
|
||||
```python
|
||||
recent_alerts = alert_manager.get_alert_history(limit=50)
|
||||
for alert in recent_alerts:
|
||||
print(f"{alert['timestamp']} - {alert['alert_type']}: {alert['message']}")
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Immediate**: Configure email and webhook for basic alerting
|
||||
2. **Short-term**: Implement Twilio SMS integration if needed
|
||||
3. **Long-term**: Implement SCADA OPC UA alarm integration in Phase 4
|
||||
|
||||
For questions or issues with alert system setup, refer to the application logs or contact the development team.
|
||||
Loading…
Reference in New Issue