Skip to content

UBU Finance Logging System

This document describes the comprehensive logging system implemented in the UBU Finance backend.

Overview

The UBU Finance logging system is designed to provide comprehensive audit trails for security, compliance, and operational monitoring. It implements a hybrid approach:

  1. Redis-based Logging: High-performance, in-memory storage for recent logs with automatic expiration
  2. Database Logging: Persistent storage for critical security and financial events
  3. File-based Logging: Traditional logging to files for long-term storage and backup
  4. Loki Integration: Log aggregation and visualization through Grafana Loki

Components

1. Redis Logger

The Redis logger (app/monitoring/redis_logger.py) provides high-performance logging with automatic expiration:

  • Stores logs in Redis lists with configurable TTL (time-to-live)
  • Automatically trims logs to prevent unbounded growth
  • Categorizes logs by type (security, auth, api, error, audit, financial)
  • Provides fast access to recent logs

Configuration: 

REDIS_LOG_DB=1           # Redis database for logs
REDIS_LOG_TTL=604800     # TTL for logs in seconds (7 days)
REDIS_MAX_LOGS=10000     # Maximum number of logs to keep per type

2. Audit Logger

The audit logger (app/monitoring/audit_logger.py) provides structured logging for security-relevant events:

  • Logs authentication events (login, logout, password reset)
  • Logs security events (rate limiting, IP blocking, account lockout)
  • Logs data access events (read, create, update, delete)
  • Logs administrative actions
  • Logs financial transactions

For critical events, logs are also persisted to the database for tamper-evident audit trails.

3. Database Audit Logs

The database audit logs (app/models/security/audit_log_model.py) provide persistent storage for security-critical events:

  • AuditLogModel: General-purpose audit logs for all types of events
  • SecurityEventModel: Specific logs for security events (rate limiting, IP blocking, etc.)
  • FinancialAuditLogModel: Specific logs for financial transactions

4. Loki and Promtail

Loki and Promtail provide log aggregation and visualization:

  • Promtail collects logs from files and containers
  • Loki stores and indexes logs
  • Grafana provides visualization through dashboards

API Endpoints

The following API endpoints are available for accessing logs:

  • GET /audit/logs: Get audit logs from the database
  • GET /audit/security-events: Get security events from the database
  • GET /audit/financial: Get financial audit logs from the database
  • GET /audit/redis: Get logs from Redis
  • GET /audit/combined: Get combined audit data from both the database and Redis

All endpoints require admin authentication.

Grafana Dashboards

A pre-configured Grafana dashboard is available for visualizing logs:

  • UBU Finance Audit Logs: Displays all audit logs with filtering by type and time range

Configuration

Logging configuration is managed through environment variables and the security_config.py file:

# Redis logging configuration
REDIS_LOG_DB=1           # Redis database for logs
REDIS_LOG_TTL=604800     # TTL for logs in seconds (7 days)
REDIS_MAX_LOGS=10000     # Maximum number of logs to keep per type
REDIS_LOG_ENABLED=true   # Enable/disable Redis logging

# File logging configuration
LOGGING_ENABLED=true     # Enable/disable logging
LOGGING_LEVEL=INFO       # Log level
LOGGING_FILE=logs/app.log # Log file path
LOGGING_MAX_SIZE_MB=10   # Maximum log file size in MB
LOGGING_BACKUP_COUNT=5   # Number of backup log files

Best Practices

  1. Log Sensitive Actions: Always log security-sensitive actions such as authentication, data access, and financial transactions.
  2. Include Context: Include relevant context in logs, such as user ID, IP address, and resource information.
  3. Structured Logging: Use structured logging (JSON) for machine-readable logs.
  4. Monitor Logs: Regularly monitor logs for security incidents and operational issues.
  5. Retention Policy: Implement appropriate retention policies for logs based on compliance requirements.

Implementation Details

Log Format

All logs are stored in JSON format with the following fields:

{
  "timestamp": "2025-05-29T11:00:00.000Z",
  "level": "INFO",
  "logger": "audit",
  "message": "User login successful",
  "event_type": "authentication",
  "action": "login",
  "status": "success",
  "user_id": "123e4567-e89b-12d3-a456-426614174000",
  "username": "john.doe",
  "ip_address": "192.168.1.1",
  "details": {
    "method": "password",
    "user_agent": "Mozilla/5.0 ..."
  }
}

Integration with Security Features

The logging system is integrated with the following security features:

  • Rate Limiting: Logs rate limit events
  • IP Whitelisting: Logs IP whitelist violations
  • Account Lockout: Logs account lockout events
  • Authentication: Logs all authentication events
  • Data Access: Logs sensitive data access events

Future Enhancements

Potential future enhancements to the logging system:

  1. Log Encryption: Encrypt sensitive log data
  2. Digital Signatures: Add digital signatures to logs for tamper detection
  3. Advanced Analytics: Implement machine learning for anomaly detection in logs
  4. Real-time Alerting: Enhance real-time alerting based on log patterns
  5. Compliance Reporting: Generate compliance reports from logs