---

name: observability description: Logging, tracing, metrics fundamentals

Observability Fundamentals

Three Pillars

1. Logs

Discrete events that happened in the system.

2. Metrics

Numeric measurements over time.

3. Traces

Request flow across services.

Logging Best Practices

Structured Logging

Copy// CORRECT - Structured, searchable
logger.info("Order processed", Map.of(
    "orderId", orderId,
    "appId", appId,
    "amount", amount,
    "duration_ms", duration
));

// WRONG - Unstructured, hard to search
logger.info("Processed order " + orderId + " for app " + appId);

Log Levels

What to Log

• Request received (INFO)
• Request completed with duration (INFO)
• Business events (INFO)
• Errors with context (ERROR)
• External service calls (DEBUG)

What NOT to Log

• Passwords, tokens, API keys
• Full credit card numbers
• Personal data (mask it)
• Successful health checks (too noisy)

Error Handling

Include Context

Copytry {
    processOrder(orderId);
} catch (OrderProcessingException e) {
    logger.error("Failed to process order", Map.of(
        "orderId", orderId,
        "appId", appId,
        "error", e.getMessage(),
        "errorType", e.getClass().getSimpleName()
    ));
    throw new ServiceException("Order processing failed", e);
}

Error Response Pattern

Copy{
    "error": {
        "code": "ORDER_NOT_FOUND",
        "message": "Order with ID 123 not found",
        "request_id": "abc-123",
        "timestamp": "2024-01-01T00:00:00Z"
    }
}

Metrics

Key Metrics to Track

Metric Naming

Copy# Pattern: [namespace]_[subsystem]_[metric]_[unit]
violet_orders_processed_total
violet_orders_processing_duration_seconds
violet_api_requests_total
violet_api_errors_total

Tracing

Correlation IDs

Copy// Pass correlation ID through request chain
String correlationId = request.getHeader("X-Correlation-ID");
if (correlationId == null) {
    correlationId = UUID.randomUUID().toString();
}
MDC.put("correlationId", correlationId);

Span Context

CopySpan span = tracer.spanBuilder("processOrder")
    .setAttribute("orderId", orderId)
    .setAttribute("appId", appId)
    .startSpan();

try (Scope scope = span.makeCurrent()) {
    // Processing logic
} finally {
    span.end();
}

Alerting Guidelines

Alert on Symptoms, Not Causes

Copy# GOOD - User-facing symptom
alert: HighErrorRate
expr: error_rate > 0.05  # 5% errors

# AVOID - Internal cause
alert: DatabaseConnectionPoolExhausted
expr: db_connections >= max_connections

Alert Severity

Health Checks

Liveness vs Readiness

Copy// Liveness: Is the process running?
@GetMapping("/health/live")
public ResponseEntity<?> liveness() {
    return ResponseEntity.ok().build();
}

// Readiness: Can it accept traffic?
@GetMapping("/health/ready")
public ResponseEntity<?> readiness() {
    if (databaseHealthy && cacheHealthy) {
        return ResponseEntity.ok().build();
    }
    return ResponseEntity.status(503).build();
}