Set up Prometheus

Prometheus setup guide

Overview

This document provides instructions for setting up Prometheus monitoring for Lobster Data Platform application.

Requirements

Prometheus server installed
Access to target applications for metrics scraping
Valid authentication tokens for secured endpoints

Configuration file structure

The Prometheus configuration is defined in prometheus.yml (https://prometheus.io/docs/prometheus/latest/configuration/configuration/) with the following key components:

Global settings

global:
  scrape_interval: 15s      # Default interval for scraping metrics
  evaluation_interval: 15s  # Default interval for evaluating rules

Scrape configurations

External system monitoring

Monitors an external system with the following configuration:

Target: <server name> like: localhost, 10.xx.yy.zz
Metrics path: /system/metrics
Scrape interval: 25 seconds
Timeout: 10 seconds
Authentication: Bearer token authentication

Security configuration

Authentication methods

The configuration supports multiple authentication methods but LDP only uses one:

Bearer token: Currently active authentication method

Token management

Store tokens securely and avoid committing them to version control
Each environment should have its own set of authentication credentials

Sample configuration file

Below is a complete sample prometheus.yml configuration file that you can use as a starting point:


# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
scrape_configs:
  - job_name: 'lobster-data-app'
    static_configs:
      - targets: [ 'localhost:8080' ]
    scrape_interval: 25s
    scrape_timeout: 10s
    metrics_path: /system/metrics
    bearer_token: 'your-application-token-here'

Configuration customization

To customize this configuration for your environment:

Replace target addresses:
- Change localhost:8080 to your application's host and port
Update authentication tokens:
- Replace your-application-token-here with your application's token
Adjust timing settings:
- Modify scrape_interval values based on your monitoring frequency needs
- Adjust scrape_timeout if your endpoints need more time to respond

Deployment steps

Prepare configuration

Update target addresses for your environment
Replace authentication tokens with valid credentials
Adjust scrape intervals based on your monitoring requirements

Validate configuration

promtool check config prometheus.yml

Start Prometheus

prometheus --config.file=prometheus.yml

Verify setup

Access Prometheus web interface (default: http://localhost:9090)
Check targets status under Status → Targets
Verify metrics are being collected

Troubleshooting

Common issues

Authentication failures: Verify bearer tokens are valid and not expired
Connection timeouts: Check network connectivity and firewall rules
Missing metrics: Ensure target applications expose metrics at the configured path

Monitoring health

Check Prometheus logs for scraping errors
Monitor target status in Prometheus UI
Verify metrics are being collected with expected labels

Best practices

Security
- Use environment-specific tokens
- Implement proper access controls
Performance
- Adjust scrape intervals based on requirements
- Monitor Prometheus resource usage
- Consider metric retention policies
Maintenance
- Regular configuration reviews
- Monitor target health
- Update authentication credentials as needed

Documentation Index