Prometheus Setup

Prev Next

Prometheus Setup Guide

Overview

This document provides instructions for setting up Prometheus monitoring for Lobster Data Platform application.

Prerequisites

  • Prometheus server installed

  • Access to target applications for metrics scraping

  • Valid authentication tokens for secured endpoints

Configuration File Structure

The Prometheus configuration is defined in prometheus.yml (https://prometheus.io/docs/prometheus/latest/configuration/configuration/) with the following key components:

Global Settings

global:
  scrape_interval: 15s      # Default interval for scraping metrics
  evaluation_interval: 15s  # Default interval for evaluating rules

Scrape Configurations

External System Monitoring

Monitors an external system with the following configuration:

  • Target: <server name> like: localhost, 10.xx.yy.zz

  • Metrics Path: /system/metrics

  • Scrape Interval: 25 seconds

  • Timeout: 10 seconds

  • Authentication: Bearer token authentication

Security Configuration

Authentication Methods

The configuration supports multiple authentication methods but LDP only uses one:

  • Bearer Token: Currently active authentication method

Token Management

  • Store tokens securely and avoid committing them to version control

  • Each environment should have its own set of authentication credentials

Sample Configuration File

Below is a complete sample prometheus.yml configuration file that you can use as a starting point:


# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
scrape_configs:
  - job_name: 'lobster-data-app'
    static_configs:
      - targets: [ 'localhost:8080' ]
    scrape_interval: 25s
    scrape_timeout: 10s
    metrics_path: /system/metrics
    bearer_token: 'your-application-token-here'

Configuration Customization

To customize this configuration for your environment:

  • Replace Target Addresses:

    • Change localhost:8080 to your application's host and port

  • Update Authentication Tokens:

    • Replace your-application-token-here with your application's token

  • Adjust Timing Settings:

    • Modify scrape_interval values based on your monitoring frequency needs

    • Adjust scrape_timeout if your endpoints need more time to respond

Deployment Steps

Prepare Configuration

  • Update target addresses for your environment

  • Replace authentication tokens with valid credentials

  • Adjust scrape intervals based on your monitoring requirements

Validate Configuration

promtool check config prometheus.yml

Start Prometheus

prometheus --config.file=prometheus.yml

Verify Setup

  • Access Prometheus web interface (default: http://localhost:9090)

  • Check targets status under Status → Targets

  • Verify metrics are being collected  

Troubleshooting

Common Issues

  1. Authentication Failures: Verify bearer tokens are valid and not expired

  2. Connection Timeouts: Check network connectivity and firewall rules

  3. Missing Metrics: Ensure target applications expose metrics at the configured path

Monitoring Health

  • Check Prometheus logs for scraping errors

  • Monitor target status in Prometheus UI

  • Verify metrics are being collected with expected labels

Best Practices

  • Security

    • Use environment-specific tokens

    • Implement proper access controls

  • Performance

    • Adjust scrape intervals based on requirements

    • Monitor Prometheus resource usage

    • Consider metric retention policies

  • Maintenance

    • Regular configuration reviews

    • Monitor target health

    • Update authentication credentials as needed