ETL/ELT (Bridge data service)

If an ETL/ELT job is to be run not directly in the Lobster Data Platform (LDP) but on a bridge (external server), the LDP delegates the processing to the bridge. Upon completion, the bridge reports the job status back to the LDP via a callback URL. This requires two configuration steps: selecting the bridge in the pipeline and specifying the callback URL in the startup.xml of the LDP.

Link the pipeline to the bridge

To run an ETL/ELT job on a bridge, set the following in the ETL/ELT pipeline:

• Process type → External server

• Remote Server Alias → The alias of the bridge on which the processing is to run.

The bridge then executes the job using its own processing functions. No additional configuration is required on the bridge itself.

Configure the callback URL for the bridge

The bridge requires a callback URL to signal job completion to the LDP. This URL is set centrally in the LDP's startup.xml file via the webServiceUrl parameter and is automatically embedded by the LDP into the ETL job payload that is passed to the bridge.

In the LDP's startup.xml file, insert or modify the following line:

<Set name="webServiceUrl">https://<LDP_IP_ADDRESS>/dw/Request</Set>

Replace <LDP_IP_ADDRESS> with the hostname or IP address of your LDP instance that is accessible from the bridge.

Requirements for the callback URL

• Fully qualified URL: Use a full URL (including the schema and host) that can be resolved and accessed from the bridge's network segment. Internal short names or localhost will not work.

• Network Connectivity: Check the firewall rules and DNS resolution between the bridge and the LDP. The bridge must be able to reach the LDP on the configured port (usually HTTPS/443).

• Endpoint Test: Test the callback endpoint and payload handling before running production ETL jobs on the bridge. This ensures that status messages are received and processed correctly.

NOTE

Without a valid webServiceUrl, the bridge cannot report that the job has been completed. In this case, the LDP does not receive a final status, even though the job was processed successfully on the bridge.

Background: ETL and ELT

ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) refer to two data integration patterns. In ETL, the data is transformed before being loaded into the target system; in ELT, the transformation takes place after loading, typically within the target system (e.g., a data warehouse). The bridge delegation and callback configuration apply equally to both variants.