Skip to content

Open Lineage Event Receiver Integration Connector

Connector details

Overview

The OpenLineage Event Receiver integration connector receives open lineage events from an event topic and publishes them to the lineage integration connectors with OpenLineage listeners registered in the same instance of the Lineage Integrator OMIS.

Figure 1

Figure 1: Operation of the OpenLineage event receiver integration connector

Configuration

This connector uses the Lineage Integrator OMIS running in the Integration Daemon.

Its connection definition to use on the administration commands that configure the API Integrator OMIS is a VirtualConnection with an embedded OpenMetadataTopicConnection.

Connection configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
{
   "connection" : 
                { 
                    "class" : "VirtualConnection",
                    "qualifiedName" : "Egeria:IntegrationConnector:Lineage:OpenLineageEventReceiver Connection",
                    "connectorType" : 
                    {
                        "class" : "ConnectorType",
                        "connectorProviderClassName" : "org.odpi.openmetadata.adapters.connectors.integration.lineage.OpenLineageEventReceiverIntegrationProvider"
                    },
                    "embeddedConnections" : [ {{topicConnection}} ]
                }
}
  • Add the connection for the open metadata topic connector in the embeddedConnections section replacing {{topicConnection}}. This will have the topic name in the endpoint's networkAddress. The example below shows the connection for the Kafka open metadata topic connector

Kafka topic connection configuration

```json linenums="1" hl_lines="11-40" { "connection" : { "class" : "VirtualConnection", "qualifiedName" : "Egeria:IntegrationConnector:Lineage:OpenLineageEventReceiver Connection", "connectorType" : { "class" : "ConnectorType", "connectorProviderClassName" : "org.odpi.openmetadata.adapters.connectors.integration.lineage.OpenLineageEventReceiverIntegrationProvider" }, "embeddedConnections" : [ { "class": "EmbeddedConnection", "embeddedConnection : { "class": "Connection", "qualifiedName": "Kafka Open Metadata Topic Connector", "connectorType": { "class": "ConnectorType", "connectorProviderClassName": "org.odpi.openmetadata.adapters.eventbus.topic.kafka.KafkaOpenMetadataTopicProvider"
}, "endpoint": { "class": "Endpoint", "address": {{openLineageTopicName}} }, "configurationProperties": { "producer": { "bootstrap.servers": {{kafkaEndpoint}} }, "local.server.id": "{{localServerId}}", "consumer": { "bootstrap.servers": {{kafkaEndpoint}} } } } }] } }

  • Add the name of the topic in {{openLineageTopicName}}; the integration daemon's server id in {{localServerId}} and the enpoint for Apache Kafka (for example localhost:9092) in {{kafkaEndpoint}}.
Back to top