Package org.odpi.openmetadata.adapters.connectors.integration.openlineage

Class FileBasedOpenLineageLogStoreConnector

java.lang.Object

org.odpi.openmetadata.frameworks.connectors.Connector

org.odpi.openmetadata.frameworks.connectors.ConnectorBase

org.odpi.openmetadata.frameworks.integration.connectors.IntegrationConnectorBase

org.odpi.openmetadata.integrationservices.lineage.connector.LineageIntegratorConnector

org.odpi.openmetadata.adapters.connectors.integration.openlineage.OpenLineageLogStoreConnectorBase

org.odpi.openmetadata.adapters.connectors.integration.openlineage.FileBasedOpenLineageLogStoreConnector

All Implemented Interfaces:

OpenLineageLogStore, AuditLoggingComponent, SecureConnectorExtension, VirtualConnectorExtension, IntegrationConnector, LineageIntegratorOMISConnector, OpenLineageEventListener

public class FileBasedOpenLineageLogStoreConnector
extends OpenLineageLogStoreConnectorBase

FileBasedOpenLineageLogStoreConnector provides a connector implementation for a file based open lineage log. The open lineage log is stored in a directory and each open lineage event record is stored as a file with a filename built from the record's unique identifier (runId), time and status. The record is stored in a subdirectory that is made from the namespace and job.

Nested Class Summary

Nested classes/interfaces inherited from class org.odpi.openmetadata.frameworks.connectors.ConnectorBase

ConnectorBase.ProtectedConnection

Field Summary

Fields inherited from class org.odpi.openmetadata.adapters.connectors.integration.openlineage.OpenLineageLogStoreConnectorBase

destinationName, myContext

Fields inherited from class org.odpi.openmetadata.frameworks.integration.connectors.IntegrationConnectorBase

auditLog, connectorName, embeddedConnectors, integrationContext

Fields inherited from class org.odpi.openmetadata.frameworks.connectors.ConnectorBase

connectedAssetProperties, connectionBean, connectionProperties, connectorInstanceId, secretsStoreConnectorMap

Constructor Summary

Constructors

Constructor	Description
FileBasedOpenLineageLogStoreConnector()	Default constructor used by the connector provider.

Method Summary

Modifier and Type	Method	Description
void	start()	Set up the name of the file store
void	storeEvent(OpenLineageRunEvent openLineageEvent, String rawEvent)	Store the open lineage event in the open lineage log store.

Methods inherited from class org.odpi.openmetadata.adapters.connectors.integration.openlineage.OpenLineageLogStoreConnectorBase

getDestinationName, initialize, logNoRawEvent, processOpenLineageRunEvent, refresh

Methods inherited from class org.odpi.openmetadata.integrationservices.lineage.connector.LineageIntegratorConnector

getContext, setContext

Methods inherited from class org.odpi.openmetadata.frameworks.integration.connectors.IntegrationConnectorBase

disconnect, engage, getConnectorComponentDescription, initializeEmbeddedConnectors, setAuditLog, setConnectorName, setContext

Methods inherited from class org.odpi.openmetadata.frameworks.connectors.ConnectorBase

disconnectConnectors, equals, getConnectedAssetProperties, getConnection, getConnectorInstanceId, hashCode, initializeConnectedAssetProperties, initializeSecretsStoreConnector, isActive, toString

Methods inherited from class org.odpi.openmetadata.frameworks.connectors.Connector

clearStatisticProperty, clearStatisticTimestamp, getConnectorStatistics, getStatisticCounter, getStatisticProperty, getStatisticTimestamp, incrementStatisticCounter, initializeStatisticCounter, setStatisticProperty, setStatisticTimestamp

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Details

FileBasedOpenLineageLogStoreConnector

public FileBasedOpenLineageLogStoreConnector()

Default constructor used by the connector provider.

Method Details

start

public void start()
           throws ConnectorCheckedException

Set up the name of the file store

Specified by:

start in interface IntegrationConnector

Overrides:

start in class OpenLineageLogStoreConnectorBase

Throws:

ConnectorCheckedException - something went wrong

storeEvent

public void storeEvent(OpenLineageRunEvent openLineageEvent,
 String rawEvent)
                throws InvalidParameterException,
PropertyServerException

Store the open lineage event in the open lineage log store. If the raw event is null, a json version of the open lineage event is generated using the Egeria beans.

Specified by:

storeEvent in interface OpenLineageLogStore

Specified by:

storeEvent in class OpenLineageLogStoreConnectorBase

Parameters:

openLineageEvent - event formatted using Egeria beans

rawEvent - event in Json form from the originator - may have facets that are not known to Egeria

Throws:

InvalidParameterException - indicates that the openLineageEvent parameter is invalid.

PropertyServerException - indicates that the log store is not available or has an error.