Package org.odpi.openmetadata.adapters.connectors.integration.openlineage

Class FileBasedOpenLineageLogStoreConnector

java.lang.Object

org.odpi.openmetadata.frameworks.connectors.Connector

org.odpi.openmetadata.frameworks.connectors.ConnectorBase

org.odpi.openmetadata.frameworks.integration.connectors.IntegrationConnectorBase

org.odpi.openmetadata.adapters.connectors.integration.openlineage.OpenLineageLogStoreConnectorBase

org.odpi.openmetadata.adapters.connectors.integration.openlineage.FileBasedOpenLineageLogStoreConnector

All Implemented Interfaces:

OpenLineageLogStore, AuditLoggingComponent, SecureConnectorExtension, VirtualConnectorExtension, IntegrationConnector, OpenLineageEventListener

public class FileBasedOpenLineageLogStoreConnector
extends OpenLineageLogStoreConnectorBase

FileBasedOpenLineageLogStoreConnector provides a connector implementation for a file based open lineage log. The open lineage log is stored in a directory and each open lineage event record is stored as a file with a filename built from the record's unique identifier (runId), time and status. The record is stored in a subdirectory that is made from the namespace and job.

Nested Class Summary

Nested classes/interfaces inherited from class org.odpi.openmetadata.frameworks.connectors.ConnectorBase

ConnectorBase.ProtectedConnection

Field Summary

Fields inherited from class org.odpi.openmetadata.adapters.connectors.integration.openlineage.OpenLineageLogStoreConnectorBase

destinationAddresses, distributorName, myContext

Fields inherited from class org.odpi.openmetadata.frameworks.integration.connectors.IntegrationConnectorBase

auditLog, catalogTargetsManager, connectorName, integrationContext, propertyHelper

Fields inherited from class org.odpi.openmetadata.frameworks.connectors.ConnectorBase

connectionBean, connectorInstanceId, embeddedConnectors, messageFormatter, secretsStoreConnectorMap

Constructor Summary

Constructors

Constructor	Description
FileBasedOpenLineageLogStoreConnector()	Default constructor used by the connector provider.

Method Summary

Modifier and Type	Method	Description
protected void	newDestinationIdentified(String logStoreDirectoryName)	Informs the subclasses that there is a new destination - in case they need to do special setup.
void	storeEvent(OpenLineageRunEvent openLineageEvent, String rawEvent, String logStoreDirectoryName)	Store the open lineage event in the open lineage log store.

Methods inherited from class org.odpi.openmetadata.adapters.connectors.integration.openlineage.OpenLineageLogStoreConnectorBase

initialize, logNoRawEvent, processOpenLineageRunEvent, refresh, start

Methods inherited from class org.odpi.openmetadata.frameworks.integration.connectors.IntegrationConnectorBase

disconnect, engage, getConnectorComponentDescription, getNetworkAddress, passEventToCatalogTargets, refreshCatalogTargets, registerCatalogTargetChangeListener, setAuditLog, setConnectorName, setContext

Methods inherited from class org.odpi.openmetadata.frameworks.connectors.ConnectorBase

combineConfigurationProperties, disconnectConnectors, equals, fromCanonicalToKebabCase, fromCanonicalToSnakeCase, fromKebabToCanonicalCase, fromSnakeToCanonicalCase, getArrayConfigurationProperty, getArrayConfigurationProperty, getArrayConfigurationProperty, getBooleanConfigurationProperty, getConnection, getConnectorInstanceId, getDateConfigurationProperty, getIntConfigurationProperty, getLongConfigurationProperty, getStringConfigurationProperty, getStringConfigurationProperty, getSuppliedPlaceholderProperties, hashCode, initializeEmbeddedConnectors, initializeSecretsStoreConnector, isActive, logExceptionRecord, logRecord, logRecord, throwMissingConfigurationProperty, throwMissingEndpointAddress, throwMissingResource, throwNoAsset, throwWrongTypeOfAsset, throwWrongTypeOfResource, throwWrongTypeOfRootSchema, toString

Methods inherited from class org.odpi.openmetadata.frameworks.connectors.Connector

clearStatisticProperty, clearStatisticTimestamp, getConnectorStatistics, getStatisticCounter, getStatisticProperty, getStatisticTimestamp, incrementStatisticCounter, initializeStatisticCounter, refreshStatistics, setStatisticProperty, setStatisticTimestamp

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.odpi.openmetadata.frameworks.connectors.VirtualConnectorExtension

initializeEmbeddedConnectors

Constructor Details

FileBasedOpenLineageLogStoreConnector

public FileBasedOpenLineageLogStoreConnector()

Default constructor used by the connector provider.

Method Details

newDestinationIdentified

protected void newDestinationIdentified(String logStoreDirectoryName)
                                 throws ConnectorCheckedException

Informs the subclasses that there is a new destination - in case they need to do special setup.

Specified by:

newDestinationIdentified in class OpenLineageLogStoreConnectorBase

Parameters:

logStoreDirectoryName - new destination

Throws:

ConnectorCheckedException - new destination not valid

storeEvent

public void storeEvent(OpenLineageRunEvent openLineageEvent,
 String rawEvent,
 String logStoreDirectoryName)
                throws InvalidParameterException,
PropertyServerException

Store the open lineage event in the open lineage log store. If the raw event is null, a json version of the open lineage event is generated using the Egeria beans.

Specified by:

storeEvent in interface OpenLineageLogStore

Specified by:

storeEvent in class OpenLineageLogStoreConnectorBase

Parameters:

openLineageEvent - event formatted using Egeria beans

rawEvent - event in Json form from the originator - may have facets that are not known to Egeria

logStoreDirectoryName - address to send the event to

Throws:

InvalidParameterException - indicates that the openLineageEvent parameter is invalid.

PropertyServerException - indicates that the log store is not available or has an error.