Class FileBasedOpenLineageLogStoreConnector
java.lang.Object
org.odpi.openmetadata.frameworks.connectors.Connector
org.odpi.openmetadata.frameworks.connectors.ConnectorBase
org.odpi.openmetadata.frameworks.integration.connectors.IntegrationConnectorBase
org.odpi.openmetadata.integrationservices.lineage.connector.LineageIntegratorConnector
org.odpi.openmetadata.adapters.connectors.integration.openlineage.OpenLineageLogStoreConnectorBase
org.odpi.openmetadata.adapters.connectors.integration.openlineage.FileBasedOpenLineageLogStoreConnector
- All Implemented Interfaces:
OpenLineageLogStore
,AuditLoggingComponent
,SecureConnectorExtension
,VirtualConnectorExtension
,IntegrationConnector
,LineageIntegratorOMISConnector
,OpenLineageEventListener
FileBasedOpenLineageLogStoreConnector provides a connector implementation for a file based open lineage log.
The open lineage log is stored in a directory and each open lineage event record is stored as a file with a filename built
from the record's unique identifier (runId), time and status. The record is stored in a subdirectory that is made from the namespace and job.
-
Nested Class Summary
Nested classes/interfaces inherited from class org.odpi.openmetadata.frameworks.connectors.ConnectorBase
ConnectorBase.ProtectedConnection
-
Field Summary
Fields inherited from class org.odpi.openmetadata.adapters.connectors.integration.openlineage.OpenLineageLogStoreConnectorBase
destinationAddresses, distributorName, myContext
Fields inherited from class org.odpi.openmetadata.frameworks.integration.connectors.IntegrationConnectorBase
auditLog, catalogTargetsManager, connectorName, integrationContext, propertyHelper
Fields inherited from class org.odpi.openmetadata.frameworks.connectors.ConnectorBase
connectedAssetProperties, connectionBean, connectionProperties, connectorInstanceId, embeddedConnectors, messageFormatter, secretsStoreConnectorMap
-
Constructor Summary
ConstructorDescriptionDefault constructor used by the connector provider. -
Method Summary
Modifier and TypeMethodDescriptionprotected void
newDestinationIdentified
(String logStoreDirectoryName) Informs the subclasses that there is a new destination - in case they need to do special setup.void
storeEvent
(OpenLineageRunEvent openLineageEvent, String rawEvent, String logStoreDirectoryName) Store the open lineage event in the open lineage log store.Methods inherited from class org.odpi.openmetadata.adapters.connectors.integration.openlineage.OpenLineageLogStoreConnectorBase
initialize, logNoRawEvent, processOpenLineageRunEvent, refresh, start
Methods inherited from class org.odpi.openmetadata.integrationservices.lineage.connector.LineageIntegratorConnector
getContext, setContext
Methods inherited from class org.odpi.openmetadata.frameworks.integration.connectors.IntegrationConnectorBase
disconnect, engage, getConnectorComponentDescription, getNetworkAddress, refreshCatalogTargets, registerCatalogTargetChangeListener, setAuditLog, setConnectorName, setContext
Methods inherited from class org.odpi.openmetadata.frameworks.connectors.ConnectorBase
disconnectConnectors, equals, getArrayConfigurationProperty, getAssetTypeName, getBooleanConfigurationProperty, getConnectedAssetProperties, getConnection, getConnectorInstanceId, getIntConfigurationProperty, getLongConfigurationProperty, getNetworkAddresses, getStringConfigurationProperty, getSuppliedPlaceholderProperties, hashCode, initializeConnectedAssetProperties, initializeEmbeddedConnectors, initializeSecretsStoreConnector, isActive, logExceptionRecord, logRecord, logRecord, throwMissingConfigurationProperty, throwMissingResource, throwNoAsset, throwWrongTypeOfAsset, throwWrongTypeOfResource, throwWrongTypeOfRootSchema, toString
Methods inherited from class org.odpi.openmetadata.frameworks.connectors.Connector
clearStatisticProperty, clearStatisticTimestamp, getConnectorStatistics, getStatisticCounter, getStatisticProperty, getStatisticTimestamp, incrementStatisticCounter, initializeStatisticCounter, setStatisticProperty, setStatisticTimestamp
Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait
Methods inherited from interface org.odpi.openmetadata.frameworks.connectors.VirtualConnectorExtension
initializeEmbeddedConnectors
-
Constructor Details
-
FileBasedOpenLineageLogStoreConnector
public FileBasedOpenLineageLogStoreConnector()Default constructor used by the connector provider.
-
-
Method Details
-
newDestinationIdentified
protected void newDestinationIdentified(String logStoreDirectoryName) throws ConnectorCheckedException Informs the subclasses that there is a new destination - in case they need to do special setup.- Specified by:
newDestinationIdentified
in classOpenLineageLogStoreConnectorBase
- Parameters:
logStoreDirectoryName
- new destination- Throws:
ConnectorCheckedException
- new destination not valid
-
storeEvent
public void storeEvent(OpenLineageRunEvent openLineageEvent, String rawEvent, String logStoreDirectoryName) throws InvalidParameterException, PropertyServerException Store the open lineage event in the open lineage log store. If the raw event is null, a json version of the open lineage event is generated using the Egeria beans.- Specified by:
storeEvent
in interfaceOpenLineageLogStore
- Specified by:
storeEvent
in classOpenLineageLogStoreConnectorBase
- Parameters:
openLineageEvent
- event formatted using Egeria beansrawEvent
- event in Json form from the originator - may have facets that are not known to EgerialogStoreDirectoryName
- address to send the event to- Throws:
InvalidParameterException
- indicates that the openLineageEvent parameter is invalid.PropertyServerException
- indicates that the log store is not available or has an error.
-