Package org.odpi.openmetadata.adapters.connectors.integration.openlineage

Class OpenLineageLogStoreConnectorBase

java.lang.Object

org.odpi.openmetadata.frameworks.connectors.Connector

org.odpi.openmetadata.frameworks.connectors.ConnectorBase

org.odpi.openmetadata.frameworks.integration.connectors.IntegrationConnectorBase

org.odpi.openmetadata.adapters.connectors.integration.openlineage.OpenLineageLogStoreConnectorBase

All Implemented Interfaces:

OpenLineageLogStore, AuditLoggingComponent, SecureConnectorExtension, VirtualConnectorExtension, IntegrationConnector, OpenLineageEventListener

Direct Known Subclasses:

APIBasedOpenLineageLogStoreConnector, FileBasedOpenLineageLogStoreConnector

public abstract class OpenLineageLogStoreConnectorBase
extends IntegrationConnectorBase
implements OpenLineageLogStore, OpenLineageEventListener

OpenLineageLogStoreConnectorBase is the base class for connectors that support the OMRSAuditLog. It has implementations of the query methods that throw "function not supported". This means that log destinations that do not support queries can ignore these methods. It also supports the start and stop method for the connector which only need to be overridden if the connector has work to do at these times

Nested Class Summary

Nested classes/interfaces inherited from class org.odpi.openmetadata.frameworks.connectors.ConnectorBase

ConnectorBase.ProtectedConnection

Field Summary

Fields

Modifier and Type	Field	Description
protected final List<String>	destinationAddresses
protected String	distributorName
protected IntegrationContext	myContext

Fields inherited from class org.odpi.openmetadata.frameworks.integration.connectors.IntegrationConnectorBase

auditLog, catalogTargetsManager, connectorName, integrationContext, propertyHelper

Fields inherited from class org.odpi.openmetadata.frameworks.connectors.ConnectorBase

connectionBean, connectorInstanceId, embeddedConnectors, messageFormatter, secretsStoreConnectorMap

Constructor Summary

Constructors

Constructor	Description
OpenLineageLogStoreConnectorBase()	Default constructor

Method Summary

Modifier and Type	Method	Description
void	initialize(String connectorInstanceId, Connection connectionDetails)	Call made by the ConnectorProvider to initialize the Connector with the base services.
protected void	logNoRawEvent(OpenLineageRunEvent openLineageEvent)	Throws an invalid parameter exception.
protected abstract void	newDestinationIdentified(String destinationAddress)	Informs the subclasses that there is a new destination - in case they need to do special setup.
void	processOpenLineageRunEvent(OpenLineageRunEvent event, String rawEvent)	Called each time an open lineage run event is published to the integration daemon.
void	refresh()	Maintains the list of catalog targets.
void	start()	Indicates that the connector is completely configured and can begin processing.
abstract void	storeEvent(OpenLineageRunEvent openLineageEvent, String rawEvent, String logDestinationAddress)	Store the open lineage event in the open lineage log store.

Methods inherited from class org.odpi.openmetadata.frameworks.integration.connectors.IntegrationConnectorBase

disconnect, engage, getConnectorComponentDescription, getNetworkAddress, passEventToCatalogTargets, refreshCatalogTargets, setAuditLog, setConnectorName, setContext

Methods inherited from class org.odpi.openmetadata.frameworks.connectors.ConnectorBase

combineConfigurationProperties, disconnectConnectors, equals, fromCamelToCanonicalCase, fromCanonicalToCamelCase, fromCanonicalToKebabCase, fromCanonicalToSnakeCase, fromKebabToCanonicalCase, fromSnakeToCanonicalCase, getArrayConfigurationProperty, getArrayConfigurationProperty, getArrayConfigurationProperty, getBooleanConfigurationProperty, getConnection, getConnectorInstanceId, getDateConfigurationProperty, getIntConfigurationProperty, getLongConfigurationProperty, getSecretsCollectionName, getSecretsLocation, getSecretsProvider, getStringConfigurationProperty, getStringConfigurationProperty, getSuppliedPlaceholderProperties, hashCode, initializeEmbeddedConnectors, initializeSecretsStoreConnector, isActive, logExceptionRecord, logRecord, logRecord, throwMissingConfigurationProperty, throwMissingEndpointAddress, throwMissingResource, throwNoAsset, throwWrongTypeOfAsset, throwWrongTypeOfResource, throwWrongTypeOfRootSchema, toString

Methods inherited from class org.odpi.openmetadata.frameworks.connectors.Connector

clearStatisticProperty, clearStatisticTimestamp, getConnectorStatistics, getStatisticCounter, getStatisticProperty, getStatisticTimestamp, incrementStatisticCounter, initializeStatisticCounter, refreshStatistics, setStatisticProperty, setStatisticTimestamp

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.odpi.openmetadata.frameworks.connectors.VirtualConnectorExtension

initializeEmbeddedConnectors

Field Details

distributorName

protected String distributorName

myContext

protected IntegrationContext myContext

destinationAddresses

protected final List<String> destinationAddresses

Constructor Details

OpenLineageLogStoreConnectorBase

public OpenLineageLogStoreConnectorBase()

Default constructor

Method Details

storeEvent

public abstract void storeEvent(OpenLineageRunEvent openLineageEvent,
 String rawEvent,
 String logDestinationAddress)
                         throws InvalidParameterException,
UserNotAuthorizedException,
PropertyServerException

Store the open lineage event in the open lineage log store. If the raw event is null, a json version of the open lineage event is generated using the Egeria beans.

Specified by:

storeEvent in interface OpenLineageLogStore

Parameters:

openLineageEvent - event formatted using Egeria beans

rawEvent - event in Json form from the originator - may have facets that are not known to Egeria

logDestinationAddress - address to send the event to

Throws:

InvalidParameterException - indicates that the openLineageEvent parameter is invalid.

UserNotAuthorizedException - indicates that the caller is not authorized to access the log store.

PropertyServerException - indicates that the log store is not available or has an error.

newDestinationIdentified

protected abstract void newDestinationIdentified(String destinationAddress)
                                          throws ConnectorCheckedException

Informs the subclasses that there is a new destination - in case they need to do special setup.

Parameters:

destinationAddress - new destination

Throws:

ConnectorCheckedException - new destination not valid

initialize

public void initialize(String connectorInstanceId,
 Connection connectionDetails)
                throws ConnectorCheckedException

Call made by the ConnectorProvider to initialize the Connector with the base services.

Overrides:

initialize in class ConnectorBase

Parameters:

connectorInstanceId - unique id for the connector instance useful for messages etc

connectionDetails - POJO for the configuration used to create the connector.

Throws:

ConnectorCheckedException - a problem within the connector.

start

public void start()
           throws ConnectorCheckedException,
UserNotAuthorizedException

Indicates that the connector is completely configured and can begin processing.

Specified by:

start in interface IntegrationConnector

Overrides:

start in class IntegrationConnectorBase

Throws:

ConnectorCheckedException - the connector detected a problem.

UserNotAuthorizedException - the connector was disconnected before/during start

refresh

public void refresh()
             throws ConnectorCheckedException

Maintains the list of catalog targets.

Specified by:

refresh in interface IntegrationConnector

Throws:

ConnectorCheckedException - there is a problem with the connector. It is not able to refresh the catalog targets.

processOpenLineageRunEvent

public void processOpenLineageRunEvent(OpenLineageRunEvent event,
 String rawEvent)

Called each time an open lineage run event is published to the integration daemon. The integration connector is able to work with the formatted event using the Egeria beans or reformat the open lineage run event using the supplied open lineage backend beans or another set of beans.

Specified by:

processOpenLineageRunEvent in interface OpenLineageEventListener

Parameters:

event - run event formatted using Egeria supplied beans (null if egeria can not format the event)

rawEvent - json payload received for the event

logNoRawEvent

protected void logNoRawEvent(OpenLineageRunEvent openLineageEvent)
                      throws InvalidParameterException

Throws an invalid parameter exception. Used by the subclasses when this class has failed to pass a raw event.

Parameters:

openLineageEvent - supplied open lineage event - may also be null

Throws:

InvalidParameterException - resulting exception