Package org.odpi.openmetadata.adapters.connectors.integration.openlineage.postgres

Class PostgresOpenLineageLogStoreConnector

java.lang.Object

org.odpi.openmetadata.frameworks.connectors.Connector

org.odpi.openmetadata.frameworks.connectors.ConnectorBase

org.odpi.openmetadata.frameworks.integration.connectors.IntegrationConnectorBase

org.odpi.openmetadata.frameworks.integration.connectors.DynamicIntegrationConnectorBase

org.odpi.openmetadata.adapters.connectors.integration.openlineage.postgres.PostgresOpenLineageLogStoreConnector

All Implemented Interfaces:

AuditLoggingComponent, SecureConnectorExtension, VirtualConnectorExtension, CatalogTargetFactory, IntegrationConnector, OpenLineageEventListener

public class PostgresOpenLineageLogStoreConnector
extends DynamicIntegrationConnectorBase
implements OpenLineageEventListener

PostgresOpenLineageLogStoreConnector stores the open lineage events published to the integration daemon in the PostgreSQL database schemas attached to it as catalog targets. Each event is stored as received, along with a digest of its run, datasets, data quality assertions and tests that the analysis services can query by time window. The schemas and their tables are created if they do not exist. Each refresh deletes the records that are older than the daysToCleanUp configuration property.

Nested Class Summary

Nested classes/interfaces inherited from class org.odpi.openmetadata.frameworks.connectors.ConnectorBase

ConnectorBase.ProtectedConnection

Field Summary

Fields inherited from class org.odpi.openmetadata.frameworks.integration.connectors.IntegrationConnectorBase

catalogTargetsManager, connectorName, integrationContext, propertyHelper

Fields inherited from class org.odpi.openmetadata.frameworks.connectors.ConnectorBase

auditLog, connectionBean, connectorInstanceId, embeddedConnectors, messageFormatter, secretsStoreConnectorMap

Constructor Summary

Constructors

Constructor	Description
PostgresOpenLineageLogStoreConnector()

Method Summary

Modifier and Type	Method	Description
RequestedCatalogTarget	getNewRequestedCatalogTargetSkeleton(CatalogTarget retrievedCatalogTarget, CatalogTargetContext catalogTargetContext, Connector connectorToTarget)	Create a catalog target processor for each database schema.
void	processOpenLineageRunEvent(OpenLineageRunEvent event, String rawEvent)	Store each open lineage event in every database schema.
void	start()	Register to receive the open lineage events.

Methods inherited from class org.odpi.openmetadata.frameworks.integration.connectors.DynamicIntegrationConnectorBase

processEvent, refresh

Methods inherited from class org.odpi.openmetadata.frameworks.integration.connectors.IntegrationConnectorBase

disconnect, engage, getConnectorComponentDescription, getNetworkAddress, passEventToCatalogTargets, refreshCatalogTargets, setAuditLog, setConnectorName, setContext

Methods inherited from class org.odpi.openmetadata.frameworks.connectors.ConnectorBase

combineConfigurationProperties, disconnectConnectors, equals, fromCamelToCanonicalCase, fromCanonicalToCamelCase, fromCanonicalToKebabCase, fromCanonicalToSnakeCase, fromKebabToCanonicalCase, fromSnakeToCanonicalCase, getArrayConfigurationProperty, getArrayConfigurationProperty, getArrayConfigurationProperty, getArrayValue, getBooleanConfigurationProperty, getConnection, getConnectorInstanceId, getDateConfigurationProperty, getIntConfigurationProperty, getLongConfigurationProperty, getSecretsCollectionName, getSecretsLocation, getSecretsProvider, getStringConfigurationProperty, getStringConfigurationProperty, getSuppliedPlaceholderProperties, hashCode, initialize, initializeEmbeddedConnectors, initializeSecretsStoreConnector, isActive, logExceptionRecord, logExceptionRecord, logRecord, logRecord, throwMissingConfigurationProperty, throwMissingElement, throwMissingEndpointAddress, throwMissingResource, throwNoAsset, throwWrongTypeOfAsset, throwWrongTypeOfResource, throwWrongTypeOfRootSchema, toString

Methods inherited from class org.odpi.openmetadata.frameworks.connectors.Connector

clearStatisticProperty, clearStatisticTimestamp, getConnectorStatistics, getStatisticCounter, getStatisticProperty, getStatisticTimestamp, incrementStatisticCounter, initializeStatisticCounter, refreshStatistics, setStatisticProperty, setStatisticTimestamp

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.odpi.openmetadata.frameworks.integration.openlineage.OpenLineageEventListener

processOpenLineageDataSetEvent, processOpenLineageJobEvent

Methods inherited from interface org.odpi.openmetadata.frameworks.connectors.VirtualConnectorExtension

initializeEmbeddedConnectors

Constructor Details

PostgresOpenLineageLogStoreConnector

public PostgresOpenLineageLogStoreConnector()

Method Details

start

public void start()
           throws ConnectorCheckedException,
UserNotAuthorizedException

Register to receive the open lineage events.

Specified by:

start in interface IntegrationConnector

Overrides:

start in class IntegrationConnectorBase

Throws:

ConnectorCheckedException - the connector detected a problem.

UserNotAuthorizedException - the connector was disconnected before/during start

getNewRequestedCatalogTargetSkeleton

public RequestedCatalogTarget getNewRequestedCatalogTargetSkeleton(CatalogTarget retrievedCatalogTarget,
 CatalogTargetContext catalogTargetContext,
 Connector connectorToTarget)

Create a catalog target processor for each database schema.

Specified by:

getNewRequestedCatalogTargetSkeleton in interface CatalogTargetFactory

Specified by:

getNewRequestedCatalogTargetSkeleton in class DynamicIntegrationConnectorBase

Parameters:

retrievedCatalogTarget - details of the open metadata elements describing the catalog target

catalogTargetContext - specialized context for this catalog target

connectorToTarget - connector to access the target resource

Returns:

new processor based on the catalog target information

processOpenLineageRunEvent

public void processOpenLineageRunEvent(OpenLineageRunEvent event,
 String rawEvent)

Store each open lineage event in every database schema. The known catalog targets are used rather than refreshing the list for each event; the list is refreshed by refresh().

Specified by:

processOpenLineageRunEvent in interface OpenLineageEventListener

Parameters:

event - run event formatted using Egeria supplied beans (null if egeria can not format the event)

rawEvent - json payload received for the event