Package org.odpi.openmetadata.adapters.connectors.lineageinsight.openlineage

Class LovelaceOpenLineageAnalysisServiceBase

java.lang.Object

org.odpi.openmetadata.frameworks.connectors.Connector

org.odpi.openmetadata.frameworks.connectors.ConnectorBase

org.odpi.openmetadata.frameworks.opengovernance.GovernanceActionServiceConnector

org.odpi.openmetadata.frameworks.opengovernance.GeneralGovernanceActionService

org.odpi.openmetadata.adapters.connectors.lineageinsight.openlineage.LovelaceOpenLineageAnalysisServiceBase

All Implemented Interfaces:

AuditLoggingComponent, SecureConnectorExtension, VirtualConnectorExtension, GovernanceActionService

Direct Known Subclasses:

LovelaceOpenLineageDataQualitySummaryService, LovelaceOpenLineageDataScopeService, LovelaceOpenLineageRunProfilerService

public abstract class LovelaceOpenLineageAnalysisServiceBase
extends GeneralGovernanceActionService

LovelaceOpenLineageAnalysisServiceBase is the common base for the Lovelace services that analyse a historical OpenLineage log store. It locates the log store (from the openLineageLogStore action target, or the logStoreDirectory request parameter), reads the events in the analysis window into an OpenLineageRunHistory, hands the history to the subclass and records the completion status. It also provides the lookup of the processes and data assets that the OpenLineage cataloguer created (or matched) for the jobs and datasets, using the same identity rules as the cataloguer - a job is a DeployedSoftwareComponent and a dataset is a DataAsset whose resourceName and namespacePath are the OpenLineage name and namespace.

Nested Class Summary

Nested classes/interfaces inherited from class org.odpi.openmetadata.frameworks.connectors.ConnectorBase

ConnectorBase.ProtectedConnection

Field Summary

Fields

Modifier and Type	Field	Description
protected static final String	ANALYSIS_SERVICE_PROPERTY
protected static final String	ANALYSIS_TIME_PROPERTY
protected static final String	ANALYSIS_WINDOW_END_PROPERTY
protected static final String	ANALYSIS_WINDOW_START_PROPERTY
protected Date	analysisTime
protected Date	windowEnd
protected Date	windowStart

Fields inherited from class org.odpi.openmetadata.frameworks.opengovernance.GeneralGovernanceActionService

governanceContext

Fields inherited from class org.odpi.openmetadata.frameworks.opengovernance.GovernanceActionServiceConnector

governanceServiceName, propertyHelper

Fields inherited from class org.odpi.openmetadata.frameworks.connectors.ConnectorBase

auditLog, connectionBean, connectorInstanceId, embeddedConnectors, messageFormatter, secretsStoreConnectorMap

Constructor Summary

Constructors

Constructor	Description
LovelaceOpenLineageAnalysisServiceBase()

Method Summary

Modifier and Type	Method	Description
protected void	addAnalysisProperties(Map<String,String> properties)	Add the standard analysis properties to a map.
protected abstract int	analyse(OpenLineageRunHistory history)	Perform the analysis on the history read from the log store.
protected OpenMetadataRootElement	findDataAsset(OpenLineageRunHistory.Key dataset)	Find the element that represents an OpenLineage dataset, using the same rules as the cataloguer (see OpenLineageDataSetResolver): the first of the physical elements that describe the dataset - an asset, or a schema attribute such as a relational table - or, if there are none, an abstraction (tabular data set or collection) that is the dataset itself.
protected OpenMetadataRootElement	findProcess(OpenLineageRunHistory.JobHistory jobHistory)	Find the element that holds the run metrics of an OpenLineage job, using the same rules as the cataloguer.
protected String	formatDate(Date date)	Format a date for messages and properties (ISO-8601, UTC).
protected String	formatDuration(long millis)	Describe a duration in milliseconds in human terms.
protected abstract String	getAnalysisDescription()	Return a short description of what the service produces, for the completion message.
protected double	getWindowDays(OpenLineageRunHistory history)	Return the length of the analysis window in days, bounded by the events actually seen.
protected void	put(Map<String,String> map, String key, double value)	Add a double value to a string map, rounded to two decimal places.
protected void	put(Map<String,String> map, String key, long value)	Add a long value to a string map.
void	start()	Read the log store and run the analysis.

Methods inherited from class org.odpi.openmetadata.frameworks.opengovernance.GeneralGovernanceActionService

disconnect, getActionTarget, getAllActionTargets, getProperty, setGovernanceContext

Methods inherited from class org.odpi.openmetadata.frameworks.opengovernance.GovernanceActionServiceConnector

getArrayRequestParameter, getBooleanRequestParameter, getConnectorComponentDescription, getIntRequestParameter, getLongRequestParameter, getStringRequestParameter, handleUnexpectedException, setAuditLog, setGovernanceServiceName, validateContext

Methods inherited from class org.odpi.openmetadata.frameworks.connectors.ConnectorBase

combineConfigurationProperties, disconnectConnectors, equals, fromCamelToCanonicalCase, fromCanonicalToCamelCase, fromCanonicalToKebabCase, fromCanonicalToSnakeCase, fromKebabToCanonicalCase, fromSnakeToCanonicalCase, getArrayConfigurationProperty, getArrayConfigurationProperty, getArrayConfigurationProperty, getBooleanConfigurationProperty, getConnection, getConnectorInstanceId, getDateConfigurationProperty, getIntConfigurationProperty, getLongConfigurationProperty, getSecretsCollectionName, getSecretsLocation, getSecretsProvider, getStringConfigurationProperty, getStringConfigurationProperty, getSuppliedPlaceholderProperties, hashCode, initialize, initializeEmbeddedConnectors, initializeSecretsStoreConnector, isActive, logExceptionRecord, logExceptionRecord, logRecord, logRecord, throwMissingConfigurationProperty, throwMissingElement, throwMissingEndpointAddress, throwMissingResource, throwNoAsset, throwWrongTypeOfAsset, throwWrongTypeOfResource, throwWrongTypeOfRootSchema, toString

Methods inherited from class org.odpi.openmetadata.frameworks.connectors.Connector

clearStatisticProperty, clearStatisticTimestamp, getConnectorStatistics, getStatisticCounter, getStatisticProperty, getStatisticTimestamp, incrementStatisticCounter, initializeStatisticCounter, refreshStatistics, setStatisticProperty, setStatisticTimestamp

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Details

ANALYSIS_TIME_PROPERTY

protected static final String ANALYSIS_TIME_PROPERTY

See Also:

• Constant Field Values

ANALYSIS_WINDOW_START_PROPERTY

protected static final String ANALYSIS_WINDOW_START_PROPERTY

See Also:

• Constant Field Values

ANALYSIS_WINDOW_END_PROPERTY

protected static final String ANALYSIS_WINDOW_END_PROPERTY

See Also:

• Constant Field Values

ANALYSIS_SERVICE_PROPERTY

protected static final String ANALYSIS_SERVICE_PROPERTY

See Also:

• Constant Field Values

windowStart

protected Date windowStart

windowEnd

protected Date windowEnd

analysisTime

protected Date analysisTime

Constructor Details

LovelaceOpenLineageAnalysisServiceBase

public LovelaceOpenLineageAnalysisServiceBase()

Method Details

analyse

protected abstract int analyse(OpenLineageRunHistory history)
                        throws InvalidParameterException,
PropertyServerException,
UserNotAuthorizedException

Perform the analysis on the history read from the log store.

Parameters:

history - events in the analysis window, grouped by job and dataset

Returns:

number of elements updated

Throws:

InvalidParameterException - invalid parameter

PropertyServerException - repository problem

UserNotAuthorizedException - security problem

getAnalysisDescription

protected abstract String getAnalysisDescription()

Return a short description of what the service produces, for the completion message.

Returns:

description

start

public void start()
           throws ConnectorCheckedException,
UserNotAuthorizedException

Read the log store and run the analysis.

Overrides:

start in class GeneralGovernanceActionService

Throws:

ConnectorCheckedException - there is a problem within the governance action service.

UserNotAuthorizedException - the connector was disconnected before/during start

findProcess

protected OpenMetadataRootElement findProcess(OpenLineageRunHistory.JobHistory jobHistory)
                                       throws InvalidParameterException,
PropertyServerException,
UserNotAuthorizedException

Find the element that holds the run metrics of an OpenLineage job, using the same rules as the cataloguer. For a step of a governance action process (reported by the governance action publisher) this is the GovernanceActionProcessStep, since each step runs separately (see OpenLineageGovernanceActionResolver). Otherwise it is the job's process: the DeployedSoftwareComponent the cataloguer created, the process named by the legacy job naming, or the single process whose resourceName and namespacePath are the job's name and namespace.

Parameters:

jobHistory - history of the job

Returns:

element or null if there is no unique element for the job

Throws:

InvalidParameterException - invalid parameter

PropertyServerException - repository problem

UserNotAuthorizedException - security problem

findDataAsset

protected OpenMetadataRootElement findDataAsset(OpenLineageRunHistory.Key dataset)
                                         throws InvalidParameterException,
PropertyServerException,
UserNotAuthorizedException

Find the element that represents an OpenLineage dataset, using the same rules as the cataloguer (see OpenLineageDataSetResolver): the first of the physical elements that describe the dataset - an asset, or a schema attribute such as a relational table - or, if there are none, an abstraction (tabular data set or collection) that is the dataset itself.

Parameters:

dataset - dataset identity

Returns:

element or null if there is no element for the dataset

Throws:

InvalidParameterException - invalid parameter

PropertyServerException - repository problem

UserNotAuthorizedException - security problem

formatDate

protected String formatDate(Date date)

Format a date for messages and properties (ISO-8601, UTC).

Parameters:

date - date (may be null)

Returns:

string

formatDuration

protected String formatDuration(long millis)

Describe a duration in milliseconds in human terms.

Parameters:

millis - milliseconds

Returns:

string such as "2 hours 5 minutes"

addAnalysisProperties

protected void addAnalysisProperties(Map<String,String> properties)

Add the standard analysis properties to a map.

Parameters:

properties - map to fill

getWindowDays

protected double getWindowDays(OpenLineageRunHistory history)

Return the length of the analysis window in days, bounded by the events actually seen.

Parameters:

history - history

Returns:

days (at least 1)

put

protected void put(Map<String,String> map,
 String key,
 long value)

Add a long value to a string map.

Parameters:

map - map

key - key

value - value

put

protected void put(Map<String,String> map,
 String key,
 double value)

Add a double value to a string map, rounded to two decimal places.

Parameters:

map - map

key - key

value - value