Package org.odpi.openmetadata.adapters.connectors.lineageinsight.openlineage

Class LovelaceOpenLineageDataQualitySummaryService

java.lang.Object

org.odpi.openmetadata.frameworks.connectors.Connector

org.odpi.openmetadata.frameworks.connectors.ConnectorBase

org.odpi.openmetadata.frameworks.opengovernance.GovernanceActionServiceConnector

org.odpi.openmetadata.frameworks.opengovernance.GeneralGovernanceActionService

org.odpi.openmetadata.adapters.connectors.lineageinsight.openlineage.LovelaceOpenLineageAnalysisServiceBase

org.odpi.openmetadata.adapters.connectors.lineageinsight.openlineage.LovelaceOpenLineageDataQualitySummaryService

All Implemented Interfaces:

AuditLoggingComponent, SecureConnectorExtension, VirtualConnectorExtension, GovernanceActionService

public class LovelaceOpenLineageDataQualitySummaryService
extends LovelaceOpenLineageAnalysisServiceBase

LovelaceOpenLineageDataQualitySummaryService analyses the data quality assertions and tests reported in an OpenLineage log store and records, for each dataset (and for each job's tests), the pass rate of each quality dimension over the analysis window. The OpenLineage cataloguer records every assertion of every run as it arrives; this service produces one survey report per element with one QualityAnnotation per dimension whose score is the pass rate, so that the current quality of a dataset can be read without scanning the run history.

Nested Class Summary

Nested classes/interfaces inherited from class org.odpi.openmetadata.frameworks.connectors.ConnectorBase

ConnectorBase.ProtectedConnection

Field Summary

Fields inherited from class org.odpi.openmetadata.adapters.connectors.lineageinsight.openlineage.LovelaceOpenLineageAnalysisServiceBase

ANALYSIS_SERVICE_PROPERTY, ANALYSIS_TIME_PROPERTY, ANALYSIS_WINDOW_END_PROPERTY, ANALYSIS_WINDOW_START_PROPERTY, analysisTime, windowEnd, windowStart

Fields inherited from class org.odpi.openmetadata.frameworks.opengovernance.GeneralGovernanceActionService

governanceContext

Fields inherited from class org.odpi.openmetadata.frameworks.opengovernance.GovernanceActionServiceConnector

governanceServiceName, propertyHelper

Fields inherited from class org.odpi.openmetadata.frameworks.connectors.ConnectorBase

auditLog, connectionBean, connectorInstanceId, embeddedConnectors, messageFormatter, secretsStoreConnectorMap

Constructor Summary

Constructors

Constructor	Description
LovelaceOpenLineageDataQualitySummaryService()	Default constructor

Method Summary

Modifier and Type	Method	Description
protected int	analyse(OpenLineageRunHistory history)	Summarise the data quality results for each dataset and job.
protected String	getAnalysisDescription()	Describe what the service produces.

Methods inherited from class org.odpi.openmetadata.adapters.connectors.lineageinsight.openlineage.LovelaceOpenLineageAnalysisServiceBase

addAnalysisProperties, findDataAsset, findProcess, formatDate, formatDuration, getWindowDays, put, put, start

Methods inherited from class org.odpi.openmetadata.frameworks.opengovernance.GeneralGovernanceActionService

disconnect, getActionTarget, getAllActionTargets, getProperty, setGovernanceContext

Methods inherited from class org.odpi.openmetadata.frameworks.opengovernance.GovernanceActionServiceConnector

getArrayRequestParameter, getBooleanRequestParameter, getConnectorComponentDescription, getIntRequestParameter, getLongRequestParameter, getStringRequestParameter, handleUnexpectedException, setAuditLog, setGovernanceServiceName, validateContext

Methods inherited from class org.odpi.openmetadata.frameworks.connectors.ConnectorBase

combineConfigurationProperties, disconnectConnectors, equals, fromCamelToCanonicalCase, fromCanonicalToCamelCase, fromCanonicalToKebabCase, fromCanonicalToSnakeCase, fromKebabToCanonicalCase, fromSnakeToCanonicalCase, getArrayConfigurationProperty, getArrayConfigurationProperty, getArrayConfigurationProperty, getBooleanConfigurationProperty, getConnection, getConnectorInstanceId, getDateConfigurationProperty, getIntConfigurationProperty, getLongConfigurationProperty, getSecretsCollectionName, getSecretsLocation, getSecretsProvider, getStringConfigurationProperty, getStringConfigurationProperty, getSuppliedPlaceholderProperties, hashCode, initialize, initializeEmbeddedConnectors, initializeSecretsStoreConnector, isActive, logExceptionRecord, logExceptionRecord, logRecord, logRecord, throwMissingConfigurationProperty, throwMissingElement, throwMissingEndpointAddress, throwMissingResource, throwNoAsset, throwWrongTypeOfAsset, throwWrongTypeOfResource, throwWrongTypeOfRootSchema, toString

Methods inherited from class org.odpi.openmetadata.frameworks.connectors.Connector

clearStatisticProperty, clearStatisticTimestamp, getConnectorStatistics, getStatisticCounter, getStatisticProperty, getStatisticTimestamp, incrementStatisticCounter, initializeStatisticCounter, refreshStatistics, setStatisticProperty, setStatisticTimestamp

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Details

LovelaceOpenLineageDataQualitySummaryService

public LovelaceOpenLineageDataQualitySummaryService()

Default constructor

Method Details

getAnalysisDescription

protected String getAnalysisDescription()

Describe what the service produces.

Specified by:

getAnalysisDescription in class LovelaceOpenLineageAnalysisServiceBase

Returns:

description

analyse

protected int analyse(OpenLineageRunHistory history)
               throws InvalidParameterException,
PropertyServerException,
UserNotAuthorizedException

Summarise the data quality results for each dataset and job.

Specified by:

analyse in class LovelaceOpenLineageAnalysisServiceBase

Parameters:

history - events in the analysis window

Returns:

number of survey reports created

Throws:

InvalidParameterException - invalid parameter

PropertyServerException - repository problem

UserNotAuthorizedException - security problem