Package org.odpi.openmetadata.adapters.connectors.lineageinsight.openlineage

Class LovelaceOpenLineageAnalysisServiceBase

java.lang.Object
org.odpi.openmetadata.frameworks.connectors.Connector
org.odpi.openmetadata.frameworks.connectors.ConnectorBase
org.odpi.openmetadata.frameworks.opengovernance.GovernanceActionServiceConnector
org.odpi.openmetadata.frameworks.opengovernance.GeneralGovernanceActionService
org.odpi.openmetadata.adapters.connectors.lineageinsight.openlineage.LovelaceOpenLineageAnalysisServiceBase
All Implemented Interfaces:
AuditLoggingComponent, SecureConnectorExtension, VirtualConnectorExtension, GovernanceActionService
Direct Known Subclasses:
LovelaceOpenLineageDataQualitySummaryService, LovelaceOpenLineageDataScopeService, LovelaceOpenLineageRunProfilerService
public abstract class LovelaceOpenLineageAnalysisServiceBase extends GeneralGovernanceActionService
LovelaceOpenLineageAnalysisServiceBase is the common base for the Lovelace services that analyse a historical OpenLineage log store. It locates the log store (from the openLineageLogStore action target, or the logStoreDirectory request parameter), reads the events in the analysis window into an OpenLineageRunHistory, hands the history to the subclass and records the completion status. It also provides the lookup of the processes and data assets that the OpenLineage cataloguer created (or matched) for the jobs and datasets, using the same identity rules as the cataloguer - a job is a DeployedSoftwareComponent and a dataset is a DataAsset whose resourceName and namespacePath are the OpenLineage name and namespace.

  • Field Details

  • Constructor Details

    • LovelaceOpenLineageAnalysisServiceBase

      public LovelaceOpenLineageAnalysisServiceBase()

  • Method Details

    • analyse

      protected abstract int analyse(OpenLineageRunHistory history) throws InvalidParameterException, PropertyServerException, UserNotAuthorizedException
      Perform the analysis on the history read from the log store.
      Parameters:
      history - events in the analysis window, grouped by job and dataset
      Returns:
      number of elements updated
      Throws:
      InvalidParameterException - invalid parameter
      PropertyServerException - repository problem
      UserNotAuthorizedException - security problem

    • getAnalysisDescription

      protected abstract String getAnalysisDescription()
      Return a short description of what the service produces, for the completion message.
      Returns:
      description

    • start

      public void start() throws ConnectorCheckedException, UserNotAuthorizedException
      Read the log store and run the analysis.
      Overrides:
      start in class GeneralGovernanceActionService
      Throws:
      ConnectorCheckedException - there is a problem within the governance action service.
      UserNotAuthorizedException - the connector was disconnected before/during start

    • withoutRelationships

      protected static <T extends GetOptions> T withoutRelationships(T options)
      Stop a read returning the relationships of each element it finds. The elements found by findProcess, findDataAsset and findUniqueMatch are only used for their header (GUID and classifications such as RunMetrics and DataScope) and properties.
      Type Parameters:
      T - type of options
      Parameters:
      options - options from the client
      Returns:
      the same options, set not to return relationships

    • findProcess

      protected OpenMetadataRootElement findProcess(OpenLineageRunHistory.JobHistory jobHistory) throws InvalidParameterException, PropertyServerException, UserNotAuthorizedException
      Find the element that holds the run metrics of an OpenLineage job, using the same rules as the cataloguer. For a step of a governance action process (reported by the governance action publisher) this is the GovernanceActionProcessStep, since each step runs separately (see OpenLineageGovernanceActionResolver). Otherwise it is the job's process: the DeployedSoftwareComponent the cataloguer created, the process named by the legacy job naming, or the single process whose resourceName and namespacePath are the job's name and namespace.
      Parameters:
      jobHistory - history of the job
      Returns:
      element or null if there is no unique element for the job
      Throws:
      InvalidParameterException - invalid parameter
      PropertyServerException - repository problem
      UserNotAuthorizedException - security problem

    • findDataAsset

      protected OpenMetadataRootElement findDataAsset(OpenLineageRunHistory.Key dataset) throws InvalidParameterException, PropertyServerException, UserNotAuthorizedException
      Find the element that represents an OpenLineage dataset, using the same rules as the cataloguer (see OpenLineageDataSetResolver): the first of the physical elements that describe the dataset - an asset, or a schema attribute such as a relational table - or, if there are none, an abstraction (tabular data set or collection) that is the dataset itself.
      Parameters:
      dataset - dataset identity
      Returns:
      element or null if there is no element for the dataset
      Throws:
      InvalidParameterException - invalid parameter
      PropertyServerException - repository problem
      UserNotAuthorizedException - security problem

    • formatDate

      protected String formatDate(Date date)
      Format a date for messages and properties (ISO-8601, UTC).
      Parameters:
      date - date (may be null)
      Returns:
      string

    • formatDuration

      protected String formatDuration(long millis)
      Describe a duration in milliseconds in human terms.
      Parameters:
      millis - milliseconds
      Returns:
      string such as "2 hours 5 minutes"

    • addAnalysisProperties

      protected void addAnalysisProperties(Map<String,String> properties)
      Add the standard analysis properties to a map.
      Parameters:
      properties - map to fill

    • getWindowDays

      protected double getWindowDays(OpenLineageRunHistory history)
      Return the length of the analysis window in days, bounded by the events actually seen.
      Parameters:
      history - history
      Returns:
      days (at least 1)

    • put

      protected void put(Map<String,String> map, String key, long value)
      Add a long value to a string map.
      Parameters:
      map - map
      key - key
      value - value

    • put

      protected void put(Map<String,String> map, String key, double value)
      Add a double value to a string map, rounded to two decimal places.
      Parameters:
      map - map
      key - key
      value - value