Package org.odpi.openmetadata.frameworks.integration.openlineage

Class OpenLineageDataSetResolver

java.lang.Object

org.odpi.openmetadata.frameworks.integration.openlineage.OpenLineageDataSetResolver

public class OpenLineageDataSetResolver
extends Object

OpenLineageDataSetResolver finds every element in open metadata that describes an OpenLineage dataset. The dataset's namespace names the server (for example postgres://host:5432) and its name the resource within it (for example sales.public.orders). The matches are divided into two lists:

• Physical elements describe the physical landscape - the database table, topic or file itself. These are the elements that the lineage from OpenLineage events is attached to.
• Abstractions - TabularDataSet and TabularDataSetCollection assets - are views over the physical data (for example the data sets of digital products). They are linked to the physical elements with DataSetContent rather than carrying lineage themselves.

The matches come from:

• the qualified name, for datasets in the "egeria" namespace;
• the resourceName and namespacePath properties, which hold the OpenLineage name and namespace for elements catalogued from OpenLineage events or by connectors that follow the OpenLineage naming conventions;
• the endpoints of the connections of the elements catalogued by Egeria's technology connectors:
  • relational tables (postgres, mssql, oracle and db2 namespaces with a {database}.{schema}.{table} name): the database's endpoint (a JDBC URL naming the host, port and database) leads to the RelationalDatabase or DeployedDatabaseSchema asset, and the table is the RelationalTable schema attribute that the JDBC integration connector catalogued beneath it ({databaseQN}::{schema}::{table} or {schemaQN}::{table});
  • relational schemas ({database}.{schema} name, as used for the endpoint of a tabular data set collection): the DeployedDatabaseSchema;
  • Kafka topics: the topic whose endpoint names the topic and whose connection's bootstrap servers include the broker;
  • local files: the data stores whose pathName is the dataset name;
  • Unity Catalog tables: the table named {deployedImplementationType}::{server URL}::{full name} by the Unity Catalog connectors.

An abstraction that is the dataset itself (for example a tabular data set collection named in the egeria namespace) is also resolved through its own endpoint to the physical element it is a view over.

The physical elements are ranked so that the first one is the element to use: elements catalogued by technology connectors come before elements that were created from OpenLineage events (whose qualified name is {typeName}::{namespace}::{name}), and within each group the oldest comes first.

Results are cached for a short time because the same datasets appear in every run of a job.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final record	OpenLineageDataSetResolver.MatchedElement	An element that describes an OpenLineage dataset.
static final record	OpenLineageDataSetResolver.ResolvedDataSet	The elements that describe an OpenLineage dataset.

Constructor Summary

Constructors

Constructor	Description
OpenLineageDataSetResolver(ConnectorContextBase context, String sourceName)	Constructor.

Method Summary

Modifier and Type	Method	Description
void	forget(String namespace, String name)	Forget a cached resolution, for example because an element has been created for the dataset.
static String	getOpenLineageQualifiedName(String typeName, String namespace, String name)	Return the qualified name that elements created from OpenLineage events are given.
boolean	isAbstraction(ElementControlHeader elementHeader)	Is the element an abstraction over physical data (a TabularDataSet or TabularDataSetCollection)?
OpenLineageDataSetResolver.ResolvedDataSet	resolve(String namespace, String name)	Find every element that describes an OpenLineage dataset.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

OpenLineageDataSetResolver

public OpenLineageDataSetResolver(ConnectorContextBase context,
 String sourceName)

Constructor.

Parameters:

context - context of the calling connector or governance service

sourceName - name of the caller (for logging and property helper calls)

Method Details

getOpenLineageQualifiedName

public static String getOpenLineageQualifiedName(String typeName,
 String namespace,
 String name)

Return the qualified name that elements created from OpenLineage events are given.

Parameters:

typeName - open metadata type of the element

namespace - OpenLineage namespace

name - OpenLineage name

Returns:

qualified name

isAbstraction

public boolean isAbstraction(ElementControlHeader elementHeader)

Is the element an abstraction over physical data (a TabularDataSet or TabularDataSetCollection)?

Parameters:

elementHeader - header of the element

Returns:

boolean

resolve

public OpenLineageDataSetResolver.ResolvedDataSet resolve(String namespace,
 String name)
                                                   throws InvalidParameterException,
PropertyServerException,
UserNotAuthorizedException

Find every element that describes an OpenLineage dataset.

Parameters:

namespace - dataset namespace

name - dataset name

Returns:

resolved dataset (the lists are empty if nothing matches)

Throws:

InvalidParameterException - invalid parameter

PropertyServerException - repository problem

UserNotAuthorizedException - security problem

forget

public void forget(String namespace,
 String name)

Forget a cached resolution, for example because an element has been created for the dataset.

Parameters:

namespace - dataset namespace

name - dataset name