Package org.odpi.openmetadata.adapters.connectors.darwin

Class DarwinProductDependencyManagerConnector

java.lang.Object
org.odpi.openmetadata.frameworks.connectors.Connector
org.odpi.openmetadata.frameworks.connectors.ConnectorBase
org.odpi.openmetadata.frameworks.integration.connectors.IntegrationConnectorBase
org.odpi.openmetadata.adapters.connectors.darwin.DarwinProductDependencyManagerConnector
All Implemented Interfaces:
AuditLoggingComponent, SecureConnectorExtension, VirtualConnectorExtension, IntegrationConnector
public class DarwinProductDependencyManagerConnector extends IntegrationConnectorBase
DarwinProductDependencyManagerConnector maintains the coarse-grained lineage that is implied by the finer-grained lineage beneath it, working upwards from the most detailed level on each refresh:
  • Schema elements to the data fabric. A DataMapping relationship between two schema elements - or between a schema element and a process - shows data being copied from one to the other. Where the two ends belong to different data fabric elements (a schema element belongs to the data asset it is anchored to; a process is its own), data flows from the first to the second, so a DataFlow relationship is maintained between them.
  • The data fabric to the system fabric. An item of IT infrastructure supports software capabilities (through the SupportedSoftwareCapability relationship), and a capability uses data assets and processes (through the CapabilityAssetUse relationship - with a useType of OWNS for a data asset, and any useType for a process). Where the lineage leads - directly or through intermediate elements - from an element used by one item's capabilities to an element used by another's, a relationship is maintained between the two items: a DataFlow for data lineage, a ProcessCall where a ProcessCall reaches the element, and a ControlFlow for a path of ControlFlow relationships.
  • Data assets to digital products. The assets that are members of a digital product (through the CollectionMembership relationship) are the start of paths followed downstream through the data lineage. A path may pass through any number of intermediate elements - processes, assets that belong to no product - but every relationship on it must belong to the same information supply chain (the iscQualifiedName of the lineage relationship). When the path reaches an asset that is a member of another product, that product depends on the product the path started from, through that information supply chain, so a DigitalProductDependency relationship is maintained between the products. The path stops there: the dependency on anything further downstream belongs to the product just reached.
Each of these is a boundary between two layers of lineage (see LineageBoundary): a lineage path in the lower layer between elements with different owners proves a relationship between the owners in the upper layer. The lineage of each lower layer is loaded once per refresh into a LineageGraph, and a LineageWalker follows the paths through it using the boundary's owner function. The layers are worked through from the bottom up, and the derived relationships of each layer are written before the layer is read by the ones above it.

The information supply chain of the finer-grained relationship is carried up onto the coarser one, and because every one of these relationship types is multi-link there is one relationship per information supply chain between the same two elements. Each level is reconciled with the relationships in the repository the same way. Relationships this connector created itself - recognized by the createdBy in their header - are removed when the finer-grained lineage no longer supports them, and missing ones are created. Relationships asserted by external users take precedence and are never removed: one whose information supply chain is not set is given the supply chain of the first finer-grained lineage that proves it.

When following the data fabric's lineage, a data set and the content it is a view over (DataSetContent) are treated as the same element, so a product whose member is an abstract data set reaches the lineage attached to the physical elements beneath it.

A relationship that names no information supply chain is treated as "supply chain unknown": it can join a path of any supply chain, and a path whose relationships name none proves a relationship of any supply chain in the layer above. Once a path's supply chain is known - from a relationship on it or from the external relationship it proves - the untagged external relationships on it are given that supply chain at the end of the refresh: filled in where one supply chain is known for them, and copied once for each supply chain where there are several.

An externally asserted relationship that the lineage beneath it does not prove is recorded as an exception, provided the layer beneath is catalogued for both of its ends - otherwise the missing detail is no evidence against it, and it is only counted as unchecked. The connector has an ExceptionType for each relationship type, created on first use, and links it to each element with unproven relationships of that type - the element the data or control reaches, or the dependent product - with an Exception relationship whose affectedRelationships property lists the unproven relationships. The exception is updated as the list changes and removed once nothing on it is left.

A relationship this connector created that someone else has since updated has been adopted. It is treated from then on as asserted by an external user: it is no longer updated or removed, and is validated like any other external relationship.

This connector works across the whole open metadata ecosystem rather than through catalog targets, which is why it extends IntegrationConnectorBase rather than DynamicIntegrationConnectorBase. Each refresh reconciles from a fresh snapshot, so nothing is carried between refreshes but the identities of the exception types.