Package org.odpi.openmetadata.adapters.connectors.mendel

Class MendelAutomatedDuplicateManagerConnector

java.lang.Object
org.odpi.openmetadata.frameworks.connectors.Connector
org.odpi.openmetadata.frameworks.connectors.ConnectorBase
org.odpi.openmetadata.frameworks.integration.connectors.IntegrationConnectorBase
org.odpi.openmetadata.adapters.connectors.mendel.MendelAutomatedDuplicateManagerConnector
All Implemented Interfaces:
AuditLoggingComponent, SecureConnectorExtension, VirtualConnectorExtension, IntegrationConnector, OpenMetadataEventListener
public class MendelAutomatedDuplicateManagerConnector extends IntegrationConnectorBase implements OpenMetadataEventListener
MendelAutomatedDuplicateManagerConnector manages the duplicate links and classifications that drive the deduplication of open metadata elements on retrieval. Each refresh makes four passes over the PeerDuplicateLink relationships in the open metadata ecosystem.
  • The links that no steward has ruled on yet - the DISCOVERED, PROPOSED and IMPORTED ones. Where the linked elements are a close enough match, the link is moved to VALIDATED and the KnownDuplicate classification is added to both elements, which is the combination that causes the retrieval processing to combine them. The rest are passed to a person appointed to the DuplicateMetadataSteward role via a to do.
  • The validated links that this connector decided itself, told apart from a steward's by the userId in their updatedBy. A close match can stop being one - a qualified name is corrected, say - and nothing else revisits a validated link, so a link whose grounds have gone is retired. A steward's decision is never reconsidered.
  • The links that have been retired - the DEPRECATED and OBSOLETE ones. The KnownDuplicate classification is removed from an element once it is no longer deduplicated by any route - no live peer link and no consolidated cluster to be reached through - so that the element stops being combined with anything.
  • The clusters of validated peer duplicates. Once a cluster reaches the configured size, its members are consolidated into a single element using the survivorship rules in MendelDuplicateConsolidator.
Each pass works from one snapshot of the duplicate links, so the links that this refresh validates are consolidated by the next refresh rather than this one.

Once the first refresh has worked through the backlog, the connector also listens for open metadata events so that a new or updated duplicate link is reviewed as soon as it appears, rather than waiting for the next refresh. The update that this connector makes to a duplicate link produces another event for the same relationship, which is ignored because the status is VALIDATED by then, so there is no loop.

This connector works across the whole open metadata ecosystem rather than through catalog targets, which is why it extends IntegrationConnectorBase rather than DynamicIntegrationConnectorBase. It registers its own listener as a result, and - like the dynamic base class - it ignores events while a refresh is in progress so that a duplicate link is not worked on by both threads at once.