Package org.odpi.openmetadata.adapters.connectors.mendel

Class MendelAutomatedDuplicateManagerConnector

java.lang.Object
org.odpi.openmetadata.frameworks.connectors.Connector
org.odpi.openmetadata.frameworks.connectors.ConnectorBase
org.odpi.openmetadata.frameworks.integration.connectors.IntegrationConnectorBase
org.odpi.openmetadata.adapters.connectors.mendel.MendelAutomatedDuplicateManagerConnector
All Implemented Interfaces:
AuditLoggingComponent, SecureConnectorExtension, VirtualConnectorExtension, IntegrationConnector, OpenMetadataEventListener
public class MendelAutomatedDuplicateManagerConnector extends IntegrationConnectorBase implements OpenMetadataEventListener
MendelAutomatedDuplicateManagerConnector manages the duplicate links and classifications that drive the deduplication of open metadata elements on retrieval. Each refresh makes three passes over the PeerDuplicateLink relationships in the open metadata ecosystem.
  • The links that no steward has ruled on yet - the DISCOVERED, PROPOSED and IMPORTED ones. Where the linked elements are a close enough match, the link is moved to VALIDATED and the KnownDuplicate classification is added to both elements, which is the combination that causes the retrieval processing to combine them. The rest are passed to a person appointed to the DuplicateMetadataSteward role via a to do.
  • The links that a steward has retired - the DEPRECATED and OBSOLETE ones. The KnownDuplicate classification is removed from an element once none of its duplicate links are live, so that the element stops being combined with anything.
  • The clusters of validated peer duplicates. Once a cluster reaches the configured size, its members are consolidated into a single element using the survivorship rules in MendelDuplicateConsolidator.
Each pass works from one snapshot of the duplicate links, so the links that this refresh validates are consolidated by the next refresh rather than this one.

Once the first refresh has worked through the backlog, the connector also listens for open metadata events so that a new or updated duplicate link is reviewed as soon as it appears, rather than waiting for the next refresh. The update that this connector makes to a duplicate link produces another event for the same relationship, which is ignored because the status is VALIDATED by then, so there is no loop.

This connector works across the whole open metadata ecosystem rather than through catalog targets, which is why it extends IntegrationConnectorBase rather than DynamicIntegrationConnectorBase. It registers its own listener as a result, and - like the dynamic base class - it ignores events while a refresh is in progress so that a duplicate link is not worked on by both threads at once.