Package org.odpi.openmetadata.adapters.connectors.unitycatalog.sync

Class OSSUnityCatalogInsideCatalogSyncBase

java.lang.Object
org.odpi.openmetadata.adapters.connectors.unitycatalog.sync.OSSUnityCatalogInsideCatalogSyncBase
Direct Known Subclasses:
OSSUnityCatalogInsideCatalogSyncFunctions, OSSUnityCatalogInsideCatalogSyncRegisteredModels, OSSUnityCatalogInsideCatalogSyncSchema, OSSUnityCatalogInsideCatalogSyncTables, OSSUnityCatalogInsideCatalogSyncVolumes, OSSUnityCatalogServerSyncCatalog
public abstract class OSSUnityCatalogInsideCatalogSyncBase extends Object
Common functions for the synchronizing between Egeria and Unity Catalog.

  • Field Details

    • propertyHelper

      protected final PropertyHelper propertyHelper

    • connectorName

      protected final String connectorName

    • context

      protected final IntegrationContext context

    • serverGUID

      protected final String serverGUID

    • catalogGUID

      protected final String catalogGUID

    • catalogTypeName

      protected final String catalogTypeName

    • metadataCollectionGUID

      protected final String metadataCollectionGUID

    • metadataCollectionQualifiedName

      protected final String metadataCollectionQualifiedName

    • catalogTargetName

      protected final String catalogTargetName

    • targetPermittedSynchronization

      protected final PermittedSynchronization targetPermittedSynchronization

    • ucConnector

      protected final OSSUnityCatalogResourceConnector ucConnector

    • ucServerEndpoint

      protected final String ucServerEndpoint

    • entityTypeName

      protected final String entityTypeName

    • deployedImplementationType

      protected final UnityCatalogDeployedImplementationType deployedImplementationType

    • templates

      protected final Map<String,String> templates

    • configurationProperties

      protected final Map<String,Object> configurationProperties

    • auditLog

      protected final AuditLog auditLog

    • openMetadataStore

      protected final OpenMetadataStore openMetadataStore

    • externalIdClient

      protected final ExternalIdClient externalIdClient

    • propertyFacetClient

      protected final PropertyFacetClient propertyFacetClient

    • assetClient

      protected final AssetClient assetClient

    • excludeNames

      protected final List<String> excludeNames

    • includeNames

      protected final List<String> includeNames

    • templateGUID

      protected final String templateGUID

    • ucFullNameToEgeriaGUID

      protected final Map<String,String> ucFullNameToEgeriaGUID

  • Constructor Details

    • OSSUnityCatalogInsideCatalogSyncBase

      public OSSUnityCatalogInsideCatalogSyncBase(String connectorName, IntegrationContext context, String catalogTargetName, String serverGUID, String catalogGUID, String metadataCollectionGUID, String metadataCollectionQualifiedName, Map<String,String> ucFullNameToEgeriaGUID, PermittedSynchronization targetPermittedSynchronization, OSSUnityCatalogResourceConnector ucConnector, String ucServerEndpoint, String entityTypeName, UnityCatalogDeployedImplementationType deployedImplementationType, Map<String,String> templates, Map<String,Object> configurationProperties, List<String> excludeNames, List<String> includeNames, AuditLog auditLog) throws UserNotAuthorizedException, InvalidParameterException
      Set up the schema synchronizer.
      Parameters:
      connectorName - name of this connector
      context - context for the connector
      catalogTargetName - the catalog target name
      serverGUID - guid of the Unity Catalog server asset that everything catalogued from it is scoped to
      catalogGUID - guid of the catalog
      metadataCollectionGUID - the managedMetadataCollectionId of the metadata collection for this UC server
      metadataCollectionQualifiedName - name of the metadata collection for this UC server
      ucFullNameToEgeriaGUID - map of full names from UC to the GUID of the entity in Egeria.
      targetPermittedSynchronization - the policy that controls the direction of metadata exchange
      ucConnector - connector for accessing UC
      entityTypeName - type name of asset entity used to represent this type of element
      deployedImplementationType - technology type
      ucServerEndpoint - the server endpoint used to constructing the qualified names
      templates - templates supplied through the catalog target
      configurationProperties - configuration properties supplied through the catalog target
      excludeNames - list of catalogs to ignore (and include all others)
      includeNames - list of catalogs to include (and ignore all others) - overrides excludeCatalogs
      auditLog - logging destination
      Throws:
      UserNotAuthorizedException - connector disconnected
      InvalidParameterException - missing template

  • Method Details

    • refreshEgeria

      protected abstract IntegrationIterator refreshEgeria(String parentGUID, String parentRelationshipTypeName, RelationshipBeanProperties relationshipProperties) throws InvalidParameterException, PropertyServerException, UserNotAuthorizedException, ConnectorCheckedException
      Review all the elements for the subtype stored in Egeria.
      Parameters:
      parentGUID - unique identifier of the parent
      parentRelationshipTypeName - relationship type between parent and elements to iterate through
      relationshipProperties - optional properties for the relationship
      Returns:
      iterator
      Throws:
      InvalidParameterException - parameter error
      PropertyServerException - repository error
      UserNotAuthorizedException - security error
      ConnectorCheckedException - logic error in properties

    • refreshUnityCatalog

      protected abstract void refreshUnityCatalog(String parentGUID, String parentRelationshipTypeName, RelationshipBeanProperties relationshipProperties, IntegrationIterator iterator) throws InvalidParameterException, PropertyServerException, UserNotAuthorizedException, ConnectorCheckedException
      Review all the elements for the subtype stored in UC.
      Parameters:
      parentGUID - unique identifier of the parent
      parentRelationshipTypeName - relationship type between parent and elements to iterate through
      relationshipProperties - optional properties for the relationship
      iterator - iterator of choice
      Throws:
      InvalidParameterException - parameter error
      PropertyServerException - repository error
      UserNotAuthorizedException - security error
      ConnectorCheckedException - logic error in properties

    • getDateFromLong

      protected Date getDateFromLong(long valueFromUC)
      Convert the long value returned by UC into a date (or null).
      Parameters:
      valueFromUC - long
      Returns:
      date

    • getQualifiedName

      protected String getQualifiedName(String fullName)
      Return the qualified name of an element using UnityCatalog's full name.
      Parameters:
      fullName - dotted name
      Returns:
      qualified name

    • deleteElementInEgeria

      protected void deleteElementInEgeria(MemberElement memberElement) throws InvalidParameterException, PropertyServerException, UserNotAuthorizedException
      Delete an element from open metadata.
      Parameters:
      memberElement - element to delete
      Throws:
      InvalidParameterException - parameter error
      PropertyServerException - repository error
      UserNotAuthorizedException - authorization error

    • addExternalIdentifier

      public void addExternalIdentifier(String openMetadataElementGUID, String parentGUID, ElementBase ucElement, String schemaName, String elementName, String elementType, String id, PermittedSynchronization instanceSynchronizationDirection) throws InvalidParameterException, UserNotAuthorizedException, PropertyServerException
      Add a new external identifier to an existing open metadata element.
      Parameters:
      openMetadataElementGUID - unique identifier (GUID) of the element in the open metadata ecosystem
      parentGUID - unique identifier of the immediate parent of openMetadataElementGUID (eg the catalog for a schema, the schema for a table/volume/function/model, or the server for the catalog itself)
      ucElement - values from Unity Catalog element
      schemaName - name of the schema
      elementName - element type (from UC)
      instanceSynchronizationDirection - direction the metadata is flowing
      Throws:
      InvalidParameterException - one of the parameters is invalid
      UserNotAuthorizedException - the user is not authorized to issue this request
      PropertyServerException - problem accessing the property server

    • buildAnchorScopeGUIDs

      protected List<String> buildAnchorScopeGUIDs(String... ancestorGUIDs)
      Build the anchorScopeGUIDs list used throughout this connector for elements catalogued from Unity Catalog: the Unity Catalog server asset, the catalog the element belongs to, any additional ancestors between the catalog and the element (for example the schema, and the table/function for a column or parameter), and the technology-type GUID for the OSS Unity Catalog server product. Nulls and duplicates are removed since not every element has all of these ancestors (for example, the catalog itself has no separate catalogGUID or ancestor GUIDs distinct from the server).
      Parameters:
      ancestorGUIDs - unique identifiers of the ancestors of the element being scoped, nearest first - eg (schemaGUID) for a table/volume/function/model, or (schemaGUID, assetGUID) for a column/parameter nested under a table/function. May be omitted or contain nulls.
      Returns:
      de-duplicated, non-null list of anchor scope GUIDs

    • findExistingSchemaType

      protected String findExistingSchemaType(SchemaTypeClient schemaTypeClient, String qualifiedName) throws InvalidParameterException, UserNotAuthorizedException, PropertyServerException
      Find an existing schema type with the requested qualified name. Used to make schema cataloguing (for example table columns, or API parameters) idempotent across repeated refreshes.
      Parameters:
      schemaTypeClient - client to search with
      qualifiedName - unique name to search for
      Returns:
      GUID of the matching schema type, or null if not found
      Throws:
      InvalidParameterException - one of the parameters is not correct
      UserNotAuthorizedException - the connector's userId is not able to work with open metadata
      PropertyServerException - there is an issue with one of the open metadata repositories

    • findExistingSchemaAttribute

      protected String findExistingSchemaAttribute(SchemaAttributeClient schemaAttributeClient, String qualifiedName) throws InvalidParameterException, UserNotAuthorizedException, PropertyServerException
      Find an existing schema attribute with the requested qualified name. Used to make schema cataloguing (for example table columns, or API parameters) idempotent across repeated refreshes.
      Parameters:
      schemaAttributeClient - client to search with
      qualifiedName - unique name to search for
      Returns:
      GUID of the matching schema attribute, or null if not found
      Throws:
      InvalidParameterException - one of the parameters is not correct
      UserNotAuthorizedException - the connector's userId is not able to work with open metadata
      PropertyServerException - there is an issue with one of the open metadata repositories

    • getExternalIdProperties

      protected ExternalIdProperties getExternalIdProperties(ElementBase ucElement, String elementType, String id)
      Populate and return the external identifier properties for a UC Schema.
      Parameters:
      ucElement - values from Unity Catalog element
      elementType - element type (from UC)
      id - element id (from UC)
      Returns:
      external identifier properties

    • getExternalIdLinkProperties

      protected ExternalIdLinkProperties getExternalIdLinkProperties(ElementBase ucElement, String schemaName, String elementName, PermittedSynchronization instanceSynchronizationDirection)
      Populate and return the external identifier properties for a UC Schema.
      Parameters:
      ucElement - values from Unity Catalog element
      schemaName - name of the schema
      elementName - element type (from UC)
      instanceSynchronizationDirection - direction the metadata is flowing
      Returns:
      external identifier properties

    • addPropertyFacet

      protected void addPropertyFacet(String parentGUID, String parentQualifiedName, BasicElementProperties basicElementProperties, Map<String,String> facetProperties) throws InvalidParameterException, PropertyServerException, UserNotAuthorizedException
      Create the property facet for an Egeria element. This holds the vendor specific properties for the element.
      Parameters:
      parentGUID - the parent (and anchor) unique name
      parentQualifiedName - qualifiedName of the parent element
      basicElementProperties - common properties of an element
      facetProperties - these are the specialist property for the linked element.
      Throws:
      InvalidParameterException
      PropertyServerException
      UserNotAuthorizedException

    • noMismatchInExternalIdentifier

      protected boolean noMismatchInExternalIdentifier(String thirdPartyExternalIdentifier, MemberElement memberElement)
      Check that the name of the third party element has not changed with respect to Egeria.
      Parameters:
      thirdPartyExternalIdentifier - id from Unity Catalog
      memberElement - element from Egeria
      Returns:
      boolean

    • throwMissingPropertyValue

      protected void throwMissingPropertyValue(String missingPropertyName, String methodName, OpenMetadataRootElement element) throws ConnectorCheckedException
      Throw an exception because a necessary property is missing. The description portrays this a s logic error.
      Parameters:
      missingPropertyName - missing property name
      methodName - calling method
      element - element in error
      Throws:
      ConnectorCheckedException - resulting exception

    • throwBadBeanClass

      protected void throwBadBeanClass(OpenMetadataType expectedType, String methodName, OpenMetadataRootElement element) throws ConnectorCheckedException
      Indicate that an element does not have properties of the expected bean class. This is documented as a logic error.
      Parameters:
      expectedType - open metadata type information
      methodName - calling method
      element - element in error
      Throws:
      ConnectorCheckedException - resulting exception

    • getUCCatalogFromMember

      protected String getUCCatalogFromMember(MemberElement memberElement) throws ConnectorCheckedException
      Extract the comment for a UC element from the description attribute in an egeria element.
      Parameters:
      memberElement - element from Egeria
      Returns:
      catalog for UC
      Throws:
      ConnectorCheckedException - logic error in properties

    • getParentGUIDFromMember

      protected String getParentGUIDFromMember(MemberElement memberElement)
      Extract the GUID of the immediate parent of an element from its Anchors classification. This is used when pushing an element created directly in open metadata out to Unity Catalog, where the only way to discover its parent (eg the schema for a table/volume/function/model) is via its anchor.
      Parameters:
      memberElement - element from Egeria
      Returns:
      parent GUID, or null if the element is not anchored

    • getUCSchemaFromMember

      protected String getUCSchemaFromMember(MemberElement memberElement) throws ConnectorCheckedException
      Extract the schema name for a UC element from the description attribute in an egeria element.
      Parameters:
      memberElement - element from Egeria
      Returns:
      schema for UC
      Throws:
      ConnectorCheckedException - logic error in properties

    • getUCNameFromMember

      protected String getUCNameFromMember(MemberElement memberElement) throws ConnectorCheckedException
      Extract the name of a UC element from the name attribute in an egeria element (which is the full name).
      Parameters:
      memberElement - element from Egeria
      Returns:
      name for UC
      Throws:
      ConnectorCheckedException - logic error in properties

    • getUCCommentFomMember

      protected String getUCCommentFomMember(MemberElement memberElement) throws ConnectorCheckedException
      Extract the comment for a UC element from the description attribute in an egeria element.
      Parameters:
      memberElement - element from Egeria
      Returns:
      comment for UC
      Throws:
      ConnectorCheckedException - logic error in properties

    • getUCPropertiesFromMember

      protected Map<String,String> getUCPropertiesFromMember(MemberElement memberElement) throws ConnectorCheckedException
      Extract the properties for a UC element from the additionalProperties attribute in an egeria element.
      Parameters:
      memberElement - element from Egeria
      Returns:
      properties for UC
      Throws:
      ConnectorCheckedException

    • getUCStorageLocationFromMember

      protected String getUCStorageLocationFromMember(MemberElement memberElement)
      Extract the storage location from the
      Parameters:
      memberElement - elements from egeria
      Returns:
      storage location (may be null

    • getCleanStorageLocation

      protected String getCleanStorageLocation(String storageLocation)
      Convert the US format for the storage location into a proper pathname.
      Parameters:
      storageLocation - storage location from UC
      Returns:
      pathname

    • getArrayConfigurationProperty

      protected List<String> getArrayConfigurationProperty(String propertyName, Map<String,Object> configurationProperties)
      Retrieve a configuration property that is a comma-separated list of strings.
      Parameters:
      propertyName - name of property
      configurationProperties - configuration properties
      Returns:
      list of strings or null if not set