Skip to content

Connector catalog

Egeria has a growing collection of connectors to third party technologies. These connectors help to accelerate the rollout of your open metadata ecosystem since they can be used to automate the extraction and distribution of metadata to the third party technologies.

A connector is a client to a third party technology. It supports a standard API that Egeria calls and it then translates these calls into requests to the third party technology. Some connectors are also able to listen for notifications from the third party technology. When a notification is received, the connector converts its content into a call to Egeria to distribute the information to the open metadata ecosystem.

Connectors enable Egeria to operate in many environments and with many types of third party technologies, just by managing the configuration of the OMAG servers. The Connector Catalog list the connector implementations supplied by the Egeria community. There are three broad categories of connectors and the connector catalog is organized accordingly:

  • Connectors that support the exchange and maintenance of metadata. This includes the integration connectors, repository connectors, discovery services and governance action services.

  • Connectors that support Egeria‚Äôs runtime. This includes the event bus connectors, cohort registry stores, configuration stores, audit log destination connectors, open metadata archive stores, REST client connectors and the cohort member remote repository connectors

  • Connectors that provide access to digital resources and their metadata that is stored in the open metadata ecosystem.

Metadata exchange and maintenance connectors

The connectors that support the exchange and maintenance of metadata help to accelerate the rollout of your open metadata ecosystem since they can be used to automate the extraction and distribution of metadata to the third party technologies.

Type Description
Integration connectors manage the metadata exchange to a third party technology through an integration service.
Repository and Event Mapper connectors integrate metadata repositories through one or more open metadata repository cohorts.
Open Discovery Services analyze the content of resources in the digital landscape and create annotations that are attached to the resource's asset metadata element in the open metadata repositories in the form of an open discovery report
Governance Action Services perform monitoring of metadata changes, validation of metadata, triage of issues, assessment and/or remediation activities as required.

Integration Connectors

The integration connectors support the exchange of metadata with third party technologies. This exchange may be inbound, outbound, synchronous, polling or event-driven.

Figure 1

Figure 1: Integration Connectors

Details of how integration connectors work is described in the developer guide.

Files

The files integration connectors run in the Files Integrator Open Metadata Integration Service (OMIS) hosted in the integration daemon.

Connector Description
Data files monitor maintains a DataFile asset for each file in the directory (or any subdirectory). When a new file is created, a new DataFile asset is created. If a file is modified, the lastModified property of the corresponding DataFile asset is updated. When a file is deleted, its corresponding DataFile asset is also deleted (or archived if it is still needed for lineage).
Data folder monitor maintains a DataFolder asset for the directory. The files and directories underneath it are assumed to be elements/records in the DataFolder asset and so each time there is a change to the files and directories under the monitored directory, it results in an update to the lastModified property of the corresponding DataFolder asset.

Databases

The database integration connectors run in the Database Integrator Open Metadata Integration Service (OMIS) hosted in the integration daemon.

Connector Description
PostgreSQL database connector automatically maintains the open metadata instances for the databases hosted on a PostgreSQL server This includes the database schemas, tables, columns, primary keys and foreign keys.

Topics

The topic integration connectors run in the Topic Integrator Open Metadata Integration Service (OMIS) hosted in the integration daemon.

Connector Description
Kafka Monitor topic integration connector automatically maintains the open metadata instances for the topics hosted on an Apache Kafka server .

APIs

The API integration connectors run in the API Integrator Open Metadata Integration Service (OMIS) hosted in the integration daemon.

Connector Description
Open API Monitor integration connector automatically maintains the open metadata instances for the APIs extracted from the Open API Specification extracted from an application.

Security Enforcement Engines

The security integration connectors run in the Security Integrator Open Metadata Integration Service (OMIS) hosted in the integration daemon.

Lineage

The lineage integration connectors run in the Lineage Integrator OMIS hosted in the integration daemon. They support Lineage Management.

Connector Description
Open Lineage Event Receiver integration connector Connector to receive open lineage events from an evvent topic and publish them to lineage integration connectors with listeners registered in the same instance of the Lineage Integrator OMIS.
Governance Action to Open Lineage integration connector Connector to listen for governance actions executing in the open metadata ecosystem, generate open lineage events for them and publish them to the integration connectors running in the same instance of Lineage Integrator OMIS that are listening for OpenLineage events.
API-based Open Lineage Log Store integration connector Connector that calls an OpenLineage compliant API to store the open lineage events that are passed to it through the OpenLineage listener that is registered with the Lineage Integrator OMIS.
File-based Open Lineage Log Store integration connector Connector that stores the open lineage events that are passed to it through the OpenLineage listener that is registered with the Lineage Integrator OMIS. Each OpenLineage event is stored in its own file in JSON format. These files are organized according to the namespace and job name in the event.
Open Lineage Cataloguer integration connector Connector to register an OpenLineage listener with the Lineage Integrator OMIS and to catalog any processes that are not already known to the open metadata ecosystem.

Repository and Event Mapper Connectors

The repository connectors provide the ability to integrate a third party metadata repository into an open metadata repository cohort.

Figure 2 shows the repository connector providing a native open metadata repository that uses a particular type of store within an Egeria Metadata Access Server.

Figure 2

Figure 2: Repository connector supporting a native open metadata repository

Connector Description
JanusGraph OMRS Repository Connector provides a native repository for a metadata server using JanusGraph as the backend.
XTDB OMRS Repository Connector provides a native repository for a metadata server that supports historical queries, using XTDB as the backend.
In-memory OMRS Repository Connector provides a simple native repository implementation that "stores" metadata in HashMaps within the JVM; it is used for testing, or for environments where metadata maintained in other repositories needs to be cached locally for performance/scalability reasons.
Read-only OMRS Repository Connector provides a native repository implementation that does not support the interfaces for create, update, delete; however, it does support the search interfaces and is able to cache metadata -- this means it can be loaded with open metadata archives to provide standard metadata definitions.

Figure 3 shows the repository connector providing a native open metadata repository that uses a particular type of store within an Egeria Metadata Access Server.

Figure 3

Figure 3: Repository connector and optional event mapper supporting an adapter to a third party metadata catalog

Connector Description
Apache Atlas OMRS Repository Connector implements read-only connectivity to the Apache Atlas metadata repository
IBM Information Governance Catalog (IGC) OMRS Repository Connector implements read-only connectivity to the metadata repository within the IBM InfoSphere Information Server suite
SAS Viya OMRS Repository Connector implements metadata exchange to the metadata repository within the SAS Viya Platform

The repository connectors implement the OMRSMetadataCollection interface to allow metadata to be communicated and exchanged according to Egeria's protocols and type definitions.

Open Discovery Services

Open discovery services are connectors that analyze the content of resources in the digital landscape and create annotations that are attached to the resource's Asset metadata element in the open metadata repositories in the form of an open discovery report.

Figure 4

Figure 4: Discovery Services

The definition of the connector interfaces for discovery services is defined in the open-discovery-services module.

Connector Description
CSV Discovery Service extracts the column names from the first line of the file, counts up the number of records in the file and extracts its last modified time.
Sequential Discovery Pipeline runs nested discovery services in a sequence (more information on discovery pipelines).

Governance Action Services

Governance action services are connectors that perform monitoring of metadata changes, validation of metadata, triage of issues, assessment and/or remediation activities on request.

Figure 5

Figure 5: Governance Action Services

They run in the Governance Action Open Metadata Engine Service (OMES) hosted by the Engine Host OMAG Server.

Connector Description
Generic Element Watchdog Governance Action Service listens for changing metadata elements and initiates governance action processes when certain events occur.
Generic Folder Watchdog Governance Action Service listens for changing assets linked to a DataFolder element and initiates governance actions when specific events occur. This may be for files directly linked to the folder or located in sub-folders.
Move/Copy File Provisioning Governance Action Service moves or copies files from one location to another and maintains the lineage of the action.
Origin Seeker Remediation Governance Action Service walks backwards through the lineage mappings to discover the origin of the data

The definition of the connector interfaces for governance action services is defined in the governance-action-framework module.

Runtime connectors

Runtime connectors support Egeria's runtime: Connectors enable Egeria to operate in many environments by providing plug-in points for the runtime services it needs to operate. Most of these connectors relate to persistent storage, or connections to distributed services.

Configuration Document Store Connectors

The configuration store connectors contain the connector implementations that manage the Configuration Documents for OMAG Servers.

Figure 7

Figure 7: Configuration Document Store Connector

There is one configuration document store connector defined for each OMAG Server Platform.

There are two implementations of the configuration document store connector provided by Egeria: one for an encrypted store (default) and the other for a plain text store.

The definition of the connector interface for these connectors is defined in the admin-services-api module in the org.odpi.openmetadata.adminservices.store Java package.

Configuring a configuration store connector in the OMAG Server Platform is described in the Administration Guide. If no connector is configured, the OMAG Server Platform uses the Encrypted File Configuration Store Connector.

Cohort Registry Store Connectors

The cohort registry store connectors store the open metadata repository cohort membership details used and maintained by the cohort registry. The cohort protocols are peer-to-peer and hence there is a cohort registry (with a cohort registry store for each member of a cohort.

Figure 8

Figure 8: Open Metadata Topic Connectors

Egeria provides a single implementation of a cohort registry store connector:

The definition of the connector interface for these connectors is defined in the repository-services-api module in the org.odpi.openmetadata.repositoryservices.connectors.stores.cohortregistrystore Java package.

Open Metadata Archive Store Connectors

The open metadata archive store connectors can read and write open metadata archives. Open metadata archives store open metadata types and instances for sharing, or for back up. These archives can be loaded into an OMAG Server at start up or added to a running OMAG Server.

Figure 9

Figure 9: Open Metadata Archive Store Connector

Egeria provides a single implementation of the open metadata archive store connector:

The definition of the connector interface for these connectors is defined in the repository-services-api module in the org.odpi.openmetadata.repositoryservices.connectors.stores.archivestore Java package.

Audit Log Destination Connectors

The audit log destination connectors support different destinations for audit log records. Multiple of these connectors can be active in an OMAG Server at any one time and they can each be configured to only process particular types of audit log records.

Figure 10

Figure 10: Audit Log Destination Connector

Below are the connector implementations provided by Egeria

The definition of the connector interface for these connectors is defined in the repository-services-api module in the org.odpi.openmetadata.repositoryservices.connectors.stores.auditlogstore Java package.

There is more information on the use of audit logging in the Diagnostic Guide.

REST Client Connectors

Egeria makes extensive use of REST API calls for synchronous (request-response) communication with its own deployed runtimes and third party technologies. The client connectors are used to issue the REST API calls. Egeria provides a single implementation for Spring.

This is embedded in Egeria's clients.

Figure 11

Figure 11: REST Client Connector

The definition of the connector interface for these connectors is defined in the rest-client-connector-api module in the org.odpi.openmetadata.adapters.connectors.restclients Java package.

Cohort Member Client Connector

Members of an Open Metadata Repository Cohort provide the other cohort members with a Connection to a connector that supports the OMRSRepositoryConnector interface during the cohort registration process. This connector translates calls to retrieve and maintain metadata in the member's repository into remote calls to the real repository.

Figure 12

Figure 12: Cohort Member Client Connector used for federating queries across the cohort

Egeria's Open Metadata Repository Services (OMRS) provides a default REST API implementation and a corresponding client:

The connection for this connector is configured in the LocalRepositoryRemoteConnection property of the cohort member's Local Repository Configuration.

The definition of the connector interface for these connectors is defined in the repository-services-api module in the org.odpi.openmetadata.repositoryservices.connectors.stores.metadatacollectionstore.repositoryconnector Java package. It is the same interface as the repository connectors that are used to provide the local repository to a metadata server so that the Open Metadata Repository Services (OMRS) issue calls to the same interface irrespective of whether the call is to the local repository or a remote cohort member.

Digital resource connectors

Digital resource connectors provide access to digital resources and their metadata that is stored in the open metadata ecosystem. These connectors are for use by external applications and tools to connect with resources and services in the digital landscape. These connectors also supply the Asset metadata from Egeria that describes these resources.

Instances of these connectors are created through the Asset Consumer OMAS or Asset Owner OMAS interfaces. They use the Connection linked to the corresponding Asset in the open metadata ecosystem. Connection objects are associated with assets in the metadata catalog using the Asset Owner OMAS, Data Manager OMAS and Asset Manager OMAS.

Files

  • The Avro file connector provides access to an Avro file that has been catalogued using open metadata.

  • The basic file connector provides support to read and write to a file using the Java File object.

  • The CSV file connector is able to retrieve data from a Comma Separated Values (CSV) file where the contents are stored in logical columns with a special character delimiter between the columns.

  • The data folder connector is for accessing data that is stored as a number of files within a folder (directory).

Databases

More coming ...

Open Metadata Topic Connectors

The Open Metadata Topic Connectors are used by Egeria to read and write events to a topic managed by an event broker. These events contain notifications relating to changes in metadata and the topic provides an asynchronous event exchange service hosted in the event broker.

Figure 6

Figure 6: Open Metadata Topic Connectors

The Open Metadata Topic Connectors connect servers into an open metadata repository cohort and exchange notifications through the Open Metadata Access Services (OMAS)'s topics called the InTopic and OutTopic.

Egeria provides a single implementation of an open metadata connector for Apache Kafka that it uses by default.

The definition of the connector interface for these connectors is defined in the repository-services-apis module in the org.odpi.openmetadata.repositoryservices.connectors.openmetadatatopic Java package.

It is configured in the Egeria OMAG Servers through the Event Bus Configuration.

Back to top