Egeria has a growing collection of connectors to third party technologies. These connectors help to accelerate the rollout of your open metadata ecosystem since they can be used to automate the extraction and distribution of metadata to the third party technologies.
A connector is a client to a third party technology. It supports a standard API that Egeria calls and it then translates these calls into requests to the third party technology. Some connectors are also able to listen for notifications from the third party technology. When a notification is received, the connector converts its content into a call to Egeria to distribute the information to the open metadata ecosystem.
Connectors enable Egeria to operate in many environments and with many types of third party technologies, just by managing the configuration of the OMAG servers. The Connector Catalog list the connector implementations supplied by the Egeria community. There are three broad categories of connectors and the connector catalog is organized accordingly:
Connectors that support the exchange and maintenance of metadata. This includes the integration connectors, repository connectors, discovery services and governance action services.
Connectors that support Egeria’s runtime. This includes the event bus connectors, cohort registry stores, configuration stores, audit log destination connectors, open metadata archive stores, REST client connectors and the cohort member remote repository connectors
Connectors that provide access to digital resources and their metadata that is stored in the open metadata ecosystem.
Metadata exchange and maintenance connectors¶
The connectors that support the exchange and maintenance of metadata help to accelerate the rollout of your open metadata ecosystem since they can be used to automate the extraction and distribution of metadata to the third party technologies.
|Integration connectors||manage the metadata exchange to a third party technology through an integration service.|
|Repository and Event Mapper connectors||integrate metadata repositories through one or more open metadata repository cohorts.|
|Open Discovery Services||analyze the content of resources in the digital landscape and create annotations that are attached to the resource's asset metadata element in the open metadata repositories in the form of an open discovery report|
|Governance Action Services||perform monitoring of metadata changes, validation of metadata, triage of issues, assessment and/or remediation activities as required.|
The integration connectors support the exchange of metadata with third party technologies. This exchange may be inbound, outbound, synchronous, polling or event-driven.
Figure 1: Integration Connectors
Details of how integration connectors work is described in the developer guide.
files integration connectors run in the Files Integrator Open Metadata Integration Service (OMIS) hosted in the integration daemon.
|Data files monitor||maintains a
|Data folder monitor||maintains a
The database integration connectors run in the Database Integrator Open Metadata Integration Service (OMIS) hosted in the integration daemon.
|PostgreSQL database connector||automatically maintains the open metadata instances for the databases hosted on a PostgreSQL server This includes the database schemas, tables, columns, primary keys and foreign keys.|
The topic integration connectors run in the Topic Integrator Open Metadata Integration Service (OMIS) hosted in the integration daemon.
|Kafka Monitor topic integration connector||automatically maintains the open metadata instances for the topics hosted on an Apache Kafka server .|
The API integration connectors run in the API Integrator Open Metadata Integration Service (OMIS) hosted in the integration daemon.
|Open API Monitor integration connector||automatically maintains the open metadata instances for the APIs extracted from the Open API Specification extracted from an application.|
Security Enforcement Engines¶
The security integration connectors run in the Security Integrator Open Metadata Integration Service (OMIS) hosted in the integration daemon.
|Open Lineage Event Receiver integration connector||Connector to receive open lineage events from an evvent topic and publish them to lineage integration connectors with listeners registered in the same instance of the Lineage Integrator OMIS.|
|Governance Action to Open Lineage integration connector||Connector to listen for governance actions executing in the open metadata ecosystem, generate open lineage events for them and publish them to the integration connectors running in the same instance of Lineage Integrator OMIS that are listening for OpenLineage events.|
|API-based Open Lineage Log Store integration connector||Connector that calls an OpenLineage compliant API to store the open lineage events that are passed to it through the OpenLineage listener that is registered with the Lineage Integrator OMIS.|
|File-based Open Lineage Log Store integration connector||Connector that stores the open lineage events that are passed to it through the OpenLineage listener that is registered with the Lineage Integrator OMIS. Each OpenLineage event is stored in its own file in JSON format. These files are organized according to the namespace and job name in the event.|
|Open Lineage Cataloguer integration connector||Connector to register an OpenLineage listener with the Lineage Integrator OMIS and to catalog any processes that are not already known to the open metadata ecosystem.|
Repository and Event Mapper Connectors¶
The repository connectors provide the ability to integrate a third party metadata repository into an open metadata repository cohort.
Figure 2 shows the repository connector providing a native open metadata repository that uses a particular type of store within an Egeria Metadata Access Server.
Figure 2: Repository connector supporting a native open metadata repository
|JanusGraph OMRS Repository Connector||provides a native repository for a metadata server using JanusGraph as the backend.|
|XTDB OMRS Repository Connector||provides a native repository for a metadata server that supports historical queries, using XTDB as the backend.|
|In-memory OMRS Repository Connector||provides a simple native repository implementation that "stores" metadata in HashMaps within the JVM; it is used for testing, or for environments where metadata maintained in other repositories needs to be cached locally for performance/scalability reasons.|
|Read-only OMRS Repository Connector||provides a native repository implementation that does not support the interfaces for create, update, delete; however, it does support the search interfaces and is able to cache metadata -- this means it can be loaded with open metadata archives to provide standard metadata definitions.|
Figure 3 shows the repository connector providing a native open metadata repository that uses a particular type of store within an Egeria Metadata Access Server.
Figure 3: Repository connector and optional event mapper supporting an adapter to a third party metadata catalog
|Apache Atlas OMRS Repository Connector||implements read-only connectivity to the Apache Atlas metadata repository|
|IBM Information Governance Catalog (IGC) OMRS Repository Connector||implements read-only connectivity to the metadata repository within the IBM InfoSphere Information Server suite|
|SAS Viya OMRS Repository Connector||implements metadata exchange to the metadata repository within the SAS Viya Platform|
Open Discovery Services¶
Open discovery services are connectors that analyze the content of resources in the digital landscape and create annotations that are attached to the resource's Asset metadata element in the open metadata repositories in the form of an open discovery report.
Figure 4: Discovery Services
The definition of the connector interfaces for discovery services is defined in the open-discovery-services module.
|CSV Discovery Service||extracts the column names from the first line of the file, counts up the number of records in the file and extracts its last modified time.|
|Sequential Discovery Pipeline||runs nested discovery services in a sequence (more information on discovery pipelines).|
Governance Action Services¶
Governance action services are connectors that perform monitoring of metadata changes, validation of metadata, triage of issues, assessment and/or remediation activities on request.
Figure 5: Governance Action Services
They run in the Governance Action Open Metadata Engine Service (OMES) hosted by the Engine Host OMAG Server.
|Generic Element Watchdog Governance Action Service||listens for changing metadata elements and initiates governance action processes when certain events occur.|
|Generic Folder Watchdog Governance Action Service||listens for changing assets linked to a
|Move/Copy File Provisioning Governance Action Service||moves or copies files from one location to another and maintains the lineage of the action.|
|Origin Seeker Remediation Governance Action Service||walks backwards through the lineage mappings to discover the origin of the data|
The definition of the connector interfaces for governance action services is defined in the governance-action-framework module.
Runtime connectors support Egeria's runtime: Connectors enable Egeria to operate in many environments by providing plug-in points for the runtime services it needs to operate. Most of these connectors relate to persistent storage, or connections to distributed services.
Configuration Document Store Connectors¶
Figure 7: Configuration Document Store Connector
There is one configuration document store connector defined for each OMAG Server Platform.
There are two implementations of the configuration document store connector provided by Egeria: one for an encrypted store (default) and the other for a plain text store.
Encrypted File Configuration Store Connector stores each configuration document as an encrypted JSON file.
File Configuration Store stores each configuration document as a clear text JSON file.
Configuring a configuration store connector in the OMAG Server Platform is described in the Administration Guide. If no connector is configured, the OMAG Server Platform uses the Encrypted File Configuration Store Connector.
Cohort Registry Store Connectors¶
The cohort registry store connectors store the open metadata repository cohort membership details used and maintained by the cohort registry. The cohort protocols are peer-to-peer and hence there is a cohort registry (with a cohort registry store for each member of a cohort.
Figure 8: Open Metadata Topic Connectors
Egeria provides a single implementation of a cohort registry store connector:
- Cohort Registry File Store Connector provides the means to store the cohort registry membership details as a JSON file.
The definition of the connector interface for these connectors is defined in the repository-services-api module in the org.odpi.openmetadata.repositoryservices.connectors.stores.cohortregistrystore Java package.
Open Metadata Archive Store Connectors¶
The open metadata archive store connectors can read and write open metadata archives. Open metadata archives store open metadata types and instances for sharing, or for back up. These archives can be loaded into an OMAG Server at start up or added to a running OMAG Server.
Figure 9: Open Metadata Archive Store Connector
Egeria provides a single implementation of the open metadata archive store connector:
- Open Metadata Archive File Store Connector stores an open metadata archive as a plain text JSON file.
The definition of the connector interface for these connectors is defined in the repository-services-api module in the org.odpi.openmetadata.repositoryservices.connectors.stores.archivestore Java package.
Audit Log Destination Connectors¶
The audit log destination connectors support different destinations for audit log records. Multiple of these connectors can be active in an OMAG Server at any one time and they can each be configured to only process particular types of audit log records.
Figure 10: Audit Log Destination Connector
Below are the connector implementations provided by Egeria
Console Audit Log Connector writes selected parts of each audit log record to stdout.
slf4j Audit Log Connector writes full log records to the slf4j ecosystem.
File Audit Log Connector creates log records as JSON files in a shared directory.
Event Topic Audit Log Connector sends each log record as an event on the supplied event topic.
The definition of the connector interface for these connectors is defined in the repository-services-api module in the org.odpi.openmetadata.repositoryservices.connectors.stores.auditlogstore Java package.
There is more information on the use of audit logging in the Diagnostic Guide.
REST Client Connectors¶
Egeria makes extensive use of REST API calls for synchronous (request-response) communication with its own deployed runtimes and third party technologies. The client connectors are used to issue the REST API calls. Egeria provides a single implementation for Spring.
- Spring REST Client Connector uses the Spring RESTClient to issue REST API calls.
This is embedded in Egeria's clients.
Figure 11: REST Client Connector
The definition of the connector interface for these connectors is defined in the rest-client-connector-api module in the org.odpi.openmetadata.adapters.connectors.restclients Java package.
Cohort Member Client Connector¶
Members of an Open Metadata Repository Cohort provide the other cohort members with a Connection to a connector that supports the OMRSRepositoryConnector interface during the cohort registration process. This connector translates calls to retrieve and maintain metadata in the member's repository into remote calls to the real repository.
Figure 12: Cohort Member Client Connector used for federating queries across the cohort
Egeria's Open Metadata Repository Services (OMRS) provides a default REST API implementation and a corresponding client:
- REST Cohort Client Connector supports remote calls to the OMRS REST API.
The connection for this connector is configured in the
LocalRepositoryRemoteConnection property of the
cohort member's Local Repository Configuration.
The definition of the connector interface for these connectors is defined in the repository-services-api module in the org.odpi.openmetadata.repositoryservices.connectors.stores.metadatacollectionstore.repositoryconnector Java package. It is the same interface as the repository connectors that are used to provide the local repository to a metadata server so that the Open Metadata Repository Services (OMRS) issue calls to the same interface irrespective of whether the call is to the local repository or a remote cohort member.
Digital resource connectors¶
Digital resource connectors provide access to digital resources and their metadata that is stored in the open metadata ecosystem. These connectors are for use by external applications and tools to connect with resources and services in the digital landscape. These connectors also supply the Asset metadata from Egeria that describes these resources.
Instances of these connectors are created through the Asset Consumer OMAS or Asset Owner OMAS interfaces. They use the Connection linked to the corresponding Asset in the open metadata ecosystem. Connection objects are associated with assets in the metadata catalog using the Asset Owner OMAS, Data Manager OMAS and Asset Manager OMAS.
The Avro file connector provides access to an Avro file that has been catalogued using open metadata.
The basic file connector provides support to read and write to a file using the Java File object.
The CSV file connector is able to retrieve data from a Comma Separated Values (CSV) file where the contents are stored in logical columns with a special character delimiter between the columns.
The data folder connector is for accessing data that is stored as a number of files within a folder (directory).
More coming ...
Open Metadata Topic Connectors¶
The Open Metadata Topic Connectors are used by Egeria to read and write events to a topic managed by an event broker. These events contain notifications relating to changes in metadata and the topic provides an asynchronous event exchange service hosted in the event broker.
Figure 6: Open Metadata Topic Connectors
The Open Metadata Topic Connectors connect servers into an open metadata repository cohort and exchange notifications through the Open Metadata Access Services (OMAS)'s topics called the InTopic and OutTopic.
Egeria provides a single implementation of an open metadata connector for Apache Kafka that it uses by default.
- The Kafka Open Metadata Topic Connector implements an Apache Kafka connector for a topic that exchanges Java Objects as JSON payloads.
The definition of the connector interface for these connectors is defined in the
repository-services-apis module in the org.odpi.openmetadata.repositoryservices.connectors.openmetadatatopic Java package.