Skip to content

Developer Guide

Egeria is designed to simplify the effort necessary to integrate different technologies so that they can actively share and consume metadata from each other.

It focuses on providing five types of integration interfaces.

  • Connectors that translate between third party APIs and open metadata APIs. These connectors are hosted in the Egeria servers and support the active exchange of metadata with these technologies.
  • Connectors for accessing popular type of data sources that also retrieve open metadata about the data source. This allows applications and tools to understand the structure, meaning, profile, quality and lineage of the data they are using.
  • Java clients for applications to call the Open Metadata Access Service (OMAS) interfaces, each of which are crafted for particular types of technology. These interfaces support both synchronous APIs, inbound event notifications and outbound asynchronous events.
  • REST APIs for the Egeria Services. These include the access services, admin services and platform services.
  • Kafka topics with JSON payloads for asynchronous communication (both in and out) with the open metadata ecosystem. Learn more ...

Using the clients

The Egeria clients wrap calls to Egeria's REST APIs and topics. The aim is to provide a language-specific interface that manages the marshalling and de-marshalling of the call parameters and responses to these services.

Using the REST APIs

Egeria supports REST APIs for making synchronous (request-response) calls between OMAG Servers and between clients and OMAG Servers.

REST APIs are intended for internal use

The REST APIs are usable directly for calling from non-Java platforms; however, they are designed for the internal use of Egeria and are not guaranteed to be backwards compatible.

The structure of the URL for an Egeria REST API varies lightly depending on whether it is a call to an OMAG Server Platform service or an OMAG Server service.

What is a connector?

Connectors are plug-in Java clients that either perform an additional service, or, more typically, enable Egeria to integrate with a third party technology.

The concept of a connector comes from the Open Connector Framework (OCF). The OCF provides a common framework for components that enable one technology to call another, arbitrary technology through a common interface. The implementation of the connector is dynamically loaded based on the connector's configuration.

Through the OCF, we can:

  1. Plug-in different technologies to Egeria.
  2. Plug-in support for open metadata into the client libraries used by applications to access data resources and services.

How OCF connectors are used in Egeria

Many subsystems in Egeria's OMAG Server Platform and servers support the first approach. They define a specialized interface for the type of connector they support.

  1. One or more connector implementations supporting that interface are then written either by the Egeria community or other organizations.
  2. When an Egeria OMAG Server is configured, details of which connector implementation to use is specified in the server's configuration document.
  3. At start up, the OMAG Server passes the connector configuration to the OCF to instantiate the required connector instance.

Connectors enable Egeria to operate in many environments and with many types of third party technologies, just by managing the configuration of the OMAG Servers.

The second approach is used by organizations that want to make use of metadata directly in applications and tools - or to externalize the security and driver properties needed to call the data source or service. In this case the OCF connector typically has the same interface as the data source's client library (unless you can do better 😄). This minimizes the learning curve for application developers. The configuration for the connector is stored in an open metadata server and the application uses the Asset Consumer OMAS client to request a new instance of the connector. The application uses the returned connector instance to access the data source or server along with the metadata stored about it.

Configuration

The configuration for a connector is managed in a connection object.

A connection contains properties about the specific use of the connector, such as user Id and password, or parameters that control the scope or resources that should be made available to the connector. It links to an optional endpoint and a mandatory connector type object.

  • ConnectorType describes the type of the connector, its supported configuration properties and its factory object (called the connector's provider). This information is used to create an instance of the connector at runtime.
  • Endpoint describes the server endpoint where the third party data source or service is accessed from.

Connector types and endpoints can be reused in multiple connections.

Structure of a connection object

Factories

Each connector implementation has a factory object called a connector provider. The connector provider has two types of methods:

  • Return a new instance of the connector based on the properties in a supplied Connection object. The Connection object has all the properties needed to create and configure the instance of the connector.
  • Return additional information about the connector's behavior and usage to make it easier to consume. For example, the standard base class for a connector provider has a method to return the ConnectorType object for this connector implementation that can be added to a Connection object used to hold the properties needed to create an instance of the connector.

Inside the connector

Each connector has its own unique implementation that is structured around a simple lifecycle that is defined by the OCF. The OCF provides the interface for a connector called Connector that has three methods: initialize, start and disconnect.

This connector interface supports the basic lifecycle of a connector. There are three phases:

  1. Initialization - During this phase, the connector is passed the context in which it is to operate. It should store this information.

    This phase is initiated by a call to the connector's initialize() method, which is called after the connector's constructor and provides the connector with a unique instance identifier (for logging) and its configuration stored in a connection. After initialize() returns, there may be other calls to pass context to the connector. For example, if the connector implements the AuditLoggingComponent , an audit log is passed to the connector.

  2. Running - The connector is completely initialized with its context, and it can start processing.

    This phase is initiated by a call to the connector's start() method. At this point it should create its client to any third party technology and begin processing. It may also start up threads if it needs to perform any background processing (such as listening for notifications). If the connector throws an exception during start, Egeria knows the connector has a configuration or operational issue and will report the error and move it to disconnected state.

  3. Disconnected - The connector must stop processing and release all of its resources.

    This phase is initiated by a call to the connector's disconnect() method.

Depending on the type of connector you are writing, there may be additional initialization calls occurring between the initialize() and the start() method. The connector may also support additional methods for its normal operation that can be called between the start() and disconnect() calls.

The OCF also provides the base class for a connector called ConnectorBase . The ConnectorBase base class manages the lifecycle state of the connector. For example, the default implementation of initialize() in the ConnectorBase class stores the supplied unique instance identifier and connection values in protected variables called connectorInstanceId and connectionProperties respectively.

Call the base class's methods in any overrides

If you override any of the initialize(), start() or disconnect() methods, be sure to call super.xxx() at the start of your implementation to call the appropriate super class method so that the state is properly maintained.

Extending Egeria using connectors

Egeria has extended the basic concept of the OCF connector and created specialized connectors for different purposes. The following types of connectors are supported by the Egeria subsystems with links to the documentation and implementation examples.

Type of Connector Description Documentation Implementation Examples
Integration Connector Implements metadata exchange with third party tools. Building Integration Connectors integration-connectors
Open Discovery Service Implements automated metadata discovery. Open Discovery Services discovery-service-connectors
Governance Action Service Implements automated governance. Governance Action Services governance-action-connectors
Configuration Document Store Persists the configuration document for an OMAG Server. Configuration Document Store Connectors configuration-store-connectors
Platform Security Connector Manages service authorization for the OMAG Server Platform. Metadata Security Connectors open-metadata-security-samples
Server Security Connector Manages service and metadata instance authorization for an OMAG Server. Metadata Security Connectors open-metadata-security-samples
Metadata Collection (repository) Store Interfaces with a metadata repository API for retrieving and storing metadata. OMRS Repository Connectors open-metadata-collection-store-connectors
Metadata Collection (repository) Event Mapper Maps events from a third party metadata repository to open metadata events. OMRS Event Mappers none
Open Metadata Archive Store Reads an open metadata archive from a particular type of store. OMRS Open Metadata Archive Store Connector open-metadata-archive-connectors
Audit Log Store Audit logging destination OMRS Audit Log Store Connector audit-log-connectors
Cohort Registry Store Local store of membership of an open metadata repository cohort. OMRS Cohort Registry Store cohort-registry-store-connectors
Open Metadata Topic Connector Connects to a topic on an external event bus such as Apache Kafka. Open Metadata Topic Connectors open-metadata- topic-connectors

You can write your own connectors to integrate additional types of technology or extend the capabilities of Egeria - and if you think your connector is more generally useful, you could consider contributing it to the Egeria project.

Back to top