Package org.odpi.openmetadata.adapters.connectors.pepys

Class PepysActivityObserverConnector

java.lang.Object

org.odpi.openmetadata.frameworks.connectors.Connector

org.odpi.openmetadata.frameworks.connectors.ConnectorBase

org.odpi.openmetadata.frameworks.integration.connectors.IntegrationConnectorBase

org.odpi.openmetadata.frameworks.integration.connectors.DynamicIntegrationConnectorBase

org.odpi.openmetadata.adapters.connectors.pepys.PepysActivityObserverConnector

All Implemented Interfaces:

AuditLoggingComponent, SecureConnectorExtension, VirtualConnectorExtension, CatalogTargetFactory, IntegrationConnector, OpenMetadataTopicListener

public class PepysActivityObserverConnector
extends DynamicIntegrationConnectorBase
implements OpenMetadataTopicListener

PepysActivityObserverConnector keeps a diary of the activity around the anchor elements (assets, digital products and glossary terms) in the open metadata ecosystem, in the manner of Samuel Pepys. Its catalog targets are the topics that the OMAG Servers write their audit logs to (registered by the OMAG Server Platform Cataloguer). It listens to each of them for the anchor element activity messages (METADATA-OBSERVABILITY-0002 to -0009 and -0011) and maintains the separate counts in the UsageActivity classification of the anchor element named in each message.

The activity is gathered in memory as the records arrive and written on each refresh, so an element that is busy costs one read and one update per refresh rather than one per record. Counts that could not be written are kept and tried again on the next refresh.

The activity of the users whose userIds match one of the patterns in the ignoredUserIdPatterns configuration property is not counted. The core content pack sets them to match the non-personal accounts (ending in npa) and the governance engines (ending in engine), so only the activity of people and external tools is recorded. The patterns are used rather than a list of userIds because the servers' userIds differ in each environment. The connector's own userId is always ignored, whether a pattern matches it or not - its reads and updates of the anchor elements are reported in the audit log too, and otherwise each update of a UsageActivity classification would be counted as more activity, without end.

Nested Class Summary

Nested classes/interfaces inherited from class org.odpi.openmetadata.frameworks.connectors.ConnectorBase

ConnectorBase.ProtectedConnection

Field Summary

Fields inherited from class org.odpi.openmetadata.frameworks.integration.connectors.IntegrationConnectorBase

catalogTargetsManager, connectorName, integrationContext, propertyHelper

Fields inherited from class org.odpi.openmetadata.frameworks.connectors.ConnectorBase

auditLog, connectionBean, connectorInstanceId, embeddedConnectors, messageFormatter, secretsStoreConnectorMap

Constructor Summary

Constructors

Constructor	Description
PepysActivityObserverConnector()

Method Summary

Modifier and Type	Method	Description
void	disconnect()	Write any activity that has not been recorded yet before the connector shuts down.
RequestedCatalogTarget	getNewRequestedCatalogTargetSkeleton(CatalogTarget retrievedCatalogTarget, CatalogTargetContext catalogTargetContext, Connector connectorToTarget)	Create a new catalog target processor for each audit log topic.
void	processEvent(String event)	Receive an audit log record from one of the topics.
void	refresh()	Refresh the catalog targets (which picks up any new audit log topics) and then write the activity observed since the last refresh to the UsageActivity classifications.
void	start()	Indicates that the connector is completely configured and can begin processing.

Methods inherited from class org.odpi.openmetadata.frameworks.integration.connectors.DynamicIntegrationConnectorBase

processEvent

Methods inherited from class org.odpi.openmetadata.frameworks.integration.connectors.IntegrationConnectorBase

engage, getConnectorComponentDescription, getNetworkAddress, passEventToCatalogTargets, refreshCatalogTargets, setAuditLog, setConnectorName, setContext

Methods inherited from class org.odpi.openmetadata.frameworks.connectors.ConnectorBase

combineConfigurationProperties, disconnectConnectors, equals, fromCamelToCanonicalCase, fromCanonicalToCamelCase, fromCanonicalToKebabCase, fromCanonicalToSnakeCase, fromKebabToCanonicalCase, fromSnakeToCanonicalCase, getArrayConfigurationProperty, getArrayConfigurationProperty, getArrayConfigurationProperty, getArrayValue, getBooleanConfigurationProperty, getConnection, getConnectorInstanceId, getDateConfigurationProperty, getIntConfigurationProperty, getLongConfigurationProperty, getSecretsCollectionName, getSecretsLocation, getSecretsProvider, getStringConfigurationProperty, getStringConfigurationProperty, getSuppliedPlaceholderProperties, hashCode, initialize, initializeEmbeddedConnectors, initializeSecretsStoreConnector, isActive, logExceptionRecord, logExceptionRecord, logRecord, logRecord, throwMissingConfigurationProperty, throwMissingElement, throwMissingEndpointAddress, throwMissingResource, throwNoAsset, throwWrongTypeOfAsset, throwWrongTypeOfResource, throwWrongTypeOfRootSchema, toString

Methods inherited from class org.odpi.openmetadata.frameworks.connectors.Connector

clearStatisticProperty, clearStatisticTimestamp, getConnectorStatistics, getStatisticCounter, getStatisticProperty, getStatisticTimestamp, incrementStatisticCounter, initializeStatisticCounter, refreshStatistics, setStatisticProperty, setStatisticTimestamp

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.odpi.openmetadata.frameworks.connectors.VirtualConnectorExtension

initializeEmbeddedConnectors

Constructor Details

PepysActivityObserverConnector

public PepysActivityObserverConnector()

Method Details

start

public void start()
           throws ConnectorCheckedException,
UserNotAuthorizedException

Indicates that the connector is completely configured and can begin processing.

Specified by:

start in interface IntegrationConnector

Overrides:

start in class IntegrationConnectorBase

Throws:

ConnectorCheckedException - the connector detected a problem.

UserNotAuthorizedException - the connector was disconnected before/during start

refresh

public void refresh()
             throws ConnectorCheckedException,
UserNotAuthorizedException

Refresh the catalog targets (which picks up any new audit log topics) and then write the activity observed since the last refresh to the UsageActivity classifications.

Specified by:

refresh in interface IntegrationConnector

Overrides:

refresh in class DynamicIntegrationConnectorBase

Throws:

ConnectorCheckedException - a problem with the connector. It is not able to refresh the metadata.

UserNotAuthorizedException - the connector was disconnected

getNewRequestedCatalogTargetSkeleton

public RequestedCatalogTarget getNewRequestedCatalogTargetSkeleton(CatalogTarget retrievedCatalogTarget,
 CatalogTargetContext catalogTargetContext,
 Connector connectorToTarget)
                                                            throws ConnectorCheckedException,
UserNotAuthorizedException

Create a new catalog target processor for each audit log topic.

Specified by:

getNewRequestedCatalogTargetSkeleton in interface CatalogTargetFactory

Specified by:

getNewRequestedCatalogTargetSkeleton in class DynamicIntegrationConnectorBase

Parameters:

retrievedCatalogTarget - details of the open metadata elements describing the catalog target

catalogTargetContext - specialized context for this catalog target

connectorToTarget - connector to access the target resource

Returns:

new processor based on the catalog target information

Throws:

ConnectorCheckedException - the connector detected a problem.

UserNotAuthorizedException - the connector was disconnected before/during start

processEvent

public void processEvent(String event)

Receive an audit log record from one of the topics. Records that are not counted anchor element activity messages are ignored, as are those that are not well-formed audit log records.

Specified by:

processEvent in interface OpenMetadataTopicListener

Parameters:

event - inbound event

disconnect

public void disconnect()
                throws ConnectorCheckedException

Write any activity that has not been recorded yet before the connector shuts down.

Specified by:

disconnect in interface IntegrationConnector

Overrides:

disconnect in class IntegrationConnectorBase

Throws:

ConnectorCheckedException - there is a problem within the connector.