Hadoop Ecosystem
Connectors for Egeria
Getting Started
Orientation
TL;DR?
Just use the right arrow on your keyboard: each slide is one step, with minimal explanation.
Need more?
If at any point you want more of an explanation, use the down arrow on your keyboard to drill-down.
Keyboard shortcuts
Aha, you want more!
This getting started guide is written using a great web-based presentation framework
called reveal.js. You can use its icons
on the slides to move around, or you can use keyboard shortcuts. Try the ? key
to bring up a cheatsheet of the keyboard shortcuts.
At any point, you can also press the m key to bring up the menu. From the menu you
can quickly jump to any particular topic.
Looking for more about how the connectors actually work, rather than this primer on keyboard shortcuts? Proceed downwards once again.
How it works
The Apache Atlas connector is implemented as a repository proxy that runs through the Egeria OMAG Platform (server chassis). It does so by being implemented as an Open Metadata Repository Connector and implementing the repository-level methods defined by the Metadata Collection interface.
How it works
The connector communicates with an existing Apache Atlas environment through its Java client. Note that the connector only supports reading metadata from Apache Atlas and is unable to create or update any metadata. Write operations can only be done by a user through Apache Atlas's supported web UIs or APIs, or through some other pre-built integration with Apache Atlas (e.g. for Hive, etc).
How it works
The connector also implements an event mapper: this uses Apache Atlas's embedded Kafka notifications capability to automatically push metadata changes out to a broader cohort. Note that only metadata that produces such notifications within Apache Atlas can trigger this capability (this does not include, for example, glossary metadata like terms and categories).
1. Obtain connector
Start by downloading the Hadoop Ecosystem connectors package:
The single file downloaded contains both connectors:
egeria-connector-hadoop-ecosystem-package-{version}-jar-with-dependencies.jar
Snapshots
Our development snapshots are hosted directly within Nexus's open source snapshot repository.
You can download them there using Maven itself, or by
navigating to the site
and looking for the latest version of the jar-with-dependencies.jar file.
Alternatively, you may want to build it yourself...
Building
$ git clone https://github.com/odpi/egeria-connector-hadoop-ecosystem.git
Cloning into 'egeria-connector-hadoop-ecosystem'...
remote: Enumerating objects: 3249, done.
remote: Counting objects: 100% (288/288), done.
remote: Compressing objects: 100% (159/159), done.
remote: Total 3249 (delta 95), reused 220 (delta 60), pack-reused 2961
Receiving objects: 100% (3249/3249), 1.30 MiB | 1.03 MiB/s, done.
Resolving deltas: 100% (1359/1359), done.
To build the connector, you will first need to clone the connector repository code itself.
Building
$ cd egeria-connector-hadoop-ecosystem
$ mvn clean install
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Egeria Connector for Hadoop Ecosystem [pom]
[INFO] Apache Atlas Repository Connector [jar]
[INFO] Egeria Connector for Hadoop Ecosystem - packaging [jar]
[INFO]
[INFO] ---------< org.odpi.egeria:egeria-connector-hadoop-ecosystem >----------
[INFO] Building Egeria Connector for Hadoop Ecosystem 2.10-SNAPSHOT [1/3]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ egeria-connector-hadoop-ecosystem ---
[INFO] Deleting .../egeria-connector-hadoop-ecosystem/target
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M3:enforce (enforce-versions) @ egeria-connector-hadoop-ecosystem ---
[INFO]
[INFO] --- apache-rat-plugin:0.13:check (rat-check) @ egeria-connector-hadoop-ecosystem ---
[INFO] Added 3 additional default licenses.
[INFO] Enabled default license matchers.
[INFO] Added 2 custom approved licenses.
[INFO] Will parse SCM ignores for exclusions...
[INFO] Parsing exclusions from .../egeria-connector-hadoop-ecosystem/.gitignore
[INFO] Finished adding exclusions from SCM ignore files.
[INFO] 103 implicit excludes (use -debug for more details).
[INFO] 26 explicit excludes (use -debug for more details).
[INFO] 27 resources included (use -debug for more details)
[INFO] Rat check: Summary over all files. Unapproved: 0, unknown: 0, generated: 0, approved: 25 licenses.
[INFO]
[INFO] --- jacoco-maven-plugin:0.8.7:prepare-agent (agent) @ egeria-connector-hadoop-ecosystem ---
[INFO] argLine set to -javaagent:/Users/cgrote/.m2/repository/org/jacoco/org.jacoco.agent/0.8.7/org.jacoco.agent-0.8.7-runtime.jar=destfile=.../egeria-connector-hadoop-ecosystem/target/jacoco.exec
[INFO]
[INFO] --- maven-source-plugin:3.2.1:jar-no-fork (attach-sources) @ egeria-connector-hadoop-ecosystem ---
[INFO]
[INFO] --- maven-javadoc-plugin:3.3.0:jar (attach-javadocs) @ egeria-connector-hadoop-ecosystem ---
[INFO] Not executing Javadoc as the project is not a Java classpath-capable package
[INFO]
[INFO] --- jacoco-maven-plugin:0.8.7:report (report) @ egeria-connector-hadoop-ecosystem ---
[INFO] Skipping JaCoCo execution due to missing execution data file.
[INFO]
[INFO] --- maven-dependency-plugin:3.1.2:analyze-only (analyze) @ egeria-connector-hadoop-ecosystem ---
[INFO] Skipping pom project
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ egeria-connector-hadoop-ecosystem ---
[INFO] Installing .../egeria-connector-hadoop-ecosystem/pom.xml to /Users/cgrote/.m2/repository/org/odpi/egeria/egeria-connector-hadoop-ecosystem/2.10-SNAPSHOT/egeria-connector-hadoop-ecosystem-2.10-SNAPSHOT.pom
[INFO]
[INFO] -------< org.odpi.egeria:egeria-connector-apache-atlas-adapter >--------
[INFO] Building Apache Atlas Repository Connector 2.10-SNAPSHOT [2/3]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ egeria-connector-apache-atlas-adapter ---
[INFO] Deleting .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/target
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M3:enforce (enforce-versions) @ egeria-connector-apache-atlas-adapter ---
[INFO]
[INFO] --- apache-rat-plugin:0.13:check (rat-check) @ egeria-connector-apache-atlas-adapter ---
[INFO] Added 3 additional default licenses.
[INFO] Enabled default license matchers.
[INFO] Added 2 custom approved licenses.
[INFO] Will parse SCM ignores for exclusions...
[INFO] Finished adding exclusions from SCM ignore files.
[INFO] 62 implicit excludes (use -debug for more details).
[INFO] 26 explicit excludes (use -debug for more details).
[INFO] 39 resources included (use -debug for more details)
[INFO] Rat check: Summary over all files. Unapproved: 0, unknown: 0, generated: 0, approved: 39 licenses.
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ egeria-connector-apache-atlas-adapter ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 4 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ egeria-connector-apache-atlas-adapter ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 27 source files to .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/target/classes
[WARNING] .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/src/main/java/org/odpi/egeria/connectors/apache/atlas/repositoryconnector/mapping/EntityMappingAtlas2OMRS.java:[364,69] getSkeletonEntitySummary(java.lang.String,java.lang.String,org.odpi.openmetadata.repositoryservices.connectors.stores.metadatacollectionstore.properties.instances.InstanceProvenanceType,java.lang.String,java.lang.String) in org.odpi.openmetadata.repositoryservices.connectors.stores.metadatacollectionstore.repositoryconnector.OMRSRepositoryHelper has been deprecated
[WARNING] .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/src/main/java/org/odpi/egeria/connectors/apache/atlas/repositoryconnector/mapping/EntityMappingAtlas2OMRS.java:[396,68] getSkeletonEntity(java.lang.String,java.lang.String,org.odpi.openmetadata.repositoryservices.connectors.stores.metadatacollectionstore.properties.instances.InstanceProvenanceType,java.lang.String,java.lang.String) in org.odpi.openmetadata.repositoryservices.connectors.stores.metadatacollectionstore.repositoryconnector.OMRSRepositoryHelper has been deprecated
[WARNING] .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/src/main/java/org/odpi/egeria/connectors/apache/atlas/repositoryconnector/mapping/EntityMappingAtlas2OMRS.java:[504,103] getNewClassification(java.lang.String,java.lang.String,java.lang.String,java.lang.String,org.odpi.openmetadata.repositoryservices.connectors.stores.metadatacollectionstore.properties.instances.ClassificationOrigin,java.lang.String,org.odpi.openmetadata.repositoryservices.connectors.stores.metadatacollectionstore.properties.instances.InstanceProperties) in org.odpi.openmetadata.repositoryservices.connectors.stores.metadatacollectionstore.repositoryconnector.OMRSRepositoryHelper has been deprecated
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ egeria-connector-apache-atlas-adapter ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 99 resources
[INFO]
[INFO] --- jacoco-maven-plugin:0.8.7:prepare-agent (agent) @ egeria-connector-apache-atlas-adapter ---
[INFO] argLine set to -javaagent:/Users/cgrote/.m2/repository/org/jacoco/org.jacoco.agent/0.8.7/org.jacoco.agent-0.8.7-runtime.jar=destfile=.../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/target/jacoco.exec
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ egeria-connector-apache-atlas-adapter ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 8 source files to .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/target/test-classes
[INFO]
[INFO] --- mockserver-maven-plugin:5.11.2:start (process-test-classes) @ egeria-connector-apache-atlas-adapter ---
[INFO] mockserver:start about to start MockServer on: serverPort [1080]
[INFO]
[INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) @ egeria-connector-apache-atlas-adapter ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.odpi.egeria.connectors.apache.atlas.repositoryconnector.AttributeMappingTest
[INFO] Running org.odpi.egeria.connectors.apache.atlas.repositoryconnector.AtlasGuidTest
[INFO] Running org.odpi.egeria.connectors.apache.atlas.repositoryconnector.ConnectorTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.238 s - in org.odpi.egeria.connectors.apache.atlas.repositoryconnector.AtlasGuidTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.441 s - in org.odpi.egeria.connectors.apache.atlas.repositoryconnector.AttributeMappingTest
Fri May 28 09:45:04 BST 2021 TestServer Startup OMRS-AUDIT-0029 The Mock Outbound EventManager event manager is initializing
Fri May 28 09:45:04 BST 2021 TestServer Startup OMRS-AUDIT-0030 Registering the Mock EventPublisher event consumer with the Mock Outbound EventManager event manager
Fri May 28 09:45:04 BST 2021 TestServer Information OMRS-ATLAS-REPOSITORY-0001 The Apache Atlas proxy is starting a new server instance
Fri May 28 09:45:04 BST 2021 TestServer Information OMRS-ATLAS-REPOSITORY-0002 The Apache Atlas proxy is attempting to connect to Apache Atlas at http://localhost:1080
Fri May 28 09:45:06 BST 2021 TestServer Information OMRS-ATLAS-REPOSITORY-0003 The Apache Atlas proxy has successfully connected to Apache Atlas at http://localhost:1080
Fri May 28 09:45:06 BST 2021 TestServer Information OMRS-ATLAS-REPOSITORY-0004 The Apache Atlas proxy has started a new instance for server null
Fri May 28 09:45:06 BST 2021 TestServer Startup OMRS-AUDIT-0024 The local repositorys event mapper connector {0} is ready to send and receive events
Fri May 28 09:45:06 BST 2021 TestServer Information OMRS-ATLAS-REPOSITORY-0009 The Apache Atlas event mapper consumer thread is starting up
Fri May 28 09:45:07 BST 2021 TestServer Information OMRS-ATLAS-REPOSITORY-0010 The Apache Atlas event mapper is running
Fri May 28 09:45:15 BST 2021 TestServer Startup OMRS-AUDIT-0029 The Mock Outbound EventManager 2 event manager is initializing
Fri May 28 09:45:15 BST 2021 TestServer Startup OMRS-AUDIT-0030 Registering the Mock EventPublisher event consumer with the Mock Outbound EventManager 2 event manager
Fri May 28 09:45:15 BST 2021 TestServer Startup OMRS-AUDIT-0030 Registering the Mock EventPublisher event consumer with the Mock Outbound EventManager 2 event manager
Fri May 28 09:45:15 BST 2021 TestServer Information OMRS-ATLAS-REPOSITORY-0002 The Apache Atlas proxy is attempting to connect to Apache Atlas at http://localhost:1080
Fri May 28 09:45:19 BST 2021 TestServer Shutdown OMRS-AUDIT-0025 The local repositorys event mapper connector {0} has disconnected the event bus connectors
Fri May 28 09:45:19 BST 2021 TestServer Information OMRS-ATLAS-REPOSITORY-0008 The Apache Atlas event mapper has shutdown its instance for server null
Fri May 28 09:45:19 BST 2021 TestServer Information OMRS-ATLAS-REPOSITORY-0005 The Apache Atlas proxy has shutdown its instance for server null
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.608 s - in org.odpi.egeria.connectors.apache.atlas.repositoryconnector.ConnectorTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ egeria-connector-apache-atlas-adapter ---
[INFO] Building jar: .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/target/egeria-connector-apache-atlas-adapter-2.10-SNAPSHOT.jar
[INFO]
[INFO] --- maven-source-plugin:3.2.1:jar-no-fork (attach-sources) @ egeria-connector-apache-atlas-adapter ---
[INFO] Building jar: .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/target/egeria-connector-apache-atlas-adapter-2.10-SNAPSHOT-sources.jar
[INFO]
[INFO] --- maven-javadoc-plugin:3.3.0:jar (attach-javadocs) @ egeria-connector-apache-atlas-adapter ---
[INFO] No previous run data found, generating javadoc.
[INFO] Building jar: .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/target/egeria-connector-apache-atlas-adapter-2.10-SNAPSHOT-javadoc.jar
[INFO]
[INFO] --- mockserver-maven-plugin:5.11.2:stop (package) @ egeria-connector-apache-atlas-adapter ---
[INFO] Stopping the MockServer
[INFO]
[INFO] --- jacoco-maven-plugin:0.8.7:report (report) @ egeria-connector-apache-atlas-adapter ---
[INFO] Loading execution data file .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/target/jacoco.exec
[INFO] Analyzed bundle 'Apache Atlas Repository Connector' with 27 classes
[INFO]
[INFO] --- maven-dependency-plugin:3.1.2:analyze-only (analyze) @ egeria-connector-apache-atlas-adapter ---
[INFO] No dependency problems found
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ egeria-connector-apache-atlas-adapter ---
[INFO] Installing .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/target/egeria-connector-apache-atlas-adapter-2.10-SNAPSHOT.jar to /Users/cgrote/.m2/repository/org/odpi/egeria/egeria-connector-apache-atlas-adapter/2.10-SNAPSHOT/egeria-connector-apache-atlas-adapter-2.10-SNAPSHOT.jar
[INFO] Installing .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/pom.xml to /Users/cgrote/.m2/repository/org/odpi/egeria/egeria-connector-apache-atlas-adapter/2.10-SNAPSHOT/egeria-connector-apache-atlas-adapter-2.10-SNAPSHOT.pom
[INFO] Installing .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/target/egeria-connector-apache-atlas-adapter-2.10-SNAPSHOT-sources.jar to /Users/cgrote/.m2/repository/org/odpi/egeria/egeria-connector-apache-atlas-adapter/2.10-SNAPSHOT/egeria-connector-apache-atlas-adapter-2.10-SNAPSHOT-sources.jar
[INFO] Installing .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/target/egeria-connector-apache-atlas-adapter-2.10-SNAPSHOT-javadoc.jar to /Users/cgrote/.m2/repository/org/odpi/egeria/egeria-connector-apache-atlas-adapter/2.10-SNAPSHOT/egeria-connector-apache-atlas-adapter-2.10-SNAPSHOT-javadoc.jar
[INFO]
[INFO] -----< org.odpi.egeria:egeria-connector-hadoop-ecosystem-package >------
[INFO] Building Egeria Connector for Hadoop Ecosystem - packaging 2.10-SNAPSHOT [3/3]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] Deleting .../egeria-connector-hadoop-ecosystem/distribution/target
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M3:enforce (enforce-versions) @ egeria-connector-hadoop-ecosystem-package ---
[INFO]
[INFO] --- apache-rat-plugin:0.13:check (rat-check) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] Added 3 additional default licenses.
[INFO] Enabled default license matchers.
[INFO] Added 2 custom approved licenses.
[INFO] Will parse SCM ignores for exclusions...
[INFO] Finished adding exclusions from SCM ignore files.
[INFO] 62 implicit excludes (use -debug for more details).
[INFO] 26 explicit excludes (use -debug for more details).
[INFO] 4 resources included (use -debug for more details)
[INFO] Rat check: Summary over all files. Unapproved: 0, unknown: 0, generated: 0, approved: 4 licenses.
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory .../egeria-connector-hadoop-ecosystem/distribution/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory .../egeria-connector-hadoop-ecosystem/distribution/src/test/resources
[INFO]
[INFO] --- jacoco-maven-plugin:0.8.7:prepare-agent (agent) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] argLine set to -javaagent:/Users/cgrote/.m2/repository/org/jacoco/org.jacoco.agent/0.8.7/org.jacoco.agent-0.8.7-runtime.jar=destfile=.../egeria-connector-hadoop-ecosystem/distribution/target/jacoco.exec
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] No tests to run.
[INFO]
[INFO] --- maven-assembly-plugin:3.3.0:single (default) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] Reading assembly descriptor: src/main/assemblies/src.xml
[INFO] Reading assembly descriptor: src/main/assemblies/javadoc.xml
[INFO] Reading assembly descriptor: src/main/assemblies/connector.xml
[INFO] Building jar: .../egeria-connector-hadoop-ecosystem/distribution/target/egeria-connector-hadoop-ecosystem-2.10-SNAPSHOT-sources.jar
[INFO] Building jar: .../egeria-connector-hadoop-ecosystem/distribution/target/egeria-connector-hadoop-ecosystem-2.10-SNAPSHOT-javadoc.jar
[INFO] Building jar: .../egeria-connector-hadoop-ecosystem/distribution/target/egeria-connector-hadoop-ecosystem-2.10-SNAPSHOT-jar-with-dependencies.jar
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ egeria-connector-hadoop-ecosystem-package ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: .../egeria-connector-hadoop-ecosystem/distribution/target/egeria-connector-hadoop-ecosystem-package-2.10-SNAPSHOT.jar
[INFO]
[INFO] --- maven-source-plugin:3.2.1:jar-no-fork (attach-sources) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] No sources in project. Archive not created.
[INFO]
[INFO] --- maven-javadoc-plugin:3.3.0:jar (attach-javadocs) @ egeria-connector-hadoop-ecosystem-package ---
[INFO]
[INFO] --- jacoco-maven-plugin:0.8.7:report (report) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] Skipping JaCoCo execution due to missing execution data file.
[INFO]
[INFO] --- maven-dependency-plugin:3.1.2:analyze-only (analyze) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] No dependency problems found
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] Installing .../egeria-connector-hadoop-ecosystem/distribution/target/egeria-connector-hadoop-ecosystem-package-2.10-SNAPSHOT.jar to /Users/cgrote/.m2/repository/org/odpi/egeria/egeria-connector-hadoop-ecosystem-package/2.10-SNAPSHOT/egeria-connector-hadoop-ecosystem-package-2.10-SNAPSHOT.jar
[INFO] Installing .../egeria-connector-hadoop-ecosystem/distribution/pom.xml to /Users/cgrote/.m2/repository/org/odpi/egeria/egeria-connector-hadoop-ecosystem-package/2.10-SNAPSHOT/egeria-connector-hadoop-ecosystem-package-2.10-SNAPSHOT.pom
[INFO] Installing .../egeria-connector-hadoop-ecosystem/distribution/target/egeria-connector-hadoop-ecosystem-2.10-SNAPSHOT-sources.jar to /Users/cgrote/.m2/repository/org/odpi/egeria/egeria-connector-hadoop-ecosystem-package/2.10-SNAPSHOT/egeria-connector-hadoop-ecosystem-package-2.10-SNAPSHOT-sources.jar
[INFO] Installing .../egeria-connector-hadoop-ecosystem/distribution/target/egeria-connector-hadoop-ecosystem-2.10-SNAPSHOT-javadoc.jar to /Users/cgrote/.m2/repository/org/odpi/egeria/egeria-connector-hadoop-ecosystem-package/2.10-SNAPSHOT/egeria-connector-hadoop-ecosystem-package-2.10-SNAPSHOT-javadoc.jar
[INFO] Installing .../egeria-connector-hadoop-ecosystem/distribution/target/egeria-connector-hadoop-ecosystem-2.10-SNAPSHOT-jar-with-dependencies.jar to /Users/cgrote/.m2/repository/org/odpi/egeria/egeria-connector-hadoop-ecosystem-package/2.10-SNAPSHOT/egeria-connector-hadoop-ecosystem-package-2.10-SNAPSHOT-jar-with-dependencies.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Egeria Connector for Hadoop Ecosystem 2.10-SNAPSHOT:
[INFO]
[INFO] Egeria Connector for Hadoop Ecosystem .............. SUCCESS [ 2.221 s]
[INFO] Apache Atlas Repository Connector .................. SUCCESS [ 39.063 s]
[INFO] Egeria Connector for Hadoop Ecosystem - packaging .. SUCCESS [ 4.452 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 45.851 s
[INFO] Finished at: 2021-05-28T09:45:33+01:00
[INFO] ------------------------------------------------------------------------
You can now build the connector. You will need to have at least Java version 1.8 installed as well as Apache Maven (at least version 3.6). Build the connector using the commands highlighted above.
Building
$ cd egeria-connector-hadoop-ecosystem
$ mvn clean install
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Egeria Connector for Hadoop Ecosystem [pom]
[INFO] Apache Atlas Repository Connector [jar]
[INFO] Egeria Connector for Hadoop Ecosystem - packaging [jar]
[INFO]
[INFO] ---------< org.odpi.egeria:egeria-connector-hadoop-ecosystem >----------
[INFO] Building Egeria Connector for Hadoop Ecosystem 2.10-SNAPSHOT [1/3]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ egeria-connector-hadoop-ecosystem ---
[INFO] Deleting .../egeria-connector-hadoop-ecosystem/target
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M3:enforce (enforce-versions) @ egeria-connector-hadoop-ecosystem ---
[INFO]
[INFO] --- apache-rat-plugin:0.13:check (rat-check) @ egeria-connector-hadoop-ecosystem ---
[INFO] Added 3 additional default licenses.
[INFO] Enabled default license matchers.
[INFO] Added 2 custom approved licenses.
[INFO] Will parse SCM ignores for exclusions...
[INFO] Parsing exclusions from .../egeria-connector-hadoop-ecosystem/.gitignore
[INFO] Finished adding exclusions from SCM ignore files.
[INFO] 103 implicit excludes (use -debug for more details).
[INFO] 26 explicit excludes (use -debug for more details).
[INFO] 27 resources included (use -debug for more details)
[INFO] Rat check: Summary over all files. Unapproved: 0, unknown: 0, generated: 0, approved: 25 licenses.
[INFO]
[INFO] --- jacoco-maven-plugin:0.8.7:prepare-agent (agent) @ egeria-connector-hadoop-ecosystem ---
[INFO] argLine set to -javaagent:/Users/cgrote/.m2/repository/org/jacoco/org.jacoco.agent/0.8.7/org.jacoco.agent-0.8.7-runtime.jar=destfile=.../egeria-connector-hadoop-ecosystem/target/jacoco.exec
[INFO]
[INFO] --- maven-source-plugin:3.2.1:jar-no-fork (attach-sources) @ egeria-connector-hadoop-ecosystem ---
[INFO]
[INFO] --- maven-javadoc-plugin:3.3.0:jar (attach-javadocs) @ egeria-connector-hadoop-ecosystem ---
[INFO] Not executing Javadoc as the project is not a Java classpath-capable package
[INFO]
[INFO] --- jacoco-maven-plugin:0.8.7:report (report) @ egeria-connector-hadoop-ecosystem ---
[INFO] Skipping JaCoCo execution due to missing execution data file.
[INFO]
[INFO] --- maven-dependency-plugin:3.1.2:analyze-only (analyze) @ egeria-connector-hadoop-ecosystem ---
[INFO] Skipping pom project
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ egeria-connector-hadoop-ecosystem ---
[INFO] Installing .../egeria-connector-hadoop-ecosystem/pom.xml to /Users/cgrote/.m2/repository/org/odpi/egeria/egeria-connector-hadoop-ecosystem/2.10-SNAPSHOT/egeria-connector-hadoop-ecosystem-2.10-SNAPSHOT.pom
[INFO]
[INFO] -------< org.odpi.egeria:egeria-connector-apache-atlas-adapter >--------
[INFO] Building Apache Atlas Repository Connector 2.10-SNAPSHOT [2/3]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ egeria-connector-apache-atlas-adapter ---
[INFO] Deleting .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/target
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M3:enforce (enforce-versions) @ egeria-connector-apache-atlas-adapter ---
[INFO]
[INFO] --- apache-rat-plugin:0.13:check (rat-check) @ egeria-connector-apache-atlas-adapter ---
[INFO] Added 3 additional default licenses.
[INFO] Enabled default license matchers.
[INFO] Added 2 custom approved licenses.
[INFO] Will parse SCM ignores for exclusions...
[INFO] Finished adding exclusions from SCM ignore files.
[INFO] 62 implicit excludes (use -debug for more details).
[INFO] 26 explicit excludes (use -debug for more details).
[INFO] 39 resources included (use -debug for more details)
[INFO] Rat check: Summary over all files. Unapproved: 0, unknown: 0, generated: 0, approved: 39 licenses.
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ egeria-connector-apache-atlas-adapter ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 4 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ egeria-connector-apache-atlas-adapter ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 27 source files to .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/target/classes
[WARNING] .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/src/main/java/org/odpi/egeria/connectors/apache/atlas/repositoryconnector/mapping/EntityMappingAtlas2OMRS.java:[364,69] getSkeletonEntitySummary(java.lang.String,java.lang.String,org.odpi.openmetadata.repositoryservices.connectors.stores.metadatacollectionstore.properties.instances.InstanceProvenanceType,java.lang.String,java.lang.String) in org.odpi.openmetadata.repositoryservices.connectors.stores.metadatacollectionstore.repositoryconnector.OMRSRepositoryHelper has been deprecated
[WARNING] .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/src/main/java/org/odpi/egeria/connectors/apache/atlas/repositoryconnector/mapping/EntityMappingAtlas2OMRS.java:[396,68] getSkeletonEntity(java.lang.String,java.lang.String,org.odpi.openmetadata.repositoryservices.connectors.stores.metadatacollectionstore.properties.instances.InstanceProvenanceType,java.lang.String,java.lang.String) in org.odpi.openmetadata.repositoryservices.connectors.stores.metadatacollectionstore.repositoryconnector.OMRSRepositoryHelper has been deprecated
[WARNING] .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/src/main/java/org/odpi/egeria/connectors/apache/atlas/repositoryconnector/mapping/EntityMappingAtlas2OMRS.java:[504,103] getNewClassification(java.lang.String,java.lang.String,java.lang.String,java.lang.String,org.odpi.openmetadata.repositoryservices.connectors.stores.metadatacollectionstore.properties.instances.ClassificationOrigin,java.lang.String,org.odpi.openmetadata.repositoryservices.connectors.stores.metadatacollectionstore.properties.instances.InstanceProperties) in org.odpi.openmetadata.repositoryservices.connectors.stores.metadatacollectionstore.repositoryconnector.OMRSRepositoryHelper has been deprecated
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ egeria-connector-apache-atlas-adapter ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 99 resources
[INFO]
[INFO] --- jacoco-maven-plugin:0.8.7:prepare-agent (agent) @ egeria-connector-apache-atlas-adapter ---
[INFO] argLine set to -javaagent:/Users/cgrote/.m2/repository/org/jacoco/org.jacoco.agent/0.8.7/org.jacoco.agent-0.8.7-runtime.jar=destfile=.../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/target/jacoco.exec
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ egeria-connector-apache-atlas-adapter ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 8 source files to .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/target/test-classes
[INFO]
[INFO] --- mockserver-maven-plugin:5.11.2:start (process-test-classes) @ egeria-connector-apache-atlas-adapter ---
[INFO] mockserver:start about to start MockServer on: serverPort [1080]
[INFO]
[INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) @ egeria-connector-apache-atlas-adapter ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.odpi.egeria.connectors.apache.atlas.repositoryconnector.AttributeMappingTest
[INFO] Running org.odpi.egeria.connectors.apache.atlas.repositoryconnector.AtlasGuidTest
[INFO] Running org.odpi.egeria.connectors.apache.atlas.repositoryconnector.ConnectorTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.238 s - in org.odpi.egeria.connectors.apache.atlas.repositoryconnector.AtlasGuidTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.441 s - in org.odpi.egeria.connectors.apache.atlas.repositoryconnector.AttributeMappingTest
Fri May 28 09:45:04 BST 2021 TestServer Startup OMRS-AUDIT-0029 The Mock Outbound EventManager event manager is initializing
Fri May 28 09:45:04 BST 2021 TestServer Startup OMRS-AUDIT-0030 Registering the Mock EventPublisher event consumer with the Mock Outbound EventManager event manager
Fri May 28 09:45:04 BST 2021 TestServer Information OMRS-ATLAS-REPOSITORY-0001 The Apache Atlas proxy is starting a new server instance
Fri May 28 09:45:04 BST 2021 TestServer Information OMRS-ATLAS-REPOSITORY-0002 The Apache Atlas proxy is attempting to connect to Apache Atlas at http://localhost:1080
Fri May 28 09:45:06 BST 2021 TestServer Information OMRS-ATLAS-REPOSITORY-0003 The Apache Atlas proxy has successfully connected to Apache Atlas at http://localhost:1080
Fri May 28 09:45:06 BST 2021 TestServer Information OMRS-ATLAS-REPOSITORY-0004 The Apache Atlas proxy has started a new instance for server null
Fri May 28 09:45:06 BST 2021 TestServer Startup OMRS-AUDIT-0024 The local repositorys event mapper connector {0} is ready to send and receive events
Fri May 28 09:45:06 BST 2021 TestServer Information OMRS-ATLAS-REPOSITORY-0009 The Apache Atlas event mapper consumer thread is starting up
Fri May 28 09:45:07 BST 2021 TestServer Information OMRS-ATLAS-REPOSITORY-0010 The Apache Atlas event mapper is running
Fri May 28 09:45:15 BST 2021 TestServer Startup OMRS-AUDIT-0029 The Mock Outbound EventManager 2 event manager is initializing
Fri May 28 09:45:15 BST 2021 TestServer Startup OMRS-AUDIT-0030 Registering the Mock EventPublisher event consumer with the Mock Outbound EventManager 2 event manager
Fri May 28 09:45:15 BST 2021 TestServer Startup OMRS-AUDIT-0030 Registering the Mock EventPublisher event consumer with the Mock Outbound EventManager 2 event manager
Fri May 28 09:45:15 BST 2021 TestServer Information OMRS-ATLAS-REPOSITORY-0002 The Apache Atlas proxy is attempting to connect to Apache Atlas at http://localhost:1080
Fri May 28 09:45:19 BST 2021 TestServer Shutdown OMRS-AUDIT-0025 The local repositorys event mapper connector {0} has disconnected the event bus connectors
Fri May 28 09:45:19 BST 2021 TestServer Information OMRS-ATLAS-REPOSITORY-0008 The Apache Atlas event mapper has shutdown its instance for server null
Fri May 28 09:45:19 BST 2021 TestServer Information OMRS-ATLAS-REPOSITORY-0005 The Apache Atlas proxy has shutdown its instance for server null
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.608 s - in org.odpi.egeria.connectors.apache.atlas.repositoryconnector.ConnectorTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ egeria-connector-apache-atlas-adapter ---
[INFO] Building jar: .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/target/egeria-connector-apache-atlas-adapter-2.10-SNAPSHOT.jar
[INFO]
[INFO] --- maven-source-plugin:3.2.1:jar-no-fork (attach-sources) @ egeria-connector-apache-atlas-adapter ---
[INFO] Building jar: .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/target/egeria-connector-apache-atlas-adapter-2.10-SNAPSHOT-sources.jar
[INFO]
[INFO] --- maven-javadoc-plugin:3.3.0:jar (attach-javadocs) @ egeria-connector-apache-atlas-adapter ---
[INFO] No previous run data found, generating javadoc.
[INFO] Building jar: .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/target/egeria-connector-apache-atlas-adapter-2.10-SNAPSHOT-javadoc.jar
[INFO]
[INFO] --- mockserver-maven-plugin:5.11.2:stop (package) @ egeria-connector-apache-atlas-adapter ---
[INFO] Stopping the MockServer
[INFO]
[INFO] --- jacoco-maven-plugin:0.8.7:report (report) @ egeria-connector-apache-atlas-adapter ---
[INFO] Loading execution data file .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/target/jacoco.exec
[INFO] Analyzed bundle 'Apache Atlas Repository Connector' with 27 classes
[INFO]
[INFO] --- maven-dependency-plugin:3.1.2:analyze-only (analyze) @ egeria-connector-apache-atlas-adapter ---
[INFO] No dependency problems found
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ egeria-connector-apache-atlas-adapter ---
[INFO] Installing .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/target/egeria-connector-apache-atlas-adapter-2.10-SNAPSHOT.jar to /Users/cgrote/.m2/repository/org/odpi/egeria/egeria-connector-apache-atlas-adapter/2.10-SNAPSHOT/egeria-connector-apache-atlas-adapter-2.10-SNAPSHOT.jar
[INFO] Installing .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/pom.xml to /Users/cgrote/.m2/repository/org/odpi/egeria/egeria-connector-apache-atlas-adapter/2.10-SNAPSHOT/egeria-connector-apache-atlas-adapter-2.10-SNAPSHOT.pom
[INFO] Installing .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/target/egeria-connector-apache-atlas-adapter-2.10-SNAPSHOT-sources.jar to /Users/cgrote/.m2/repository/org/odpi/egeria/egeria-connector-apache-atlas-adapter/2.10-SNAPSHOT/egeria-connector-apache-atlas-adapter-2.10-SNAPSHOT-sources.jar
[INFO] Installing .../egeria-connector-hadoop-ecosystem/apache-atlas-adapter/target/egeria-connector-apache-atlas-adapter-2.10-SNAPSHOT-javadoc.jar to /Users/cgrote/.m2/repository/org/odpi/egeria/egeria-connector-apache-atlas-adapter/2.10-SNAPSHOT/egeria-connector-apache-atlas-adapter-2.10-SNAPSHOT-javadoc.jar
[INFO]
[INFO] -----< org.odpi.egeria:egeria-connector-hadoop-ecosystem-package >------
[INFO] Building Egeria Connector for Hadoop Ecosystem - packaging 2.10-SNAPSHOT [3/3]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] Deleting .../egeria-connector-hadoop-ecosystem/distribution/target
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M3:enforce (enforce-versions) @ egeria-connector-hadoop-ecosystem-package ---
[INFO]
[INFO] --- apache-rat-plugin:0.13:check (rat-check) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] Added 3 additional default licenses.
[INFO] Enabled default license matchers.
[INFO] Added 2 custom approved licenses.
[INFO] Will parse SCM ignores for exclusions...
[INFO] Finished adding exclusions from SCM ignore files.
[INFO] 62 implicit excludes (use -debug for more details).
[INFO] 26 explicit excludes (use -debug for more details).
[INFO] 4 resources included (use -debug for more details)
[INFO] Rat check: Summary over all files. Unapproved: 0, unknown: 0, generated: 0, approved: 4 licenses.
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory .../egeria-connector-hadoop-ecosystem/distribution/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory .../egeria-connector-hadoop-ecosystem/distribution/src/test/resources
[INFO]
[INFO] --- jacoco-maven-plugin:0.8.7:prepare-agent (agent) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] argLine set to -javaagent:/Users/cgrote/.m2/repository/org/jacoco/org.jacoco.agent/0.8.7/org.jacoco.agent-0.8.7-runtime.jar=destfile=.../egeria-connector-hadoop-ecosystem/distribution/target/jacoco.exec
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] No tests to run.
[INFO]
[INFO] --- maven-assembly-plugin:3.3.0:single (default) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] Reading assembly descriptor: src/main/assemblies/src.xml
[INFO] Reading assembly descriptor: src/main/assemblies/javadoc.xml
[INFO] Reading assembly descriptor: src/main/assemblies/connector.xml
[INFO] Building jar: .../egeria-connector-hadoop-ecosystem/distribution/target/egeria-connector-hadoop-ecosystem-2.10-SNAPSHOT-sources.jar
[INFO] Building jar: .../egeria-connector-hadoop-ecosystem/distribution/target/egeria-connector-hadoop-ecosystem-2.10-SNAPSHOT-javadoc.jar
[INFO] Building jar: .../egeria-connector-hadoop-ecosystem/distribution/target/egeria-connector-hadoop-ecosystem-2.10-SNAPSHOT-jar-with-dependencies.jar
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ egeria-connector-hadoop-ecosystem-package ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: .../egeria-connector-hadoop-ecosystem/distribution/target/egeria-connector-hadoop-ecosystem-package-2.10-SNAPSHOT.jar
[INFO]
[INFO] --- maven-source-plugin:3.2.1:jar-no-fork (attach-sources) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] No sources in project. Archive not created.
[INFO]
[INFO] --- maven-javadoc-plugin:3.3.0:jar (attach-javadocs) @ egeria-connector-hadoop-ecosystem-package ---
[INFO]
[INFO] --- jacoco-maven-plugin:0.8.7:report (report) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] Skipping JaCoCo execution due to missing execution data file.
[INFO]
[INFO] --- maven-dependency-plugin:3.1.2:analyze-only (analyze) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] No dependency problems found
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ egeria-connector-hadoop-ecosystem-package ---
[INFO] Installing .../egeria-connector-hadoop-ecosystem/distribution/target/egeria-connector-hadoop-ecosystem-package-2.10-SNAPSHOT.jar to /Users/cgrote/.m2/repository/org/odpi/egeria/egeria-connector-hadoop-ecosystem-package/2.10-SNAPSHOT/egeria-connector-hadoop-ecosystem-package-2.10-SNAPSHOT.jar
[INFO] Installing .../egeria-connector-hadoop-ecosystem/distribution/pom.xml to /Users/cgrote/.m2/repository/org/odpi/egeria/egeria-connector-hadoop-ecosystem-package/2.10-SNAPSHOT/egeria-connector-hadoop-ecosystem-package-2.10-SNAPSHOT.pom
[INFO] Installing .../egeria-connector-hadoop-ecosystem/distribution/target/egeria-connector-hadoop-ecosystem-2.10-SNAPSHOT-sources.jar to /Users/cgrote/.m2/repository/org/odpi/egeria/egeria-connector-hadoop-ecosystem-package/2.10-SNAPSHOT/egeria-connector-hadoop-ecosystem-package-2.10-SNAPSHOT-sources.jar
[INFO] Installing .../egeria-connector-hadoop-ecosystem/distribution/target/egeria-connector-hadoop-ecosystem-2.10-SNAPSHOT-javadoc.jar to /Users/cgrote/.m2/repository/org/odpi/egeria/egeria-connector-hadoop-ecosystem-package/2.10-SNAPSHOT/egeria-connector-hadoop-ecosystem-package-2.10-SNAPSHOT-javadoc.jar
[INFO] Installing .../egeria-connector-hadoop-ecosystem/distribution/target/egeria-connector-hadoop-ecosystem-2.10-SNAPSHOT-jar-with-dependencies.jar to /Users/cgrote/.m2/repository/org/odpi/egeria/egeria-connector-hadoop-ecosystem-package/2.10-SNAPSHOT/egeria-connector-hadoop-ecosystem-package-2.10-SNAPSHOT-jar-with-dependencies.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Egeria Connector for Hadoop Ecosystem 2.10-SNAPSHOT:
[INFO]
[INFO] Egeria Connector for Hadoop Ecosystem .............. SUCCESS [ 2.221 s]
[INFO] Apache Atlas Repository Connector .................. SUCCESS [ 39.063 s]
[INFO] Egeria Connector for Hadoop Ecosystem - packaging .. SUCCESS [ 4.452 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 45.851 s
[INFO] Finished at: 2021-05-28T09:45:33+01:00
[INFO] ------------------------------------------------------------------------
Note from the output that the build creates the file .../egeria-connector-hadoop-ecosystem/distribution/target/egeria-connector-hadoop-ecosystem-2.10-SNAPSHOT-jar-with-dependencies.jar.
This is the connector, which you can use just as if you had downloaded it.
2. Obtain server chassis
Download Egeria's server chassis (you will need at least the same version as the connector, or later):
The single file downloaded is the server chassis:
server-chassis-spring-{version}.jar
Snapshots
Our development snapshots are hosted on Nexus's open source repository. You can download them there
using Maven itself, or by
navigating to the site
and looking for the latest version of the jar-with-dependencies.jar file.
Alternatively, you may want to build it yourself...
Building
$ git clone https://github.com/odpi/egeria.git
Cloning into 'egeria'...
remote: Enumerating objects: 22, done.
remote: Counting objects: 100% (22/22), done.
remote: Compressing objects: 100% (15/15), done.
remote: Total 318834 (delta 0), reused 12 (delta 0), pack-reused 318812
Receiving objects: 100% (318834/318834), 185.92 MiB | 20.81 MiB/s, done.
Resolving deltas: 100% (165808/165808), done.
Updating files: 100% (7494/7494), done.
$ cd egeria
$ mvn clean install
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Egeria [pom]
[INFO] Open Metadata Implementation [pom]
[INFO] Open Metadata Frameworks [pom]
[INFO] Audit Log Framework (ALF) [jar]
[INFO] Open Connector Framework (OCF) [jar]
[INFO] Open Discovery Framework (ODF) [jar]
[INFO] Governance Action Framework (GAF) [jar]
[INFO] Open Metadata Test Resources [pom]
[INFO] Open Metadata Unit Test Utilities [jar]
[INFO] Repository Services (OMRS) [pom]
[INFO] Repository Services General APIs [jar]
... truncated ...
[INFO] ---------------< org.odpi.egeria:server-chassis-spring >----------------
[INFO] Building OMAG Server Platform Chassis for Spring 2.9-SNAPSHOT [335/383]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ server-chassis-spring ---
... truncated ...
[INFO] --- maven-install-plugin:3.0.0-M1:install (default-install) @ server-chassis-spring ---
[INFO] Installing .../egeria/open-metadata-implementation/server-chassis/server-chassis-spring/target/server-chassis-spring-2.9-SNAPSHOT.jar to /Users/cgrote/.m2/repository/org/odpi/egeria/server-chassis-spring/2.9-SNAPSHOT/server-chassis-spring-2.9-SNAPSHOT.jar
[INFO] Installing .../egeria/open-metadata-implementation/server-chassis/server-chassis-spring/pom.xml to /Users/cgrote/.m2/repository/org/odpi/egeria/server-chassis-spring/2.9-SNAPSHOT/server-chassis-spring-2.9-SNAPSHOT.pom
[INFO] Installing .../egeria/open-metadata-implementation/server-chassis/server-chassis-spring/target/server-chassis-spring-2.9-SNAPSHOT-sources.jar to /Users/cgrote/.m2/repository/org/odpi/egeria/server-chassis-spring/2.9-SNAPSHOT/server-chassis-spring-2.9-SNAPSHOT-sources.jar
[INFO] Installing .../egeria/open-metadata-implementation/server-chassis/server-chassis-spring/target/server-chassis-spring-2.9-SNAPSHOT-javadoc.jar to /Users/cgrote/.m2/repository/org/odpi/egeria/server-chassis-spring/2.9-SNAPSHOT/server-chassis-spring-2.9-SNAPSHOT-javadoc.jar
[INFO]
[INFO] ------------------< org.odpi.egeria:user-interfaces >-------------------
[INFO] Building Open Metadata User Interfaces 2.9-SNAPSHOT [336/383]
[INFO] --------------------------------[ pom ]---------------------------------
... truncated ...
[INFO] OMAS FV and SV Tests ............................... SUCCESS [ 0.260 s]
[INFO] Asset Consumer OMAS FVT ............................ SUCCESS [01:05 min]
[INFO] Asset Manager OMAS FVT ............................. SUCCESS [01:02 min]
[INFO] Asset Owner OMAS FVT ............................... SUCCESS [ 43.237 s]
[INFO] Data Engine OMAS FVT ............................... SUCCESS [02:55 min]
[INFO] Data Manager OMAS FVT .............................. SUCCESS [ 44.278 s]
[INFO] Digital Architecture OMAS FVT ...................... SUCCESS [ 52.186 s]
[INFO] Subject Area OMAS FVT .............................. SUCCESS [02:03 min]
[INFO] Open Metadata Open Types Test Resources ............ SUCCESS [ 0.196 s]
[INFO] Open Types Test generator .......................... SUCCESS [ 3.430 s]
[INFO] Open Types Test execution .......................... SUCCESS [ 37.326 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 26:47 min
[INFO] Finished at: 2021-02-12T18:06:19Z
[INFO] ------------------------------------------------------------------------
To build the server chassis, you will need to have at least Java version 1.8 installed as well as Apache Maven (at least version 3.6). Clone Egeria's GitHub repository and then build the project using the commands highlighted above.
(Note that this builds all of Egeria's core, which is significantly larger than the connector. The output has therefore been heavily truncated in various places.)
Building
$ git clone https://github.com/odpi/egeria.git
Cloning into 'egeria'...
remote: Enumerating objects: 22, done.
remote: Counting objects: 100% (22/22), done.
remote: Compressing objects: 100% (15/15), done.
remote: Total 318834 (delta 0), reused 12 (delta 0), pack-reused 318812
Receiving objects: 100% (318834/318834), 185.92 MiB | 20.81 MiB/s, done.
Resolving deltas: 100% (165808/165808), done.
Updating files: 100% (7494/7494), done.
$ cd egeria
$ mvn clean install
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Egeria [pom]
[INFO] Open Metadata Implementation [pom]
[INFO] Open Metadata Frameworks [pom]
[INFO] Audit Log Framework (ALF) [jar]
[INFO] Open Connector Framework (OCF) [jar]
[INFO] Open Discovery Framework (ODF) [jar]
[INFO] Governance Action Framework (GAF) [jar]
[INFO] Open Metadata Test Resources [pom]
[INFO] Open Metadata Unit Test Utilities [jar]
[INFO] Repository Services (OMRS) [pom]
[INFO] Repository Services General APIs [jar]
... truncated ...
[INFO] ---------------< org.odpi.egeria:server-chassis-spring >----------------
[INFO] Building OMAG Server Platform Chassis for Spring 2.9-SNAPSHOT [335/383]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ server-chassis-spring ---
... truncated ...
[INFO] --- maven-install-plugin:3.0.0-M1:install (default-install) @ server-chassis-spring ---
[INFO] Installing .../egeria/open-metadata-implementation/server-chassis/server-chassis-spring/target/server-chassis-spring-2.9-SNAPSHOT.jar to /Users/cgrote/.m2/repository/org/odpi/egeria/server-chassis-spring/2.9-SNAPSHOT/server-chassis-spring-2.9-SNAPSHOT.jar
[INFO] Installing .../egeria/open-metadata-implementation/server-chassis/server-chassis-spring/pom.xml to /Users/cgrote/.m2/repository/org/odpi/egeria/server-chassis-spring/2.9-SNAPSHOT/server-chassis-spring-2.9-SNAPSHOT.pom
[INFO] Installing .../egeria/open-metadata-implementation/server-chassis/server-chassis-spring/target/server-chassis-spring-2.9-SNAPSHOT-sources.jar to /Users/cgrote/.m2/repository/org/odpi/egeria/server-chassis-spring/2.9-SNAPSHOT/server-chassis-spring-2.9-SNAPSHOT-sources.jar
[INFO] Installing .../egeria/open-metadata-implementation/server-chassis/server-chassis-spring/target/server-chassis-spring-2.9-SNAPSHOT-javadoc.jar to /Users/cgrote/.m2/repository/org/odpi/egeria/server-chassis-spring/2.9-SNAPSHOT/server-chassis-spring-2.9-SNAPSHOT-javadoc.jar
[INFO]
[INFO] ------------------< org.odpi.egeria:user-interfaces >-------------------
[INFO] Building Open Metadata User Interfaces 2.9-SNAPSHOT [336/383]
[INFO] --------------------------------[ pom ]---------------------------------
... truncated ...
[INFO] OMAS FV and SV Tests ............................... SUCCESS [ 0.260 s]
[INFO] Asset Consumer OMAS FVT ............................ SUCCESS [01:05 min]
[INFO] Asset Manager OMAS FVT ............................. SUCCESS [01:02 min]
[INFO] Asset Owner OMAS FVT ............................... SUCCESS [ 43.237 s]
[INFO] Data Engine OMAS FVT ............................... SUCCESS [02:55 min]
[INFO] Data Manager OMAS FVT .............................. SUCCESS [ 44.278 s]
[INFO] Digital Architecture OMAS FVT ...................... SUCCESS [ 52.186 s]
[INFO] Subject Area OMAS FVT .............................. SUCCESS [02:03 min]
[INFO] Open Metadata Open Types Test Resources ............ SUCCESS [ 0.196 s]
[INFO] Open Types Test generator .......................... SUCCESS [ 3.430 s]
[INFO] Open Types Test execution .......................... SUCCESS [ 37.326 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 26:47 min
[INFO] Finished at: 2021-02-12T18:06:19Z
[INFO] ------------------------------------------------------------------------
Note from the output that the build creates the file .../egeria/open-metadata-implementation/server-chassis/server-chassis-spring/target/server-chassis-spring-{version}.jar.
This is Egeria's server chassis, which you can use just as if you had downloaded it.
3. Configure security
To get an initial environment up-and-running just download the truststore.p12
file from: https://github.com/odpi/egeria/raw/main/truststore.p12.
In addition, set an environment variable STRICT_SSL=false.
Transport-level security for Egeria
The truststore.p12 file provides a local truststore for Java. This allows the self-signed certificate embedded within the server chassis (by default) to be trusted.
Without this trust, interactions with the server chassis (such as the REST calls that are made
through Java to handle interaction between the chassis and the connector) will result in an
SSLHandshakeException.
While this truststore.p12 file allows SSL-encrypted communication, the fact that
it relies on a self-signed certificate means that there is no independent source of trust
in the interactions (which would typically be achieved through an independent Certificate
Authority).
Additional details on TLS for Egeria can be found in the Egeria documentation.
Transport-level security for Apache Atlas
While the truststore.p12 file sets up TLS for Egeria itself, there is also the
communication between the connector and the Apache Atlas environment to consider.
The simplest method to configure this layer of the transport is to disable SSL verification
itself, which you can do by exporting the environment variable STRICT_SSL=false
or passing the argument -Dstrict.ssl=false to the java command when running the
server chassis.
Both of these effectively disable security, however, so for a more robust (albeit also more cumbersome) approach see the further details on other options in the pages below.
Configure Java keystore
With this approach you can tell Java to trust the certificate of one or more systems, even when those systems only have self-signed certificates. This is more secure than entirely disabling SSL verification, as it provides a sort of whitelist for certificates to trust; however, it does not rely on an independent party to verify that trust. Done without proper diligence, there is therefore still some risk in this approach.
Begin by obtaining the self-signed certificate of the Atlas host (from any machine with a JVM: could be the Egeria OMAG Server Platform so long as you're confident it is reaching the correct host):
$ keytool -printcert -sslserver my.atlas.host.com:21000 -rfc > atlas_self_signed.crt
Note that you need to use the correct hostname and port number of your Apache Atlas environment. ("Correct hostname" means the hostname by which that server knows itself -- it should not be an IP address.)
Configure Java keystore
Next, import the certificate into the Java keystore used by the environment. For example, in an Alpine openjdk docker
image, this would be /etc/ssl/certs/java/cacerts:
$ keytool -trustcacerts -keystore /etc/ssl/certs/java/cacerts -storepass changeit -noprompt -alias my.atlas.host.com -importcert -file atlas_self_signed.crt
$ rm atlas_self_signed.crt
- The keystore location should be specified, and it is best to use the one for the environment itself (if a user-specific keystore it will likely depend on the user under which the JVM itself is running whether it is picked up or not).
- The default keystore password in Java is
changeit; obviously replace this with the password for your actual keystore if it has been changed... (Both the-storepassand-nopromptcan be skipped if you would rather be prompted for the password and confirmation of importing the certificate.) - The
-aliasshould be specified and must match the hostname of the Atlas machine itself: if different, the certificate won't match and won't be validated.
Use an independent certificate authority
This approach is generally accepted as the most robust and secure. It relies on a third party (a "certificate authority") to sign the certificates to be used by your environments, and leverages inherent trust in these third parties in order to validate such signed certificates automatically.
However, this option also relies on obtaining (typically at a cost) such certificate signatures, and updating the configuration of your system(s) to leverage these new signed certificates rather than the self-signed certificates they possess by default.
More information on doing this specifically for Apache Atlas can be found in Cloudera's documentation.
After configuring such a signed certificate, you should be able to connect to the Atlas environment without any further configuration of either the connector, OMAG Server Platform, or Java keystores.
4. Start the server chassis
Ensure the 3 files are in the same directory, and run:
$ ls -1
egeria-connector-hadoop-ecosystem-package-{version}-jar-with-dependencies.jar
server-chassis-spring-{version}.jar
truststore.p12
$ java -Dloader.path=. -jar server-chassis-spring-{version}.jar
ODPi Egeria
____ __ ___ ___ ______ _____ ____ _ _ ___
/ __ \ / |/ // | / ____/ / ___/ ___ ____ _ __ ___ ____ / _ \ / / __ / / / _ /__ ____ _ _
/ / / // /|_/ // /| | / / __ \__ \ / _ \ / __/| | / // _ \ / __/ / /_/ // // | / _\ / /_ / | / _// || |
/ /_/ // / / // ___ |/ /_/ / ___/ // __// / | |/ // __// / / __ // // / \ / /_ / _// / // / / / / /
\____//_/ /_//_/ |_|\____/ /____/ \___//_/ |___/ \___//_/ /_/ /_/ \__/\//___//_/ \__//_/ /_/ /_/
:: Powered by Spring Boot (v2.3.3.RELEASE) ::
2021-02-12 20:16:37.243 INFO 53787 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 9443 (https)
2021-02-12 20:16:49.162 INFO 53787 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 9443 (https) with context path ''
Fri Feb 12 20:16:40 GMT 2021 No OMAG servers listed in startup configuration
Fri Feb 12 20:16:49 GMT 2021 OMAG server platform ready for more configuration
Wait for these two final lines of output to appear before proceeding.
Startup options
$ ls -1
egeria-connector-hadoop-ecosystem-package-{version}-jar-with-dependencies.jar
server-chassis-spring-{version}.jar
truststore.p12
$ java -Dloader.path=. -jar server-chassis-spring-{version}.jar
ODPi Egeria
____ __ ___ ___ ______ _____ ____ _ _ ___
/ __ \ / |/ // | / ____/ / ___/ ___ ____ _ __ ___ ____ / _ \ / / __ / / / _ /__ ____ _ _
/ / / // /|_/ // /| | / / __ \__ \ / _ \ / __/| | / // _ \ / __/ / /_/ // // | / _\ / /_ / | / _// || |
/ /_/ // / / // ___ |/ /_/ / ___/ // __// / | |/ // __// / / __ // // / \ / /_ / _// / // / / / / /
\____//_/ /_//_/ |_|\____/ /____/ \___//_/ |___/ \___//_/ /_/ /_/ \__/\//___//_/ \__//_/ /_/ /_/
:: Powered by Spring Boot (v2.3.3.RELEASE) ::
2021-02-12 20:16:37.243 INFO 53787 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 9443 (https)
2021-02-12 20:16:49.162 INFO 53787 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 9443 (https) with context path ''
Fri Feb 12 20:16:40 GMT 2021 No OMAG servers listed in startup configuration
Fri Feb 12 20:16:49 GMT 2021 OMAG server platform ready for more configuration
Of course, the {version} should be replaced by whatever version of the server
chassis you are running.
The -Dloader.path=. indicates that the server chassis should look for any
connectors in the current directory. If you wanted to instead place the connector in a
separate location (for example /lib), you could change the . to
point to that other location (-Dloader.path=/lib).
Startup options
$ ls -1
egeria-connector-hadoop-ecosystem-package-{version}-jar-with-dependencies.jar
server-chassis-spring-{version}.jar
truststore.p12
$ java -Dloader.path=. -jar server-chassis-spring-{version}.jar
ODPi Egeria
____ __ ___ ___ ______ _____ ____ _ _ ___
/ __ \ / |/ // | / ____/ / ___/ ___ ____ _ __ ___ ____ / _ \ / / __ / / / _ /__ ____ _ _
/ / / // /|_/ // /| | / / __ \__ \ / _ \ / __/| | / // _ \ / __/ / /_/ // // | / _\ / /_ / | / _// || |
/ /_/ // / / // ___ |/ /_/ / ___/ // __// / | |/ // __// / / __ // // / \ / /_ / _// / // / / / / /
\____//_/ /_//_/ |_|\____/ /____/ \___//_/ |___/ \___//_/ /_/ /_/ \__/\//___//_/ \__//_/ /_/ /_/
:: Powered by Spring Boot (v2.3.3.RELEASE) ::
2021-02-12 20:16:37.243 INFO 53787 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 9443 (https)
2021-02-12 20:16:49.162 INFO 53787 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 9443 (https) with context path ''
Fri Feb 12 20:16:40 GMT 2021 No OMAG servers listed in startup configuration
Fri Feb 12 20:16:49 GMT 2021 OMAG server platform ready for more configuration
These final two lines of output indicate that the server chassis has completed starting up and is now ready to be configured.
Any attempt to configure the server chassis before these lines are output is likely to result in failures.
5. Configure Atlas connector
Make these API calls to the admin services of the running server chassis to configure the Apache Atlas connector:
$ curl -k -X POST -H "Content-Type: application/json" --data '{"producer":{"bootstrap.servers":"localhost:9092"},"consumer":{"bootstrap.servers":"localhost:9092"}}' "https://localhost:9443/open-metadata/admin-services/users/admin/servers/atlas/event-bus?connectorProvider=org.odpi.openmetadata.adapters.eventbus.topic.kafka.KafkaOpenMetadataTopicProvider&topicURLRoot=OMRSTopic"
{"class":"VoidResponse","relatedHTTPCode":200}
$ curl -k -X POST "https://localhost:9443/open-metadata/admin-services/users/admin/servers/atlas/cohorts/mycohort"
{"class":"VoidResponse","relatedHTTPCode":200}
$ curl -k -X POST -H "Content-Type: application/json" --data '{"class":"Connection","connectorType":{"class":"ConnectorType","connectorProviderClassName":"org.odpi.egeria.connectors.apache.atlas.repositoryconnector.ApacheAtlasOMRSRepositoryConnectorProvider"},"endpoint":{"class":"Endpoint","address":"atlas:21000","protocol":"http"},"userId":"admin","clearPassword":"admin"}' "https://localhost:9443/open-metadata/admin-services/users/admin/servers/atlas/local-repository/mode/repository-proxy/connection"
{"class":"VoidResponse","relatedHTTPCode":200}
(Remember to replace the hostnames and credentials, in particular for your Apache Atlas environment.)
Configure event bus
This first call to the API will configure the event bus of Egeria.
$ curl -k -X POST -H "Content-Type: application/json" --data '{"producer":{"bootstrap.servers":"localhost:9092"},"consumer":{"bootstrap.servers":"localhost:9092"}}' "https://localhost:9443/open-metadata/admin-services/users/admin/servers/atlas/event-bus?connectorProvider=org.odpi.openmetadata.adapters.eventbus.topic.kafka.KafkaOpenMetadataTopicProvider&topicURLRoot=OMRSTopic"
{"class":"VoidResponse","relatedHTTPCode":200}
The event bus is how Egeria coordinates communication amongst its various servers and repositories: for example, ensuring that any new type definitions are registered with each repository capable of handling them, notifying other repositories when the metadata in one repository changes, etc.
https://localhost:9443/open-metadata/admin-services/users/admin/servers/atlas/event-bus?\
connectorProvider=org.odpi.openmetadata.adapters.eventbus.topic.kafka.KafkaOpenMetadataTopicProvider\
&topicURLRoot=OMRSTopic
The URL parameter connectorProvider defines the type of event bus to use (in
this case Apache Kafka), while the topicURLRoot gives a prefix name for the Kafka topic.
{
"producer": {
"bootstrap.servers": "localhost:9092"
},
"consumer": {
"bootstrap.servers": "localhost:9092"
}
}
The JSON payload gives details about how to connect to Apache Kafka, in this case assuming it is running on local machine (localhost) on its default port (9092).
Configure cohort
This next call configures the cohort with which this Egeria server chassis should register.
$ curl -k -X POST "https://localhost:9443/open-metadata/admin-services/users/admin/servers/atlas/cohorts/mycohort"
{"class":"VoidResponse","relatedHTTPCode":200}
The cohort is a network of interacting metadata servers, which could be some subset of the overall Egeria landscape.
https://localhost:9443/open-metadata/admin-services/users/admin/servers/atlas/cohorts/\
mycohort
We only need to supply a name for the cohort, which forms part of the URL itself: in this
example using mycohort.
Configure Apache Atlas connector
This final call to the API configures the behavior of the Apache Atlas proxy itself.
$ curl -k -X POST -H "Content-Type: application/json" --data '{"class":"Connection","connectorType":{"class":"ConnectorType","connectorProviderClassName":"org.odpi.egeria.connectors.apache.atlas.repositoryconnector.ApacheAtlasOMRSRepositoryConnectorProvider"},"endpoint":{"class":"Endpoint","address":"atlas:21000","protocol":"http"},"userId":"admin","clearPassword":"admin"}' "https://localhost:9443/open-metadata/admin-services/users/admin/servers/atlas/local-repository/mode/repository-proxy/connection"
{"class":"VoidResponse","relatedHTTPCode":200}
The URL to which we post indicates that we will use the Egeria server chassis's built-in repository proxy capability to access the Apache Atlas repository connector.
https://localhost:9443/open-metadata/admin-services/users/admin/servers/atlas/local-repository/\
mode/repository-proxy/connection
The JSON payload's contents define how this proxy itself should be configured: specifically,
which Java class should be used for the repository connection. Here we can see the payload
refers to the ApacheAtlasOMRSRepositoryConnectorProvider, which therefore tells the
plugin to use this class -- specific to the Atlas repository connector -- in order to configure
its repository connection.
{
"class": "Connection",
"connectorType": {
"class": "ConnectorType",
"connectorProviderClassName": "org.odpi.egeria.connectors.apache.atlas.repositoryconnector.ApacheAtlasOMRSRepositoryConnectorProvider"
}
}
Configure Apache Atlas connector
{
"class": "Connection",
"connectorType": {
"class": "ConnectorType",
"connectorProviderClassName": "org.odpi.egeria.connectors.apache.atlas.repositoryconnector.ApacheAtlasOMRSRepositoryConnectorProvider"
},
"endpoint": {
"class": "Endpoint",
"address": "atlas:21000",
"protocol": "http"
},
"userId": "admin",
"clearPassword": "admin"
}
Details about how to connect to the Apache Atlas environment must also be provided in the connection's configuration: the endpoint details covering the hostname and port of the environment, and the username and password through which we can access its REST APIs.
6. Start the connector
Make the following API call to the admin services of the running server chassis to start the configured Apache Atlas connector:
$ curl -k -X POST "https://localhost:9443/open-metadata/admin-services/users/admin/servers/atlas/instance"
{"class":"SuccessMessageResponse","relatedHTTPCode":200,"successMessage":"Thu Mar 11 12:50:46 GMT 2021 atlas is running the following services: [Open Metadata Repository Services (OMRS)]"}
It may take 10-15 seconds to complete, but the example response above indicates that the connector instance is now running.
Sample log output
Back in the console where the server chassis is running, you should see the audit log printing out a large amount of information as the startup is running. Most of this is related to the registration of type definition details with the repository: this would also be where you should see any errors if the connector is unable to start successfully.
This final line indicates that the server is now running.
Explore
You should now be able to start accessing metadata in Apache Atlas via Egeria. Without any additional dependencies, this is most easily illustrated using direct REST API interactions with the server chassis's repository services. We suggest the following steps:
- Run a query for an Egeria type (e.g.
Databases) - See the resulting data from Apache Atlas in an Egeria representation
1. Query via Egeria
Run a query for all databases in Apache Atlas whose name is "default", by posting a payload like the following:
{
"class": "EntityPropertyFindRequest",
"typeGUID": "0921c83f-b2db-4086-a52c-0d10e52ca078",
"pageSize": 10,
"matchCriteria": "ALL",
"matchProperties": {
"class": "InstanceProperties",
"instanceProperties": {
"name": {
"class": "PrimitivePropertyValue",
"instancePropertyCategory": "PRIMITIVE",
"primitiveDefCategory": "OM_PRIMITIVE_TYPE_STRING",
"primitiveValue": "\\Qdefault\\E"
}
}
}
}
To this API endpoint:/servers/atlas/open-metadata/repository-services/users/admin/instances/entities/by-property
$ curl -k -X POST -H "Content-Type: application/json" --data '{"class":"EntityPropertyFindRequest","typeGUID":"0921c83f-b2db-4086-a52c-0d10e52ca078","pageSize":10,"matchCriteria":"ALL","matchProperties":{"class":"InstanceProperties","instanceProperties":{"name":{"class":"PrimitivePropertyValue","instancePropertyCategory":"PRIMITIVE","primitiveDefCategory":"OM_PRIMITIVE_TYPE_STRING","primitiveValue":"\\Qdefault\\E"}}}}' "https://localhost:9443/servers/atlas/open-metadata/repository-services/users/admin/instances/entities/by-property"
{"class":"EntityListResponse","relatedHTTPCode":200,"offset":0,"pageSize":10,"entities":[{"class":"EntityDetail","headerVersion":1,"type":{"class":"InstanceType","typeDefCategory":"ENTITY_DEF","typeDefGUID":"0921c83f-b2db-4086-a52c-0d10e52ca078","typeDefName":"Database","typeDefVersion":1,"typeDefDescription":"A data store containing relational data.","typeDefSuperTypes":[{"headerVersion":1,"guid":"30756d0b-362b-4bfa-a0de-fce6a8f47b47","name":"DataStore","status":"ACTIVE_TYPEDEF"},{"headerVersion":1,"guid":"896d14c2-7522-4f6c-8519-757711943fe6","name":"Asset","status":"ACTIVE_TYPEDEF"},{"headerVersion":1,"guid":"a32316b8-dc8c-48c5-b12b-71c1b2a080bf","name":"Referenceable","status":"ACTIVE_TYPEDEF"}],"validInstanceProperties":["qualifiedName","additionalProperties","name","description","owner","ownerType","zoneMembership","latestChange","createTime","modifiedTime","type","version","instance","importedFrom"]},"instanceProvenanceType":"LOCAL_COHORT","metadataCollectionId":"78317da1-eeb4-4999-b2d4-46fa5e69d5f3","metadataCollectionName":"atlas","createdBy":"atlas","updatedBy":"admin","createTime":"2018-11-29T19:01:39.200+00:00","updateTime":"2021-05-28T09:21:08.727+00:00","version":0,"guid":"97a74eed-9cd5-4168-8974-1c256d149217","instanceURL":"http://atlas:21000/api/atlas/v2/entity/guid/97a74eed-9cd5-4168-8974-1c256d149217","properties":{"class":"InstanceProperties","instanceProperties":{"owner":{"class":"PrimitivePropertyValue","instancePropertyCategory":"PRIMITIVE","typeGUID":"b34a64b9-554a-42b1-8f8a-7d5c2339f9c4","typeName":"string","primitiveDefCategory":"OM_PRIMITIVE_TYPE_STRING","primitiveValue":"atlas"},"modifiedTime":{"class":"PrimitivePropertyValue","instancePropertyCategory":"PRIMITIVE","primitiveDefCategory":"OM_PRIMITIVE_TYPE_DATE","primitiveValue":1543518095533},"qualifiedName":{"class":"PrimitivePropertyValue","instancePropertyCategory":"PRIMITIVE","typeGUID":"b34a64b9-554a-42b1-8f8a-7d5c2339f9c4","typeName":"string","primitiveDefCategory":"OM_PRIMITIVE_TYPE_STRING","primitiveValue":"default@Sandbox"},"name":{"class":"PrimitivePropertyValue","instancePropertyCategory":"PRIMITIVE","typeGUID":"b34a64b9-554a-42b1-8f8a-7d5c2339f9c4","typeName":"string","primitiveDefCategory":"OM_PRIMITIVE_TYPE_STRING","primitiveValue":"default"},"description":{"class":"PrimitivePropertyValue","instancePropertyCategory":"PRIMITIVE","typeGUID":"b34a64b9-554a-42b1-8f8a-7d5c2339f9c4","typeName":"string","primitiveDefCategory":"OM_PRIMITIVE_TYPE_STRING","primitiveValue":"default"},"additionalProperties":{"class":"MapPropertyValue","instancePropertyCategory":"MAP","mapValues":{"class":"InstanceProperties","instanceProperties":{"createTime":{"class":"PrimitivePropertyValue","instancePropertyCategory":"PRIMITIVE","typeGUID":"b34a64b9-554a-42b1-8f8a-7d5c2339f9c4","typeName":"string","primitiveDefCategory":"OM_PRIMITIVE_TYPE_STRING","primitiveValue":"0"},"clusterName":{"class":"PrimitivePropertyValue","instancePropertyCategory":"PRIMITIVE","typeGUID":"b34a64b9-554a-42b1-8f8a-7d5c2339f9c4","typeName":"string","primitiveDefCategory":"OM_PRIMITIVE_TYPE_STRING","primitiveValue":"Sandbox"}},"propertyCount":2,"propertyNames":["createTime","clusterName"]},"mapElementCount":2}},"propertyCount":6,"propertyNames":["owner","modifiedTime","qualifiedName","name","description","additionalProperties"]},"status":"ACTIVE"}]}
2. Review query results
Note that the results are returned in Egeria's type structure, as Databases.
These are the same results that Egeria would see from Apache Atlas when running a federated query out
to a cohort that is running the Apache Atlas proxy connector, and would therefore be able to combine
with other metadata from other members of the cohort.
{
"class": "EntityListResponse",
"relatedHTTPCode": 200,
"offset": 0,
"pageSize": 10,
"entities":
[
{
"class": "EntityDetail",
"headerVersion": 1,
"type":
{
"class": "InstanceType",
"typeDefCategory": "ENTITY_DEF",
"typeDefGUID": "0921c83f-b2db-4086-a52c-0d10e52ca078",
"typeDefName": "Database",
"typeDefVersion": 1,
"typeDefDescription": "A data store containing relational data.",
"typeDefSuperTypes":
[
{
"headerVersion": 1,
"guid": "30756d0b-362b-4bfa-a0de-fce6a8f47b47",
"name": "DataStore",
"status": "ACTIVE_TYPEDEF"
},
{
"headerVersion": 1,
"guid": "896d14c2-7522-4f6c-8519-757711943fe6",
"name": "Asset",
"status": "ACTIVE_TYPEDEF"
},
{
"headerVersion": 1,
"guid": "a32316b8-dc8c-48c5-b12b-71c1b2a080bf",
"name": "Referenceable",
"status": "ACTIVE_TYPEDEF"
}
],
"validInstanceProperties":
[
"qualifiedName",
"additionalProperties",
"name",
"description",
"owner",
"ownerType",
"zoneMembership",
"latestChange",
"createTime",
"modifiedTime",
"type",
"version",
"instance",
"importedFrom"
]
},
"instanceProvenanceType": "LOCAL_COHORT",
"metadataCollectionId": "78317da1-eeb4-4999-b2d4-46fa5e69d5f3",
"metadataCollectionName": "atlas",
"createdBy": "atlas",
"updatedBy": "admin",
"createTime": "2018-11-29T19:01:39.200+00:00",
"updateTime": "2021-05-28T09:21:08.727+00:00",
"version": 0,
"guid": "97a74eed-9cd5-4168-8974-1c256d149217",
"instanceURL": "http://atlas:21000/api/atlas/v2/entity/guid/97a74eed-9cd5-4168-8974-1c256d149217",
"properties":
{
"class": "InstanceProperties",
"instanceProperties":
{
"owner":
{
"class": "PrimitivePropertyValue",
"instancePropertyCategory": "PRIMITIVE",
"typeGUID": "b34a64b9-554a-42b1-8f8a-7d5c2339f9c4",
"typeName": "string",
"primitiveDefCategory": "OM_PRIMITIVE_TYPE_STRING",
"primitiveValue": "atlas"
},
"modifiedTime":
{
"class": "PrimitivePropertyValue",
"instancePropertyCategory": "PRIMITIVE",
"primitiveDefCategory": "OM_PRIMITIVE_TYPE_DATE",
"primitiveValue": 1543518095533
},
"qualifiedName":
{
"class": "PrimitivePropertyValue",
"instancePropertyCategory": "PRIMITIVE",
"typeGUID": "b34a64b9-554a-42b1-8f8a-7d5c2339f9c4",
"typeName": "string",
"primitiveDefCategory": "OM_PRIMITIVE_TYPE_STRING",
"primitiveValue": "default@Sandbox"
},
"name":
{
"class": "PrimitivePropertyValue",
"instancePropertyCategory": "PRIMITIVE",
"typeGUID": "b34a64b9-554a-42b1-8f8a-7d5c2339f9c4",
"typeName": "string",
"primitiveDefCategory": "OM_PRIMITIVE_TYPE_STRING",
"primitiveValue": "default"
},
"description":
{
"class": "PrimitivePropertyValue",
"instancePropertyCategory": "PRIMITIVE",
"typeGUID": "b34a64b9-554a-42b1-8f8a-7d5c2339f9c4",
"typeName": "string",
"primitiveDefCategory": "OM_PRIMITIVE_TYPE_STRING",
"primitiveValue": "default"
},
"additionalProperties":
{
"class": "MapPropertyValue",
"instancePropertyCategory": "MAP",
"mapValues":
{
"class": "InstanceProperties",
"instanceProperties":
{
"createTime":
{
"class": "PrimitivePropertyValue",
"instancePropertyCategory": "PRIMITIVE",
"typeGUID": "b34a64b9-554a-42b1-8f8a-7d5c2339f9c4",
"typeName": "string",
"primitiveDefCategory": "OM_PRIMITIVE_TYPE_STRING",
"primitiveValue": "0"
},
"clusterName":
{
"class": "PrimitivePropertyValue",
"instancePropertyCategory": "PRIMITIVE",
"typeGUID": "b34a64b9-554a-42b1-8f8a-7d5c2339f9c4",
"typeName": "string",
"primitiveDefCategory": "OM_PRIMITIVE_TYPE_STRING",
"primitiveValue": "Sandbox"
}
},
"propertyCount": 2,
"propertyNames":
[
"createTime",
"clusterName"
]
},
"mapElementCount": 2
}
},
"propertyCount": 6,
"propertyNames":
[
"owner",
"modifiedTime",
"qualifiedName",
"name",
"description",
"additionalProperties"
]
},
"status": "ACTIVE"
}
]
}