API Plugin for Apache Nifi
This documentation is archived.
All new implementation please use the following documentation link: http://help.lingk.io/en/articles/128-on-premise-adapter-powered-by-apache-nifi-installation-guide
Lingk Adapter
The Lingk Adapter is used to connect on-premise SIS/ERP/databases with Lingk integrations solutions using secure REST APIs. The Lingk Adapter contains an API dashboard that supports configuration of desired data flow pipelines required for each data scenario. This document intends to contains complete information required to configure the Lingk Adapter.
This guide walks you through:
Installing the Lingk Adapter.
Configuring the Lingk Adapter security, network settings, and back-end access for internal configuration.
Configuring the Lingk Adapter to externally connect with the LingkSync Speed Wizard.
Configuring the APIs served by the Lingk Adapter.
By the end of this configuration you should have a Lingk Adapter API endpoint to provide to your application administrator to complete LingkSync configuration.
If you are a Colleague by Ellucian users, please visit Colleague Implementation notes.
Table of Contents
Intended Audience
Prior knowledge of Linux system administration is required to execute the procedures mentioned in this document. Docker knowledge is desired but not mandatory.
Document should be used by:
Lingk Adapter system administrator
Client IT and SIS personnel
Architecture Overview
Implementation Procedure
Pre-requisites
Server Requirement
Minimum system requirement for running the Lingk Adapter is as follows:
2 CPU Cores
4 GB Memory
At least 10 GB of hard disk storage
Operating System Requirement
The Lingk Adapter is tested to be working on Ubuntu 16.04 (Xenial Xerus) or later and Windows 10 (Insider Edition), but should be able to work on any operating system as long as the Docker requirements are fulfilled.
For Linux
Lingk recommends to use a Linux based distribution (Ubuntu 16.x server) and proceeding sections have instructions with Ubuntu 16.04 as host operating system.
For Windows
For Windows installation, see our separate Docker on Windows Installation Guide for Lingk Adapter.
For AWS environments, the Amazon Linux AMI is a recommended starting point.
Docker Requirement
The Lingk Adapter is built using Docker Server version 1.12.6 but should work on later releases as well.
Browser Requirements
Lingk Adapter management console has been tested on below browsers;
Internet Explorer 9+
Mozilla FireFox 24+
Google Chrome 36+
Safari 8
Installation Steps
Estimated Time: 15-20 minutes
Lingk uses Git and Amazon Web Services (AWS) to host adapter container repositories. Following is three step process to install Lingk Adapter on client premises.
When you have completed this step, you will be able to:
Run the adapter in a local environment
Run the SQL Dashboard
Configure Banner SQL Queries
Additional SSL and DNS configuration is needed to:
Run the adapter in a production environment
Provide an public endpoint to your LingkSync app administrator
Before you begin...
Git and Amazon Bucket Locations
See below for information on where to download the Lingk Adapter installation components using Git and AWS.
Firewall Ports
Ports 22, 80, 8083, and 9000 need to be open internally to your network so that your users can configure and maintain the Lingk Adapter. Port 3000 needs to be opened externally to your users (for running LingkSync wizards) as well as Lingk's Transformer Engine. Lingk’s Transformer Engine runs on Amazon AWS (unless a hybrid configuration is used). Amazon hosts a list of IP ranges for their services here: https://ip-ranges.amazonaws.com/ip-ranges.json, requests from our transformer will come from the ranges listed for region: us-east-1 and service: ec2.
Getting Started
If you are installing on a Windows machine, see the separate document, Docker on Windows Installation Guide for information on installing Docker.
Step 1 - Install Docker and docker-compose
For Linux
Install docker as per operating system based instructions from here;
https://docs.docker.com/engine/installation/#platform-support-matrix
Verify that docker has been installed
To create additional ldap users, host machine should also have ldap-utils installed. For instance, below command should install ldap-utils on ubuntu flavors supporting aptitude
or
For Windows
Please see the Docker on Windows Installation Guide.
Step 2 - Install Lingk Adapter
For Linux
As a prerequisite, install docker-compose and docker on a server where installation of LingkSync is desired. Afterwards, follow below steps to deploy the latest version of LingkSync.
Copy and paste the command below into your terminal to obtain the GIT repository for the Lingk Adapter installation process.
After downloading the git repository, you will need to change permissions for the script
On Linux, during execution, your script will ask for the location of the LingkSync adapter file and the Lingk LDAP file.
Get the URLs for the latest versions from Resources page in the Lingk console.
For Windows
As a prerequisite, install Docker for Windows on a Windows machine where installation of LingkSync is desired.
Copy and paste the command below into your terminal to obtain the GIT repository for the Lingk Adapter installation process.
After downloading the git repository, you will need to change permissions for __the script
If installing on Windows, you will need to manually create a folder named "downloaded" within the lingkadapter-docker folder on your machine:
Now download the latest versions of the LingkSync adapter and Lingk LDAP files from the Resources page in the Lingk console into the "downloaded" folder. Once downloaded, rename the files as:
Execute the script
All Platforms
Verify your installation
Step 3: Open Lingk Adapter in your Browser
After completing above steps to install Lingk’s adapter, you can access it using a web browser from https://YOURMACHINEIP:8083/lingk/. In order to change the port, simply replace 8083 with the desired port. NOTE: If using Windows, you will access the REST Adapter on the local machine via https://YOURDOCKERIP/lingk/ (see the Docker on Windows Installation Guide for details).
To login, use the default administrator credentials of:
To change the password and/or create more users, see Create Users in LDAP.
IMPORTANT: Reset the default Lingk Adapter Administrator password. Please see the instruction later in this document for Managing Users. |
Configuring Lingk APIs
About the Lingk Adapter APIs
Lingk Adapter APIs enable on-premise datasets to be available via RESTful APIs to wizards, recipes and connectors through the Lingk Transformer Engine. Lingk Adapter APIs can be configured using the API Dashboard that ships with the Lingk Adapter. On-premise dataset can be accessed by a SQL (i.e. query, view or store procedure) or delimited file. Each dataset is provided an API endpoint by the Lingk Adapter. Additionally, some on-premise systems may have unique data access requirements.
Lingk Adapter APIs are secured between the Lingk Adapter and Lingk Transformer Engine using JWT security and provide per message signatures.
Lingk Adapter APIs are used solely for the purposes of the Lingk connectors and the Lingk Platform. Enabling these APIs to be reused for other purposes in the planning stages. Please submit a request to support@lingk.io, if you have business or technical requirements that could reuse these APIs.
Constructing Data Sets from Queries, Flat-files and Operations
Traditionally, in point to point integration, how you build datasets and the names of the output columns is not that important. With the Lingk Adapter, there are benefits to considering the names and values of the data that is returned.
To improve the integration experience, Speed Wizards can provide automapping for commonly named fields.
With a small amount of effort you can take common attributes in your datasets and enable them to be automatically aligned to Salesforce objects by aligning names and values with the list below. Additionally, your datasets will be easier to understand in future integrations.
Default Speed Wizard mapping with Salesforce by resource and field name:
Lingk Adapter API Name | Field Name | LingkSync Salesforce Mapping |
Applicants | ExternalId | Contact -> Lingk External Id |
Applicants | FirstName | Contact -> First Name |
Applicants | LastName | Contact -> Last Name |
Applicants | Gender | Contact -> Gender |
Institutions | Name | Account: Educational Institution -> Account Name |
Departments | Name | Account: University Department -> Account Name |
Courses | ExternalId | Course -> Lingk External Id |
Courses | Description | Course -> Description |
Courses | Credits | Course -> Credit Hours |
Courses | CourseTitle | Course -> Course Name |
Terms | ExternalId | Term -> Lingk External Id |
Terms | Name | Term -> Term Name |
Course Sections | ExternalId | Course Offering -> Lingk External Id |
Course Sections | StartDate | Course Offering -> Start Date |
Course Sections | EndDate | Course Offering -> End Date |
Course Sections | Capacity | Course Offering -> Capacity |
Unlike other tools, this isn’t about just source and target name matching. LingkSync recognizes fields that have names and values associated with the Common Education Data Standard (CEDS). You can learn more about CEDS at https://ceds.ed.gov.
Configuration Options for Data Backends for APIs
The Lingk Adapter has 2 modes of operation:
Direct SQL queries: for on-demand querying against live data
Flat file queries: for on-demand querying against file-based snapshots
Under both of these modes, the data is accessible via REST APIs to the LingkSync app. Flat file query mode would remove any impact on SIS operations and utilize a file to be dropped to a director available to a local on-premise environment.
Direct SQL Query Operation | Flat File Query Operation | |
Lingk API Secret Configuration | Yes | Yes |
On-Demand Database Querying | Yes | No |
On-Demand File Querying | No | Yes |
API Tool Configuration | SQL | File |
One-time API Configurations
Complete all the one-time configurations before configuring individual APIs.
Getting Your Lingk Adapter API Credentials
You will receive your Lingk Adapter API secret from the provider configuration under Environments in your workspaces.
Configuring Your Data Flow
1. Download the Lingk Adapter Flow template for your database from the Resources page.
4. Double-click into the process group
Note: All database templates support flat files endpoints.
6. Configure your controller services Click on the gear for your processor group. A. Enable HTTP Services
B. Enable HTTPS Services Use lingkadapter123 as the password for TrustStore and KeyStore
C. Counfigre the database connection.
7. Start all processors in the flow
Database Connections
If you are configuring the Lingk Adapter to connect directly to your database using queries, views or stored procedures, you will need to configure your database connections.
Configure your database driver
Oracle (for PeopleSoft Campus Solutions and Banner by Ellucian)
Postgres
Microsoft SQL Server
Other JDBC providers
Configuring Database Connectivity
Here are the instructions for an Oracle database.
1. Open the Main Flow configuration dialog as shown below.
2. Click on the pencil bolt icon for DBCPConnectionPool Controller Service
Note: You may also need to enable the StandardHttpContextMapPort3000 controller service, if it is disabled.
3. Enter your JDBC connection string, username and password into the Lingk Adapter processor Oracle Connection Strings Example Oracle connection strings (Service Name)
Example connection string (SID)_
For more information on Oracle Connection strings go to: https://docs.oracle.com/cd/B28359_01/java.111/b31224/jdbcthin.htm
Postgres Connection Strings
Microsoft SQL Server Connection Strings
Example connection string (database user)
Example connection string (domain user)
4. Click Apply.
5. Click on the Lightning icon for the same Controller service
6. Enable the Controller Service to load the changes into associated processors
7. Press the "Play" button on the main flow to start all configured processors in the flow.
Now you are able to start configuring which datasets you want to expose through the Lingk Adapter.
Configuring An API with an SQL Database Query Backend
We will walk through the configuration of the Lingk Adapter for managing APIs used in the LingkSync product.
Open the API dashboard in separate browser tab with the following URL (if configured for Private IP) https://<machineip|domain>:9000/dashboard/
Login using your Lingk Adapter credentials
[add login screen for dashboard]
Delete any preset APIs from the list that do not align to your data scenario and add any APIs to the list to align to your data scenario.
Test any SQL queries and views using the API Dashboard to generate API metadata. Use the GET HTTP verb for pulling data from Oracle. See the section titled "Constructing Data Sets from Queries, Flat-files and Operations" on constructing user-friendly and reusable endpoints.
Once all endpoints are complete, configure the API endpoint in the Lingk HQ. Find your provider under your workspace's environment navigation.
Configure API Security for Lingk Transformer Engine
1. Stop the Client Secret processor to enable updates
2. Click Configure on the processor
3. Create a new Property in the Lingk Adapter processor for your LingkSync using the [+] icon.
4. Enter your Lingk API Secret into the Lingk Adapter processor.
5. Apply Changes
6. Restart the Client Secret processor
Configuring An API with a File-based Query Backend
Open the API dashboard in separate browser tab with the following URL (if configured for Private IP) https://<machineip|domain>:9000/dashboard/
Create a new API and specify the Endpoint Type of "File"
Default File Drop Directory
The default directory for storing CSV files to serve as an API is /opt/nifi-1.x.x/extras/files/csv. You can upload a CSV file onto your Lingk Adapter instance with the following docker command:
docker cp yourFile.csv {DOCKER_TAG}:/opt/nifi-1.x.x/extras/files/csv/yourFile.csv
You can target the output of scheduled processes to this directory and keep the file name the same to keep your files updated.
For network drive support, please contact Lingk support at support@lingk.io .
Adding a File-based API Endpoint
Specify the names, files and fields using the File Tool to configure the API endpoint. See the section titled "Constructing Data Sets from Queries, Flat-files and Operations" on constructing user-friendly and reusable endpoints.
Test your file-based endpoint with "Preview Data"
Click "Save" when you are finished with the endpoint
Additional Installation Options
Manage Users in LDAP
IMPORTANT: Reset both the default Administrator password. |
By default a user ‘admin (password: password)’ has been assigned administrator privileges and created automatically in Lingk ldap. Admin user can assign rights to more users that are created in Lingk’s LDAP.
In order to create more users in Lingk LDAP, update the file named conf/allinone.ldif in the lingkadapter-ldap docker container with below data for each user replacing <username>, <username@lingk.io> and <anypassword> fields with relevant information. Ideally any two users records should be separated by a blank line.
once the file is saved on disk, below command should create all users mentioned in ldif file;
ldapadd -h localhost -p 389 -c -x -D cn=admin,dc=lingk,dc=io -W -f users.ldif
In order to change password for a user (for instance below procedure can be applied to change admin password)
ldappasswd -h localhost -p 389 -x -D "cn=admin,ou=people,dc=lingk,dc=io" -W -A -S
ldap server asks response to following prompts;
SSL Certificate Generation
Note: This step is not required if SSL Private Mode is followed. For SSL Dedicated Domain Mode, this section is mandatory.
Lingk's recommended approach is to use certificates from a trusted Certificate Authority (CA) and Lingk uses http://www.letsencrypt.org as CA. Below procedure is applicable if client intends to use their own certificates.
Lets assume that client wants to generate certificates for subdomain adapter.yourwebsitedomain.com to host lingkadapter. Following are a set of steps that need to be carried out for certificate generation.
Step 1: Copy utility script repository on server that is hosting the domain/subdomain
curl --silent [https://raw.githubusercontent.com/srvrco/getssl/master/getssl](https://raw.githubusercontent.com/srvrco/getssl/master/getssl) > getssl ;
chmod 700 getssl
Step 2: Generate certificate configuration files
./getssl -c adapter.yourwebsitedomain.com
Step 3: Edit the getssl.cfg file
Changed the ACCOUNT_EMAIL directive to actual e-mail address
/root/.getssl/getssl.cfg
Change Server location from ‘staging’ to production or else it will not be trusted
The staging server is best for testing (hence set as default) CA="https://acme-staging.api.letsencrypt.org"
This server issues full certificates, however has rate limits CA="https://acme-v01.api.letsencrypt.org"
Step 3a: Edit the getssl.cfg file for the domain
Comment the SANS= directive, because we will have exactly 1 domain.
/root/.getssl/yourwebsitedomain.com/getssl.cfg
Specify ACL directive to:
ACL=('/some/directoryAsRoot/.well-known/acme-challenge')
Un-comment following directives:
Save this file.
Step 4: Start a Small Server Thread (Temporary)
Create a separate terminal window for this step to start a process that will be used for subsequent steps.
For ACL generation, we start a small server that letsencrypt will use to verify whether we are on the server hosting the lingkadapter
Note: If python is not found, in ubuntu you can install python-minimal using below command;
$ apt install python-minimal
Once this has been started, let this command run on this terminal and continue from next step in another terminal window.
Step 5: Generate SSL Certificates for the domain
Run the following command to generate SSL Certificates for the domain.
$ getssl adapter.yourwebsitedomain.com
Certificates will be generated in location /etc/ssl/. Following two files will be of significance
adapter.yourwebsitedomain.com_chain.pem
adapter.yourwebsitedomain.com.pem
Step 6: Stop Small Server Thread
Once Step 5 is executed successful and certificates have been verified, open the terminal that was left in Step 4 and press 'Ctrl+c' to terminate the server thread. This terminal window is no longer needed.
Step 7: Generate JAVA Keystore and formats as desired by Adapter
Lingk Adapter has embedded Java-based server that needs Java Keystore (JKS). In order to generate keystore.jks from the generated certificates in the above steps 1-6, follow these two steps.
$ keytool -import -file adapter.yourwebsitedomain.com_chain.pem -alias cacert -keystore truststore.jks -storepass lingkadapter123
Convert the key to pkcs12 format.
$ openssl pkcs12 -export -inkey adapter.yourwebsitedomain.com.pem -in adapter.yourwebsitedomain.com.pem -out cert_key.p12 use password: lingkadapter123
The two generated files (i.e. keystore.jks and cert_key.p12) can be used as newly generated certificates. Place these two files under folder ‘certs’.
Changing the Default Mapped Ports
When you are in an environment that has limited external ports, you may need to change the ports mapped in your Lingk Adapter. The instructions below will help you map the API port to the standard HTTPS port. You still may need to use localhost on the server for all configuration of the APIs with the solution below.
There are two solutions to exposing your API port (port 3000) externally on a default port (443) and you can choose which one is most expedient for you.
A. Delete the Lingk Adapter container and recreate it with this command:
B. Modify the ports on your existing Lingk Adapter container per the following:
https://mybrainimage.wordpress.com/2017/02/05/docker-change-port-mapping-for-an-existing-container/
Event Based Integration Lingk Adapter to the Lingk Platform
Refer to the "Lingk Adapter for Event-Based Integration Guide"
Upgrade the LingkSync Adapter to the Latest Version
Refer to the "Lingk Adapter Upgrade Guide"
Resources
The Lingk Adapter Distribution of Apache Nifi
The Lingk Adapter is a higher education distribution of the Apache Nifi project. If you would like to learn more about Apache Nifi concepts please go to:
https://nifi.apache.org/docs/nifi-docs/html/overview.html
Last updated
