> For the complete documentation index, see [llms.txt](https://developers.lingk.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developers.lingk.io/lingk-adapter.md). # API Plugin for Apache Nifi ## This documentation is archived. All new implementation please use the following documentation link: ## Lingk Adapter The Lingk Adapter is used to connect on-premise SIS/ERP/databases with Lingk integrations solutions using secure REST APIs. The Lingk Adapter contains an API dashboard that supports configuration of desired data flow pipelines required for each data scenario. This document intends to contains complete information required to configure the Lingk Adapter. This guide walks you through: 1. Installing the Lingk Adapter. 2. Configuring the Lingk Adapter security, network settings, and back-end access for internal configuration. 3. Configuring the Lingk Adapter to externally connect with the LingkSync Speed Wizard. 4. Configuring the APIs served by the Lingk Adapter. By the end of this configuration you should have a Lingk Adapter API endpoint to provide to your application administrator to complete LingkSync configuration. If you are a Colleague by Ellucian users, please visit [Colleague Implementation notes](https://github.com/lingk/dev/tree/1afeb15bd5d5a5c3dc78b70bdc93e1e0c89bddd1/adapter/colleague-implementation-notes.html). ### Table of Contents ### Intended Audience Prior knowledge of Linux system administration is required to execute the procedures mentioned in this document. Docker knowledge is desired but not mandatory. Document should be used by: * Lingk Adapter system administrator * Client IT and SIS personnel ### Architecture Overview ### Implementation Procedure #### Pre-requisites **Server Requirement** Minimum system requirement for running the Lingk Adapter is as follows: * 2 CPU Cores * 4 GB Memory * At least 10 GB of hard disk storage **Operating System Requirement** The Lingk Adapter is tested to be working on Ubuntu 16.04 (Xenial Xerus) or later and Windows 10 (Insider Edition), but should be able to work on any operating system as long as the Docker requirements are fulfilled. ***For Linux*** Lingk recommends to use a Linux based distribution (Ubuntu 16.x server) and proceeding sections have instructions with Ubuntu 16.04 as host operating system. ***For Windows*** For Windows installation, see our separate [Docker on Windows Installation Guide for Lingk Adapter](/lingk-adapter/lingk-on-windows-installation-guide.md). For AWS environments, the Amazon Linux AMI is a recommended starting point. **Docker Requirement** The Lingk Adapter is built using Docker Server version 1.12.6 but should work on later releases as well. **Browser Requirements** Lingk Adapter management console has been tested on below browsers; * Internet Explorer 9+ * Mozilla FireFox 24+ * Google Chrome 36+ * Safari 8 #### Installation Steps Estimated Time: 15-20 minutes Lingk uses Git and Amazon Web Services (AWS) to host adapter container repositories. Following is three step process to install Lingk Adapter on client premises. When you have completed this step, you will be able to: * Run the adapter in a local environment * Run the SQL Dashboard * Configure Banner SQL Queries Additional SSL and DNS configuration is needed to: * Run the adapter in a production environment * Provide an public endpoint to your LingkSync app administrator **Before you begin...** **Git and Amazon Bucket Locations** See below for information on where to download the Lingk Adapter installation components using Git and AWS. **Firewall Ports** Ports 22, 80, 8083, and 9000 need to be open internally to your network so that your users can configure and maintain the Lingk Adapter. Port 3000 needs to be opened externally to your users (for running LingkSync wizards) as well as Lingk's Transformer Engine. Lingk’s Transformer Engine runs on Amazon AWS (unless a hybrid configuration is used). Amazon hosts a list of IP ranges for their services here: , requests from our transformer will come from the ranges listed for region: us-east-1 and service: ec2. **Getting Started** If you are installing on a Windows machine, see the separate document, [Docker on Windows Installation Guide](/lingk-adapter/lingk-on-windows-installation-guide.md) for information on installing Docker. **Step 1 - Install Docker and docker-compose** ***For Linux*** Install docker as per operating system based instructions from here; Verify that docker has been installed ``` # sudo docker info ``` To create additional ldap users, host machine should also have ldap-utils installed. For instance, below command should install ldap-utils on ubuntu flavors supporting aptitude ``` # sudo apt-get install ldap-utils ``` or ``` # sudo apt-get update # sudo apt-get ldap-utils ``` ***For Windows*** Please see the [Docker on Windows Installation Guide](/lingk-adapter/lingk-on-windows-installation-guide.md). **Step 2 - Install Lingk Adapter** ***For Linux*** As a prerequisite, install docker-compose and docker on a server where installation of LingkSync is desired. Afterwards, follow below steps to deploy the latest version of LingkSync. **Copy and paste** the command below into your terminal to obtain the GIT repository for the Lingk Adapter installation process. ``` # git clone https://github.com/lingkio/lingkadapter-docker.git ``` After downloading the git repository, you will need to change permissions for the script ``` # cd /lingkadapter-docker # chmod 755 adapter-install.sh ``` On Linux, during execution, your script will ask for the location of the LingkSync adapter file and the Lingk LDAP file. Get the URLs for the latest versions from [Resources](https://app.lingk.io/resources) page in the Lingk console. ``` Enter path of Lingksync adapter file [enter URL from Resources page for lingkadapter-lingksync Docker image] Enter path of Lingk ldap file [enter URL from Resources page for lingkadapter-ldap docker Docker image] ``` ***For Windows*** As a prerequisite, install Docker for Windows on a Windows machine where installation of LingkSync is desired. **Copy and paste** the command below into your terminal to obtain the GIT repository for the Lingk Adapter installation process. ``` # git clone https://github.com/lingkio/lingkadapter-docker.git ``` After downloading the git repository, you will need to change permissions for \_\_the script ``` # cd /lingkadapter-docker # Give full control to adapter-install-docker-windows.bat ``` If installing on Windows, you will need to manually create a folder named "downloaded" within the lingkadapter-docker folder on your machine: ``` # mkdir downloaded ``` Now download the latest versions of the LingkSync adapter and Lingk LDAP files from the [Resources](https://app.lingk.io/resources) page in the Lingk console into the "downloaded" folder. Once downloaded, rename the files as: ``` # lingkldap.tar.gz # lingksync.tar.gz ``` Execute the script ``` # ./adapter-install-docker-windows.bat ``` ***All Platforms*** Verify your installation ``` # docker ps -a ``` **Step 3: Open Lingk Adapter in your Browser** After completing above steps to install Lingk’s adapter, you can access it using a web browser from . In order to change the port, simply replace 8083 with the desired port. **NOTE:** If using Windows, you will access the REST Adapter on the local machine via (see the [Docker on Windows Installation Guide](/lingk-adapter/lingk-on-windows-installation-guide.md) for details). To login, use the default administrator credentials of: ``` username: admin password: password ``` To change the password and/or create more users, see **Create Users in LDAP.** | IMPORTANT: Reset the default Lingk Adapter Administrator password. Please see the instruction later in this document for Managing Users. | | ---------------------------------------------------------------------------------------------------------------------------------------- | #### Configuring Lingk APIs **About the Lingk Adapter APIs** Lingk Adapter APIs enable on-premise datasets to be available via RESTful APIs to wizards, recipes and connectors through the Lingk Transformer Engine. Lingk Adapter APIs can be configured using the API Dashboard that ships with the Lingk Adapter. On-premise dataset can be accessed by a SQL (i.e. query, view or store procedure) or delimited file. Each dataset is provided an API endpoint by the Lingk Adapter. Additionally, some on-premise systems may have unique data access requirements. Lingk Adapter APIs are secured between the Lingk Adapter and Lingk Transformer Engine using JWT security and provide per message signatures. Lingk Adapter APIs are used solely for the purposes of the Lingk connectors and the Lingk Platform. Enabling these APIs to be reused for other purposes in the planning stages. Please submit a request to , if you have business or technical requirements that could reuse these APIs. **Constructing Data Sets from Queries, Flat-files and Operations** Traditionally, in point to point integration, how you build datasets and the names of the output columns is not that important. With the Lingk Adapter, there are benefits to considering the names and values of the data that is returned. **To improve the integration experience, Speed Wizards can provide automapping for commonly named fields.** With a small amount of effort you can take common attributes in your datasets and enable them to be automatically aligned to Salesforce objects by aligning names and values with the list below. Additionally, your datasets will be easier to understand in future integrations. Default Speed Wizard mapping with Salesforce by resource and field name: | Lingk Adapter API Name | Field Name | LingkSync Salesforce Mapping | | ---------------------- | ----------- | ------------------------------------------------ | | Applicants | ExternalId | Contact -> Lingk External Id | | Applicants | FirstName | Contact -> First Name | | Applicants | LastName | Contact -> Last Name | | Applicants | Gender | Contact -> Gender | | Institutions | Name | Account: Educational Institution -> Account Name | | Departments | Name | Account: University Department -> Account Name | | Courses | ExternalId | Course -> Lingk External Id | | Courses | Description | Course -> Description | | Courses | Credits | Course -> Credit Hours | | Courses | CourseTitle | Course -> Course Name | | Terms | ExternalId | Term -> Lingk External Id | | Terms | Name | Term -> Term Name | | Course Sections | ExternalId | Course Offering -> Lingk External Id | | Course Sections | StartDate | Course Offering -> Start Date | | Course Sections | EndDate | Course Offering -> End Date | | Course Sections | Capacity | Course Offering -> Capacity | Unlike other tools, this isn’t about just source and target name matching. LingkSync recognizes fields that have names and values associated with the Common Education Data Standard (CEDS). You can learn more about CEDS at . **Configuration Options for Data Backends for APIs** The Lingk Adapter has 2 modes of operation: 1. Direct SQL queries: for on-demand querying against live data 2. Flat file queries: for on-demand querying against file-based snapshots Under both of these modes, the data is accessible via REST APIs to the LingkSync app. Flat file query mode would remove any impact on SIS operations and utilize a file to be dropped to a director available to a local on-premise environment. | | Direct SQL Query Operation | Flat File Query Operation | | ------------------------------ | -------------------------- | ------------------------- | | Lingk API Secret Configuration | Yes | Yes | | On-Demand Database Querying | Yes | No | | On-Demand File Querying | No | Yes | | API Tool Configuration | SQL | File | **One-time API Configurations** Complete all the one-time configurations before configuring individual APIs. **Getting Your Lingk Adapter API Credentials** You will receive your Lingk Adapter API secret from the provider configuration under Environments in your workspaces. **Configuring Your Data Flow** **1. Download the Lingk Adapter Flow template for your database from the** [**Resources**](https://app.lingk.io/resources) **page.** **2. Upload the template XML file using the upload template button on the "Operate" window.** ![](/files/-M3s1ipJ5_MD29DYwl02) **3. Drop a "Process Group" on the main screen and name it.** ![](/files/-M3s1ipLy1u8f2sblkuP) **4. Double-click into the process group** **5. Drop the template for your database onto the screen** For the current list of available templates on the Resources in the Lingk console. ![](/files/-M3s1ipOArtKgt_i7T0H) Note: All database templates support flat files endpoints. **6. Configure your controller services** Click on the gear for your processor group. A. Enable HTTP Services B. Enable HTTPS Services Use `lingkadapter123` as the password for TrustStore and KeyStore C. Counfigre the database connection. **7. Start all processors in the flow** ![image alt text](/files/-M3s1ipQPO0_g9jPEFpw) **Database Connections** If you are configuring the Lingk Adapter to connect directly to your database using queries, views or stored procedures, you will need to configure your database connections. Configure your database driver * Oracle (for PeopleSoft Campus Solutions and Banner by Ellucian) * Postgres * Microsoft SQL Server * Other JDBC providers **Configuring Database Connectivity** Here are the instructions for an Oracle database. **1. Open the Main Flow configuration dialog as shown below.** **2. Click on the pencil bolt icon for DBCPConnectionPool Controller Service** Note: You may also need to enable the `StandardHttpContextMapPort3000` controller service, if it is disabled. **3. Enter your JDBC connection string, username and password into the Lingk Adapter processor** **Oracle Connection Strings** *Example Oracle connection strings (Service Name)* ``` jdbc:oracle:thin:scott/tiger@//myhost:1521/myservicename jdbc:oracle:thin:@//myhost:1521/myservicename jdbc:oracle:thin:@myhost:1521/myservicename ``` *Example connection string (SID)\_* ``` jdbc:oracle:thin:@myhost:1521:ORCL ``` For more information on Oracle Connection strings go to: **Postgres Connection Strings** **Microsoft SQL Server Connection Strings** *Example connection string (database user)* ``` jdbc:sqlserver://localhost:1433;databaseName=AdventureWorks ``` *Example connection string (domain user)* ``` jdbc:sqlserver://localhost:1433;databaseName=AdventureWorks ;integratedSecurity=true ``` The Database Driver location should be preloaded with the JDBC driver driver associated with your flow template. ![image alt text](/files/-M3s1ipSMRKoOW_3fuOE) **4. Click Apply.** **5. Click on the Lightning icon for the same Controller service** **6. Enable the Controller Service to load the changes into associated processors** ![image alt text](/files/-M3s1ipUq7x4vHvxV8qS) **7. Press the "Play" button on the main flow to start all configured processors in the flow.** Now you are able to start configuring which datasets you want to expose through the Lingk Adapter. **Configuring An API with an SQL Database Query Backend** We will walk through the configuration of the Lingk Adapter for managing APIs used in the LingkSync product. Open the API dashboard in separate browser tab with the following URL (if configured for Private IP) `https://:9000/dashboard/` Login using your Lingk Adapter credentials \[add login screen for dashboard] **Delete** any preset APIs from the list that do not align to your data scenario and **add** any APIs to the list to align to your data scenario. ![image alt text](/files/-M3s1ipWEUJBBedfDG0e) Test any SQL queries and views using the API Dashboard to generate API metadata. Use the GET HTTP verb for pulling data from Oracle. See the section titled "Constructing Data Sets from Queries, Flat-files and Operations" on constructing user-friendly and reusable endpoints. ![image alt text](/files/-M3s1ipYmfCl90CEdgNr) Once all endpoints are complete, configure the API endpoint in the Lingk HQ. Find your provider under your workspace's environment navigation. **Configure API Security for Lingk Transformer Engine** **1. Stop the Client Secret processor to enable updates** ![image alt text](/files/-M3s1ip_4mfSi9A-nODw) **2. Click Configure on the processor** **3. Create a new Property in the Lingk Adapter processor for your LingkSync using the \[+] icon.** ![image alt text](/files/-M3s1ipbJfCutuv9PstD) **4. Enter your Lingk API Secret into the Lingk Adapter processor.** **5. Apply Changes** ![image alt text](/files/-M3s1ipdT6TS6ZAWwA-I) **6. Restart the Client Secret processor** ![image alt text](/files/-M3s1ipfuTMpWQSYmSFx) **Configuring An API with a File-based Query Backend** Open the API dashboard in separate browser tab with the following URL (if configured for Private IP) `https://:9000/dashboard/` Create a new API and specify the Endpoint Type of "File" **Default File Drop Directory** The default directory for storing CSV files to serve as an API is /opt/nifi-1.x.x/extras/files/csv. You can upload a CSV file onto your Lingk Adapter instance with the following docker command: `docker cp yourFile.csv {DOCKER_TAG}:/opt/nifi-1.x.x/extras/files/csv/yourFile.csv` You can target the output of scheduled processes to this directory and keep the file name the same to keep your files updated. For network drive support, please contact Lingk support at . **Adding a File-based API Endpoint** Specify the names, files and fields using the File Tool to configure the API endpoint. See the section titled "Constructing Data Sets from Queries, Flat-files and Operations" on constructing user-friendly and reusable endpoints. Test your file-based endpoint with "Preview Data" ![image alt text](/files/-M3s1iphA06UojeeGfdX) Click "Save" when you are finished with the endpoint #### Additional Installation Options **Manage Users in LDAP** | IMPORTANT: Reset both the default Administrator password. | | --------------------------------------------------------- | By default a user ‘admin (password: password)’ has been assigned administrator privileges and created automatically in Lingk ldap. Admin user can assign rights to more users that are created in Lingk’s LDAP. In order to create more users in Lingk LDAP, update the file named *conf/allinone.ldif* in the lingkadapter-ldap docker container with below data for each user replacing ``, `` and `` fields with relevant information. Ideally any two users records should be separated by a blank line. ``` dn: cn=username,ou=people,dc=lingk,dc=io objectclass: inetOrgPerson objectclass: organizationalPerson objectclass: person objectclass: top cn: username description: A New User sn: username uid: username mail: [username@lingk.io](mailto:username@lingk.io) userpassword: anypassword ``` once the file is saved on disk, below command should create all users mentioned in ldif file; `ldapadd -h localhost -p 389 -c -x -D cn=admin,dc=lingk,dc=io -W -f users.ldif` In order to change password for a user (for instance below procedure can be applied to change admin password) ldappasswd -h localhost -p 389 -x -D "cn=admin,ou=people,dc=lingk,dc=io" -W -A -S ldap server asks response to following prompts; ``` Old password: Re-enter old password: New password: Re-enter new password: Enter LDAP Password: ``` **SSL Certificate Generation** Note: This step is not required if SSL Private Mode is followed. For SSL Dedicated Domain Mode, this section is mandatory. Lingk's recommended approach is to use certificates from a trusted Certificate Authority (CA) and Lingk uses [http://www.letsencrypt.org](http://www.letsencrypt.org/) as CA. Below procedure is applicable if client intends to use their own certificates. Lets assume that client wants to generate certificates for subdomain adapter.yourwebsitedomain.com to host lingkadapter. Following are a set of steps that need to be carried out for certificate generation. **Step 1: Copy utility script repository on server that is hosting the domain/subdomain** `curl --silent [https://raw.githubusercontent.com/srvrco/getssl/master/getssl](https://raw.githubusercontent.com/srvrco/getssl/master/getssl) > getssl ;` `chmod 700 getssl` **Step 2: Generate certificate configuration files** `./getssl -c adapter.yourwebsitedomain.com` **Step 3: Edit the getssl.cfg file** Changed the ACCOUNT\_EMAIL directive to actual e-mail address `/root/.getssl/getssl.cfg` Change Server location from ‘staging’ to production or else it will not be trusted The staging server is best for testing (hence set as default) CA="[https://acme-staging.api.letsencrypt.org](https://acme-staging.api.letsencrypt.org/)" This server issues full certificates, however has rate limits CA="[https://acme-v01.api.letsencrypt.org](https://acme-v01.api.letsencrypt.org/)" **Step 3a: Edit the getssl.cfg file for the domain** Comment the SANS= directive, because we will have exactly 1 domain. /root/.getssl/yourwebsitedomain.com/getssl.cfg Specify ACL directive to: ACL=('/some/directoryAsRoot/.well-known/acme-challenge') Un-comment following directives: ``` DOMAIN_CERT_LOCATION="/etc/ssl/adapter.yourwebsitedomain.com.crt" DOMAIN_KEY_LOCATION="/etc/ssl/adapter.yourwebsitedomain.com.key" CA_CERT_LOCATION="/etc/ssl/chain.crt" DOMAIN_CHAIN_LOCATION="/etc/ssl/" # this is the domain cert and CA cert DOMAIN_PEM_LOCATION="/etc/ssl/" # this is the domain_key, domain cert and CA cert ``` Save this file. **Step 4: Start a Small Server Thread (Temporary)** Create a separate terminal window for this step to start a process that will be used for subsequent steps. For ACL generation, we start a small server that letsencrypt will use to verify whether we are on the server hosting the lingkadapter ``` $ mkdir /some $ mkdir /some/directoryAsRoot $ mkdir /some/directoryAsRoot/.well-known $ cd /some/directoryAsRoot $ python -m SimpleHTTPServer 80 ``` Note: If python is not found, in ubuntu you can install python-minimal using below command; $ apt install python-minimal Once this has been started, let this command run on this terminal and continue from next step in another terminal window. **Step 5: Generate SSL Certificates for the domain** Run the following command to generate SSL Certificates for the domain. `$ getssl adapter.yourwebsitedomain.com` Certificates will be generated in location /etc/ssl/. Following two files will be of significance `adapter.yourwebsitedomain.com_chain.pem` `adapter.yourwebsitedomain.com.pem` **Step 6: Stop Small Server Thread** Once Step 5 is executed successful and certificates have been verified, open the terminal that was left in Step 4 and press 'Ctrl+c' to terminate the server thread. This terminal window is no longer needed. **Step 7: Generate JAVA Keystore and formats as desired by Adapter** Lingk Adapter has embedded Java-based server that needs Java Keystore (JKS). In order to generate keystore.jks from the generated certificates in the above steps 1-6, follow these two steps. $ keytool -import -file adapter.yourwebsitedomain.com\_chain.pem -alias cacert -keystore truststore.jks -storepass lingkadapter123 Convert the key to pkcs12 format. $ openssl pkcs12 -export -inkey adapter.yourwebsitedomain.com.pem -in adapter.yourwebsitedomain.com.pem -out cert\_key.p12 use password: lingkadapter123 The two generated files (i.e. keystore.jks and cert\_key.p12) can be used as newly generated certificates. Place these two files under folder ‘certs’. **Changing the Default Mapped Ports** When you are in an environment that has limited external ports, you may need to change the ports mapped in your Lingk Adapter. The instructions below will help you map the API port to the standard HTTPS port. You still may need to use localhost on the server for all configuration of the APIs with the solution below. There are two solutions to exposing your API port (port 3000) externally on a default port (443) and you can choose which one is most expedient for you. A. Delete the Lingk Adapter container and recreate it with this command: ``` # docker run \ -idt \ -p 8083:443 \ -p 9000:9000 \ -p 443:3000 \ --link lingkldap:lingk \ --name lingkadapter-lingksync \ quay.io/lingkio/lingkadapter-lingksync:latest ``` B. Modify the ports on your existing Lingk Adapter container per the following: **Event Based Integration Lingk Adapter to the Lingk Platform** Refer to the "Lingk Adapter for Event-Based Integration Guide" **Upgrade the LingkSync Adapter to the Latest Version** Refer to the "Lingk Adapter Upgrade Guide" #### Resources **The Lingk Adapter Distribution of Apache Nifi** The Lingk Adapter is a higher education distribution of the Apache Nifi project. If you would like to learn more about Apache Nifi concepts please go to: ![image alt text](/files/-M3s1ipjSo3y3Q1moqLN)