External API Connections
From version 2.2, encryption(in-flight, at-rest) for all jobs and catalog is enabled. All the existing jobs(User created, and also system created) were updated with encryption related settings, and all the newly created jobs will have encryption enabled automatically.
External API connections are used to import data from APIs to Amorphic Dataset. Only API Authentication of type BASIC is supported as of now.
Below are the ways to create an External API
BASIC
To create an External API connection, user has to enter API Endpoint, HTTP Method and Query String Parameters. Below image shows how to create an External API Connection
| Attribute | Description |
|---|---|
| Connection Name | Name of the connection Amorphic |
| Connection Type | Type of connection. In this case it is ExternalAPI |
| Description | Connection related information user wants to store |
| Authorized Users | Amorphic users to whom user wants to have access to this connection |
| API Endpoint | Endpoint URL from which data needs to be extracted |
| API Authentication | As of version 1.1.3 only BASIC is supported |
| Method | HTTP Method, as of version 1.1.3 only GET and POST are allowed |
| Query Params | Query string parameters which the API URL takes as input. |
| Version | Enables the user to select what version of ingestion scripts to use (Amorphic specific). For any new feature/Glue version that gets added to the underlying ingestion script, new version will be added to the Amorphic. |
External API details
Edit
There is an option to edit an External API Connection. To edit an External API Connection, click the edit button which on the right corner.
Description and Authorised users of an External API Connection can be changed.
Upgrade
User can upgrade a connection (if available) which will be shown in the options. Upgrade option is visible only if there is any new version available else it is not shown.
Connection upgrade upgrades the underlying Glue version and the data ingestion script with new features.
Downgrade
User can downgrade a connection to previous version if they think the upgrade is not serving the purpose. A connection can only be downgraded if it has been upgraded. For a connection that has been created we can't downgrade it. The option to downgrade is available on the top right corner if the connection is downgrade compatible.
Deletion
On the top right corner, there is a button with trash can on it. Click on it to delete.
Connection Versions
1.1
In this version of external api connections, we added auto-reload feature for datasets of type reload.
From this version onwards, data reloads process will trigger automatically as soon as the file upload finishes through the external api connections. So that user don't need to manually trigger reload process after completion of file upload when ingesting data through external api ingestion connection.
1.2
In this version we made code changes in the underlying glue script for the support dataset custom partitioning.
From this version onwards, the data will be loaded into into a S3 LZ with the prefix containing the partition key(if you specified any) for the targets which supports dataset partitioning.
Eg. For the partition keys KeyA, KeyB with the values ValueA, ValueB respectively, the S3 prefix will be in the format
Domain/DatasetName/KeyA=ValueA/KeyB=ValueB/upload_date=Unix_Timestamp/UserName/FileType/.
To understand more about custom data partitioning, read the docs about dataset custom partitioning here.
1.3
In this version of external API connection, we added support of skip LZ feature.
This feature enables users to directly upload data to the data lake zone by skipping the data validation. Please refer Skip LZ related docs for more details.