Setting up SFTP Imports

[MarcusEdwards]

Overview

Crownpeak DXM provides a facility for external content or data to be imported from an SFTP source and brought into the content management lifecycle. SFTP is used so that this process can operate on a pull model, which is preferable from a performance and scalability point of view.

NOTE: We only support SFTP imports now. Support for FTP has been dropped for security reasons but there are legacy configurations fields are still visible.

SFTP Imports can be used to implement many different use cases. The first step is deciding what data will be imported and how the data will be used in the CMS. Because of the possibility to perform bulk operations in an import, it is common to import text-based data that has some structure, for example:

• Delimited content -- commonly comma- or tab-separated
• XML
• JSON
• Plain text

The import can also bring in binary file and create digital assets (non-templated assets).

Pre-requisites

Before you can configure the SFTP import, you will need to have a number of elements prepared.

1. A template that has an ftp_import.aspx handler
2. A model blueprint that uses your template
3. A folder where you import content will be placed -- this does not need to be the final location of content created during the import process.

Import Template

At a minimum, the template needs to have the ftp_import.aspx handler defined. Although the template could have other handlers defined, it is highly recommended that you have a separate import template to the content template(s) your import will be used.

In the ftp_import.aspx handler, you can access the data from the FTP file through the FTPInputContext's FtpBody property. The content is provided as a string so you will need to parse this if you need to extract rows and fields from it. Util.CreateListFromCsv, Util.DeserializeDataContractJson or Util.DeserializeDataContractXml are useful but use whatever text parser you need.

Once you have extracted fields, you can use Asset.CreateNewAsset to create new assets; or Asset.Load and Asset.SaveContent to update existing assets. What you can also do is treat each import as a single asset rather than as a batch, in which case you will be extracting fields and calling asset.SaveContent. You will almost certainly want to use asset.Rename as well.

Set the FTPInputContext AutoCreateAsset property to false if you don't want the FTP import process to create an asset for the underlying FTP import file. However, it can be useful to have this in the CMS so that if the import process fails or has errors, it can be re-run after any corrective action has been taken.

Import Model

Once you have the import template ready, create a model blueprint that uses your template. The purpose of this model is to ensure that the asset that is created from the FTP import picks up the correct template and workflow settings.

Import Folder

There is nothing special about this folder, but it is needed during the configuration phase. 

For batch imports, it is recommended that the folder is a generic container, "imports" for example. If you are importing individual assets, then the folder could of course be more descriptive, "articles" for example.

DXM Configuration

SFTP import configurations can be set up in the Settings > Publishing > SFTP Imports panel.

Click on the "Add FTP Import" button to create a new configuration, or click on an existing one to edit. Once you are editing a particular configuration, these are the properties you can edit:

Field		Description
Name	Required	The name of the import configuration. Used for reference purposes only.
Expires On		Allows you to pick a date and time after which the import will not be triggered.
Polls	Required	Pick the frequency the import configuration will be triggered by the DXM scheduler. Options range from 1 day to 5 minutes
Max number of files PER DAY		The maximum number of files that can be imported by this configuration in any given day. This is useful for limiting the import in the event there is a sudden content surge or a problem with the upstream system.
Max number of files		The maximum number of files this import configuration can import in total. Again, useful if you need to place a limit on the process to protect against runaway processes.
Owner of the imported files	Required	Pick the existing CMS user that the new assets will appear to be created by.
Folder where generated documents are stored	Required	See import folder from the pre-requisites.
Model that will be used when creating the documents	Required	See import model from the pre-requisites
Once the file has been processed	Required	Choose from one of the actions the import process should perform on the remote file after import. Most configurations will either move the file to a "remove" or processed folder so that the file isn't imported again on the next cycle; or will delete the remove file.
Remote "remove" folder	Required (if the "Move to remote remove folder" action is selected)	The name of the remote folder to move the import file to.
SFTP Hostname	Required	The hostname or IP address of the remote SFTP server.
SFTP Remote Folder		The folder to change to on the remote server to start lookng for import candidates. If not specified, the process will use the default folder after login.
SFTP Username		The username to use when authenticating with the SFTP server.
SFTP Password		The password to use with the SFTP username when authenticating with the SFTP server.
Use Active connection	NO LONGER USED	Whether to use FTP active or passive connection.
Use SFTP		This property must be checked as we no longer support FTP configurations.
Binary import		If checked, this indicates that the import should NOT run the ftp_import.aspx handler but rather just create a digital asset with the imported file.
Send errors to email		An email address that a notification of any errors or faults in an import will be sent to. Although not required, it is strongly recommened that you use this facility.

Once you have entered all the configuration, you can test the connection; or trigger an import usng the buttons at the bottom of the configuration panel. This is a crucial part of the development and configuration process so that you can be sure that all the information is correct and also that the DXM servers can establish a connection to the remote server.

Once you save the import configuration, DXM will generate a unique token for the configuration. This is visible in the import overview panel as well as the individual configuration edit panel. You will need this token in the next section.

Remote Configuration

Once you have the DXM configuration set up and have the generated token, you can start delivering files into the SFTP server folder. For text files, the filename must include the generated configuration token in order for DXM to match up files and import configurations, for example: articles-20190812-TK9qMId7zxgvGRSJ.xml

NOTE: If you have a binary import configuration, you do NOT need to have the token appear in the filename. For this reason, it is strongly recommended that digital assets for import are placed in their own folder on the SFTP server.

[kunalshah]

Thanks for this post.

Is there an alternative to Polling, like is there an event that can trigger the import that can be used programitically?

[MarcusEdwards]

You can trigger the FTP import process manually from the FTP Import configuration screen.

The only API that is available to trigger an SFTP import is using the Access API method: /accessapi/settings/TriggerFtpImport/{id}. The method takes as parameter the unique asset id of the SFTP configuration. This can be found by using the Access API method /accessapi/settings/publishing/sftp-import.

Using the Access API from a template running in the CMS context is possible, but you will need to go through the full Access API process including authenticating.

*NOTE: All these URLs are pointing to the CPUK training instance and should be modified to target your CMS instance.

[kunalshah]

Thanks Marcus.