FTP and SFTP Tasks

Overview

This document describes the tasks available for FTP (File Transfer Protocol) and SFTP (SSH File Transfer Protocol) operations within the CargoXplorer TMS System. These tasks enable secure file transfers between the TMS and external systems.

YAML Structure

FTP and SFTP tasks follow a consistent structure that allows for flexible configuration of file transfer operations.

Attribute Description

FTP/Connect

The FTP/Connect task establishes a connection to an FTP server.

task: "FTP/Connect@1"
name: ftpConnect
inputs:
  host: "ftp.example.com"
  port: 21 # Optional - Default is 21
  username: "ftpuser"
  password: "{{ secrets.FTP_PASSWORD }}"
  passive: true # Optional - Use passive mode
  secure: false # Optional - Use explicit FTPS (FTP over TLS)
  timeout: 30 # Optional - Connection timeout in seconds
outputs:
  - name: "connection"
    mapping: "connection"

Attribute Details:

Input Parameter	Description
`host`	FTP server hostname or IP address
`port`	Port number for the FTP server
`username`	Username for authentication
`password`	Password for authentication
`passive`	Whether to use passive mode
`secure`	Whether to use FTPS (FTP over TLS)
`timeout`	Connection timeout in seconds

Output:

{
  "connection": {
    "id": "ftp-conn-123456",
    "host": "ftp.example.com",
    "connected": true
  }
}

FTP/Disconnect

The FTP/Disconnect task closes an FTP connection.

task: "FTP/Disconnect@1"
name: ftpDisconnect
inputs:
  connection: "{{ ftpConnect.connection }}"

FTP/ListFiles

The FTP/ListFiles task retrieves a list of files from an FTP server directory.

task: "FTP/ListFiles@1"
name: ftpListFiles
inputs:
  connection: "{{ ftpConnect.connection }}"
  directory: "/incoming" # Optional - Default is root directory
  pattern: "*.edi" # Optional - File pattern to match
  recursive: false # Optional - List files recursively
outputs:
  - name: "files"
    mapping: "files"

Attribute Details:

Input Parameter	Description
`connection`	FTP connection object from FTP/Connect
`directory`	Directory path to list files from
`pattern`	File pattern to filter results
`recursive`	Whether to list files recursively

Output:

{
  "files": [
    {
      "name": "order123.edi",
      "path": "/incoming/order123.edi",
      "size": 1024,
      "lastModified": "2023-05-15T10:30:00Z",
      "type": "file"
    },
    {
      "name": "shipment456.edi",
      "path": "/incoming/shipment456.edi",
      "size": 2048,
      "lastModified": "2023-05-15T11:45:00Z",
      "type": "file"
    }
  ]
}

FTP/DownloadFile

The FTP/DownloadFile task downloads a file from an FTP server.

task: "FTP/DownloadFile@1"
name: ftpDownloadFile
inputs:
  connection: "{{ ftpConnect.connection }}"
  remotePath: "/incoming/order123.edi"
  localPath: "/tmp/downloads/order123.edi" # Optional - If not provided, file content is returned
  deleteAfterDownload: false # Optional - Delete remote file after download
  overwrite: true # Optional - Overwrite local file if exists
outputs:
  - name: "fileInfo"
    mapping: "fileInfo"
  - name: "content"
    mapping: "content" # Only if localPath is not provided

Attribute Details:

Input Parameter	Description
`connection`	FTP connection object from FTP/Connect
`remotePath`	Path to the file on the FTP server
`localPath`	Local path to save the downloaded file
`deleteAfterDownload`	Whether to delete the remote file after download
`overwrite`	Whether to overwrite existing local file

Output:

{
  "fileInfo": {
    "name": "order123.edi",
    "path": "/tmp/downloads/order123.edi",
    "size": 1024,
    "lastModified": "2023-05-15T10:30:00Z"
  },
  "content": "ISA*00*          *00*          *ZZ*ABCDEFGH      *ZZ*123456789      *210101*1030*U*00401*000000001*0*P*>\n..."
}

FTP/UploadFile

The FTP/UploadFile task uploads a file to an FTP server.

task: "FTP/UploadFile@1"
name: ftpUploadFile
inputs:
  connection: "{{ ftpConnect.connection }}"
  localPath: "/tmp/uploads/shipment456.edi" # Either localPath or content must be provided
  content: "ISA*00*          *00*          *ZZ*ABCDEFGH      *ZZ*123456789      *210101*1030*U*00401*000000001*0*P*>\n..." # Either localPath or content must be provided
  remotePath: "/outgoing/shipment456.edi"
  createDirectories: true # Optional - Create remote directories if they don't exist
  overwrite: true # Optional - Overwrite remote file if exists
outputs:
  - name: "fileInfo"
    mapping: "fileInfo"

Attribute Details:

Input Parameter	Description
`connection`	FTP connection object from FTP/Connect
`localPath`	Local path of the file to upload
`content`	File content to upload (alternative to localPath)
`remotePath`	Path to save the file on the FTP server
`createDirectories`	Whether to create remote directories if needed
`overwrite`	Whether to overwrite existing remote file

Output:

{
  "fileInfo": {
    "name": "shipment456.edi",
    "path": "/outgoing/shipment456.edi",
    "size": 2048,
    "uploaded": true,
    "timestamp": "2023-05-15T14:30:00Z"
  }
}

FTP/DeleteFile

The FTP/DeleteFile task deletes a file from an FTP server.

task: "FTP/DeleteFile@1"
name: ftpDeleteFile
inputs:
  connection: "{{ ftpConnect.connection }}"
  remotePath: "/incoming/order123.edi"
outputs:
  - name: "success"
    mapping: "success"

Attribute Details:

Input Parameter	Description
`connection`	FTP connection object from FTP/Connect
`remotePath`	Path to the file to delete on the FTP server

Output:

{
  "success": true
}

FTP/MoveFile

The FTP/MoveFile task moves or renames a file on an FTP server.

task: "FTP/MoveFile@1"
name: ftpMoveFile
inputs:
  connection: "{{ ftpConnect.connection }}"
  sourcePath: "/incoming/order123.edi"
  destinationPath: "/processed/order123.edi"
  createDirectories: true # Optional - Create destination directories if they don't exist
  overwrite: true # Optional - Overwrite destination file if exists
outputs:
  - name: "success"
    mapping: "success"
  - name: "fileInfo"
    mapping: "fileInfo"

Attribute Details:

Input Parameter	Description
`connection`	FTP connection object from FTP/Connect
`sourcePath`	Path to the source file on the FTP server
`destinationPath`	Path to move/rename the file to
`createDirectories`	Whether to create destination directories
`overwrite`	Whether to overwrite existing destination file

Output:

{
  "success": true,
  "fileInfo": {
    "name": "order123.edi",
    "path": "/processed/order123.edi",
    "size": 1024,
    "lastModified": "2023-05-15T15:30:00Z"
  }
}

SFTP/Connect

The SFTP/Connect task establishes a connection to an SFTP server.

task: "SFTP/Connect@1"
name: sftpConnect
inputs:
  host: "sftp.example.com"
  port: 22 # Optional - Default is 22
  username: "sftpuser"
  authentication:
    type: "password" # Options: password, privateKey
    password: "{{ secrets.SFTP_PASSWORD }}" # Required if type is password
    privateKey: "{{ secrets.SFTP_PRIVATE_KEY }}" # Required if type is privateKey
    passphrase: "{{ secrets.SFTP_KEY_PASSPHRASE }}" # Optional - For encrypted private keys
  timeout: 30 # Optional - Connection timeout in seconds
  hostKeyVerification:
    enabled: true # Optional - Verify host key
    hostKey: "ssh-rsa AAAAB3NzaC1yc2E..." # Optional - Expected host key
outputs:
  - name: "connection"
    mapping: "connection"

Attribute Details:

Input Parameter	Description
`host`	SFTP server hostname or IP address
`port`	Port number for the SFTP server
`username`	Username for authentication
`authentication.type`	Authentication method (password or privateKey)
`authentication.password`	Password for password authentication
`authentication.privateKey`	Private key for key-based authentication
`authentication.passphrase`	Passphrase for encrypted private key
`timeout`	Connection timeout in seconds
`hostKeyVerification.enabled`	Whether to verify the server's host key
`hostKeyVerification.hostKey`	Expected host key for verification

Output:

{
  "connection": {
    "id": "sftp-conn-123456",
    "host": "sftp.example.com",
    "connected": true
  }
}

SFTP/Disconnect

The SFTP/Disconnect task closes an SFTP connection.

task: "SFTP/Disconnect@1"
name: sftpDisconnect
inputs:
  connection: "{{ sftpConnect.connection }}"

SFTP/ListFiles

The SFTP/ListFiles task retrieves a list of files from an SFTP server directory.

task: "SFTP/ListFiles@1"
name: sftpListFiles
inputs:
  connection: "{{ sftpConnect.connection }}"
  directory: "/incoming" # Optional - Default is root directory
  pattern: "*.edi" # Optional - File pattern to match
  recursive: false # Optional - List files recursively
outputs:
  - name: "files"
    mapping: "files"

Attribute Details:

Input Parameter	Description
`connection`	SFTP connection object from SFTP/Connect
`directory`	Directory path to list files from
`pattern`	File pattern to filter results
`recursive`	Whether to list files recursively

Output:

{
  "files": [
    {
      "name": "order123.edi",
      "path": "/incoming/order123.edi",
      "size": 1024,
      "lastModified": "2023-05-15T10:30:00Z",
      "permissions": "rw-r--r--",
      "type": "file"
    },
    {
      "name": "shipment456.edi",
      "path": "/incoming/shipment456.edi",
      "size": 2048,
      "lastModified": "2023-05-15T11:45:00Z",
      "permissions": "rw-r--r--",
      "type": "file"
    }
  ]
}

SFTP/DownloadFile

The SFTP/DownloadFile task downloads a file from an SFTP server.

task: "SFTP/DownloadFile@1"
name: sftpDownloadFile
inputs:
  connection: "{{ sftpConnect.connection }}"
  remotePath: "/incoming/order123.edi"
  localPath: "/tmp/downloads/order123.edi" # Optional - If not provided, file content is returned
  deleteAfterDownload: false # Optional - Delete remote file after download
  overwrite: true # Optional - Overwrite local file if exists
  preserveTimestamp: true # Optional - Preserve file timestamp
outputs:
  - name: "fileInfo"
    mapping: "fileInfo"
  - name: "content"
    mapping: "content" # Only if localPath is not provided

Attribute Details:

Input Parameter	Description
`connection`	SFTP connection object from SFTP/Connect
`remotePath`	Path to the file on the SFTP server
`localPath`	Local path to save the downloaded file
`deleteAfterDownload`	Whether to delete the remote file after download
`overwrite`	Whether to overwrite existing local file
`preserveTimestamp`	Whether to preserve the file's timestamp

Output:

{
  "fileInfo": {
    "name": "order123.edi",
    "path": "/tmp/downloads/order123.edi",
    "size": 1024,
    "lastModified": "2023-05-15T10:30:00Z",
    "permissions": "rw-r--r--"
  },
  "content": "ISA*00*          *00*          *ZZ*ABCDEFGH      *ZZ*123456789      *210101*1030*U*00401*000000001*0*P*>\n..."
}

SFTP/UploadFile

The SFTP/UploadFile task uploads a file to an SFTP server.

task: "SFTP/UploadFile@1"
name: sftpUploadFile
inputs:
  connection: "{{ sftpConnect.connection }}"
  localPath: "/tmp/uploads/shipment456.edi" # Either localPath or content must be provided
  content: "ISA*00*          *00*          *ZZ*ABCDEFGH      *ZZ*123456789      *210101*1030*U*00401*000000001*0*P*>\n..." # Either localPath or content must be provided
  remotePath: "/outgoing/shipment456.edi"
  permissions: "0644" # Optional - Set file permissions (octal)
  createDirectories: true # Optional - Create remote directories if they don't exist
  overwrite: true # Optional - Overwrite remote file if exists
outputs:
  - name: "fileInfo"
    mapping: "fileInfo"

Attribute Details:

Input Parameter	Description
`connection`	SFTP connection object from SFTP/Connect
`localPath`	Local path of the file to upload
`content`	File content to upload (alternative to localPath)
`remotePath`	Path to save the file on the SFTP server
`permissions`	File permissions in octal format
`createDirectories`	Whether to create remote directories if needed
`overwrite`	Whether to overwrite existing remote file

Output:

{
  "fileInfo": {
    "name": "shipment456.edi",
    "path": "/outgoing/shipment456.edi",
    "size": 2048,
    "uploaded": true,
    "permissions": "rw-r--r--",
    "timestamp": "2023-05-15T14:30:00Z"
  }
}

SFTP/DeleteFile

The SFTP/DeleteFile task deletes a file from an SFTP server.

task: "SFTP/DeleteFile@1"
name: sftpDeleteFile
inputs:
  connection: "{{ sftpConnect.connection }}"
  remotePath: "/incoming/order123.edi"
outputs:
  - name: "success"
    mapping: "success"

Attribute Details:

Input Parameter	Description
`connection`	SFTP connection object from SFTP/Connect
`remotePath`	Path to the file to delete on the SFTP server

Output:

{
  "success": true
}

SFTP/MoveFile

The SFTP/MoveFile task moves or renames a file on an SFTP server.

task: "SFTP/MoveFile@1"
name: sftpMoveFile
inputs:
  connection: "{{ sftpConnect.connection }}"
  sourcePath: "/incoming/order123.edi"
  destinationPath: "/processed/order123.edi"
  createDirectories: true # Optional - Create destination directories if they don't exist
  overwrite: true # Optional - Overwrite destination file if exists
outputs:
  - name: "success"
    mapping: "success"
  - name: "fileInfo"
    mapping: "fileInfo"

Attribute Details:

Input Parameter	Description
`connection`	SFTP connection object from SFTP/Connect
`sourcePath`	Path to the source file on the SFTP server
`destinationPath`	Path to move/rename the file to
`createDirectories`	Whether to create destination directories
`overwrite`	Whether to overwrite existing destination file

Output:

{
  "success": true,
  "fileInfo": {
    "name": "order123.edi",
    "path": "/processed/order123.edi",
    "size": 1024,
    "lastModified": "2023-05-15T15:30:00Z",
    "permissions": "rw-r--r--"
  }
}

Examples

Downloading EDI Files from SFTP Server

task: "SFTP/Connect@1"
name: connectToPartnerSFTP
inputs:
  host: "{{ partner.sftpHost }}"
  port: 22
  username: "{{ partner.sftpUsername }}"
  authentication:
    type: "password"
    password: "{{ secrets.PARTNER_SFTP_PASSWORD }}"
  timeout: 60
outputs:
  - name: "sftpConnection"
    mapping: "connection"

task: "SFTP/ListFiles@1"
name: listEDIFiles
inputs:
  connection: "{{ connectToPartnerSFTP.sftpConnection }}"
  directory: "/incoming"
  pattern: "*.edi"
outputs:
  - name: "ediFiles"
    mapping: "files"

task: "Utilities/ForEach@1"
name: processEDIFiles
inputs:
  items: "{{ listEDIFiles.ediFiles }}"
  parallel: true
  maxParallel: 5
  task:
    task: "SFTP/DownloadFile@1"
    name: downloadEDIFile
    inputs:
      connection: "{{ connectToPartnerSFTP.sftpConnection }}"
      remotePath: "{{ item.path }}"
      deleteAfterDownload: true
    outputs:
      - name: "fileContent"
        mapping: "content"

    task: "EDI/Parse@1"
    name: parseEDIFile
    inputs:
      ediData: "{{ downloadEDIFile.fileContent }}"
      validateSchema: true
    outputs:
      - name: "parsedData"
        mapping: "result"
      - name: "transactionType"
        mapping: "transactionType"

task: "SFTP/Disconnect@1"
name: disconnectFromSFTP
inputs:
  connection: "{{ connectToPartnerSFTP.sftpConnection }}"

Uploading EDI Acknowledgments to FTP Server

task: "FTP/Connect@1"
name: connectToPartnerFTP
inputs:
  host: "{{ partner.ftpHost }}"
  port: 21
  username: "{{ partner.ftpUsername }}"
  password: "{{ secrets.PARTNER_FTP_PASSWORD }}"
  passive: true
  secure: true
outputs:
  - name: "ftpConnection"
    mapping: "connection"

task: "EDI/Acknowledge@1"
name: generateAcknowledgment
inputs:
  ediData: "{{ workflow.input.originalEDI }}"
  acknowledgmentType: "997"
  status: "A"
outputs:
  - name: "acknowledgmentEDI"
    mapping: "ediString"

task: "FTP/UploadFile@1"
name: uploadAcknowledgment
inputs:
  connection: "{{ connectToPartnerFTP.ftpConnection }}"
  content: "{{ generateAcknowledgment.acknowledgmentEDI }}"
  remotePath: "/outgoing/ACK_{{ formatDate now 'YYYYMMDDHHmmss' }}.edi"
  createDirectories: true
outputs:
  - name: "uploadInfo"
    mapping: "fileInfo"

task: "FTP/Disconnect@1"
name: disconnectFromFTP
inputs:
  connection: "{{ connectToPartnerFTP.ftpConnection }}"

Best Practices

Secure Credentials: Always use environment variables or secrets management for storing credentials.
Connection Management: Always disconnect from FTP/SFTP servers after operations are complete.
Error Handling: Implement proper error handling for network issues and file operation failures.
File Naming: Use consistent file naming conventions with timestamps to avoid conflicts.
Directory Structure: Maintain a clear directory structure on FTP/SFTP servers (e.g., /incoming, /outgoing, /processed, /errors).
Logging: Log all file transfer operations for audit and troubleshooting purposes.
Timeout Configuration: Set appropriate timeouts based on expected file sizes and network conditions.
Host Key Verification: For SFTP, always verify host keys in production environments to prevent man-in-the-middle attacks.

Overview​

YAML Structure​

Attribute Description​

FTP/Connect​

FTP/Disconnect​

FTP/ListFiles​

FTP/DownloadFile​

FTP/UploadFile​

FTP/DeleteFile​

FTP/MoveFile​

SFTP/Connect​

SFTP/Disconnect​

SFTP/ListFiles​

SFTP/DownloadFile​

SFTP/UploadFile​

SFTP/DeleteFile​

SFTP/MoveFile​

Examples​

Downloading EDI Files from SFTP Server​

Uploading EDI Acknowledgments to FTP Server​

Best Practices​

Related Topics​

Overview

YAML Structure

Attribute Description

FTP/Connect

FTP/Disconnect

FTP/ListFiles

FTP/DownloadFile

FTP/UploadFile

FTP/DeleteFile

FTP/MoveFile

SFTP/Connect

SFTP/Disconnect

SFTP/ListFiles

SFTP/DownloadFile

SFTP/UploadFile

SFTP/DeleteFile

SFTP/MoveFile

Examples

Downloading EDI Files from SFTP Server

Uploading EDI Acknowledgments to FTP Server

Best Practices

Related Topics