File Transfer (FTP) Tasks

Overview

This document describes the FTP tasks available for FTP, FTPS, and SFTP operations within the CargoXplorer TMS System. These tasks provide a single interface for secure file transfers between the TMS and external systems across multiple protocols.

YAML Structure

File transfer tasks use a unified structure that automatically handles protocol-specific configurations based on the connection type specified.

Attribute Description

FileTransfer/Connect

The FileTransfer/Connect task establishes a connection to an FTP, FTPS, or SFTP server.

task: "FileTransfer/Connect@1"
name: fileTransferConnect
inputs:
  host: "ftp.example.com"
  protocol: "sftp" # Options: ftp, ftps, sftp
  port: 22 # Optional - Auto-detected based on protocol (21=FTP, 990=FTPS, 22=SFTP)
  username: "ftpuser"
  authentication:
    type: "password" # Options: password, privateKey (SFTP only)
    password: "{{ secrets.FTP_PASSWORD }}"
    privateKey: "{{ secrets.SFTP_PRIVATE_KEY }}" # SFTP only
    passphrase: "{{ secrets.SFTP_KEY_PASSPHRASE }}" # SFTP only
  options:
    passive: true # FTP/FTPS only - Use passive mode
    timeout: 30 # Connection timeout in seconds
    hostKeyVerification: # SFTP only
      enabled: true # Verify host key
      hostKey: "ssh-rsa AAAAB3NzaC1yc2E..." # Expected host key
outputs:
  - name: "connection"
    mapping: "connection"

Attribute Details:

Input Parameter	Type	Description	Protocols
`host`	string	Server hostname or IP address	All
`protocol`	string	Protocol type: ftp, ftps, sftp	All
`port`	integer	Port number (auto-detected if not specified)	All
`username`	string	Username for authentication	All
`authentication.type`	string	Authentication method	All
`authentication.password`	string	Password for authentication	All
`authentication.privateKey`	string	Private key content (PEM format)	SFTP
`authentication.passphrase`	string	Passphrase for encrypted private key	SFTP
`options.passive`	boolean	Use passive mode (default: true)	FTP/FTPS
`options.timeout`	integer	Connection timeout in seconds (default: 30)	All
`options.hostKeyVerification.enabled`	boolean	Verify server host key (default: true)	SFTP
`options.hostKeyVerification.hostKey`	string	Expected host key for verification	SFTP

Protocol-Specific Defaults:

Protocol	Default Port	Default Options
`ftp`	21	passive: true, secure: false
`ftps`	990	passive: true, secure: true
`sftp`	22	hostKeyVerification.enabled: true

Output:

{
  "connection": {
    "id": "ft-conn-123456",
    "protocol": "sftp",
    "host": "ftp.example.com",
    "port": 22,
    "connected": true,
    "features": ["upload", "download", "list", "delete", "move"]
  }
}

FileTransfer/Disconnect

The FileTransfer/Disconnect task closes a file transfer connection.

task: "FileTransfer/Disconnect@1"
name: fileTransferDisconnect
inputs:
  connection: "{{ fileTransferConnect.connection }}"

FileTransfer/ListFiles

The FileTransfer/ListFiles task retrieves a list of files from a server directory.

task: "FileTransfer/ListFiles@1"
name: fileTransferListFiles
inputs:
  connection: "{{ fileTransferConnect.connection }}"
  directory: "/incoming" # Optional - Default is root directory
  pattern: "*.edi" # Optional - File pattern to match (supports wildcards)
  recursive: false # Optional - List files recursively
  includeDirectories: false # Optional - Include directories in results
  sortBy: "lastModified" # Optional - Sort by: name, size, lastModified
  sortOrder: "desc" # Optional - Sort order: asc, desc
outputs:
  - name: "files"
    mapping: "files"
  - name: "totalSize"
    mapping: "totalSize"
  - name: "fileCount"
    mapping: "fileCount"

Attribute Details:

Input Parameter	Type	Description
`connection`	object	File transfer connection object from FileTransfer/Connect
`directory`	string	Directory path to list files from
`pattern`	string	File pattern to filter results (supports wildcards)
`recursive`	boolean	Whether to list files recursively
`includeDirectories`	boolean	Include directories in results
`sortBy`	string	Sort files by: name, size, lastModified
`sortOrder`	string	Sort order: asc, desc

Output:

{
  "files": [
    {
      "name": "order123.edi",
      "path": "/incoming/order123.edi",
      "size": 1024,
      "lastModified": "2023-05-15T10:30:00Z",
      "permissions": "rw-r--r--",
      "type": "file",
      "checksum": "sha256:abc123..."
    }
  ],
  "totalSize": 3072,
  "fileCount": 2
}

FileTransfer/DownloadFile

The FileTransfer/DownloadFile task downloads a file from a server.

task: "FileTransfer/DownloadFile@1"
name: fileTransferDownload
inputs:
  connection: "{{ fileTransferConnect.connection }}"
  remotePath: "/incoming/order123.edi"
  localPath: "/tmp/downloads/order123.edi" # Optional - If not provided, file content is returned
  options:
    deleteAfterDownload: false # Delete remote file after download
    overwrite: true # Overwrite local file if exists
    preserveTimestamp: true # Preserve file timestamp (SFTP)
    verifyChecksum: true # Verify file integrity
    resumeTransfer: false # Resume interrupted transfers
outputs:
  - name: "fileInfo"
    mapping: "fileInfo"
  - name: "content"
    mapping: "content" # Only if localPath not provided

Attribute Details:

Input Parameter	Type	Description
`connection`	object	File transfer connection object from FileTransfer/Connect
`remotePath`	string	Path to the file on the server
`localPath`	string	Local path to save the downloaded file (optional)
`options.deleteAfterDownload`	boolean	Whether to delete the remote file after download
`options.overwrite`	boolean	Whether to overwrite existing local file
`options.preserveTimestamp`	boolean	Whether to preserve the file's timestamp
`options.verifyChecksum`	boolean	Whether to verify file integrity
`options.resumeTransfer`	boolean	Whether to resume interrupted transfers

Output:

{
  "fileInfo": {
    "name": "order123.edi",
    "path": "/tmp/downloads/order123.edi",
    "size": 1024,
    "lastModified": "2023-05-15T10:30:00Z",
    "permissions": "rw-r--r--",
    "checksum": "sha256:abc123..."
  },
  "content": "ISA*00*          *00*          *ZZ*ABCDEFGH      *ZZ*123456789      *210101*1030*U*00401*000000001*0*P*>\n..."
}

FileTransfer/UploadFile

The FileTransfer/UploadFile task uploads a file to a server.

task: "FileTransfer/UploadFile@1"
name: fileTransferUpload
inputs:
  connection: "{{ fileTransferConnect.connection }}"
  localPath: "/tmp/uploads/shipment456.edi" # Either localPath or content must be provided
  content: "ISA*00*..." # Either localPath or content must be provided
  remotePath: "/outgoing/shipment456.edi"
  options:
    permissions: "0644" # File permissions (SFTP only)
    createDirectories: true # Create remote directories if needed
    overwrite: true # Overwrite remote file if exists
    verifyChecksum: true # Verify upload integrity
    resumeTransfer: false # Resume interrupted transfers
outputs:
  - name: "fileInfo"
    mapping: "fileInfo"

Attribute Details:

Input Parameter	Type	Description
`connection`	object	File transfer connection object from FileTransfer/Connect
`localPath`	string	Local path of the file to upload
`content`	string	File content to upload (alternative to localPath)
`remotePath`	string	Path to save the file on the server
`options.permissions`	string	File permissions in octal format (SFTP only)
`options.createDirectories`	boolean	Whether to create remote directories if needed
`options.overwrite`	boolean	Whether to overwrite existing remote file
`options.verifyChecksum`	boolean	Whether to verify upload integrity
`options.resumeTransfer`	boolean	Whether to resume interrupted transfers

Output:

{
  "fileInfo": {
    "name": "shipment456.edi",
    "path": "/outgoing/shipment456.edi",
    "size": 2048,
    "uploaded": true,
    "permissions": "rw-r--r--",
    "timestamp": "2023-05-15T14:30:00Z",
    "checksum": "sha256:def456..."
  }
}

FileTransfer/DeleteFile

The FileTransfer/DeleteFile task deletes a file from a server.

task: "FileTransfer/DeleteFile@1"
name: fileTransferDelete
inputs:
  connection: "{{ fileTransferConnect.connection }}"
  remotePath: "/incoming/order123.edi"
  options:
    verify: true # Verify file exists before deletion
outputs:
  - name: "success"
    mapping: "success"
  - name: "fileInfo"
    mapping: "fileInfo"

Attribute Details:

Input Parameter	Type	Description
`connection`	object	File transfer connection object from FileTransfer/Connect
`remotePath`	string	Path to the file to delete on the server
`options.verify`	boolean	Whether to verify file exists before deletion

Output:

{
  "success": true,
  "fileInfo": {
    "name": "order123.edi",
    "path": "/incoming/order123.edi",
    "deletedAt": "2023-05-15T16:30:00Z"
  }
}

FileTransfer/MoveFile

The FileTransfer/MoveFile task moves or renames a file on a server.

task: "FileTransfer/MoveFile@1"
name: fileTransferMove
inputs:
  connection: "{{ fileTransferConnect.connection }}"
  sourcePath: "/incoming/order123.edi"
  destinationPath: "/processed/order123.edi"
  options:
    createDirectories: true # Create destination directories if they don't exist
    overwrite: true # Overwrite destination file if exists
    atomic: true # Ensure atomic move operation
outputs:
  - name: "success"
    mapping: "success"
  - name: "fileInfo"
    mapping: "fileInfo"

Attribute Details:

Input Parameter	Type	Description
`connection`	object	File transfer connection object from FileTransfer/Connect
`sourcePath`	string	Path to the source file on the server
`destinationPath`	string	Path to move/rename the file to
`options.createDirectories`	boolean	Whether to create destination directories
`options.overwrite`	boolean	Whether to overwrite existing destination file
`options.atomic`	boolean	Whether to ensure atomic move operation

Output:

{
  "success": true,
  "fileInfo": {
    "name": "order123.edi",
    "sourcePath": "/incoming/order123.edi",
    "destinationPath": "/processed/order123.edi",
    "size": 1024,
    "lastModified": "2023-05-15T15:30:00Z",
    "permissions": "rw-r--r--"
  }
}

Examples

Multi-Protocol EDI Processing Workflow

This example demonstrates connecting to different protocols for incoming and outgoing file transfers.

# Connect to SFTP for incoming files
task: "FileTransfer/Connect@1"
name: connectIncoming
inputs:
  host: "{{ partner.incomingHost }}"
  protocol: "sftp"
  username: "{{ partner.username }}"
  authentication:
    type: "privateKey"
    privateKey: "{{ secrets.PARTNER_PRIVATE_KEY }}"
    passphrase: "{{ secrets.KEY_PASSPHRASE }}"
  options:
    timeout: 60
    hostKeyVerification:
      enabled: true
      hostKey: "{{ partner.expectedHostKey }}"
outputs:
  - name: "incomingConnection"
    mapping: "connection"

# Connect to FTPS for outgoing files
task: "FileTransfer/Connect@1"
name: connectOutgoing
inputs:
  host: "{{ partner.outgoingHost }}"
  protocol: "ftps"
  port: 990
  username: "{{ partner.username }}"
  authentication:
    type: "password"
    password: "{{ secrets.PARTNER_PASSWORD }}"
  options:
    passive: true
    timeout: 60
outputs:
  - name: "outgoingConnection"
    mapping: "connection"

# List and download EDI files
task: "FileTransfer/ListFiles@1"
name: listIncomingFiles
inputs:
  connection: "{{ connectIncoming.incomingConnection }}"
  directory: "/incoming"
  pattern: "*.edi"
  sortBy: "lastModified"
  sortOrder: "asc"
outputs:
  - name: "ediFiles"
    mapping: "files"

task: "Utilities/ForEach@1"
name: processFiles
inputs:
  items: "{{ listIncomingFiles.ediFiles }}"
  parallel: true
  maxParallel: 3
  task:
    task: "FileTransfer/DownloadFile@1"
    name: downloadFile
    inputs:
      connection: "{{ connectIncoming.incomingConnection }}"
      remotePath: "{{ item.path }}"
      options:
        deleteAfterDownload: false
        verifyChecksum: true
    outputs:
      - name: "fileContent"
        mapping: "content"

    task: "EDI/Parse@1"
    name: parseEDI
    inputs:
      ediData: "{{ downloadFile.fileContent }}"
      validateSchema: true
    outputs:
      - name: "parsedData"
        mapping: "result"

    task: "FileTransfer/MoveFile@1"
    name: moveToProcessed
    inputs:
      connection: "{{ connectIncoming.incomingConnection }}"
      sourcePath: "{{ item.path }}"
      destinationPath: "/processed/{{ item.name }}"
      options:
        createDirectories: true
        atomic: true

# Generate and upload acknowledgment
task: "EDI/Acknowledge@1"
name: generateAck
inputs:
  ediData: "{{ processFiles.parsedData }}"
  acknowledgmentType: "997"
  status: "A"
outputs:
  - name: "ackEDI"
    mapping: "ediString"

task: "FileTransfer/UploadFile@1"
name: uploadAck
inputs:
  connection: "{{ connectOutgoing.outgoingConnection }}"
  content: "{{ generateAck.ackEDI }}"
  remotePath: "/outgoing/ACK_{{ formatDate now 'yyyyMMddHHmmss' }}.edi"
  options:
    createDirectories: true
    verifyChecksum: true
outputs:
  - name: "uploadResult"
    mapping: "fileInfo"

# Disconnect from both servers
task: "FileTransfer/Disconnect@1"
name: disconnectIncoming
inputs:
  connection: "{{ connectIncoming.incomingConnection }}"

task: "FileTransfer/Disconnect@1"
name: disconnectOutgoing
inputs:
  connection: "{{ connectOutgoing.outgoingConnection }}"

Simple FTP File Upload

This example shows a basic file upload using FTP protocol.

task: "FileTransfer/Connect@1"
name: connectFTP
inputs:
  host: "ftp.partner.com"
  protocol: "ftp"
  username: "{{ secrets.FTP_USERNAME }}"
  authentication:
    type: "password"
    password: "{{ secrets.FTP_PASSWORD }}"
  options:
    passive: true
    timeout: 30
outputs:
  - name: "ftpConnection"
    mapping: "connection"

task: "Document/Generate@1"
name: generateReport
inputs:
  template: "shipment-report"
  data: "{{ workflow.input.shipmentData }}"
  format: "csv"
outputs:
  - name: "reportContent"
    mapping: "content"

task: "FileTransfer/UploadFile@1"
name: uploadReport
inputs:
  connection: "{{ connectFTP.ftpConnection }}"
  content: "{{ generateReport.reportContent }}"
  remotePath: "/reports/shipment_{{ formatDate now 'yyyyMMdd' }}.csv"
  options:
    createDirectories: true
    overwrite: true
outputs:
  - name: "uploadInfo"
    mapping: "fileInfo"

task: "FileTransfer/Disconnect@1"
name: disconnectFTP
inputs:
  connection: "{{ connectFTP.ftpConnection }}"

SFTP with Advanced Security

This example demonstrates SFTP with enhanced security features.

task: "FileTransfer/Connect@1"
name: secureConnect
inputs:
  host: "secure.partner.com"
  protocol: "sftp"
  port: 2222 # Custom SFTP port
  username: "{{ secrets.SFTP_USERNAME }}"
  authentication:
    type: "privateKey"
    privateKey: "{{ secrets.SFTP_PRIVATE_KEY }}"
    passphrase: "{{ secrets.SFTP_KEY_PASSPHRASE }}"
  options:
    timeout: 45
    hostKeyVerification:
      enabled: true
      hostKey: "{{ secrets.PARTNER_HOST_KEY }}"
outputs:
  - name: "secureConnection"
    mapping: "connection"

task: "FileTransfer/DownloadFile@1"
name: secureDownload
inputs:
  connection: "{{ secureConnect.secureConnection }}"
  remotePath: "/secure/confidential.xml"
  localPath: "/tmp/secure/confidential.xml"
  options:
    deleteAfterDownload: true
    preserveTimestamp: true
    verifyChecksum: true
outputs:
  - name: "downloadInfo"
    mapping: "fileInfo"

task: "FileTransfer/Disconnect@1"
name: secureDisconnect
inputs:
  connection: "{{ secureConnect.secureConnection }}"

Best Practices

1. Protocol Selection

FTP: Use only for legacy systems that don't support secure protocols
- Limited security features
- Passwords transmitted in plain text
- Suitable for internal networks only
FTPS: Preferred for secure FTP when SFTP is not available
- Encrypted data transmission
- Compatible with existing FTP infrastructure
- Supports both implicit and explicit modes
SFTP: Most secure option, recommended for new integrations
- SSH-based encryption
- Key-based authentication support
- Host key verification
- File permission management

2. Security Configuration

SFTP with Key-Based Authentication (Recommended)

authentication:
  type: "privateKey"
  privateKey: "{{ secrets.SFTP_PRIVATE_KEY }}"
  passphrase: "{{ secrets.KEY_PASSPHRASE }}"
options:
  hostKeyVerification:
    enabled: true
    hostKey: "{{ secrets.EXPECTED_HOST_KEY }}"

FTPS with Strong Settings

protocol: "ftps"
port: 990 # Implicit FTPS
options:
  passive: true
  timeout: 30

Security Checklist

Never store credentials in plain text
Use environment variables or secure secret management
Rotate credentials regularly
Enable host key verification for SFTP
Use private networks or VPN for sensitive data

3. Connection Management

Always use FileTransfer/Disconnect to close connections
Set appropriate timeouts based on file sizes and network conditions
Use connection pooling for high-volume operations
Implement connection retry logic for network failures

task: "FileTransfer/Connect@1"
name: connectWithRetry
inputs:
  host: "{{ partner.host }}"
  protocol: "sftp"
  # ... other inputs
onError:
  action: "retry"
  maxAttempts: 3
  delaySeconds: 5
  escalate: true

4. File Operations

Enable Integrity Verification

options:
  verifyChecksum: true
  resumeTransfer: true # For large files

Atomic Operations

options:
  atomic: true # Ensures move operations are atomic

Error Handling

Implement proper error handling and retry logic
Validate file integrity after transfers
Use temporary directories for processing
Clean up temporary files after operations

5. Directory Structure

Maintain a clear directory structure on file transfer servers:

/incoming/     # New files to process
/processing/   # Files currently being processed
/processed/    # Successfully processed files
/error/        # Files that failed processing
/outgoing/     # Files to send to partners
/archive/      # Historical files for audit

6. Monitoring and Logging

Log all file transfer operations with timestamps
Monitor transfer speeds and connection reliability
Set up alerts for failed transfers
Track file processing metrics
Maintain audit trails for compliance

task: "Utilities/Log@1"
name: logTransfer
inputs:
  level: "info"
  message: "File transfer completed: {{ uploadInfo.fileInfo.name }}"
  metadata:
    fileSize: "{{ uploadInfo.fileInfo.size }}"
    transferTime: "{{ uploadInfo.duration }}"
    protocol: "{{ connection.protocol }}"

7. Performance Optimization

Use parallel processing for multiple file operations
Implement file size limits to prevent timeouts
Consider compression for large files
Use resume capabilities for interrupted transfers
Optimize batch sizes for list operations

task: "Utilities/ForEach@1"
name: parallelDownloads
inputs:
  items: "{{ fileList.files }}"
  parallel: true
  maxParallel: 3 # Limit concurrent connections

Security Considerations

Authentication Security

Never store credentials in plain text
Use environment variables or secure secret management systems
Rotate credentials regularly according to security policies
Use key-based authentication for SFTP when possible
Implement strong passphrase policies for encrypted keys

Network Security

Enable host key verification for SFTP connections
Use FTPS instead of plain FTP for encrypted transfers
Implement connection timeouts to prevent hanging connections
Consider using VPN or private networks for sensitive data
Monitor for suspicious connection patterns

File Security

Verify file checksums for critical transfers
Use temporary directories for file processing
Clean up temporary files after processing
Implement file size limits to prevent abuse
Encrypt sensitive files before transmission

Access Control

Implement least privilege access principles
Use separate credentials for different partners
Regularly audit file transfer permissions
Monitor file access patterns
Implement session logging and tracking

Error Handling

Connection Errors

Common connection issues and their solutions:

Connection Timeout: Increase timeout values or check network connectivity
Authentication Failed: Verify credentials and authentication method
Host Key Verification Failed: Update expected host key or disable verification temporarily
Protocol Not Supported: Verify server protocol support and configuration

Transfer Errors

File Not Found: Verify file paths and permissions
Insufficient Space: Check available disk space on target systems
Permission Denied: Verify file and directory permissions
Transfer Interrupted: Use resume capabilities for large files

Retry Strategies

# Exponential backoff retry
onError:
  action: "retry"
  maxAttempts: 3
  delaySeconds: [5, 10, 20]
  escalate: true

# Custom error handling
onError:
  action: "custom"
  handler: "handleTransferError"
  context:
    operation: "download"
    filePath: "{{ remotePath }}"

Error Logging

Maintain comprehensive error logs for troubleshooting:

task: "Utilities/Log@1"
name: logError
inputs:
  level: "error"
  message: "File transfer failed: {{ error.message }}"
  metadata:
    operation: "{{ operation }}"
    filePath: "{{ filePath }}"
    errorCode: "{{ error.code }}"
    timestamp: "{{ now }}"

Performance Tuning

Connection Optimization

Use persistent connections for multiple operations
Implement connection pooling for high-volume scenarios
Set appropriate buffer sizes for network conditions
Monitor connection establishment times

Transfer Optimization

Use parallel transfers for multiple files
Implement compression for text-based files
Optimize chunk sizes for large file transfers
Use resume capabilities to handle interruptions

Resource Management

Limit concurrent connections to prevent server overload
Implement rate limiting for high-frequency operations
Monitor memory usage during large file transfers
Use streaming for very large files to reduce memory footprint

Overview​

YAML Structure​

Attribute Description​

FileTransfer/Connect​

FileTransfer/Disconnect​

FileTransfer/ListFiles​

FileTransfer/DownloadFile​

FileTransfer/UploadFile​

FileTransfer/DeleteFile​

FileTransfer/MoveFile​

Examples​

Multi-Protocol EDI Processing Workflow​

Simple FTP File Upload​

SFTP with Advanced Security​

Best Practices​

1. Protocol Selection​

2. Security Configuration​

SFTP with Key-Based Authentication (Recommended)​

FTPS with Strong Settings​

Security Checklist​

3. Connection Management​

4. File Operations​

Enable Integrity Verification​

Atomic Operations​

Error Handling​

5. Directory Structure​

6. Monitoring and Logging​

7. Performance Optimization​

Security Considerations​

Authentication Security​

Network Security​

File Security​

Access Control​

Error Handling​

Connection Errors​

Transfer Errors​

Retry Strategies​

Error Logging​

Performance Tuning​

Connection Optimization​

Transfer Optimization​

Resource Management​

Overview

YAML Structure

Attribute Description

FileTransfer/Connect

FileTransfer/Disconnect

FileTransfer/ListFiles

FileTransfer/DownloadFile

FileTransfer/UploadFile

FileTransfer/DeleteFile

FileTransfer/MoveFile

Examples

Multi-Protocol EDI Processing Workflow

Simple FTP File Upload

SFTP with Advanced Security

Best Practices

1. Protocol Selection

2. Security Configuration

SFTP with Key-Based Authentication (Recommended)

FTPS with Strong Settings

Security Checklist

3. Connection Management

4. File Operations

Enable Integrity Verification

Atomic Operations

Error Handling

5. Directory Structure

6. Monitoring and Logging

7. Performance Optimization

Security Considerations

Authentication Security

Network Security

File Security

Access Control

Error Handling

Connection Errors

Transfer Errors

Retry Strategies

Error Logging

Performance Tuning

Connection Optimization

Transfer Optimization

Resource Management