Automated Data Export to Amazon S3

Overview

The Amazon S3 Export feature enables automated, scheduled exports of your FlexPrice data directly to Amazon S3 buckets. This robust, enterprise-grade solution allows you to configure scheduled tasks to export data at regular intervals, trigger manual exports on-demand, and track all export operations with comprehensive execution history. Key Benefits:

Automated Backups: Schedule regular exports to maintain up-to-date data backups
Flexible Scheduling: Export data hourly or daily based on your business needs
Secure Storage: Data encrypted at rest with multiple encryption options
Compression Support: Reduce storage costs with GZIP compression
Execution Tracking: Monitor all export runs with detailed execution history
On-Demand Exports: Trigger manual exports for immediate data exports

Supported Data Types

FlexPrice currently supports exporting the following data types to S3:

Invoices

Finalized invoices without line items, taxes, and payment details. Example Export (First Row):

id,tenant_id,environment_id,customer_id,subscription_id,invoice_number,invoice_type,invoice_status,payment_status,currency,amount_due,amount_paid,amount_remaining,subtotal,total,total_discount,total_tax,adjustment_amount,refunded_amount,billing_sequence,description,due_date,paid_at,voided_at,finalized_at,billing_period,period_start,period_end,invoice_pdf_url,billing_reason,metadata,idempotency_key,version,status,created_by,updated_by,created_at,updated_at
inv_01K76TCFSQX6JY5C43WT2G1EE8,tenant_01K0F7AQCZR5THSG9MHEZE8VKR,env_01K0F7AQEQFTV06P178C1KTR6X,cust_01K4CGWJ4493R9V2Y5PZW7CNRZ,subs_01K4M3DVXZNFRN50P9T5EKB9HV,INV-202510-00079,SUBSCRIPTION,FINALIZED,PENDING,usd,10,0,10,10,10,0,0,0,0,2,Invoice for subscription subs_01K4M3DVXZNFRN50P9T5EKB9HV,2025-10-09T05:47:27Z,,,2025-10-10T10:24:09Z,MONTHLY,2025-09-08T05:47:27Z,2025-10-08T05:47:27Z,https://pay.stripe.com/invoice/...,SUBSCRIPTION_CYCLE,"{""stripe_hosted_invoice_url"":""...""}",subscription_invoice-ff29843f8cde4859,3,published,d95842bc-7d6c-4a4e-8dc9-eb9d0780d9f3,d95842bc-7d6c-4a4e-8dc9-eb9d0780d9f3,2025-10-10T10:24:08Z,2025-10-10T10:24:12Z

Full Example File: Download invoice-example.csv to see the complete CSV structure.

Events

Feature usage events and billing-related activity data. Example Export (First Row):

id,tenant_id,environment_id,external_customer_id,customer_id,subscription_id,sub_line_item_id,price_id,meter_id,feature_id,event_name,source,timestamp,ingested_at,period_id,qty_total,properties,unique_hash
event_01K8D3HHYJVAC90K3X8R0YZA5N,tenant_01K0F7AQCZR5THSG9MHEZE8VKR,env_01K0F7AQEQFTV06P178C1KTR6X,cus_TIciSJDL8QS8OU,cust_01K8D2PERMJ6TX8ATP9C8WCP1Z,subs_01K8D3BJ78M7F0FCPFBBXJWJPG,subs_line_01K8D2Q72ESN6FPP72RA2SWWQR,price_01K42Q7VQEDETDGSA80A8WRJ2Y,meter_01K42Q6R916Q4KB9MC6S46PWQ2,feat_01K42Q6R941C3QS6Q59CGGSPFB,api_call,onboarding,2025-10-25T07:15:20Z,1970-01-01T00:00:00Z,1761376322000,20,"{""api_calls"":20}",baae5efc04d556dc97407d259169b46390946ec4330c974304e70b037aebadcb

Full Example File: Download events-example.csv to see the complete CSV structure.

Setting Up S3 Export

Prerequisites

Before configuring S3 exports, ensure you have:

An AWS account with S3 access
An S3 bucket created in your desired region
AWS credentials (Access Key ID and Secret Access Key)
Appropriate IAM permissions for S3 operations

Step 1: Create an S3 Connection

An S3 connection stores your AWS credentials and serves as the authentication mechanism for all export operations.

Required Information:

Connection Name: A descriptive name for your S3 connection (e.g., “Production S3 Export”)
AWS Access Key ID: Your AWS access key ID
- Format: Starts with AKIA for permanent credentials
- Format: Starts with ASIA for temporary/session credentials
- Required for authentication
AWS Secret Access Key: Your AWS secret access key
- This credential is encrypted and stored securely
- Never exposed in API responses
AWS Session Token (Optional): Required only for temporary credentials
- Use when working with assumed roles or temporary credentials
- Leave empty for permanent IAM user credentials

Security Best Practice: Create a dedicated IAM user with minimal permissions for S3 exports. Grant only the necessary S3 permissions (s3:PutObject, s3:GetObject, s3:ListBucket).

AWS IAM Policy Example

Create an IAM policy with the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "FlexPriceS3ExportAccess",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket-name/*",
        "arn:aws:s3:::your-bucket-name"
      ]
    },
    {
      "Sid": "HeadObject",
      "Effect": "Allow",
      "Action": "s3:HeadObject",
      "Resource": "*"
    }
  ]
}

Replace your-bucket-name with your actual S3 bucket name. Ensure the IAM user or role has this policy attached before creating the connection.

Step 2: Configure a Scheduled Task

Once your S3 connection is established, create a scheduled task to automate exports.

Configuration Options

1. Connection

Select the S3 connection created in Step 1
Each scheduled task can use different connections if needed

2. Entity Type (Required) Choose what data to export:

Entity Type	Description	Data Included
invoice	Finalized invoices	All invoice data, without line items, taxes, payments
events	Feature usage events	Event data, usage metrics, customer activity

3. Schedule Interval (Required) Select how frequently exports should run:

Interval	Frequency	Cron Schedule	Best For
hourly	Every hour at :15 minutes past	`15 * * * *`	Real-time data needs, high-frequency updates
daily	Every day at 00:15 AM	`15 0 * * *`	Standard backups, daily reporting

Why the 15-minute buffer?Scheduled exports include a 15-minute buffer after the interval boundary to ensure all data has been ingested and processed. For example, an hourly export scheduled at 10:15 AM will export data from 9:00 AM to 10:00 AM.

4. S3 Configuration (Required) Configure S3-specific settings for your exports:

Bucket (Required)

The name of your S3 bucket where files will be stored
Example: flexprice-exports-prod
Must exist before creating the scheduled task

Region (Required)

AWS region where your bucket is located
Example: us-west-2, eu-central-1, ap-south-1
Must match your bucket’s region

Key Prefix (Optional)

Directory path prefix for organizing files in your bucket
Example: flexprice-exports/invoices/
Example: production/events/2024/
Files will be stored as: {key_prefix}/{filename}.csv

Use key prefixes to organize exports by environment, data type, or date. This makes it easier to manage and query exported data.

Compression (Optional)

Control file compression to reduce storage costs:

Option	Description	File Extension	Compression Ratio
none	No compression (default)	`.csv`	N/A (original size)
gzip	GZIP compression	`.csv.gz`	~70-80% size reduction

When to use compression:

Large datasets with many records
Storage cost optimization
Bandwidth-limited environments

When to skip compression:

Small datasets (< 1MB)
Files need to be immediately readable
Downstream systems don’t support GZIP

Encryption (Optional)

Secure your data at rest with server-side encryption:

Option	Description	Key Management	Best For
AES256	S3-managed AES-256 encryption (default)	AWS manages keys	Standard security needs
aws:kms	AWS KMS encryption	You control keys via KMS	Compliance requirements
aws:kms:dsse	KMS with dual-layer encryption	Enhanced KMS security	Maximum security needs

All encryption options are server-side encryption (SSE). Data is encrypted at rest in S3. AES256 is the default and recommended for most use cases.

5. Enable/Disable Toggle

Enable the scheduled task to start automatic exports
Disable to pause exports without deleting the configuration
Can be toggled at any time

Step 3: Verify Configuration

After creating a scheduled task:

Verify Connection: Ensure your S3 connection is validated and active
Check Schedule: Confirm the interval matches your requirements
Review S3 Settings: Double-check bucket name, region, and paths
Test with Manual Export: Trigger a manual export to verify the setup

Viewing Export Details

Once you’ve created an export task, you can view its configuration and status by clicking on the task. The export details page shows:

Basic Information: Status, entity type, interval, and connection
S3 Configuration: Bucket, region, key prefix, compression, and encryption settings
Timestamps: Creation date and last update time

Manual Export

Manual Export allows you to trigger an immediate export without waiting for the next scheduled execution. This is useful for:

Testing your export configuration
Backfilling historical data
Responding to urgent data requests
Exporting specific time ranges

How to Trigger a Manual Export

Navigate to the export details page and click the “Manual Export” button to trigger a manual export.

Option 1: Automatic Time Range (Default)

Trigger a manual export without specifying time ranges. The system automatically calculates the export window based on the task’s interval: Example - Hourly Task:

Current time: 10:30 AM
Export window: 10:00 AM to 11:00 AM (current hour)

Example - Daily Task:

Current date: October 16, 2025
Export window: October 16, 00:00 to October 17, 00:00 (current day)

API Request:

POST /api/v1/tasks/scheduled/{task_id}/run

Response:

{
  "workflow_id": "task_export_01HK8DBMBN87CY36DEFY62PEXBT",
  "message": "Manual export triggered successfully",
  "start_time": "2025-10-16T10:00:00Z",
  "end_time": "2025-10-16T11:00:00Z",
  "mode": "manual"
}

Option 2: Custom Time Range

Specify exact start and end times for the export: API Request:

POST /api/v1/tasks/scheduled/{task_id}/run
Content-Type: application/json

{
  "start_time": "2025-10-01T00:00:00Z",
  "end_time": "2025-10-15T23:59:59Z"
}

Response:

{
  "workflow_id": "task_export_01HK8DBMBN87CY36DEFY62PEXBT",
  "message": "Manual export triggered successfully",
  "start_time": "2025-10-01T00:00:00Z",
  "end_time": "2025-10-15T23:59:59Z",
  "mode": "custom"
}

Custom time ranges must be valid timestamps in RFC3339 format (ISO 8601). The end time must be after the start time.

Manual Export Behavior

Independent Execution: Manual exports don’t affect the regular schedule
Parallel Execution: Can run alongside scheduled exports
Tracked Separately: Appears in the Runs tab with “manual” or “custom” mode
Same Configuration: Uses the same S3 settings as the scheduled task
Immediate Processing: Starts execution within seconds

Runs Tab - Execution History

The Runs tab displays a comprehensive history of all export executions for a scheduled task, including both automatic scheduled runs and manual exports.

Each run record shows the execution status, time range, file details, and any errors that occurred. Use this to monitor export health and access exported files.

Accessing Exported Files

Each completed run provides a direct S3 file URL: File Naming Convention:

{entity_type}-{start_time}-{end_time}.csv

Examples:

invoice-251016100000-251016110000.csv
events-251016000000-251017000000.csv.gz

Time Format in Filename:

YYMMDDHHMMSS (Year, Month, Day, Hour, Minute, Second)
Example: 251016100000 = October 16, 2025, 10:00:00

Troubleshooting Failed Runs

When a run fails, the error summary provides details about what went wrong: Common Failure Reasons:

Error	Cause	Solution
Connection Failed	Invalid AWS credentials	Verify and update S3 connection credentials
Bucket Not Found	S3 bucket doesn’t exist or wrong region	Check bucket name and region in task configuration
Permission Denied	Insufficient IAM permissions	Review and update IAM policy
No Data Found	No records in the specified time range	Verify the time range or check data availability
Timeout	Export took longer than 15 minutes	Contact support or split into smaller time ranges

Best Practices

Use Descriptive Connection NamesName your S3 connections based on environment and purpose (e.g., “Production Invoice Exports”, “Staging Events Backup”).

Organize with Key PrefixesUse key prefixes to organize exports by environment, data type, and date:

production/invoices/2025/
staging/events/

Enable Compression for Large DatasetsUse GZIP compression for exports with thousands of records to reduce storage costs by 70-80%.

Choose the Right Interval

Hourly: Real-time analytics, high-frequency reporting
Daily: Standard backups, daily batch processing

Test with Manual ExportAlways test a new scheduled task with a manual export before relying on automatic execution.

Monitor Execution HistoryRegularly review the Runs tab to ensure exports are completing successfully and adjust configuration if needed.

Use IAM Best Practices

Create dedicated IAM users for exports
Grant minimal required permissions
Rotate credentials regularly
Enable CloudTrail for audit logging

Plan for S3 Lifecycle PoliciesConfigure S3 lifecycle policies to automatically archive or delete old exports:

Transition to S3 Glacier after 90 days
Delete files older than 1 year

Troubleshooting

Scheduled Task Not Running

Possible causes:

Task is disabled
S3 connection is invalid
Temporal workflow service is down

Solution:

Verify the task is enabled in the dashboard
Check the S3 connection status
Test with a manual export to isolate the issue
Review Temporal workflow logs

Files Not Appearing in S3

Possible causes:

Incorrect bucket name or region
Missing IAM permissions
Key prefix typo

Solution:

Verify bucket name matches exactly (case-sensitive)
Confirm region matches the bucket’s region
Review IAM policy and ensure s3:PutObject permission
Check key prefix for typos or trailing slashes

Export Timeout Errors

Possible causes:

Very large time range (months of data)
Thousands of records in a single export
S3 upload speed issues

Solution:

Reduce the time range for custom exports
Use hourly intervals instead of daily for high-volume data
Enable GZIP compression to reduce upload time
Contact support if issue persists

Invalid Credentials Error

Possible causes:

AWS credentials expired or rotated
Incorrect Access Key ID or Secret Key
Session token expired (for temporary credentials)

Solution:

Verify credentials in AWS IAM console
Update the S3 connection with new credentials
For temporary credentials, refresh the session token
Ensure the IAM user/role still exists

API Reference

Create Scheduled Task

POST /api/v1/tasks/scheduled
Content-Type: application/json
Authorization: Bearer {token}

{
  "connection_id": "conn_01HK8DBMBN87CY36DEFY62PEXBT",
  "entity_type": "invoice",
  "interval": "daily",
  "enabled": true,
  "job_config": {
    "bucket": "flexprice-exports",
    "region": "us-west-2",
    "key_prefix": "invoices/",
    "compression": "gzip",
    "encryption": "AES256"
  }
}

Update Scheduled Task

PATCH /api/v1/tasks/scheduled/{task_id}
Content-Type: application/json
Authorization: Bearer {token}

{
  "enabled": false
}

Trigger Manual Export

POST /api/v1/tasks/scheduled/{task_id}/run
Content-Type: application/json
Authorization: Bearer {token}

{
  "start_time": "2025-10-01T00:00:00Z",
  "end_time": "2025-10-15T23:59:59Z"
}

List Scheduled Tasks

GET /api/v1/tasks/scheduled?entity_type=invoice&status=published
Authorization: Bearer {token}

Get Task Execution History

GET /api/v1/tasks?scheduled_task_id={task_id}&task_type=EXPORT
Authorization: Bearer {token}

Summary

S3 Export provides a comprehensive solution for automated data exports to Amazon S3. Key takeaways:

Two Data Types: Export invoices and events data
Flexible Scheduling: Hourly or daily automated exports
Secure Storage: Multiple encryption options for data at rest
Manual Export Capability: Trigger manual exports anytime
Execution Tracking: Comprehensive run history and monitoring
Enterprise-Ready: Compression, encryption, and scalable architecture
Easy Integration: Standard CSV format for downstream processing

By implementing S3 Export, you can automate data backups, feed data warehouses, meet compliance requirements, and integrate FlexPrice data with your broader data infrastructure.

Introduction

Getting Started

Event Ingestion

Product Catalogue

Customers

Subscriptions

Wallet

Invoices

Webhooks

Settings

Data Exports

​Overview

​Supported Data Types

​Invoices

​Events

​Setting Up S3 Export

​Prerequisites

​Step 1: Create an S3 Connection

​AWS IAM Policy Example

​Step 2: Configure a Scheduled Task

​Configuration Options

Bucket (Required)

Region (Required)

Key Prefix (Optional)

Compression (Optional)

Encryption (Optional)

​Step 3: Verify Configuration

​Viewing Export Details

​Manual Export

​How to Trigger a Manual Export

​Option 1: Automatic Time Range (Default)

​Option 2: Custom Time Range

​Manual Export Behavior

​Runs Tab - Execution History

​Accessing Exported Files

​Troubleshooting Failed Runs

​Best Practices

​Troubleshooting

​Scheduled Task Not Running

​Files Not Appearing in S3

​Export Timeout Errors

​Invalid Credentials Error

​API Reference

​Create Scheduled Task

​Update Scheduled Task

​Trigger Manual Export

​List Scheduled Tasks

​Get Task Execution History

​Summary

Overview

Supported Data Types

Invoices

Events

Setting Up S3 Export

Prerequisites

Step 1: Create an S3 Connection

AWS IAM Policy Example

Step 2: Configure a Scheduled Task

Configuration Options

Step 3: Verify Configuration

Viewing Export Details

Manual Export

How to Trigger a Manual Export

Option 1: Automatic Time Range (Default)

Option 2: Custom Time Range

Manual Export Behavior

Runs Tab - Execution History

Accessing Exported Files

Troubleshooting Failed Runs

Best Practices

Troubleshooting

Scheduled Task Not Running

Files Not Appearing in S3

Export Timeout Errors

Invalid Credentials Error

API Reference

Create Scheduled Task

Update Scheduled Task

Trigger Manual Export

List Scheduled Tasks

Get Task Execution History

Summary