S3 Bucket Connector

Store device telemetry in Amazon S3 for cost-effective, long-term archival. Data is automatically organized by date partitions and supports JSON, CSV, and Parquet formats for seamless analytics integration.

What is S3

Amazon S3 (Simple Storage Service) is an object storage service offering industry-leading scalability, data availability, and durability. It stores unlimited data objects (up to 5TB each) and is ideal for archiving IoT telemetry data at scale.

Key Features:

Unlimited Scalability: Store petabytes of telemetry data without provisioning storage capacity.

Storage Classes: Automatically tier data from Standard → Infrequent Access → Glacier based on access patterns.

Analytics Integration: Query data directly with Athena, process with Glue ETL, or visualize with QuickSight.

Lifecycle Policies: Auto-archive or delete data after configured retention periods.

Encryption: Server-side encryption (SSE-S3, SSE-KMS) for data at rest.

How it Works

When the S3 connector is enabled, /IOTCONNECT™ buffers and writes telemetry data to your S3 bucket in organized partitions:

Device sends telemetry → /IOTCONNECT™ ingests and validates the data.

Data is buffered → Records are batched for efficient S3 writes (configurable buffer interval).

Partitioned storage → Data is written to paths like: s3://bucket/year=2026/month=06/day=26/deviceId/data.json

Query-ready → Partition structure enables efficient queries with Athena or Spark without scanning all data.

Video Walkthrough: Amazon S3 Connector | Configuration

Watch the complete walkthrough video showing the end-to-end process of configuring the Amazon S3 connector and validating device data storage with the IoTConnect platform.

User Manual

Step 1: Navigate to Connector Settings

Go to Settings > Configuration > Live Data Streaming in the /IOTCONNECT™ portal.

Step 2: Enable S3

Find the S3 connector card and toggle the switch to ON (green).

Step 3: Select Data Types

Choose which data to route: Telemetry, Faulty, Diagnostics, Command History.

Step 4: Run CloudFormation Stack

Click the CloudFormation link. This creates:

An S3 bucket with date-based partition structure

Bucket policy for cross-account access

IAM role with PutObject and ListBucket permissions

Optional lifecycle rules for automatic data tiering

Step 5: Confirm Account ID

Account ID auto-populates after stack completion.

Step 6: Save

Click Save. Telemetry data begins flowing to your S3 bucket.

Use Cases

Long-term Archival: Keep years of telemetry data for compliance, audit, and historical analysis.

Data Lake Foundation: Build a centralized data lake for cross-device analytics and ML training.

Cost-effective Backup: Use Glacier storage class for rarely accessed but retained data at minimal cost.

BI Integration: Connect QuickSight, Tableau, or Power BI to S3 data via Athena for dashboards.

Regulatory Compliance: Meet data retention requirements with immutable storage and versioning.

ⓘ NOTE

S3 pricing is based on storage volume and access frequency. Use lifecycle policies to automatically move older data to cheaper storage classes. Enable versioning for data integrity.