Data Lake Gen 2 Connector

Store device telemetry in Azure Data Lake Storage Gen 2 for big data analytics at petabyte scale. Hierarchical namespace with HDFS-compatible access enables efficient analytics with Spark, Databricks, and Synapse.

What is Data Lake Gen 2

Azure Data Lake Storage Gen 2 combines the power of a Hadoop-compatible file system with the scalability and cost-effectiveness of Azure Blob Storage. It is purpose-built for enterprise big data analytics workloads.

Key Features:

Hierarchical Namespace: True directory structure with atomic rename/move operations for efficient ETL.

Petabyte Scale: Store massive volumes of IoT telemetry with no upper storage limit.

HDFS Compatible: Native ABFS driver for direct access from Spark, Hadoop, and Databricks.

Fine-grained ACLs: POSIX-compatible access control at file and directory level.

Multi-format Support: Store Parquet, Delta Lake, ORC, Avro, CSV, and JSON formats.

How it Works

When the Data Lake connector is enabled, /IOTCONNECT™ writes telemetry data to your Data Lake in analytics-optimized format:

Device sends telemetry → /IOTCONNECT™ ingests and processes data.

Data partitioned → Records are organized by date, device, and template for efficient querying.

Written to Data Lake → Data lands in your file system: filesystem/raw/year/month/day/deviceId/

Analytics ready → Query directly from Synapse SQL, Spark, or mount in Databricks workspace.

User Manual

Step 1: Navigate to Connector Settings

Go to Settings > Configuration > Live Data Streaming in the /IOTCONNECT™ portal.

Step 2: Enable Data Lake Gen 2

Find the Data Lake Gen 2 connector card and toggle the switch to ON (green).

Step 3: Select Data Types

Choose which data to store: Telemetry, Faulty, Diagnostics, Command History.

Step 4: Run ARM Template

Click the ARM template link. This deploys:

A Storage Account with hierarchical namespace enabled (Data Lake Gen 2)

File system container with directory structure

Service Principal with Storage Blob Data Contributor role

POSIX ACLs on directories for security

Step 5: Confirm Subscription ID

Subscription ID auto-populates after deployment.

Step 6: Save

Click Save. Telemetry begins flowing to your Data Lake.

Use Cases

Big Data Analytics: Process petabytes of historical telemetry with Spark or Databricks clusters.

ML Training: Build large training datasets from historical device data for machine learning models.

Data Warehousing: Feed Azure Synapse dedicated SQL pools for structured BI reporting.

Delta Lake: Use Delta format for ACID transactions, time-travel, and schema enforcement on IoT data.

Cross-team Analytics: Share data across teams with fine-grained ACL permissions on directories.

ⓘ NOTE

Data Lake Gen 2 combines Blob Storage pricing with hierarchical namespace performance. Use Parquet format for columnar analytics and Delta Lake for transactional consistency on IoT data.