Partner Solution

Contextualized OT Data Pipeline into Snowflake

How can we reliably collect vendor-agnostic OT data from the shopfloor, enrich it with asset and production context at the edge, and deliver a governed, queryable copy into Snowflake for analytics, ML and enterprise consumption?

Overview

A hybrid edge-to-cloud architecture where Industrial Edge captures, normalizes and contextualizes OT telemetry and events, then delivers them to Snowflake using both file-based and streaming ingestion patterns. Snowflake serves as the enterprise data cloud for landing, curated and analytical layers, enabling near‑real‑time analytics, model scoring, operational apps and integrations with MES/ERP/SCADA systems. The design prioritizes data consistency, security, resilience and vendor-agnostic interoperability.

Learn more

Detailed architecture

Edge collection and contextualization (Industrial Edge)

Industrial Edge runs on-prem devices close to the shopfloor and connects to vendor-agnostic automation equipment via OT connectors (OPC UA, Modbus, EtherNet/IP, etc.). It acquires raw telemetry, alarms and events.
At the edge, data is pre-processed: filtering, compression, timestamp normalization, enrichment with asset metadata (asset hierarchies, work order / batch context), and local aggregation to reduce cloud bandwidth.
An internal databus (MQTT / Unified Namespace) or Industrial Information Hub propagates harmonized topic streams for downstream components and local consumers.

See detailed architecture

architecture hub snowflake detailed graphic no text

Description

Protocol and format bridging

FFT DataBridge (Edge App) transforms and prepares data for file-based ingestion. It buffers to handle connection loss, batches data intelligently, authenticates securely with Snowflake key‑pair auth and writes to cloud stages for Snowpipe processing.
Snowflake Connector (Edge App) subscribes to the databus for continuous streams, performs schema validation and health checks, buffers short outages in memory, and uses Snowpipe Streaming to insert rows directly into Snowflake with low latency.

Snowflake data platform

JSON-based ingestion via Stage + Pipe (FFT DataBridge): raw payloads land in a staging area; Snowpipe / Tasks create curated tables and historical archives.
Direct row-based ingestion (Snowpipe Streaming via Snowflake Connector): continuous, low-latency availability of operational rows for dashboards and monitoring.
Transformation pipelines (SQL, Snowpark, Streams & Tasks) produce curated, time-aligned and context-enriched datasets for BI and ML.
Snowflake provides governance (access control, masking, lineage), scaling and cross-cloud capabilities for enterprise consumption.
Snowflake runs natively and consistently across the major cloud providers (AWS, Microsoft Azure and Google Cloud Platform), offering true cross‑cloud deployment, replication and data mobility.

Values & benefits

Single, governed source of truth

Snowflake stores raw and curated OT data, enabling consistent analytics and reporting.

Reduced cloud cost and bandwidth

Edge-side aggregation and filtering lower data volume while preserving fidelity where needed.

Faster ML and analytics

Snowpark and Streams & Tasks accelerate feature engineering, model training and deployment.

Operational resilience

Edge buffering (ring memory, file buffers) and retry logic maintain continuity during network outages.

Security and governance

End-to-end encryption, key-pair authentication, role-based access and Snowflake controls protect sensitive operational data.

Components

Industrial Edge

• Hosts OT connectors, apps, local databus and buffer storage

• Responsibilities: secure acquisition, enrichment, filtering, aggregation and orchestration

• Managed centrally via Industrial Edge Management

FFT DataBridge (Edge App)

• Buffers data using ring memory and file persistence to survive extended outages

• Handles retries, connection loss, batching and scheme packaging

• Authenticates securely using Snowflake key-pair authentication

Snowflake Connector (Edge App)

• Subscribes to the Industrial Edge databus for continuous data streams

• Buffers data in memory for short outages

• Performs schema validation and health checks via JDBC

• Streams data rows directly into Snowflake via Snowpipe

Snowflake Data Cloud

• Landing/Raw zone: preserves original JSON payloads and metadata for auditability

• Curated/Analytical zone: structured tables used for reporting, ML and operational apps

• JSON-based ingestion

• Direct row-based ingestion