Category: delta

Azure Data Factory is a cloud-based data integration service used to create data-driven workflows for orchestrating and automating data movement and transformation across different data stores and compute services.

Key capabilities

• Data ingestion
• Data orchestration
• Data transformation
• Scheduling and monitoring

The main components are:

A pipeline is a logical grouping of activities that together perform a task such as data ingestion or transformation.

Integration Runtime is the compute infrastructure used by ADF to perform data integration tasks.

• Azure IR Fully managed compute in Azure
• Self-hosted IR Runs on on-premises machines
• Azure SSIS IR Used to run SSIS packages

Types:

A linked service defines the connection information needed for ADF to connect to external resources, example:

• Azure SQL Database
• Data Lake
• databricks
• On-Prem database

A dataset represents the structure of data within a data store.

Solution 1: Watermark column
last_modified_date:
Max(Last_modified)
WHERE last_modified > last_run_time

Solution 2: CDC
transaction log
change tracking

Solution 3: File-based incremental
folder partition by date

Try-Catch pattern
Failure path
Retry policy

Activity
├ success → next step
└ failure → error handling pipeline

Triggers are used to automatically start pipelines.

A tumbling window trigger runs pipelines at fixed time intervals and processes data in discrete time windows.

ADF supports parameters at:

• Linked Service
• plpeline
• dataset

Parallel copy: pipeline success; pipeline duration; failure rate
Staging: use blob staging
Partitioning: split large tables

Monitoring options:
ADF Monitor UI
Azure Monitor
Log Analytics

Databricks offers several ways to store data. First, there’s the Databricks File System for storing and managing files. Then, there’s Delta Lake, an open-source storage layer that adds ACID transactions to Apache Spark, making it more reliable. Databricks also integrates with cloud storage services like AWS S3, Azure Blob Storage, and Google Cloud Storage. Plus, you can connect to a range of external databases, both relational and NoSQL, using JDBC.

Databricks Delta, now known as Delta Lake, is an open-source storage layer that brings ACID transactions to Apache Spark and big data workloads. It enhances Azure Databricks by providing features like:

• ACID transactions for data reliability and consistency.
• Scalable metadata handling for large tables.
• Time travel for data versioning and historical data analysis.
• Schema enforcement and evolution.
• Improved performance with data skipping and Z-ordering

there are several alternative technologies that provide Delta Lake–style Lakehouse capabilities (ACID + schema enforcement + time travel + scalable storage + SQL engine). such as,

• Apache Iceberg
• Apache Hudi
• Snowflake (Iceberg Tables / Unistore)
• BigQuery + BigLake
• AWS Redshift + Lake Formation + Apache Iceberg
• Microsoft Fabric (OneLake + Delta/DQ/DLTS)

Delta lake tables are tables that store data in the delta format. Delta Lake is an extension to existing data lakes,

Delta Live Tables (DLT) is a framework in Azure Databricks for building reliable, automated, and scalable data pipelines using Delta Lake tables.

It simplifies ETL development by managing data dependencies, orchestration, quality checks, and monitoring automatically.

1. Bronze Layer (Raw Data): Ingest raw data from various sources into the Bronze layer. This data is stored as-is, without any transformation.
2. Silver Layer (Cleaned Data, as known Enriched layer): Clean and enrich the data from the Bronze layer. Apply transformations, data cleansing, and filtering to create more refined datasets.
3. Gold Layer (Aggregated Data, as known Curated layer): Aggregate and further transform the data from the Silver layer to create high-level business tables or machine learning features. This layer is used for analytics and reporting.

Z-Ordering in Databricks (specifically for Delta Lake tables) is an optimization technique designed to co-locate related information into the same set of data files on disk.

OPTIMIZE mytable
ZORDER BY (col1, col2);

Liquid Clustering is Databricks’ next-generation data layout optimization for Delta Lake.
It replaces (and is far superior to) Z-Order.

## At creation time:
CREATE TABLE sales
CLUSTER BY (customer_id, event_date)
AS SELECT * FROM source;

## For existing tables:
ALTER TABLE sales
CLUSTER BY (customer_id, event_date);

## Trigger the actual clustering:
OPTIMIZE sales;

## Remove clustering:
ALTER TABLE sales
CLUSTER BY NONE;

Dataframe refers to a specified form of tables employed to store the data within Databricks during runtime. In this data structure, the data will be arranged into two-dimensional rows and columns to achieve better accessibility.

RDD, Resilient Distributed Dataset, is a fault-tolerant, immutable collection of elements partitioned across the nodes of a cluster. RDDs are the basic building blocks that power all of Spark’s computations.

Dataset is an extension of the DataFrame API that provides compile-time type safety and object-oriented programming benefits.

A cache is a temporary storage that holds frequently accessed data, aiming to reduce latency and enhance speed. Caching involves the process of storing data in cache memory.

This is the standard Apache Spark feature. It stores data in the JVM Heap (Memory).

This is a Databricks-specific optimization. It automatically stores copies of remote files (Parquet/Delta) on the local NVMe SSDs of the worker nodes.

both .cache() and .persist() are used to save intermediate results to avoid re-computing the entire lineage. The fundamental difference is that .cache() is a specific, pre-configured version of .persist().

.cache(): This is a shorthand for .persist(StorageLevel.MEMORY_ONLY). It tries to store your data in the JVM heap as deserialized objects.

.persist(): This is the more flexible version. It allows you to pass a StorageLevel to decide exactly how and where the data should be stored (RAM, Disk, or both).

When optimizing Databricks workloads, I focus on several layers. First, I optimize data layout using partitioning and Z-ordering on Delta tables. Second, I improve Spark performance by using broadcast joins, filtering data early, and caching intermediate results. Third, I tune cluster resources such as autoscaling and Photon engine. Finally, I run Delta maintenance commands like OPTIMIZE and VACUUM to manage small files and improve query performance.

Data Layout Optimization:
Partitioning:
CREATE TABLE sales
USING DELTA
PARTITIONED BY (date)

Z-Ordering:
OPTIMIZE sales
ZORDER BY (customer_id);

Liquid Clustering:

Photon is a high-performance query engine built in C++ that accelerates SQL queries and data processing workloads in Azure Databricks. It improves performance by using vectorized processing and optimized execution for modern hardware.

1. Use Azure Key Vault to store and manage secrets securely.
2. Integrate Azure Key Vault with Azure Databricks using Databricks-backed or Azure-backed scopes.
3. Access secrets in notebooks and jobs using the dbutils.secrets API.
   dbutils.secrets.get(scope=”<scope-name>”, key=”<key-name>”)
4. Ensure that secret access policies are strictly controlled and audited.

• Data Classification: Classify data based on sensitivity and compliance requirements.
• Access Controls: Implement role-based access control (RBAC) using Azure Active Directory.
• Data Lineage: Use tools like Databricks Lineage to track data transformations and movement.
• Audit Logs: Enable and monitor audit logs to track access and changes to data.
• Compliance Policies: Implement Azure Policies and Azure Purview for data governance and compliance monitoring.

1. Repartitioning: Repartition the data to balance the workload across nodes and reduce skew.
2. Broadcast Joins: Use broadcast joins for small datasets to avoid shuffle operations.
3. Caching: Cache intermediate results to reduce the need for recomputation.
4. Shuffle Partitions: Increase the number of shuffle partitions to distribute the workload more evenly.
5. Skew Handling: Identify and handle skewed data by adding salt keys or custom partitioning strategies.

• Assessment: Evaluate the existing Hadoop workloads and identify components to be migrated.
• Data Transfer: Use Azure Data Factory or Azure Databricks to transfer data from on-premises HDFS to ADLS.
• Code Migration: Convert Hadoop jobs (e.g., MapReduce, Hive) to Spark jobs and test them in Databricks.
• Optimization: Optimize the Spark jobs for performance and cost-efficiency.
• Validation: Validate the migrated workloads to ensure they produce the same results as on-premises.
• Deployment: Deploy the migrated workloads to production and monitor their performance.