Posted on by William Chen

Data mesh

Data mesh is a decentralized data architecture and organizational model that shifts data ownership from a single central IT team to individual business domains.

Its core idea is: Data should be owned and managed by the business domains that understand that data best, while still following common enterprise governance standards.

Instead of one central data team owning everything, different business domains become responsible for their own data products.

Enterprise
│
├── Customer Domain
│   └── Customer Data Product
│
├── Finance Domain
│   └── Finance Data Product
│
├── Risk Domain
│   └── Risk Data Product
│
├── HR Domain
│   └── Employee Data Product
│
└── Operations Domain
    └── Operations Data Product

Data Mesh, on the other hand, becomes: Domain owns the data, Enterprise governs the data.

Finance Domain → looks after Finance Data
HR Domain → looks after HR Data
Risk Domain → looks after Risk Data
Customer Domain → looks after Customer Data

What is Data Mesh trying to address?

  • Domain Ownership
    Data is owned by the business domains.
  • Data as a Product
    Data must be treated like a product.
  • Self-Serve Data Platform
    The enterprise provides a centralized platform enabling domains to autonomously publish and govern their data.
  • Federated Computational Governance
    The enterprise continues to define standardized rules.
    • Security
    • Privacy
    • Governance
    • Compliance
    • Data Quality
    • Metadata
    • Lineage

What is Domain-Driven Modeling?

Domain-Driven Modeling means: Designing data models around the business domains, business concepts, business processes, and business rules rather than starting from source-system tables.

Domain vs Canonical Model

Canonical Model ≠ Data Mesh

Data Mesh –>Reorganize data ownership, data products, the platform, and governance.

                 Enterprise
                     │
             Data Mesh Governance
                     │
        ┌────────────┼────────────┐
        ↓            ↓            ↓
    Customer       Finance       Risk
     Domain         Domain       Domain
        │            │            │
     Canonical    Canonical    Canonical
      Model         Model        Model
        │            │            │
    Data Product  Data Product  Data Product

Data Mesh targets a different set of challenges: data is owned by the business domains, it must be treated like a product, and the enterprise provides a unified self-serve platform. Ultimately, it requires reorganizing data ownership, data products, the platform, and governance.

What exactly are we modeling in Domain-Driven Design?

Business Domain

First, determine: What business area are we modeling?

  • Customer
  • Payment
  • Risk
  • Mortgage
  • Finance

Business Capability

Within a domain, we can further examine business capabilities

Customer Domain
│
├── Customer Onboarding
├── Customer Identification
├── Customer Profile Management
├── Customer Relationship Management
└── Customer Communication

Capability indicates: What the business domain can do.
Capability 表示：这个业务领域能够做什么。

Business Concept

Then, identify the truly critical business concepts within the domain.
然后识别 Domain 中真正重要的业务概念。

# Customer Domain：
Customer
Customer Profile
Customer Relationship
Customer Address
Customer Contact

# Payment Domain：
Payment
Payment Instruction
Payment Method
Payment Transaction
Payment Status

Business Event

Taking it a step further: What events occur within the domain?

e.g.

  • CustomerCreated
  • CustomerUpdated
  • CustomerAddressChanged
  • PaymentInitiated
  • PaymentCompleted
  • PaymentFailed

In Domain-Driven Design (DDD), things that “happen” are specifically called “Domain Events.”

Using the word “events” makes your phrasing sound highly sophisticated and immediately signals the right architectural concept to other engineers.

Business Rule

Domain has business rules. e.g.

Payment amount > $10,000
↓
Additional approval required

Data Entity

Finally, data modeling

Customer
Account
Payment
Address
Transaction

THen define 

Attributes
Keys
Relationships
Cardinality
Constraints
History

Therefore, the overall process is as follows:

Business Domain
       ↓
Business Capability
       ↓
Business Concept
       ↓
Business Event
       ↓
Business Rule
       ↓
Domain Entity
       ↓
Data Model
       ↓
Data Product

Complete example

Step 1 — Identify Domain

Customer Domain: Manage customer identity, profile and relationship information.

Step 2 — Identify Business Concepts

Customer
Customer Profile
Customer Address
Customer Contact
Customer Relationship

Step 3 — Identify Business Events

CustomerCreated
CustomerUpdated
CustomerAddressChanged
CustomerStatusChanged

Step 4 — Identify Business Rules

e.g. Customer must have a unique enterprise customer ID.

Customer status must be one of:

  • ACTIVE
  • INACTIVE
  • SUSPENDED
  • CLOSED

Step 5 — Build Domain Model

Customer
│
├── Customer Profile
│
├── Customer Address
│
├── Customer Contact
│
└── Customer Relationship

Step 6 — Map Source Systems

CRM
   │
   ├── CRM_CUSTOMER
   └── CRM_CONTACT
             │
             ↓
Core Banking ─────→ Customer Domain
   │
   └── CUSTOMER
             │
             ↓
Mortgage
   │
   └── BORROWER

Step 7 — Canonical Model

Customer
------------------------
customer_id
customer_type
first_name
last_name
date_of_birth
status
created_date
updated_date

Step 8 — Data Product

Customer Domain provides “Customer Data Product”.

e.g. customer.gold_customer. it should has

  • Owner
  • Schema
  • Description
  • Data Quality
  • SLA
  • Security
  • Lineage
  • Documentation
  • Access Policy

As a result, the downstream Risk team does not need direct access to:

  • CRM
  • Core Banking
  • Mortgage

Instead, they consume: Customer Data Product

How to implement in Databricks

                    Enterprise Data Mesh
                            │
              Federated Governance
                            │
        ┌───────────────────┼───────────────────┐
        ↓                   ↓                   ↓
 Customer Domain       Finance Domain       Risk Domain
        │                   │                   │
        ↓                   ↓                   ↓
 Unity Catalog         Unity Catalog        Unity Catalog
        │                   │                   │
        ↓                   ↓                   ↓
 Delta Tables          Delta Tables         Delta Tables
        │                   │                   │
        ↓                   ↓                   ↓
 Data Products         Data Products        Data Products

Catalog / Schema designing

e.g.

customer_catalog
│
├── bronze
├── silver
└── gold

Do not think “One Domain = One Catalog”,
No, implementation choice, it is NOT data Mesh define

Domain-owned tables

-- ============================================================
-- PURPOSE:
-- Create the Customer domain's curated customer entity.
--
-- KEY IDEA:
-- The Customer domain owns the business meaning and quality
-- of this dataset.
-- ============================================================

CREATE TABLE customer_catalog.gold.customer
(
    -- Enterprise-wide business identifier.
    customer_id STRING NOT NULL,

    -- Business classification of the customer.
    customer_type STRING,

    -- Standardized customer name.
    first_name STRING,
    last_name STRING,

    -- Standardized date of birth.
    date_of_birth DATE,

    -- Controlled business status.
    status STRING,

    -- Audit columns.
    created_date TIMESTAMP,
    updated_date TIMESTAMP
)
USING DELTA;

Key Takeaways

Data Mesh is primarily about how an enterprise organizes ownership, products, platforms, and governance around data domains.

Domain: Business responsibility boundary

Domain-Driven Modeling: Business first –> Data model second

Canonical Model: Standardized enterprise representation of data

Data Product: A Data Product is not simply a Delta table.

Proudly powered by WordPress