Databricks for Healthcare Data

Column Masking for PHI Fields

-- Create column masking function for member names
CREATE OR REPLACE FUNCTION
marts_catalog.security.phi_name_mask(col STRING)
RETURN CASE
  WHEN is_member('phi-analyst-group') THEN col
  WHEN is_member('claims-analyst-group') THEN '***MASKED***'
  ELSE NULL
END;

-- Apply to PHI columns in member dimension
ALTER TABLE marts_catalog.member.dim_member
  ALTER COLUMN mbr_first_nm
  SET MASK marts_catalog.security.phi_name_mask;

ALTER TABLE marts_catalog.member.dim_member
  ALTER COLUMN mbr_last_nm
  SET MASK marts_catalog.security.phi_name_mask;

-- Date masking for patients over 89 (HIPAA de-id)
CREATE OR REPLACE FUNCTION
marts_catalog.security.phi_dob_mask(dob DATE)
RETURN CASE
  WHEN is_member('phi-analyst-group') THEN dob
  ELSE DATE_TRUNC('YEAR', dob)  -- show year only
END;

ALTER TABLE marts_catalog.member.dim_member
  ALTER COLUMN mbr_dob
  SET MASK marts_catalog.security.phi_dob_mask;

Row Filters for Population-Level Access

-- Row filter: analysts see only their authorized populations
CREATE OR REPLACE FUNCTION
marts_catalog.security.member_population_filter(
  plan_cd STRING, mbr_state_cd STRING)
RETURN
  is_member('phi-analyst-group')
  OR is_member('data-engineer-group')
  OR EXISTS (
    SELECT 1
    FROM marts_catalog.security.user_population_access
    WHERE user_group   = current_user()
      AND (allowed_plan = plan_cd OR allowed_plan = 'ALL')
      AND (allowed_state = mbr_state_cd OR allowed_state = 'ALL')
  );

-- Apply row filter to member dimension
ALTER TABLE marts_catalog.member.dim_member
  SET ROW FILTER
  marts_catalog.security.member_population_filter
  ON (plan_cd, mbr_state_cd);

Is Databricks HIPAA compliant?

Databricks offers a HIPAA-eligible configuration and will sign a Business Associate Agreement with covered entities and business associates handling protected health information. Databricks runs on your cloud provider infrastructure (AWS, Azure, or GCP) and leverages that provider's HIPAA-eligible services. HIPAA compliance is a shared responsibility — Databricks secures the platform while your organization configures Unity Catalog access controls, encryption, audit logging, and data governance policies appropriate for PHI workloads.

What is Databricks Unity Catalog and why does it matter for healthcare?

Unity Catalog is Databricks's unified governance layer providing centralized access control, data lineage, auditing, and discovery across all Databricks workspaces. For healthcare organizations, Unity Catalog enables fine-grained PHI access controls through row-level security and column masking policies, maintains complete data lineage from raw EDI ingestion through analytics marts, provides a centralized audit log of all PHI-touching queries for HIPAA audit control requirements, and enables data discovery with sensitivity classification for PHI fields across your entire Databricks lakehouse.

What is Delta Lake and how does it help healthcare data engineering?

Delta Lake is an open-source storage layer that adds ACID transaction support, schema enforcement, data versioning, and time travel capabilities to Apache Spark and cloud object storage. For healthcare data engineering, Delta Lake enables reliable upserts for slowly changing dimension member and provider tables, time travel queries for point-in-time eligibility analysis and HEDIS measurement year reproducibility, schema enforcement preventing corrupt EDI or HL7 data from reaching downstream analytics, and Z-ordering optimization that dramatically reduces query scan costs for large claims fact tables.

How does the medallion architecture work for healthcare data on Databricks?

The medallion architecture organizes healthcare data into three progressive layers: the bronze layer stores raw ingested data exactly as received from source systems including raw EDI 837 files, HL7 messages, and EHR extracts; the silver layer applies cleaning, validation, ISO-11179 column renaming, and type enforcement to produce standardized records; and the gold layer contains business-level dimensional models including fact_claim_header, dim_member with SCD Type 2 history, and HEDIS measure fact tables ready for analytics consumption. Each layer is implemented as Delta Lake tables with appropriate access controls.

Should I use Databricks or Snowflake for healthcare data?

The choice depends on your workload mix. Databricks excels at large-scale data engineering pipelines, machine learning and AI workloads, streaming data processing with Structured Streaming, and Python/Spark-heavy transformation work. Snowflake excels at SQL-first analytics, BI dashboard serving with high concurrency, data sharing with payer or provider partners, and teams where analysts outnumber data engineers. Many healthcare organizations use both: Databricks for ingestion, transformation, and ML, and Snowflake or Databricks SQL for analyst-facing query serving.

How do I implement row-level security for PHI in Databricks Unity Catalog?

Use Unity Catalog Row Filters to restrict which rows a user or group can query based on policy logic evaluated at runtime. Create a row filter function that joins the querying user's group membership to an access control mapping table defining authorized member populations, plan codes, or geographic regions. Apply the row filter to tables containing PHI using ALTER TABLE SET ROW FILTER. Row filters in Unity Catalog enforce restrictions even on direct table queries, making them more robust than view-based security for HIPAA minimum necessary access compliance.

How do I connect dbt to Databricks for healthcare transformations?

Use the dbt-databricks adapter which connects to Databricks SQL warehouses or All-Purpose clusters via the Databricks SQL connector. Configure your profiles.yml with your Databricks workspace host, HTTP path from your SQL warehouse or cluster, and a personal access token or service principal OAuth credentials. Set the default materialization to incremental for large healthcare fact tables and configure Delta Lake-specific settings including liquid clustering on service date and payer columns. Use dbt's grant configuration to enforce Unity Catalog access controls on all model outputs.

What is Databricks Structured Streaming for healthcare?

Databricks Structured Streaming enables continuous, near-real-time data processing where new data is processed as it arrives rather than in scheduled batch jobs. Healthcare use cases include streaming HL7 ADT messages from interface engines for real-time bed management analytics, processing claims status change events from clearinghouses within minutes of receipt, streaming lab result feeds from laboratory information systems for clinical alerting, and ingesting patient monitoring device data for ICU analytics. Structured Streaming writes to Delta Lake tables enabling downstream queries on continuously updated data.