Showing Posts From

Bias mitigation

We stand at a peculiar inflection point in machine learning history. For decades, the prevailing dogma was "more data beats better algorithms." Today, the constraint has inverted: we are drowning in parameter counts but starving for high-fidelity, labeled, and compliant training signals. The modern practitioner doesn't just need data; they need distributional coverage—the long tail of edge cases, the underrepresented demographics, the adversarial perturbations that live in the interstices of the manifold. Real-world collection is slow, expensive, legally fraught (GDPR, CCPA, HIPAA), and inherently biased by the collection mechanism itself. Enter Synthetic Data Generation (SDG): not merely a data augmentation trick, but a rigorous statistical discipline aimed at learning the joint probability distribution $P(X, Y)$ of a domain to sample from it with controllable fidelity. This article dissects the architectures, the evaluation metrics, and the deployment pipelines that separate toy generators from production-grade synthetic data factories. The Statistical Mechanics of Distributional Fidelity At its core, SDG is a density estimation problem. We seek a generator $G_\theta: Z \to X$ mapping a latent prior $p(z)$ (usually Gaussian or Uniform) to the data space $X$, such that the pushforward distribution $p_g(x)$ minimizes a statistical distance to the true data distribution $p_{data}(x)$. In the early GAN era (Goodfellow et al., 2014), this was framed as a minimax game: $\min_G \max_D V(D, G) = \mathbb{E}{x \sim p{data}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))]$. While theoretically converging to Jensen-Shannon divergence minimization, the practical reality of mode collapse and vanishing gradients plagued tabular and high-dimensional structured data. Modern production systems have largely pivoted to Diffusion Probabilistic Models (DPMs) and Conditional Normalizing Flows (CNFs) for tabular/time-series data, and Latent Diffusion Models (LDMs) for high-dimensional perceptual data (images, video, point clouds). The diffusion objective—optimizing the Evidence Lower Bound (ELBO) on the negative log-likelihood via a variational bound on the KL divergence between the forward noising process $q(x_t|x_{t-1})$ and the learned reverse denoising process $p_\theta(x_{t-1}|x_t)$—provides a stable training dynamic and explicit likelihood evaluation, crucial for detecting distributional shift.For tabular data—the backbone of enterprise ML (finance, healthcare, logistics)—the state-of-the-art is TabDDPM (Kotelnikov et al., 2023) or CoDi (Kim et al., 2023). These architectures handle mixed-type variables (continuous, categorical, ordinal) by embedding categoricals into continuous spaces, applying diffusion in the latent space, and projecting back via a classifier head or argmax sampling. Crucially, they model the joint distribution $P(X_{cont}, X_{cat})$, preserving complex correlation structures (e.g., the non-linear dependency between zip_code, income, and credit_score) that marginal generators (like CTGAN or TVAE) often fail to capture. Architecting the Synthetic Pipeline: From Latent Space to Lakehouse A production SDG pipeline is not a Jupyter notebook; it is a distributed data engineering system. It requires versioned model artifacts, automated quality gates, and lineage tracking. We typically structure this as a Medallion Architecture (Bronze/Silver/Gold) adapted for synthetic assets.Bronze (Raw Ingestion & Profiling): Ingest raw production data. Compute statistical fingerprints: marginal distributions (KS-test, Chi-squared), correlation matrices (Pearson, Spearman, Cramér's V), and—critically—Privacy Metrics (k-anonymity, l-diversity, $\epsilon$-Differential Privacy budget accounting). Silver (Training & Validation): Train the generative model. This is where the compute lives. Distributed training (FSDP/DeepSpeed) on A100/H100 clusters. Validation here is not just loss curves. We run Downstream Task Utility (DTU) tests: train a downstream classifier (XGBoost, TabPFN) on synthetic data, evaluate on real holdout set. The metric is the Utility Gap: $\Delta_{AUC} = AUC_{real} - AUC_{syn}$. Target: $\Delta_{AUC} < 0.02$. Gold (Certified Synthetic Datasets): The output artifact. Parquet files partitioned by semantic keys (e.g., event_date, tenant_id), accompanied by a Data Card (Gebru et al., 2021) documenting provenance, fidelity scores, privacy guarantees, and intended use cases.Code Block: Synthetic Data Quality Gate (Python/Great Expectations) This snippet represents a CI/CD gate preventing low-fidelity synthetic data from polluting the feature store. It uses Great Expectations for statistical validation and scikit-learn for the downstream utility check. # synthetic_quality_gate.py import pandas as pd import great_expectations as gx from great_expectations.core import ExpectationSuite from sklearn.model_selection import train_test_split from sklearn.ensemble import HistGradientBoostingClassifier from sklearn.metrics import roc_auc_score import mlflow import numpy as npclass SyntheticDataValidator: def __init__(self, real_data_path: str, synth_data_path: str, target_col: str, utility_threshold: float = 0.02, privacy_epsilon: float = 1.0): self.real_df = pd.read_parquet(real_data_path) self.synth_df = pd.read_parquet(synth_data_path) self.target = target_col self.utility_thresh = utility_threshold self.context = gx.get_context() def _statistical_fidelity_suite(self) -> ExpectationSuite: """Define expectations for marginal and joint distributions.""" suite = self.context.suites.add(ExpectationSuite(name="stat_fidelity")) # 1. Marginal Distribution Parity (KS Test for continuous, Chi2 for categorical) for col in self.real_df.columns: if col == self.target: continue if pd.api.types.is_numeric_dtype(self.real_df[col]): suite.add_expectation( gx.expectations.ExpectColumnKolmogorovSmirnovTestPValueToBeGreaterThan( column=col, partition_object=self.real_df[col], p_value=0.05 ) ) else: suite.add_expectation( gx.expectations.ExpectColumnChiSquareTestPValueToBeGreaterThan( column=col, partition_object=self.real_df[col], p_value=0.05 ) ) # 2. Correlation Structure Preservation (Cramér's V / Pearson) # Custom expectation logic usually required here for matrix comparison return suite def _downstream_utility_check(self) -> float: """Train on Synthetic, Test on Real. Return Utility Gap.""" X_syn = self.synth_df.drop(columns=[self.target]) y_syn = self.synth_df[self.target] X_real = self.real_df.drop(columns=[self.target]) y_real = self.real_df[self.target] # Align columns (handle one-hot encoding drift) X_syn, X_real = X_syn.align(X_real, join='inner', axis=1) model = HistGradientBoostingClassifier(random_state=42, max_iter=200) model.fit(X_syn, y_syn) preds = model.predict_proba(X_real)[:, 1] auc_syn = roc_auc_score(y_real, preds) # Baseline: Train on Real, Test on Real model_real = HistGradientBoostingClassifier(random_state=42, max_iter=200) model_real.fit(X_real, y_real) auc_real = roc_auc_score(y_real, model_real.predict_proba(X_real)[:, 1]) return auc_real - auc_syn def run_validation(self) -> bool: mlflow.start_run(run_name="synthetic_quality_gate") # 1. Statistical Checks suite = self._statistical_fidelity_suite() validator = self.context.get_validator(batch_request=self.synth_df, expectation_suite=suite) results = validator.validate() stat_pass = results.success mlflow.log_metric("statistical_fidelity_pass", int(stat_pass)) # 2. Utility Check utility_gap = self._downstream_utility_check() mlflow.log_metric("utility_gap_auc", utility_gap) utility_pass = utility_gap < self.utility_thresh mlflow.log_metric("utility_pass", int(utility_pass)) # 3. Privacy Check (Simplified: DCR - Distance to Closest Record) # In production, use `opacus` or `diffprivlib` for formal DP-SGD guarantees dcr = self._compute_dcr() mlflow.log_metric("min_dcr", dcr) privacy_pass = dcr > 0.05 # Heuristic threshold mlflow.end_run() if not (stat_pass and utility_pass and privacy_pass): raise ValueError(f"Quality Gate Failed: Stat={stat_pass}, Util={utility_pass}, Priv={privacy_pass}") return True def _compute_dcr(self) -> float: # Simplified Euclidean DCR on normalized numeric subset num_cols = self.real_df.select_dtypes(include=np.number).columns if len(num_cols) == 0: return 1.0 from scipy.spatial import KDTree tree = KDTree(self.real_df[num_cols].values) dists, _ = tree.query(self.synth_df[num_cols].values, k=1) return np.min(dists)if __name__ == "__main__": validator = SyntheticDataValidator( real_data_path="s3://lakehouse/gold/real/train.parquet", synth_data_path="s3://lakehouse/gold/synthetic/v1/train.parquet", target_col="churn_label" ) validator.run_validation() print("✅ Synthetic Data Certified for Production Use.")Bias Mitigation: Counterfactual Generation and Causal Regularization "Bias" in training sets is not a monolith. We distinguish Selection Bias ($P_{train}(X) \neq P_{pop}(X)$), Label Bias ($P_{train}(Y|X) \neq P_{true}(Y|X)$), and Measurement Bias (systematic noise in sensors/annotation). Synthetic data offers a unique lever: interventional generation. Instead of merely learning $P(X|Y)$, we can enforce constraints via Causal Normalizing Flows or Constrained Diffusion. Consider a credit scoring dataset where Gender is a protected attribute $A$. The real data exhibits $P(Y=1|A=Female) < P(Y=1|A=Male)$ due to historical discrimination. A naive generator replicates this disparity. To mitigate, we define a Structural Causal Model (SCM): $$ X \leftarrow f_X(U_X) $$ $$ A \leftarrow f_A(X, U_A) $$ $$ Y \leftarrow f_Y(X, A, U_Y) $$ We want a synthetic distribution $P^(X, A, Y)$ satisfying Counterfactual Fairness (Kusner et al., 2017): $P^(Y_{A \leftarrow a} | X, A=a') = P^*(Y_{A \leftarrow a'} | X, A=a')$. Implementation strategy: Adversarial Debiasing in Latent Space. We add a discriminator $D_{adv}$ to the generator loss that tries to predict $A$ from the latent representation $z$ (or the generated $X$). The generator minimizes $\mathcal{L}{gen} - \lambda \mathcal{L}{adv}$. For diffusion models, this translates to Classifier-Free Guidance where the guidance vector is the gradient of the adversarial loss w.r.t. the noise prediction $\epsilon_\theta(x_t, t)$. Furthermore, for Scarcity (the long tail), we use Few-Shot Diffusion (e.g., DreamBooth/LoRA adapters) or Retrieval-Augmented Generation (RAG) for tabular data: retrieve nearest neighbors from the real dataset for a rare class, condition the generator on this context window, and generate synthetic "neighbors" to densify the minority class manifold. This is effectively Manifold Mixup in the latent space of a pre-trained autoencoder. The Privacy-Utility Frontier: Differential Privacy as a First-Class Citizen You cannot discuss synthetic data in 2024+ without Differential Privacy (DP). Regulators (EU AI Act, US Executive Orders) demand provable bounds. DP-SGD (Abadi et al., 2016) applied to the generator training is the standard: clip per-sample gradients to norm $C$, add Gaussian noise $\mathcal{N}(0, \sigma^2 C^2 I)$. The privacy accountant (RDP or GDP) tracks $(\epsilon, \delta)$. However, DP destroys utility for high-dimensional data ($\epsilon < 10$ often kills fidelity). The architectural pattern winning in industry (e.g., Google, Apple, Ant Group) is Private Pre-training + Public Fine-tuning (PATE-GAN style) or DP-Fine-Tuning of Foundation Models.Pre-train a massive generative foundation model (e.g., Tabular Foundation Model like TabPFN-v2 or a custom LLM for structured data) on public or synthetic data. No privacy cost. Fine-tune the last $k$ layers (or LoRA adapters) on private data using DP-SGD. Generate from the fine-tuned model. The privacy cost $\epsilon$ is incurred only on the small adapter parameters, drastically reducing noise magnitude for equivalent $\epsilon$.This shifts the Privacy-Utility curve significantly. We observe $\Delta_{AUC} < 0.03$ at $\epsilon=1.0$ for tabular classification tasks using DP-LoRA on a TabDDPM backbone, versus $\Delta_{AUC} > 0.15$ for full DP-SGD training from scratch. Code Block: DP-LoRA Fine-tuning Config (YAML/Hydra) Configuration for a differentially private LoRA fine-tuning run using opacus and peft on a hypothetical Tabular Diffusion backbone. # config/dp_lora_finetune.yaml defaults: - /model: tabddpm_base - /privacy: dp_sgd_rdp - /optimizer: adamw - _self_experiment_name: "tabddpm_dp_lora_credit_scoring" seed: 42 device: "cuda" mixed_precision: "bf16"model: _target_: src.models.TabDDPM checkpoint_path: "s3://model-registry/tabddpm-base-v1.pt" freeze_backbone: true lora_config: r: 16 alpha: 32 dropout: 0.05 target_modules: ["denoiser.mlp.layers.*.linear", "denoiser.time_embed"] bias: "none" task_type: "FEATURE_EXTRACTION"privacy: _target_: opacus.PrivacyEngine noise_multiplier: 1.2 # Sigma max_grad_norm: 1.0 # Clipping threshold C target_epsilon: 2.0 # Target epsilon for accounting target_delta: 1e-5 epochs: 50 sample_rate: 0.01 # Batch size / Dataset size accountant: "rdp" # Renyi DP Accountant secure_mode: true # Secure RNG for noiseoptimizer: _target_: torch.optim.AdamW lr: 1e-4 weight_decay: ### Related Insights- [Rescue Book Review and Summary](/blog/rescue) - [Masala Dosa: The Crispy Crepe That Conquered the World](/blog/masala-dosa-the-crispy-crepe-that-conquered-the-world) - [Mrs. Frisby and the Rats of NIMH Book Review and Summary](/blog/mrs-frisby-and-the-rats-of-nimh)

We use cookies (including third-party cookies) to provide you with the best possible experience and to serve personalized ads. By clicking "Accept", you consent to the use of all cookies. Visit our Privacy Policy for more information.