Training-based versus training-free
differential privacy for data synthesis

Introduction

Why does this matter?

Device telemetry, including hardware specs, usage patterns, and network activity, helps product teams improve software, but it contains sensitive information about individuals. Releasing raw telemetry poses privacy risks. Differential privacy (DP) lets us publish synthetic data that preserves statistical utility while mathematically bounding the information leaked about any single record.

The privacy-utility tension

Differential privacy adds carefully calibrated noise so that an algorithm’s output is nearly identical whether or not any single record is in the dataset. The privacy “cost” is measured by a parameter $\epsilon$ (epsilon). Smaller $\epsilon$ means stronger privacy but potentially lower utility. We use $(\epsilon, \delta)$-differential privacy with $\epsilon = 4.0$ and $\delta = 10^{-5}$, matched across all methods for fair comparison.

Two paradigms for synthetic data

Training-based methods inject calibrated noise during model optimization, as in DP-SGD with VAEs. Training-free methods leverage foundation models through black-box API access, as in Private Evolution with LLMs. We compare four concrete instantiations of these paradigms on a realistic multi-table telemetry benchmark.

Data

Intel DCA telemetry corpus

We use the Intel DCA corpus, a multi-table telemetry dataset with 8 relational tables covering hardware specifications, software configurations, network activity, power consumption, and user behavior across thousands of devices.

Tables are linked by a shared system_id foreign key. This relational structure is central to our evaluation, and many benchmark queries require joining 2 to 3 tables.

The 21-query SQL benchmark

We designed a benchmark of 21 SQL queries spanning five types, each with type-appropriate metrics and pass/fail thresholds.

Formal benchmark definition

A query passes only if all its relevant metrics meet their thresholds.

Data handling

There is no data leakage in our setup. Synthetic data is generated from the full dataset, and benchmark queries test statistical fidelity rather than prediction. All published results are aggregate statistics or synthetic data, and no individual records are ever exposed. During preprocessing, continuous columns are normalized to [0, 1] and categorical columns are label-encoded. Missing values are imputed per table.

Methods

Privacy budget comparison

All methods use $\epsilon = 4.0$, $\delta = 10^{-5}$. Training-based methods (DP-SGD) track cumulative privacy loss via Rényi DP accountants. Training-free methods (PE) spend their budget through the exponential mechanism’s selection step.

What the team built vs. reused

The team built the 21-query benchmark, the per-table DP-SGD pipeline, the Private Evolution integration, and the evaluation harness. We reused the MST implementation from the private-pgm library, the VAE architecture adapted from SDV, and Opacus for DP-SGD gradient clipping.

Results

Overall pass rates

Wide-table DP-VAE and both PE variants were evaluated on 10 single-table queries. Per-table DP-SGD and MST were evaluated on all 21.

Continuous performance metrics

Beyond pass/fail, we report average scores and error metrics to understand how close each method gets.

Avg score ranges $0$–$1$ (higher is better, $1$ = perfect fidelity). Median normalized error measures how far off the synthetic data is on average (lower is better).

Per-query breakdown

Full scores for all 21 queries are shown below, now including both PE variants (vanilla baseline and conditional). Green indicates a pass, red indicates a fail, amber indicates a partial result, and gray means the query was not evaluated.

What changed after early attempts

We started with only the wide-table DP-VAE, which achieved a pass rate of just 1 out of 10 and prompted us to explore per-table splitting. Splitting improved single-table queries dramatically but revealed the join problem. Adding the MST histogram baseline matched per-table DP-SGD on pass rate (6/21) but with different query strengths. Vanilla PE showed promise but was limited by uniform noise allocation (1/10 passes). Conditional PE, which partitions the privacy budget across query groups using zCDP composition, tripled the pass rate to 3/10 and achieved the lowest normalized error on 4 of 9 evaluated queries. The overarching takeaway is that no single method dominates, and a hybrid routing strategy using different methods for different query types is the most promising path forward.

Discussion

Key takeaways

1. Per-table independence breaks cross-table queries

Per-table DP-SGD and MST generated high-fidelity data within individual tables but failed on queries joining multiple tables. Foreign-key mismatches caused inflated relative errors and low Jaccard overlap.

2. Wide-table sparsity collapses continuous metrics

Flattening all tables into one wide table introduced extreme sparsity. The VAE encoded most columns as near-zero, destroying continuous distributions (power consumption, network throughput, RAM usage).

3. Training-free methods have a knowledge ceiling

Both PE variants relied on an LLM's pre-trained knowledge. When real data contained categories unseen by the model (geographic regions, rare OS versions, niche applications), the method produced 0% group coverage on those queries. Conditional PE improved utility through smarter budget allocation but could not overcome fundamental coverage gaps in the LLM's training data.

4. Simple distributions are tractable

Queries on categorical distributions with few categories (browser usage, vendor percentages) passed across multiple methods. This suggests distribution synthesis is largely solved for non-sparse data.

5. DP noise is necessary but limiting

Even at $\epsilon = 4.0$ (moderate privacy), noise masked subtle correlations and rare subgroups. Query-specific privacy budgets via advanced composition may help allocate noise more efficiently.

Limitations

We only evaluated at a single privacy budget of $\epsilon = 4.0$, and performance at stricter budgets ($\epsilon < 1$) may differ substantially. Our results are also specific to the Intel DCA corpus, and other domains like healthcare or finance may show different method rankings. We did not implement methods that reason about foreign keys during synthesis, such as DP joins or relational GANs. Private Evolution used a general-purpose LLM, and domain-specific fine-tuning could improve category coverage. Finally, our 21 queries, while diverse, may not cover all production workload patterns, so results should be interpreted as indicative rather than exhaustive.

Method comparison

Recommendations for practitioners

No single method dominates. We recommend routing queries to different methods based on type. For simple aggregates and distributions, MST is fast, interpretable, and achieves good TV scores. For row-level analytics, per-table DP-SGD offers the best within-table fidelity. For ranking queries, MST provides good Jaccard overlap on top-k items. For targeted single-table workloads, conditional PE offers competitive accuracy with the lowest median normalized error (3.79) of any method we tested. Cross-table joins remain an open challenge that requires future relational DP methods, as current approaches are insufficient.

Future directions

Relational differential privacy

Develop methods that reason about foreign-key relationships directly, applying DP at the join level. This is the most impactful open problem identified by our work.

Domain-adapted LLMs for PE

Fine-tune LLMs on domain-specific schemas and category lists to improve PE's coverage of rare values. Conditional PE's query-aware budget partitioning could be extended with more granular group definitions and adaptive allocation strategies.

Multi-epsilon evaluation

Characterize the privacy-utility frontier by sweeping $\epsilon$ from $0.1$ to $10$ for each method.

Broader datasets

Validate findings on healthcare, financial, and census datasets to test generalizability.

References

Dwork, C., and Roth, A. (2014). The Algorithmic Foundations of Differential Privacy. Foundations and Trends in Theoretical Computer Science.

Abadi, M., et al. (2016). Deep Learning with Differential Privacy. CCS 2016.

McKenna, R., Miklau, G., and Sheldon, D. (2021). Winning the NIST Contest: A Scalable and General Approach to Differentially Private Synthetic Data. VLDB 2021.

Xie, Z., et al. (2024). Private Evolution: Training-Free Differentially Private Synthetic Data via LLMs.

Project GitHub repository | Full technical report (PDF)