SIMULACRA - Synthetic Data Generation Pipeline
A synthetic data studio that profiles a seed dataset's statistical DNA, generates 10,000+ privacy-safe records via Gaussian-copula or LLM engines, and automatically re-prompts and regenerates when KL-divergence drift exceeds tolerance.
- Python
- FastAPI
- pandas
- Gaussian copula
- React
SYSTEM BUILD
What went into it
- 01
Profiles seed CSVs — type inference, distributions, missingness maps — then synthesizes with an offline Gaussian-copula engine or a profile-conditioned LLM batch agent.
- 02
Validates every run with ε-smoothed KL-divergence drift scoring plus a near-duplicate privacy guard, keeping the best checkpoint across refinement rounds.
- 03
Closes the loop automatically: drift over tolerance injects corrective constraints, steps down temperature, and re-runs only the failing columns.
- 04
Streams live job telemetry over SSE to a studio UI with hand-rolled SVG histogram overlays comparing real versus synthetic distributions.
MEASURED OUTPUT
What changed
Turned a 50-row seed file into validated 10,000-row synthetic datasets with per-column drift scores and an export-ready fidelity report.
Covered by 51 green core-engine tests plus an end-to-end Playwright journey from dataset upload to download.