|
| 1 | +--- |
| 2 | +title: 'Constrained Posterior Sampling: Time Series Generation with Hard Constraints' |
| 3 | +description: NeurIPS 2025 paper |
| 4 | +categories: blog |
| 5 | +--- |
| 6 | + |
| 7 | +*By Sai Shankar Narasimhan* |
| 8 | + |
| 9 | +# Forcing Diffusion Models to Respect the Rules: Constrained Posterior Sampling for Time Series |
| 10 | + |
| 11 | +Proceedings of the 39th Conference on Neural Information Processing Systems (NeurIPS 2025) |
| 12 | + |
| 13 | +**TLDR:** Constrained Posterior Sampling (CPS) is a diffusion-based sampling algorithm that aims to generate time series samples that can satisfy user-specified constraints. |
| 14 | + |
| 15 | +[arXiv](https://arxiv.org/abs/2410.12652) | |
| 16 | + |
| 17 | + |
| 18 | +## Motivation |
| 19 | +Imagine you are a quantitative researcher who wants to stress-test trading strategies. You would want access to a tool that can precisely generate high-fidelity stock price time series for prompts like |
| 20 | + |
| 21 | +**Generate Tesla’s stock price for the next month with 5% volatility.** |
| 22 | + |
| 23 | +Additionally, for the stocks domain, there are some inherent rules that a generated sample should adhere to, such as the opening and closing prices for a day should be bounded by the highest and lowest prices for that day. This problem exists in almost all engineering domains, where certain domain-specific constraints may arise due to the laws of physics or the underlying process. We refer to this as the **constrained time series generation problem**. |
| 24 | + |
| 25 | +## Expectations Of A Desired Approach |
| 26 | + |
| 27 | +- **Training-free:** Training a generative model for specific constraints lacks scalability. For example, a model trained for mean constraints cannot easily adapt to argmax constraints. |
| 28 | +- **Hyperparameter-free:** The choice of guidance weights in guidance-based approaches with diffusion models significantly affects the sample quality. However, optimizing these weights becomes computationally intractable when dealing with hundreds of constraints. |
| 29 | +- **Should be Independent of external models:** Prior works often employ external models, like the discriminator in Generative Adversarial Networks (GANs), alongside the generative model to enforce realism, necessitating additional training and complex sampling procedures. |
| 30 | + |
| 31 | + |
| 32 | +<figure style="text-align: center;"> |
| 33 | + <img src="{{site.baseurl}}/images/post/cps_intro_fig.png" alt="High level description plot" height="auto" style="margin: auto; display: block;"> |
| 34 | + <figcaption>Figure 1 - High-Level Description of Constrained Posterior Sampling (CPS): Here, we show an example where CPS generates a daily stock price time series with natural constraints, such as the bounds on the opening and closing prices of the stock.</figcaption> |
| 35 | + <p></p> |
| 36 | +</figure> |
| 37 | + |
| 38 | +This post features **Constrained Posterior Sampling**, a training-free, plug-and-play approach that enforces hard constraints on the outputs from your diffusion generative models without compromising sample quality. |
| 39 | + |
| 40 | +## Constrained Posterior Sampling |
| 41 | + |
| 42 | +CPS builds on diffusion models – a class of generative models that iteratively refine noise into data. In a diffusion model, you start with random noise and repeatedly “denoise” it to produce a sample. **The key insight of CPS is: after each small denoising step, gently push the output back to satisfy the constraints before further denoising.** In other words, at every step of the generation, we enforce the rules, at least a little. By the end, the output should follow all the rules by construction. |
| 43 | + |
| 44 | +## Under the Hood: How CPS Enforces Constraints in Diffusion Sampling |
| 45 | + |
| 46 | +Given a specific constraint set from which we aim to generate a realistic time series sample, we perform the following steps: |
| 47 | +- **Estimate the posterior mean.** At each denoising step, we obtain an estimate of the clean sample. |
| 48 | +- **Project the posterior mean estimate.** We project the posterior mean estimate towards the constraint set. This is performed using off-the-shelf optimization routines. |
| 49 | +- **Perform DDIM sampling.** Using the projected posterior mean estimate, we obtain the noisy sample for the next denoising step using DDIM sampling. We refer the reader to our manuscript for further technical details. |
| 50 | + |
| 51 | +This process continues for all the denoising steps. During projection, CPS performs an unconstrained optimization step that ensures minimal perturbation of the posterior mean estimate such that the constraint violation is reduced. This is concisely depicted in Figure 2. |
| 52 | + |
| 53 | +<figure style="text-align: center;"> |
| 54 | + <img src="{{site.baseurl}}/images/post/cps_approach_fig.png" alt="CPS Approach" height="auto" style="margin: auto; display: block;"> |
| 55 | + <figcaption>Figure 2 - Constrained Posterior Sampling: We show the graphical model for one step of denoising in CPS: check Algorithm 1 in our manuscript.</figcaption> |
| 56 | + <p></p> |
| 57 | +</figure> |
| 58 | + |
| 59 | +To ensure minimal effects on the sample quality, we introduce penalty coefficients that minimize the perturbations of the posterior mean estimate during the initial denoising steps when the signal-to-noise ratio is very low. Towards the final denoising steps, the penalty coefficients are very large to ensure constraint satisfaction. |
| 60 | + |
| 61 | +Note that CPS satisfies all the requirements listed above for a desired approach. |
| 62 | + |
| 63 | +- CPS is a **training-free procedure** and can be extended to any constraint class. |
| 64 | +- The proposed approach is **hyperparameter-free**. The penalty coefficients are fixed using the noise schedule of the diffusion model, and **CPS leverages existing off-the-shelf optimization solvers** to perform the projection step. |
| 65 | +- CPS does not require any external model to improve sample realism. In fact, we note that **the adverse effects of projection are mitigated through the subsequent denoising steps**. |
| 66 | + |
| 67 | + |
| 68 | + |
| 69 | +## Applications and Results Across Domains: |
| 70 | + |
| 71 | +In our work, we showcase the efficiency of CPS in terms of sample quality and diversity on six diverse datasets, spanning environmental, traffic, and finance domains. Particularly, in terms of sample realism and utility metrics (check Figure 3), we show that CPS outperforms SOTA approaches by 10%. |
| 72 | + |
| 73 | +<figure style="text-align: center;"> |
| 74 | + <img src="{{site.baseurl}}/images/post/cps_results_1.png" alt="CPS Main Qualitative Results" height="auto" style="margin: auto; display: block;"> |
| 75 | + <figcaption>Figure 3 - Main Qualitative Results: CPS provides high-fidelity synthetic time series samples that match real time series data. The real test samples from which the constraints are extracted are shown in blue. The samples generated using the extracted constraints are shown in red. Across all datasets, the baselines suffer from the adversarial effects of the projection step, whereas CPS generates high-quality samples.</figcaption> |
| 76 | + <p></p> |
| 77 | +</figure> |
| 78 | + |
| 79 | +Additionally, in terms of tracking real time series samples, we show that CPS outperforms SOTA methods by 42%. Specifically, CPS does not suffer from sample quality degradation for a large number of constraints, while other approaches break down in such settings (Figure 4). We refer the readers to Figures 3 and 4 to observe the sample quality and tracking abilities of CPS. |
| 80 | + |
| 81 | +<figure style="text-align: center;"> |
| 82 | + <img src="{{site.baseurl}}/images/post/cps_results_2.png" alt="CPS Main Qualitative Results" height="auto" style="margin: auto; display: block;"> |
| 83 | + <figcaption>Figure 4: CPS tracks the real data samples as the number of constraints increases. Increasing the number of constraints reduces the size of the constraint set, and an ideal approach should effectively generate samples that resemble the real time series samples that belong to the constraint set. Here, we show a qualitative example from the Stocks dataset. Observe that CPS accurately tracks the real sample that concurs with the specified constraints, while other approaches suffer.</figcaption> |
| 84 | + <p></p> |
| 85 | +</figure> |
| 86 | + |
0 commit comments