Towards data efficient generative models

For DMs, we address the data efficiency challenge by better minimizing the total denoising score matching error during training. <br><br>Firstly, we apply information game theory to analyze the GANs' min-max game and unveil a novel perspective on GAN training: the min-max game in existing GANs training is unfair. Based on this, we present a novel GAN called Information Gap GAN (IGGAN) to enhance the min-max game in GANs training.

Specifically, IGGAN comprises a generator ($G$) and two discriminators ($D_1$ and $D_2$), each applying distinct data augmentation (DA) techniques. The information gap caused by various DA methods between $D_1$ and $D_2$ can alter the information received by each player in the min-max game, leading to incomplete information for all three players—$G$, $D_1$, and $D_2$—in IGGAN. As a result, IGGAN outperforms other existing methods, achieving better Inception Scores and lower Fréchet Inception Distances.<br><br>Secondly we introduce a straightforward yet effective approach named Dual Adaptive Noise Injection (DANI) to mitigate the overfitting of $D$ in DE-GANs training.

Specifically, DANI incorporates two adaptive strategies: adaptive injection probability and adaptive noise strength. By dynamically adjusting these factors, DANI effectively alleviates the overfitting of $D$. The strong performance and generalization ability of DANI demonstrate its effectiveness in improving GAN training under limited data conditions. <br><br>Thirdly, we first illustrate that applying Data augmentation (DA) in DE-GANs can introduce out-of-distribution samples, leading to the undesirable leaking of augmentation issue in DE-GANs training.

To address this, we present a straightforward yet powerful technique named Adaptive Negative Data Augmentation (ANDA) for DE-GANs. Specifically, ANDA enhances the augmented distribution of generated data by incorporating the augmented distribution of negative real data, which is generated by applying Negative Data Augmentation (NDA) to real data. This approach allows potential leaking samples to be presented as ``fake” instances to the discriminator in an adaptive manner, preventing the generator ($G$) from learning these samples and improving overall performance.

We demonstrate that ANDA is effective in mitigating the leaking of augmentation issue. <br><br>Finally, we focus on training diffusion models with limited data. We present a novel theoretical insight for diffusion models that two factors, i.e., the denoiser function hypothesis space and the number of training samples, can affect the denoising score matching error of all training samples. Based on this theoretical insight, it is clearly evident that the total denoising score matching error is hard to be minimized within the denoiser function hypothesis space of existing methods when training diffusion models with limited data.

To address this, we design a new diffusion model with limited data called Limited Data Diffusion (LD-Diffusion), which consists of two main components: a compressing model and a novel mixed augmentation with fixed probability (MAFP) strategy. Comprehensive experiments show that LD-Diffusion greatly enhances the training of diffusion models with limited data, achieving state-of-the-art performance compared to other diffusion models.