Beyond general recognition tasks, specialized domains including privacy-constrained medical applications and fine-grained settings often encounter data scarcity, especially for tail classes. To obtain less biased and more reliable models under such scarcity, practitioners leverage diffusion models to generate underrepresented data. Specifically, recent studies fine-tune pretrained diffusion models with LoRA on few-shot real sets to synthesize additional images. While a single LoRA trained on a single image captures fine-grained details, it offers limited diversity, as class-wise LoRA trained over all shots of a diverse images as it encodes class priors yet tends to overlook fine details. To combine both benefits, we separate the adapter into class-shared LoRA A for class-level priors and per-image LoRAs B for image-specific characteristics. To expunge different data shifts in the shared LoRA A, we propose a semantic boosting by preserving class bounding boxes during training. For generation, we compose A with a mixture of B using coefficients drawn from a Dirichlet distribution. Across diverse datasets, our synthesized images exhibit both diverse and detail-rich while staying aligned with the few-shot real distribution, yielding robust gains in downstream classification accuracy.
