AudioDiffuser

Companion open source code for the paper "A Review on Score-based Generative Models for Audio Applications." A modular diffusion codebase designed for audio using denoising score matching formulation discussed in EDM. This repository uses hydra-lightning config management and is suitable for developping new models efficiently.

Setup

Install dependencies

# clone project
git clone https://github.com/AgentCooper2002/AudioDiffuser
cd AudioDiffuser

# [OPTIONAL] create conda environment
conda create -n diffaudio python=3.8
conda activate diffaudio

# install pytorch (>=2.0.1), e.g. with cuda=11.7, we have:
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia

# install requirements
pip install -r requirements.txt

How to run

Run experiment and evaluation

Train model with chosen experiment configuration from configs/experiment/

CUDA_VISIBLE_DEVICES=0,3 python src/train.py trainer=ddp.yaml trainer.devices=2 experiment=example.yaml +trainer.precision=16-mixed +trainer.accumulate_grad_batches=4

For RTX 4090, add NCCL_P2P_DISABLE=1 (verified, ref here) otherwise, DDP will stuck.

Or train model with single GPU resume from a checkpoint:

CUDA_VISIBLE_DEVICES=3 python src/train.py experiment=example1.yaml +trainer.precision=16-mixed ckpt_path="/path/to/ckpt/name.ckpt"

Or evaluation:

CUDA_VISIBLE_DEVICES=3 python src/eval.py ckpt_path='dummy.ckpt' +trainer.precision=16 experiment=example2.yaml

Particularly, grid search for tuning hyperparameters during sampling:

CUDA_VISIBLE_DEVICES=2 python src/eval.py --multirun ckpt_path='ckpt.pt' +trainer.precision=16-mixed experiment=experiment.yaml model.sampler.param1=3,6,9 model.sampler.param2=1.0,1.1

Code References

• k-Diffusion by Katherine Crowson
• EDM by Nvdia
• Audio Diffusion by Flavio
• EfficientUNet by lucidrians
• UNet in ADM by openai
• Unconditional diffwave by philsyn

Resources

This repo is generated with lightning-hydra-template.