Can't quantize nemotron 3.5

[supersimple33]

System Info

Hello when trying to quantize nemotron 3.5 lightning the model barely reduces in size and it seems most is still saved in bf16. (version 0.50.1 running on fedora with a100s) [transformer 5.15.0, torch 2.13)

Reproduction

    tokenizer = AutoTokenizer.from_pretrained(nemo_dir)
    model = AutoModelForCausalLM.from_pretrained(
        nemo_dir,
        quantization_config=quant_config,
    )

    model.save_pretrained(
        quantized_output_dir,
        safe_serialization=True,
    ) # the output here is only roughly 3GB less 

Expected behavior

The model should be at least half the size