fineweb

Star

Here are 13 public repositories matching this topic...

kmkrofficial / LiteGPT

Star

LiteGPT: A 124M Small Language Model (SLM) pre-trained on FineWeb and fine-tuned on Alpaca.

pytorch alpaca gpt2 llm nanogpt fineweb

Updated Dec 16, 2025
Python

evanatyourservice / llm-jax

Star

Train a SmolLM-style llm on fineweb-edu in JAX/Flax with an assortment of optimizers.

optimization flax gpt jax huggingface large-language-models llm fineweb

Updated Jul 24, 2025
Python

A 134M-parameter GPT pretrained from scratch on FineWeb-Edu on one free-tier GPU, then audited: 4 bugs found in its own training run, every reported number shipped with the command that reproduces it.

machine-learning deep-learning reproducible-research pytorch transformer gpt language-model from-scratch perplexity ml-engineering pretraining llm nanogpt fineweb

Updated Aug 2, 2026
Jupyter Notebook

nazmiefearmutcu / awareness

Sponsor

Star

A private index of the public web — backfill history + live-tail RSS/Common Crawl/GDELT onto your laptop, query it like a table. One Python process, fully local.

python rss text-mining osint data-engineering web-scraping gdelt web-archiving data-pipeline iceberg common-crawl apache-iceberg duckdb fineweb

Updated Aug 6, 2026
Python

lmmx / bbcfw

Star

Exploring the BBC News subset of the FineWeb dataset (via HuggingFaceFW/fineweb's dated subsets on HF), originally a Common Crawl dataset.

news bbc-news fineweb

Updated Feb 17, 2025
Python

aidendorian / Marcella-66M-SLM

Star

A 66M parameter decoder-only transformer language model implemented from scratch in PyTorch. Features a custom SentencePiece tokenizer, RoPE positional embeddings, SwiGLU feed-forward network, per-layer KV cache for efficient autoregressive inference, and a Svelte-based streaming chat interface.

transformers torch pytorch pretrained-models slm language-model rope alpaca sdpa finetuning sentencepiece transformer-models kv-cache small-language-models fineweb flashattention marcella

Updated May 13, 2026
Python

MasterSkepticista / gpt2

Star

Training GPT-2 on FineWeb-Edu in JAX/Flax

flax jax gpt2 flash-attention fineweb

Updated Aug 3, 2026
Python

chris-colinsky / openai-parameter-golf

Star

OpenAI Parameter Golf competition entry — adaptive Hessian-sensitivity GPTQ clipping on PR #1855's stack. 1.06310 BPB (3-seed mean), PR #1962.

machine-learning transformer quantization language-model ai-engineering llm gptq fineweb claude-code openai-challenge

Updated May 6, 2026
Python

Cubix1729 / gpt-training

Star

A minimal GPT model pretraining pipeline (mostly taken from Andrej Karpathy's build-nanogpt repo)

gpt-2 llm llm-training fineweb pretraining-llms gpt-2-small

Updated May 18, 2026
Python

Rian-T / benchmark-in-a-haystack

Star

Do quality filters pull benchmark questions into pretraining corpora? Injects MMLU, GSM8K, GPQA, ARC, HellaSwag, PIQA and TruthfulQA items into a corpus and ranks them with six quality classifiers, including DCLM, FineWeb-Edu and Gaperon.