Low-resource NLP project for Assamese text generation using web-scraped corpus creation, n-gram language modeling, probabilistic word prediction, and perplexity evaluation.
Assamese Language Modeling and Text Generation Overview
A Natural Language Processing project focused on building a low-resource language model for Assamese text generation.
The system collects Assamese textual data through web scraping, constructs a custom corpus, generates statistical language models using n-grams, and produces probabilistic Assamese text generation.
The project demonstrates foundational language modeling concepts in a low-resource Indic language setting.
Problem Statement
Most modern NLP systems focus heavily on English and high-resource languages.
Low-resource languages such as Assamese have limited NLP resources, datasets, and pretrained models.
This project explores classical probabilistic language modeling techniques to build a lightweight text generation system for Assamese language processing.
Features
✔ Assamese web corpus generation ✔ Assamese text scraping from online sources ✔ Unicode-based Assamese tokenization ✔ Unigram generation ✔ Bigram generation ✔ Trigram language modeling ✔ Next-word probability prediction ✔ Sentence generation using probabilistic sampling ✔ Perplexity evaluation using smoothing techniques ✔ Low-resource language NLP experimentation
Concepts Used Natural Language Processing Language Modeling Probabilistic Modeling N-Grams Text Generation Web Scraping Tokenization Perplexity Evaluation Low Resource Language Processing
Workflow Step 1 — Corpus Collection
Assamese text is scraped from online Assamese news sources.
Step 2 — Text Processing
Text is cleaned and tokenized using Assamese Unicode patterns.
Step 3 — N-Gram Generation
The system creates:
Unigrams Bigrams Trigrams Step 4 — Language Model Training
A probabilistic language model learns word transition probabilities.
Step 5 — Text Generation
The trained model generates Assamese text based on learned probability distributions.
Step 6 — Evaluation
Perplexity score measures language model quality.
Applications Indic Language NLP Low Resource Language Modeling Text Generation Systems Language Preservation Research Regional Language AI Systems Statistical Language Processing Future Improvements Transformer-based Assamese language model Fine-tuning small LLMs for Assamese BERT-based Assamese embeddings Neural language modeling using LSTMs Indic multilingual transformer integration