Skip to content

Beginner course: Module 1 to 5 notebooks with pre-run code #115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ESzyszka wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Beginner course: Module 1 to 5 notebooks with pre-run code #115

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
fbf2ca2
Add files via upload
ESzyszka Jul 27, 2026
d32bf11
Update Module 4 code to match the md file
ESzyszka Aug 11, 2026
acfdd82
updated module 1 notebook
manas95826 Aug 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
378 changes: 378 additions & 0 deletions Beginner-course/Module1.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,378 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Module 1: Let's Understand Search\n",
"\n",
"Companion notebook for **Module 1** of the Qdrant Beginners Course.\n",
"\n",
"Understand why traditional search struggles and how modern semantic search improves it. This notebook walks through embeddings, cosine similarity, and the limits of similarity search, with runnable code for every example.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setup\n",
"\n",
"The embedding model runs on your CPU. You don't need a GPU or API keys.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip install fastembed numpy -q"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. What Is Search?\n",
"\n",
"Search is the act of finding the right information out of everything you have, given a question. You type \"car repair\" into a box, and something has to decide which of your thousands of documents, products, or messages actually answers that.\n",
"\n",
"Every search system, no matter how it's built internally, does the same two things:\n",
"\n",
"1. **Retrieve**: narrow a huge collection down to a shortlist of documents that might be relevant.\n",
"2. **Rank**: order that shortlist so the best answer ends up near the top.\n",
"\n",
"This module is about one question: how does a system decide what \"relevant\" means? We'll start with the simplest possible answer, watch it fail, and build up from there. No prior knowledge of vector search engines or indexing algorithms is assumed.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. The Problem: Why Keyword Search Struggles\n",
"\n",
"Keyword search retrieves documents by exact word match: it checks whether the literal terms in your query appear in the document, with no understanding of what those terms mean. That works when people use the same terms as the content they need.\n",
"\n",
"That creates a few common problems:\n",
"\n",
"- **Different words, same meaning:** \"car repair\" misses \"automobile maintenance.\"\n",
"- **Same word, different meaning:** \"Apple stock\" can retrieve fruit content instead of financial information.\n",
"- **Same words, different order:** \"dog bites man\" and \"man bites dog\" contain the same terms, but mean very different things.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. How Traditional Search Improved\n",
"\n",
"Keyword search picked up real upgrades over the years: faster lookups, relevance ranking, tolerance for typos, matching on word roots. Each made matching faster or more forgiving, but none of them taught the system what words mean. A keyword system can't know \"car\" and \"automobile\" are synonyms unless someone hard-codes that fact, and you can't hard-code an entire language.\n",
"\n",
"That's the gap semantic search closes. Instead of asking \"Does this document contain the same words?\" it asks \"Does this document mean the same thing?\" Nobody hand-codes the fact that \"car\" and \"automobile\" are related, the model learns it from the text it was trained on, and sentences with related meaning end up as vectors that sit close together, even when they share no words.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. How It Works: Embeddings\n",
"\n",
"### What Is an Embedding?\n",
"\n",
"An embedding is a vector: a list of numbers that captures meaning. Semantic search works by converting text into embeddings, text with similar meaning produces embeddings that sit close together in high-dimensional space, and text with different meaning produces embeddings that sit far apart. Each position in that list is a dimension; no single one maps to a human concept like \"color,\" meaning comes from all of them combined.\n",
"\n",
"### The Embedding Model\n",
"\n",
"An embedding model takes a piece of text and returns a fixed-length array of floating-point numbers. The exact numbers matter less than the relationships between them.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from fastembed import TextEmbedding\n",
"\n",
"model = TextEmbedding(model_name=\"sentence-transformers/all-MiniLM-L6-v2\")\n",
"\n",
"# model.embed() takes a list of strings and returns one vector per string\n",
"query_vec = list(model.embed([\"car repair\"]))[0]\n",
"doc_vec = list(model.embed([\"automobile maintenance\"]))[0]\n",
"\n",
"print(len(query_vec), len(doc_vec)) # check both vectors are the same length: 384 dimensions each\n",
"print(query_vec[:5]) # peek at the first 5 of the query's 384 floats\n",
"print(doc_vec[:5]) # peek at the first 5 of the document's 384 floats"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Model Size: A Tradeoff\n",
"\n",
"Embedding models come in different sizes. Smaller models (128\u2013384 dimensions, like the one above) are fast and cheap to run. Larger ones (1024+ dimensions) can capture more nuance and context, at the cost of more compute and memory. Dimension count alone isn't a quality signal, a well-trained small model can beat a poorly trained large one.\n",
"\n",
"### Why This Model\n",
"\n",
"This module uses `sentence-transformers/all-MiniLM-L6-v2` because it's small enough to run on a CPU with no API keys or GPU, and accurate enough to demonstrate semantic search clearly. When you start your own project, see [Points, Vectors and Payloads](https://qdrant.tech/course/essentials/day-1/embedding-models/) for how to weigh size, language, and domain fit when picking a model.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Comparing Meaning: Distance Metrics\n",
"\n",
"Once we have vectors, we need a way to measure how similar two of them are. Different metrics suit different situations.\n",
"\n",
"### Cosine Similarity\n",
"\n",
"The most common metric for text. It measures the angle between two vectors and ignores their length, focusing purely on direction. Scores range from -1 to 1: 1.0 means the vectors point the same way, 0.0 means unrelated, -1 means opposite. In practice, normalized text-embedding models like the one used here rarely produce negative scores, so unrelated text usually lands as a small positive number instead.\n",
"\n",
"$$\n",
"\\text{cosine\\_similarity}(A, B) = \\frac{A \\cdot B}{\\lVert A \\rVert \\, \\lVert B \\rVert}\n",
"$$\n",
"\n",
"For example, embedding \"car repair\" and \"automobile maintenance\" and comparing the two vectors with this formula yields a similarity score around 0.73, far higher than an unrelated pair would score, reflecting their shared meaning despite having no words in common.\n",
"\n",
"### Try It Yourself: Compare Cosine Scores\n",
"\n",
"Reuse the embedding snippet from section 4 to embed three query/document pairs, then score each pair with cosine similarity, using the formula above implemented directly with NumPy.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from fastembed import TextEmbedding\n",
"import numpy as np\n",
"\n",
"model = TextEmbedding(model_name=\"sentence-transformers/all-MiniLM-L6-v2\")\n",
"\n",
"def cosine_similarity(a, b):\n",
" return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))\n",
"\n",
"pairs = [\n",
" (\"car repair\", \"automobile maintenance\"), # synonyms\n",
" (\"cheap flights to New York\", \"affordable airfare to NYC\"), # paraphrase\n",
" (\"cheap flights to New York\", \"best pizza in Chicago\"), # unrelated\n",
"]\n",
"\n",
"for query, document in pairs:\n",
" query_vec = list(model.embed([query]))[0]\n",
" doc_vec = list(model.embed([document]))[0]\n",
" score = cosine_similarity(query_vec, doc_vec)\n",
" print(f\"{score:.3f} | {query!r} vs {document!r}\")\n",
"\n",
"# Expected output:\n",
"# 0.733 | 'car repair' vs 'automobile maintenance'\n",
"# 0.821 | 'cheap flights to New York' vs 'affordable airfare to NYC'\n",
"# 0.332 | 'cheap flights to New York' vs 'best pizza in Chicago'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**What to look for:**\n",
"\n",
"- The synonym and paraphrase pairs (0.733, 0.821) score high despite sharing almost no words.\n",
"- The unrelated pair (0.332) scores far lower, reflecting different meaning.\n",
"\n",
"**Your turn:** swap in a polysemy case. Score `\"apple stock\"` against both `\"shares of a tech company\"` and `\"a crisp red fruit\"`. Which comes out higher, and does it match the sense you meant?\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Your turn: try the polysemy case here\n",
"query = \"apple stock\"\n",
"candidates = [\"shares of a tech company\", \"a crisp red fruit\"]\n",
"\n",
"query_vec = list(model.embed([query]))[0]\n",
"for candidate in candidates:\n",
" candidate_vec = list(model.embed([candidate]))[0]\n",
" score = cosine_similarity(query_vec, candidate_vec)\n",
" print(f\"{score:.3f} | {candidate!r}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Distance Metric Comparison\n",
"\n",
"| Metric | Best for | Notes |\n",
"|--------|----------|-------|\n",
"| Cosine | Text similarity, NLP (Natural Language Processing) models | Robust to different vector magnitudes. Most common default. |\n",
"| Dot product | Vectors already normalized to unit length | Numerically identical to cosine similarity once vectors are unit length, same score, not a separate metric. |\n",
"| Euclidean (L2) | Image embeddings, spatial data | Sensitive to magnitude; works best with models trained for it. |\n",
"| Manhattan (L1) | Grid-like or count-based data | Sums absolute differences per dimension rather than squaring them first, making it less affected by extreme values in any single dimension. |\n",
"\n",
"**Cosine vs. dot product:** for vectors normalized to unit length, dot product produces the exact same ranking as cosine similarity, it's a cheaper way to compute the same result, not a different metric. That's why Qdrant normalizes vectors on upload and computes a \"Cosine\" collection as a dot product internally.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 6. Why Similarity Alone Is Not Enough\n",
"\n",
"Sections 1 and 2 showed keyword search failing on synonyms, paraphrasing, polysemy, and word order. It's tempting to read that as \"semantic search replaces keyword search.\" It doesn't, each is strong exactly where the other is weak, as the next two cases show. (Filtering by recency, permissions, or other payload values, and combining that with ranking signals, is a separate layer on top of similarity, later modules cover it once you have a collection to filter.)\n",
"\n",
"### Word Order and Negation Still Trip It Up\n",
"\n",
"Section 2 said keyword search can't tell \"dog bites man\" from \"man bites dog.\" You'd expect semantic search to fix that. It mostly doesn't:\n",
"\n",
"| Pair | Cosine similarity |\n",
"|------|-------------------|\n",
"| \"dog bites man\" vs \"man bites dog\" | 0.907 |\n",
"| \"safe for kids\" vs \"harmful to kids\" | 0.779 |\n",
"| \"dog bites man\" vs \"a canine attacked a person\" | 0.570 |\n",
"\n",
"The first two rows score high, even though each pair means something different: one flips who's doing the biting, the other flips safe into dangerous. The model mostly notices that the two sentences share almost all the same words, so it calls them similar, even though a person would read them as opposites right away.\n",
"\n",
"The third row is the sharpest version of the problem: \"a canine attacked a person\" is a genuine paraphrase of \"dog bites man,\" meaning the same thing in different words, yet it scores lower (0.570) than the reversed, opposite-meaning sentence (0.907). Shared words move the score more than shared meaning does.\n",
"\n",
"So: semantic search is great at synonyms and paraphrasing, but shaky on word order and negation. Don't rely on a similarity score alone anywhere it actually matters whether something is \"safe\" or \"not safe.\"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Reproduce the table above\n",
"word_order_pairs = [\n",
" (\"dog bites man\", \"man bites dog\"),\n",
" (\"safe for kids\", \"harmful to kids\"),\n",
" (\"dog bites man\", \"a canine attacked a person\"),\n",
"]\n",
"\n",
"for a, b in word_order_pairs:\n",
" a_vec = list(model.embed([a]))[0]\n",
" b_vec = list(model.embed([b]))[0]\n",
" score = cosine_similarity(a_vec, b_vec)\n",
" print(f\"{score:.3f} | {a!r} vs {b!r}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Exact Matching: Where Keyword Search Wins\n",
"\n",
"Not every query needs semantic understanding. A query for an exact SKU (Stock Keeping Unit, the unique code a retailer assigns to one specific product), like \"SKU-48291,\" needs an exact match instead. Try it yourself: embed the query and three candidate SKUs, then compare their cosine scores.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from fastembed import TextEmbedding\n",
"import numpy as np\n",
"\n",
"model = TextEmbedding(model_name=\"sentence-transformers/all-MiniLM-L6-v2\")\n",
"\n",
"def cosine_similarity(a, b):\n",
" return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))\n",
"\n",
"query = \"SKU-48291 issue\"\n",
"candidates = [\"SKU-48292\", \"SKU-48291\", \"SKU-48290\"] # wrong, correct, wrong\n",
"\n",
"query_vec = list(model.embed([query]))[0]\n",
"\n",
"for candidate in candidates:\n",
" candidate_vec = list(model.embed([candidate]))[0]\n",
" score = cosine_similarity(query_vec, candidate_vec)\n",
" print(f\"{score:.3f} | {candidate}\")\n",
"\n",
"# Real output:\n",
"# 0.730 | SKU-48292 (wrong product)\n",
"# 0.734 | SKU-48291 (correct product)\n",
"# 0.765 | SKU-48290 (wrong product, scores HIGHEST)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Semantic similarity here doesn't just drift toward the wrong SKU, it ranks the wrong one first. A keyword-style exact filter gets it right, trivially: restrict the results to points where the `sku` payload field equals `\"SKU-48291\"`, and only that one product comes back, no embedding, no similarity score, just an exact match. Module 2 builds a real filtered Qdrant query once there's a collection with payloads to filter.\n",
"\n",
"Keyword matching isn't obsolete, it's exactly the right tool when a query needs to hit one precise token. Dense similarity finds the general neighborhood of relevant results; exact keyword matching finds the right point within it. Neither replaces the other.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 7. When a System Needs Both\n",
"\n",
"The SKU example above is why some production systems run semantic and exact retrieval together rather than picking one: a single collection can serve queries that need meaning and queries that need one precise token. This isn't a universal requirement, plenty of systems only ever need one or the other, but when a use case needs both, that's called **hybrid search**, combining **dense** (semantic/vector) retrieval with **sparse** (keyword-style, e.g. BM25) retrieval.\n",
"\n",
"Hybrid search is the next module's topic.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 8. References & Further Reading\n",
"\n",
"**Qdrant docs:**\n",
"\n",
"- [Qdrant Documentation Overview](https://qdrant.tech/documentation/overview/) \u2014 How Qdrant's vector search engine fits together: collections, points, payloads, and APIs.\n",
"- [Distance Metrics](https://qdrant.tech/course/essentials/day-1/distance-metrics/) \u2014 Cosine, dot product, and Euclidean distance compared, and how to pick the right one for your embedding model.\n",
"- [Filtering](https://qdrant.tech/documentation/search/filtering/) \u2014 Payload filter syntax, indexed fields, and combining filters with vector queries.\n",
"\n",
"**Go deeper:**\n",
"\n",
"- [Vector Embeddings Explained](https://qdrant.tech/articles/what-are-embeddings/)\n",
"- [What Is a Vector Database?](https://qdrant.tech/articles/what-is-a-vector-database/)\n",
"- [FastEmbed: Qdrant's Efficient Python Library for Embedding Generation](https://qdrant.tech/articles/fastembed/)\n",
"- [Fine-Tuning Sparse Embeddings for E-Commerce Search, Part 1: Why Sparse Embeddings Beat BM25](https://qdrant.tech/articles/sparse-embeddings-ecommerce-part-1/)\n",
"- [What Is RAG in AI?](https://qdrant.tech/articles/what-is-rag-in-ai/)\n",
"\n",
"**Definitions:**\n",
"\n",
"- [Cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity)\n",
"- [Dot product](https://en.wikipedia.org/wiki/Dot_product)\n",
"- [Euclidean distance](https://en.wikipedia.org/wiki/Euclidean_distance)\n",
"- [Manhattan (taxicab) distance](https://en.wikipedia.org/wiki/Taxicab_geometry)\n",
"\n",
"## What's Next: Module 2\n",
"\n",
"In the next module, we'll break down:\n",
"\n",
"- What is a vector, and why does it have hundreds to thousands of dimensions?\n",
"- How do dimensions actually represent meaning?\n",
"- How similarity really works under the hood, and when it fails.\n",
"- Your first Qdrant collection: points, payloads, and your first query.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
},
"colab": {
"provenance": [],
"name": "Module1.ipynb"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Loading