diff --git a/Beginner-course/Module1.ipynb b/Beginner-course/Module1.ipynb new file mode 100644 index 0000000..cfdd175 --- /dev/null +++ b/Beginner-course/Module1.ipynb @@ -0,0 +1,378 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Module 1: Let's Understand Search\n", + "\n", + "Companion notebook for **Module 1** of the Qdrant Beginners Course.\n", + "\n", + "Understand why traditional search struggles and how modern semantic search improves it. This notebook walks through embeddings, cosine similarity, and the limits of similarity search, with runnable code for every example.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup\n", + "\n", + "The embedding model runs on your CPU. You don't need a GPU or API keys.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!pip install fastembed numpy -q" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1. What Is Search?\n", + "\n", + "Search is the act of finding the right information out of everything you have, given a question. You type \"car repair\" into a box, and something has to decide which of your thousands of documents, products, or messages actually answers that.\n", + "\n", + "Every search system, no matter how it's built internally, does the same two things:\n", + "\n", + "1. **Retrieve**: narrow a huge collection down to a shortlist of documents that might be relevant.\n", + "2. **Rank**: order that shortlist so the best answer ends up near the top.\n", + "\n", + "This module is about one question: how does a system decide what \"relevant\" means? We'll start with the simplest possible answer, watch it fail, and build up from there. No prior knowledge of vector search engines or indexing algorithms is assumed.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2. The Problem: Why Keyword Search Struggles\n", + "\n", + "Keyword search retrieves documents by exact word match: it checks whether the literal terms in your query appear in the document, with no understanding of what those terms mean. That works when people use the same terms as the content they need.\n", + "\n", + "That creates a few common problems:\n", + "\n", + "- **Different words, same meaning:** \"car repair\" misses \"automobile maintenance.\"\n", + "- **Same word, different meaning:** \"Apple stock\" can retrieve fruit content instead of financial information.\n", + "- **Same words, different order:** \"dog bites man\" and \"man bites dog\" contain the same terms, but mean very different things.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 3. How Traditional Search Improved\n", + "\n", + "Keyword search picked up real upgrades over the years: faster lookups, relevance ranking, tolerance for typos, matching on word roots. Each made matching faster or more forgiving, but none of them taught the system what words mean. A keyword system can't know \"car\" and \"automobile\" are synonyms unless someone hard-codes that fact, and you can't hard-code an entire language.\n", + "\n", + "That's the gap semantic search closes. Instead of asking \"Does this document contain the same words?\" it asks \"Does this document mean the same thing?\" Nobody hand-codes the fact that \"car\" and \"automobile\" are related, the model learns it from the text it was trained on, and sentences with related meaning end up as vectors that sit close together, even when they share no words.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 4. How It Works: Embeddings\n", + "\n", + "### What Is an Embedding?\n", + "\n", + "An embedding is a vector: a list of numbers that captures meaning. Semantic search works by converting text into embeddings, text with similar meaning produces embeddings that sit close together in high-dimensional space, and text with different meaning produces embeddings that sit far apart. Each position in that list is a dimension; no single one maps to a human concept like \"color,\" meaning comes from all of them combined.\n", + "\n", + "### The Embedding Model\n", + "\n", + "An embedding model takes a piece of text and returns a fixed-length array of floating-point numbers. The exact numbers matter less than the relationships between them.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from fastembed import TextEmbedding\n", + "\n", + "model = TextEmbedding(model_name=\"sentence-transformers/all-MiniLM-L6-v2\")\n", + "\n", + "# model.embed() takes a list of strings and returns one vector per string\n", + "query_vec = list(model.embed([\"car repair\"]))[0]\n", + "doc_vec = list(model.embed([\"automobile maintenance\"]))[0]\n", + "\n", + "print(len(query_vec), len(doc_vec)) # check both vectors are the same length: 384 dimensions each\n", + "print(query_vec[:5]) # peek at the first 5 of the query's 384 floats\n", + "print(doc_vec[:5]) # peek at the first 5 of the document's 384 floats" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Model Size: A Tradeoff\n", + "\n", + "Embedding models come in different sizes. Smaller models (128\u2013384 dimensions, like the one above) are fast and cheap to run. Larger ones (1024+ dimensions) can capture more nuance and context, at the cost of more compute and memory. Dimension count alone isn't a quality signal, a well-trained small model can beat a poorly trained large one.\n", + "\n", + "### Why This Model\n", + "\n", + "This module uses `sentence-transformers/all-MiniLM-L6-v2` because it's small enough to run on a CPU with no API keys or GPU, and accurate enough to demonstrate semantic search clearly. When you start your own project, see [Points, Vectors and Payloads](https://qdrant.tech/course/essentials/day-1/embedding-models/) for how to weigh size, language, and domain fit when picking a model.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 5. Comparing Meaning: Distance Metrics\n", + "\n", + "Once we have vectors, we need a way to measure how similar two of them are. Different metrics suit different situations.\n", + "\n", + "### Cosine Similarity\n", + "\n", + "The most common metric for text. It measures the angle between two vectors and ignores their length, focusing purely on direction. Scores range from -1 to 1: 1.0 means the vectors point the same way, 0.0 means unrelated, -1 means opposite. In practice, normalized text-embedding models like the one used here rarely produce negative scores, so unrelated text usually lands as a small positive number instead.\n", + "\n", + "$$\n", + "\\text{cosine\\_similarity}(A, B) = \\frac{A \\cdot B}{\\lVert A \\rVert \\, \\lVert B \\rVert}\n", + "$$\n", + "\n", + "For example, embedding \"car repair\" and \"automobile maintenance\" and comparing the two vectors with this formula yields a similarity score around 0.73, far higher than an unrelated pair would score, reflecting their shared meaning despite having no words in common.\n", + "\n", + "### Try It Yourself: Compare Cosine Scores\n", + "\n", + "Reuse the embedding snippet from section 4 to embed three query/document pairs, then score each pair with cosine similarity, using the formula above implemented directly with NumPy.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from fastembed import TextEmbedding\n", + "import numpy as np\n", + "\n", + "model = TextEmbedding(model_name=\"sentence-transformers/all-MiniLM-L6-v2\")\n", + "\n", + "def cosine_similarity(a, b):\n", + " return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))\n", + "\n", + "pairs = [\n", + " (\"car repair\", \"automobile maintenance\"), # synonyms\n", + " (\"cheap flights to New York\", \"affordable airfare to NYC\"), # paraphrase\n", + " (\"cheap flights to New York\", \"best pizza in Chicago\"), # unrelated\n", + "]\n", + "\n", + "for query, document in pairs:\n", + " query_vec = list(model.embed([query]))[0]\n", + " doc_vec = list(model.embed([document]))[0]\n", + " score = cosine_similarity(query_vec, doc_vec)\n", + " print(f\"{score:.3f} | {query!r} vs {document!r}\")\n", + "\n", + "# Expected output:\n", + "# 0.733 | 'car repair' vs 'automobile maintenance'\n", + "# 0.821 | 'cheap flights to New York' vs 'affordable airfare to NYC'\n", + "# 0.332 | 'cheap flights to New York' vs 'best pizza in Chicago'" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**What to look for:**\n", + "\n", + "- The synonym and paraphrase pairs (0.733, 0.821) score high despite sharing almost no words.\n", + "- The unrelated pair (0.332) scores far lower, reflecting different meaning.\n", + "\n", + "**Your turn:** swap in a polysemy case. Score `\"apple stock\"` against both `\"shares of a tech company\"` and `\"a crisp red fruit\"`. Which comes out higher, and does it match the sense you meant?\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Your turn: try the polysemy case here\n", + "query = \"apple stock\"\n", + "candidates = [\"shares of a tech company\", \"a crisp red fruit\"]\n", + "\n", + "query_vec = list(model.embed([query]))[0]\n", + "for candidate in candidates:\n", + " candidate_vec = list(model.embed([candidate]))[0]\n", + " score = cosine_similarity(query_vec, candidate_vec)\n", + " print(f\"{score:.3f} | {candidate!r}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Distance Metric Comparison\n", + "\n", + "| Metric | Best for | Notes |\n", + "|--------|----------|-------|\n", + "| Cosine | Text similarity, NLP (Natural Language Processing) models | Robust to different vector magnitudes. Most common default. |\n", + "| Dot product | Vectors already normalized to unit length | Numerically identical to cosine similarity once vectors are unit length, same score, not a separate metric. |\n", + "| Euclidean (L2) | Image embeddings, spatial data | Sensitive to magnitude; works best with models trained for it. |\n", + "| Manhattan (L1) | Grid-like or count-based data | Sums absolute differences per dimension rather than squaring them first, making it less affected by extreme values in any single dimension. |\n", + "\n", + "**Cosine vs. dot product:** for vectors normalized to unit length, dot product produces the exact same ranking as cosine similarity, it's a cheaper way to compute the same result, not a different metric. That's why Qdrant normalizes vectors on upload and computes a \"Cosine\" collection as a dot product internally.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 6. Why Similarity Alone Is Not Enough\n", + "\n", + "Sections 1 and 2 showed keyword search failing on synonyms, paraphrasing, polysemy, and word order. It's tempting to read that as \"semantic search replaces keyword search.\" It doesn't, each is strong exactly where the other is weak, as the next two cases show. (Filtering by recency, permissions, or other payload values, and combining that with ranking signals, is a separate layer on top of similarity, later modules cover it once you have a collection to filter.)\n", + "\n", + "### Word Order and Negation Still Trip It Up\n", + "\n", + "Section 2 said keyword search can't tell \"dog bites man\" from \"man bites dog.\" You'd expect semantic search to fix that. It mostly doesn't:\n", + "\n", + "| Pair | Cosine similarity |\n", + "|------|-------------------|\n", + "| \"dog bites man\" vs \"man bites dog\" | 0.907 |\n", + "| \"safe for kids\" vs \"harmful to kids\" | 0.779 |\n", + "| \"dog bites man\" vs \"a canine attacked a person\" | 0.570 |\n", + "\n", + "The first two rows score high, even though each pair means something different: one flips who's doing the biting, the other flips safe into dangerous. The model mostly notices that the two sentences share almost all the same words, so it calls them similar, even though a person would read them as opposites right away.\n", + "\n", + "The third row is the sharpest version of the problem: \"a canine attacked a person\" is a genuine paraphrase of \"dog bites man,\" meaning the same thing in different words, yet it scores lower (0.570) than the reversed, opposite-meaning sentence (0.907). Shared words move the score more than shared meaning does.\n", + "\n", + "So: semantic search is great at synonyms and paraphrasing, but shaky on word order and negation. Don't rely on a similarity score alone anywhere it actually matters whether something is \"safe\" or \"not safe.\"\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Reproduce the table above\n", + "word_order_pairs = [\n", + " (\"dog bites man\", \"man bites dog\"),\n", + " (\"safe for kids\", \"harmful to kids\"),\n", + " (\"dog bites man\", \"a canine attacked a person\"),\n", + "]\n", + "\n", + "for a, b in word_order_pairs:\n", + " a_vec = list(model.embed([a]))[0]\n", + " b_vec = list(model.embed([b]))[0]\n", + " score = cosine_similarity(a_vec, b_vec)\n", + " print(f\"{score:.3f} | {a!r} vs {b!r}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Exact Matching: Where Keyword Search Wins\n", + "\n", + "Not every query needs semantic understanding. A query for an exact SKU (Stock Keeping Unit, the unique code a retailer assigns to one specific product), like \"SKU-48291,\" needs an exact match instead. Try it yourself: embed the query and three candidate SKUs, then compare their cosine scores.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from fastembed import TextEmbedding\n", + "import numpy as np\n", + "\n", + "model = TextEmbedding(model_name=\"sentence-transformers/all-MiniLM-L6-v2\")\n", + "\n", + "def cosine_similarity(a, b):\n", + " return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))\n", + "\n", + "query = \"SKU-48291 issue\"\n", + "candidates = [\"SKU-48292\", \"SKU-48291\", \"SKU-48290\"] # wrong, correct, wrong\n", + "\n", + "query_vec = list(model.embed([query]))[0]\n", + "\n", + "for candidate in candidates:\n", + " candidate_vec = list(model.embed([candidate]))[0]\n", + " score = cosine_similarity(query_vec, candidate_vec)\n", + " print(f\"{score:.3f} | {candidate}\")\n", + "\n", + "# Real output:\n", + "# 0.730 | SKU-48292 (wrong product)\n", + "# 0.734 | SKU-48291 (correct product)\n", + "# 0.765 | SKU-48290 (wrong product, scores HIGHEST)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Semantic similarity here doesn't just drift toward the wrong SKU, it ranks the wrong one first. A keyword-style exact filter gets it right, trivially: restrict the results to points where the `sku` payload field equals `\"SKU-48291\"`, and only that one product comes back, no embedding, no similarity score, just an exact match. Module 2 builds a real filtered Qdrant query once there's a collection with payloads to filter.\n", + "\n", + "Keyword matching isn't obsolete, it's exactly the right tool when a query needs to hit one precise token. Dense similarity finds the general neighborhood of relevant results; exact keyword matching finds the right point within it. Neither replaces the other.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 7. When a System Needs Both\n", + "\n", + "The SKU example above is why some production systems run semantic and exact retrieval together rather than picking one: a single collection can serve queries that need meaning and queries that need one precise token. This isn't a universal requirement, plenty of systems only ever need one or the other, but when a use case needs both, that's called **hybrid search**, combining **dense** (semantic/vector) retrieval with **sparse** (keyword-style, e.g. BM25) retrieval.\n", + "\n", + "Hybrid search is the next module's topic.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 8. References & Further Reading\n", + "\n", + "**Qdrant docs:**\n", + "\n", + "- [Qdrant Documentation Overview](https://qdrant.tech/documentation/overview/) \u2014 How Qdrant's vector search engine fits together: collections, points, payloads, and APIs.\n", + "- [Distance Metrics](https://qdrant.tech/course/essentials/day-1/distance-metrics/) \u2014 Cosine, dot product, and Euclidean distance compared, and how to pick the right one for your embedding model.\n", + "- [Filtering](https://qdrant.tech/documentation/search/filtering/) \u2014 Payload filter syntax, indexed fields, and combining filters with vector queries.\n", + "\n", + "**Go deeper:**\n", + "\n", + "- [Vector Embeddings Explained](https://qdrant.tech/articles/what-are-embeddings/)\n", + "- [What Is a Vector Database?](https://qdrant.tech/articles/what-is-a-vector-database/)\n", + "- [FastEmbed: Qdrant's Efficient Python Library for Embedding Generation](https://qdrant.tech/articles/fastembed/)\n", + "- [Fine-Tuning Sparse Embeddings for E-Commerce Search, Part 1: Why Sparse Embeddings Beat BM25](https://qdrant.tech/articles/sparse-embeddings-ecommerce-part-1/)\n", + "- [What Is RAG in AI?](https://qdrant.tech/articles/what-is-rag-in-ai/)\n", + "\n", + "**Definitions:**\n", + "\n", + "- [Cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity)\n", + "- [Dot product](https://en.wikipedia.org/wiki/Dot_product)\n", + "- [Euclidean distance](https://en.wikipedia.org/wiki/Euclidean_distance)\n", + "- [Manhattan (taxicab) distance](https://en.wikipedia.org/wiki/Taxicab_geometry)\n", + "\n", + "## What's Next: Module 2\n", + "\n", + "In the next module, we'll break down:\n", + "\n", + "- What is a vector, and why does it have hundreds to thousands of dimensions?\n", + "- How do dimensions actually represent meaning?\n", + "- How similarity really works under the hood, and when it fails.\n", + "- Your first Qdrant collection: points, payloads, and your first query.\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + }, + "colab": { + "provenance": [], + "name": "Module1.ipynb" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/Beginner-course/Module2.ipynb b/Beginner-course/Module2.ipynb new file mode 100644 index 0000000..b4a8509 --- /dev/null +++ b/Beginner-course/Module2.ipynb @@ -0,0 +1,265 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "cells": [ + { + "cell_type": "markdown", + "source": "# Module 2: First Principles of Vector Search\n\n## What you will do\n\n1. Turn text into a vector and look at what a collection needs to know about it.\n2. Create your first Qdrant collection.\n3. Upsert points with vectors and payloads.\n4. Run a top-K query, then constrain it with a payload filter.\n5. See why the payload index has to exist before you ingest.\n\n**Tip:** every cell below is already run, so you can read it straight through. It uses Qdrant in local mode, so there is nothing to sign up for and no API key to paste.\n\nCompanion notebook to the [Module 2 lesson](https://qdrant.tech/course/beginners/module-2/).", + "metadata": { + "id": "ujOeHwdFcAef" + } + }, + { + "cell_type": "markdown", + "source": "## Setup\n\n`qdrant-client[fastembed]` bundles local embedding models. Passing a `models.Document` lets the client embed text for us before upload and at query time, so we never handle raw vectors by hand. The model is `all-MiniLM-L6-v2`, the same one from Module 1.", + "metadata": { + "id": "AZhnM6Jy8c1r" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "ihtYlKNxHddm" + }, + "execution_count": null, + "source": "!pip install -q \"qdrant-client[fastembed]\" ", + "outputs": [] + }, + { + "cell_type": "markdown", + "source": "## 1. What a Collection Needs to Know\n\nA collection is a container for points, and it is declared up front with two things it can never change silently: how many dimensions each vector has, and which distance metric compares them.\n\nThose two numbers are not stylistic. Get the size wrong and every upsert fails. Get the metric wrong and your results are quietly worse rather than broken.", + "metadata": { + "id": "QAtKCsXRpJG2" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "Tr8hzUjEcPVJ", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "e5b997df" + }, + "execution_count": null, + "source": "from qdrant_client import QdrantClient, models\n\nMODEL = \"sentence-transformers/all-MiniLM-L6-v2\"\nVECTOR_SIZE = 384 # fixed by the model\nDISTANCE = models.Distance.COSINE # the default for text embeddings\n\n# Local mode: an in-process Qdrant, ideal for notebooks and CI.\nclient = QdrantClient(\":memory:\")\n\n# On Qdrant Cloud you would swap in:\n# client = QdrantClient(url=\"https://YOUR-CLUSTER.cloud.qdrant.io:6333\",\n# api_key=\"YOUR_API_KEY\")\n\nclient.create_collection(\n collection_name=\"articles\",\n vectors_config=models.VectorParams(size=VECTOR_SIZE, distance=DISTANCE),\n)\n\nprint(client.get_collection(\"articles\").config.params.vectors)", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "size=384 distance= hnsw_config=None quantization_config=None on_disk=None datatype=None multivector_config=None\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 2. Index What You Will Filter, Before You Ingest\n\nThe collection is empty, and this is the moment to declare the payload fields you plan to filter on.\n\nThe ordering matters. Qdrant adds filter-aware edges to its vector index based on indexed payload values, and it can only add them for indexes that already exist when that index is built. Create a payload index after ingesting and you have to rebuild the vector index to get the benefit.\n\nLocal mode ignores payload indexes entirely, so the call below is a no-op here and will emit a warning. Write it anyway: it is the habit that transfers to a real server, and Module 4 goes into why.", + "metadata": { + "id": "DPdRO9YrxPbC" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "khdVipy2eBI2", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "dd465ef2" + }, + "execution_count": null, + "source": "import warnings\n\nwith warnings.catch_warnings():\n warnings.simplefilter(\"ignore\") # local mode warns that indexes do nothing here\n client.create_payload_index(\n collection_name=\"articles\",\n field_name=\"category\",\n field_schema=models.PayloadSchemaType.KEYWORD,\n )\n\nprint(\"payload index declared on 'category'\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "payload index declared on 'category'\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 3. Points: Vector Plus Payload\n\nA point is an id, one or more vectors, and a payload. The vector is what gets searched. The payload is everything you want back, or want to filter on, and it is ordinary JSON.\n\nNote what goes in the payload below. `title` is there to be returned to the user. `category` is there to be filtered. `published` is there to be sorted or range-filtered later. None of them affect similarity.", + "metadata": { + "id": "kooEKqazwJ7Q" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "FMPd3W2XyygN", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "01325002" + }, + "execution_count": null, + "source": "documents = [\n {\"id\": 1, \"title\": \"Car repair guide\", \"category\": \"automotive\", \"published\": 2023},\n {\"id\": 2, \"title\": \"Automobile maintenance 101\", \"category\": \"automotive\", \"published\": 2024},\n {\"id\": 3, \"title\": \"How to cook pasta\", \"category\": \"food\", \"published\": 2024},\n {\"id\": 4, \"title\": \"Best pizza in Chicago\", \"category\": \"food\", \"published\": 2022},\n]\n\nclient.upload_points(\n collection_name=\"articles\",\n points=[\n models.PointStruct(\n id=doc[\"id\"],\n vector=models.Document(text=doc[\"title\"], model=MODEL),\n payload=doc,\n )\n for doc in documents\n ],\n)\n\nprint(\"points in collection:\", client.count(\"articles\").count)", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "points in collection: 4\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 4. Your First Query\n\nSame model at query time as at ingestion time. This is the rule that breaks the most beginner pipelines: two different models produce two different vector spaces, and nothing will warn you, you will just get nonsense rankings.\n\nThe query below shares no words with any stored title.", + "metadata": { + "id": "gwb2Mjpvwh09" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "CDfgvqZkb6Gj", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "808fdb45" + }, + "execution_count": null, + "source": "results = client.query_points(\n collection_name=\"articles\",\n query=models.Document(text=\"fixing my vehicle\", model=MODEL),\n limit=4,\n)\n\nfor r in results.points:\n print(f\"{r.score:.3f} {r.payload['title']:30} ({r.payload['category']})\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "0.699 Car repair guide (automotive)\n0.565 Automobile maintenance 101 (automotive)\n0.045 Best pizza in Chicago (food)\n0.032 How to cook pasta (food)\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Both automotive articles come back above both food articles, and the query contained neither the word \"car\" nor \"automobile\". That is the Module 1 embedding doing its job, now with storage and ranking around it.\n\nAlso worth noticing: every result carries its payload back. You did not have to look anything up in a second database.", + "metadata": { + "id": "kVHWunY90XYT" + } + }, + { + "cell_type": "markdown", + "source": "## 5. Filtering\n\nA filter is a hard constraint, not a ranking hint. It is evaluated while the search runs, so excluded points never occupy a slot in your top-K.\n\nRun the same query twice, once unfiltered and once scoped to `food`, and watch the entire result set change rather than just reorder.", + "metadata": { + "id": "oGwbbrqQ7B69" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "7wnommM30DOX", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "1aecc72c" + }, + "execution_count": null, + "source": "from qdrant_client.models import Filter, FieldCondition, MatchValue\n\ndef search(text, query_filter=None, limit=4):\n return client.query_points(\n collection_name=\"articles\",\n query=models.Document(text=text, model=MODEL),\n query_filter=query_filter,\n limit=limit,\n ).points\n\nfood_only = Filter(must=[FieldCondition(key=\"category\", match=MatchValue(value=\"food\"))])\n\nprint(\"UNFILTERED 'fixing my vehicle':\")\nfor r in search(\"fixing my vehicle\"):\n print(f\" {r.score:.3f} {r.payload['title']:30} ({r.payload['category']})\")\n\nprint()\nprint(\"FILTERED to category=food:\")\nfor r in search(\"fixing my vehicle\", query_filter=food_only):\n print(f\" {r.score:.3f} {r.payload['title']:30} ({r.payload['category']})\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "UNFILTERED 'fixing my vehicle':\n 0.699 Car repair guide (automotive)\n 0.565 Automobile maintenance 101 (automotive)\n 0.045 Best pizza in Chicago (food)\n 0.032 How to cook pasta (food)\n\nFILTERED to category=food:\n 0.045 Best pizza in Chicago (food)\n 0.032 How to cook pasta (food)\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "The filtered run returns only food articles, and their scores are unchanged from the unfiltered run. The filter did not rescore anything. It removed candidates.\n\nThat distinction matters later: because filtering happens during retrieval rather than after it, a filtered search still returns a full top-K of valid results instead of a short list of leftovers.", + "metadata": { + "id": "uX8yy6Skhj4Y" + } + }, + { + "cell_type": "markdown", + "source": "## 6. Range Filters and Combining Conditions\n\n`must` is AND, `should` is OR, and `must_not` excludes. Numeric and date fields take ranges. They compose in one filter object.", + "metadata": { + "id": "jZ8OvIia8OG5" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "A2ZnpsoKqzi4", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "579e6e88" + }, + "execution_count": null, + "source": "from qdrant_client.models import Range\n\nrecent_automotive = Filter(\n must=[\n FieldCondition(key=\"category\", match=MatchValue(value=\"automotive\")),\n FieldCondition(key=\"published\", range=Range(gte=2024)),\n ]\n)\n\nprint(\"automotive AND published >= 2024:\")\nfor r in search(\"fixing my vehicle\", query_filter=recent_automotive):\n print(f\" {r.score:.3f} {r.payload['title']:30} ({r.payload['published']})\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "automotive AND published >= 2024:\n 0.565 Automobile maintenance 101 (2024)\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "One result, because only one point satisfies both conditions. Note that `published` was never indexed, and in local mode that costs nothing. On a real server it would work but scan, and on Qdrant Cloud with strict mode on it would be rejected outright. Every field you filter on needs an index.", + "metadata": { + "id": "GbBlaXkDSIuG" + } + }, + { + "cell_type": "markdown", + "source": "## 7. Similarity Under the Hood\n\nQdrant does not compare your query against every stored vector. It walks an HNSW graph, a layered structure where each layer is a sparser shortcut over the one below, so search starts coarse and refines.\n\nTwo consequences worth carrying forward.\n\nIt is **approximate**. HNSW trades a small amount of recall for a large amount of speed, so a top-K is very likely, not certainly, the true nearest neighbours.\n\nIt is **not always used**. Below a size threshold, scanning every vector is genuinely faster, so Qdrant does that instead. That is why a tiny notebook collection like this one returns exact results, and why timing measurements here tell you nothing about production.", + "metadata": { + "id": "IWgIprVFIV4B" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "LFFQCHD6R26q", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "8fd21713" + }, + "execution_count": null, + "source": "info = client.get_collection(\"articles\")\nprint(\"points: \", info.points_count)\nprint(\"vector size: \", info.config.params.vectors.size)\nprint(\"distance: \", info.config.params.vectors.distance)\nprint(\"hnsw m: \", info.config.hnsw_config.m)\nprint(\"hnsw ef_construct: \", info.config.hnsw_config.ef_construct)\nprint(\"full_scan_threshold:\", info.config.hnsw_config.full_scan_threshold, \"(KB)\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "points: 4\nvector size: 384\ndistance: Cosine\nhnsw m: 16\nhnsw ef_construct: 100\nfull_scan_threshold: 10000 (KB)\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "`m` is how many edges each node keeps, `ef_construct` is how hard the graph works while being built, and `full_scan_threshold` is the size below which Qdrant skips the graph. You will tune these in later work; for now the point is that they are collection-level settings you inherit by default.", + "metadata": { + "id": "ePW3jSOi27n7" + } + }, + { + "cell_type": "markdown", + "source": "## Your turn\n\nTwo changes to try.\n\nAdd a point in a third category and rerun the unfiltered query. Does it land where you expect relative to the existing four?\n\nThen change `DISTANCE` to `models.Distance.EUCLID`, recreate the collection, and re-ingest. Compare the scores to the cosine run. They will not be on the same scale, which is exactly why the metric is declared once per collection rather than per query.", + "metadata": { + "id": "yE9ZkAFvmtSb" + } + }, + { + "cell_type": "markdown", + "source": "## What's next: Module 3\n\n- Where dense-only search fails: exact codes, model numbers, and SKUs\n- Sparse vectors, BM25, and the inverted index\n- Hybrid search: running dense and sparse together and fusing the results\n\n[Continue to Module 3](https://qdrant.tech/course/beginners/module-3/)", + "metadata": { + "id": "IBbuMFd9W8gq" + } + } + ] +} diff --git a/Beginner-course/Module3.ipynb b/Beginner-course/Module3.ipynb new file mode 100644 index 0000000..d84bb0b --- /dev/null +++ b/Beginner-course/Module3.ipynb @@ -0,0 +1,393 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "cells": [ + { + "cell_type": "markdown", + "source": "# Module 3: Sparse vs Dense vs Hybrid Search\n\n## What you will do\n\n1. See the gap that dense-only search leaves on exact names and codes.\n2. Compare the two families of search, dense (meaning) and sparse (exact tokens).\n3. Build a hybrid collection and fuse both by rank.\n4. Run the dense vs hybrid experiment and watch the ranking change.\n5. Prove where a filter has to go in a hybrid query, by breaking it on purpose.\n\n**Tip for Colab:** run cells top to bottom. The first install downloads small embedding models, so it takes a minute.\n\nCompanion notebook to the [Module 3 lesson](https://qdrant.tech/course/beginners/module-3/).", + "metadata": { + "id": "ujOeHwdFcAef" + } + }, + { + "cell_type": "markdown", + "source": "## Setup\n\n`qdrant-client[fastembed]` bundles local embedding models. Passing a `models.Document` lets the client embed text for us before upload and at query time, so we never manage the vectors by hand.", + "metadata": { + "id": "AZhnM6Jy8c1r" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "ihtYlKNxHddm" + }, + "execution_count": null, + "source": "!pip install -q \"qdrant-client[fastembed]\" ", + "outputs": [] + }, + { + "cell_type": "markdown", + "source": "## 1. Where We Left Off\n\nIn Module 2 you built a full pipeline: raw text to vector to store to top-K query. Dense-only retrieval is great for semantic and contextual search, but it struggles on precise product names and model numbers.\n\nTake the query `iPhone 15`. The user wants exactly this product: no synonyms, no paraphrasing. Dense-only search tends to return the whole product line, because \"iPhone 14\", \"iPhone 15\", and \"iPhone 15 Pro Max\" sit close together in embedding space. IDs, codes, and specific model names need exact matching, not semantic neighborhood. That is the gap sparse search fills, and we will reproduce it live in Section 4.", + "metadata": { + "id": "QAtKCsXRpJG2" + } + }, + { + "cell_type": "markdown", + "source": "## 2. The Two Families of Search\n\n### Dense search (semantic)\n\nA dense vector has a small, fixed number of dimensions (for example 384), and every dimension holds a value. Two texts with similar meaning produce vectors that are close in space, even if they share no words. That is why `car repair` sits near `automobile maintenance`.\n\n### Sparse search (keyword based)\n\nSparse vectors are token based. Each dimension maps to a token, and only the tokens that actually appear carry a non-zero value. A vocabulary can be tens of thousands of tokens, but a given text activates only the handful it contains, so sparse vectors are stored as two parallel arrays: the `indices` of the non-zero dimensions and the `values` at those positions.\n\n```python\nsparse_vector = {\n \"indices\": [142, 9325, 44001], # token IDs: 'nike', 'pegasus', '40'\n \"values\": [2.3, 1.2, 0.8], # weight per token\n}\n```\n\nSparse similarity in Qdrant is always the dot product. There is no metric to choose, unlike the dense side where you pick Cosine, Dot, or Euclidean.\n\n#### Sparse models: BM25, SPLADE, miniCOIL\n\n| Model | How it assigns weights | Notes |\n|-------|------------------------|-------|\n| BM25 | Statistical: term frequency and inverse document frequency, no training | Classic, fast, interpretable. Scores tokens exactly as written. FastEmbed handle `Qdrant/bm25`. |\n| SPLADE | Neural: a transformer expands text with related terms and weights them | Captures some synonymy while staying sparse. More compute than BM25. |\n| miniCOIL | Neural, contextualized term weighting on BM25's exact vocabulary | Context aware exact match without full expansion cost. FastEmbed handle `Qdrant/minicoil-v1`. |\n\nminiCOIL is Qdrant's recommendation for new projects. We use BM25 here because it needs no model inference at all, which keeps this notebook fast and every score traceable by hand.\n\n#### How sparse is indexed\n\nQdrant uses an inverted index: for every token it keeps a posting list of every point where that token has a non-zero weight. A query only walks the posting lists for tokens it contains, skipping every point that shares none. HNSW (Module 2) is approximate, but the sparse index is exact.\n\n### Key insight\n\nDense = meaning. Sparse = exact matching. Neither is complete alone. Every real query carries both semantic intent (what the user means) and exact constraints (what the user needs precisely).", + "metadata": { + "id": "Tr8hzUjEcPVJ" + } + }, + { + "cell_type": "markdown", + "source": "## 3. Hybrid Search: Dense + Sparse\n\nHybrid search runs dense and sparse retrieval in the same request, then fuses the two ranked candidate lists into one result set: semantic understanding with exact-match precision. Payload filters constrain both retrievers while they run, which Section 7 covers.\n\n**Reciprocal Rank Fusion (RRF)** merges the dense list and the sparse list using each candidate's *position* in the two lists, not its raw score. A document ranked high by both retrievers rises to the top. Because it ignores raw scores, RRF is robust to the fact that dense and sparse scores live on completely different scales.", + "metadata": { + "id": "2tRKJC06DPdR" + } + }, + { + "cell_type": "markdown", + "source": "## 4. Setting Up Hybrid Search in Qdrant\n\n### Step 1: create a hybrid collection\n\nDeclare both a dense and a sparse vector config on one collection. Every point will carry both.\n\nTwo details in the cell below are easy to skip and expensive to skip. The sparse config needs `modifier=models.Modifier.IDF`, because BM25-style vectors store only term frequency and Qdrant applies the inverse-document-frequency half of the formula at query time. Without it you are not scoring BM25. And the payload index on `in_stock` is declared before any data is ingested, which is the ordering Module 4 explains.", + "metadata": { + "id": "O9YrxPbCkhdV" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "ipy2eBI2Y1rz", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "5ef22337" + }, + "execution_count": null, + "source": "import warnings\nfrom qdrant_client import QdrantClient, models\n\nDENSE_MODEL = \"sentence-transformers/all-MiniLM-L6-v2\" # 384-dim dense embeddings\nSPARSE_MODEL = \"Qdrant/bm25\" # exact-token sparse\n\nclient = QdrantClient(\":memory:\")\n\nclient.create_collection(\n collection_name=\"products\",\n vectors_config={\n \"dense\": models.VectorParams(size=384, distance=models.Distance.COSINE),\n },\n sparse_vectors_config={\n \"sparse\": models.SparseVectorParams(\n modifier=models.Modifier.IDF # required for BM25 scoring\n ),\n },\n)\n\nwith warnings.catch_warnings():\n warnings.simplefilter(\"ignore\") # local mode warns that indexes do nothing here\n client.create_payload_index(\n collection_name=\"products\",\n field_name=\"in_stock\",\n field_schema=models.PayloadSchemaType.BOOL,\n )\n\nprint(\"Hybrid collection 'products' created, with in_stock indexed.\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Hybrid collection 'products' created, with in_stock indexed.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### Step 2: insert points with both vectors\n\nEach point carries a dense embedding and a sparse vector. We pass a `models.Document` and name the model. The client embeds it locally with FastEmbed before upload. The catalog below mixes a phone product line (to reproduce the `iPhone 15` problem) with running shoes (for the Nike example) and includes an exact SKU.", + "metadata": { + "id": "KqazwJ7QFMPd" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "3W2XyygNdemk", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "5002150d" + }, + "execution_count": null, + "source": "catalog = [\n {\"id\": 1, \"name\": \"iPhone 14\", \"sku\": \"APL-IP14\", \"price\": 699, \"in_stock\": True},\n {\"id\": 2, \"name\": \"iPhone 15\", \"sku\": \"APL-IP15\", \"price\": 799, \"in_stock\": True},\n {\"id\": 3, \"name\": \"iPhone 15 Pro Max\", \"sku\": \"APL-IP15PM\", \"price\": 1199, \"in_stock\": True},\n {\"id\": 4, \"name\": \"iPhone 13 mini\", \"sku\": \"APL-IP13M\", \"price\": 599, \"in_stock\": False},\n {\"id\": 5, \"name\": \"Nike Pegasus 40 running shoes\", \"sku\": \"NK-PEG40\", \"price\": 130, \"in_stock\": True},\n {\"id\": 6, \"name\": \"Nike Pegasus 39 running shoes\", \"sku\": \"NK-PEG39\", \"price\": 110, \"in_stock\": True},\n {\"id\": 7, \"name\": \"Adidas Ultraboost running shoes\", \"sku\": \"AD-UB22\", \"price\": 180, \"in_stock\": True},\n {\"id\": 8, \"name\": \"Widget assembly part SKU-48291\", \"sku\": \"SKU-48291\", \"price\": 12, \"in_stock\": True},\n]\n\nclient.upload_points(\n collection_name=\"products\",\n points=[\n models.PointStruct(\n id=item[\"id\"],\n vector={\n \"dense\": models.Document(text=item[\"name\"], model=DENSE_MODEL),\n \"sparse\": models.Document(text=item[\"name\"] + \" \" + item[\"sku\"], model=SPARSE_MODEL),\n },\n payload=item,\n )\n for item in catalog\n ],\n)\nprint(\"Upserted\", client.count(\"products\").count, \"products.\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Upserted 8 products.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### Step 3: three ways to search the same collection\n\nA small helper prints results compactly. Then the same query runs three ways: dense-only, sparse-only, and hybrid.\n\nLook closely at `hybrid()`. The filter is passed into **each** `Prefetch`, not to `query_points` as a top-level `query_filter`. That placement is the whole lesson of Section 7, and we break it on purpose later to show what goes wrong.", + "metadata": { + "id": "Mjpvwh09CDfg" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "vqZkb6GjHbG8" + }, + "execution_count": null, + "source": "def show(title, points):\n print(title)\n for r in points:\n print(f\" [{r.score:.4f}] {r.payload['name']} (sku={r.payload['sku']}, in_stock={r.payload['in_stock']})\")\n print()\n\ndef dense_only(text, query_filter=None, limit=5):\n return client.query_points(\n \"products\",\n query=models.Document(text=text, model=DENSE_MODEL),\n using=\"dense\",\n query_filter=query_filter, # no prefetch here, so top level is correct\n limit=limit,\n ).points\n\ndef sparse_only(text, query_filter=None, limit=5):\n return client.query_points(\n \"products\",\n query=models.Document(text=text, model=SPARSE_MODEL),\n using=\"sparse\",\n query_filter=query_filter, # no prefetch here either\n limit=limit,\n ).points\n\ndef hybrid(text, query_filter=None, limit=5, fusion=\"rrf\"):\n # The filter goes INSIDE each prefetch, so both retrievers only ever\n # consider valid points. See Section 7 for why the top level is wrong here.\n fuse = (models.RrfQuery(rrf=models.Rrf()) if fusion == \"rrf\"\n else models.FusionQuery(fusion=models.Fusion.DBSF))\n return client.query_points(\n \"products\",\n prefetch=[\n models.Prefetch(query=models.Document(text=text, model=DENSE_MODEL),\n using=\"dense\", filter=query_filter, limit=20),\n models.Prefetch(query=models.Document(text=text, model=SPARSE_MODEL),\n using=\"sparse\", filter=query_filter, limit=20),\n ],\n query=fuse,\n limit=limit,\n ).points", + "outputs": [] + }, + { + "cell_type": "markdown", + "source": "The dense-only run reproduces the problem from Section 1: semantically similar phones cluster together, and the exact model the user typed does not reliably sit on top. Sparse-only, by contrast, locks onto the literal tokens.", + "metadata": { + "id": "1RqwkVHWunY9" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "0XYToGwbbrqQ", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "f7eff533" + }, + "execution_count": null, + "source": "show(\"DENSE-ONLY 'iPhone 15' (semantic neighborhood, exact model can drift):\", dense_only(\"iPhone 15\"))\nshow(\"SPARSE-ONLY 'iPhone 15' (exact tokens win):\", sparse_only(\"iPhone 15\"))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "DENSE-ONLY 'iPhone 15' (semantic neighborhood, exact model can drift):\n [1.0000] iPhone 15 (sku=APL-IP15, in_stock=True)\n [0.8760] iPhone 14 (sku=APL-IP14, in_stock=True)\n [0.8149] iPhone 15 Pro Max (sku=APL-IP15PM, in_stock=True)\n [0.6801] iPhone 13 mini (sku=APL-IP13M, in_stock=False)\n [0.1764] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n\nSPARSE-ONLY 'iPhone 15' (exact tokens win):\n [3.3050] iPhone 15 (sku=APL-IP15, in_stock=True)\n [3.2874] iPhone 15 Pro Max (sku=APL-IP15PM, in_stock=True)\n [1.1605] iPhone 14 (sku=APL-IP14, in_stock=True)\n [1.1574] iPhone 13 mini (sku=APL-IP13M, in_stock=False)\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "The clearest case is an exact code. A dense model has never really \"seen\" `SKU-48291` as a meaningful concept, so it drifts. Sparse matches the literal token exactly.", + "metadata": { + "id": "mmM30DOXfO4W" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "UDlWuX8yy6Sk", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "22ec2dfa" + }, + "execution_count": null, + "source": "show(\"DENSE-ONLY 'SKU-48291' (drifts):\", dense_only(\"SKU-48291\"))\nshow(\"SPARSE-ONLY 'SKU-48291' (exact hit):\", sparse_only(\"SKU-48291\"))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "DENSE-ONLY 'SKU-48291' (drifts):\n [0.4544] Widget assembly part SKU-48291 (sku=SKU-48291, in_stock=True)\n [0.2397] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.1948] Nike Pegasus 40 running shoes (sku=NK-PEG40, in_stock=True)\n [0.1403] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n [0.1266] iPhone 15 (sku=APL-IP15, in_stock=True)\n\nSPARSE-ONLY 'SKU-48291' (exact hit):\n [6.7829] Widget assembly part SKU-48291 (sku=SKU-48291, in_stock=True)\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Now hybrid. Both prefetches run as part of one request and return up to 20 candidates each. RRF merges them by rank, then `limit` takes the top results. Because each prefetch carries the filter, out-of-stock products never enter either candidate set.", + "metadata": { + "id": "vIia8OG5A2Zn" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "psoKqzi4vCK4", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "6e888072" + }, + "execution_count": null, + "source": "from qdrant_client.models import Filter, FieldCondition, MatchValue\n\nin_stock_filter = Filter(must=[FieldCondition(key=\"in_stock\", match=MatchValue(value=True))])\n\nshow(\"HYBRID 'Nike Pegasus 40 size 10' (semantic + exact, in stock only):\",\n hybrid(\"Nike Pegasus 40 size 10\", query_filter=in_stock_filter))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "HYBRID 'Nike Pegasus 40 size 10' (semantic + exact, in stock only):\n [1.0000] Nike Pegasus 40 running shoes (sku=NK-PEG40, in_stock=True)\n [0.6667] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.2500] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n [0.2000] iPhone 15 (sku=APL-IP15, in_stock=True)\n [0.1667] iPhone 14 (sku=APL-IP14, in_stock=True)\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### Try it: watch the ranking change\n\nThe experiment from the lesson, side by side. Compare where the exact target lands under dense-only versus hybrid.", + "metadata": { + "id": "aXkDSIuGIWgI" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "prVFIV4BLFFQ", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "787fbef4" + }, + "execution_count": null, + "source": "q = \"Nike Pegasus 40\"\nshow(f\"DENSE-ONLY '{q}':\", dense_only(q))\nshow(f\"HYBRID '{q}':\", hybrid(q))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "DENSE-ONLY 'Nike Pegasus 40':\n [0.8815] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.8713] Nike Pegasus 40 running shoes (sku=NK-PEG40, in_stock=True)\n [0.6043] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n [0.2403] iPhone 15 (sku=APL-IP15, in_stock=True)\n [0.2044] iPhone 14 (sku=APL-IP14, in_stock=True)\n\n" + }, + { + "output_type": "stream", + "name": "stdout", + "text": "HYBRID 'Nike Pegasus 40':\n [0.8333] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.8333] Nike Pegasus 40 running shoes (sku=NK-PEG40, in_stock=True)\n [0.2500] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n [0.2000] iPhone 15 (sku=APL-IP15, in_stock=True)\n [0.1667] iPhone 14 (sku=APL-IP14, in_stock=True)\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### Try it: put the filter in the wrong place\n\nNow the part worth doing yourself, because nothing raises an error when you get it wrong.\n\n`hybrid_wrong()` below is identical to `hybrid()` except the filter moves from inside the prefetches to a top-level `query_filter`. Then we mark `iPhone 15` out of stock and run both against an in-stock-only filter. The correct version drops it. The broken version should not.", + "metadata": { + "id": "I60ihBeoePW3" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "jSOi27n7yE9Z", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "268535b0" + }, + "execution_count": null, + "source": "def hybrid_wrong(text, query_filter=None, limit=5):\n \"\"\"Same query, filter at the top level instead of inside each prefetch.\"\"\"\n return client.query_points(\n \"products\",\n prefetch=[\n models.Prefetch(query=models.Document(text=text, model=DENSE_MODEL),\n using=\"dense\", limit=20),\n models.Prefetch(query=models.Document(text=text, model=SPARSE_MODEL),\n using=\"sparse\", limit=20),\n ],\n query=models.RrfQuery(rrf=models.Rrf()),\n query_filter=query_filter, # too late: prefetches have already run\n limit=limit,\n ).points\n\nclient.set_payload(\"products\", payload={\"in_stock\": False}, points=[2]) # iPhone 15\n\nshow(\"CORRECT filter inside each prefetch (iPhone 15 is gone):\",\n hybrid(\"iPhone 15\", query_filter=in_stock_filter))\nshow(\"BROKEN filter at the top level (iPhone 15 comes back):\",\n hybrid_wrong(\"iPhone 15\", query_filter=in_stock_filter))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "CORRECT filter inside each prefetch (iPhone 15 is gone):\n [0.8333] iPhone 14 (sku=APL-IP14, in_stock=True)\n [0.8333] iPhone 15 Pro Max (sku=APL-IP15PM, in_stock=True)\n [0.2500] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.2000] Nike Pegasus 40 running shoes (sku=NK-PEG40, in_stock=True)\n [0.1667] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n\n" + }, + { + "output_type": "stream", + "name": "stdout", + "text": "BROKEN filter at the top level (iPhone 15 comes back):\n [1.0000] iPhone 15 (sku=APL-IP15, in_stock=False)\n [0.5833] iPhone 14 (sku=APL-IP14, in_stock=True)\n [0.5833] iPhone 15 Pro Max (sku=APL-IP15PM, in_stock=True)\n [0.4000] iPhone 13 mini (sku=APL-IP13M, in_stock=False)\n [0.1667] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "There it is. Same filter, same query, one line of difference, and the broken version returns a product that is out of stock, without a warning or an error.\n\nThe reason is execution order. Once a query has prefetches, Qdrant runs those first and applies the main query to their results. A top-level filter therefore never reaches the retrievers: each one searches the whole catalog, and the filter only trims the already-fused list at the end. That is post-filtering, and with a selective filter it can leave you with nothing at all.\n\nThe rule is simple. No prefetch, use `query_filter`. Prefetch, put the filter in every prefetch.", + "metadata": { + "id": "IBbuMFd9W8gq" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "cWqiA4Yqj4JR", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "10b61fac" + }, + "execution_count": null, + "source": "client.set_payload(\"products\", payload={\"in_stock\": True}, points=[2]) # restore iPhone 15\nprint(\"iPhone 15 back in stock:\", client.retrieve(\"products\", ids=[2])[0].payload[\"in_stock\"])", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "iPhone 15 back in stock: True\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 5. Fusion Strategies\n\nOnce both retrievers return candidates, a fusion algorithm merges them into one ranked list. Qdrant supports two.\n\n| Strategy | How it works | When to use it |\n|----------|--------------|----------------|\n| RRF (Reciprocal Rank Fusion) | Combines rankings only, ignores raw scores. Robust, hard to game. | Default. Safe when dense and sparse score scales differ. |\n| DBSF (Distribution-Based Score Fusion) | Normalizes score distributions before merging. Sensitive to relative score gaps. | When score gaps meaningfully encode relevance and both retrievers are well calibrated. |\n\nStart with unweighted RRF, because dense and sparse scores live on different scales and raw-score fusion without normalization is unreliable. RRF also accepts a `k` constant and per-prefetch `weights`, so you can favour the stronger retriever once you have an evaluation set to tune against. Move to DBSF or tuned weights only after measuring, and tune on a different split from the one you measure on.\n\nBoth strategies run below on the same query so you can compare.", + "metadata": { + "id": "f1e1CvI5qiGo" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "gkdmtsVrFlvb", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "40b837e1" + }, + "execution_count": null, + "source": "show(\"RRF fusion:\", hybrid(\"Nike Pegasus 40 size 10\", fusion=\"rrf\"))\nshow(\"DBSF fusion:\", hybrid(\"Nike Pegasus 40 size 10\", fusion=\"dbsf\"))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "RRF fusion:\n [1.0000] Nike Pegasus 40 running shoes (sku=NK-PEG40, in_stock=True)\n [0.6667] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.2500] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n [0.2000] iPhone 15 (sku=APL-IP15, in_stock=True)\n [0.1667] iPhone 14 (sku=APL-IP14, in_stock=True)\n\n" + }, + { + "output_type": "stream", + "name": "stdout", + "text": "DBSF fusion:\n [1.3562] Nike Pegasus 40 running shoes (sku=NK-PEG40, in_stock=True)\n [1.1153] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.6049] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n [0.4129] iPhone 15 (sku=APL-IP15, in_stock=True)\n [0.4019] iPhone 14 (sku=APL-IP14, in_stock=True)\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 6. Beyond Text: Multimodal Search\n\nThe same primitive, embed data then store as a vector then search by similarity, applies to any modality: images (CLIP, SigLIP), video frames, audio fingerprints, or text. Qdrant stores whatever vectors your embedding model produces, and the retrieval mechanics are identical.\n\n**Data to Embedding Model to Vector to Qdrant.** The modality changes, the system does not.\n\n### Named vectors\n\nWhen two representations must be searchable together, store them as named vectors on the same point, then query against whichever one you want. Below we demonstrate the mechanic with two text views of each product, a short `title` and a longer `description`, so it runs with no heavy image models. In a real multimodal system you would swap the `description` model for an image encoder such as CLIP; the collection and query code stay the same shape.", + "metadata": { + "id": "YAEZyFQ8vZRN" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "zvdieTpkf01P", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "4347274f" + }, + "execution_count": null, + "source": "client.create_collection(\n collection_name=\"catalog\",\n vectors_config={\n \"title\": models.VectorParams(size=384, distance=models.Distance.COSINE),\n \"description\": models.VectorParams(size=384, distance=models.Distance.COSINE),\n },\n)\n\nclient.upload_points(\n collection_name=\"catalog\",\n points=[\n models.PointStruct(\n id=42,\n vector={\n \"title\": models.Document(text=\"Red Nike running shoe\", model=DENSE_MODEL),\n \"description\": models.Document(text=\"Lightweight breathable trainer for road running in bright red\", model=DENSE_MODEL),\n },\n payload={\"sku\": \"NK-RED-10\", \"price\": 120},\n ),\n models.PointStruct(\n id=43,\n vector={\n \"title\": models.Document(text=\"Blue hiking boot\", model=DENSE_MODEL),\n \"description\": models.Document(text=\"Waterproof ankle support boot for rough mountain trails\", model=DENSE_MODEL),\n },\n payload={\"sku\": \"HK-BLU-9\", \"price\": 150},\n ),\n ],\n)\n\n# Query one named vector; the other is simply not searched on this call.\nres = client.query_points(\n \"catalog\",\n query=models.Document(text=\"shoe for running on pavement\", model=DENSE_MODEL),\n using=\"description\",\n limit=2,\n).points\nfor r in res:\n print(f\"[{r.score:.4f}] {r.payload['sku']}\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "[0.5492] HK-BLU-9\n[0.4119] NK-RED-10\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Each named vector is its own space, and vectors from different models are not comparable. Swapping `description` for a CLIP image encoder would look like this, using the same collection and query shape:\n\n```python\n# ingestion\n\"image\": embed_image(product_photo) # CLIP get_image_features\n\n# querying by text against those image vectors\nclient.query_points(\"catalog\", query=embed_text_with_clip(\"red running shoe\"),\n using=\"image\", limit=10)\n```\n\nThe important part is that the query text must go through **CLIP's text encoder**, not the sentence transformer, or it lands in a different space and the scores are meaningless. Module 5 builds this out properly.", + "metadata": { + "id": "8Hp7twaxDFmF" + } + }, + { + "cell_type": "markdown", + "source": "## 7. Filtering Works with Any Retrieval Method\n\nPayload filters are not a hybrid-only feature. The same conditions apply to dense-only, sparse-only, or hybrid retrieval, and they are evaluated as hard constraints *while* the search runs, not as a separate step afterward. Because out-of-scope points never take a slot in your top-K, results stay both relevant and valid: in stock, within permissions, within a date range.\n\nWhat changes between the three is *where* the filter goes, which is what the broken example earlier demonstrated. Dense-only and sparse-only have no prefetch, so `query_filter` is correct. Hybrid has prefetches, so the filter belongs in each one.\n\nThe three calls below apply the identical filter to all three retrieval methods.", + "metadata": { + "id": "aqfycbsoKGUO" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "S2yu9jSNcZ3M", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "bc288dcd" + }, + "execution_count": null, + "source": "show(\"DENSE + filter:\", dense_only(\"iPhone\", query_filter=in_stock_filter))\nshow(\"SPARSE + filter:\", sparse_only(\"iPhone\", query_filter=in_stock_filter))\nshow(\"HYBRID + filter:\", hybrid(\"iPhone\", query_filter=in_stock_filter))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "DENSE + filter:\n [0.8036] iPhone 15 (sku=APL-IP15, in_stock=True)\n [0.7914] iPhone 14 (sku=APL-IP14, in_stock=True)\n [0.5714] iPhone 15 Pro Max (sku=APL-IP15PM, in_stock=True)\n [0.1663] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.1396] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n\nSPARSE + filter:\n [1.1605] iPhone 15 (sku=APL-IP15, in_stock=True)\n [1.1605] iPhone 14 (sku=APL-IP14, in_stock=True)\n [1.1543] iPhone 15 Pro Max (sku=APL-IP15PM, in_stock=True)\n\nHYBRID + filter:\n [1.0000] iPhone 15 (sku=APL-IP15, in_stock=True)\n [0.6667] iPhone 14 (sku=APL-IP14, in_stock=True)\n [0.5000] iPhone 15 Pro Max (sku=APL-IP15PM, in_stock=True)\n [0.2000] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.1667] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## What's next: Module 4\n\n- The five layers of a vector search stack\n- A worked design: a multilingual news search system, decision by decision\n- Filtering in production: how the query planner picks a strategy, and multitenancy\n- The production RAG pipeline, and deployment options\n\n[Continue to Module 4](https://qdrant.tech/course/beginners/module-4/)", + "metadata": { + "id": "K3QQobiwgZIM" + } + } + ] +} diff --git a/Beginner-course/Module4.ipynb b/Beginner-course/Module4.ipynb new file mode 100644 index 0000000..e5b7560 --- /dev/null +++ b/Beginner-course/Module4.ipynb @@ -0,0 +1,1110 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "b4c1d4b2", + "metadata": {}, + "source": [ + "# Module 4: Designing a Vector Search System\n", + "\n", + "You know the parts from Modules 1 to 3: embeddings, collections, HNSW, hybrid retrieval, and filters. This module is about judgment, going from \"we have articles and analysts\" to a payload schema, a retrieval pipeline, and a deployment mode.\n", + "\n", + "## What You Will Do\n", + "\n", + "1. See the five layers of the stack as a diagnostic checklist.\n", + "2. Design and build a news search system by answering five questions.\n", + "3. Filter the way production systems do, and see what happens when you get it wrong.\n", + "4. Assemble a production RAG retrieval pipeline.\n", + "5. Review the deployment options and work through a knowledge check.\n", + "\n", + "**Running this in Colab:** run the cells top to bottom. The first two cells take a minute, because they download the embedding models." + ] + }, + { + "cell_type": "markdown", + "id": "8abb2140", + "metadata": {}, + "source": [ + "## Setup\n", + "\n", + "The notebook runs against Qdrant in **local mode**, an in-memory Qdrant that needs no server. It is the fastest way to follow along, and it has one limit worth knowing before you start: local mode is a Python reimplementation, not the engine. Payload indexes have no effect there, and search is exact rather than approximate. Every index call below is still the right habit, and Section 5 covers when to move off it.\n", + "\n", + "To run against a real cluster instead, create a free one at [cloud.qdrant.io](https://cloud.qdrant.io/), then swap in the commented lines below." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "fd85e0de", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-11T09:29:07.591536Z", + "iopub.status.busy": "2026-08-11T09:29:07.591306Z", + "iopub.status.idle": "2026-08-11T09:29:07.597029Z", + "shell.execute_reply": "2026-08-11T09:29:07.596084Z" + } + }, + "outputs": [], + "source": [ + "!pip install -q \"qdrant-client[fastembed]\"" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "fb19a50f", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-11T09:29:07.598654Z", + "iopub.status.busy": "2026-08-11T09:29:07.598507Z", + "iopub.status.idle": "2026-08-11T09:29:10.904642Z", + "shell.execute_reply": "2026-08-11T09:29:10.902935Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Dense vector dimension: 384\n" + ] + } + ], + "source": [ + "from qdrant_client import QdrantClient, models\n", + "from fastembed import TextEmbedding, SparseTextEmbedding\n", + "\n", + "# Local mode: in-memory Qdrant, no server needed.\n", + "client = QdrantClient(\":memory:\")\n", + "\n", + "# To use a real cluster instead, comment out the line above and uncomment these.\n", + "# Colab stores secrets under the key icon in the left sidebar.\n", + "#\n", + "# from google.colab import userdata\n", + "# client = QdrantClient(\n", + "# url=userdata.get(\"QDRANT_URL\"),\n", + "# api_key=userdata.get(\"QDRANT_API_KEY\"),\n", + "# )\n", + "\n", + "# A small, fast English model. Any embedding model works, this one keeps the\n", + "# Colab download short. It is also FastEmbed's default.\n", + "dense_model = TextEmbedding(\"BAAI/bge-small-en-v1.5\")\n", + "\n", + "# BM25 gives us exact-token matching alongside dense semantics.\n", + "sparse_model = SparseTextEmbedding(\"Qdrant/bm25\")\n", + "\n", + "# Ask the model for its dimensionality rather than hardcoding it, so the\n", + "# collection config can never drift out of sync with the model.\n", + "DENSE_DIM = len(list(dense_model.embed([\"hello\"]))[0])\n", + "print(\"Dense vector dimension:\", DENSE_DIM)" + ] + }, + { + "cell_type": "markdown", + "id": "29ae8d4a", + "metadata": {}, + "source": [ + "## 1. The Layers of the Stack\n", + "\n", + "Every vector search system is built from the same five layers. When something is slow, wrong, or expensive, the first question is always which layer the problem is in.\n", + "\n", + "| Layer | What lives here |\n", + "|---|---|\n", + "| Query | Embedding the query, dense vs. sparse vs. hybrid, fusion, limits |\n", + "| Indexing | The HNSW graph for vectors, payload indexes for filter fields |\n", + "| Storage | Vectors, payloads, and IDs on disk and in memory |\n", + "| Knowledge | The data itself: chunking, embedding model choice, payload schema |\n", + "| Distribution | Sharding, replication, multi-node clusters |\n", + "\n", + "Every decision below is tagged with the layer it belongs to." + ] + }, + { + "cell_type": "markdown", + "id": "c5c11f1e", + "metadata": {}, + "source": [ + "## 2. Worked Example: Designing a News Search System\n", + "\n", + "> Analysts at a research firm need to search global news that arrives continuously. They ask in plain language (\"port congestion in Southeast Asia\") and they scope every search by country, topic, date range, and source. Some queries name one specific thing, a company ticker or a ship name, that has to match exactly.\n", + "\n", + "Five questions turn that brief into a system.\n", + "\n", + "### Question 1: What Do the Queries Look Like?\n", + "\n", + "Both kinds. \"Port congestion in Southeast Asia\" is semantic intent, which is dense territory. \"MAERSK-B.CO\" is an exact token that carries no meaning a dense model can use.\n", + "\n", + "**Decision:** hybrid search, with a named dense vector and a named sparse vector on every point, fused at query time. *(Query layer.)*\n", + "\n", + "Naming the vectors is what lets a single point carry both and a single query use both. The `modifier=IDF` setting is required for correct BM25 scoring: sparse vectors store term frequency, and Qdrant applies the inverse document frequency half at query time." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "7d821206", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-11T09:29:10.907847Z", + "iopub.status.busy": "2026-08-11T09:29:10.906502Z", + "iopub.status.idle": "2026-08-11T09:29:10.913169Z", + "shell.execute_reply": "2026-08-11T09:29:10.912164Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Collection 'news' created.\n" + ] + } + ], + "source": [ + "client.create_collection(\n", + " collection_name=\"news\",\n", + " vectors_config={\n", + " \"dense\": models.VectorParams(size=DENSE_DIM, distance=models.Distance.COSINE),\n", + " },\n", + " sparse_vectors_config={\n", + " \"sparse\": models.SparseVectorParams(\n", + " # Required for correct BM25 scoring\n", + " modifier=models.Modifier.IDF\n", + " ),\n", + " },\n", + ")\n", + "print(\"Collection 'news' created.\")" + ] + }, + { + "cell_type": "markdown", + "id": "90208c76", + "metadata": {}, + "source": [ + "### Question 2: What Must the System Filter On?\n", + "\n", + "From the brief: country, topic, date range, and source. These are hard rules, not ranking signals. An analyst scoping to \"Vietnam, last seven days\" means exactly that. Hard rules go in the payload, and every field you filter on gets a payload index.\n", + "\n", + "The schema, decided now, before ingestion:\n", + "\n", + "```\n", + "payload:\n", + " country string (indexed)\n", + " topic string (indexed)\n", + " source string (indexed)\n", + " published_at datetime (indexed)\n", + " headline string (returned, never filtered)\n", + " lead string (returned, never filtered)\n", + " body string (returned, never filtered)\n", + "```\n", + "\n", + "That is a knowledge-layer decision (what to store) and an indexing-layer decision (what to index).\n", + "\n", + "The next cell prints a warning: `Payload indexes have no effect in the local Qdrant`. That is expected and it is a local-mode artifact, not a mistake. Creating the indexes before ingestion is the correct habit, and on a real cluster it is what makes filtered search fast." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "aae7fcb1", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-11T09:29:10.914754Z", + "iopub.status.busy": "2026-08-11T09:29:10.914578Z", + "iopub.status.idle": "2026-08-11T09:29:10.920518Z", + "shell.execute_reply": "2026-08-11T09:29:10.919497Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Payload indexes created for country, topic, source, published_at.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/tmp/ipykernel_576/4200797916.py:2: UserWarning: Payload indexes have no effect in the local Qdrant. Please use server Qdrant if you need payload indexes.\n", + " client.create_payload_index(\n" + ] + } + ], + "source": [ + "for field in [\"country\", \"topic\", \"source\"]:\n", + " client.create_payload_index(\n", + " collection_name=\"news\",\n", + " field_name=field,\n", + " field_schema=models.PayloadSchemaType.KEYWORD,\n", + " )\n", + "\n", + "client.create_payload_index(\n", + " collection_name=\"news\",\n", + " field_name=\"published_at\",\n", + " field_schema=models.PayloadSchemaType.DATETIME,\n", + ")\n", + "print(\"Payload indexes created for country, topic, source, published_at.\")" + ] + }, + { + "cell_type": "markdown", + "id": "b1a0a376", + "metadata": {}, + "source": [ + "### Question 3: What Is the Workload Shape?\n", + "\n", + "Millions of articles, text only, arriving continuously, and analysts expect this morning's news to be searchable this morning.\n", + "\n", + "**Decisions:** one collection, and continuous upserts rather than periodic rebuilds. *(Storage and knowledge layers.)*\n", + "\n", + "Here is a small sample standing in for that stream. Note the exact tokens hiding in the text: a ticker and a ship name." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "6ae895a9", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-11T09:29:10.922347Z", + "iopub.status.busy": "2026-08-11T09:29:10.922185Z", + "iopub.status.idle": "2026-08-11T09:29:10.931652Z", + "shell.execute_reply": "2026-08-11T09:29:10.930523Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "9 sample articles across 8 countries.\n" + ] + } + ], + "source": [ + "articles = [\n", + " {\n", + " \"headline\": \"Hai Phong container backlog grows for a third week\",\n", + " \"lead\": \"Congestion at the northern Vietnamese port deepened as container volumes climbed, delaying vessel departures.\",\n", + " \"body\": \"Terminal operators said yard utilization reached the high nineties, forcing arriving vessels to wait at anchor. Freight forwarders reported delays of two to four days on outbound bookings.\",\n", + " \"country\": \"VN\", \"topic\": \"shipping\", \"source\": \"reuters\",\n", + " \"published_at\": \"2026-07-20T08:00:00Z\", \"tenant_id\": \"asia-desk\",\n", + " },\n", + " {\n", + " \"headline\": \"Southeast Asian port congestion slows regional transport\",\n", + " \"lead\": \"Delays at major Southeast Asian ports are pushing back container ship arrivals across the region.\",\n", + " \"body\": \"Analysts point to a combination of higher volumes and reduced berth availability. Shipping lines have added waiting time to their schedules.\",\n", + " \"country\": \"JP\", \"topic\": \"shipping\", \"source\": \"nikkei\",\n", + " \"published_at\": \"2026-07-21T09:30:00Z\", \"tenant_id\": \"asia-desk\",\n", + " },\n", + " {\n", + " \"headline\": \"Shanghai port sets a monthly logistics throughput record\",\n", + " \"lead\": \"The Port of Shanghai handled record volumes this month as operators expanded yard capacity.\",\n", + " \"body\": \"Terminal managers credited longer gate hours and a new rail link for the improvement. Throughput growth has outpaced the national average for six consecutive quarters.\",\n", + " \"country\": \"CN\", \"topic\": \"logistics\", \"source\": \"caixin\",\n", + " \"published_at\": \"2026-07-19T11:00:00Z\", \"tenant_id\": \"asia-desk\",\n", + " },\n", + " {\n", + " \"headline\": \"Shipping group unit faces delisting speculation\",\n", + " \"lead\": \"Shares tied to MAERSK-B.CO swung on speculation that a subsidiary vehicle could be delisted.\",\n", + " \"body\": \"Traders cited an unusual volume spike in the final hour of trading. The company declined to comment.\",\n", + " \"country\": \"DK\", \"topic\": \"markets\", \"source\": \"reuters\",\n", + " \"published_at\": \"2026-07-22T07:15:00Z\", \"tenant_id\": \"europe-desk\",\n", + " },\n", + " {\n", + " \"headline\": \"Carrier unit faces delisting speculation\",\n", + " \"lead\": \"Shares tied to HLAG-D.DE swung on speculation that a subsidiary vehicle could be delisted.\",\n", + " \"body\": \"Traders cited unusual volume late in the session. The company declined to comment.\",\n", + " \"country\": \"DE\", \"topic\": \"markets\", \"source\": \"handelsblatt\",\n", + " \"published_at\": \"2026-07-22T08:05:00Z\", \"tenant_id\": \"europe-desk\",\n", + " },\n", + " {\n", + " \"headline\": \"Ever Given rerouted through Singapore to avoid delays\",\n", + " \"lead\": \"The container ship Ever Given was rerouted through the Port of Singapore this week.\",\n", + " \"body\": \"The diversion adds roughly two days to the voyage but avoids a longer wait at anchor. Singapore has absorbed a growing share of regional transhipment traffic.\",\n", + " \"country\": \"SG\", \"topic\": \"shipping\", \"source\": \"straits-times\",\n", + " \"published_at\": \"2026-07-22T10:45:00Z\", \"tenant_id\": \"asia-desk\",\n", + " },\n", + " {\n", + " \"headline\": \"Logistics operator reports steady quarterly earnings\",\n", + " \"lead\": \"A North American logistics operator announced quarterly earnings, citing steady freight demand.\",\n", + " \"body\": \"Revenue was broadly flat against the same quarter last year. The company reiterated its full year guidance.\",\n", + " \"country\": \"US\", \"topic\": \"markets\", \"source\": \"press-release-wire\",\n", + " \"published_at\": \"2026-06-30T12:00:00Z\", \"tenant_id\": \"americas-desk\",\n", + " },\n", + " {\n", + " \"headline\": \"Laem Chabang congestion rises on higher container volumes\",\n", + " \"lead\": \"Thailand's largest port is handling more containers than its yards were sized for, lengthening waits.\",\n", + " \"body\": \"Operators have extended gate hours and added weekend shifts. Exporters report longer lead times on shipments to Europe.\",\n", + " \"country\": \"TH\", \"topic\": \"shipping\", \"source\": \"bangkok-post\",\n", + " \"published_at\": \"2026-07-18T06:20:00Z\", \"tenant_id\": \"asia-desk\",\n", + " },\n", + " {\n", + " \"headline\": \"Hamburg reports handling delays as arrivals climb\",\n", + " \"lead\": \"The Port of Hamburg is reporting delays in cargo handling as the number of arriving ships rises.\",\n", + " \"body\": \"Dock labor shortages have compounded the problem. Rail connections inland are running close to capacity.\",\n", + " \"country\": \"DE\", \"topic\": \"logistics\", \"source\": \"handelsblatt\",\n", + " \"published_at\": \"2026-07-15T14:10:00Z\", \"tenant_id\": \"europe-desk\",\n", + " },\n", + "]\n", + "print(len(articles), \"sample articles across\", len(set(a[\"country\"] for a in articles)), \"countries.\")" + ] + }, + { + "cell_type": "markdown", + "id": "53221a3f", + "metadata": {}, + "source": [ + "One knowledge-layer decision hides in the ingestion step: **what you embed matters as much as how you search it.**\n", + "\n", + "A news article runs 800 words, and the ticker from Question 1 is one token inside it. Embed the whole body and that token is averaged into a vector about shipping in general. Embed the headline and the lead, keep the full text in the payload, and the dense vector stays about one story while the sparse vector still sees every token.\n", + "\n", + "Two of these articles are deliberately hard: they have near-identical headlines and leads, and differ only in the ticker. Watch what that does to the dense retriever later." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "694b2102", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-11T09:29:10.933049Z", + "iopub.status.busy": "2026-08-11T09:29:10.932833Z", + "iopub.status.idle": "2026-08-11T09:29:11.122256Z", + "shell.execute_reply": "2026-08-11T09:29:11.121047Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Upserted" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + " 9 points. Collection count: 9\n" + ] + } + ], + "source": [ + "def to_sparse_vector(text, is_query=False):\n", + " \"\"\"Convert text to a Qdrant SparseVector using BM25.\n", + " BM25 scores queries and documents differently, so queries use query_embed.\"\"\"\n", + " emb = next(sparse_model.query_embed(text)) if is_query else next(sparse_model.embed([text]))\n", + " return models.SparseVector(indices=emb.indices.tolist(), values=emb.values.tolist())\n", + "\n", + "points = []\n", + "for i, art in enumerate(articles):\n", + " # The knowledge-layer decision, made concrete: embed headline and lead only.\n", + " embed_text = art[\"headline\"] + \". \" + art[\"lead\"]\n", + " points.append(\n", + " models.PointStruct(\n", + " id=i,\n", + " vector={\n", + " \"dense\": next(dense_model.embed([embed_text])).tolist(),\n", + " \"sparse\": to_sparse_vector(embed_text),\n", + " },\n", + " payload=art,\n", + " )\n", + " )\n", + "\n", + "client.upsert(collection_name=\"news\", points=points)\n", + "print(\"Upserted\", len(points), \"points. Collection count:\", client.count(\"news\").count)" + ] + }, + { + "cell_type": "markdown", + "id": "c90fd7ac", + "metadata": {}, + "source": [ + "### Question 4: What Does the Retrieval Pipeline Look Like?\n", + "\n", + "Start with the simplest pipeline that fits the query analysis: hybrid, from Question 1, plus filters, from Question 2, fused with Reciprocal Rank Fusion. No reranker yet.\n", + "\n", + "One `query_points` call does all of it:\n", + "\n", + "- Two `Prefetch` branches, one per named vector, each pulling 50 candidates.\n", + "- `RrfQuery`, which merges the two candidate lists by rank rather than by score, so it does not matter that cosine similarity and BM25 live on different scales.\n", + "- The filter **inside each `Prefetch`**, so both retrievers search only the valid subset. Where that filter goes matters more than it looks, and the next section shows what happens when it goes somewhere else." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "6eb185bc", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-11T09:29:11.124574Z", + "iopub.status.busy": "2026-08-11T09:29:11.123873Z", + "iopub.status.idle": "2026-08-11T09:29:11.145783Z", + "shell.execute_reply": "2026-08-11T09:29:11.144755Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Query: 'port congestion in Southeast Asia'\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "[1.0000] JP shipping nikkei | Southeast Asian port congestion slows regional transport\n", + "[0.5833] TH shipping bangkok-post | Laem Chabang congestion rises on higher container volumes\n", + "[0.5000] SG shipping straits-times | Ever Given rerouted through Singapore to avoid delays\n", + "[0.4500] VN shipping reuters | Hai Phong container backlog grows for a third week\n", + "[0.3667] CN logistics caixin | Shanghai port sets a monthly logistics throughput record\n" + ] + } + ], + "source": [ + "def search(query_text, query_filter=None, limit=5):\n", + " \"\"\"Hybrid search: dense + sparse prefetch, fused with RRF.\n", + " The filter goes inside each Prefetch, not on the outer query.\"\"\"\n", + " dense_q = next(dense_model.query_embed(query_text)).tolist()\n", + " sparse_q = to_sparse_vector(query_text, is_query=True)\n", + " response = client.query_points(\n", + " collection_name=\"news\",\n", + " prefetch=[\n", + " models.Prefetch(query=dense_q, using=\"dense\", filter=query_filter, limit=50),\n", + " models.Prefetch(query=sparse_q, using=\"sparse\", filter=query_filter, limit=50),\n", + " ],\n", + " query=models.RrfQuery(rrf=models.Rrf()),\n", + " limit=limit,\n", + " )\n", + " return response.points\n", + "\n", + "\n", + "def show(results):\n", + " for r in results:\n", + " p = r.payload\n", + " print(f\"[{r.score:.4f}] {p['country']} {p['topic']:<9} {p['source']:<18} | {p['headline']}\")\n", + "\n", + "\n", + "print(\"Query: 'port congestion in Southeast Asia'\\n\")\n", + "show(search(\"port congestion in Southeast Asia\"))" + ] + }, + { + "cell_type": "markdown", + "id": "d46f0862", + "metadata": {}, + "source": [ + "### Try It: Why Hybrid, Not Dense Alone\n", + "\n", + "Now the exact-token case, and the reason Question 1 chose hybrid.\n", + "\n", + "Run the same query three ways: dense only, sparse only, and hybrid. Two articles in the collection have near-identical headlines and leads, differing only in the ticker. Look at the **gaps between the scores**, not just the order." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "2ea22801", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-11T09:29:11.147923Z", + "iopub.status.busy": "2026-08-11T09:29:11.147060Z", + "iopub.status.idle": "2026-08-11T09:29:11.180905Z", + "shell.execute_reply": "2026-08-11T09:29:11.180044Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "DENSE ONLY\n", + " [0.8120] Shares tied to MAERSK-B.CO swung on speculation that a subsidi\n", + " [0.6956] Shares tied to HLAG-D.DE swung on speculation that a subsidiar\n", + " [0.6071] The container ship Ever Given was rerouted through the Port of\n", + "\n", + "SPARSE ONLY (BM25)\n", + " [11.7931] Shares tied to MAERSK-B.CO swung on speculation that a subsidi\n", + " [2.5926] Shares tied to HLAG-D.DE swung on speculation that a subsidiar\n", + "\n", + "HYBRID (RRF over both)\n", + "[1.0000] DK markets reuters | Shipping group unit faces delisting speculation\n", + "[0.6667] DE markets handelsblatt | Carrier unit faces delisting speculation\n", + "[0.2500] SG shipping straits-times | Ever Given rerouted through Singapore to avoid delays\n" + ] + } + ], + "source": [ + "QUERY = \"MAERSK-B.CO delisting\"\n", + "dense_q = next(dense_model.query_embed(QUERY)).tolist()\n", + "sparse_q = to_sparse_vector(QUERY, is_query=True)\n", + "\n", + "print(\"DENSE ONLY\")\n", + "for p in client.query_points(\"news\", query=dense_q, using=\"dense\", limit=3).points:\n", + " print(f\" [{p.score:.4f}] {p.payload['lead'][:62]}\")\n", + "\n", + "print(\"\\nSPARSE ONLY (BM25)\")\n", + "for p in client.query_points(\"news\", query=sparse_q, using=\"sparse\", limit=3).points:\n", + " print(f\" [{p.score:.4f}] {p.payload['lead'][:62]}\")\n", + "\n", + "print(\"\\nHYBRID (RRF over both)\")\n", + "show(search(QUERY, limit=3))" + ] + }, + { + "cell_type": "markdown", + "id": "1944b15b", + "metadata": {}, + "source": [ + "Dense puts the right article first, but only just. The correct article scores 0.8120 and the near-identical decoy scores 0.6956, a gap of 0.12, because a ticker is a string with no semantics for a dense model to work with. BM25 scores them 11.79 and 2.59, separating them by more than 4x, because it is matching the literal token.\n", + "\n", + "On nine articles that thin dense margin still lands the right answer. On nine million, with thousands of similar market stories, it is noise. That gap is the argument for hybrid, and it is why Question 1 is the most consequential question in the list." + ] + }, + { + "cell_type": "markdown", + "id": "2b8713e6", + "metadata": {}, + "source": [ + "### Question 5: What Are the Deployment Constraints?\n", + "\n", + "A research firm with a small engineering team, no data residency restrictions, and a \"please do not page us at night\" budget points to a managed deployment. The design and the deployment mode are independent decisions: everything above runs unchanged against this in-memory client, a Docker container, or Qdrant Cloud. Section 5 lays out the full option space. *(Distribution layer.)*\n", + "\n", + "### The Design on One Page\n", + "\n", + "| Question | Answer for this system | Layer |\n", + "|---|---|---|\n", + "| Query type | Mixed semantic and exact, so hybrid with Reciprocal Rank Fusion | Query |\n", + "| Filter scope | country, topic, source, date, indexed before ingestion | Knowledge, indexing |\n", + "| Workload shape | Millions of text chunks, continuous ingestion | Storage, knowledge |\n", + "| Pipeline | Hybrid with per-prefetch filters, headline and lead embedded, no reranker yet | Query, knowledge |\n", + "| Deployment | Managed, and independent of the design | Distribution |" + ] + }, + { + "cell_type": "markdown", + "id": "b2b37ddd", + "metadata": {}, + "source": [ + "## 3. Filtering\n", + "\n", + "Filtering is the feature that decides whether your results are correct, so it deserves more than a passing mention.\n", + "\n", + "The naive approach is post-filtering: retrieve the top K by similarity, then throw away whatever fails the filter. With a selective filter, one country out of 200, the top K can contain zero valid results, and there is no K that guarantees correctness. Qdrant does not work that way. A query planner picks a strategy per segment based on how many points it estimates the filter will match, and on which payload indexes exist.\n", + "\n", + "### The Filter Toolbox\n", + "\n", + "| Condition | Logic | Example |\n", + "|---|---|---|\n", + "| must | AND, all conditions true | country = VN AND topic = shipping |\n", + "| should | OR, at least one true | topic = shipping OR topic = logistics |\n", + "| must_not | Exclude matches | Exclude source = press-release-wire |\n", + "| Range | Numeric or datetime bounds | published_at within the last seven days |\n", + "| MatchAny | Value in a set | source in [reuters, nikkei, caixin] |\n", + "| Geo | Radius, bounding box, or polygon | Events within 100 km of a port |\n", + "\n", + "The cells below run several of these against the news collection." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "d7df4beb", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-11T09:29:11.182656Z", + "iopub.status.busy": "2026-08-11T09:29:11.182496Z", + "iopub.status.idle": "2026-08-11T09:29:11.198817Z", + "shell.execute_reply": "2026-08-11T09:29:11.197542Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "must (country=VN AND topic=shipping):\n", + "[1.0000] VN shipping reuters | Hai Phong container backlog grows for a third week\n" + ] + } + ], + "source": [ + "# must: AND. Only Vietnamese shipping news.\n", + "f_must = models.Filter(\n", + " must=[\n", + " models.FieldCondition(key=\"country\", match=models.MatchValue(value=\"VN\")),\n", + " models.FieldCondition(key=\"topic\", match=models.MatchValue(value=\"shipping\")),\n", + " ]\n", + ")\n", + "print(\"must (country=VN AND topic=shipping):\")\n", + "show(search(\"port congestion\", query_filter=f_must))" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "0f11523f", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-11T09:29:11.201223Z", + "iopub.status.busy": "2026-08-11T09:29:11.201034Z", + "iopub.status.idle": "2026-08-11T09:29:11.220026Z", + "shell.execute_reply": "2026-08-11T09:29:11.218976Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "should (topic=shipping OR topic=logistics):\n", + "[1.0000] JP shipping nikkei | Southeast Asian port congestion slows regional transport\n", + "[0.5833] TH shipping bangkok-post | Laem Chabang congestion rises on higher container volumes\n", + "[0.4762] DE logistics handelsblatt | Hamburg reports handling delays as arrivals climb\n", + "[0.4167] VN shipping reuters | Hai Phong container backlog grows for a third week\n", + "[0.4000] CN logistics caixin | Shanghai port sets a monthly logistics throughput record\n" + ] + } + ], + "source": [ + "# should: OR. Either shipping or logistics.\n", + "f_should = models.Filter(\n", + " should=[\n", + " models.FieldCondition(key=\"topic\", match=models.MatchValue(value=\"shipping\")),\n", + " models.FieldCondition(key=\"topic\", match=models.MatchValue(value=\"logistics\")),\n", + " ]\n", + ")\n", + "print(\"should (topic=shipping OR topic=logistics):\")\n", + "show(search(\"congestion at ports\", query_filter=f_should))" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "62f84634", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-11T09:29:11.222218Z", + "iopub.status.busy": "2026-08-11T09:29:11.221564Z", + "iopub.status.idle": "2026-08-11T09:29:11.242740Z", + "shell.execute_reply": "2026-08-11T09:29:11.241811Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "published since July 19, from a wire we trust, no press releases:\n", + "[1.0000] JP shipping nikkei | Southeast Asian port congestion slows regional transport\n", + "[0.5833] VN shipping reuters | Hai Phong container backlog grows for a third week\n", + "[0.5333] SG shipping straits-times | Ever Given rerouted through Singapore to avoid delays\n", + "[0.4500] CN logistics caixin | Shanghai port sets a monthly logistics throughput record\n", + "[0.1667] DK markets reuters | Shipping group unit faces delisting speculation\n" + ] + } + ], + "source": [ + "# must_not + Range + MatchAny, all evaluated together during the search.\n", + "f_combined = models.Filter(\n", + " must=[\n", + " models.FieldCondition(\n", + " key=\"published_at\",\n", + " range=models.DatetimeRange(gte=\"2026-07-19T00:00:00Z\"),\n", + " ),\n", + " models.FieldCondition(\n", + " key=\"source\",\n", + " match=models.MatchAny(any=[\"reuters\", \"nikkei\", \"caixin\", \"straits-times\"]),\n", + " ),\n", + " ],\n", + " must_not=[\n", + " models.FieldCondition(key=\"source\", match=models.MatchValue(value=\"press-release-wire\")),\n", + " ],\n", + ")\n", + "print(\"published since July 19, from a wire we trust, no press releases:\")\n", + "show(search(\"port congestion in Southeast Asia\", query_filter=f_combined))" + ] + }, + { + "cell_type": "markdown", + "id": "456ca1df", + "metadata": {}, + "source": [ + "### Common Mistake: Filters in the Wrong Place\n", + "\n", + "Every search above passed its filter **inside each `Prefetch`**. There is a second place the client will happily accept one: `query_filter`, on the outer query, next to `prefetch`.\n", + "\n", + "Put the filter inside every `Prefetch`. It narrows what each retriever searches, so both come back with 50 candidates that already satisfy the constraint, and it behaves the same way on every deployment mode.\n", + "\n", + "Local mode ignores an outer filter and raises no error, so a notebook can return results that violate its own filter while looking perfectly healthy. The cell below runs the same VN-and-shipping filter both ways so you can see it." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "b1ec6c98", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-11T09:29:11.244373Z", + "iopub.status.busy": "2026-08-11T09:29:11.244198Z", + "iopub.status.idle": "2026-08-11T09:29:11.273435Z", + "shell.execute_reply": "2026-08-11T09:29:11.272468Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "FILTER ON THE OUTER QUERY, in local mode:\n", + "[1.0000] JP shipping nikkei | Southeast Asian port congestion slows regional transport\n", + "[0.5833] TH shipping bangkok-post | Laem Chabang congestion rises on higher container volumes\n", + "[0.4762] DE logistics handelsblatt | Hamburg reports handling delays as arrivals climb\n", + "[0.4167] VN shipping reuters | Hai Phong container backlog grows for a third week\n", + "[0.4000] CN logistics caixin | Shanghai port sets a monthly logistics throughput record\n", + " -> 5 results, 4 of them break the filter\n", + "\n", + "FILTER INSIDE EACH PREFETCH:\n", + "[1.0000] VN shipping reuters | Hai Phong container backlog grows for a third week\n", + " -> 1 results, 0 of them break the filter\n" + ] + } + ], + "source": [ + "dense_q = next(dense_model.query_embed(\"port congestion\")).tolist()\n", + "sparse_q = to_sparse_vector(\"port congestion\", is_query=True)\n", + "\n", + "# The wrong place: query_filter on the outer query, alongside prefetch.\n", + "wrong = client.query_points(\n", + " collection_name=\"news\",\n", + " prefetch=[\n", + " models.Prefetch(query=dense_q, using=\"dense\", limit=50),\n", + " models.Prefetch(query=sparse_q, using=\"sparse\", limit=50),\n", + " ],\n", + " query=models.RrfQuery(rrf=models.Rrf()),\n", + " query_filter=f_must,\n", + " limit=5,\n", + ").points\n", + "\n", + "print(\"FILTER ON THE OUTER QUERY, in local mode:\")\n", + "show(wrong)\n", + "violations = [p for p in wrong if p.payload[\"country\"] != \"VN\" or p.payload[\"topic\"] != \"shipping\"]\n", + "print(f\" -> {len(wrong)} results, {len(violations)} of them break the filter\\n\")\n", + "\n", + "print(\"FILTER INSIDE EACH PREFETCH:\")\n", + "right = search(\"port congestion\", query_filter=f_must)\n", + "show(right)\n", + "violations = [p for p in right if p.payload[\"country\"] != \"VN\" or p.payload[\"topic\"] != \"shipping\"]\n", + "print(f\" -> {len(right)} results, {len(violations)} of them break the filter\")" + ] + }, + { + "cell_type": "markdown", + "id": "a13364fd", + "metadata": {}, + "source": [ + "### A Special Case: Scoping by User or Tenant\n", + "\n", + "In almost any multi-user product you have to scope every query to one customer's data. The instinct is a collection per customer, which becomes millions of collections and is unmanageable. The pattern instead:\n", + "\n", + "1. Add a `tenant_id` payload field to every point at ingestion. Already done above.\n", + "2. Create a payload index on it with `is_tenant=True`, which tells Qdrant this field identifies tenants so it can keep each tenant's data together on disk. Supported for the `keyword` and `uuid` index types.\n", + "3. Filter on it at every query. Never omit it.\n", + "\n", + "Steps 1 and 3 alone are already correct. Step 2 is what keeps them fast as the tenant count grows." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "16ee601b", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-11T09:29:11.274835Z", + "iopub.status.busy": "2026-08-11T09:29:11.274675Z", + "iopub.status.idle": "2026-08-11T09:29:11.300161Z", + "shell.execute_reply": "2026-08-11T09:29:11.299165Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "europe-desk view of 'port delays':\n", + "[1.0000] DE logistics handelsblatt | Hamburg reports handling delays as arrivals climb\n", + "[0.3333] DK markets reuters | Shipping group unit faces delisting speculation\n", + "[0.2500] DE markets handelsblatt | Carrier unit faces delisting speculation\n", + "\n", + "asia-desk view of 'port delays':\n", + "[1.0000] JP shipping nikkei | Southeast Asian port congestion slows regional transport\n", + "[0.5833] VN shipping reuters | Hai Phong container backlog grows for a third week\n", + "[0.5833] SG shipping straits-times | Ever Given rerouted through Singapore to avoid delays\n", + "[0.3667] TH shipping bangkok-post | Laem Chabang congestion rises on higher container volumes\n", + "[0.3667] CN logistics caixin | Shanghai port sets a monthly logistics throughput record\n" + ] + } + ], + "source": [ + "client.create_payload_index(\n", + " collection_name=\"news\",\n", + " field_name=\"tenant_id\",\n", + " field_schema=models.KeywordIndexParams(\n", + " type=models.KeywordIndexType.KEYWORD,\n", + " is_tenant=True,\n", + " ),\n", + ")\n", + "\n", + "\n", + "def tenant_search(query_text, tenant_id, limit=5):\n", + " return search(\n", + " query_text,\n", + " query_filter=models.Filter(\n", + " must=[models.FieldCondition(key=\"tenant_id\", match=models.MatchValue(value=tenant_id))]\n", + " ),\n", + " limit=limit,\n", + " )\n", + "\n", + "\n", + "print(\"europe-desk view of 'port delays':\")\n", + "show(tenant_search(\"port delays\", tenant_id=\"europe-desk\"))\n", + "print(\"\\nasia-desk view of 'port delays':\")\n", + "show(tenant_search(\"port delays\", tenant_id=\"asia-desk\"))" + ] + }, + { + "cell_type": "markdown", + "id": "6e15d660", + "metadata": {}, + "source": [ + "Two desks, one collection, and neither sees the other's articles.\n", + "\n", + "### Key Insight\n", + "\n", + "Design the payload schema before you ingest, driven by one question: what will I need to filter on? Time, geography, identity, permissions, and status flags are the usual suspects. Adding a payload field later is easy. Discovering at query time that you never stored the publication date means re-ingesting everything." + ] + }, + { + "cell_type": "markdown", + "id": "6f6064aa", + "metadata": {}, + "source": [ + "## 4. The Production RAG Pipeline\n", + "\n", + "Retrieval-Augmented Generation (RAG) retrieves relevant passages from a vector search engine and hands them to a large language model as context, so the model answers from your data instead of only from what it memorized during training.\n", + "\n", + "The production shape, using everything above:\n", + "\n", + "1. **Query understanding:** pull hard constraints (dates, country, topic) into a filter. Embed the query as a dense vector and a sparse vector.\n", + "2. **Hybrid retrieval:** dense and sparse prefetch with the filter on each, fused with Reciprocal Rank Fusion. One `query_points` call.\n", + "3. **Optional reranking:** a cross-encoder, a model that scores a query and a passage together rather than separately, reorders the top candidates. Add it only when evaluation shows fused results need refinement.\n", + "4. **Generation:** the top passages go in as context and the model writes the answer.\n", + "\n", + "### Rule of Thumb\n", + "\n", + "When RAG quality disappoints, improve step 2 before reaching for a bigger model in step 4. Retrieval quality caps answer quality: the model cannot cite a passage it never received.\n", + "\n", + "The cell below runs steps 1, 2, and the prompt assembly for step 4. It needs no API key." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "7d587768", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-11T09:29:11.301808Z", + "iopub.status.busy": "2026-08-11T09:29:11.301613Z", + "iopub.status.idle": "2026-08-11T09:29:11.319486Z", + "shell.execute_reply": "2026-08-11T09:29:11.318530Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Answer the question using only the sources below. Cite the source name.\n", + "\n", + "Sources:\n", + "- (nikkei, JP, 2026-07-21) Southeast Asian port congestion slows regional transport. Delays at major Southeast Asian ports are pushing back container ship arrivals across the region.\n", + "- (reuters, VN, 2026-07-20) Hai Phong container backlog grows for a third week. Congestion at the northern Vietnamese port deepened as container volumes climbed, delaying vessel departures.\n", + "- (bangkok-post, TH, 2026-07-18) Laem Chabang congestion rises on higher container volumes. Thailand's largest port is handling more containers than its yards were sized for, lengthening waits.\n", + "\n", + "Question: What is happening with port congestion in Southeast Asia?\n", + "Answer:\n" + ] + } + ], + "source": [ + "def rag_context(question, query_filter=None, k=3):\n", + " \"\"\"Steps 1 and 2: retrieve the top-k passages for a question.\"\"\"\n", + " hits = search(question, query_filter=query_filter, limit=k)\n", + " passages = [\n", + " f\"- ({h.payload['source']}, {h.payload['country']}, {h.payload['published_at'][:10]}) \"\n", + " f\"{h.payload['headline']}. {h.payload['lead']}\"\n", + " for h in hits\n", + " ]\n", + " return \"\\n\".join(passages), hits\n", + "\n", + "\n", + "def build_prompt(question, query_filter=None, k=3):\n", + " context, hits = rag_context(question, query_filter=query_filter, k=k)\n", + " prompt = (\n", + " \"Answer the question using only the sources below. Cite the source name.\\n\\n\"\n", + " f\"Sources:\\n{context}\\n\\n\"\n", + " f\"Question: {question}\\nAnswer:\"\n", + " )\n", + " return prompt, hits\n", + "\n", + "\n", + "prompt, hits = build_prompt(\"What is happening with port congestion in Southeast Asia?\")\n", + "print(prompt)" + ] + }, + { + "cell_type": "markdown", + "id": "cc9a9eee", + "metadata": {}, + "source": [ + "Step 4 is the only part that needs an API key, so it is optional. To run it, add your key to the Colab secrets panel (the key icon in the left sidebar) under the name `LLM_API_KEY`, then uncomment the block." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "ed2582b6", + "metadata": { + "execution": { + "iopub.execute_input": "2026-08-11T09:29:11.321704Z", + "iopub.status.busy": "2026-08-11T09:29:11.321554Z", + "iopub.status.idle": "2026-08-11T09:29:11.327215Z", + "shell.execute_reply": "2026-08-11T09:29:11.325922Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Prompt is ready. Uncomment the block above and add a key to generate an answer.\n" + ] + } + ], + "source": [ + "# Optional step 4: hand the prompt to a model. This cell runs without a key.\n", + "#\n", + "# from google.colab import userdata\n", + "# from anthropic import Anthropic\n", + "#\n", + "# llm = Anthropic(api_key=userdata.get(\"LLM_API_KEY\"))\n", + "# message = llm.messages.create(\n", + "# model=\"claude-sonnet-4-5\",\n", + "# max_tokens=300,\n", + "# messages=[{\"role\": \"user\", \"content\": prompt}],\n", + "# )\n", + "# print(message.content[0].text)\n", + "\n", + "print(\"Prompt is ready. Uncomment the block above and add a key to generate an answer.\")" + ] + }, + { + "cell_type": "markdown", + "id": "fe6e49a4", + "metadata": {}, + "source": [ + "## 5. Deployment Options\n", + "\n", + "The design above runs unchanged on any of these. Which one is right depends on your constraints, not on sophistication.\n", + "\n", + "| Deployment Mode | Use When | Avoid When |\n", + "|---|---|---|\n", + "| Local Mode | Prototyping, notebooks, CI tests, teaching | Production or benchmarking |\n", + "| Docker, self-hosted | Full infrastructure control, air-gapped or regulated environments | You do not yet have monitoring and backups |\n", + "| Managed Cloud | Small ops team, standard requirements | Data cannot leave your infrastructure |\n", + "| Hybrid Cloud | Data residency or security policy requires your own infrastructure | Managed cloud would do, with less operational overhead |\n", + "| Private Cloud or on-prem | Strictest requirements: defense, healthcare, finance | A lighter mode meets your needs |\n", + "| Edge | On-device search, offline, very low latency | You need distributed search |\n", + "\n", + "One caveat about local mode, since it is what this notebook used. It is a Python reimplementation rather than the engine: search is exact rather than approximate, payload indexes have no effect, and a filter on the outer query is ignored. Everything in Section 3 is still worth understanding while working locally, but verify indexing and filtering against a real server before trusting numbers from a notebook." + ] + }, + { + "cell_type": "markdown", + "id": "170652a1", + "metadata": {}, + "source": [ + "## 6. Knowledge Check\n", + "\n", + "Work through these before starting the capstone in Module 5.\n", + "\n", + "**1. Name the five layers, and place \"chunk articles instead of embedding them whole\" in the right one.**\n", + "Query, indexing, storage, knowledge, distribution. Chunking is a knowledge-layer decision, and mistakes there cannot be fixed by tuning any other layer.\n", + "\n", + "**2. Why hybrid retrieval from the start, rather than dense only?**\n", + "Query analysis showed a mix of semantic intent and exact tokens. A ticker carries no semantics for a dense model, so the dense margin between the right article and a near-identical one is thin. The sparse half separates them decisively.\n", + "\n", + "**3. What is wrong with post-filtering, and how does Qdrant avoid it?**\n", + "Post-filtering retrieves the top K first and discards invalid results, so a selective filter can leave zero valid points. Qdrant's query planner instead picks a strategy per segment from the estimated filter cardinality.\n", + "\n", + "**4. In a hybrid query, where does the filter belong?**\n", + "Inside each `Prefetch`. It narrows what each retriever searches, and it behaves the same way on every deployment mode. Local mode ignores an outer `query_filter` without raising an error, which is how a notebook ends up printing results that break its own filter.\n", + "\n", + "**5. Why create payload indexes before ingestion rather than after?**\n", + "The filterable HNSW graph gains filter-aware edges only for indexes that exist when the graph is built. An index created later still filters and still feeds the planner's cardinality estimate, but getting those extra edges means rebuilding the HNSW index.\n", + "\n", + "**6. Why does per-tenant scoping use a payload filter instead of one collection per tenant?**\n", + "Collections do not scale to millions of tenants operationally. An indexed `tenant_id` field, filtered on at every query, gives isolation in one collection. `is_tenant=True` is what keeps it fast as the tenant count grows.\n", + "\n", + "**7. In the RAG pipeline, which step do you improve first when answers disappoint?**\n", + "Step 2, retrieval. The model cannot cite a passage it never received." + ] + }, + { + "cell_type": "markdown", + "id": "8607c08e", + "metadata": {}, + "source": [ + "## What Is Next: Module 5\n", + "\n", + "The capstone extends this system. Same five questions, bigger answers:\n", + "\n", + "- Ingest daily news, audio, and satellite imagery about suppliers, so three modalities instead of one\n", + "- Embed each modality into named vectors on shared points\n", + "- Cluster signals into risk themes across suppliers\n", + "\n", + "Before you start it, create a free cluster at [cloud.qdrant.io](https://cloud.qdrant.io/) so the capstone runs against a real server rather than local mode." + ] + } + ], + "metadata": { + "colab": { + "provenance": [], + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/Beginner-course/Module5.ipynb b/Beginner-course/Module5.ipynb new file mode 100644 index 0000000..1c76286 --- /dev/null +++ b/Beginner-course/Module5.ipynb @@ -0,0 +1,358 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "cells": [ + { + "cell_type": "markdown", + "source": "# Module 5: Capstone, Multimodal Supplier Risk Intelligence\n\nA factory in Vietnam catches fire on a Tuesday. By that afternoon it is in a Japanese trade publication and on a Chinese forum. By Thursday it reaches the English business press. By Friday your procurement team finds out.\n\nThis notebook builds the system that finds Tuesday.\n\n## What you will do\n\n1. Create one collection with three named vectors: dense text, sparse text, and CLIP image.\n2. Ingest multilingual news and satellite tiles onto shared points.\n3. Query in English and retrieve Japanese and Chinese sources, with no translation.\n4. Query images with text, through CLIP's shared space.\n5. Cluster signals into risk themes and use a cluster centroid as a query.\n6. Run the analyst hybrid query, and break its filter on purpose to see why placement matters.\n\n**Tip for Colab:** the first cells download `multilingual-e5-large` (about 2 GB) plus two small CLIP encoders, so give the setup a few minutes. Everything after that is fast.\n\nCompanion notebook to the [Module 5 lesson](https://qdrant.tech/course/beginners/module-5/).", + "metadata": { + "id": "ujOeHwdFcAef" + } + }, + { + "cell_type": "markdown", + "source": "## Setup\n\nThree models, each chosen for a reason.\n\n`intfloat/multilingual-e5-large` inherits 100 languages from XLM-RoBERTa and puts all of them in one vector space, which is the point of the whole system. `Qdrant/bm25` gives exact-token matching for supplier codes and ticker symbols. And a matched pair of CLIP encoders, one for images and one for text, puts pictures and words in a *second* shared space so a text query can retrieve a photo.\n\nNothing here needs a GPU or an API key.", + "metadata": { + "id": "AZhnM6Jy8c1r" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "ihtYlKNxHddm" + }, + "execution_count": null, + "source": "!pip install -q \"qdrant-client[fastembed]\" scikit-learn pillow ", + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "QAtKCsXRpJG2", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "b4f16c27" + }, + "execution_count": null, + "source": "import warnings\nimport numpy as np\nfrom qdrant_client import QdrantClient, models\nfrom fastembed import TextEmbedding, SparseTextEmbedding, ImageEmbedding\n\nwarnings.filterwarnings(\"ignore\")\n\nTEXT_MODEL = \"intfloat/multilingual-e5-large\" # 1024-dim, 100 languages\nSPARSE_MODEL = \"Qdrant/bm25\" # exact tokens\nCLIP_VISION = \"Qdrant/clip-ViT-B-32-vision\" # 512-dim images\nCLIP_TEXT = \"Qdrant/clip-ViT-B-32-text\" # 512-dim text, same space\n\ntext_model = TextEmbedding(TEXT_MODEL)\nsparse_model = SparseTextEmbedding(SPARSE_MODEL)\nclip_vision = ImageEmbedding(CLIP_VISION)\nclip_text = TextEmbedding(CLIP_TEXT)\n\nTEXT_DIM = len(next(text_model.embed([\"passage: warm up\"])))\nCLIP_DIM = len(next(clip_text.embed([\"warm up\"])))\nprint(\"text dim:\", TEXT_DIM, \"| clip dim:\", CLIP_DIM)", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "text dim: 1024 | clip dim: 512\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### The e5 prefixes are not optional\n\ne5 was trained with `query:` on search text and `passage:` on stored text. FastEmbed does not add them for you, and if you skip them nothing errors.\n\nMeasuring the effect takes a little care, because the thing that matters is not how high a relevant document scores. It is how far a relevant document sits above an irrelevant one, since that gap is what ranking depends on. So we score one relevant passage and one irrelevant passage against the same query, both ways, and compare the margin.", + "metadata": { + "id": "cPVJ2tRKJC06" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "DPdRO9YrxPbC", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "210c236d" + }, + "execution_count": null, + "source": "def unit(v):\n v = np.asarray(v, dtype=np.float32)\n return v / np.linalg.norm(v)\n\ndef embed_text(t, is_query=False):\n \"\"\"e5 needs an explicit prefix. This is the only place we add it.\"\"\"\n prefix = \"query: \" if is_query else \"passage: \"\n return unit(next(text_model.embed([prefix + t])))\n\ndef embed_raw(t):\n return unit(next(text_model.embed([t]))) # deliberately no prefix\n\nquestion = \"factory fire halted production\"\nrelevant = \"工場火災により生産が停止し、出荷の遅延が続いている。\"\nirrelevant = \"The supplier reaffirmed full-year guidance and reported steady quarterly demand.\"\n\nqv = embed_text(question, is_query=True)\nwith_rel, with_irr = float(qv @ embed_text(relevant)), float(qv @ embed_text(irrelevant))\n\nqr = embed_raw(question)\nraw_rel, raw_irr = float(qr @ embed_raw(relevant)), float(qr @ embed_raw(irrelevant))\n\nprint(f\"with prefixes relevant={with_rel:.3f} irrelevant={with_irr:.3f} margin={with_rel - with_irr:+.3f}\")\nprint(f\"without prefixes relevant={raw_rel:.3f} irrelevant={raw_irr:.3f} margin={raw_rel - raw_irr:+.3f}\")\nprint()\nprint(\"Skipping the prefixes RAISES both scores and NARROWS the gap between them.\")\nprint(\"Absolute similarity is not the target. Separation is.\")\nprint()\nprint(\"Note the scale too: e5 similarities compress into roughly 0.7 to 1.0,\")\nprint(\"so never read one of these numbers as a percentage of relevance.\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "with prefixes relevant=0.804 irrelevant=0.719 margin=+0.085\nwithout prefixes relevant=0.823 irrelevant=0.753 margin=+0.070\n\nSkipping the prefixes RAISES both scores and NARROWS the gap between them.\nAbsolute similarity is not the target. Separation is.\n\nNote the scale too: e5 similarities compress into roughly 0.7 to 1.0,\nso never read one of these numbers as a percentage of relevance.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 1. One Collection, Not Three\n\nThree modalities, and the instinct is three collections because that feels tidier.\n\nA single event produces evidence in several modalities at once. That factory fire is a news article, a satellite image, and a line in an earnings call. Split by modality and you have split one event across three collections: every query hits all three and stitches results back together in your own code, and your filters get written three times.\n\nNamed vectors solve it. One point carries a dense text vector, a sparse one, and a CLIP image vector, and all of them share a single payload.\n\nTwo details below decide whether this works in production. The sparse config needs `modifier=models.Modifier.IDF`, or you are not scoring BM25. And every payload index is created **now**, before any data arrives, because Qdrant can only add filter-aware edges to the vector index for indexes that already exist when it is built. Here there are two dense vectors, so a late index means rebuilding two graphs.\n\n`risk_score` is the one people forget, because nothing filters on it until Section 5. Miss it and the analyst query does not get slower, it fails: Qdrant Cloud enables strict mode by default, and strict mode rejects any query that filters an unindexed field.", + "metadata": { + "id": "eBI2Y1rzw27j" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "kooEKqazwJ7Q", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "89a0ecdc" + }, + "execution_count": null, + "source": "client = QdrantClient(\":memory:\")\n\nclient.create_collection(\n collection_name=\"supplier_signals\",\n vectors_config={\n \"text_dense\": models.VectorParams(size=TEXT_DIM, distance=models.Distance.COSINE),\n \"image\": models.VectorParams(size=CLIP_DIM, distance=models.Distance.COSINE),\n },\n sparse_vectors_config={\n \"text_sparse\": models.SparseVectorParams(modifier=models.Modifier.IDF),\n },\n)\n\nfor field in [\"supplier_id\", \"source_type\", \"language\", \"country\", \"facility_id\"]:\n client.create_payload_index(\"supplier_signals\", field_name=field,\n field_schema=models.PayloadSchemaType.KEYWORD)\n\nclient.create_payload_index(\"supplier_signals\", field_name=\"published_at\",\n field_schema=models.PayloadSchemaType.DATETIME)\nclient.create_payload_index(\"supplier_signals\", field_name=\"risk_score\",\n field_schema=models.PayloadSchemaType.FLOAT)\nclient.create_payload_index(\"supplier_signals\", field_name=\"cluster_id\",\n field_schema=models.PayloadSchemaType.INTEGER)\n\nprint(\"Collection created with 3 named vectors and 8 payload indexes, before ingestion.\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Collection created with 3 named vectors and 8 payload indexes, before ingestion.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 2. Ingesting Multilingual Signals\n\nThe signals below stand in for a day's feed. Note the shape of the story: the Japanese and Chinese sources are reporting a shutdown, while the English sources are reporting routine quarterly news. That gap is what Section 6 goes looking for.\n\n`source_type` is drawn from one fixed vocabulary, `news` or `satellite` here, because a filter written against a value nobody ingests returns nothing and warns you about nothing.", + "metadata": { + "id": "yygNdemkvdaj" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "gwb2Mjpvwh09", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "771154d2" + }, + "execution_count": null, + "source": "signals = [\n # SUP-7291: local-language sources are ahead of the English ones\n dict(supplier_id=\"SUP-7291\", language=\"ja\", country=\"JP\", source_type=\"news\",\n published_at=\"2026-07-21T09:00:00Z\", risk_score=0.88,\n text=\"工場火災により生産が停止し、出荷の遅延が続いている。復旧の見込みは立っていない。\"),\n dict(supplier_id=\"SUP-7291\", language=\"zh\", country=\"CN\", source_type=\"news\",\n published_at=\"2026-07-21T14:00:00Z\", risk_score=0.82,\n text=\"供应商工厂因火灾停产，交货期限推迟，客户已收到延误通知。\"),\n dict(supplier_id=\"SUP-7291\", language=\"ja\", country=\"JP\", source_type=\"news\",\n published_at=\"2026-07-22T02:00:00Z\", risk_score=0.71,\n text=\"労働争議が長引き、工場の稼働率が低下している。組合との交渉は難航している。\"),\n dict(supplier_id=\"SUP-7291\", language=\"vi\", country=\"VN\", source_type=\"news\",\n published_at=\"2026-07-20T08:00:00Z\", risk_score=0.64,\n text=\"Cang Hai Phong bi tac nghen, cac chuyen hang cua nha cung cap bi cham tre.\"),\n dict(supplier_id=\"SUP-7291\", language=\"en\", country=\"US\", source_type=\"news\",\n published_at=\"2026-07-22T07:15:00Z\", risk_score=0.12,\n text=\"The supplier reaffirmed full-year guidance and reported steady quarterly demand.\"),\n dict(supplier_id=\"SUP-7291\", language=\"en\", country=\"GB\", source_type=\"news\",\n published_at=\"2026-07-22T11:00:00Z\", risk_score=0.10,\n text=\"Analysts described the quarter as routine, with no change to the outlook for the group.\"),\n dict(supplier_id=\"SUP-7291\", language=\"en\", country=\"US\", source_type=\"news\",\n published_at=\"2026-07-19T16:00:00Z\", risk_score=0.55,\n text=\"Ticker SUP7291.T slipped modestly on higher freight costs across the region.\"),\n # a second supplier, so tenant-style scoping has something to exclude\n dict(supplier_id=\"SUP-0002\", language=\"zh\", country=\"CN\", source_type=\"news\",\n published_at=\"2026-07-21T10:00:00Z\", risk_score=0.79,\n text=\"另一家供应商的工厂发生停电，生产线暂时中断。\"),\n dict(supplier_id=\"SUP-0002\", language=\"en\", country=\"DE\", source_type=\"news\",\n published_at=\"2026-07-18T09:00:00Z\", risk_score=0.20,\n text=\"A European logistics operator announced a routine expansion of warehouse capacity.\"),\n]\n\ndef to_sparse(text, is_query=False):\n emb = next(sparse_model.query_embed(text)) if is_query else next(sparse_model.embed([text]))\n return models.SparseVector(indices=emb.indices.tolist(), values=emb.values.tolist())\n\npoints = []\nfor i, s in enumerate(signals):\n points.append(models.PointStruct(\n id=i,\n vector={\n \"text_dense\": embed_text(s[\"text\"]).tolist(), # passage: prefix\n \"text_sparse\": to_sparse(s[\"text\"]),\n },\n payload={**s, \"summary\": s[\"text\"][:60]},\n ))\n\nclient.upsert(\"supplier_signals\", points=points)\nprint(\"ingested\", client.count(\"supplier_signals\").count, \"text signals across\",\n len({s[\"language\"] for s in signals}), \"languages\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "ingested 9 text signals across 4 languages\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 3. Satellite Tiles Through CLIP\n\nReal satellite imagery is not something a notebook can ship, so we generate four crude tiles instead. They are enough to show the mechanics and, as you will see, enough to show a real limitation too.\n\nThese points carry only the `image` vector. They live in the same collection as the text signals and share the same payload schema, which is the whole argument for named vectors.", + "metadata": { + "id": "b6GjHbG81Rqw" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "kVHWunY90XYT", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "3850044b" + }, + "execution_count": null, + "source": "from PIL import Image, ImageDraw, ImageFilter\nimport os\n\nos.makedirs(\"tiles\", exist_ok=True)\n\ndef tile_smoke():\n im = Image.new(\"RGB\", (224, 224), (70, 72, 78)); d = ImageDraw.Draw(im)\n d.rectangle([40, 150, 184, 224], fill=(48, 48, 52))\n for y, r in [(140, 20), (115, 28), (88, 36), (60, 44)]:\n d.ellipse([112 - r, y - r, 112 + r, y + r], fill=(190, 190, 195))\n return im.filter(ImageFilter.GaussianBlur(6))\n\ndef tile_farmland():\n im = Image.new(\"RGB\", (224, 224), (94, 140, 60)); d = ImageDraw.Draw(im)\n for x in range(0, 224, 16):\n d.rectangle([x, 0, x + 8, 224], fill=(120, 168, 74))\n return im.filter(ImageFilter.GaussianBlur(1))\n\ndef tile_harbour():\n im = Image.new(\"RGB\", (224, 224), (40, 90, 150)); d = ImageDraw.Draw(im)\n for x, y in [(30, 60), (120, 100), (70, 160), (150, 40)]:\n d.rectangle([x, y, x + 50, y + 18], fill=(210, 210, 215))\n return im.filter(ImageFilter.GaussianBlur(1))\n\ndef tile_fire():\n im = Image.new(\"RGB\", (224, 224), (30, 20, 15)); d = ImageDraw.Draw(im)\n for r, c in [(90, (120, 30, 0)), (65, (200, 70, 0)), (40, (255, 150, 0)), (20, (255, 230, 120))]:\n d.ellipse([112 - r, 150 - r, 112 + r, 150 + r], fill=c)\n return im.filter(ImageFilter.GaussianBlur(8))\n\ntiles = [\n (\"smoke_plume\", tile_smoke, \"SUP-7291\", \"FAC-01\"),\n (\"fire\", tile_fire, \"SUP-7291\", \"FAC-01\"),\n (\"harbour\", tile_harbour, \"SUP-7291\", \"FAC-02\"),\n (\"farmland\", tile_farmland, \"SUP-0002\", \"FAC-09\"),\n]\n\npaths = []\nfor name, fn, _, _ in tiles:\n p = f\"tiles/{name}.png\"\n fn().save(p)\n paths.append(p)\n\nvecs = [unit(v).tolist() for v in clip_vision.embed(paths)]\n\nclient.upsert(\"supplier_signals\", points=[\n models.PointStruct(\n id=100 + i,\n vector={\"image\": vec},\n payload=dict(supplier_id=sup, facility_id=fac, source_type=\"satellite\",\n language=\"n/a\", country=\"VN\",\n published_at=\"2026-07-21T00:00:00Z\",\n risk_score=0.9 if name in (\"fire\", \"smoke_plume\") else 0.1,\n summary=f\"synthetic satellite tile: {name}\"),\n )\n for i, (vec, (name, _, sup, fac)) in enumerate(zip(vecs, tiles))\n])\n\nprint(\"collection now holds\", client.count(\"supplier_signals\").count, \"points (text + imagery)\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "collection now holds 13 points (text + imagery)\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 4. Two Kinds of Cross-Modal Query\n\n### Text against images, via CLIP\n\nThe query text goes through CLIP's **text** encoder, not e5. That is the part people get wrong: each named vector is its own space, and a query only means something in the space it was embedded for. Sending an e5 vector at the `image` vector would return numbers, and they would be noise.", + "metadata": { + "id": "7B697wnommM3" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "0DOXfO4WUDlW", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "5cf66fb2" + }, + "execution_count": null, + "source": "def image_search(text, limit=4):\n qv = unit(next(clip_text.embed([text]))).tolist() # CLIP text encoder\n return client.query_points(\"supplier_signals\", query=qv, using=\"image\",\n limit=limit).points\n\nfor q in [\"an orange fire burning\", \"ships in blue water at a port\", \"grey smoke rising into the sky\"]:\n print(f\"{q!r}\")\n for r in image_search(q):\n print(f\" {r.score:.3f} {r.payload['summary']}\")\n print()", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "'an orange fire burning'\n 0.267 synthetic satellite tile: fire\n 0.209 synthetic satellite tile: farmland\n 0.191 synthetic satellite tile: smoke_plume\n 0.188 synthetic satellite tile: harbour\n\n'ships in blue water at a port'\n 0.209 synthetic satellite tile: harbour\n 0.193 synthetic satellite tile: smoke_plume\n 0.180 synthetic satellite tile: farmland\n 0.141 synthetic satellite tile: fire\n\n'grey smoke rising into the sky'\n 0.211 synthetic satellite tile: fire\n 0.204 synthetic satellite tile: farmland\n 0.195 synthetic satellite tile: smoke_plume\n 0.194 synthetic satellite tile: harbour\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Two of those three are right, and the third is the useful one.\n\n\"an orange fire burning\" and \"ships in blue water at a port\" retrieve the correct tiles. \"grey smoke rising into the sky\" does not: it prefers the fire tile.\n\nThat is not a bug in Qdrant or in the query. CLIP was trained on photographs, and these tiles are flat drawings made with a few ellipses. They sit outside the distribution the model learned, so its judgments get unreliable. Swap in real satellite imagery and this improves immediately.\n\nThe lesson generalizes past this notebook: cross-modal retrieval quality depends on your images resembling the model's training data, and the only way to know is to evaluate on your own.", + "metadata": { + "id": "hj4YjZ8OvIia" + } + }, + { + "cell_type": "markdown", + "source": "### English query against local-language text\n\nNow the capability the whole system exists for. The query is English, the corpus is not, and there is no translation step anywhere.\n\nThe filter has no prefetch above it, so `query_filter` is the correct placement here.", + "metadata": { + "id": "8OG5A2ZnpsoK" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "qzi4vCK4A4FG", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "80720c27" + }, + "execution_count": null, + "source": "def text_search(query_en, languages=None, supplier=None, limit=5):\n must = []\n if supplier:\n must.append(models.FieldCondition(key=\"supplier_id\",\n match=models.MatchValue(value=supplier)))\n if languages:\n must.append(models.FieldCondition(key=\"language\",\n match=models.MatchAny(any=languages)))\n return client.query_points(\n \"supplier_signals\",\n query=embed_text(query_en, is_query=True).tolist(), # query: prefix\n using=\"text_dense\",\n query_filter=models.Filter(must=must) if must else None,\n limit=limit,\n ).points\n\nprint(\"English query -> Japanese and Chinese sources only:\\n\")\nfor r in text_search(\"factory shutdown production halt\",\n languages=[\"ja\", \"zh\"], supplier=\"SUP-7291\"):\n p = r.payload\n print(f\" {r.score:.3f} [{p['language']}] risk={p['risk_score']:.2f} {p['summary']}\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "English query -> Japanese and Chinese sources only:\n\n" + }, + { + "output_type": "stream", + "name": "stdout", + "text": " 0.786 [zh] risk=0.82 供应商工厂因火灾停产，交货期限推迟，客户已收到延误通知。\n 0.776 [ja] risk=0.88 工場火災により生産が停止し、出荷の遅延が続いている。復旧の見込みは立っていない。\n 0.758 [ja] risk=0.71 労働争議が長引き、工場の稼働率が低下している。組合との交渉は難航している。\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 5. Clustering Signals Into Risk Themes\n\nClustering groups signals that describe the same underlying event even when they arrive in different languages from different sources.\n\nThree practical points, all of them easy to get wrong.\n\n**Page the scroll.** It returns a batch and an offset, and you keep going until the offset comes back empty. One capped call quietly clusters a busy supplier on partial data.\n\n**Normalize before k-means.** These vectors are built for cosine similarity, but k-means measures Euclidean distance. Without normalizing you are partly clustering by vector length instead of direction.\n\n**Drop the supplier filter to go wider.** A theme shared across suppliers shows up as one cluster pulling in signals from several of them at once.", + "metadata": { + "id": "SIuGIWgIprVF" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "IV4BLFFQCHD6", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "bef48fd2" + }, + "execution_count": null, + "source": "from sklearn.cluster import KMeans\n\ndef scroll_all(scroll_filter=None, page=64):\n \"\"\"Page until the offset comes back empty.\"\"\"\n out, offset = [], None\n while True:\n batch, offset = client.scroll(\"supplier_signals\", scroll_filter=scroll_filter,\n with_vectors=True, limit=page, offset=offset)\n out.extend(batch)\n if offset is None:\n return out\n\ndef dense_matrix(points):\n \"\"\"Unit-normalized text_dense vectors, with the ids they belong to.\"\"\"\n ids, vecs = [], []\n for p in points:\n if p.vector and \"text_dense\" in p.vector: # image-only points have none\n ids.append(p.id)\n vecs.append(p.vector[\"text_dense\"])\n if not vecs:\n return [], None\n arr = np.asarray(vecs, dtype=np.float32)\n arr /= np.linalg.norm(arr, axis=1, keepdims=True)\n return ids, arr\n\ntext_only = models.Filter(must=[models.FieldCondition(\n key=\"source_type\", match=models.MatchValue(value=\"news\"))])\n\npts = scroll_all(text_only)\nids, arr = dense_matrix(pts)\nprint(f\"scrolled {len(pts)} points, {len(ids)} carry a text vector\")\n\nlabels = KMeans(n_clusters=3, n_init=10, random_state=42).fit_predict(arr)\n\n# One set_payload call per cluster, not one per point\nfor label in sorted({int(l) for l in labels}):\n client.set_payload(\"supplier_signals\", payload={\"cluster_id\": label},\n points=[i for i, l in zip(ids, labels) if int(l) == label])\n\nby_id = {p.id: p.payload for p in pts}\nfor label in sorted({int(l) for l in labels}):\n print(f\"\\ncluster {label}:\")\n for i, l in zip(ids, labels):\n if int(l) == label:\n p = by_id[i]\n print(f\" [{p['language']}] risk={p['risk_score']:.2f} {p['summary'][:52]}\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "scrolled 9 points, 9 carry a text vector\n\ncluster 0:\n [vi] risk=0.64 Cang Hai Phong bi tac nghen, cac chuyen hang cua nha\n\ncluster 1:\n [en] risk=0.12 The supplier reaffirmed full-year guidance and repor\n [en] risk=0.10 Analysts described the quarter as routine, with no c\n [en] risk=0.55 Ticker SUP7291.T slipped modestly on higher freight \n [en] risk=0.20 A European logistics operator announced a routine ex\n\ncluster 2:\n [ja] risk=0.88 工場火災により生産が停止し、出荷の遅延が続いている。復旧の見込みは立っていない。\n [zh] risk=0.82 供应商工厂因火灾停产，交货期限推迟，客户已收到延误通知。\n [ja] risk=0.71 労働争議が長引き、工場の稼働率が低下している。組合との交渉は難航している。\n [zh] risk=0.79 另一家供应商的工厂发生停电，生产线暂时中断。\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Notice that the clusters cross language boundaries. The Japanese fire report and the Chinese shutdown report land together because they describe the same event, which is exactly what a single multilingual vector space buys you.\n\n### The centroid as a query\n\nA cluster centroid is just another vector, so you can hand it straight back to Qdrant as a query and pull in more of the same theme. That is how you go from \"here are today's clusters\" to \"find everything that looks like this emerging story\".", + "metadata": { + "id": "hBeoePW3jSOi" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "27n7yE9ZkAFv", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "35b08705" + }, + "execution_count": null, + "source": "target = int(labels[0])\nmember_rows = [i for i, l in zip(ids, labels) if int(l) == target]\ncentroid = arr[[ids.index(i) for i in member_rows]].mean(axis=0)\ncentroid = centroid / np.linalg.norm(centroid)\n\nprint(f\"querying with the centroid of cluster {target}:\\n\")\nfor r in client.query_points(\"supplier_signals\", query=centroid.tolist(),\n using=\"text_dense\", limit=5).points:\n p = r.payload\n mark = \" <- in the cluster\" if r.id in member_rows else \"\"\n print(f\" {r.score:.3f} [{p['language']}] {p['summary'][:48]}{mark}\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "querying with the centroid of cluster 2:\n\n 0.962 [zh] 供应商工厂因火灾停产，交货期限推迟，客户已收到延误通知。 <- in the cluster\n 0.961 [zh] 另一家供应商的工厂发生停电，生产线暂时中断。 <- in the cluster\n 0.957 [ja] 工場火災により生産が停止し、出荷の遅延が続いている。復旧の見込みは立っていない。 <- in the cluster\n 0.947 [ja] 労働争議が長引き、工場の稼働率が低下している。組合との交渉は難航している。 <- in the cluster\n 0.798 [vi] Cang Hai Phong bi tac nghen, cac chuyen hang cua\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 6. The Analyst Query, and the Mistake That Fails Silently\n\nThe query analysts actually run: hybrid retrieval over dense and sparse, scoped to one supplier, elevated risk only.\n\nOne detail decides whether it works. The filter goes **inside each prefetch**, not on the outer query. Prefetches run first and the outer query is applied to their results, so a top-level filter arrives too late: both retrievers search every supplier and every risk level, and the filter only trims the fused list at the end.\n\nBoth versions run below. Watch the `risk` and `supplier` columns in the broken one.", + "metadata": { + "id": "MFd9W8gqcWqi" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "A4Yqj4JRfdQA", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "1fac1d1d" + }, + "execution_count": null, + "source": "def analyst_search(query_text, supplier, min_risk=0.5, limit=5, broken=False):\n dense_q = embed_text(query_text, is_query=True).tolist()\n sparse_q = to_sparse(query_text, is_query=True)\n risk_filter = models.Filter(must=[\n models.FieldCondition(key=\"supplier_id\", match=models.MatchValue(value=supplier)),\n models.FieldCondition(key=\"risk_score\", range=models.Range(gte=min_risk)),\n ])\n if broken:\n return client.query_points(\n \"supplier_signals\",\n prefetch=[\n models.Prefetch(query=dense_q, using=\"text_dense\", limit=50),\n models.Prefetch(query=sparse_q, using=\"text_sparse\", limit=50),\n ],\n query=models.RrfQuery(rrf=models.Rrf()),\n query_filter=risk_filter, # too late\n limit=limit,\n ).points\n return client.query_points(\n \"supplier_signals\",\n prefetch=[\n models.Prefetch(query=dense_q, using=\"text_dense\",\n filter=risk_filter, limit=50),\n models.Prefetch(query=sparse_q, using=\"text_sparse\",\n filter=risk_filter, limit=50),\n ],\n query=models.RrfQuery(rrf=models.Rrf()),\n limit=limit,\n ).points\n\ndef report(title, rows):\n print(title)\n if not rows:\n print(\" (no results)\")\n for r in rows:\n p = r.payload\n print(f\" {r.score:.4f} {p['supplier_id']} risk={p['risk_score']:.2f} \"\n f\"[{p['language']}] {p['summary'][:42]}\")\n print()\n\nreport(\"CORRECT filter inside each prefetch:\",\n analyst_search(\"production halt at the factory\", \"SUP-7291\"))\nreport(\"BROKEN same filter at the top level:\",\n analyst_search(\"production halt at the factory\", \"SUP-7291\", broken=True))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "CORRECT filter inside each prefetch:\n 0.5000 SUP-7291 risk=0.82 [zh] 供应商工厂因火灾停产，交货期限推迟，客户已收到延误通知。\n 0.3333 SUP-7291 risk=0.88 [ja] 工場火災により生産が停止し、出荷の遅延が続いている。復旧の見込みは立っていない。\n 0.2500 SUP-7291 risk=0.55 [en] Ticker SUP7291.T slipped modestly on highe\n 0.2000 SUP-7291 risk=0.71 [ja] 労働争議が長引き、工場の稼働率が低下している。組合との交渉は難航している。\n 0.1667 SUP-7291 risk=0.64 [vi] Cang Hai Phong bi tac nghen, cac chuyen ha\n\n" + }, + { + "output_type": "stream", + "name": "stdout", + "text": "BROKEN same filter at the top level:\n 0.5000 SUP-0002 risk=0.79 [zh] 另一家供应商的工厂发生停电，生产线暂时中断。\n 0.3333 SUP-7291 risk=0.82 [zh] 供应商工厂因火灾停产，交货期限推迟，客户已收到延误通知。\n 0.2500 SUP-7291 risk=0.88 [ja] 工場火災により生産が停止し、出荷の遅延が続いている。復旧の見込みは立っていない。\n 0.2000 SUP-7291 risk=0.55 [en] Ticker SUP7291.T slipped modestly on highe\n 0.1667 SUP-7291 risk=0.71 [ja] 労働争議が長引き、工場の稼働率が低下している。組合との交渉は難航している。\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "The broken version returns rows that violate the filter, with no error and no warning. On a real corpus, where one supplier is a thousandth of the collection instead of most of it, the same mistake usually returns nothing at all, and an analyst concludes there is no news.\n\nThe rule, one more time. No prefetch, use `query_filter`. Prefetch, put the filter in every prefetch.", + "metadata": { + "id": "CvI5qiGogkdm" + } + }, + { + "cell_type": "markdown", + "source": "## 7. Reading the Gap\n\nThe payoff. Run the same query twice, once scoped to English sources and once to Japanese and Chinese, and compare.\n\nThis is not a different algorithm. It is the same query and the same vector space with a different `language` filter.", + "metadata": { + "id": "tsVrFlvbpaTI" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "lD5gYAEZyFQ8", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "5dba6502" + }, + "execution_count": null, + "source": "query = \"factory shutdown, production halted, delivery delays\"\n\nreport(\"ENGLISH sources only:\",\n text_search(query, languages=[\"en\"], supplier=\"SUP-7291\"))\nreport(\"JAPANESE and CHINESE sources only:\",\n text_search(query, languages=[\"ja\", \"zh\"], supplier=\"SUP-7291\"))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "ENGLISH sources only:\n 0.7789 SUP-7291 risk=0.55 [en] Ticker SUP7291.T slipped modestly on highe\n 0.7239 SUP-7291 risk=0.12 [en] The supplier reaffirmed full-year guidance\n 0.6827 SUP-7291 risk=0.10 [en] Analysts described the quarter as routine,\n\nJAPANESE and CHINESE sources only:\n 0.8059 SUP-7291 risk=0.82 [zh] 供应商工厂因火灾停产，交货期限推迟，客户已收到延误通知。\n 0.7946 SUP-7291 risk=0.88 [ja] 工場火災により生産が停止し、出荷の遅延が続いている。復旧の見込みは立っていない。\n 0.7731 SUP-7291 risk=0.71 [ja] 労働争議が長引き、工場の稼働率が低下している。組合との交渉は難航している。\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "The English sources are reporting reaffirmed guidance and a routine quarter. The Japanese and Chinese sources are reporting a fire, halted production, and delayed deliveries, and they were published earlier.\n\nThat gap is the early warning, and finding it needed no translation pipeline, no separate per-language index, and no second database. One collection, one multilingual model, one filter.", + "metadata": { + "id": "eTpkf01PrpsC" + } + }, + { + "cell_type": "markdown", + "source": "## Course Close\n\nLook at what is in that collection. Text, sparse tokens, and imagery on shared points. Filters that hold. Hybrid retrieval. Clustering. Cross-language and cross-modal search.\n\nSix primitives got you here: collection, point, vector, payload, index, query.\n\nModule 1 asked why keyword search misses. Module 2 opened up the vector. Module 3 put dense and sparse together. Module 4 turned that into a design you could defend. This module ran the whole thing on three modalities at once.\n\nThe next system someone hands you is these six, arranged differently.\n\n### Your turn\n\nSwap the synthetic tiles for real satellite imagery and rerun Section 4, then check whether the smoke query starts behaving.\n\nAdd a `tenant_id` field with `is_tenant=True` and scope every query to one desk, the way Module 4 did.\n\nThen try `models.Rrf(weights=[3.0, 1.0])` in the analyst query to favour dense over sparse, and see which retriever your data actually prefers.", + "metadata": { + "id": "jBq78Hp7twax" + } + } + ] +}