From fbf2ca2aa987b791818425f4ac067eeec3fee74c Mon Sep 17 00:00:00 2001 From: Ewa Szyszka <79250896+ESzyszka@users.noreply.github.com> Date: Mon, 27 Jul 2026 16:12:18 +0200 Subject: [PATCH 1/3] Add files via upload --- Beginner-course/Module1.ipynb | 317 ++++++++++++++++++++++++ Beginner-course/Module2.ipynb | 265 ++++++++++++++++++++ Beginner-course/Module3.ipynb | 393 ++++++++++++++++++++++++++++++ Beginner-course/Module4.ipynb | 443 ++++++++++++++++++++++++++++++++++ Beginner-course/Module5.ipynb | 358 +++++++++++++++++++++++++++ 5 files changed, 1776 insertions(+) create mode 100644 Beginner-course/Module1.ipynb create mode 100644 Beginner-course/Module2.ipynb create mode 100644 Beginner-course/Module3.ipynb create mode 100644 Beginner-course/Module4.ipynb create mode 100644 Beginner-course/Module5.ipynb diff --git a/Beginner-course/Module1.ipynb b/Beginner-course/Module1.ipynb new file mode 100644 index 0000000..f9db5c4 --- /dev/null +++ b/Beginner-course/Module1.ipynb @@ -0,0 +1,317 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "cells": [ + { + "cell_type": "markdown", + "source": "# Module 1: Let's Understand Search\n\n## What you will do\n\n1. Watch keyword search miss documents that plainly answer the query.\n2. Turn text into a vector and look at what actually comes back.\n3. Score meaning with cosine similarity, first by hand, then with the library helper.\n4. Re-run the Section 1 failures semantically and watch them pass.\n5. Meet the SKU problem, where similarity on its own is not enough.\n\n**Tip:** every cell below is already run, so you can read this straight through. Run it yourself and you should see the same numbers.\n\nCompanion notebook to the [Module 1 lesson](https://qdrant.tech/course/beginners/module-1/).", + "metadata": { + "id": "i0VpEBOWfbZA" + } + }, + { + "cell_type": "markdown", + "source": "## Setup\n\nModule 1 stays deliberately small: one embedding model, no vector database yet. `sentence-transformers` gives us `all-MiniLM-L6-v2`, a 384-dimension model fast enough to run on a free Colab CPU. Qdrant arrives in Module 2.", + "metadata": { + "id": "VaBSo63bbH6x" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "nAbnBEoonCrb" + }, + "execution_count": null, + "source": "!pip install -q sentence-transformers", + "outputs": [] + }, + { + "cell_type": "markdown", + "source": "## 1. The Problem: Why Keyword Search Struggles\n\nTraditional search matches exact words. If the query string appears in the document it is a hit, and if it does not it is a miss, however close the meaning is.\n\nHere is the whole of keyword search, in three lines.", + "metadata": { + "id": "SS6AZPsK20FK" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "cpXzkIRPxBFW", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "86700bf9" + }, + "execution_count": null, + "source": "documents = [\n \"automobile maintenance guide\",\n \"affordable airfare to New York\",\n \"apple harvest season guide\",\n]\n\ndef keyword_search(query, docs):\n \"\"\"All of keyword search: does the query string appear verbatim?\"\"\"\n return [d for d in docs if query.lower() in d.lower()]\n\nfor query in [\"car repair\", \"cheap flights NYC\", \"Apple stock\"]:\n hits = keyword_search(query, documents)\n print(f\"{query!r:22} -> {hits if hits else 'NO MATCH'}\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "'car repair' -> NO MATCH\n'cheap flights NYC' -> NO MATCH\n'Apple stock' -> NO MATCH\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Three queries. Three documents that answer them. Zero hits.\n\nNothing is broken. The engine is doing exactly what it was asked: comparing characters. That single limitation shows up in four ways.\n\n- **Synonyms.** \"car\" and \"automobile\" mean the same thing and share no letters.\n- **Paraphrasing.** \"cheap flights\" and \"affordable airfare\" are the same intent in different words.\n- **Polysemy.** \"apple\" is a company, a fruit, and a record label. Characters cannot tell you which one was meant.\n- **Word order.** \"dog bites man\" and \"man bites dog\" contain identical words.\n\nImprovements to keyword search, inverted indexes for speed, BM25 for ranking, stemming, and fuzzy matching, all raise the ceiling without moving it. They still work on words. You could hard-code that \"car\" means \"automobile\", then do it again for the next pair, and the next. You cannot hard-code a language.\n\nSo semantic search changes the question. Not *does this document contain the same words*, but *does this document mean the same thing*.", + "metadata": { + "id": "ykF8VH1oF7JC" + } + }, + { + "cell_type": "markdown", + "source": "## 2. How It Works: Embeddings\n\nAn embedding model takes a piece of text and returns a fixed-length list of floating-point numbers. Similar meanings produce vectors that sit close together in that space.", + "metadata": { + "id": "qH7aWY2TYGIA" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "d1JmFEwvHMLC", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "0329dccd" + }, + "execution_count": null, + "source": "from sentence_transformers import SentenceTransformer, util\n\nmodel = SentenceTransformer(\"all-MiniLM-L6-v2\")\n\nquery_vec = model.encode(\"car repair\")\ndoc_vec = model.encode(\"automobile maintenance\")\n\nprint(\"dimensions:\", len(query_vec))\nprint(\"first five:\", query_vec[:5])", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "dimensions: 384\nfirst five: [-0.12456685 0.03955904 0.08732592 -0.02361241 -0.07054088]\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "384 numbers. That is the vector.\n\nDo not go looking for the dimension that stores colour, or urgency. Nobody designed these. The model learned them from data, and meaning lives in the combination of all 384 rather than in any single one. A 384-dimension model has 384 such aspects, and only the whole set together encodes meaning.", + "metadata": { + "id": "p0PfbaUpgMve" + } + }, + { + "cell_type": "markdown", + "source": "## 3. Comparing Meaning: Cosine Similarity\n\nCosine similarity measures the angle between two vectors and ignores their length. Vectors pointing the same way score near 1. Unrelated directions score near 0.\n\nThe formula is short enough to write out, which is worth doing once so the library call stops being magic.", + "metadata": { + "id": "jGkqSCtDbxAl" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "g3FmLYbbjSjR", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "a83fc83a" + }, + "execution_count": null, + "source": "import numpy as np\n\ndef cosine_by_hand(a, b):\n return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))\n\nby_hand = cosine_by_hand(query_vec, doc_vec)\nby_library = util.cos_sim(query_vec, doc_vec).item()\n\nprint(f\"by hand: {by_hand:.6f}\")\nprint(f\"cos_sim: {by_library:.6f}\")\nprint(f\"agree: {np.isclose(by_hand, by_library)}\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "by hand: 0.733420\ncos_sim: 0.733420\nagree: True\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Roughly 0.73 for two phrases with no words in common. That number is the whole point of the module.\n\nNow three pairs at once: a synonym pair, a paraphrase pair, and a pair that has nothing to do with each other.", + "metadata": { + "id": "yJtNds82s1t6" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "Uzpa2cY7Jk18", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "b8653a63" + }, + "execution_count": null, + "source": "pairs = [\n (\"car repair\", \"automobile maintenance\"), # synonyms\n (\"cheap flights to New York\", \"affordable airfare to NYC\"), # paraphrase\n (\"cheap flights to New York\", \"best pizza in Chicago\"), # unrelated\n]\n\nfor query, document in pairs:\n score = util.cos_sim(model.encode(query), model.encode(document)).item()\n print(f\"{score:.3f} | {query!r} vs {document!r}\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "0.733 | 'car repair' vs 'automobile maintenance'\n0.821 | 'cheap flights to New York' vs 'affordable airfare to NYC'\n0.332 | 'cheap flights to New York' vs 'best pizza in Chicago'\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "The first two pairs share little or no vocabulary and both score high. That is the synonym and the paraphrase that keyword search missed in Section 1. The third scores far lower, so the model is separating meaning rather than matching surface words.\n\nWhich means we can go back and re-run those three failures.", + "metadata": { + "id": "gPsEu23bmu9W" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "vnPZ5v2QE9oS", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "12e3c9d5" + }, + "execution_count": null, + "source": "# Same three documents as Section 1, plus a second \"apple\" sense so the\n# polysemy case has somewhere correct to land.\ndocuments = [\n \"automobile maintenance guide\",\n \"affordable airfare to New York\",\n \"apple harvest season guide\",\n \"technology shares closed higher today\",\n]\n\ndoc_vecs = model.encode(documents)\n\nfor query in [\"car repair\", \"cheap flights NYC\", \"Apple stock\"]:\n sims = util.cos_sim(model.encode(query), doc_vecs)[0]\n ranked = sorted(zip(sims.tolist(), documents), reverse=True)\n print(f\"{query!r}\")\n for score, doc in ranked:\n print(f\" {score:6.3f} {doc}\")\n print()", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "'car repair'\n 0.609 automobile maintenance guide\n 0.065 apple harvest season guide\n 0.060 technology shares closed higher today\n -0.032 affordable airfare to New York\n\n'cheap flights NYC'\n 0.800 affordable airfare to New York\n 0.117 apple harvest season guide\n 0.080 technology shares closed higher today\n -0.021 automobile maintenance guide\n\n'Apple stock'\n 0.401 technology shares closed higher today\n 0.374 apple harvest season guide\n 0.145 affordable airfare to New York\n 0.048 automobile maintenance guide\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Every query now finds its document, and each one was a total miss under keyword search.\n\nTwo things in that output are worth pausing on.\n\n**Negative scores exist.** `car repair` against `affordable airfare to New York` comes out slightly below zero. Cosine similarity ranges from -1 to 1 in principle, where -1 means the vectors point in opposite directions. With normalized text models these stay small and near zero, so treat a value like -0.03 as \"unrelated\" rather than \"opposite\".\n\n**The polysemy win is narrow.** `Apple stock` prefers the technology document over the fruit document, but only by about 0.03. The model got the sense right, and it was close. Hold on to that, because Section 4 is built on exactly this kind of narrow gap.", + "metadata": { + "id": "vs1L73iIgce1" + } + }, + { + "cell_type": "markdown", + "source": "### Where dense search struggles too\n\nThe four failure modes were framed as keyword problems, but word order is not solved by switching to embeddings. Compare a sentence with its meaning reversed, and with a genuine paraphrase.", + "metadata": { + "id": "WZvMWxJnfq3I" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "5CrWZaPfh2co", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "f6123b1e" + }, + "execution_count": null, + "source": "reversed_pair = (\"dog bites man\", \"man bites dog\")\nparaphrase_pair = (\"dog bites man\", \"a canine attacked a person\")\n\nfor a, b in [reversed_pair, paraphrase_pair]:\n print(f\"{util.cos_sim(model.encode(a), model.encode(b)).item():.3f} | {a!r} vs {b!r}\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "0.907 | 'dog bites man' vs 'man bites dog'\n0.570 | 'dog bites man' vs 'a canine attacked a person'\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "The reversed sentence scores higher than the correct paraphrase.\n\nThe model rates *opposite meaning, identical words* as more similar than *same meaning, different words*. Embeddings shifted the problem rather than removing it, and that is worth knowing before you trust similarity with anything important.", + "metadata": { + "id": "x84spDgOca8s" + } + }, + { + "cell_type": "markdown", + "source": "## 4. Why Similarity Alone Is Not Enough\n\nHere is the failure that costs real money. A user types an exact product code and wants that product.", + "metadata": { + "id": "KBtd488gnM8H" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "QPqHtpfr8BON", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "f6455de4" + }, + "execution_count": null, + "source": "skus = [\"SKU-48290\", \"SKU-48291\", \"SKU-48292\", \"SKU-48293\"]\ncatalog = [f\"{sku} replacement part, ships in 2 days\" for sku in skus]\n\nquery = model.encode(\"SKU-48291 issue\")\nsims = util.cos_sim(query, model.encode(catalog))[0]\n\nscored = sorted(zip(sims.tolist(), skus), reverse=True)\nfor score, sku in scored:\n marker = \" <-- the one they asked for\" if sku == \"SKU-48291\" else \"\"\n print(f\"{score:.3f} {sku}{marker}\")\n\nprint()\nprint(f\"spread between best and worst: {scored[0][0] - scored[-1][0]:.3f}\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "0.626 SKU-48290\n0.624 SKU-48291 <-- the one they asked for\n0.616 SKU-48293\n0.610 SKU-48292\n\nspread between best and worst: 0.016\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "The requested SKU is not first, and look at the spread: all four codes land within a couple of hundredths of each other.\n\nThat is not a bug in the model. Those strings are nearly identical, so their vectors are nearly identical, and the model has no idea that the final digit is the only part that matters. Semantically they are the same thing. Commercially, three of the four are the wrong product.\n\nNo amount of model tuning fixes this, because the model is not wrong. What is missing is a hard constraint.", + "metadata": { + "id": "uHJKpbpeIeeN" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "sXE1jFXe6kW9", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "d518e4e2" + }, + "execution_count": null, + "source": "TARGET = \"SKU-48291\"\n\n# An exact constraint on a metadata field, not a similarity score\nfiltered = [(score, sku) for score, sku in scored if sku == TARGET]\n\nfor score, sku in filtered:\n print(f\"{score:.3f} {sku}\")\n\nprint()\nprint(\"In Qdrant that constraint is a payload filter, evaluated while the search runs:\")\nprint(' must=[FieldCondition(key=\"sku\", match=MatchValue(value=\"SKU-48291\"))]')\nprint(\"You will write exactly that in Module 2.\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "0.624 SKU-48291\n\nIn Qdrant that constraint is a payload filter, evaluated while the search runs:\n must=[FieldCondition(key=\"sku\", match=MatchValue(value=\"SKU-48291\"))]\nYou will write exactly that in Module 2.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### Key insight\n\nDense similarity finds the neighbourhood. Filters, exact matches, and payload constraints find the right point inside it. Production search needs both, which is why real systems are hybrid: dense retrieval for meaning, sparse retrieval for exact tokens, and filters constraining both.", + "metadata": { + "id": "4bt3X40UQlAj" + } + }, + { + "cell_type": "markdown", + "source": "## Your turn\n\nOne exercise, then Module 2.\n\nSwap in your own polysemy case below. Score `\"apple stock\"` against both senses and see which wins, then try a word your own domain overloads: *charge*, *bank*, *lead*, *plant*. Does the model pick the sense you meant?", + "metadata": { + "id": "SPpd7YIH0Cyu" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "pbOzJdwihqZy", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "693088a7" + }, + "execution_count": null, + "source": "candidates = [\n \"shares of a technology company\",\n \"a crisp red fruit\",\n]\n\nfor candidate in candidates:\n score = util.cos_sim(model.encode(\"apple stock\"), model.encode(candidate)).item()\n print(f\"{score:.3f} | 'apple stock' vs {candidate!r}\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "0.480 | 'apple stock' vs 'shares of a technology company'\n0.225 | 'apple stock' vs 'a crisp red fruit'\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## What's next: Module 2\n\n- What a vector is, and why it has hundreds of dimensions\n- How similarity works under the hood, and when it fails\n- Your first Qdrant collection: points, payloads, and your first query\n\n[Continue to Module 2](https://qdrant.tech/course/beginners/module-2/)", + "metadata": { + "id": "QToEDnzI44rO" + } + } + ] +} diff --git a/Beginner-course/Module2.ipynb b/Beginner-course/Module2.ipynb new file mode 100644 index 0000000..b4a8509 --- /dev/null +++ b/Beginner-course/Module2.ipynb @@ -0,0 +1,265 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "cells": [ + { + "cell_type": "markdown", + "source": "# Module 2: First Principles of Vector Search\n\n## What you will do\n\n1. Turn text into a vector and look at what a collection needs to know about it.\n2. Create your first Qdrant collection.\n3. Upsert points with vectors and payloads.\n4. Run a top-K query, then constrain it with a payload filter.\n5. See why the payload index has to exist before you ingest.\n\n**Tip:** every cell below is already run, so you can read it straight through. It uses Qdrant in local mode, so there is nothing to sign up for and no API key to paste.\n\nCompanion notebook to the [Module 2 lesson](https://qdrant.tech/course/beginners/module-2/).", + "metadata": { + "id": "ujOeHwdFcAef" + } + }, + { + "cell_type": "markdown", + "source": "## Setup\n\n`qdrant-client[fastembed]` bundles local embedding models. Passing a `models.Document` lets the client embed text for us before upload and at query time, so we never handle raw vectors by hand. The model is `all-MiniLM-L6-v2`, the same one from Module 1.", + "metadata": { + "id": "AZhnM6Jy8c1r" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "ihtYlKNxHddm" + }, + "execution_count": null, + "source": "!pip install -q \"qdrant-client[fastembed]\" ", + "outputs": [] + }, + { + "cell_type": "markdown", + "source": "## 1. What a Collection Needs to Know\n\nA collection is a container for points, and it is declared up front with two things it can never change silently: how many dimensions each vector has, and which distance metric compares them.\n\nThose two numbers are not stylistic. Get the size wrong and every upsert fails. Get the metric wrong and your results are quietly worse rather than broken.", + "metadata": { + "id": "QAtKCsXRpJG2" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "Tr8hzUjEcPVJ", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "e5b997df" + }, + "execution_count": null, + "source": "from qdrant_client import QdrantClient, models\n\nMODEL = \"sentence-transformers/all-MiniLM-L6-v2\"\nVECTOR_SIZE = 384 # fixed by the model\nDISTANCE = models.Distance.COSINE # the default for text embeddings\n\n# Local mode: an in-process Qdrant, ideal for notebooks and CI.\nclient = QdrantClient(\":memory:\")\n\n# On Qdrant Cloud you would swap in:\n# client = QdrantClient(url=\"https://YOUR-CLUSTER.cloud.qdrant.io:6333\",\n# api_key=\"YOUR_API_KEY\")\n\nclient.create_collection(\n collection_name=\"articles\",\n vectors_config=models.VectorParams(size=VECTOR_SIZE, distance=DISTANCE),\n)\n\nprint(client.get_collection(\"articles\").config.params.vectors)", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "size=384 distance= hnsw_config=None quantization_config=None on_disk=None datatype=None multivector_config=None\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 2. Index What You Will Filter, Before You Ingest\n\nThe collection is empty, and this is the moment to declare the payload fields you plan to filter on.\n\nThe ordering matters. Qdrant adds filter-aware edges to its vector index based on indexed payload values, and it can only add them for indexes that already exist when that index is built. Create a payload index after ingesting and you have to rebuild the vector index to get the benefit.\n\nLocal mode ignores payload indexes entirely, so the call below is a no-op here and will emit a warning. Write it anyway: it is the habit that transfers to a real server, and Module 4 goes into why.", + "metadata": { + "id": "DPdRO9YrxPbC" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "khdVipy2eBI2", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "dd465ef2" + }, + "execution_count": null, + "source": "import warnings\n\nwith warnings.catch_warnings():\n warnings.simplefilter(\"ignore\") # local mode warns that indexes do nothing here\n client.create_payload_index(\n collection_name=\"articles\",\n field_name=\"category\",\n field_schema=models.PayloadSchemaType.KEYWORD,\n )\n\nprint(\"payload index declared on 'category'\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "payload index declared on 'category'\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 3. Points: Vector Plus Payload\n\nA point is an id, one or more vectors, and a payload. The vector is what gets searched. The payload is everything you want back, or want to filter on, and it is ordinary JSON.\n\nNote what goes in the payload below. `title` is there to be returned to the user. `category` is there to be filtered. `published` is there to be sorted or range-filtered later. None of them affect similarity.", + "metadata": { + "id": "kooEKqazwJ7Q" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "FMPd3W2XyygN", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "01325002" + }, + "execution_count": null, + "source": "documents = [\n {\"id\": 1, \"title\": \"Car repair guide\", \"category\": \"automotive\", \"published\": 2023},\n {\"id\": 2, \"title\": \"Automobile maintenance 101\", \"category\": \"automotive\", \"published\": 2024},\n {\"id\": 3, \"title\": \"How to cook pasta\", \"category\": \"food\", \"published\": 2024},\n {\"id\": 4, \"title\": \"Best pizza in Chicago\", \"category\": \"food\", \"published\": 2022},\n]\n\nclient.upload_points(\n collection_name=\"articles\",\n points=[\n models.PointStruct(\n id=doc[\"id\"],\n vector=models.Document(text=doc[\"title\"], model=MODEL),\n payload=doc,\n )\n for doc in documents\n ],\n)\n\nprint(\"points in collection:\", client.count(\"articles\").count)", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "points in collection: 4\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 4. Your First Query\n\nSame model at query time as at ingestion time. This is the rule that breaks the most beginner pipelines: two different models produce two different vector spaces, and nothing will warn you, you will just get nonsense rankings.\n\nThe query below shares no words with any stored title.", + "metadata": { + "id": "gwb2Mjpvwh09" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "CDfgvqZkb6Gj", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "808fdb45" + }, + "execution_count": null, + "source": "results = client.query_points(\n collection_name=\"articles\",\n query=models.Document(text=\"fixing my vehicle\", model=MODEL),\n limit=4,\n)\n\nfor r in results.points:\n print(f\"{r.score:.3f} {r.payload['title']:30} ({r.payload['category']})\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "0.699 Car repair guide (automotive)\n0.565 Automobile maintenance 101 (automotive)\n0.045 Best pizza in Chicago (food)\n0.032 How to cook pasta (food)\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Both automotive articles come back above both food articles, and the query contained neither the word \"car\" nor \"automobile\". That is the Module 1 embedding doing its job, now with storage and ranking around it.\n\nAlso worth noticing: every result carries its payload back. You did not have to look anything up in a second database.", + "metadata": { + "id": "kVHWunY90XYT" + } + }, + { + "cell_type": "markdown", + "source": "## 5. Filtering\n\nA filter is a hard constraint, not a ranking hint. It is evaluated while the search runs, so excluded points never occupy a slot in your top-K.\n\nRun the same query twice, once unfiltered and once scoped to `food`, and watch the entire result set change rather than just reorder.", + "metadata": { + "id": "oGwbbrqQ7B69" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "7wnommM30DOX", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "1aecc72c" + }, + "execution_count": null, + "source": "from qdrant_client.models import Filter, FieldCondition, MatchValue\n\ndef search(text, query_filter=None, limit=4):\n return client.query_points(\n collection_name=\"articles\",\n query=models.Document(text=text, model=MODEL),\n query_filter=query_filter,\n limit=limit,\n ).points\n\nfood_only = Filter(must=[FieldCondition(key=\"category\", match=MatchValue(value=\"food\"))])\n\nprint(\"UNFILTERED 'fixing my vehicle':\")\nfor r in search(\"fixing my vehicle\"):\n print(f\" {r.score:.3f} {r.payload['title']:30} ({r.payload['category']})\")\n\nprint()\nprint(\"FILTERED to category=food:\")\nfor r in search(\"fixing my vehicle\", query_filter=food_only):\n print(f\" {r.score:.3f} {r.payload['title']:30} ({r.payload['category']})\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "UNFILTERED 'fixing my vehicle':\n 0.699 Car repair guide (automotive)\n 0.565 Automobile maintenance 101 (automotive)\n 0.045 Best pizza in Chicago (food)\n 0.032 How to cook pasta (food)\n\nFILTERED to category=food:\n 0.045 Best pizza in Chicago (food)\n 0.032 How to cook pasta (food)\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "The filtered run returns only food articles, and their scores are unchanged from the unfiltered run. The filter did not rescore anything. It removed candidates.\n\nThat distinction matters later: because filtering happens during retrieval rather than after it, a filtered search still returns a full top-K of valid results instead of a short list of leftovers.", + "metadata": { + "id": "uX8yy6Skhj4Y" + } + }, + { + "cell_type": "markdown", + "source": "## 6. Range Filters and Combining Conditions\n\n`must` is AND, `should` is OR, and `must_not` excludes. Numeric and date fields take ranges. They compose in one filter object.", + "metadata": { + "id": "jZ8OvIia8OG5" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "A2ZnpsoKqzi4", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "579e6e88" + }, + "execution_count": null, + "source": "from qdrant_client.models import Range\n\nrecent_automotive = Filter(\n must=[\n FieldCondition(key=\"category\", match=MatchValue(value=\"automotive\")),\n FieldCondition(key=\"published\", range=Range(gte=2024)),\n ]\n)\n\nprint(\"automotive AND published >= 2024:\")\nfor r in search(\"fixing my vehicle\", query_filter=recent_automotive):\n print(f\" {r.score:.3f} {r.payload['title']:30} ({r.payload['published']})\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "automotive AND published >= 2024:\n 0.565 Automobile maintenance 101 (2024)\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "One result, because only one point satisfies both conditions. Note that `published` was never indexed, and in local mode that costs nothing. On a real server it would work but scan, and on Qdrant Cloud with strict mode on it would be rejected outright. Every field you filter on needs an index.", + "metadata": { + "id": "GbBlaXkDSIuG" + } + }, + { + "cell_type": "markdown", + "source": "## 7. Similarity Under the Hood\n\nQdrant does not compare your query against every stored vector. It walks an HNSW graph, a layered structure where each layer is a sparser shortcut over the one below, so search starts coarse and refines.\n\nTwo consequences worth carrying forward.\n\nIt is **approximate**. HNSW trades a small amount of recall for a large amount of speed, so a top-K is very likely, not certainly, the true nearest neighbours.\n\nIt is **not always used**. Below a size threshold, scanning every vector is genuinely faster, so Qdrant does that instead. That is why a tiny notebook collection like this one returns exact results, and why timing measurements here tell you nothing about production.", + "metadata": { + "id": "IWgIprVFIV4B" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "LFFQCHD6R26q", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "8fd21713" + }, + "execution_count": null, + "source": "info = client.get_collection(\"articles\")\nprint(\"points: \", info.points_count)\nprint(\"vector size: \", info.config.params.vectors.size)\nprint(\"distance: \", info.config.params.vectors.distance)\nprint(\"hnsw m: \", info.config.hnsw_config.m)\nprint(\"hnsw ef_construct: \", info.config.hnsw_config.ef_construct)\nprint(\"full_scan_threshold:\", info.config.hnsw_config.full_scan_threshold, \"(KB)\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "points: 4\nvector size: 384\ndistance: Cosine\nhnsw m: 16\nhnsw ef_construct: 100\nfull_scan_threshold: 10000 (KB)\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "`m` is how many edges each node keeps, `ef_construct` is how hard the graph works while being built, and `full_scan_threshold` is the size below which Qdrant skips the graph. You will tune these in later work; for now the point is that they are collection-level settings you inherit by default.", + "metadata": { + "id": "ePW3jSOi27n7" + } + }, + { + "cell_type": "markdown", + "source": "## Your turn\n\nTwo changes to try.\n\nAdd a point in a third category and rerun the unfiltered query. Does it land where you expect relative to the existing four?\n\nThen change `DISTANCE` to `models.Distance.EUCLID`, recreate the collection, and re-ingest. Compare the scores to the cosine run. They will not be on the same scale, which is exactly why the metric is declared once per collection rather than per query.", + "metadata": { + "id": "yE9ZkAFvmtSb" + } + }, + { + "cell_type": "markdown", + "source": "## What's next: Module 3\n\n- Where dense-only search fails: exact codes, model numbers, and SKUs\n- Sparse vectors, BM25, and the inverted index\n- Hybrid search: running dense and sparse together and fusing the results\n\n[Continue to Module 3](https://qdrant.tech/course/beginners/module-3/)", + "metadata": { + "id": "IBbuMFd9W8gq" + } + } + ] +} diff --git a/Beginner-course/Module3.ipynb b/Beginner-course/Module3.ipynb new file mode 100644 index 0000000..d84bb0b --- /dev/null +++ b/Beginner-course/Module3.ipynb @@ -0,0 +1,393 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "cells": [ + { + "cell_type": "markdown", + "source": "# Module 3: Sparse vs Dense vs Hybrid Search\n\n## What you will do\n\n1. See the gap that dense-only search leaves on exact names and codes.\n2. Compare the two families of search, dense (meaning) and sparse (exact tokens).\n3. Build a hybrid collection and fuse both by rank.\n4. Run the dense vs hybrid experiment and watch the ranking change.\n5. Prove where a filter has to go in a hybrid query, by breaking it on purpose.\n\n**Tip for Colab:** run cells top to bottom. The first install downloads small embedding models, so it takes a minute.\n\nCompanion notebook to the [Module 3 lesson](https://qdrant.tech/course/beginners/module-3/).", + "metadata": { + "id": "ujOeHwdFcAef" + } + }, + { + "cell_type": "markdown", + "source": "## Setup\n\n`qdrant-client[fastembed]` bundles local embedding models. Passing a `models.Document` lets the client embed text for us before upload and at query time, so we never manage the vectors by hand.", + "metadata": { + "id": "AZhnM6Jy8c1r" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "ihtYlKNxHddm" + }, + "execution_count": null, + "source": "!pip install -q \"qdrant-client[fastembed]\" ", + "outputs": [] + }, + { + "cell_type": "markdown", + "source": "## 1. Where We Left Off\n\nIn Module 2 you built a full pipeline: raw text to vector to store to top-K query. Dense-only retrieval is great for semantic and contextual search, but it struggles on precise product names and model numbers.\n\nTake the query `iPhone 15`. The user wants exactly this product: no synonyms, no paraphrasing. Dense-only search tends to return the whole product line, because \"iPhone 14\", \"iPhone 15\", and \"iPhone 15 Pro Max\" sit close together in embedding space. IDs, codes, and specific model names need exact matching, not semantic neighborhood. That is the gap sparse search fills, and we will reproduce it live in Section 4.", + "metadata": { + "id": "QAtKCsXRpJG2" + } + }, + { + "cell_type": "markdown", + "source": "## 2. The Two Families of Search\n\n### Dense search (semantic)\n\nA dense vector has a small, fixed number of dimensions (for example 384), and every dimension holds a value. Two texts with similar meaning produce vectors that are close in space, even if they share no words. That is why `car repair` sits near `automobile maintenance`.\n\n### Sparse search (keyword based)\n\nSparse vectors are token based. Each dimension maps to a token, and only the tokens that actually appear carry a non-zero value. A vocabulary can be tens of thousands of tokens, but a given text activates only the handful it contains, so sparse vectors are stored as two parallel arrays: the `indices` of the non-zero dimensions and the `values` at those positions.\n\n```python\nsparse_vector = {\n \"indices\": [142, 9325, 44001], # token IDs: 'nike', 'pegasus', '40'\n \"values\": [2.3, 1.2, 0.8], # weight per token\n}\n```\n\nSparse similarity in Qdrant is always the dot product. There is no metric to choose, unlike the dense side where you pick Cosine, Dot, or Euclidean.\n\n#### Sparse models: BM25, SPLADE, miniCOIL\n\n| Model | How it assigns weights | Notes |\n|-------|------------------------|-------|\n| BM25 | Statistical: term frequency and inverse document frequency, no training | Classic, fast, interpretable. Scores tokens exactly as written. FastEmbed handle `Qdrant/bm25`. |\n| SPLADE | Neural: a transformer expands text with related terms and weights them | Captures some synonymy while staying sparse. More compute than BM25. |\n| miniCOIL | Neural, contextualized term weighting on BM25's exact vocabulary | Context aware exact match without full expansion cost. FastEmbed handle `Qdrant/minicoil-v1`. |\n\nminiCOIL is Qdrant's recommendation for new projects. We use BM25 here because it needs no model inference at all, which keeps this notebook fast and every score traceable by hand.\n\n#### How sparse is indexed\n\nQdrant uses an inverted index: for every token it keeps a posting list of every point where that token has a non-zero weight. A query only walks the posting lists for tokens it contains, skipping every point that shares none. HNSW (Module 2) is approximate, but the sparse index is exact.\n\n### Key insight\n\nDense = meaning. Sparse = exact matching. Neither is complete alone. Every real query carries both semantic intent (what the user means) and exact constraints (what the user needs precisely).", + "metadata": { + "id": "Tr8hzUjEcPVJ" + } + }, + { + "cell_type": "markdown", + "source": "## 3. Hybrid Search: Dense + Sparse\n\nHybrid search runs dense and sparse retrieval in the same request, then fuses the two ranked candidate lists into one result set: semantic understanding with exact-match precision. Payload filters constrain both retrievers while they run, which Section 7 covers.\n\n**Reciprocal Rank Fusion (RRF)** merges the dense list and the sparse list using each candidate's *position* in the two lists, not its raw score. A document ranked high by both retrievers rises to the top. Because it ignores raw scores, RRF is robust to the fact that dense and sparse scores live on completely different scales.", + "metadata": { + "id": "2tRKJC06DPdR" + } + }, + { + "cell_type": "markdown", + "source": "## 4. Setting Up Hybrid Search in Qdrant\n\n### Step 1: create a hybrid collection\n\nDeclare both a dense and a sparse vector config on one collection. Every point will carry both.\n\nTwo details in the cell below are easy to skip and expensive to skip. The sparse config needs `modifier=models.Modifier.IDF`, because BM25-style vectors store only term frequency and Qdrant applies the inverse-document-frequency half of the formula at query time. Without it you are not scoring BM25. And the payload index on `in_stock` is declared before any data is ingested, which is the ordering Module 4 explains.", + "metadata": { + "id": "O9YrxPbCkhdV" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "ipy2eBI2Y1rz", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "5ef22337" + }, + "execution_count": null, + "source": "import warnings\nfrom qdrant_client import QdrantClient, models\n\nDENSE_MODEL = \"sentence-transformers/all-MiniLM-L6-v2\" # 384-dim dense embeddings\nSPARSE_MODEL = \"Qdrant/bm25\" # exact-token sparse\n\nclient = QdrantClient(\":memory:\")\n\nclient.create_collection(\n collection_name=\"products\",\n vectors_config={\n \"dense\": models.VectorParams(size=384, distance=models.Distance.COSINE),\n },\n sparse_vectors_config={\n \"sparse\": models.SparseVectorParams(\n modifier=models.Modifier.IDF # required for BM25 scoring\n ),\n },\n)\n\nwith warnings.catch_warnings():\n warnings.simplefilter(\"ignore\") # local mode warns that indexes do nothing here\n client.create_payload_index(\n collection_name=\"products\",\n field_name=\"in_stock\",\n field_schema=models.PayloadSchemaType.BOOL,\n )\n\nprint(\"Hybrid collection 'products' created, with in_stock indexed.\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Hybrid collection 'products' created, with in_stock indexed.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### Step 2: insert points with both vectors\n\nEach point carries a dense embedding and a sparse vector. We pass a `models.Document` and name the model. The client embeds it locally with FastEmbed before upload. The catalog below mixes a phone product line (to reproduce the `iPhone 15` problem) with running shoes (for the Nike example) and includes an exact SKU.", + "metadata": { + "id": "KqazwJ7QFMPd" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "3W2XyygNdemk", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "5002150d" + }, + "execution_count": null, + "source": "catalog = [\n {\"id\": 1, \"name\": \"iPhone 14\", \"sku\": \"APL-IP14\", \"price\": 699, \"in_stock\": True},\n {\"id\": 2, \"name\": \"iPhone 15\", \"sku\": \"APL-IP15\", \"price\": 799, \"in_stock\": True},\n {\"id\": 3, \"name\": \"iPhone 15 Pro Max\", \"sku\": \"APL-IP15PM\", \"price\": 1199, \"in_stock\": True},\n {\"id\": 4, \"name\": \"iPhone 13 mini\", \"sku\": \"APL-IP13M\", \"price\": 599, \"in_stock\": False},\n {\"id\": 5, \"name\": \"Nike Pegasus 40 running shoes\", \"sku\": \"NK-PEG40\", \"price\": 130, \"in_stock\": True},\n {\"id\": 6, \"name\": \"Nike Pegasus 39 running shoes\", \"sku\": \"NK-PEG39\", \"price\": 110, \"in_stock\": True},\n {\"id\": 7, \"name\": \"Adidas Ultraboost running shoes\", \"sku\": \"AD-UB22\", \"price\": 180, \"in_stock\": True},\n {\"id\": 8, \"name\": \"Widget assembly part SKU-48291\", \"sku\": \"SKU-48291\", \"price\": 12, \"in_stock\": True},\n]\n\nclient.upload_points(\n collection_name=\"products\",\n points=[\n models.PointStruct(\n id=item[\"id\"],\n vector={\n \"dense\": models.Document(text=item[\"name\"], model=DENSE_MODEL),\n \"sparse\": models.Document(text=item[\"name\"] + \" \" + item[\"sku\"], model=SPARSE_MODEL),\n },\n payload=item,\n )\n for item in catalog\n ],\n)\nprint(\"Upserted\", client.count(\"products\").count, \"products.\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Upserted 8 products.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### Step 3: three ways to search the same collection\n\nA small helper prints results compactly. Then the same query runs three ways: dense-only, sparse-only, and hybrid.\n\nLook closely at `hybrid()`. The filter is passed into **each** `Prefetch`, not to `query_points` as a top-level `query_filter`. That placement is the whole lesson of Section 7, and we break it on purpose later to show what goes wrong.", + "metadata": { + "id": "Mjpvwh09CDfg" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "vqZkb6GjHbG8" + }, + "execution_count": null, + "source": "def show(title, points):\n print(title)\n for r in points:\n print(f\" [{r.score:.4f}] {r.payload['name']} (sku={r.payload['sku']}, in_stock={r.payload['in_stock']})\")\n print()\n\ndef dense_only(text, query_filter=None, limit=5):\n return client.query_points(\n \"products\",\n query=models.Document(text=text, model=DENSE_MODEL),\n using=\"dense\",\n query_filter=query_filter, # no prefetch here, so top level is correct\n limit=limit,\n ).points\n\ndef sparse_only(text, query_filter=None, limit=5):\n return client.query_points(\n \"products\",\n query=models.Document(text=text, model=SPARSE_MODEL),\n using=\"sparse\",\n query_filter=query_filter, # no prefetch here either\n limit=limit,\n ).points\n\ndef hybrid(text, query_filter=None, limit=5, fusion=\"rrf\"):\n # The filter goes INSIDE each prefetch, so both retrievers only ever\n # consider valid points. See Section 7 for why the top level is wrong here.\n fuse = (models.RrfQuery(rrf=models.Rrf()) if fusion == \"rrf\"\n else models.FusionQuery(fusion=models.Fusion.DBSF))\n return client.query_points(\n \"products\",\n prefetch=[\n models.Prefetch(query=models.Document(text=text, model=DENSE_MODEL),\n using=\"dense\", filter=query_filter, limit=20),\n models.Prefetch(query=models.Document(text=text, model=SPARSE_MODEL),\n using=\"sparse\", filter=query_filter, limit=20),\n ],\n query=fuse,\n limit=limit,\n ).points", + "outputs": [] + }, + { + "cell_type": "markdown", + "source": "The dense-only run reproduces the problem from Section 1: semantically similar phones cluster together, and the exact model the user typed does not reliably sit on top. Sparse-only, by contrast, locks onto the literal tokens.", + "metadata": { + "id": "1RqwkVHWunY9" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "0XYToGwbbrqQ", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "f7eff533" + }, + "execution_count": null, + "source": "show(\"DENSE-ONLY 'iPhone 15' (semantic neighborhood, exact model can drift):\", dense_only(\"iPhone 15\"))\nshow(\"SPARSE-ONLY 'iPhone 15' (exact tokens win):\", sparse_only(\"iPhone 15\"))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "DENSE-ONLY 'iPhone 15' (semantic neighborhood, exact model can drift):\n [1.0000] iPhone 15 (sku=APL-IP15, in_stock=True)\n [0.8760] iPhone 14 (sku=APL-IP14, in_stock=True)\n [0.8149] iPhone 15 Pro Max (sku=APL-IP15PM, in_stock=True)\n [0.6801] iPhone 13 mini (sku=APL-IP13M, in_stock=False)\n [0.1764] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n\nSPARSE-ONLY 'iPhone 15' (exact tokens win):\n [3.3050] iPhone 15 (sku=APL-IP15, in_stock=True)\n [3.2874] iPhone 15 Pro Max (sku=APL-IP15PM, in_stock=True)\n [1.1605] iPhone 14 (sku=APL-IP14, in_stock=True)\n [1.1574] iPhone 13 mini (sku=APL-IP13M, in_stock=False)\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "The clearest case is an exact code. A dense model has never really \"seen\" `SKU-48291` as a meaningful concept, so it drifts. Sparse matches the literal token exactly.", + "metadata": { + "id": "mmM30DOXfO4W" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "UDlWuX8yy6Sk", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "22ec2dfa" + }, + "execution_count": null, + "source": "show(\"DENSE-ONLY 'SKU-48291' (drifts):\", dense_only(\"SKU-48291\"))\nshow(\"SPARSE-ONLY 'SKU-48291' (exact hit):\", sparse_only(\"SKU-48291\"))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "DENSE-ONLY 'SKU-48291' (drifts):\n [0.4544] Widget assembly part SKU-48291 (sku=SKU-48291, in_stock=True)\n [0.2397] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.1948] Nike Pegasus 40 running shoes (sku=NK-PEG40, in_stock=True)\n [0.1403] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n [0.1266] iPhone 15 (sku=APL-IP15, in_stock=True)\n\nSPARSE-ONLY 'SKU-48291' (exact hit):\n [6.7829] Widget assembly part SKU-48291 (sku=SKU-48291, in_stock=True)\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Now hybrid. Both prefetches run as part of one request and return up to 20 candidates each. RRF merges them by rank, then `limit` takes the top results. Because each prefetch carries the filter, out-of-stock products never enter either candidate set.", + "metadata": { + "id": "vIia8OG5A2Zn" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "psoKqzi4vCK4", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "6e888072" + }, + "execution_count": null, + "source": "from qdrant_client.models import Filter, FieldCondition, MatchValue\n\nin_stock_filter = Filter(must=[FieldCondition(key=\"in_stock\", match=MatchValue(value=True))])\n\nshow(\"HYBRID 'Nike Pegasus 40 size 10' (semantic + exact, in stock only):\",\n hybrid(\"Nike Pegasus 40 size 10\", query_filter=in_stock_filter))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "HYBRID 'Nike Pegasus 40 size 10' (semantic + exact, in stock only):\n [1.0000] Nike Pegasus 40 running shoes (sku=NK-PEG40, in_stock=True)\n [0.6667] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.2500] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n [0.2000] iPhone 15 (sku=APL-IP15, in_stock=True)\n [0.1667] iPhone 14 (sku=APL-IP14, in_stock=True)\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### Try it: watch the ranking change\n\nThe experiment from the lesson, side by side. Compare where the exact target lands under dense-only versus hybrid.", + "metadata": { + "id": "aXkDSIuGIWgI" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "prVFIV4BLFFQ", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "787fbef4" + }, + "execution_count": null, + "source": "q = \"Nike Pegasus 40\"\nshow(f\"DENSE-ONLY '{q}':\", dense_only(q))\nshow(f\"HYBRID '{q}':\", hybrid(q))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "DENSE-ONLY 'Nike Pegasus 40':\n [0.8815] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.8713] Nike Pegasus 40 running shoes (sku=NK-PEG40, in_stock=True)\n [0.6043] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n [0.2403] iPhone 15 (sku=APL-IP15, in_stock=True)\n [0.2044] iPhone 14 (sku=APL-IP14, in_stock=True)\n\n" + }, + { + "output_type": "stream", + "name": "stdout", + "text": "HYBRID 'Nike Pegasus 40':\n [0.8333] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.8333] Nike Pegasus 40 running shoes (sku=NK-PEG40, in_stock=True)\n [0.2500] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n [0.2000] iPhone 15 (sku=APL-IP15, in_stock=True)\n [0.1667] iPhone 14 (sku=APL-IP14, in_stock=True)\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### Try it: put the filter in the wrong place\n\nNow the part worth doing yourself, because nothing raises an error when you get it wrong.\n\n`hybrid_wrong()` below is identical to `hybrid()` except the filter moves from inside the prefetches to a top-level `query_filter`. Then we mark `iPhone 15` out of stock and run both against an in-stock-only filter. The correct version drops it. The broken version should not.", + "metadata": { + "id": "I60ihBeoePW3" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "jSOi27n7yE9Z", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "268535b0" + }, + "execution_count": null, + "source": "def hybrid_wrong(text, query_filter=None, limit=5):\n \"\"\"Same query, filter at the top level instead of inside each prefetch.\"\"\"\n return client.query_points(\n \"products\",\n prefetch=[\n models.Prefetch(query=models.Document(text=text, model=DENSE_MODEL),\n using=\"dense\", limit=20),\n models.Prefetch(query=models.Document(text=text, model=SPARSE_MODEL),\n using=\"sparse\", limit=20),\n ],\n query=models.RrfQuery(rrf=models.Rrf()),\n query_filter=query_filter, # too late: prefetches have already run\n limit=limit,\n ).points\n\nclient.set_payload(\"products\", payload={\"in_stock\": False}, points=[2]) # iPhone 15\n\nshow(\"CORRECT filter inside each prefetch (iPhone 15 is gone):\",\n hybrid(\"iPhone 15\", query_filter=in_stock_filter))\nshow(\"BROKEN filter at the top level (iPhone 15 comes back):\",\n hybrid_wrong(\"iPhone 15\", query_filter=in_stock_filter))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "CORRECT filter inside each prefetch (iPhone 15 is gone):\n [0.8333] iPhone 14 (sku=APL-IP14, in_stock=True)\n [0.8333] iPhone 15 Pro Max (sku=APL-IP15PM, in_stock=True)\n [0.2500] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.2000] Nike Pegasus 40 running shoes (sku=NK-PEG40, in_stock=True)\n [0.1667] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n\n" + }, + { + "output_type": "stream", + "name": "stdout", + "text": "BROKEN filter at the top level (iPhone 15 comes back):\n [1.0000] iPhone 15 (sku=APL-IP15, in_stock=False)\n [0.5833] iPhone 14 (sku=APL-IP14, in_stock=True)\n [0.5833] iPhone 15 Pro Max (sku=APL-IP15PM, in_stock=True)\n [0.4000] iPhone 13 mini (sku=APL-IP13M, in_stock=False)\n [0.1667] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "There it is. Same filter, same query, one line of difference, and the broken version returns a product that is out of stock, without a warning or an error.\n\nThe reason is execution order. Once a query has prefetches, Qdrant runs those first and applies the main query to their results. A top-level filter therefore never reaches the retrievers: each one searches the whole catalog, and the filter only trims the already-fused list at the end. That is post-filtering, and with a selective filter it can leave you with nothing at all.\n\nThe rule is simple. No prefetch, use `query_filter`. Prefetch, put the filter in every prefetch.", + "metadata": { + "id": "IBbuMFd9W8gq" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "cWqiA4Yqj4JR", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "10b61fac" + }, + "execution_count": null, + "source": "client.set_payload(\"products\", payload={\"in_stock\": True}, points=[2]) # restore iPhone 15\nprint(\"iPhone 15 back in stock:\", client.retrieve(\"products\", ids=[2])[0].payload[\"in_stock\"])", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "iPhone 15 back in stock: True\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 5. Fusion Strategies\n\nOnce both retrievers return candidates, a fusion algorithm merges them into one ranked list. Qdrant supports two.\n\n| Strategy | How it works | When to use it |\n|----------|--------------|----------------|\n| RRF (Reciprocal Rank Fusion) | Combines rankings only, ignores raw scores. Robust, hard to game. | Default. Safe when dense and sparse score scales differ. |\n| DBSF (Distribution-Based Score Fusion) | Normalizes score distributions before merging. Sensitive to relative score gaps. | When score gaps meaningfully encode relevance and both retrievers are well calibrated. |\n\nStart with unweighted RRF, because dense and sparse scores live on different scales and raw-score fusion without normalization is unreliable. RRF also accepts a `k` constant and per-prefetch `weights`, so you can favour the stronger retriever once you have an evaluation set to tune against. Move to DBSF or tuned weights only after measuring, and tune on a different split from the one you measure on.\n\nBoth strategies run below on the same query so you can compare.", + "metadata": { + "id": "f1e1CvI5qiGo" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "gkdmtsVrFlvb", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "40b837e1" + }, + "execution_count": null, + "source": "show(\"RRF fusion:\", hybrid(\"Nike Pegasus 40 size 10\", fusion=\"rrf\"))\nshow(\"DBSF fusion:\", hybrid(\"Nike Pegasus 40 size 10\", fusion=\"dbsf\"))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "RRF fusion:\n [1.0000] Nike Pegasus 40 running shoes (sku=NK-PEG40, in_stock=True)\n [0.6667] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.2500] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n [0.2000] iPhone 15 (sku=APL-IP15, in_stock=True)\n [0.1667] iPhone 14 (sku=APL-IP14, in_stock=True)\n\n" + }, + { + "output_type": "stream", + "name": "stdout", + "text": "DBSF fusion:\n [1.3562] Nike Pegasus 40 running shoes (sku=NK-PEG40, in_stock=True)\n [1.1153] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.6049] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n [0.4129] iPhone 15 (sku=APL-IP15, in_stock=True)\n [0.4019] iPhone 14 (sku=APL-IP14, in_stock=True)\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 6. Beyond Text: Multimodal Search\n\nThe same primitive, embed data then store as a vector then search by similarity, applies to any modality: images (CLIP, SigLIP), video frames, audio fingerprints, or text. Qdrant stores whatever vectors your embedding model produces, and the retrieval mechanics are identical.\n\n**Data to Embedding Model to Vector to Qdrant.** The modality changes, the system does not.\n\n### Named vectors\n\nWhen two representations must be searchable together, store them as named vectors on the same point, then query against whichever one you want. Below we demonstrate the mechanic with two text views of each product, a short `title` and a longer `description`, so it runs with no heavy image models. In a real multimodal system you would swap the `description` model for an image encoder such as CLIP; the collection and query code stay the same shape.", + "metadata": { + "id": "YAEZyFQ8vZRN" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "zvdieTpkf01P", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "4347274f" + }, + "execution_count": null, + "source": "client.create_collection(\n collection_name=\"catalog\",\n vectors_config={\n \"title\": models.VectorParams(size=384, distance=models.Distance.COSINE),\n \"description\": models.VectorParams(size=384, distance=models.Distance.COSINE),\n },\n)\n\nclient.upload_points(\n collection_name=\"catalog\",\n points=[\n models.PointStruct(\n id=42,\n vector={\n \"title\": models.Document(text=\"Red Nike running shoe\", model=DENSE_MODEL),\n \"description\": models.Document(text=\"Lightweight breathable trainer for road running in bright red\", model=DENSE_MODEL),\n },\n payload={\"sku\": \"NK-RED-10\", \"price\": 120},\n ),\n models.PointStruct(\n id=43,\n vector={\n \"title\": models.Document(text=\"Blue hiking boot\", model=DENSE_MODEL),\n \"description\": models.Document(text=\"Waterproof ankle support boot for rough mountain trails\", model=DENSE_MODEL),\n },\n payload={\"sku\": \"HK-BLU-9\", \"price\": 150},\n ),\n ],\n)\n\n# Query one named vector; the other is simply not searched on this call.\nres = client.query_points(\n \"catalog\",\n query=models.Document(text=\"shoe for running on pavement\", model=DENSE_MODEL),\n using=\"description\",\n limit=2,\n).points\nfor r in res:\n print(f\"[{r.score:.4f}] {r.payload['sku']}\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "[0.5492] HK-BLU-9\n[0.4119] NK-RED-10\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Each named vector is its own space, and vectors from different models are not comparable. Swapping `description` for a CLIP image encoder would look like this, using the same collection and query shape:\n\n```python\n# ingestion\n\"image\": embed_image(product_photo) # CLIP get_image_features\n\n# querying by text against those image vectors\nclient.query_points(\"catalog\", query=embed_text_with_clip(\"red running shoe\"),\n using=\"image\", limit=10)\n```\n\nThe important part is that the query text must go through **CLIP's text encoder**, not the sentence transformer, or it lands in a different space and the scores are meaningless. Module 5 builds this out properly.", + "metadata": { + "id": "8Hp7twaxDFmF" + } + }, + { + "cell_type": "markdown", + "source": "## 7. Filtering Works with Any Retrieval Method\n\nPayload filters are not a hybrid-only feature. The same conditions apply to dense-only, sparse-only, or hybrid retrieval, and they are evaluated as hard constraints *while* the search runs, not as a separate step afterward. Because out-of-scope points never take a slot in your top-K, results stay both relevant and valid: in stock, within permissions, within a date range.\n\nWhat changes between the three is *where* the filter goes, which is what the broken example earlier demonstrated. Dense-only and sparse-only have no prefetch, so `query_filter` is correct. Hybrid has prefetches, so the filter belongs in each one.\n\nThe three calls below apply the identical filter to all three retrieval methods.", + "metadata": { + "id": "aqfycbsoKGUO" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "S2yu9jSNcZ3M", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "bc288dcd" + }, + "execution_count": null, + "source": "show(\"DENSE + filter:\", dense_only(\"iPhone\", query_filter=in_stock_filter))\nshow(\"SPARSE + filter:\", sparse_only(\"iPhone\", query_filter=in_stock_filter))\nshow(\"HYBRID + filter:\", hybrid(\"iPhone\", query_filter=in_stock_filter))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "DENSE + filter:\n [0.8036] iPhone 15 (sku=APL-IP15, in_stock=True)\n [0.7914] iPhone 14 (sku=APL-IP14, in_stock=True)\n [0.5714] iPhone 15 Pro Max (sku=APL-IP15PM, in_stock=True)\n [0.1663] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.1396] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n\nSPARSE + filter:\n [1.1605] iPhone 15 (sku=APL-IP15, in_stock=True)\n [1.1605] iPhone 14 (sku=APL-IP14, in_stock=True)\n [1.1543] iPhone 15 Pro Max (sku=APL-IP15PM, in_stock=True)\n\nHYBRID + filter:\n [1.0000] iPhone 15 (sku=APL-IP15, in_stock=True)\n [0.6667] iPhone 14 (sku=APL-IP14, in_stock=True)\n [0.5000] iPhone 15 Pro Max (sku=APL-IP15PM, in_stock=True)\n [0.2000] Nike Pegasus 39 running shoes (sku=NK-PEG39, in_stock=True)\n [0.1667] Adidas Ultraboost running shoes (sku=AD-UB22, in_stock=True)\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## What's next: Module 4\n\n- The five layers of a vector search stack\n- A worked design: a multilingual news search system, decision by decision\n- Filtering in production: how the query planner picks a strategy, and multitenancy\n- The production RAG pipeline, and deployment options\n\n[Continue to Module 4](https://qdrant.tech/course/beginners/module-4/)", + "metadata": { + "id": "K3QQobiwgZIM" + } + } + ] +} diff --git a/Beginner-course/Module4.ipynb b/Beginner-course/Module4.ipynb new file mode 100644 index 0000000..4f53ac2 --- /dev/null +++ b/Beginner-course/Module4.ipynb @@ -0,0 +1,443 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "cells": [ + { + "cell_type": "markdown", + "source": "# Module 4: Designing a Vector Search System\n\nVector search systems rarely break at the distance metric. They break because nobody decided what the payload had to hold until a few million documents were already ingested. This module is about the decisions that are cheap to make now and expensive to revisit later.\n\n## What you will do\n\n1. See the five layers of the stack as a mental checklist.\n2. Design and build a multilingual news search system by answering five questions.\n3. Practice filtering the way production systems do it, including the mistake that fails silently.\n4. Assemble a production RAG retrieval pipeline.\n\n**Tip for Colab:** run the cells top to bottom. The first install cell takes a minute because it downloads the embedding models.\n\nCompanion notebook to the [Module 4 lesson](https://qdrant.tech/course/beginners/module-4/).", + "metadata": { + "id": "ujOeHwdFcAef" + } + }, + { + "cell_type": "markdown", + "source": "## Setup\n\nWe install `qdrant-client` with the FastEmbed extra. FastEmbed gives us local, CPU friendly embedding models so the notebook produces real vectors rather than random numbers.", + "metadata": { + "id": "AZhnM6Jy8c1r" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "ihtYlKNxHddm" + }, + "execution_count": null, + "source": "!pip install -q \"qdrant-client[fastembed]\" ", + "outputs": [] + }, + { + "cell_type": "markdown", + "source": "## 1. The Layers of the Stack\n\nEvery vector search system, from a notebook prototype to a deployment serving millions of queries, is built from the same five layers. When something is slow, wrong, or expensive, the first diagnostic question is always the same: which layer is the problem in?\n\n- **Query layer**: turns user intent into a search. Embedding the query, choosing dense vs sparse vs hybrid, fusing results, setting limits.\n- **Indexing layer**: makes that search fast, through the HNSW graph for vectors and payload indexes for the fields you filter on. Get this wrong and results are usually still correct, just slow.\n- **Storage layer**: holds the points themselves, the vectors, payloads, and IDs, across memory and disk.\n- **Knowledge layer**: shapes the data before it ever becomes a vector: chunking, embedding model choice, payload schema. No amount of tuning elsewhere fixes a mistake here. Garbage in, garbage retrieved.\n- **Distribution layer**: spreads the system across more than one machine: sharding, replication, multi node clusters. You will not need it on day one.\n\n### Key insight\n\nEvery design decision belongs to a layer. \"Add a payload index\" is an indexing decision. \"Switch to a multilingual embedding model\" is a knowledge decision. \"Move to three nodes\" is a distribution decision. Once you sort decisions into layers, intimidating architecture diagrams become checklists.", + "metadata": { + "id": "QAtKCsXRpJG2" + } + }, + { + "cell_type": "markdown", + "source": "## 2. Worked Example: Designing a Multilingual News Search System\n\nHere is the brief, the kind you would get on a real project:\n\n> Analysts at a research firm need to search global news that arrives continuously, in many languages. They ask in English (\"port congestion in Southeast Asia\") but expect matches from any language, and they scope every search by country, topic, date range, and source. Some queries name one specific thing, a company ticker or a ship name, that has to match exactly.\n\nFive questions turn it into a design, and each answer is a decision that belongs to one layer.\n\n### Question 1: What do the queries look like?\n\nBoth natural language and exact tokens. \"Port congestion in Southeast Asia\" is semantic intent, which is dense territory. \"MAERSK-B.CO delisting\" is an exact token that dense search will blur into neighboring tickers. That is the SKU problem from Module 3.\n\n**Decision:** hybrid from the start. Two named vectors on every point, one dense and one sparse, with their rankings fused at query time. This is a query layer decision.", + "metadata": { + "id": "Tr8hzUjEcPVJ" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "2tRKJC06DPdR", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "afd46a07" + }, + "execution_count": null, + "source": "import warnings\nfrom qdrant_client import QdrantClient, models\nfrom fastembed import TextEmbedding, SparseTextEmbedding\n\n# Local mode: an in-process Qdrant, ideal for prototyping, notebooks and teaching.\n# It is a Python reimplementation rather than the engine: search is exact instead\n# of approximate, and payload indexes have no effect. Everything we write below is\n# still the right shape for a real server.\nclient = QdrantClient(\":memory:\")\n\n# Knowledge layer decision made concrete: a MULTILINGUAL dense model, so an\n# English query can retrieve a Japanese or Vietnamese article with no translation.\n# We use a small multilingual model to keep the Colab download fast. Production\n# would reach for something like multilingual-e5-large.\ndense_model = TextEmbedding(\"sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2\")\n\n# A sparse model gives us exact-token matching alongside dense semantics.\n# BM25 is lightweight and language agnostic, a good fit for mixed-language news.\nsparse_model = SparseTextEmbedding(\"Qdrant/bm25\")\n\n# Ask the model for its dimensionality rather than hardcoding it, so the\n# collection config can never drift out of sync with the model.\nDENSE_DIM = len(list(dense_model.embed([\"hello\"]))[0])\nprint(\"Dense vector dimension:\", DENSE_DIM)", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Dense vector dimension: 384\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Now create the collection with **named vectors**: one dense, one sparse. Naming them lets a single point carry both, and lets a single query fuse both.\n\nThe sparse config needs `modifier=models.Modifier.IDF`. BM25-style sparse vectors store only term frequency, and Qdrant applies the inverse-document-frequency half of the formula at query time. Leave the modifier off and you are not scoring BM25, you are scoring raw term counts.", + "metadata": { + "id": "khdVipy2eBI2" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "Y1rzw27jkooE", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "940659fb" + }, + "execution_count": null, + "source": "client.create_collection(\n collection_name=\"news\",\n vectors_config={\n \"dense\": models.VectorParams(size=DENSE_DIM, distance=models.Distance.COSINE),\n },\n sparse_vectors_config={\n \"sparse\": models.SparseVectorParams(\n modifier=models.Modifier.IDF # required for BM25 scoring\n ),\n },\n)\nprint(\"Collection 'news' created.\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Collection 'news' created.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### Question 2: What must the system filter on?\n\nFrom the brief: country, topic, date range, and source. These are hard rules, not similarity signals. An analyst scoping to \"Vietnam, last seven days\" means exactly that. Hard rules go in the payload, and every field we filter on gets a payload index.\n\nThe schema, decided now, before ingestion:\n\n```yaml\npayload:\n country: string # indexed\n language: string # indexed\n topic: string # indexed\n source: string # indexed\n tenant_id: string # indexed, and marked as a tenant field\n published_at: datetime # indexed\n summary: string # not indexed: returned, never filtered\n```\n\nCreate the indexes now too, in the same setup step, before a single point is uploaded. This ordering is not a style preference. Qdrant extends the HNSW graph with extra edges derived from indexed payload values, and it can only add those edges for indexes that already exist when the graph is built. Create a payload index after ingesting and you have to rebuild the HNSW index to get any benefit from it.\n\nNote `tenant_id` is in this list even though multitenancy does not come up until Section 3. That is the point: it is nearly free now and expensive later.", + "metadata": { + "id": "FMPd3W2XyygN" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "demkvdajgwb2", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "924551df" + }, + "execution_count": null, + "source": "with warnings.catch_warnings():\n warnings.simplefilter(\"ignore\") # local mode warns that indexes do nothing here\n\n for field in [\"country\", \"language\", \"topic\", \"source\"]:\n client.create_payload_index(\n collection_name=\"news\",\n field_name=field,\n field_schema=models.PayloadSchemaType.KEYWORD,\n )\n\n client.create_payload_index(\n collection_name=\"news\",\n field_name=\"published_at\",\n field_schema=models.PayloadSchemaType.DATETIME,\n )\n\n # is_tenant tells Qdrant this field identifies tenants, so it can keep each\n # tenant's data close together on disk. Supported for keyword and uuid.\n client.create_payload_index(\n collection_name=\"news\",\n field_name=\"tenant_id\",\n field_schema=models.KeywordIndexParams(\n type=models.KeywordIndexType.KEYWORD,\n is_tenant=True,\n ),\n )\n\nprint(\"Payload indexes created before ingestion: country, language, topic, source, published_at, tenant_id.\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Payload indexes created before ingestion: country, language, topic, source, published_at, tenant_id.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Skipping an index costs more than a slow scan. The payload index is also what lets Qdrant estimate how many points a filter will match, and that estimate is what the query planner uses to choose a strategy at all. Without it the planner is guessing, and it can fall back to comparing the query against every vector in the collection.\n\nBecause the query still returns results, just slowly, a missing index can sit in production unnoticed for months. Strict mode closes that gap: set `unindexed_filtering_retrieve` to `false` and Qdrant rejects any query that filters on an unindexed field instead of quietly degrading. Qdrant Cloud applies this by default.", + "metadata": { + "id": "CDfgvqZkb6Gj" + } + }, + { + "cell_type": "markdown", + "source": "### Question 3: What is the workload shape?\n\nMillions of articles, text only for now, arriving continuously. Analysts expect this morning's news to be searchable this morning.\n\n**Decisions:** one collection, and continuous upserts rather than periodic rebuilds. Two things still need designing. The initial backfill of millions of articles is a bulk load, not a stream, so batch the upserts and consider disabling indexing for the duration so the optimizer builds the graph once at the end. And Qdrant does index as it ingests, but not instantly: the optimizer builds an HNSW index for a segment only once that segment passes the indexing threshold, and unindexed segments are served by full scan in the meantime.\n\nHere is a small, multilingual sample that stands in for that stream. Note the mix of languages and the exact tokens (a ticker, a ship name) hiding in the text.", + "metadata": { + "id": "HbG81RqwkVHW" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "unY90XYToGwb", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "044bf7ef" + }, + "execution_count": null, + "source": "articles = [\n {\n \"country\": \"VN\", \"language\": \"vi\", \"topic\": \"shipping\", \"source\": \"reuters\",\n \"published_at\": \"2026-07-20T08:00:00Z\", \"tenant_id\": \"asia-desk\",\n \"summary\": \"Tac nghen cang tai Hai Phong khi luong hang tang manh.\",\n \"text\": \"Cang Hai Phong o Viet Nam bi tac nghen nghiem trong trong tuan nay khi luong container tang vot, gay cham tre cho tau thuyen.\",\n },\n {\n \"country\": \"JP\", \"language\": \"ja\", \"topic\": \"shipping\", \"source\": \"nikkei\",\n \"published_at\": \"2026-07-21T09:30:00Z\", \"tenant_id\": \"asia-desk\",\n \"summary\": \"東南アジアの港湾混雑が輸送を遅らせている。\",\n \"text\": \"東南アジアの主要港で混雑が深刻化し、コンテナ船の到着が遅れている。特にベトナムとタイの港で影響が大きい。\",\n },\n {\n \"country\": \"CN\", \"language\": \"zh\", \"topic\": \"logistics\", \"source\": \"caixin\",\n \"published_at\": \"2026-07-19T11:00:00Z\", \"tenant_id\": \"asia-desk\",\n \"summary\": \"上海港物流吞吐量创新高。\",\n \"text\": \"上海港本月物流吞吐量创下新高，港口运营商正在扩大堆场以缓解拥堵压力。\",\n },\n {\n \"country\": \"DK\", \"language\": \"en\", \"topic\": \"markets\", \"source\": \"reuters\",\n \"published_at\": \"2026-07-22T07:15:00Z\", \"tenant_id\": \"europe-desk\",\n \"summary\": \"Maersk shares move on delisting speculation.\",\n \"text\": \"Shares tied to the ticker MAERSK-B.CO moved sharply amid speculation about a possible delisting of a subsidiary vehicle.\",\n },\n {\n \"country\": \"SG\", \"language\": \"en\", \"topic\": \"shipping\", \"source\": \"straits-times\",\n \"published_at\": \"2026-07-22T10:45:00Z\", \"tenant_id\": \"asia-desk\",\n \"summary\": \"Vessel Ever Given reroutes through Singapore.\",\n \"text\": \"The container ship Ever Given was rerouted through the Port of Singapore this week to avoid congestion further north.\",\n },\n {\n \"country\": \"US\", \"language\": \"en\", \"topic\": \"markets\", \"source\": \"press-release-wire\",\n \"published_at\": \"2026-06-30T12:00:00Z\", \"tenant_id\": \"americas-desk\",\n \"summary\": \"Company announces quarterly logistics earnings.\",\n \"text\": \"A logistics operator announced quarterly earnings, citing steady demand across North American freight corridors.\",\n },\n {\n \"country\": \"TH\", \"language\": \"th\", \"topic\": \"shipping\", \"source\": \"bangkok-post\",\n \"published_at\": \"2026-07-18T06:20:00Z\", \"tenant_id\": \"asia-desk\",\n \"summary\": \"ความแออัดที่ท่าเรือแหลมฉบังเพิ่มขึ้น\",\n \"text\": \"ท่าเรือแหลมฉบังของไทยเผชิญความแออัดมากขึ้นเนื่องจากปริมาณตู้คอนเทนเนอร์ที่เพิ่มสูงขึ้นในสัปดาห์นี้\",\n },\n {\n \"country\": \"DE\", \"language\": \"de\", \"topic\": \"logistics\", \"source\": \"handelsblatt\",\n \"published_at\": \"2026-07-15T14:10:00Z\", \"tenant_id\": \"europe-desk\",\n \"summary\": \"Hamburger Hafen meldet Verzoegerungen.\",\n \"text\": \"Der Hamburger Hafen meldet Verzoegerungen bei der Abfertigung, da die Zahl der ankommenden Schiffe deutlich gestiegen ist.\",\n },\n]\nprint(f\"{len(articles)} sample articles across {len(set(a['language'] for a in articles))} languages.\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "8 sample articles across 6 languages.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Embed each article into both a dense and a sparse vector, and upsert them as points. In production this same loop runs continuously as articles arrive.", + "metadata": { + "id": "7wnommM30DOX" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "fO4WUDlWuX8y", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "6fb222ec" + }, + "execution_count": null, + "source": "def to_sparse_vector(text, is_query=False):\n \"\"\"Convert text to a Qdrant SparseVector using BM25.\n BM25 scores queries and documents slightly differently, so we use\n query_embed for queries and embed for documents.\"\"\"\n emb = next(sparse_model.query_embed(text)) if is_query else next(sparse_model.embed([text]))\n return models.SparseVector(indices=emb.indices.tolist(), values=emb.values.tolist())\n\npoints = []\nfor i, art in enumerate(articles):\n dense_vec = next(dense_model.embed([art[\"text\"]])).tolist()\n sparse_vec = to_sparse_vector(art[\"text\"])\n points.append(\n models.PointStruct(\n id=i,\n vector={\"dense\": dense_vec, \"sparse\": sparse_vec},\n payload=art,\n )\n )\n\nclient.upsert(collection_name=\"news\", points=points)\nprint(f\"Upserted {len(points)} points. Collection count:\", client.count(\"news\").count)", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Upserted 8 points. Collection count: 8\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### Question 4: What does the retrieval pipeline look like?\n\nStart with the simplest pipeline that fits the query analysis: hybrid (from Question 1) plus filters (from Question 2), fused by rank. No reranker yet. Add complexity when data proves it is needed, never in advance.\n\nThe `query_points` call below does it all in one round trip:\n\n- Two `Prefetch` branches, one per named vector, each pulling 50 candidates.\n- `RrfQuery` to merge the two candidate lists into one ranking.\n- The filter passed **into each prefetch**, so both retrievers only ever consider valid points.\n\nThat last point is the one to slow down on, and Section 3 breaks it on purpose to show why.\n\nNotice the payoff of the multilingual model: we query in **English** and retrieve articles written in Japanese, Vietnamese, Thai, and more, with no translation step.", + "metadata": { + "id": "jZ8OvIia8OG5" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "A2ZnpsoKqzi4", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "579e6e88" + }, + "execution_count": null, + "source": "def search(query_text, query_filter=None, limit=5):\n \"\"\"Hybrid search: dense + sparse prefetch, fused by rank, filter in each branch.\"\"\"\n dense_q = next(dense_model.query_embed(query_text)).tolist()\n sparse_q = to_sparse_vector(query_text, is_query=True)\n response = client.query_points(\n collection_name=\"news\",\n prefetch=[\n models.Prefetch(query=dense_q, using=\"dense\",\n filter=query_filter, limit=50),\n models.Prefetch(query=sparse_q, using=\"sparse\",\n filter=query_filter, limit=50),\n ],\n query=models.RrfQuery(rrf=models.Rrf()),\n limit=limit,\n )\n return response.points\n\ndef show(results):\n if not results:\n print(\" (no results)\")\n for r in results:\n p = r.payload\n print(f\"[{r.score:.4f}] {p['country']}/{p['language']:<2} {p['topic']:<9} | {p['summary']}\")\n\nprint(\"Query (English): 'port congestion in Southeast Asia'\\n\")\nshow(search(\"port congestion in Southeast Asia\"))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Query (English): 'port congestion in Southeast Asia'\n\n[0.7500] SG/en shipping | Vessel Ever Given reroutes through Singapore.\n[0.5000] JP/ja shipping | 東南アジアの港湾混雑が輸送を遅らせている。\n[0.3333] TH/th shipping | ความแออัดที่ท่าเรือแหลมฉบังเพิ่มขึ้น\n[0.2000] CN/zh logistics | 上海港物流吞吐量创新高。\n[0.1667] DE/de logistics | Hamburger Hafen meldet Verzoegerungen.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Now the exact-token case. A dense-only search would blur `MAERSK-B.CO` into semantically similar finance text. The sparse half of the hybrid locks onto the literal token, and rank fusion pushes the correct article to the top. This is why we chose hybrid from the start.", + "metadata": { + "id": "GbBlaXkDSIuG" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "IWgIprVFIV4B", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "988b787f" + }, + "execution_count": null, + "source": "print(\"Query (exact token): 'MAERSK-B.CO delisting'\\n\")\nshow(search(\"MAERSK-B.CO delisting\"))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Query (exact token): 'MAERSK-B.CO delisting'\n\n[1.0000] DK/en markets | Maersk shares move on delisting speculation.\n[0.3333] DE/de logistics | Hamburger Hafen meldet Verzoegerungen.\n[0.2500] VN/vi shipping | Tac nghen cang tai Hai Phong khi luong hang tang manh.\n[0.2000] SG/en shipping | Vessel Ever Given reroutes through Singapore.\n[0.1667] TH/th shipping | ความแออัดที่ท่าเรือแหลมฉบังเพิ่มขึ้น\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Reciprocal Rank Fusion is the right default here because it works from ranks rather than raw scores, so it does not care that cosine similarity and BM25 live on different scales. Two alternatives are worth knowing once you can measure quality: weighted RRF, which favours the stronger retriever via `weights` on `models.Rrf()`, and Distribution-Based Score Fusion, `models.FusionQuery(fusion=models.Fusion.DBSF)`, which normalizes each retriever's score distribution instead. Neither reliably beats the other, so choose with an evaluation set rather than by reputation.", + "metadata": { + "id": "R26qI60ihBeo" + } + }, + { + "cell_type": "markdown", + "source": "### Question 5: What are the deployment constraints?\n\nA research firm with a small engineering team, no data residency restrictions, and a \"please do not page us at night\" budget points to a **managed** deployment. The key idea: the design and the deployment mode are independent decisions. Everything above runs unchanged whether the backend is this in-process client, a Docker container, or Qdrant Cloud.\n\n### The design on one page\n\n| Question | Answer for This System | Layer |\n|----------|------------------------|-------|\n| Query type | Mixed semantic + exact, so hybrid with Reciprocal Rank Fusion | Query |\n| Filter scope | country, topic, source, date, indexed before ingestion | Knowledge, indexing |\n| Workload shape | Millions of text chunks: bulk backfill, then continuous ingestion | Storage, knowledge |\n| Pipeline | Hybrid + per-prefetch filters, multilingual model, no reranker yet | Query, knowledge |\n| Deployment | Managed, design independent of the choice | Distribution |\n\n### Key insight\n\nEvery piece of this design already appeared in an earlier module: the Module 2 pipeline, the Module 3 hybrid pattern, and a payload schema. What is new is the order of operations. You decided what to filter on and created the indexes before the first point went in. That ordering is the difference between a system that scales and one that gets re-ingested three times.", + "metadata": { + "id": "ePW3jSOi27n7" + } + }, + { + "cell_type": "markdown", + "source": "## 3. Filtering\n\nFiltering is the feature that decides whether your results are **correct**, not an accessory to similarity.\n\n### How Qdrant combines filters with vector search\n\nThe naive approach is post-filtering: retrieve the top K by similarity, then discard whatever fails the filter. With a selective filter, say one country out of 200, the top K can contain zero valid results, and there is no K that guarantees correctness.\n\nQdrant does not work that way. A query planner chooses a strategy for each segment, based on the estimated cardinality of the filter and which payload indexes exist:\n\n- When the filter matches a large share of the collection, Qdrant walks the HNSW graph as usual and skips points that fail the filter during traversal.\n- When the filter matches very few points, it can skip the graph entirely and retrieve through the payload index, which is cheaper at that selectivity.\n- When a segment is small enough, a full scan wins outright.\n\nThe middle ground is the hard case, because a strict filter can disconnect the HNSW graph and leave relevant points unreachable. Qdrant handles it by extending the graph with additional edges derived from indexed payload values, which is why the indexes had to exist before ingestion. Those extra edges are added per index, not per combination of indexes, so two or more highly selective filters can still reach a disconnected component. The ACORN algorithm (v1.16) exists for that case.\n\nSo: filtering on every query is safe and fast, as long as every filtered field is indexed and the indexes existed before ingestion. It is not free, and combinations of highly selective filters deserve measurement rather than assumption.\n\n### The filter toolbox\n\n| Condition | Logic | Example |\n|-----------|-------|---------|\n| must | AND, all conditions true | country = VN AND topic = shipping |\n| should | OR, at least one true (`min_should` sets a higher minimum) | topic = shipping OR topic = logistics |\n| must_not | Exclude matches | Exclude source = press-release-wire |\n| Range | Numeric or datetime bounds | published_at within the last seven days |\n| MatchAny | Value in a set | language in [ja, zh, th, vi] |\n| Geo | Radius, bounding box, or polygon | Events within 100 km of a port |\n\nThe cells below run several of these against the news collection.", + "metadata": { + "id": "yE9ZkAFvmtSb" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "IBbuMFd9W8gq", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "0c426ed4" + }, + "execution_count": null, + "source": "# must: AND. Only Vietnamese shipping news.\nf_must = models.Filter(\n must=[\n models.FieldCondition(key=\"country\", match=models.MatchValue(value=\"VN\")),\n models.FieldCondition(key=\"topic\", match=models.MatchValue(value=\"shipping\")),\n ]\n)\nprint(\"must (country=VN AND topic=shipping):\")\nshow(search(\"port congestion\", query_filter=f_must))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "must (country=VN AND topic=shipping):\n" + }, + { + "output_type": "stream", + "name": "stdout", + "text": "[0.5000] VN/vi shipping | Tac nghen cang tai Hai Phong khi luong hang tang manh.\n" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "j4JRfdQAe6NX", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "1d1d758e" + }, + "execution_count": null, + "source": "# should: OR. Either shipping or logistics topics.\nf_should = models.Filter(\n should=[\n models.FieldCondition(key=\"topic\", match=models.MatchValue(value=\"shipping\")),\n models.FieldCondition(key=\"topic\", match=models.MatchValue(value=\"logistics\")),\n ]\n)\nprint(\"should (topic=shipping OR topic=logistics):\")\nshow(search(\"congestion at ports\", query_filter=f_should))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "should (topic=shipping OR topic=logistics):\n" + }, + { + "output_type": "stream", + "name": "stdout", + "text": "[0.7000] SG/en shipping | Vessel Ever Given reroutes through Singapore.\n[0.5000] JP/ja shipping | 東南アジアの港湾混雑が輸送を遅らせている。\n[0.3333] CN/zh logistics | 上海港物流吞吐量创新高。\n[0.2500] TH/th shipping | ความแออัดที่ท่าเรือแหลมฉบังเพิ่มขึ้น\n[0.1667] DE/de logistics | Hamburger Hafen meldet Verzoegerungen.\n" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "qiGogkdmtsVr", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "825040b8" + }, + "execution_count": null, + "source": "# must_not + Range + MatchAny combined, evaluated together while the search runs.\nf_combined = models.Filter(\n must=[\n models.FieldCondition(\n key=\"published_at\",\n range=models.DatetimeRange(gte=\"2026-07-15T00:00:00Z\"),\n ),\n models.FieldCondition(\n key=\"language\",\n match=models.MatchAny(any=[\"ja\", \"zh\", \"th\", \"vi\"]),\n ),\n ],\n must_not=[\n models.FieldCondition(key=\"source\", match=models.MatchValue(value=\"press-release-wire\")),\n ],\n)\nprint(\"recent AND asian-language AND not a press-release wire:\")\nshow(search(\"port congestion in Southeast Asia\", query_filter=f_combined))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "recent AND asian-language AND not a press-release wire:\n[0.5000] JP/ja shipping | 東南アジアの港湾混雑が輸送を遅らせている。\n[0.3333] TH/th shipping | ความแออัดที่ท่าเรือแหลมฉบังเพิ่มขึ้น\n[0.2500] CN/zh logistics | 上海港物流吞吐量创新高。\n[0.2000] VN/vi shipping | Tac nghen cang tai Hai Phong khi luong hang tang manh.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### Common mistake: filters in the wrong place\n\nEvery filtered search above worked because `search()` passes the filter into each `Prefetch`. Move it to a top-level `query_filter` and it silently stops constraining the retrievers.\n\nWhenever a query has at least one prefetch, Qdrant runs the prefetches first and applies the main query to their results. A top-level filter therefore never reaches them: each searches the whole collection, returns its 50 candidates, and the filter only trims the fused set at the end. That is post-filtering, with exactly the failure mode described above.\n\nBelow, the same selective filter runs both ways. The correct version returns only Vietnamese shipping news. The broken version does not, and nothing raises an error.", + "metadata": { + "id": "lD5gYAEZyFQ8" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "vZRNzvdieTpk", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "1dda4347" + }, + "execution_count": null, + "source": "def search_wrong(query_text, query_filter=None, limit=5):\n \"\"\"Identical to search(), except the filter sits at the top level.\"\"\"\n dense_q = next(dense_model.query_embed(query_text)).tolist()\n sparse_q = to_sparse_vector(query_text, is_query=True)\n return client.query_points(\n collection_name=\"news\",\n prefetch=[\n models.Prefetch(query=dense_q, using=\"dense\", limit=50),\n models.Prefetch(query=sparse_q, using=\"sparse\", limit=50),\n ],\n query=models.RrfQuery(rrf=models.Rrf()),\n query_filter=query_filter, # too late: the prefetches already ran\n limit=limit,\n ).points\n\nprint(\"CORRECT filter inside each prefetch (country=VN AND topic=shipping):\")\nshow(search(\"port congestion\", query_filter=f_must))\nprint()\nprint(\"BROKEN same filter at the top level:\")\nshow(search_wrong(\"port congestion\", query_filter=f_must))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "CORRECT filter inside each prefetch (country=VN AND topic=shipping):\n[0.5000] VN/vi shipping | Tac nghen cang tai Hai Phong khi luong hang tang manh.\n\nBROKEN same filter at the top level:\n[0.7000] SG/en shipping | Vessel Ever Given reroutes through Singapore.\n[0.5000] JP/ja shipping | 東南アジアの港湾混雑が輸送を遅らせている。\n[0.3333] CN/zh logistics | 上海港物流吞吐量创新高。\n[0.2500] TH/th shipping | ความแออัดที่ท่าเรือแหลมฉบังเพิ่มขึ้น\n[0.1667] DE/de logistics | Hamburger Hafen meldet Verzoegerungen.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "The rule is short. No prefetch, use `query_filter`. Prefetch, put the filter in every prefetch.", + "metadata": { + "id": "jBq78Hp7twax" + } + }, + { + "cell_type": "markdown", + "source": "### A special case: scoping by user or tenant\n\nIn almost any multi user product you must scope every query to one user's or one customer's data. The instinct is a collection per user, which becomes millions of collections and is operationally unmanageable. The standard pattern instead:\n\n1. Add a `tenant_id` payload field to every point at ingestion. (Done, above.)\n2. Create a payload index on it with `is_tenant=True`. (Done, before ingestion.)\n3. Filter on it at every query. Never omit it.\n\nSteps 1 and 3 alone are already correct: they just get slower than they need to as tenant count grows. Step 2 is what keeps it fast, and it is the one people skip.", + "metadata": { + "id": "DFmFaqfycbso" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "KGUOS2yu9jSN", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "0deabc28" + }, + "execution_count": null, + "source": "def tenant_search(query_text, tenant_id, limit=5):\n return search(\n query_text,\n query_filter=models.Filter(\n must=[models.FieldCondition(key=\"tenant_id\", match=models.MatchValue(value=tenant_id))]\n ),\n limit=limit,\n )\n\nprint(\"europe-desk view of 'port delays':\")\nshow(tenant_search(\"port delays\", tenant_id=\"europe-desk\"))\nprint(\"\\nasia-desk view of 'port delays':\")\nshow(tenant_search(\"port delays\", tenant_id=\"asia-desk\"))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "europe-desk view of 'port delays':\n[0.5000] DE/de logistics | Hamburger Hafen meldet Verzoegerungen.\n[0.3333] DK/en markets | Maersk shares move on delisting speculation.\n\nasia-desk view of 'port delays':\n[0.7000] SG/en shipping | Vessel Ever Given reroutes through Singapore.\n[0.5000] JP/ja shipping | 東南アジアの港湾混雑が輸送を遅らせている。\n[0.3333] TH/th shipping | ความแออัดที่ท่าเรือแหลมฉบังเพิ่มขึ้น\n[0.2500] CN/zh logistics | 上海港物流吞吐量创新高。\n[0.1667] VN/vi shipping | Tac nghen cang tai Hai Phong khi luong hang tang manh.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### Key insight\n\nDesign the payload schema before you ingest, driven by one question: what will I need to filter on? Time, geography, identity, permissions, and status flags are the usual suspects.\n\nAdding a payload *field* later is easy. Adding a payload *index* later means rebuilding the HNSW graph, and discovering at query time that you never stored `language` at all means re-ingesting everything.", + "metadata": { + "id": "FZXZK3QQobiw" + } + }, + { + "cell_type": "markdown", + "source": "## 4. The Production RAG Pipeline\n\nRetrieval-Augmented Generation (RAG) retrieves relevant passages from a vector search engine and hands them to an LLM as context, so the model answers from your data instead of relying only on what it memorized during training.\n\nThe production shape, using everything above:\n\n1. **Query understanding:** extract hard constraints (dates, country, topic) into a filter. Embed the query as dense and sparse vectors.\n2. **Hybrid retrieval:** dense + sparse prefetch, each carrying the filter, fused by rank. One `query_points` call.\n3. **Optional reranking:** a cross-encoder scores the top candidates and keeps the best few. Add this only when evaluation shows fused results need refinement.\n4. **LLM generation:** the top passages go in as context, the model generates the answer with sources.\n\n### Rule of thumb\n\nWhen RAG quality disappoints, improve step 2 before reaching for a bigger model in step 4. Retrieval quality caps answer quality: the model cannot cite a passage it never received.\n\nThe cell below implements steps 1, 2, and the prompt assembly for step 4. It runs with no API key and prints the exact prompt an LLM would receive.", + "metadata": { + "id": "gZIMMQEaXUFH" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "OeTpeqTmT8Ex", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "7ac9a124" + }, + "execution_count": null, + "source": "def rag_context(question, query_filter=None, k=3):\n \"\"\"Steps 1 and 2: retrieve the top-k passages for a question.\"\"\"\n hits = search(question, query_filter=query_filter, limit=k)\n passages = []\n for h in hits:\n p = h.payload\n passages.append(f\"- ({p['source']}, {p['country']}, {p['published_at'][:10]}) {p['summary']}\")\n return \"\\n\".join(passages), hits\n\ndef build_prompt(question, query_filter=None, k=3):\n context, hits = rag_context(question, query_filter=query_filter, k=k)\n prompt = (\n \"Answer the question using only the sources below. Cite the source name.\\n\\n\"\n f\"Sources:\\n{context}\\n\\n\"\n f\"Question: {question}\\nAnswer:\"\n )\n return prompt, hits\n\nprompt, hits = build_prompt(\"What is happening with port congestion in Southeast Asia?\")\nprint(prompt)", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Answer the question using only the sources below. Cite the source name.\n\nSources:\n- (straits-times, SG, 2026-07-22) Vessel Ever Given reroutes through Singapore.\n- (nikkei, JP, 2026-07-21) 東南アジアの港湾混雑が輸送を遅らせている。\n- (bangkok-post, TH, 2026-07-18) ความแออัดที่ท่าเรือแหลมฉบังเพิ่มขึ้น\n\nQuestion: What is happening with port congestion in Southeast Asia?\nAnswer:\n" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "UsJadqPQPsGC", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "71e3fe07" + }, + "execution_count": null, + "source": "# OPTIONAL step 4: hand the prompt to a real LLM. This block is inert unless you\n# add an API key, so the notebook still runs end to end without one.\n#\n# import os\n# from anthropic import Anthropic\n# llm = Anthropic(api_key=os.environ[\"ANTHROPIC_API_KEY\"])\n# msg = llm.messages.create(\n# model=\"claude-sonnet-4-6\",\n# max_tokens=300,\n# messages=[{\"role\": \"user\", \"content\": prompt}],\n# )\n# print(msg.content[0].text)\n\nprint(\"Prompt is ready. Uncomment the block above and set an API key to generate an answer.\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Prompt is ready. Uncomment the block above and set an API key to generate an answer.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## What's next: Module 5\n\nThe capstone extends this system to three modalities at once: news, audio, and satellite imagery on shared points, clustered into risk themes and queried across languages.\n\n[Continue to Module 5](https://qdrant.tech/course/beginners/module-5/)", + "metadata": { + "id": "Y8Bqn6nKiG7i" + } + } + ] +} diff --git a/Beginner-course/Module5.ipynb b/Beginner-course/Module5.ipynb new file mode 100644 index 0000000..1c76286 --- /dev/null +++ b/Beginner-course/Module5.ipynb @@ -0,0 +1,358 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "cells": [ + { + "cell_type": "markdown", + "source": "# Module 5: Capstone, Multimodal Supplier Risk Intelligence\n\nA factory in Vietnam catches fire on a Tuesday. By that afternoon it is in a Japanese trade publication and on a Chinese forum. By Thursday it reaches the English business press. By Friday your procurement team finds out.\n\nThis notebook builds the system that finds Tuesday.\n\n## What you will do\n\n1. Create one collection with three named vectors: dense text, sparse text, and CLIP image.\n2. Ingest multilingual news and satellite tiles onto shared points.\n3. Query in English and retrieve Japanese and Chinese sources, with no translation.\n4. Query images with text, through CLIP's shared space.\n5. Cluster signals into risk themes and use a cluster centroid as a query.\n6. Run the analyst hybrid query, and break its filter on purpose to see why placement matters.\n\n**Tip for Colab:** the first cells download `multilingual-e5-large` (about 2 GB) plus two small CLIP encoders, so give the setup a few minutes. Everything after that is fast.\n\nCompanion notebook to the [Module 5 lesson](https://qdrant.tech/course/beginners/module-5/).", + "metadata": { + "id": "ujOeHwdFcAef" + } + }, + { + "cell_type": "markdown", + "source": "## Setup\n\nThree models, each chosen for a reason.\n\n`intfloat/multilingual-e5-large` inherits 100 languages from XLM-RoBERTa and puts all of them in one vector space, which is the point of the whole system. `Qdrant/bm25` gives exact-token matching for supplier codes and ticker symbols. And a matched pair of CLIP encoders, one for images and one for text, puts pictures and words in a *second* shared space so a text query can retrieve a photo.\n\nNothing here needs a GPU or an API key.", + "metadata": { + "id": "AZhnM6Jy8c1r" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "ihtYlKNxHddm" + }, + "execution_count": null, + "source": "!pip install -q \"qdrant-client[fastembed]\" scikit-learn pillow ", + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "QAtKCsXRpJG2", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "b4f16c27" + }, + "execution_count": null, + "source": "import warnings\nimport numpy as np\nfrom qdrant_client import QdrantClient, models\nfrom fastembed import TextEmbedding, SparseTextEmbedding, ImageEmbedding\n\nwarnings.filterwarnings(\"ignore\")\n\nTEXT_MODEL = \"intfloat/multilingual-e5-large\" # 1024-dim, 100 languages\nSPARSE_MODEL = \"Qdrant/bm25\" # exact tokens\nCLIP_VISION = \"Qdrant/clip-ViT-B-32-vision\" # 512-dim images\nCLIP_TEXT = \"Qdrant/clip-ViT-B-32-text\" # 512-dim text, same space\n\ntext_model = TextEmbedding(TEXT_MODEL)\nsparse_model = SparseTextEmbedding(SPARSE_MODEL)\nclip_vision = ImageEmbedding(CLIP_VISION)\nclip_text = TextEmbedding(CLIP_TEXT)\n\nTEXT_DIM = len(next(text_model.embed([\"passage: warm up\"])))\nCLIP_DIM = len(next(clip_text.embed([\"warm up\"])))\nprint(\"text dim:\", TEXT_DIM, \"| clip dim:\", CLIP_DIM)", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "text dim: 1024 | clip dim: 512\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "### The e5 prefixes are not optional\n\ne5 was trained with `query:` on search text and `passage:` on stored text. FastEmbed does not add them for you, and if you skip them nothing errors.\n\nMeasuring the effect takes a little care, because the thing that matters is not how high a relevant document scores. It is how far a relevant document sits above an irrelevant one, since that gap is what ranking depends on. So we score one relevant passage and one irrelevant passage against the same query, both ways, and compare the margin.", + "metadata": { + "id": "cPVJ2tRKJC06" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "DPdRO9YrxPbC", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "210c236d" + }, + "execution_count": null, + "source": "def unit(v):\n v = np.asarray(v, dtype=np.float32)\n return v / np.linalg.norm(v)\n\ndef embed_text(t, is_query=False):\n \"\"\"e5 needs an explicit prefix. This is the only place we add it.\"\"\"\n prefix = \"query: \" if is_query else \"passage: \"\n return unit(next(text_model.embed([prefix + t])))\n\ndef embed_raw(t):\n return unit(next(text_model.embed([t]))) # deliberately no prefix\n\nquestion = \"factory fire halted production\"\nrelevant = \"工場火災により生産が停止し、出荷の遅延が続いている。\"\nirrelevant = \"The supplier reaffirmed full-year guidance and reported steady quarterly demand.\"\n\nqv = embed_text(question, is_query=True)\nwith_rel, with_irr = float(qv @ embed_text(relevant)), float(qv @ embed_text(irrelevant))\n\nqr = embed_raw(question)\nraw_rel, raw_irr = float(qr @ embed_raw(relevant)), float(qr @ embed_raw(irrelevant))\n\nprint(f\"with prefixes relevant={with_rel:.3f} irrelevant={with_irr:.3f} margin={with_rel - with_irr:+.3f}\")\nprint(f\"without prefixes relevant={raw_rel:.3f} irrelevant={raw_irr:.3f} margin={raw_rel - raw_irr:+.3f}\")\nprint()\nprint(\"Skipping the prefixes RAISES both scores and NARROWS the gap between them.\")\nprint(\"Absolute similarity is not the target. Separation is.\")\nprint()\nprint(\"Note the scale too: e5 similarities compress into roughly 0.7 to 1.0,\")\nprint(\"so never read one of these numbers as a percentage of relevance.\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "with prefixes relevant=0.804 irrelevant=0.719 margin=+0.085\nwithout prefixes relevant=0.823 irrelevant=0.753 margin=+0.070\n\nSkipping the prefixes RAISES both scores and NARROWS the gap between them.\nAbsolute similarity is not the target. Separation is.\n\nNote the scale too: e5 similarities compress into roughly 0.7 to 1.0,\nso never read one of these numbers as a percentage of relevance.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 1. One Collection, Not Three\n\nThree modalities, and the instinct is three collections because that feels tidier.\n\nA single event produces evidence in several modalities at once. That factory fire is a news article, a satellite image, and a line in an earnings call. Split by modality and you have split one event across three collections: every query hits all three and stitches results back together in your own code, and your filters get written three times.\n\nNamed vectors solve it. One point carries a dense text vector, a sparse one, and a CLIP image vector, and all of them share a single payload.\n\nTwo details below decide whether this works in production. The sparse config needs `modifier=models.Modifier.IDF`, or you are not scoring BM25. And every payload index is created **now**, before any data arrives, because Qdrant can only add filter-aware edges to the vector index for indexes that already exist when it is built. Here there are two dense vectors, so a late index means rebuilding two graphs.\n\n`risk_score` is the one people forget, because nothing filters on it until Section 5. Miss it and the analyst query does not get slower, it fails: Qdrant Cloud enables strict mode by default, and strict mode rejects any query that filters an unindexed field.", + "metadata": { + "id": "eBI2Y1rzw27j" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "kooEKqazwJ7Q", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "89a0ecdc" + }, + "execution_count": null, + "source": "client = QdrantClient(\":memory:\")\n\nclient.create_collection(\n collection_name=\"supplier_signals\",\n vectors_config={\n \"text_dense\": models.VectorParams(size=TEXT_DIM, distance=models.Distance.COSINE),\n \"image\": models.VectorParams(size=CLIP_DIM, distance=models.Distance.COSINE),\n },\n sparse_vectors_config={\n \"text_sparse\": models.SparseVectorParams(modifier=models.Modifier.IDF),\n },\n)\n\nfor field in [\"supplier_id\", \"source_type\", \"language\", \"country\", \"facility_id\"]:\n client.create_payload_index(\"supplier_signals\", field_name=field,\n field_schema=models.PayloadSchemaType.KEYWORD)\n\nclient.create_payload_index(\"supplier_signals\", field_name=\"published_at\",\n field_schema=models.PayloadSchemaType.DATETIME)\nclient.create_payload_index(\"supplier_signals\", field_name=\"risk_score\",\n field_schema=models.PayloadSchemaType.FLOAT)\nclient.create_payload_index(\"supplier_signals\", field_name=\"cluster_id\",\n field_schema=models.PayloadSchemaType.INTEGER)\n\nprint(\"Collection created with 3 named vectors and 8 payload indexes, before ingestion.\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "Collection created with 3 named vectors and 8 payload indexes, before ingestion.\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 2. Ingesting Multilingual Signals\n\nThe signals below stand in for a day's feed. Note the shape of the story: the Japanese and Chinese sources are reporting a shutdown, while the English sources are reporting routine quarterly news. That gap is what Section 6 goes looking for.\n\n`source_type` is drawn from one fixed vocabulary, `news` or `satellite` here, because a filter written against a value nobody ingests returns nothing and warns you about nothing.", + "metadata": { + "id": "yygNdemkvdaj" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "gwb2Mjpvwh09", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "771154d2" + }, + "execution_count": null, + "source": "signals = [\n # SUP-7291: local-language sources are ahead of the English ones\n dict(supplier_id=\"SUP-7291\", language=\"ja\", country=\"JP\", source_type=\"news\",\n published_at=\"2026-07-21T09:00:00Z\", risk_score=0.88,\n text=\"工場火災により生産が停止し、出荷の遅延が続いている。復旧の見込みは立っていない。\"),\n dict(supplier_id=\"SUP-7291\", language=\"zh\", country=\"CN\", source_type=\"news\",\n published_at=\"2026-07-21T14:00:00Z\", risk_score=0.82,\n text=\"供应商工厂因火灾停产，交货期限推迟，客户已收到延误通知。\"),\n dict(supplier_id=\"SUP-7291\", language=\"ja\", country=\"JP\", source_type=\"news\",\n published_at=\"2026-07-22T02:00:00Z\", risk_score=0.71,\n text=\"労働争議が長引き、工場の稼働率が低下している。組合との交渉は難航している。\"),\n dict(supplier_id=\"SUP-7291\", language=\"vi\", country=\"VN\", source_type=\"news\",\n published_at=\"2026-07-20T08:00:00Z\", risk_score=0.64,\n text=\"Cang Hai Phong bi tac nghen, cac chuyen hang cua nha cung cap bi cham tre.\"),\n dict(supplier_id=\"SUP-7291\", language=\"en\", country=\"US\", source_type=\"news\",\n published_at=\"2026-07-22T07:15:00Z\", risk_score=0.12,\n text=\"The supplier reaffirmed full-year guidance and reported steady quarterly demand.\"),\n dict(supplier_id=\"SUP-7291\", language=\"en\", country=\"GB\", source_type=\"news\",\n published_at=\"2026-07-22T11:00:00Z\", risk_score=0.10,\n text=\"Analysts described the quarter as routine, with no change to the outlook for the group.\"),\n dict(supplier_id=\"SUP-7291\", language=\"en\", country=\"US\", source_type=\"news\",\n published_at=\"2026-07-19T16:00:00Z\", risk_score=0.55,\n text=\"Ticker SUP7291.T slipped modestly on higher freight costs across the region.\"),\n # a second supplier, so tenant-style scoping has something to exclude\n dict(supplier_id=\"SUP-0002\", language=\"zh\", country=\"CN\", source_type=\"news\",\n published_at=\"2026-07-21T10:00:00Z\", risk_score=0.79,\n text=\"另一家供应商的工厂发生停电，生产线暂时中断。\"),\n dict(supplier_id=\"SUP-0002\", language=\"en\", country=\"DE\", source_type=\"news\",\n published_at=\"2026-07-18T09:00:00Z\", risk_score=0.20,\n text=\"A European logistics operator announced a routine expansion of warehouse capacity.\"),\n]\n\ndef to_sparse(text, is_query=False):\n emb = next(sparse_model.query_embed(text)) if is_query else next(sparse_model.embed([text]))\n return models.SparseVector(indices=emb.indices.tolist(), values=emb.values.tolist())\n\npoints = []\nfor i, s in enumerate(signals):\n points.append(models.PointStruct(\n id=i,\n vector={\n \"text_dense\": embed_text(s[\"text\"]).tolist(), # passage: prefix\n \"text_sparse\": to_sparse(s[\"text\"]),\n },\n payload={**s, \"summary\": s[\"text\"][:60]},\n ))\n\nclient.upsert(\"supplier_signals\", points=points)\nprint(\"ingested\", client.count(\"supplier_signals\").count, \"text signals across\",\n len({s[\"language\"] for s in signals}), \"languages\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "ingested 9 text signals across 4 languages\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 3. Satellite Tiles Through CLIP\n\nReal satellite imagery is not something a notebook can ship, so we generate four crude tiles instead. They are enough to show the mechanics and, as you will see, enough to show a real limitation too.\n\nThese points carry only the `image` vector. They live in the same collection as the text signals and share the same payload schema, which is the whole argument for named vectors.", + "metadata": { + "id": "b6GjHbG81Rqw" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "kVHWunY90XYT", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "3850044b" + }, + "execution_count": null, + "source": "from PIL import Image, ImageDraw, ImageFilter\nimport os\n\nos.makedirs(\"tiles\", exist_ok=True)\n\ndef tile_smoke():\n im = Image.new(\"RGB\", (224, 224), (70, 72, 78)); d = ImageDraw.Draw(im)\n d.rectangle([40, 150, 184, 224], fill=(48, 48, 52))\n for y, r in [(140, 20), (115, 28), (88, 36), (60, 44)]:\n d.ellipse([112 - r, y - r, 112 + r, y + r], fill=(190, 190, 195))\n return im.filter(ImageFilter.GaussianBlur(6))\n\ndef tile_farmland():\n im = Image.new(\"RGB\", (224, 224), (94, 140, 60)); d = ImageDraw.Draw(im)\n for x in range(0, 224, 16):\n d.rectangle([x, 0, x + 8, 224], fill=(120, 168, 74))\n return im.filter(ImageFilter.GaussianBlur(1))\n\ndef tile_harbour():\n im = Image.new(\"RGB\", (224, 224), (40, 90, 150)); d = ImageDraw.Draw(im)\n for x, y in [(30, 60), (120, 100), (70, 160), (150, 40)]:\n d.rectangle([x, y, x + 50, y + 18], fill=(210, 210, 215))\n return im.filter(ImageFilter.GaussianBlur(1))\n\ndef tile_fire():\n im = Image.new(\"RGB\", (224, 224), (30, 20, 15)); d = ImageDraw.Draw(im)\n for r, c in [(90, (120, 30, 0)), (65, (200, 70, 0)), (40, (255, 150, 0)), (20, (255, 230, 120))]:\n d.ellipse([112 - r, 150 - r, 112 + r, 150 + r], fill=c)\n return im.filter(ImageFilter.GaussianBlur(8))\n\ntiles = [\n (\"smoke_plume\", tile_smoke, \"SUP-7291\", \"FAC-01\"),\n (\"fire\", tile_fire, \"SUP-7291\", \"FAC-01\"),\n (\"harbour\", tile_harbour, \"SUP-7291\", \"FAC-02\"),\n (\"farmland\", tile_farmland, \"SUP-0002\", \"FAC-09\"),\n]\n\npaths = []\nfor name, fn, _, _ in tiles:\n p = f\"tiles/{name}.png\"\n fn().save(p)\n paths.append(p)\n\nvecs = [unit(v).tolist() for v in clip_vision.embed(paths)]\n\nclient.upsert(\"supplier_signals\", points=[\n models.PointStruct(\n id=100 + i,\n vector={\"image\": vec},\n payload=dict(supplier_id=sup, facility_id=fac, source_type=\"satellite\",\n language=\"n/a\", country=\"VN\",\n published_at=\"2026-07-21T00:00:00Z\",\n risk_score=0.9 if name in (\"fire\", \"smoke_plume\") else 0.1,\n summary=f\"synthetic satellite tile: {name}\"),\n )\n for i, (vec, (name, _, sup, fac)) in enumerate(zip(vecs, tiles))\n])\n\nprint(\"collection now holds\", client.count(\"supplier_signals\").count, \"points (text + imagery)\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "collection now holds 13 points (text + imagery)\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 4. Two Kinds of Cross-Modal Query\n\n### Text against images, via CLIP\n\nThe query text goes through CLIP's **text** encoder, not e5. That is the part people get wrong: each named vector is its own space, and a query only means something in the space it was embedded for. Sending an e5 vector at the `image` vector would return numbers, and they would be noise.", + "metadata": { + "id": "7B697wnommM3" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "0DOXfO4WUDlW", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "5cf66fb2" + }, + "execution_count": null, + "source": "def image_search(text, limit=4):\n qv = unit(next(clip_text.embed([text]))).tolist() # CLIP text encoder\n return client.query_points(\"supplier_signals\", query=qv, using=\"image\",\n limit=limit).points\n\nfor q in [\"an orange fire burning\", \"ships in blue water at a port\", \"grey smoke rising into the sky\"]:\n print(f\"{q!r}\")\n for r in image_search(q):\n print(f\" {r.score:.3f} {r.payload['summary']}\")\n print()", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "'an orange fire burning'\n 0.267 synthetic satellite tile: fire\n 0.209 synthetic satellite tile: farmland\n 0.191 synthetic satellite tile: smoke_plume\n 0.188 synthetic satellite tile: harbour\n\n'ships in blue water at a port'\n 0.209 synthetic satellite tile: harbour\n 0.193 synthetic satellite tile: smoke_plume\n 0.180 synthetic satellite tile: farmland\n 0.141 synthetic satellite tile: fire\n\n'grey smoke rising into the sky'\n 0.211 synthetic satellite tile: fire\n 0.204 synthetic satellite tile: farmland\n 0.195 synthetic satellite tile: smoke_plume\n 0.194 synthetic satellite tile: harbour\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Two of those three are right, and the third is the useful one.\n\n\"an orange fire burning\" and \"ships in blue water at a port\" retrieve the correct tiles. \"grey smoke rising into the sky\" does not: it prefers the fire tile.\n\nThat is not a bug in Qdrant or in the query. CLIP was trained on photographs, and these tiles are flat drawings made with a few ellipses. They sit outside the distribution the model learned, so its judgments get unreliable. Swap in real satellite imagery and this improves immediately.\n\nThe lesson generalizes past this notebook: cross-modal retrieval quality depends on your images resembling the model's training data, and the only way to know is to evaluate on your own.", + "metadata": { + "id": "hj4YjZ8OvIia" + } + }, + { + "cell_type": "markdown", + "source": "### English query against local-language text\n\nNow the capability the whole system exists for. The query is English, the corpus is not, and there is no translation step anywhere.\n\nThe filter has no prefetch above it, so `query_filter` is the correct placement here.", + "metadata": { + "id": "8OG5A2ZnpsoK" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "qzi4vCK4A4FG", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "80720c27" + }, + "execution_count": null, + "source": "def text_search(query_en, languages=None, supplier=None, limit=5):\n must = []\n if supplier:\n must.append(models.FieldCondition(key=\"supplier_id\",\n match=models.MatchValue(value=supplier)))\n if languages:\n must.append(models.FieldCondition(key=\"language\",\n match=models.MatchAny(any=languages)))\n return client.query_points(\n \"supplier_signals\",\n query=embed_text(query_en, is_query=True).tolist(), # query: prefix\n using=\"text_dense\",\n query_filter=models.Filter(must=must) if must else None,\n limit=limit,\n ).points\n\nprint(\"English query -> Japanese and Chinese sources only:\\n\")\nfor r in text_search(\"factory shutdown production halt\",\n languages=[\"ja\", \"zh\"], supplier=\"SUP-7291\"):\n p = r.payload\n print(f\" {r.score:.3f} [{p['language']}] risk={p['risk_score']:.2f} {p['summary']}\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "English query -> Japanese and Chinese sources only:\n\n" + }, + { + "output_type": "stream", + "name": "stdout", + "text": " 0.786 [zh] risk=0.82 供应商工厂因火灾停产，交货期限推迟，客户已收到延误通知。\n 0.776 [ja] risk=0.88 工場火災により生産が停止し、出荷の遅延が続いている。復旧の見込みは立っていない。\n 0.758 [ja] risk=0.71 労働争議が長引き、工場の稼働率が低下している。組合との交渉は難航している。\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 5. Clustering Signals Into Risk Themes\n\nClustering groups signals that describe the same underlying event even when they arrive in different languages from different sources.\n\nThree practical points, all of them easy to get wrong.\n\n**Page the scroll.** It returns a batch and an offset, and you keep going until the offset comes back empty. One capped call quietly clusters a busy supplier on partial data.\n\n**Normalize before k-means.** These vectors are built for cosine similarity, but k-means measures Euclidean distance. Without normalizing you are partly clustering by vector length instead of direction.\n\n**Drop the supplier filter to go wider.** A theme shared across suppliers shows up as one cluster pulling in signals from several of them at once.", + "metadata": { + "id": "SIuGIWgIprVF" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "IV4BLFFQCHD6", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "bef48fd2" + }, + "execution_count": null, + "source": "from sklearn.cluster import KMeans\n\ndef scroll_all(scroll_filter=None, page=64):\n \"\"\"Page until the offset comes back empty.\"\"\"\n out, offset = [], None\n while True:\n batch, offset = client.scroll(\"supplier_signals\", scroll_filter=scroll_filter,\n with_vectors=True, limit=page, offset=offset)\n out.extend(batch)\n if offset is None:\n return out\n\ndef dense_matrix(points):\n \"\"\"Unit-normalized text_dense vectors, with the ids they belong to.\"\"\"\n ids, vecs = [], []\n for p in points:\n if p.vector and \"text_dense\" in p.vector: # image-only points have none\n ids.append(p.id)\n vecs.append(p.vector[\"text_dense\"])\n if not vecs:\n return [], None\n arr = np.asarray(vecs, dtype=np.float32)\n arr /= np.linalg.norm(arr, axis=1, keepdims=True)\n return ids, arr\n\ntext_only = models.Filter(must=[models.FieldCondition(\n key=\"source_type\", match=models.MatchValue(value=\"news\"))])\n\npts = scroll_all(text_only)\nids, arr = dense_matrix(pts)\nprint(f\"scrolled {len(pts)} points, {len(ids)} carry a text vector\")\n\nlabels = KMeans(n_clusters=3, n_init=10, random_state=42).fit_predict(arr)\n\n# One set_payload call per cluster, not one per point\nfor label in sorted({int(l) for l in labels}):\n client.set_payload(\"supplier_signals\", payload={\"cluster_id\": label},\n points=[i for i, l in zip(ids, labels) if int(l) == label])\n\nby_id = {p.id: p.payload for p in pts}\nfor label in sorted({int(l) for l in labels}):\n print(f\"\\ncluster {label}:\")\n for i, l in zip(ids, labels):\n if int(l) == label:\n p = by_id[i]\n print(f\" [{p['language']}] risk={p['risk_score']:.2f} {p['summary'][:52]}\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "scrolled 9 points, 9 carry a text vector\n\ncluster 0:\n [vi] risk=0.64 Cang Hai Phong bi tac nghen, cac chuyen hang cua nha\n\ncluster 1:\n [en] risk=0.12 The supplier reaffirmed full-year guidance and repor\n [en] risk=0.10 Analysts described the quarter as routine, with no c\n [en] risk=0.55 Ticker SUP7291.T slipped modestly on higher freight \n [en] risk=0.20 A European logistics operator announced a routine ex\n\ncluster 2:\n [ja] risk=0.88 工場火災により生産が停止し、出荷の遅延が続いている。復旧の見込みは立っていない。\n [zh] risk=0.82 供应商工厂因火灾停产，交货期限推迟，客户已收到延误通知。\n [ja] risk=0.71 労働争議が長引き、工場の稼働率が低下している。組合との交渉は難航している。\n [zh] risk=0.79 另一家供应商的工厂发生停电，生产线暂时中断。\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "Notice that the clusters cross language boundaries. The Japanese fire report and the Chinese shutdown report land together because they describe the same event, which is exactly what a single multilingual vector space buys you.\n\n### The centroid as a query\n\nA cluster centroid is just another vector, so you can hand it straight back to Qdrant as a query and pull in more of the same theme. That is how you go from \"here are today's clusters\" to \"find everything that looks like this emerging story\".", + "metadata": { + "id": "hBeoePW3jSOi" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "27n7yE9ZkAFv", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "35b08705" + }, + "execution_count": null, + "source": "target = int(labels[0])\nmember_rows = [i for i, l in zip(ids, labels) if int(l) == target]\ncentroid = arr[[ids.index(i) for i in member_rows]].mean(axis=0)\ncentroid = centroid / np.linalg.norm(centroid)\n\nprint(f\"querying with the centroid of cluster {target}:\\n\")\nfor r in client.query_points(\"supplier_signals\", query=centroid.tolist(),\n using=\"text_dense\", limit=5).points:\n p = r.payload\n mark = \" <- in the cluster\" if r.id in member_rows else \"\"\n print(f\" {r.score:.3f} [{p['language']}] {p['summary'][:48]}{mark}\")", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "querying with the centroid of cluster 2:\n\n 0.962 [zh] 供应商工厂因火灾停产，交货期限推迟，客户已收到延误通知。 <- in the cluster\n 0.961 [zh] 另一家供应商的工厂发生停电，生产线暂时中断。 <- in the cluster\n 0.957 [ja] 工場火災により生産が停止し、出荷の遅延が続いている。復旧の見込みは立っていない。 <- in the cluster\n 0.947 [ja] 労働争議が長引き、工場の稼働率が低下している。組合との交渉は難航している。 <- in the cluster\n 0.798 [vi] Cang Hai Phong bi tac nghen, cac chuyen hang cua\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "## 6. The Analyst Query, and the Mistake That Fails Silently\n\nThe query analysts actually run: hybrid retrieval over dense and sparse, scoped to one supplier, elevated risk only.\n\nOne detail decides whether it works. The filter goes **inside each prefetch**, not on the outer query. Prefetches run first and the outer query is applied to their results, so a top-level filter arrives too late: both retrievers search every supplier and every risk level, and the filter only trims the fused list at the end.\n\nBoth versions run below. Watch the `risk` and `supplier` columns in the broken one.", + "metadata": { + "id": "MFd9W8gqcWqi" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "A4Yqj4JRfdQA", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "1fac1d1d" + }, + "execution_count": null, + "source": "def analyst_search(query_text, supplier, min_risk=0.5, limit=5, broken=False):\n dense_q = embed_text(query_text, is_query=True).tolist()\n sparse_q = to_sparse(query_text, is_query=True)\n risk_filter = models.Filter(must=[\n models.FieldCondition(key=\"supplier_id\", match=models.MatchValue(value=supplier)),\n models.FieldCondition(key=\"risk_score\", range=models.Range(gte=min_risk)),\n ])\n if broken:\n return client.query_points(\n \"supplier_signals\",\n prefetch=[\n models.Prefetch(query=dense_q, using=\"text_dense\", limit=50),\n models.Prefetch(query=sparse_q, using=\"text_sparse\", limit=50),\n ],\n query=models.RrfQuery(rrf=models.Rrf()),\n query_filter=risk_filter, # too late\n limit=limit,\n ).points\n return client.query_points(\n \"supplier_signals\",\n prefetch=[\n models.Prefetch(query=dense_q, using=\"text_dense\",\n filter=risk_filter, limit=50),\n models.Prefetch(query=sparse_q, using=\"text_sparse\",\n filter=risk_filter, limit=50),\n ],\n query=models.RrfQuery(rrf=models.Rrf()),\n limit=limit,\n ).points\n\ndef report(title, rows):\n print(title)\n if not rows:\n print(\" (no results)\")\n for r in rows:\n p = r.payload\n print(f\" {r.score:.4f} {p['supplier_id']} risk={p['risk_score']:.2f} \"\n f\"[{p['language']}] {p['summary'][:42]}\")\n print()\n\nreport(\"CORRECT filter inside each prefetch:\",\n analyst_search(\"production halt at the factory\", \"SUP-7291\"))\nreport(\"BROKEN same filter at the top level:\",\n analyst_search(\"production halt at the factory\", \"SUP-7291\", broken=True))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "CORRECT filter inside each prefetch:\n 0.5000 SUP-7291 risk=0.82 [zh] 供应商工厂因火灾停产，交货期限推迟，客户已收到延误通知。\n 0.3333 SUP-7291 risk=0.88 [ja] 工場火災により生産が停止し、出荷の遅延が続いている。復旧の見込みは立っていない。\n 0.2500 SUP-7291 risk=0.55 [en] Ticker SUP7291.T slipped modestly on highe\n 0.2000 SUP-7291 risk=0.71 [ja] 労働争議が長引き、工場の稼働率が低下している。組合との交渉は難航している。\n 0.1667 SUP-7291 risk=0.64 [vi] Cang Hai Phong bi tac nghen, cac chuyen ha\n\n" + }, + { + "output_type": "stream", + "name": "stdout", + "text": "BROKEN same filter at the top level:\n 0.5000 SUP-0002 risk=0.79 [zh] 另一家供应商的工厂发生停电，生产线暂时中断。\n 0.3333 SUP-7291 risk=0.82 [zh] 供应商工厂因火灾停产，交货期限推迟，客户已收到延误通知。\n 0.2500 SUP-7291 risk=0.88 [ja] 工場火災により生産が停止し、出荷の遅延が続いている。復旧の見込みは立っていない。\n 0.2000 SUP-7291 risk=0.55 [en] Ticker SUP7291.T slipped modestly on highe\n 0.1667 SUP-7291 risk=0.71 [ja] 労働争議が長引き、工場の稼働率が低下している。組合との交渉は難航している。\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "The broken version returns rows that violate the filter, with no error and no warning. On a real corpus, where one supplier is a thousandth of the collection instead of most of it, the same mistake usually returns nothing at all, and an analyst concludes there is no news.\n\nThe rule, one more time. No prefetch, use `query_filter`. Prefetch, put the filter in every prefetch.", + "metadata": { + "id": "CvI5qiGogkdm" + } + }, + { + "cell_type": "markdown", + "source": "## 7. Reading the Gap\n\nThe payoff. Run the same query twice, once scoped to English sources and once to Japanese and Chinese, and compare.\n\nThis is not a different algorithm. It is the same query and the same vector space with a different `language` filter.", + "metadata": { + "id": "tsVrFlvbpaTI" + } + }, + { + "cell_type": "code", + "metadata": { + "id": "lD5gYAEZyFQ8", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "5dba6502" + }, + "execution_count": null, + "source": "query = \"factory shutdown, production halted, delivery delays\"\n\nreport(\"ENGLISH sources only:\",\n text_search(query, languages=[\"en\"], supplier=\"SUP-7291\"))\nreport(\"JAPANESE and CHINESE sources only:\",\n text_search(query, languages=[\"ja\", \"zh\"], supplier=\"SUP-7291\"))", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": "ENGLISH sources only:\n 0.7789 SUP-7291 risk=0.55 [en] Ticker SUP7291.T slipped modestly on highe\n 0.7239 SUP-7291 risk=0.12 [en] The supplier reaffirmed full-year guidance\n 0.6827 SUP-7291 risk=0.10 [en] Analysts described the quarter as routine,\n\nJAPANESE and CHINESE sources only:\n 0.8059 SUP-7291 risk=0.82 [zh] 供应商工厂因火灾停产，交货期限推迟，客户已收到延误通知。\n 0.7946 SUP-7291 risk=0.88 [ja] 工場火災により生産が停止し、出荷の遅延が続いている。復旧の見込みは立っていない。\n 0.7731 SUP-7291 risk=0.71 [ja] 労働争議が長引き、工場の稼働率が低下している。組合との交渉は難航している。\n\n" + } + ] + }, + { + "cell_type": "markdown", + "source": "The English sources are reporting reaffirmed guidance and a routine quarter. The Japanese and Chinese sources are reporting a fire, halted production, and delayed deliveries, and they were published earlier.\n\nThat gap is the early warning, and finding it needed no translation pipeline, no separate per-language index, and no second database. One collection, one multilingual model, one filter.", + "metadata": { + "id": "eTpkf01PrpsC" + } + }, + { + "cell_type": "markdown", + "source": "## Course Close\n\nLook at what is in that collection. Text, sparse tokens, and imagery on shared points. Filters that hold. Hybrid retrieval. Clustering. Cross-language and cross-modal search.\n\nSix primitives got you here: collection, point, vector, payload, index, query.\n\nModule 1 asked why keyword search misses. Module 2 opened up the vector. Module 3 put dense and sparse together. Module 4 turned that into a design you could defend. This module ran the whole thing on three modalities at once.\n\nThe next system someone hands you is these six, arranged differently.\n\n### Your turn\n\nSwap the synthetic tiles for real satellite imagery and rerun Section 4, then check whether the smoke query starts behaving.\n\nAdd a `tenant_id` field with `is_tenant=True` and scope every query to one desk, the way Module 4 did.\n\nThen try `models.Rrf(weights=[3.0, 1.0])` in the analyst query to favour dense over sparse, and see which retriever your data actually prefers.", + "metadata": { + "id": "jBq78Hp7twax" + } + } + ] +} From d32bf11fc356d47bd6a299a2a1eeb0aa375fd985 Mon Sep 17 00:00:00 2001 From: Ewa Szyszka <79250896+ESzyszka@users.noreply.github.com> Date: Tue, 11 Aug 2026 11:46:12 +0200 Subject: [PATCH 2/3] Update Module 4 code to match the md file --- Beginner-course/Module4.ipynb | 1153 ++++++++++++++++++++++++++------- 1 file changed, 910 insertions(+), 243 deletions(-) diff --git a/Beginner-course/Module4.ipynb b/Beginner-course/Module4.ipynb index 4f53ac2..e5b7560 100644 --- a/Beginner-course/Module4.ipynb +++ b/Beginner-course/Module4.ipynb @@ -1,443 +1,1110 @@ { - "nbformat": 4, - "nbformat_minor": 0, - "metadata": { - "colab": { - "provenance": [] - }, - "kernelspec": { - "display_name": "Python 3", - "name": "python3" - }, - "language_info": { - "name": "python" - } - }, "cells": [ { "cell_type": "markdown", - "source": "# Module 4: Designing a Vector Search System\n\nVector search systems rarely break at the distance metric. They break because nobody decided what the payload had to hold until a few million documents were already ingested. This module is about the decisions that are cheap to make now and expensive to revisit later.\n\n## What you will do\n\n1. See the five layers of the stack as a mental checklist.\n2. Design and build a multilingual news search system by answering five questions.\n3. Practice filtering the way production systems do it, including the mistake that fails silently.\n4. Assemble a production RAG retrieval pipeline.\n\n**Tip for Colab:** run the cells top to bottom. The first install cell takes a minute because it downloads the embedding models.\n\nCompanion notebook to the [Module 4 lesson](https://qdrant.tech/course/beginners/module-4/).", - "metadata": { - "id": "ujOeHwdFcAef" - } + "id": "b4c1d4b2", + "metadata": {}, + "source": [ + "# Module 4: Designing a Vector Search System\n", + "\n", + "You know the parts from Modules 1 to 3: embeddings, collections, HNSW, hybrid retrieval, and filters. This module is about judgment, going from \"we have articles and analysts\" to a payload schema, a retrieval pipeline, and a deployment mode.\n", + "\n", + "## What You Will Do\n", + "\n", + "1. See the five layers of the stack as a diagnostic checklist.\n", + "2. Design and build a news search system by answering five questions.\n", + "3. Filter the way production systems do, and see what happens when you get it wrong.\n", + "4. Assemble a production RAG retrieval pipeline.\n", + "5. Review the deployment options and work through a knowledge check.\n", + "\n", + "**Running this in Colab:** run the cells top to bottom. The first two cells take a minute, because they download the embedding models." + ] }, { "cell_type": "markdown", - "source": "## Setup\n\nWe install `qdrant-client` with the FastEmbed extra. FastEmbed gives us local, CPU friendly embedding models so the notebook produces real vectors rather than random numbers.", - "metadata": { - "id": "AZhnM6Jy8c1r" - } + "id": "8abb2140", + "metadata": {}, + "source": [ + "## Setup\n", + "\n", + "The notebook runs against Qdrant in **local mode**, an in-memory Qdrant that needs no server. It is the fastest way to follow along, and it has one limit worth knowing before you start: local mode is a Python reimplementation, not the engine. Payload indexes have no effect there, and search is exact rather than approximate. Every index call below is still the right habit, and Section 5 covers when to move off it.\n", + "\n", + "To run against a real cluster instead, create a free one at [cloud.qdrant.io](https://cloud.qdrant.io/), then swap in the commented lines below." + ] }, { "cell_type": "code", - "metadata": { - "id": "ihtYlKNxHddm" - }, "execution_count": null, - "source": "!pip install -q \"qdrant-client[fastembed]\" ", - "outputs": [] - }, - { - "cell_type": "markdown", - "source": "## 1. The Layers of the Stack\n\nEvery vector search system, from a notebook prototype to a deployment serving millions of queries, is built from the same five layers. When something is slow, wrong, or expensive, the first diagnostic question is always the same: which layer is the problem in?\n\n- **Query layer**: turns user intent into a search. Embedding the query, choosing dense vs sparse vs hybrid, fusing results, setting limits.\n- **Indexing layer**: makes that search fast, through the HNSW graph for vectors and payload indexes for the fields you filter on. Get this wrong and results are usually still correct, just slow.\n- **Storage layer**: holds the points themselves, the vectors, payloads, and IDs, across memory and disk.\n- **Knowledge layer**: shapes the data before it ever becomes a vector: chunking, embedding model choice, payload schema. No amount of tuning elsewhere fixes a mistake here. Garbage in, garbage retrieved.\n- **Distribution layer**: spreads the system across more than one machine: sharding, replication, multi node clusters. You will not need it on day one.\n\n### Key insight\n\nEvery design decision belongs to a layer. \"Add a payload index\" is an indexing decision. \"Switch to a multilingual embedding model\" is a knowledge decision. \"Move to three nodes\" is a distribution decision. Once you sort decisions into layers, intimidating architecture diagrams become checklists.", + "id": "fd85e0de", "metadata": { - "id": "QAtKCsXRpJG2" - } - }, - { - "cell_type": "markdown", - "source": "## 2. Worked Example: Designing a Multilingual News Search System\n\nHere is the brief, the kind you would get on a real project:\n\n> Analysts at a research firm need to search global news that arrives continuously, in many languages. They ask in English (\"port congestion in Southeast Asia\") but expect matches from any language, and they scope every search by country, topic, date range, and source. Some queries name one specific thing, a company ticker or a ship name, that has to match exactly.\n\nFive questions turn it into a design, and each answer is a decision that belongs to one layer.\n\n### Question 1: What do the queries look like?\n\nBoth natural language and exact tokens. \"Port congestion in Southeast Asia\" is semantic intent, which is dense territory. \"MAERSK-B.CO delisting\" is an exact token that dense search will blur into neighboring tickers. That is the SKU problem from Module 3.\n\n**Decision:** hybrid from the start. Two named vectors on every point, one dense and one sparse, with their rankings fused at query time. This is a query layer decision.", - "metadata": { - "id": "Tr8hzUjEcPVJ" - } + "execution": { + "iopub.execute_input": "2026-08-11T09:29:07.591536Z", + "iopub.status.busy": "2026-08-11T09:29:07.591306Z", + "iopub.status.idle": "2026-08-11T09:29:07.597029Z", + "shell.execute_reply": "2026-08-11T09:29:07.596084Z" + } + }, + "outputs": [], + "source": [ + "!pip install -q \"qdrant-client[fastembed]\"" + ] }, { "cell_type": "code", + "execution_count": 2, + "id": "fb19a50f", "metadata": { - "id": "2tRKJC06DPdR", - "colab": { - "base_uri": "https://localhost:8080/" - }, - "outputId": "afd46a07" + "execution": { + "iopub.execute_input": "2026-08-11T09:29:07.598654Z", + "iopub.status.busy": "2026-08-11T09:29:07.598507Z", + "iopub.status.idle": "2026-08-11T09:29:10.904642Z", + "shell.execute_reply": "2026-08-11T09:29:10.902935Z" + } }, - "execution_count": null, - "source": "import warnings\nfrom qdrant_client import QdrantClient, models\nfrom fastembed import TextEmbedding, SparseTextEmbedding\n\n# Local mode: an in-process Qdrant, ideal for prototyping, notebooks and teaching.\n# It is a Python reimplementation rather than the engine: search is exact instead\n# of approximate, and payload indexes have no effect. Everything we write below is\n# still the right shape for a real server.\nclient = QdrantClient(\":memory:\")\n\n# Knowledge layer decision made concrete: a MULTILINGUAL dense model, so an\n# English query can retrieve a Japanese or Vietnamese article with no translation.\n# We use a small multilingual model to keep the Colab download fast. Production\n# would reach for something like multilingual-e5-large.\ndense_model = TextEmbedding(\"sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2\")\n\n# A sparse model gives us exact-token matching alongside dense semantics.\n# BM25 is lightweight and language agnostic, a good fit for mixed-language news.\nsparse_model = SparseTextEmbedding(\"Qdrant/bm25\")\n\n# Ask the model for its dimensionality rather than hardcoding it, so the\n# collection config can never drift out of sync with the model.\nDENSE_DIM = len(list(dense_model.embed([\"hello\"]))[0])\nprint(\"Dense vector dimension:\", DENSE_DIM)", "outputs": [ { - "output_type": "stream", "name": "stdout", - "text": "Dense vector dimension: 384\n" + "output_type": "stream", + "text": [ + "Dense vector dimension: 384\n" + ] } + ], + "source": [ + "from qdrant_client import QdrantClient, models\n", + "from fastembed import TextEmbedding, SparseTextEmbedding\n", + "\n", + "# Local mode: in-memory Qdrant, no server needed.\n", + "client = QdrantClient(\":memory:\")\n", + "\n", + "# To use a real cluster instead, comment out the line above and uncomment these.\n", + "# Colab stores secrets under the key icon in the left sidebar.\n", + "#\n", + "# from google.colab import userdata\n", + "# client = QdrantClient(\n", + "# url=userdata.get(\"QDRANT_URL\"),\n", + "# api_key=userdata.get(\"QDRANT_API_KEY\"),\n", + "# )\n", + "\n", + "# A small, fast English model. Any embedding model works, this one keeps the\n", + "# Colab download short. It is also FastEmbed's default.\n", + "dense_model = TextEmbedding(\"BAAI/bge-small-en-v1.5\")\n", + "\n", + "# BM25 gives us exact-token matching alongside dense semantics.\n", + "sparse_model = SparseTextEmbedding(\"Qdrant/bm25\")\n", + "\n", + "# Ask the model for its dimensionality rather than hardcoding it, so the\n", + "# collection config can never drift out of sync with the model.\n", + "DENSE_DIM = len(list(dense_model.embed([\"hello\"]))[0])\n", + "print(\"Dense vector dimension:\", DENSE_DIM)" ] }, { "cell_type": "markdown", - "source": "Now create the collection with **named vectors**: one dense, one sparse. Naming them lets a single point carry both, and lets a single query fuse both.\n\nThe sparse config needs `modifier=models.Modifier.IDF`. BM25-style sparse vectors store only term frequency, and Qdrant applies the inverse-document-frequency half of the formula at query time. Leave the modifier off and you are not scoring BM25, you are scoring raw term counts.", - "metadata": { - "id": "khdVipy2eBI2" - } + "id": "29ae8d4a", + "metadata": {}, + "source": [ + "## 1. The Layers of the Stack\n", + "\n", + "Every vector search system is built from the same five layers. When something is slow, wrong, or expensive, the first question is always which layer the problem is in.\n", + "\n", + "| Layer | What lives here |\n", + "|---|---|\n", + "| Query | Embedding the query, dense vs. sparse vs. hybrid, fusion, limits |\n", + "| Indexing | The HNSW graph for vectors, payload indexes for filter fields |\n", + "| Storage | Vectors, payloads, and IDs on disk and in memory |\n", + "| Knowledge | The data itself: chunking, embedding model choice, payload schema |\n", + "| Distribution | Sharding, replication, multi-node clusters |\n", + "\n", + "Every decision below is tagged with the layer it belongs to." + ] + }, + { + "cell_type": "markdown", + "id": "c5c11f1e", + "metadata": {}, + "source": [ + "## 2. Worked Example: Designing a News Search System\n", + "\n", + "> Analysts at a research firm need to search global news that arrives continuously. They ask in plain language (\"port congestion in Southeast Asia\") and they scope every search by country, topic, date range, and source. Some queries name one specific thing, a company ticker or a ship name, that has to match exactly.\n", + "\n", + "Five questions turn that brief into a system.\n", + "\n", + "### Question 1: What Do the Queries Look Like?\n", + "\n", + "Both kinds. \"Port congestion in Southeast Asia\" is semantic intent, which is dense territory. \"MAERSK-B.CO\" is an exact token that carries no meaning a dense model can use.\n", + "\n", + "**Decision:** hybrid search, with a named dense vector and a named sparse vector on every point, fused at query time. *(Query layer.)*\n", + "\n", + "Naming the vectors is what lets a single point carry both and a single query use both. The `modifier=IDF` setting is required for correct BM25 scoring: sparse vectors store term frequency, and Qdrant applies the inverse document frequency half at query time." + ] }, { "cell_type": "code", + "execution_count": 3, + "id": "7d821206", "metadata": { - "id": "Y1rzw27jkooE", - "colab": { - "base_uri": "https://localhost:8080/" - }, - "outputId": "940659fb" + "execution": { + "iopub.execute_input": "2026-08-11T09:29:10.907847Z", + "iopub.status.busy": "2026-08-11T09:29:10.906502Z", + "iopub.status.idle": "2026-08-11T09:29:10.913169Z", + "shell.execute_reply": "2026-08-11T09:29:10.912164Z" + } }, - "execution_count": null, - "source": "client.create_collection(\n collection_name=\"news\",\n vectors_config={\n \"dense\": models.VectorParams(size=DENSE_DIM, distance=models.Distance.COSINE),\n },\n sparse_vectors_config={\n \"sparse\": models.SparseVectorParams(\n modifier=models.Modifier.IDF # required for BM25 scoring\n ),\n },\n)\nprint(\"Collection 'news' created.\")", "outputs": [ { - "output_type": "stream", "name": "stdout", - "text": "Collection 'news' created.\n" + "output_type": "stream", + "text": [ + "Collection 'news' created.\n" + ] } + ], + "source": [ + "client.create_collection(\n", + " collection_name=\"news\",\n", + " vectors_config={\n", + " \"dense\": models.VectorParams(size=DENSE_DIM, distance=models.Distance.COSINE),\n", + " },\n", + " sparse_vectors_config={\n", + " \"sparse\": models.SparseVectorParams(\n", + " # Required for correct BM25 scoring\n", + " modifier=models.Modifier.IDF\n", + " ),\n", + " },\n", + ")\n", + "print(\"Collection 'news' created.\")" ] }, { "cell_type": "markdown", - "source": "### Question 2: What must the system filter on?\n\nFrom the brief: country, topic, date range, and source. These are hard rules, not similarity signals. An analyst scoping to \"Vietnam, last seven days\" means exactly that. Hard rules go in the payload, and every field we filter on gets a payload index.\n\nThe schema, decided now, before ingestion:\n\n```yaml\npayload:\n country: string # indexed\n language: string # indexed\n topic: string # indexed\n source: string # indexed\n tenant_id: string # indexed, and marked as a tenant field\n published_at: datetime # indexed\n summary: string # not indexed: returned, never filtered\n```\n\nCreate the indexes now too, in the same setup step, before a single point is uploaded. This ordering is not a style preference. Qdrant extends the HNSW graph with extra edges derived from indexed payload values, and it can only add those edges for indexes that already exist when the graph is built. Create a payload index after ingesting and you have to rebuild the HNSW index to get any benefit from it.\n\nNote `tenant_id` is in this list even though multitenancy does not come up until Section 3. That is the point: it is nearly free now and expensive later.", - "metadata": { - "id": "FMPd3W2XyygN" - } + "id": "90208c76", + "metadata": {}, + "source": [ + "### Question 2: What Must the System Filter On?\n", + "\n", + "From the brief: country, topic, date range, and source. These are hard rules, not ranking signals. An analyst scoping to \"Vietnam, last seven days\" means exactly that. Hard rules go in the payload, and every field you filter on gets a payload index.\n", + "\n", + "The schema, decided now, before ingestion:\n", + "\n", + "```\n", + "payload:\n", + " country string (indexed)\n", + " topic string (indexed)\n", + " source string (indexed)\n", + " published_at datetime (indexed)\n", + " headline string (returned, never filtered)\n", + " lead string (returned, never filtered)\n", + " body string (returned, never filtered)\n", + "```\n", + "\n", + "That is a knowledge-layer decision (what to store) and an indexing-layer decision (what to index).\n", + "\n", + "The next cell prints a warning: `Payload indexes have no effect in the local Qdrant`. That is expected and it is a local-mode artifact, not a mistake. Creating the indexes before ingestion is the correct habit, and on a real cluster it is what makes filtered search fast." + ] }, { "cell_type": "code", + "execution_count": 4, + "id": "aae7fcb1", "metadata": { - "id": "demkvdajgwb2", - "colab": { - "base_uri": "https://localhost:8080/" - }, - "outputId": "924551df" + "execution": { + "iopub.execute_input": "2026-08-11T09:29:10.914754Z", + "iopub.status.busy": "2026-08-11T09:29:10.914578Z", + "iopub.status.idle": "2026-08-11T09:29:10.920518Z", + "shell.execute_reply": "2026-08-11T09:29:10.919497Z" + } }, - "execution_count": null, - "source": "with warnings.catch_warnings():\n warnings.simplefilter(\"ignore\") # local mode warns that indexes do nothing here\n\n for field in [\"country\", \"language\", \"topic\", \"source\"]:\n client.create_payload_index(\n collection_name=\"news\",\n field_name=field,\n field_schema=models.PayloadSchemaType.KEYWORD,\n )\n\n client.create_payload_index(\n collection_name=\"news\",\n field_name=\"published_at\",\n field_schema=models.PayloadSchemaType.DATETIME,\n )\n\n # is_tenant tells Qdrant this field identifies tenants, so it can keep each\n # tenant's data close together on disk. Supported for keyword and uuid.\n client.create_payload_index(\n collection_name=\"news\",\n field_name=\"tenant_id\",\n field_schema=models.KeywordIndexParams(\n type=models.KeywordIndexType.KEYWORD,\n is_tenant=True,\n ),\n )\n\nprint(\"Payload indexes created before ingestion: country, language, topic, source, published_at, tenant_id.\")", "outputs": [ { - "output_type": "stream", "name": "stdout", - "text": "Payload indexes created before ingestion: country, language, topic, source, published_at, tenant_id.\n" + "output_type": "stream", + "text": [ + "Payload indexes created for country, topic, source, published_at.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/tmp/ipykernel_576/4200797916.py:2: UserWarning: Payload indexes have no effect in the local Qdrant. Please use server Qdrant if you need payload indexes.\n", + " client.create_payload_index(\n" + ] } + ], + "source": [ + "for field in [\"country\", \"topic\", \"source\"]:\n", + " client.create_payload_index(\n", + " collection_name=\"news\",\n", + " field_name=field,\n", + " field_schema=models.PayloadSchemaType.KEYWORD,\n", + " )\n", + "\n", + "client.create_payload_index(\n", + " collection_name=\"news\",\n", + " field_name=\"published_at\",\n", + " field_schema=models.PayloadSchemaType.DATETIME,\n", + ")\n", + "print(\"Payload indexes created for country, topic, source, published_at.\")" ] }, { "cell_type": "markdown", - "source": "Skipping an index costs more than a slow scan. The payload index is also what lets Qdrant estimate how many points a filter will match, and that estimate is what the query planner uses to choose a strategy at all. Without it the planner is guessing, and it can fall back to comparing the query against every vector in the collection.\n\nBecause the query still returns results, just slowly, a missing index can sit in production unnoticed for months. Strict mode closes that gap: set `unindexed_filtering_retrieve` to `false` and Qdrant rejects any query that filters on an unindexed field instead of quietly degrading. Qdrant Cloud applies this by default.", - "metadata": { - "id": "CDfgvqZkb6Gj" - } - }, - { - "cell_type": "markdown", - "source": "### Question 3: What is the workload shape?\n\nMillions of articles, text only for now, arriving continuously. Analysts expect this morning's news to be searchable this morning.\n\n**Decisions:** one collection, and continuous upserts rather than periodic rebuilds. Two things still need designing. The initial backfill of millions of articles is a bulk load, not a stream, so batch the upserts and consider disabling indexing for the duration so the optimizer builds the graph once at the end. And Qdrant does index as it ingests, but not instantly: the optimizer builds an HNSW index for a segment only once that segment passes the indexing threshold, and unindexed segments are served by full scan in the meantime.\n\nHere is a small, multilingual sample that stands in for that stream. Note the mix of languages and the exact tokens (a ticker, a ship name) hiding in the text.", - "metadata": { - "id": "HbG81RqwkVHW" - } + "id": "b1a0a376", + "metadata": {}, + "source": [ + "### Question 3: What Is the Workload Shape?\n", + "\n", + "Millions of articles, text only, arriving continuously, and analysts expect this morning's news to be searchable this morning.\n", + "\n", + "**Decisions:** one collection, and continuous upserts rather than periodic rebuilds. *(Storage and knowledge layers.)*\n", + "\n", + "Here is a small sample standing in for that stream. Note the exact tokens hiding in the text: a ticker and a ship name." + ] }, { "cell_type": "code", + "execution_count": 5, + "id": "6ae895a9", "metadata": { - "id": "unY90XYToGwb", - "colab": { - "base_uri": "https://localhost:8080/" - }, - "outputId": "044bf7ef" + "execution": { + "iopub.execute_input": "2026-08-11T09:29:10.922347Z", + "iopub.status.busy": "2026-08-11T09:29:10.922185Z", + "iopub.status.idle": "2026-08-11T09:29:10.931652Z", + "shell.execute_reply": "2026-08-11T09:29:10.930523Z" + } }, - "execution_count": null, - "source": "articles = [\n {\n \"country\": \"VN\", \"language\": \"vi\", \"topic\": \"shipping\", \"source\": \"reuters\",\n \"published_at\": \"2026-07-20T08:00:00Z\", \"tenant_id\": \"asia-desk\",\n \"summary\": \"Tac nghen cang tai Hai Phong khi luong hang tang manh.\",\n \"text\": \"Cang Hai Phong o Viet Nam bi tac nghen nghiem trong trong tuan nay khi luong container tang vot, gay cham tre cho tau thuyen.\",\n },\n {\n \"country\": \"JP\", \"language\": \"ja\", \"topic\": \"shipping\", \"source\": \"nikkei\",\n \"published_at\": \"2026-07-21T09:30:00Z\", \"tenant_id\": \"asia-desk\",\n \"summary\": \"東南アジアの港湾混雑が輸送を遅らせている。\",\n \"text\": \"東南アジアの主要港で混雑が深刻化し、コンテナ船の到着が遅れている。特にベトナムとタイの港で影響が大きい。\",\n },\n {\n \"country\": \"CN\", \"language\": \"zh\", \"topic\": \"logistics\", \"source\": \"caixin\",\n \"published_at\": \"2026-07-19T11:00:00Z\", \"tenant_id\": \"asia-desk\",\n \"summary\": \"上海港物流吞吐量创新高。\",\n \"text\": \"上海港本月物流吞吐量创下新高，港口运营商正在扩大堆场以缓解拥堵压力。\",\n },\n {\n \"country\": \"DK\", \"language\": \"en\", \"topic\": \"markets\", \"source\": \"reuters\",\n \"published_at\": \"2026-07-22T07:15:00Z\", \"tenant_id\": \"europe-desk\",\n \"summary\": \"Maersk shares move on delisting speculation.\",\n \"text\": \"Shares tied to the ticker MAERSK-B.CO moved sharply amid speculation about a possible delisting of a subsidiary vehicle.\",\n },\n {\n \"country\": \"SG\", \"language\": \"en\", \"topic\": \"shipping\", \"source\": \"straits-times\",\n \"published_at\": \"2026-07-22T10:45:00Z\", \"tenant_id\": \"asia-desk\",\n \"summary\": \"Vessel Ever Given reroutes through Singapore.\",\n \"text\": \"The container ship Ever Given was rerouted through the Port of Singapore this week to avoid congestion further north.\",\n },\n {\n \"country\": \"US\", \"language\": \"en\", \"topic\": \"markets\", \"source\": \"press-release-wire\",\n \"published_at\": \"2026-06-30T12:00:00Z\", \"tenant_id\": \"americas-desk\",\n \"summary\": \"Company announces quarterly logistics earnings.\",\n \"text\": \"A logistics operator announced quarterly earnings, citing steady demand across North American freight corridors.\",\n },\n {\n \"country\": \"TH\", \"language\": \"th\", \"topic\": \"shipping\", \"source\": \"bangkok-post\",\n \"published_at\": \"2026-07-18T06:20:00Z\", \"tenant_id\": \"asia-desk\",\n \"summary\": \"ความแออัดที่ท่าเรือแหลมฉบังเพิ่มขึ้น\",\n \"text\": \"ท่าเรือแหลมฉบังของไทยเผชิญความแออัดมากขึ้นเนื่องจากปริมาณตู้คอนเทนเนอร์ที่เพิ่มสูงขึ้นในสัปดาห์นี้\",\n },\n {\n \"country\": \"DE\", \"language\": \"de\", \"topic\": \"logistics\", \"source\": \"handelsblatt\",\n \"published_at\": \"2026-07-15T14:10:00Z\", \"tenant_id\": \"europe-desk\",\n \"summary\": \"Hamburger Hafen meldet Verzoegerungen.\",\n \"text\": \"Der Hamburger Hafen meldet Verzoegerungen bei der Abfertigung, da die Zahl der ankommenden Schiffe deutlich gestiegen ist.\",\n },\n]\nprint(f\"{len(articles)} sample articles across {len(set(a['language'] for a in articles))} languages.\")", "outputs": [ { - "output_type": "stream", "name": "stdout", - "text": "8 sample articles across 6 languages.\n" + "output_type": "stream", + "text": [ + "9 sample articles across 8 countries.\n" + ] } + ], + "source": [ + "articles = [\n", + " {\n", + " \"headline\": \"Hai Phong container backlog grows for a third week\",\n", + " \"lead\": \"Congestion at the northern Vietnamese port deepened as container volumes climbed, delaying vessel departures.\",\n", + " \"body\": \"Terminal operators said yard utilization reached the high nineties, forcing arriving vessels to wait at anchor. Freight forwarders reported delays of two to four days on outbound bookings.\",\n", + " \"country\": \"VN\", \"topic\": \"shipping\", \"source\": \"reuters\",\n", + " \"published_at\": \"2026-07-20T08:00:00Z\", \"tenant_id\": \"asia-desk\",\n", + " },\n", + " {\n", + " \"headline\": \"Southeast Asian port congestion slows regional transport\",\n", + " \"lead\": \"Delays at major Southeast Asian ports are pushing back container ship arrivals across the region.\",\n", + " \"body\": \"Analysts point to a combination of higher volumes and reduced berth availability. Shipping lines have added waiting time to their schedules.\",\n", + " \"country\": \"JP\", \"topic\": \"shipping\", \"source\": \"nikkei\",\n", + " \"published_at\": \"2026-07-21T09:30:00Z\", \"tenant_id\": \"asia-desk\",\n", + " },\n", + " {\n", + " \"headline\": \"Shanghai port sets a monthly logistics throughput record\",\n", + " \"lead\": \"The Port of Shanghai handled record volumes this month as operators expanded yard capacity.\",\n", + " \"body\": \"Terminal managers credited longer gate hours and a new rail link for the improvement. Throughput growth has outpaced the national average for six consecutive quarters.\",\n", + " \"country\": \"CN\", \"topic\": \"logistics\", \"source\": \"caixin\",\n", + " \"published_at\": \"2026-07-19T11:00:00Z\", \"tenant_id\": \"asia-desk\",\n", + " },\n", + " {\n", + " \"headline\": \"Shipping group unit faces delisting speculation\",\n", + " \"lead\": \"Shares tied to MAERSK-B.CO swung on speculation that a subsidiary vehicle could be delisted.\",\n", + " \"body\": \"Traders cited an unusual volume spike in the final hour of trading. The company declined to comment.\",\n", + " \"country\": \"DK\", \"topic\": \"markets\", \"source\": \"reuters\",\n", + " \"published_at\": \"2026-07-22T07:15:00Z\", \"tenant_id\": \"europe-desk\",\n", + " },\n", + " {\n", + " \"headline\": \"Carrier unit faces delisting speculation\",\n", + " \"lead\": \"Shares tied to HLAG-D.DE swung on speculation that a subsidiary vehicle could be delisted.\",\n", + " \"body\": \"Traders cited unusual volume late in the session. The company declined to comment.\",\n", + " \"country\": \"DE\", \"topic\": \"markets\", \"source\": \"handelsblatt\",\n", + " \"published_at\": \"2026-07-22T08:05:00Z\", \"tenant_id\": \"europe-desk\",\n", + " },\n", + " {\n", + " \"headline\": \"Ever Given rerouted through Singapore to avoid delays\",\n", + " \"lead\": \"The container ship Ever Given was rerouted through the Port of Singapore this week.\",\n", + " \"body\": \"The diversion adds roughly two days to the voyage but avoids a longer wait at anchor. Singapore has absorbed a growing share of regional transhipment traffic.\",\n", + " \"country\": \"SG\", \"topic\": \"shipping\", \"source\": \"straits-times\",\n", + " \"published_at\": \"2026-07-22T10:45:00Z\", \"tenant_id\": \"asia-desk\",\n", + " },\n", + " {\n", + " \"headline\": \"Logistics operator reports steady quarterly earnings\",\n", + " \"lead\": \"A North American logistics operator announced quarterly earnings, citing steady freight demand.\",\n", + " \"body\": \"Revenue was broadly flat against the same quarter last year. The company reiterated its full year guidance.\",\n", + " \"country\": \"US\", \"topic\": \"markets\", \"source\": \"press-release-wire\",\n", + " \"published_at\": \"2026-06-30T12:00:00Z\", \"tenant_id\": \"americas-desk\",\n", + " },\n", + " {\n", + " \"headline\": \"Laem Chabang congestion rises on higher container volumes\",\n", + " \"lead\": \"Thailand's largest port is handling more containers than its yards were sized for, lengthening waits.\",\n", + " \"body\": \"Operators have extended gate hours and added weekend shifts. Exporters report longer lead times on shipments to Europe.\",\n", + " \"country\": \"TH\", \"topic\": \"shipping\", \"source\": \"bangkok-post\",\n", + " \"published_at\": \"2026-07-18T06:20:00Z\", \"tenant_id\": \"asia-desk\",\n", + " },\n", + " {\n", + " \"headline\": \"Hamburg reports handling delays as arrivals climb\",\n", + " \"lead\": \"The Port of Hamburg is reporting delays in cargo handling as the number of arriving ships rises.\",\n", + " \"body\": \"Dock labor shortages have compounded the problem. Rail connections inland are running close to capacity.\",\n", + " \"country\": \"DE\", \"topic\": \"logistics\", \"source\": \"handelsblatt\",\n", + " \"published_at\": \"2026-07-15T14:10:00Z\", \"tenant_id\": \"europe-desk\",\n", + " },\n", + "]\n", + "print(len(articles), \"sample articles across\", len(set(a[\"country\"] for a in articles)), \"countries.\")" ] }, { "cell_type": "markdown", - "source": "Embed each article into both a dense and a sparse vector, and upsert them as points. In production this same loop runs continuously as articles arrive.", - "metadata": { - "id": "7wnommM30DOX" - } + "id": "53221a3f", + "metadata": {}, + "source": [ + "One knowledge-layer decision hides in the ingestion step: **what you embed matters as much as how you search it.**\n", + "\n", + "A news article runs 800 words, and the ticker from Question 1 is one token inside it. Embed the whole body and that token is averaged into a vector about shipping in general. Embed the headline and the lead, keep the full text in the payload, and the dense vector stays about one story while the sparse vector still sees every token.\n", + "\n", + "Two of these articles are deliberately hard: they have near-identical headlines and leads, and differ only in the ticker. Watch what that does to the dense retriever later." + ] }, { "cell_type": "code", + "execution_count": 6, + "id": "694b2102", "metadata": { - "id": "fO4WUDlWuX8y", - "colab": { - "base_uri": "https://localhost:8080/" - }, - "outputId": "6fb222ec" + "execution": { + "iopub.execute_input": "2026-08-11T09:29:10.933049Z", + "iopub.status.busy": "2026-08-11T09:29:10.932833Z", + "iopub.status.idle": "2026-08-11T09:29:11.122256Z", + "shell.execute_reply": "2026-08-11T09:29:11.121047Z" + } }, - "execution_count": null, - "source": "def to_sparse_vector(text, is_query=False):\n \"\"\"Convert text to a Qdrant SparseVector using BM25.\n BM25 scores queries and documents slightly differently, so we use\n query_embed for queries and embed for documents.\"\"\"\n emb = next(sparse_model.query_embed(text)) if is_query else next(sparse_model.embed([text]))\n return models.SparseVector(indices=emb.indices.tolist(), values=emb.values.tolist())\n\npoints = []\nfor i, art in enumerate(articles):\n dense_vec = next(dense_model.embed([art[\"text\"]])).tolist()\n sparse_vec = to_sparse_vector(art[\"text\"])\n points.append(\n models.PointStruct(\n id=i,\n vector={\"dense\": dense_vec, \"sparse\": sparse_vec},\n payload=art,\n )\n )\n\nclient.upsert(collection_name=\"news\", points=points)\nprint(f\"Upserted {len(points)} points. Collection count:\", client.count(\"news\").count)", "outputs": [ { + "name": "stdout", "output_type": "stream", + "text": [ + "Upserted" + ] + }, + { "name": "stdout", - "text": "Upserted 8 points. Collection count: 8\n" + "output_type": "stream", + "text": [ + " 9 points. Collection count: 9\n" + ] } + ], + "source": [ + "def to_sparse_vector(text, is_query=False):\n", + " \"\"\"Convert text to a Qdrant SparseVector using BM25.\n", + " BM25 scores queries and documents differently, so queries use query_embed.\"\"\"\n", + " emb = next(sparse_model.query_embed(text)) if is_query else next(sparse_model.embed([text]))\n", + " return models.SparseVector(indices=emb.indices.tolist(), values=emb.values.tolist())\n", + "\n", + "points = []\n", + "for i, art in enumerate(articles):\n", + " # The knowledge-layer decision, made concrete: embed headline and lead only.\n", + " embed_text = art[\"headline\"] + \". \" + art[\"lead\"]\n", + " points.append(\n", + " models.PointStruct(\n", + " id=i,\n", + " vector={\n", + " \"dense\": next(dense_model.embed([embed_text])).tolist(),\n", + " \"sparse\": to_sparse_vector(embed_text),\n", + " },\n", + " payload=art,\n", + " )\n", + " )\n", + "\n", + "client.upsert(collection_name=\"news\", points=points)\n", + "print(\"Upserted\", len(points), \"points. Collection count:\", client.count(\"news\").count)" ] }, { "cell_type": "markdown", - "source": "### Question 4: What does the retrieval pipeline look like?\n\nStart with the simplest pipeline that fits the query analysis: hybrid (from Question 1) plus filters (from Question 2), fused by rank. No reranker yet. Add complexity when data proves it is needed, never in advance.\n\nThe `query_points` call below does it all in one round trip:\n\n- Two `Prefetch` branches, one per named vector, each pulling 50 candidates.\n- `RrfQuery` to merge the two candidate lists into one ranking.\n- The filter passed **into each prefetch**, so both retrievers only ever consider valid points.\n\nThat last point is the one to slow down on, and Section 3 breaks it on purpose to show why.\n\nNotice the payoff of the multilingual model: we query in **English** and retrieve articles written in Japanese, Vietnamese, Thai, and more, with no translation step.", - "metadata": { - "id": "jZ8OvIia8OG5" - } + "id": "c90fd7ac", + "metadata": {}, + "source": [ + "### Question 4: What Does the Retrieval Pipeline Look Like?\n", + "\n", + "Start with the simplest pipeline that fits the query analysis: hybrid, from Question 1, plus filters, from Question 2, fused with Reciprocal Rank Fusion. No reranker yet.\n", + "\n", + "One `query_points` call does all of it:\n", + "\n", + "- Two `Prefetch` branches, one per named vector, each pulling 50 candidates.\n", + "- `RrfQuery`, which merges the two candidate lists by rank rather than by score, so it does not matter that cosine similarity and BM25 live on different scales.\n", + "- The filter **inside each `Prefetch`**, so both retrievers search only the valid subset. Where that filter goes matters more than it looks, and the next section shows what happens when it goes somewhere else." + ] }, { "cell_type": "code", + "execution_count": 7, + "id": "6eb185bc", "metadata": { - "id": "A2ZnpsoKqzi4", - "colab": { - "base_uri": "https://localhost:8080/" - }, - "outputId": "579e6e88" + "execution": { + "iopub.execute_input": "2026-08-11T09:29:11.124574Z", + "iopub.status.busy": "2026-08-11T09:29:11.123873Z", + "iopub.status.idle": "2026-08-11T09:29:11.145783Z", + "shell.execute_reply": "2026-08-11T09:29:11.144755Z" + } }, - "execution_count": null, - "source": "def search(query_text, query_filter=None, limit=5):\n \"\"\"Hybrid search: dense + sparse prefetch, fused by rank, filter in each branch.\"\"\"\n dense_q = next(dense_model.query_embed(query_text)).tolist()\n sparse_q = to_sparse_vector(query_text, is_query=True)\n response = client.query_points(\n collection_name=\"news\",\n prefetch=[\n models.Prefetch(query=dense_q, using=\"dense\",\n filter=query_filter, limit=50),\n models.Prefetch(query=sparse_q, using=\"sparse\",\n filter=query_filter, limit=50),\n ],\n query=models.RrfQuery(rrf=models.Rrf()),\n limit=limit,\n )\n return response.points\n\ndef show(results):\n if not results:\n print(\" (no results)\")\n for r in results:\n p = r.payload\n print(f\"[{r.score:.4f}] {p['country']}/{p['language']:<2} {p['topic']:<9} | {p['summary']}\")\n\nprint(\"Query (English): 'port congestion in Southeast Asia'\\n\")\nshow(search(\"port congestion in Southeast Asia\"))", "outputs": [ { + "name": "stdout", "output_type": "stream", + "text": [ + "Query: 'port congestion in Southeast Asia'\n" + ] + }, + { "name": "stdout", - "text": "Query (English): 'port congestion in Southeast Asia'\n\n[0.7500] SG/en shipping | Vessel Ever Given reroutes through Singapore.\n[0.5000] JP/ja shipping | 東南アジアの港湾混雑が輸送を遅らせている。\n[0.3333] TH/th shipping | ความแออัดที่ท่าเรือแหลมฉบังเพิ่มขึ้น\n[0.2000] CN/zh logistics | 上海港物流吞吐量创新高。\n[0.1667] DE/de logistics | Hamburger Hafen meldet Verzoegerungen.\n" + "output_type": "stream", + "text": [ + "\n", + "[1.0000] JP shipping nikkei | Southeast Asian port congestion slows regional transport\n", + "[0.5833] TH shipping bangkok-post | Laem Chabang congestion rises on higher container volumes\n", + "[0.5000] SG shipping straits-times | Ever Given rerouted through Singapore to avoid delays\n", + "[0.4500] VN shipping reuters | Hai Phong container backlog grows for a third week\n", + "[0.3667] CN logistics caixin | Shanghai port sets a monthly logistics throughput record\n" + ] } + ], + "source": [ + "def search(query_text, query_filter=None, limit=5):\n", + " \"\"\"Hybrid search: dense + sparse prefetch, fused with RRF.\n", + " The filter goes inside each Prefetch, not on the outer query.\"\"\"\n", + " dense_q = next(dense_model.query_embed(query_text)).tolist()\n", + " sparse_q = to_sparse_vector(query_text, is_query=True)\n", + " response = client.query_points(\n", + " collection_name=\"news\",\n", + " prefetch=[\n", + " models.Prefetch(query=dense_q, using=\"dense\", filter=query_filter, limit=50),\n", + " models.Prefetch(query=sparse_q, using=\"sparse\", filter=query_filter, limit=50),\n", + " ],\n", + " query=models.RrfQuery(rrf=models.Rrf()),\n", + " limit=limit,\n", + " )\n", + " return response.points\n", + "\n", + "\n", + "def show(results):\n", + " for r in results:\n", + " p = r.payload\n", + " print(f\"[{r.score:.4f}] {p['country']} {p['topic']:<9} {p['source']:<18} | {p['headline']}\")\n", + "\n", + "\n", + "print(\"Query: 'port congestion in Southeast Asia'\\n\")\n", + "show(search(\"port congestion in Southeast Asia\"))" ] }, { "cell_type": "markdown", - "source": "Now the exact-token case. A dense-only search would blur `MAERSK-B.CO` into semantically similar finance text. The sparse half of the hybrid locks onto the literal token, and rank fusion pushes the correct article to the top. This is why we chose hybrid from the start.", - "metadata": { - "id": "GbBlaXkDSIuG" - } + "id": "d46f0862", + "metadata": {}, + "source": [ + "### Try It: Why Hybrid, Not Dense Alone\n", + "\n", + "Now the exact-token case, and the reason Question 1 chose hybrid.\n", + "\n", + "Run the same query three ways: dense only, sparse only, and hybrid. Two articles in the collection have near-identical headlines and leads, differing only in the ticker. Look at the **gaps between the scores**, not just the order." + ] }, { "cell_type": "code", + "execution_count": 8, + "id": "2ea22801", "metadata": { - "id": "IWgIprVFIV4B", - "colab": { - "base_uri": "https://localhost:8080/" - }, - "outputId": "988b787f" + "execution": { + "iopub.execute_input": "2026-08-11T09:29:11.147923Z", + "iopub.status.busy": "2026-08-11T09:29:11.147060Z", + "iopub.status.idle": "2026-08-11T09:29:11.180905Z", + "shell.execute_reply": "2026-08-11T09:29:11.180044Z" + } }, - "execution_count": null, - "source": "print(\"Query (exact token): 'MAERSK-B.CO delisting'\\n\")\nshow(search(\"MAERSK-B.CO delisting\"))", "outputs": [ { - "output_type": "stream", "name": "stdout", - "text": "Query (exact token): 'MAERSK-B.CO delisting'\n\n[1.0000] DK/en markets | Maersk shares move on delisting speculation.\n[0.3333] DE/de logistics | Hamburger Hafen meldet Verzoegerungen.\n[0.2500] VN/vi shipping | Tac nghen cang tai Hai Phong khi luong hang tang manh.\n[0.2000] SG/en shipping | Vessel Ever Given reroutes through Singapore.\n[0.1667] TH/th shipping | ความแออัดที่ท่าเรือแหลมฉบังเพิ่มขึ้น\n" + "output_type": "stream", + "text": [ + "DENSE ONLY\n", + " [0.8120] Shares tied to MAERSK-B.CO swung on speculation that a subsidi\n", + " [0.6956] Shares tied to HLAG-D.DE swung on speculation that a subsidiar\n", + " [0.6071] The container ship Ever Given was rerouted through the Port of\n", + "\n", + "SPARSE ONLY (BM25)\n", + " [11.7931] Shares tied to MAERSK-B.CO swung on speculation that a subsidi\n", + " [2.5926] Shares tied to HLAG-D.DE swung on speculation that a subsidiar\n", + "\n", + "HYBRID (RRF over both)\n", + "[1.0000] DK markets reuters | Shipping group unit faces delisting speculation\n", + "[0.6667] DE markets handelsblatt | Carrier unit faces delisting speculation\n", + "[0.2500] SG shipping straits-times | Ever Given rerouted through Singapore to avoid delays\n" + ] } + ], + "source": [ + "QUERY = \"MAERSK-B.CO delisting\"\n", + "dense_q = next(dense_model.query_embed(QUERY)).tolist()\n", + "sparse_q = to_sparse_vector(QUERY, is_query=True)\n", + "\n", + "print(\"DENSE ONLY\")\n", + "for p in client.query_points(\"news\", query=dense_q, using=\"dense\", limit=3).points:\n", + " print(f\" [{p.score:.4f}] {p.payload['lead'][:62]}\")\n", + "\n", + "print(\"\\nSPARSE ONLY (BM25)\")\n", + "for p in client.query_points(\"news\", query=sparse_q, using=\"sparse\", limit=3).points:\n", + " print(f\" [{p.score:.4f}] {p.payload['lead'][:62]}\")\n", + "\n", + "print(\"\\nHYBRID (RRF over both)\")\n", + "show(search(QUERY, limit=3))" ] }, { "cell_type": "markdown", - "source": "Reciprocal Rank Fusion is the right default here because it works from ranks rather than raw scores, so it does not care that cosine similarity and BM25 live on different scales. Two alternatives are worth knowing once you can measure quality: weighted RRF, which favours the stronger retriever via `weights` on `models.Rrf()`, and Distribution-Based Score Fusion, `models.FusionQuery(fusion=models.Fusion.DBSF)`, which normalizes each retriever's score distribution instead. Neither reliably beats the other, so choose with an evaluation set rather than by reputation.", - "metadata": { - "id": "R26qI60ihBeo" - } + "id": "1944b15b", + "metadata": {}, + "source": [ + "Dense puts the right article first, but only just. The correct article scores 0.8120 and the near-identical decoy scores 0.6956, a gap of 0.12, because a ticker is a string with no semantics for a dense model to work with. BM25 scores them 11.79 and 2.59, separating them by more than 4x, because it is matching the literal token.\n", + "\n", + "On nine articles that thin dense margin still lands the right answer. On nine million, with thousands of similar market stories, it is noise. That gap is the argument for hybrid, and it is why Question 1 is the most consequential question in the list." + ] }, { "cell_type": "markdown", - "source": "### Question 5: What are the deployment constraints?\n\nA research firm with a small engineering team, no data residency restrictions, and a \"please do not page us at night\" budget points to a **managed** deployment. The key idea: the design and the deployment mode are independent decisions. Everything above runs unchanged whether the backend is this in-process client, a Docker container, or Qdrant Cloud.\n\n### The design on one page\n\n| Question | Answer for This System | Layer |\n|----------|------------------------|-------|\n| Query type | Mixed semantic + exact, so hybrid with Reciprocal Rank Fusion | Query |\n| Filter scope | country, topic, source, date, indexed before ingestion | Knowledge, indexing |\n| Workload shape | Millions of text chunks: bulk backfill, then continuous ingestion | Storage, knowledge |\n| Pipeline | Hybrid + per-prefetch filters, multilingual model, no reranker yet | Query, knowledge |\n| Deployment | Managed, design independent of the choice | Distribution |\n\n### Key insight\n\nEvery piece of this design already appeared in an earlier module: the Module 2 pipeline, the Module 3 hybrid pattern, and a payload schema. What is new is the order of operations. You decided what to filter on and created the indexes before the first point went in. That ordering is the difference between a system that scales and one that gets re-ingested three times.", - "metadata": { - "id": "ePW3jSOi27n7" - } + "id": "2b8713e6", + "metadata": {}, + "source": [ + "### Question 5: What Are the Deployment Constraints?\n", + "\n", + "A research firm with a small engineering team, no data residency restrictions, and a \"please do not page us at night\" budget points to a managed deployment. The design and the deployment mode are independent decisions: everything above runs unchanged against this in-memory client, a Docker container, or Qdrant Cloud. Section 5 lays out the full option space. *(Distribution layer.)*\n", + "\n", + "### The Design on One Page\n", + "\n", + "| Question | Answer for this system | Layer |\n", + "|---|---|---|\n", + "| Query type | Mixed semantic and exact, so hybrid with Reciprocal Rank Fusion | Query |\n", + "| Filter scope | country, topic, source, date, indexed before ingestion | Knowledge, indexing |\n", + "| Workload shape | Millions of text chunks, continuous ingestion | Storage, knowledge |\n", + "| Pipeline | Hybrid with per-prefetch filters, headline and lead embedded, no reranker yet | Query, knowledge |\n", + "| Deployment | Managed, and independent of the design | Distribution |" + ] }, { "cell_type": "markdown", - "source": "## 3. Filtering\n\nFiltering is the feature that decides whether your results are **correct**, not an accessory to similarity.\n\n### How Qdrant combines filters with vector search\n\nThe naive approach is post-filtering: retrieve the top K by similarity, then discard whatever fails the filter. With a selective filter, say one country out of 200, the top K can contain zero valid results, and there is no K that guarantees correctness.\n\nQdrant does not work that way. A query planner chooses a strategy for each segment, based on the estimated cardinality of the filter and which payload indexes exist:\n\n- When the filter matches a large share of the collection, Qdrant walks the HNSW graph as usual and skips points that fail the filter during traversal.\n- When the filter matches very few points, it can skip the graph entirely and retrieve through the payload index, which is cheaper at that selectivity.\n- When a segment is small enough, a full scan wins outright.\n\nThe middle ground is the hard case, because a strict filter can disconnect the HNSW graph and leave relevant points unreachable. Qdrant handles it by extending the graph with additional edges derived from indexed payload values, which is why the indexes had to exist before ingestion. Those extra edges are added per index, not per combination of indexes, so two or more highly selective filters can still reach a disconnected component. The ACORN algorithm (v1.16) exists for that case.\n\nSo: filtering on every query is safe and fast, as long as every filtered field is indexed and the indexes existed before ingestion. It is not free, and combinations of highly selective filters deserve measurement rather than assumption.\n\n### The filter toolbox\n\n| Condition | Logic | Example |\n|-----------|-------|---------|\n| must | AND, all conditions true | country = VN AND topic = shipping |\n| should | OR, at least one true (`min_should` sets a higher minimum) | topic = shipping OR topic = logistics |\n| must_not | Exclude matches | Exclude source = press-release-wire |\n| Range | Numeric or datetime bounds | published_at within the last seven days |\n| MatchAny | Value in a set | language in [ja, zh, th, vi] |\n| Geo | Radius, bounding box, or polygon | Events within 100 km of a port |\n\nThe cells below run several of these against the news collection.", - "metadata": { - "id": "yE9ZkAFvmtSb" - } + "id": "b2b37ddd", + "metadata": {}, + "source": [ + "## 3. Filtering\n", + "\n", + "Filtering is the feature that decides whether your results are correct, so it deserves more than a passing mention.\n", + "\n", + "The naive approach is post-filtering: retrieve the top K by similarity, then throw away whatever fails the filter. With a selective filter, one country out of 200, the top K can contain zero valid results, and there is no K that guarantees correctness. Qdrant does not work that way. A query planner picks a strategy per segment based on how many points it estimates the filter will match, and on which payload indexes exist.\n", + "\n", + "### The Filter Toolbox\n", + "\n", + "| Condition | Logic | Example |\n", + "|---|---|---|\n", + "| must | AND, all conditions true | country = VN AND topic = shipping |\n", + "| should | OR, at least one true | topic = shipping OR topic = logistics |\n", + "| must_not | Exclude matches | Exclude source = press-release-wire |\n", + "| Range | Numeric or datetime bounds | published_at within the last seven days |\n", + "| MatchAny | Value in a set | source in [reuters, nikkei, caixin] |\n", + "| Geo | Radius, bounding box, or polygon | Events within 100 km of a port |\n", + "\n", + "The cells below run several of these against the news collection." + ] }, { "cell_type": "code", + "execution_count": 9, + "id": "d7df4beb", "metadata": { - "id": "IBbuMFd9W8gq", - "colab": { - "base_uri": "https://localhost:8080/" - }, - "outputId": "0c426ed4" + "execution": { + "iopub.execute_input": "2026-08-11T09:29:11.182656Z", + "iopub.status.busy": "2026-08-11T09:29:11.182496Z", + "iopub.status.idle": "2026-08-11T09:29:11.198817Z", + "shell.execute_reply": "2026-08-11T09:29:11.197542Z" + } }, - "execution_count": null, - "source": "# must: AND. Only Vietnamese shipping news.\nf_must = models.Filter(\n must=[\n models.FieldCondition(key=\"country\", match=models.MatchValue(value=\"VN\")),\n models.FieldCondition(key=\"topic\", match=models.MatchValue(value=\"shipping\")),\n ]\n)\nprint(\"must (country=VN AND topic=shipping):\")\nshow(search(\"port congestion\", query_filter=f_must))", "outputs": [ { - "output_type": "stream", "name": "stdout", - "text": "must (country=VN AND topic=shipping):\n" - }, - { "output_type": "stream", - "name": "stdout", - "text": "[0.5000] VN/vi shipping | Tac nghen cang tai Hai Phong khi luong hang tang manh.\n" + "text": [ + "must (country=VN AND topic=shipping):\n", + "[1.0000] VN shipping reuters | Hai Phong container backlog grows for a third week\n" + ] } + ], + "source": [ + "# must: AND. Only Vietnamese shipping news.\n", + "f_must = models.Filter(\n", + " must=[\n", + " models.FieldCondition(key=\"country\", match=models.MatchValue(value=\"VN\")),\n", + " models.FieldCondition(key=\"topic\", match=models.MatchValue(value=\"shipping\")),\n", + " ]\n", + ")\n", + "print(\"must (country=VN AND topic=shipping):\")\n", + "show(search(\"port congestion\", query_filter=f_must))" ] }, { "cell_type": "code", + "execution_count": 10, + "id": "0f11523f", "metadata": { - "id": "j4JRfdQAe6NX", - "colab": { - "base_uri": "https://localhost:8080/" - }, - "outputId": "1d1d758e" + "execution": { + "iopub.execute_input": "2026-08-11T09:29:11.201223Z", + "iopub.status.busy": "2026-08-11T09:29:11.201034Z", + "iopub.status.idle": "2026-08-11T09:29:11.220026Z", + "shell.execute_reply": "2026-08-11T09:29:11.218976Z" + } }, - "execution_count": null, - "source": "# should: OR. Either shipping or logistics topics.\nf_should = models.Filter(\n should=[\n models.FieldCondition(key=\"topic\", match=models.MatchValue(value=\"shipping\")),\n models.FieldCondition(key=\"topic\", match=models.MatchValue(value=\"logistics\")),\n ]\n)\nprint(\"should (topic=shipping OR topic=logistics):\")\nshow(search(\"congestion at ports\", query_filter=f_should))", "outputs": [ { - "output_type": "stream", "name": "stdout", - "text": "should (topic=shipping OR topic=logistics):\n" - }, - { "output_type": "stream", - "name": "stdout", - "text": "[0.7000] SG/en shipping | Vessel Ever Given reroutes through Singapore.\n[0.5000] JP/ja shipping | 東南アジアの港湾混雑が輸送を遅らせている。\n[0.3333] CN/zh logistics | 上海港物流吞吐量创新高。\n[0.2500] TH/th shipping | ความแออัดที่ท่าเรือแหลมฉบังเพิ่มขึ้น\n[0.1667] DE/de logistics | Hamburger Hafen meldet Verzoegerungen.\n" + "text": [ + "should (topic=shipping OR topic=logistics):\n", + "[1.0000] JP shipping nikkei | Southeast Asian port congestion slows regional transport\n", + "[0.5833] TH shipping bangkok-post | Laem Chabang congestion rises on higher container volumes\n", + "[0.4762] DE logistics handelsblatt | Hamburg reports handling delays as arrivals climb\n", + "[0.4167] VN shipping reuters | Hai Phong container backlog grows for a third week\n", + "[0.4000] CN logistics caixin | Shanghai port sets a monthly logistics throughput record\n" + ] } + ], + "source": [ + "# should: OR. Either shipping or logistics.\n", + "f_should = models.Filter(\n", + " should=[\n", + " models.FieldCondition(key=\"topic\", match=models.MatchValue(value=\"shipping\")),\n", + " models.FieldCondition(key=\"topic\", match=models.MatchValue(value=\"logistics\")),\n", + " ]\n", + ")\n", + "print(\"should (topic=shipping OR topic=logistics):\")\n", + "show(search(\"congestion at ports\", query_filter=f_should))" ] }, { "cell_type": "code", + "execution_count": 11, + "id": "62f84634", "metadata": { - "id": "qiGogkdmtsVr", - "colab": { - "base_uri": "https://localhost:8080/" - }, - "outputId": "825040b8" + "execution": { + "iopub.execute_input": "2026-08-11T09:29:11.222218Z", + "iopub.status.busy": "2026-08-11T09:29:11.221564Z", + "iopub.status.idle": "2026-08-11T09:29:11.242740Z", + "shell.execute_reply": "2026-08-11T09:29:11.241811Z" + } }, - "execution_count": null, - "source": "# must_not + Range + MatchAny combined, evaluated together while the search runs.\nf_combined = models.Filter(\n must=[\n models.FieldCondition(\n key=\"published_at\",\n range=models.DatetimeRange(gte=\"2026-07-15T00:00:00Z\"),\n ),\n models.FieldCondition(\n key=\"language\",\n match=models.MatchAny(any=[\"ja\", \"zh\", \"th\", \"vi\"]),\n ),\n ],\n must_not=[\n models.FieldCondition(key=\"source\", match=models.MatchValue(value=\"press-release-wire\")),\n ],\n)\nprint(\"recent AND asian-language AND not a press-release wire:\")\nshow(search(\"port congestion in Southeast Asia\", query_filter=f_combined))", "outputs": [ { - "output_type": "stream", "name": "stdout", - "text": "recent AND asian-language AND not a press-release wire:\n[0.5000] JP/ja shipping | 東南アジアの港湾混雑が輸送を遅らせている。\n[0.3333] TH/th shipping | ความแออัดที่ท่าเรือแหลมฉบังเพิ่มขึ้น\n[0.2500] CN/zh logistics | 上海港物流吞吐量创新高。\n[0.2000] VN/vi shipping | Tac nghen cang tai Hai Phong khi luong hang tang manh.\n" + "output_type": "stream", + "text": [ + "published since July 19, from a wire we trust, no press releases:\n", + "[1.0000] JP shipping nikkei | Southeast Asian port congestion slows regional transport\n", + "[0.5833] VN shipping reuters | Hai Phong container backlog grows for a third week\n", + "[0.5333] SG shipping straits-times | Ever Given rerouted through Singapore to avoid delays\n", + "[0.4500] CN logistics caixin | Shanghai port sets a monthly logistics throughput record\n", + "[0.1667] DK markets reuters | Shipping group unit faces delisting speculation\n" + ] } + ], + "source": [ + "# must_not + Range + MatchAny, all evaluated together during the search.\n", + "f_combined = models.Filter(\n", + " must=[\n", + " models.FieldCondition(\n", + " key=\"published_at\",\n", + " range=models.DatetimeRange(gte=\"2026-07-19T00:00:00Z\"),\n", + " ),\n", + " models.FieldCondition(\n", + " key=\"source\",\n", + " match=models.MatchAny(any=[\"reuters\", \"nikkei\", \"caixin\", \"straits-times\"]),\n", + " ),\n", + " ],\n", + " must_not=[\n", + " models.FieldCondition(key=\"source\", match=models.MatchValue(value=\"press-release-wire\")),\n", + " ],\n", + ")\n", + "print(\"published since July 19, from a wire we trust, no press releases:\")\n", + "show(search(\"port congestion in Southeast Asia\", query_filter=f_combined))" ] }, { "cell_type": "markdown", - "source": "### Common mistake: filters in the wrong place\n\nEvery filtered search above worked because `search()` passes the filter into each `Prefetch`. Move it to a top-level `query_filter` and it silently stops constraining the retrievers.\n\nWhenever a query has at least one prefetch, Qdrant runs the prefetches first and applies the main query to their results. A top-level filter therefore never reaches them: each searches the whole collection, returns its 50 candidates, and the filter only trims the fused set at the end. That is post-filtering, with exactly the failure mode described above.\n\nBelow, the same selective filter runs both ways. The correct version returns only Vietnamese shipping news. The broken version does not, and nothing raises an error.", - "metadata": { - "id": "lD5gYAEZyFQ8" - } + "id": "456ca1df", + "metadata": {}, + "source": [ + "### Common Mistake: Filters in the Wrong Place\n", + "\n", + "Every search above passed its filter **inside each `Prefetch`**. There is a second place the client will happily accept one: `query_filter`, on the outer query, next to `prefetch`.\n", + "\n", + "Put the filter inside every `Prefetch`. It narrows what each retriever searches, so both come back with 50 candidates that already satisfy the constraint, and it behaves the same way on every deployment mode.\n", + "\n", + "Local mode ignores an outer filter and raises no error, so a notebook can return results that violate its own filter while looking perfectly healthy. The cell below runs the same VN-and-shipping filter both ways so you can see it." + ] }, { "cell_type": "code", + "execution_count": 12, + "id": "b1ec6c98", "metadata": { - "id": "vZRNzvdieTpk", - "colab": { - "base_uri": "https://localhost:8080/" - }, - "outputId": "1dda4347" + "execution": { + "iopub.execute_input": "2026-08-11T09:29:11.244373Z", + "iopub.status.busy": "2026-08-11T09:29:11.244198Z", + "iopub.status.idle": "2026-08-11T09:29:11.273435Z", + "shell.execute_reply": "2026-08-11T09:29:11.272468Z" + } }, - "execution_count": null, - "source": "def search_wrong(query_text, query_filter=None, limit=5):\n \"\"\"Identical to search(), except the filter sits at the top level.\"\"\"\n dense_q = next(dense_model.query_embed(query_text)).tolist()\n sparse_q = to_sparse_vector(query_text, is_query=True)\n return client.query_points(\n collection_name=\"news\",\n prefetch=[\n models.Prefetch(query=dense_q, using=\"dense\", limit=50),\n models.Prefetch(query=sparse_q, using=\"sparse\", limit=50),\n ],\n query=models.RrfQuery(rrf=models.Rrf()),\n query_filter=query_filter, # too late: the prefetches already ran\n limit=limit,\n ).points\n\nprint(\"CORRECT filter inside each prefetch (country=VN AND topic=shipping):\")\nshow(search(\"port congestion\", query_filter=f_must))\nprint()\nprint(\"BROKEN same filter at the top level:\")\nshow(search_wrong(\"port congestion\", query_filter=f_must))", "outputs": [ { - "output_type": "stream", "name": "stdout", - "text": "CORRECT filter inside each prefetch (country=VN AND topic=shipping):\n[0.5000] VN/vi shipping | Tac nghen cang tai Hai Phong khi luong hang tang manh.\n\nBROKEN same filter at the top level:\n[0.7000] SG/en shipping | Vessel Ever Given reroutes through Singapore.\n[0.5000] JP/ja shipping | 東南アジアの港湾混雑が輸送を遅らせている。\n[0.3333] CN/zh logistics | 上海港物流吞吐量创新高。\n[0.2500] TH/th shipping | ความแออัดที่ท่าเรือแหลมฉบังเพิ่มขึ้น\n[0.1667] DE/de logistics | Hamburger Hafen meldet Verzoegerungen.\n" + "output_type": "stream", + "text": [ + "FILTER ON THE OUTER QUERY, in local mode:\n", + "[1.0000] JP shipping nikkei | Southeast Asian port congestion slows regional transport\n", + "[0.5833] TH shipping bangkok-post | Laem Chabang congestion rises on higher container volumes\n", + "[0.4762] DE logistics handelsblatt | Hamburg reports handling delays as arrivals climb\n", + "[0.4167] VN shipping reuters | Hai Phong container backlog grows for a third week\n", + "[0.4000] CN logistics caixin | Shanghai port sets a monthly logistics throughput record\n", + " -> 5 results, 4 of them break the filter\n", + "\n", + "FILTER INSIDE EACH PREFETCH:\n", + "[1.0000] VN shipping reuters | Hai Phong container backlog grows for a third week\n", + " -> 1 results, 0 of them break the filter\n" + ] } + ], + "source": [ + "dense_q = next(dense_model.query_embed(\"port congestion\")).tolist()\n", + "sparse_q = to_sparse_vector(\"port congestion\", is_query=True)\n", + "\n", + "# The wrong place: query_filter on the outer query, alongside prefetch.\n", + "wrong = client.query_points(\n", + " collection_name=\"news\",\n", + " prefetch=[\n", + " models.Prefetch(query=dense_q, using=\"dense\", limit=50),\n", + " models.Prefetch(query=sparse_q, using=\"sparse\", limit=50),\n", + " ],\n", + " query=models.RrfQuery(rrf=models.Rrf()),\n", + " query_filter=f_must,\n", + " limit=5,\n", + ").points\n", + "\n", + "print(\"FILTER ON THE OUTER QUERY, in local mode:\")\n", + "show(wrong)\n", + "violations = [p for p in wrong if p.payload[\"country\"] != \"VN\" or p.payload[\"topic\"] != \"shipping\"]\n", + "print(f\" -> {len(wrong)} results, {len(violations)} of them break the filter\\n\")\n", + "\n", + "print(\"FILTER INSIDE EACH PREFETCH:\")\n", + "right = search(\"port congestion\", query_filter=f_must)\n", + "show(right)\n", + "violations = [p for p in right if p.payload[\"country\"] != \"VN\" or p.payload[\"topic\"] != \"shipping\"]\n", + "print(f\" -> {len(right)} results, {len(violations)} of them break the filter\")" ] }, { "cell_type": "markdown", - "source": "The rule is short. No prefetch, use `query_filter`. Prefetch, put the filter in every prefetch.", - "metadata": { - "id": "jBq78Hp7twax" - } - }, - { - "cell_type": "markdown", - "source": "### A special case: scoping by user or tenant\n\nIn almost any multi user product you must scope every query to one user's or one customer's data. The instinct is a collection per user, which becomes millions of collections and is operationally unmanageable. The standard pattern instead:\n\n1. Add a `tenant_id` payload field to every point at ingestion. (Done, above.)\n2. Create a payload index on it with `is_tenant=True`. (Done, before ingestion.)\n3. Filter on it at every query. Never omit it.\n\nSteps 1 and 3 alone are already correct: they just get slower than they need to as tenant count grows. Step 2 is what keeps it fast, and it is the one people skip.", - "metadata": { - "id": "DFmFaqfycbso" - } + "id": "a13364fd", + "metadata": {}, + "source": [ + "### A Special Case: Scoping by User or Tenant\n", + "\n", + "In almost any multi-user product you have to scope every query to one customer's data. The instinct is a collection per customer, which becomes millions of collections and is unmanageable. The pattern instead:\n", + "\n", + "1. Add a `tenant_id` payload field to every point at ingestion. Already done above.\n", + "2. Create a payload index on it with `is_tenant=True`, which tells Qdrant this field identifies tenants so it can keep each tenant's data together on disk. Supported for the `keyword` and `uuid` index types.\n", + "3. Filter on it at every query. Never omit it.\n", + "\n", + "Steps 1 and 3 alone are already correct. Step 2 is what keeps them fast as the tenant count grows." + ] }, { "cell_type": "code", + "execution_count": 13, + "id": "16ee601b", "metadata": { - "id": "KGUOS2yu9jSN", - "colab": { - "base_uri": "https://localhost:8080/" - }, - "outputId": "0deabc28" + "execution": { + "iopub.execute_input": "2026-08-11T09:29:11.274835Z", + "iopub.status.busy": "2026-08-11T09:29:11.274675Z", + "iopub.status.idle": "2026-08-11T09:29:11.300161Z", + "shell.execute_reply": "2026-08-11T09:29:11.299165Z" + } }, - "execution_count": null, - "source": "def tenant_search(query_text, tenant_id, limit=5):\n return search(\n query_text,\n query_filter=models.Filter(\n must=[models.FieldCondition(key=\"tenant_id\", match=models.MatchValue(value=tenant_id))]\n ),\n limit=limit,\n )\n\nprint(\"europe-desk view of 'port delays':\")\nshow(tenant_search(\"port delays\", tenant_id=\"europe-desk\"))\nprint(\"\\nasia-desk view of 'port delays':\")\nshow(tenant_search(\"port delays\", tenant_id=\"asia-desk\"))", "outputs": [ { - "output_type": "stream", "name": "stdout", - "text": "europe-desk view of 'port delays':\n[0.5000] DE/de logistics | Hamburger Hafen meldet Verzoegerungen.\n[0.3333] DK/en markets | Maersk shares move on delisting speculation.\n\nasia-desk view of 'port delays':\n[0.7000] SG/en shipping | Vessel Ever Given reroutes through Singapore.\n[0.5000] JP/ja shipping | 東南アジアの港湾混雑が輸送を遅らせている。\n[0.3333] TH/th shipping | ความแออัดที่ท่าเรือแหลมฉบังเพิ่มขึ้น\n[0.2500] CN/zh logistics | 上海港物流吞吐量创新高。\n[0.1667] VN/vi shipping | Tac nghen cang tai Hai Phong khi luong hang tang manh.\n" + "output_type": "stream", + "text": [ + "europe-desk view of 'port delays':\n", + "[1.0000] DE logistics handelsblatt | Hamburg reports handling delays as arrivals climb\n", + "[0.3333] DK markets reuters | Shipping group unit faces delisting speculation\n", + "[0.2500] DE markets handelsblatt | Carrier unit faces delisting speculation\n", + "\n", + "asia-desk view of 'port delays':\n", + "[1.0000] JP shipping nikkei | Southeast Asian port congestion slows regional transport\n", + "[0.5833] VN shipping reuters | Hai Phong container backlog grows for a third week\n", + "[0.5833] SG shipping straits-times | Ever Given rerouted through Singapore to avoid delays\n", + "[0.3667] TH shipping bangkok-post | Laem Chabang congestion rises on higher container volumes\n", + "[0.3667] CN logistics caixin | Shanghai port sets a monthly logistics throughput record\n" + ] } + ], + "source": [ + "client.create_payload_index(\n", + " collection_name=\"news\",\n", + " field_name=\"tenant_id\",\n", + " field_schema=models.KeywordIndexParams(\n", + " type=models.KeywordIndexType.KEYWORD,\n", + " is_tenant=True,\n", + " ),\n", + ")\n", + "\n", + "\n", + "def tenant_search(query_text, tenant_id, limit=5):\n", + " return search(\n", + " query_text,\n", + " query_filter=models.Filter(\n", + " must=[models.FieldCondition(key=\"tenant_id\", match=models.MatchValue(value=tenant_id))]\n", + " ),\n", + " limit=limit,\n", + " )\n", + "\n", + "\n", + "print(\"europe-desk view of 'port delays':\")\n", + "show(tenant_search(\"port delays\", tenant_id=\"europe-desk\"))\n", + "print(\"\\nasia-desk view of 'port delays':\")\n", + "show(tenant_search(\"port delays\", tenant_id=\"asia-desk\"))" ] }, { "cell_type": "markdown", - "source": "### Key insight\n\nDesign the payload schema before you ingest, driven by one question: what will I need to filter on? Time, geography, identity, permissions, and status flags are the usual suspects.\n\nAdding a payload *field* later is easy. Adding a payload *index* later means rebuilding the HNSW graph, and discovering at query time that you never stored `language` at all means re-ingesting everything.", - "metadata": { - "id": "FZXZK3QQobiw" - } + "id": "6e15d660", + "metadata": {}, + "source": [ + "Two desks, one collection, and neither sees the other's articles.\n", + "\n", + "### Key Insight\n", + "\n", + "Design the payload schema before you ingest, driven by one question: what will I need to filter on? Time, geography, identity, permissions, and status flags are the usual suspects. Adding a payload field later is easy. Discovering at query time that you never stored the publication date means re-ingesting everything." + ] }, { "cell_type": "markdown", - "source": "## 4. The Production RAG Pipeline\n\nRetrieval-Augmented Generation (RAG) retrieves relevant passages from a vector search engine and hands them to an LLM as context, so the model answers from your data instead of relying only on what it memorized during training.\n\nThe production shape, using everything above:\n\n1. **Query understanding:** extract hard constraints (dates, country, topic) into a filter. Embed the query as dense and sparse vectors.\n2. **Hybrid retrieval:** dense + sparse prefetch, each carrying the filter, fused by rank. One `query_points` call.\n3. **Optional reranking:** a cross-encoder scores the top candidates and keeps the best few. Add this only when evaluation shows fused results need refinement.\n4. **LLM generation:** the top passages go in as context, the model generates the answer with sources.\n\n### Rule of thumb\n\nWhen RAG quality disappoints, improve step 2 before reaching for a bigger model in step 4. Retrieval quality caps answer quality: the model cannot cite a passage it never received.\n\nThe cell below implements steps 1, 2, and the prompt assembly for step 4. It runs with no API key and prints the exact prompt an LLM would receive.", - "metadata": { - "id": "gZIMMQEaXUFH" - } + "id": "6f6064aa", + "metadata": {}, + "source": [ + "## 4. The Production RAG Pipeline\n", + "\n", + "Retrieval-Augmented Generation (RAG) retrieves relevant passages from a vector search engine and hands them to a large language model as context, so the model answers from your data instead of only from what it memorized during training.\n", + "\n", + "The production shape, using everything above:\n", + "\n", + "1. **Query understanding:** pull hard constraints (dates, country, topic) into a filter. Embed the query as a dense vector and a sparse vector.\n", + "2. **Hybrid retrieval:** dense and sparse prefetch with the filter on each, fused with Reciprocal Rank Fusion. One `query_points` call.\n", + "3. **Optional reranking:** a cross-encoder, a model that scores a query and a passage together rather than separately, reorders the top candidates. Add it only when evaluation shows fused results need refinement.\n", + "4. **Generation:** the top passages go in as context and the model writes the answer.\n", + "\n", + "### Rule of Thumb\n", + "\n", + "When RAG quality disappoints, improve step 2 before reaching for a bigger model in step 4. Retrieval quality caps answer quality: the model cannot cite a passage it never received.\n", + "\n", + "The cell below runs steps 1, 2, and the prompt assembly for step 4. It needs no API key." + ] }, { "cell_type": "code", + "execution_count": 14, + "id": "7d587768", "metadata": { - "id": "OeTpeqTmT8Ex", - "colab": { - "base_uri": "https://localhost:8080/" - }, - "outputId": "7ac9a124" + "execution": { + "iopub.execute_input": "2026-08-11T09:29:11.301808Z", + "iopub.status.busy": "2026-08-11T09:29:11.301613Z", + "iopub.status.idle": "2026-08-11T09:29:11.319486Z", + "shell.execute_reply": "2026-08-11T09:29:11.318530Z" + } }, - "execution_count": null, - "source": "def rag_context(question, query_filter=None, k=3):\n \"\"\"Steps 1 and 2: retrieve the top-k passages for a question.\"\"\"\n hits = search(question, query_filter=query_filter, limit=k)\n passages = []\n for h in hits:\n p = h.payload\n passages.append(f\"- ({p['source']}, {p['country']}, {p['published_at'][:10]}) {p['summary']}\")\n return \"\\n\".join(passages), hits\n\ndef build_prompt(question, query_filter=None, k=3):\n context, hits = rag_context(question, query_filter=query_filter, k=k)\n prompt = (\n \"Answer the question using only the sources below. Cite the source name.\\n\\n\"\n f\"Sources:\\n{context}\\n\\n\"\n f\"Question: {question}\\nAnswer:\"\n )\n return prompt, hits\n\nprompt, hits = build_prompt(\"What is happening with port congestion in Southeast Asia?\")\nprint(prompt)", "outputs": [ { - "output_type": "stream", "name": "stdout", - "text": "Answer the question using only the sources below. Cite the source name.\n\nSources:\n- (straits-times, SG, 2026-07-22) Vessel Ever Given reroutes through Singapore.\n- (nikkei, JP, 2026-07-21) 東南アジアの港湾混雑が輸送を遅らせている。\n- (bangkok-post, TH, 2026-07-18) ความแออัดที่ท่าเรือแหลมฉบังเพิ่มขึ้น\n\nQuestion: What is happening with port congestion in Southeast Asia?\nAnswer:\n" + "output_type": "stream", + "text": [ + "Answer the question using only the sources below. Cite the source name.\n", + "\n", + "Sources:\n", + "- (nikkei, JP, 2026-07-21) Southeast Asian port congestion slows regional transport. Delays at major Southeast Asian ports are pushing back container ship arrivals across the region.\n", + "- (reuters, VN, 2026-07-20) Hai Phong container backlog grows for a third week. Congestion at the northern Vietnamese port deepened as container volumes climbed, delaying vessel departures.\n", + "- (bangkok-post, TH, 2026-07-18) Laem Chabang congestion rises on higher container volumes. Thailand's largest port is handling more containers than its yards were sized for, lengthening waits.\n", + "\n", + "Question: What is happening with port congestion in Southeast Asia?\n", + "Answer:\n" + ] } + ], + "source": [ + "def rag_context(question, query_filter=None, k=3):\n", + " \"\"\"Steps 1 and 2: retrieve the top-k passages for a question.\"\"\"\n", + " hits = search(question, query_filter=query_filter, limit=k)\n", + " passages = [\n", + " f\"- ({h.payload['source']}, {h.payload['country']}, {h.payload['published_at'][:10]}) \"\n", + " f\"{h.payload['headline']}. {h.payload['lead']}\"\n", + " for h in hits\n", + " ]\n", + " return \"\\n\".join(passages), hits\n", + "\n", + "\n", + "def build_prompt(question, query_filter=None, k=3):\n", + " context, hits = rag_context(question, query_filter=query_filter, k=k)\n", + " prompt = (\n", + " \"Answer the question using only the sources below. Cite the source name.\\n\\n\"\n", + " f\"Sources:\\n{context}\\n\\n\"\n", + " f\"Question: {question}\\nAnswer:\"\n", + " )\n", + " return prompt, hits\n", + "\n", + "\n", + "prompt, hits = build_prompt(\"What is happening with port congestion in Southeast Asia?\")\n", + "print(prompt)" + ] + }, + { + "cell_type": "markdown", + "id": "cc9a9eee", + "metadata": {}, + "source": [ + "Step 4 is the only part that needs an API key, so it is optional. To run it, add your key to the Colab secrets panel (the key icon in the left sidebar) under the name `LLM_API_KEY`, then uncomment the block." ] }, { "cell_type": "code", + "execution_count": 15, + "id": "ed2582b6", "metadata": { - "id": "UsJadqPQPsGC", - "colab": { - "base_uri": "https://localhost:8080/" - }, - "outputId": "71e3fe07" + "execution": { + "iopub.execute_input": "2026-08-11T09:29:11.321704Z", + "iopub.status.busy": "2026-08-11T09:29:11.321554Z", + "iopub.status.idle": "2026-08-11T09:29:11.327215Z", + "shell.execute_reply": "2026-08-11T09:29:11.325922Z" + } }, - "execution_count": null, - "source": "# OPTIONAL step 4: hand the prompt to a real LLM. This block is inert unless you\n# add an API key, so the notebook still runs end to end without one.\n#\n# import os\n# from anthropic import Anthropic\n# llm = Anthropic(api_key=os.environ[\"ANTHROPIC_API_KEY\"])\n# msg = llm.messages.create(\n# model=\"claude-sonnet-4-6\",\n# max_tokens=300,\n# messages=[{\"role\": \"user\", \"content\": prompt}],\n# )\n# print(msg.content[0].text)\n\nprint(\"Prompt is ready. Uncomment the block above and set an API key to generate an answer.\")", "outputs": [ { - "output_type": "stream", "name": "stdout", - "text": "Prompt is ready. Uncomment the block above and set an API key to generate an answer.\n" + "output_type": "stream", + "text": [ + "Prompt is ready. Uncomment the block above and add a key to generate an answer.\n" + ] } + ], + "source": [ + "# Optional step 4: hand the prompt to a model. This cell runs without a key.\n", + "#\n", + "# from google.colab import userdata\n", + "# from anthropic import Anthropic\n", + "#\n", + "# llm = Anthropic(api_key=userdata.get(\"LLM_API_KEY\"))\n", + "# message = llm.messages.create(\n", + "# model=\"claude-sonnet-4-5\",\n", + "# max_tokens=300,\n", + "# messages=[{\"role\": \"user\", \"content\": prompt}],\n", + "# )\n", + "# print(message.content[0].text)\n", + "\n", + "print(\"Prompt is ready. Uncomment the block above and add a key to generate an answer.\")" ] }, { "cell_type": "markdown", - "source": "## What's next: Module 5\n\nThe capstone extends this system to three modalities at once: news, audio, and satellite imagery on shared points, clustered into risk themes and queried across languages.\n\n[Continue to Module 5](https://qdrant.tech/course/beginners/module-5/)", - "metadata": { - "id": "Y8Bqn6nKiG7i" - } + "id": "fe6e49a4", + "metadata": {}, + "source": [ + "## 5. Deployment Options\n", + "\n", + "The design above runs unchanged on any of these. Which one is right depends on your constraints, not on sophistication.\n", + "\n", + "| Deployment Mode | Use When | Avoid When |\n", + "|---|---|---|\n", + "| Local Mode | Prototyping, notebooks, CI tests, teaching | Production or benchmarking |\n", + "| Docker, self-hosted | Full infrastructure control, air-gapped or regulated environments | You do not yet have monitoring and backups |\n", + "| Managed Cloud | Small ops team, standard requirements | Data cannot leave your infrastructure |\n", + "| Hybrid Cloud | Data residency or security policy requires your own infrastructure | Managed cloud would do, with less operational overhead |\n", + "| Private Cloud or on-prem | Strictest requirements: defense, healthcare, finance | A lighter mode meets your needs |\n", + "| Edge | On-device search, offline, very low latency | You need distributed search |\n", + "\n", + "One caveat about local mode, since it is what this notebook used. It is a Python reimplementation rather than the engine: search is exact rather than approximate, payload indexes have no effect, and a filter on the outer query is ignored. Everything in Section 3 is still worth understanding while working locally, but verify indexing and filtering against a real server before trusting numbers from a notebook." + ] + }, + { + "cell_type": "markdown", + "id": "170652a1", + "metadata": {}, + "source": [ + "## 6. Knowledge Check\n", + "\n", + "Work through these before starting the capstone in Module 5.\n", + "\n", + "**1. Name the five layers, and place \"chunk articles instead of embedding them whole\" in the right one.**\n", + "Query, indexing, storage, knowledge, distribution. Chunking is a knowledge-layer decision, and mistakes there cannot be fixed by tuning any other layer.\n", + "\n", + "**2. Why hybrid retrieval from the start, rather than dense only?**\n", + "Query analysis showed a mix of semantic intent and exact tokens. A ticker carries no semantics for a dense model, so the dense margin between the right article and a near-identical one is thin. The sparse half separates them decisively.\n", + "\n", + "**3. What is wrong with post-filtering, and how does Qdrant avoid it?**\n", + "Post-filtering retrieves the top K first and discards invalid results, so a selective filter can leave zero valid points. Qdrant's query planner instead picks a strategy per segment from the estimated filter cardinality.\n", + "\n", + "**4. In a hybrid query, where does the filter belong?**\n", + "Inside each `Prefetch`. It narrows what each retriever searches, and it behaves the same way on every deployment mode. Local mode ignores an outer `query_filter` without raising an error, which is how a notebook ends up printing results that break its own filter.\n", + "\n", + "**5. Why create payload indexes before ingestion rather than after?**\n", + "The filterable HNSW graph gains filter-aware edges only for indexes that exist when the graph is built. An index created later still filters and still feeds the planner's cardinality estimate, but getting those extra edges means rebuilding the HNSW index.\n", + "\n", + "**6. Why does per-tenant scoping use a payload filter instead of one collection per tenant?**\n", + "Collections do not scale to millions of tenants operationally. An indexed `tenant_id` field, filtered on at every query, gives isolation in one collection. `is_tenant=True` is what keeps it fast as the tenant count grows.\n", + "\n", + "**7. In the RAG pipeline, which step do you improve first when answers disappoint?**\n", + "Step 2, retrieval. The model cannot cite a passage it never received." + ] + }, + { + "cell_type": "markdown", + "id": "8607c08e", + "metadata": {}, + "source": [ + "## What Is Next: Module 5\n", + "\n", + "The capstone extends this system. Same five questions, bigger answers:\n", + "\n", + "- Ingest daily news, audio, and satellite imagery about suppliers, so three modalities instead of one\n", + "- Embed each modality into named vectors on shared points\n", + "- Cluster signals into risk themes across suppliers\n", + "\n", + "Before you start it, create a free cluster at [cloud.qdrant.io](https://cloud.qdrant.io/) so the capstone runs against a real server rather than local mode." + ] } - ] + ], + "metadata": { + "colab": { + "provenance": [], + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 } From acfdd828e79e6a1f88b5f4f4ba8f0fc7096f6d84 Mon Sep 17 00:00:00 2001 From: Manas Chopra Date: Tue, 11 Aug 2026 18:25:52 +0530 Subject: [PATCH 3/3] updated module 1 notebook --- Beginner-course/Module1.ipynb | 535 +++++++++++++++++++--------------- 1 file changed, 298 insertions(+), 237 deletions(-) diff --git a/Beginner-course/Module1.ipynb b/Beginner-course/Module1.ipynb index f9db5c4..cfdd175 100644 --- a/Beginner-course/Module1.ipynb +++ b/Beginner-course/Module1.ipynb @@ -1,317 +1,378 @@ { - "nbformat": 4, - "nbformat_minor": 0, - "metadata": { - "colab": { - "provenance": [] - }, - "kernelspec": { - "display_name": "Python 3", - "name": "python3" - }, - "language_info": { - "name": "python" - } - }, "cells": [ { "cell_type": "markdown", - "source": "# Module 1: Let's Understand Search\n\n## What you will do\n\n1. Watch keyword search miss documents that plainly answer the query.\n2. Turn text into a vector and look at what actually comes back.\n3. Score meaning with cosine similarity, first by hand, then with the library helper.\n4. Re-run the Section 1 failures semantically and watch them pass.\n5. Meet the SKU problem, where similarity on its own is not enough.\n\n**Tip:** every cell below is already run, so you can read this straight through. Run it yourself and you should see the same numbers.\n\nCompanion notebook to the [Module 1 lesson](https://qdrant.tech/course/beginners/module-1/).", - "metadata": { - "id": "i0VpEBOWfbZA" - } - }, - { - "cell_type": "markdown", - "source": "## Setup\n\nModule 1 stays deliberately small: one embedding model, no vector database yet. `sentence-transformers` gives us `all-MiniLM-L6-v2`, a 384-dimension model fast enough to run on a free Colab CPU. Qdrant arrives in Module 2.", - "metadata": { - "id": "VaBSo63bbH6x" - } - }, - { - "cell_type": "code", - "metadata": { - "id": "nAbnBEoonCrb" - }, - "execution_count": null, - "source": "!pip install -q sentence-transformers", - "outputs": [] + "metadata": {}, + "source": [ + "# Module 1: Let's Understand Search\n", + "\n", + "Companion notebook for **Module 1** of the Qdrant Beginners Course.\n", + "\n", + "Understand why traditional search struggles and how modern semantic search improves it. This notebook walks through embeddings, cosine similarity, and the limits of similarity search, with runnable code for every example.\n" + ] }, { "cell_type": "markdown", - "source": "## 1. The Problem: Why Keyword Search Struggles\n\nTraditional search matches exact words. If the query string appears in the document it is a hit, and if it does not it is a miss, however close the meaning is.\n\nHere is the whole of keyword search, in three lines.", - "metadata": { - "id": "SS6AZPsK20FK" - } + "metadata": {}, + "source": [ + "## Setup\n", + "\n", + "The embedding model runs on your CPU. You don't need a GPU or API keys.\n" + ] }, { "cell_type": "code", - "metadata": { - "id": "cpXzkIRPxBFW", - "colab": { - "base_uri": "https://localhost:8080/" - }, - "outputId": "86700bf9" - }, "execution_count": null, - "source": "documents = [\n \"automobile maintenance guide\",\n \"affordable airfare to New York\",\n \"apple harvest season guide\",\n]\n\ndef keyword_search(query, docs):\n \"\"\"All of keyword search: does the query string appear verbatim?\"\"\"\n return [d for d in docs if query.lower() in d.lower()]\n\nfor query in [\"car repair\", \"cheap flights NYC\", \"Apple stock\"]:\n hits = keyword_search(query, documents)\n print(f\"{query!r:22} -> {hits if hits else 'NO MATCH'}\")", - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": "'car repair' -> NO MATCH\n'cheap flights NYC' -> NO MATCH\n'Apple stock' -> NO MATCH\n" - } + "metadata": {}, + "outputs": [], + "source": [ + "!pip install fastembed numpy -q" ] }, { "cell_type": "markdown", - "source": "Three queries. Three documents that answer them. Zero hits.\n\nNothing is broken. The engine is doing exactly what it was asked: comparing characters. That single limitation shows up in four ways.\n\n- **Synonyms.** \"car\" and \"automobile\" mean the same thing and share no letters.\n- **Paraphrasing.** \"cheap flights\" and \"affordable airfare\" are the same intent in different words.\n- **Polysemy.** \"apple\" is a company, a fruit, and a record label. Characters cannot tell you which one was meant.\n- **Word order.** \"dog bites man\" and \"man bites dog\" contain identical words.\n\nImprovements to keyword search, inverted indexes for speed, BM25 for ranking, stemming, and fuzzy matching, all raise the ceiling without moving it. They still work on words. You could hard-code that \"car\" means \"automobile\", then do it again for the next pair, and the next. You cannot hard-code a language.\n\nSo semantic search changes the question. Not *does this document contain the same words*, but *does this document mean the same thing*.", - "metadata": { - "id": "ykF8VH1oF7JC" - } + "metadata": {}, + "source": [ + "## 1. What Is Search?\n", + "\n", + "Search is the act of finding the right information out of everything you have, given a question. You type \"car repair\" into a box, and something has to decide which of your thousands of documents, products, or messages actually answers that.\n", + "\n", + "Every search system, no matter how it's built internally, does the same two things:\n", + "\n", + "1. **Retrieve**: narrow a huge collection down to a shortlist of documents that might be relevant.\n", + "2. **Rank**: order that shortlist so the best answer ends up near the top.\n", + "\n", + "This module is about one question: how does a system decide what \"relevant\" means? We'll start with the simplest possible answer, watch it fail, and build up from there. No prior knowledge of vector search engines or indexing algorithms is assumed.\n" + ] }, { "cell_type": "markdown", - "source": "## 2. How It Works: Embeddings\n\nAn embedding model takes a piece of text and returns a fixed-length list of floating-point numbers. Similar meanings produce vectors that sit close together in that space.", - "metadata": { - "id": "qH7aWY2TYGIA" - } - }, - { - "cell_type": "code", - "metadata": { - "id": "d1JmFEwvHMLC", - "colab": { - "base_uri": "https://localhost:8080/" - }, - "outputId": "0329dccd" - }, - "execution_count": null, - "source": "from sentence_transformers import SentenceTransformer, util\n\nmodel = SentenceTransformer(\"all-MiniLM-L6-v2\")\n\nquery_vec = model.encode(\"car repair\")\ndoc_vec = model.encode(\"automobile maintenance\")\n\nprint(\"dimensions:\", len(query_vec))\nprint(\"first five:\", query_vec[:5])", - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": "dimensions: 384\nfirst five: [-0.12456685 0.03955904 0.08732592 -0.02361241 -0.07054088]\n" - } + "metadata": {}, + "source": [ + "## 2. The Problem: Why Keyword Search Struggles\n", + "\n", + "Keyword search retrieves documents by exact word match: it checks whether the literal terms in your query appear in the document, with no understanding of what those terms mean. That works when people use the same terms as the content they need.\n", + "\n", + "That creates a few common problems:\n", + "\n", + "- **Different words, same meaning:** \"car repair\" misses \"automobile maintenance.\"\n", + "- **Same word, different meaning:** \"Apple stock\" can retrieve fruit content instead of financial information.\n", + "- **Same words, different order:** \"dog bites man\" and \"man bites dog\" contain the same terms, but mean very different things.\n" ] }, { "cell_type": "markdown", - "source": "384 numbers. That is the vector.\n\nDo not go looking for the dimension that stores colour, or urgency. Nobody designed these. The model learned them from data, and meaning lives in the combination of all 384 rather than in any single one. A 384-dimension model has 384 such aspects, and only the whole set together encodes meaning.", - "metadata": { - "id": "p0PfbaUpgMve" - } + "metadata": {}, + "source": [ + "## 3. How Traditional Search Improved\n", + "\n", + "Keyword search picked up real upgrades over the years: faster lookups, relevance ranking, tolerance for typos, matching on word roots. Each made matching faster or more forgiving, but none of them taught the system what words mean. A keyword system can't know \"car\" and \"automobile\" are synonyms unless someone hard-codes that fact, and you can't hard-code an entire language.\n", + "\n", + "That's the gap semantic search closes. Instead of asking \"Does this document contain the same words?\" it asks \"Does this document mean the same thing?\" Nobody hand-codes the fact that \"car\" and \"automobile\" are related, the model learns it from the text it was trained on, and sentences with related meaning end up as vectors that sit close together, even when they share no words.\n" + ] }, { "cell_type": "markdown", - "source": "## 3. Comparing Meaning: Cosine Similarity\n\nCosine similarity measures the angle between two vectors and ignores their length. Vectors pointing the same way score near 1. Unrelated directions score near 0.\n\nThe formula is short enough to write out, which is worth doing once so the library call stops being magic.", - "metadata": { - "id": "jGkqSCtDbxAl" - } + "metadata": {}, + "source": [ + "## 4. How It Works: Embeddings\n", + "\n", + "### What Is an Embedding?\n", + "\n", + "An embedding is a vector: a list of numbers that captures meaning. Semantic search works by converting text into embeddings, text with similar meaning produces embeddings that sit close together in high-dimensional space, and text with different meaning produces embeddings that sit far apart. Each position in that list is a dimension; no single one maps to a human concept like \"color,\" meaning comes from all of them combined.\n", + "\n", + "### The Embedding Model\n", + "\n", + "An embedding model takes a piece of text and returns a fixed-length array of floating-point numbers. The exact numbers matter less than the relationships between them.\n" + ] }, { "cell_type": "code", - "metadata": { - "id": "g3FmLYbbjSjR", - "colab": { - "base_uri": "https://localhost:8080/" - }, - "outputId": "a83fc83a" - }, "execution_count": null, - "source": "import numpy as np\n\ndef cosine_by_hand(a, b):\n return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))\n\nby_hand = cosine_by_hand(query_vec, doc_vec)\nby_library = util.cos_sim(query_vec, doc_vec).item()\n\nprint(f\"by hand: {by_hand:.6f}\")\nprint(f\"cos_sim: {by_library:.6f}\")\nprint(f\"agree: {np.isclose(by_hand, by_library)}\")", - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": "by hand: 0.733420\ncos_sim: 0.733420\nagree: True\n" - } + "metadata": {}, + "outputs": [], + "source": [ + "from fastembed import TextEmbedding\n", + "\n", + "model = TextEmbedding(model_name=\"sentence-transformers/all-MiniLM-L6-v2\")\n", + "\n", + "# model.embed() takes a list of strings and returns one vector per string\n", + "query_vec = list(model.embed([\"car repair\"]))[0]\n", + "doc_vec = list(model.embed([\"automobile maintenance\"]))[0]\n", + "\n", + "print(len(query_vec), len(doc_vec)) # check both vectors are the same length: 384 dimensions each\n", + "print(query_vec[:5]) # peek at the first 5 of the query's 384 floats\n", + "print(doc_vec[:5]) # peek at the first 5 of the document's 384 floats" ] }, { "cell_type": "markdown", - "source": "Roughly 0.73 for two phrases with no words in common. That number is the whole point of the module.\n\nNow three pairs at once: a synonym pair, a paraphrase pair, and a pair that has nothing to do with each other.", - "metadata": { - "id": "yJtNds82s1t6" - } - }, - { - "cell_type": "code", - "metadata": { - "id": "Uzpa2cY7Jk18", - "colab": { - "base_uri": "https://localhost:8080/" - }, - "outputId": "b8653a63" - }, - "execution_count": null, - "source": "pairs = [\n (\"car repair\", \"automobile maintenance\"), # synonyms\n (\"cheap flights to New York\", \"affordable airfare to NYC\"), # paraphrase\n (\"cheap flights to New York\", \"best pizza in Chicago\"), # unrelated\n]\n\nfor query, document in pairs:\n score = util.cos_sim(model.encode(query), model.encode(document)).item()\n print(f\"{score:.3f} | {query!r} vs {document!r}\")", - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": "0.733 | 'car repair' vs 'automobile maintenance'\n0.821 | 'cheap flights to New York' vs 'affordable airfare to NYC'\n0.332 | 'cheap flights to New York' vs 'best pizza in Chicago'\n" - } + "metadata": {}, + "source": [ + "### Model Size: A Tradeoff\n", + "\n", + "Embedding models come in different sizes. Smaller models (128\u2013384 dimensions, like the one above) are fast and cheap to run. Larger ones (1024+ dimensions) can capture more nuance and context, at the cost of more compute and memory. Dimension count alone isn't a quality signal, a well-trained small model can beat a poorly trained large one.\n", + "\n", + "### Why This Model\n", + "\n", + "This module uses `sentence-transformers/all-MiniLM-L6-v2` because it's small enough to run on a CPU with no API keys or GPU, and accurate enough to demonstrate semantic search clearly. When you start your own project, see [Points, Vectors and Payloads](https://qdrant.tech/course/essentials/day-1/embedding-models/) for how to weigh size, language, and domain fit when picking a model.\n" ] }, { "cell_type": "markdown", - "source": "The first two pairs share little or no vocabulary and both score high. That is the synonym and the paraphrase that keyword search missed in Section 1. The third scores far lower, so the model is separating meaning rather than matching surface words.\n\nWhich means we can go back and re-run those three failures.", - "metadata": { - "id": "gPsEu23bmu9W" - } + "metadata": {}, + "source": [ + "## 5. Comparing Meaning: Distance Metrics\n", + "\n", + "Once we have vectors, we need a way to measure how similar two of them are. Different metrics suit different situations.\n", + "\n", + "### Cosine Similarity\n", + "\n", + "The most common metric for text. It measures the angle between two vectors and ignores their length, focusing purely on direction. Scores range from -1 to 1: 1.0 means the vectors point the same way, 0.0 means unrelated, -1 means opposite. In practice, normalized text-embedding models like the one used here rarely produce negative scores, so unrelated text usually lands as a small positive number instead.\n", + "\n", + "$$\n", + "\\text{cosine\\_similarity}(A, B) = \\frac{A \\cdot B}{\\lVert A \\rVert \\, \\lVert B \\rVert}\n", + "$$\n", + "\n", + "For example, embedding \"car repair\" and \"automobile maintenance\" and comparing the two vectors with this formula yields a similarity score around 0.73, far higher than an unrelated pair would score, reflecting their shared meaning despite having no words in common.\n", + "\n", + "### Try It Yourself: Compare Cosine Scores\n", + "\n", + "Reuse the embedding snippet from section 4 to embed three query/document pairs, then score each pair with cosine similarity, using the formula above implemented directly with NumPy.\n" + ] }, { "cell_type": "code", - "metadata": { - "id": "vnPZ5v2QE9oS", - "colab": { - "base_uri": "https://localhost:8080/" - }, - "outputId": "12e3c9d5" - }, "execution_count": null, - "source": "# Same three documents as Section 1, plus a second \"apple\" sense so the\n# polysemy case has somewhere correct to land.\ndocuments = [\n \"automobile maintenance guide\",\n \"affordable airfare to New York\",\n \"apple harvest season guide\",\n \"technology shares closed higher today\",\n]\n\ndoc_vecs = model.encode(documents)\n\nfor query in [\"car repair\", \"cheap flights NYC\", \"Apple stock\"]:\n sims = util.cos_sim(model.encode(query), doc_vecs)[0]\n ranked = sorted(zip(sims.tolist(), documents), reverse=True)\n print(f\"{query!r}\")\n for score, doc in ranked:\n print(f\" {score:6.3f} {doc}\")\n print()", - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": "'car repair'\n 0.609 automobile maintenance guide\n 0.065 apple harvest season guide\n 0.060 technology shares closed higher today\n -0.032 affordable airfare to New York\n\n'cheap flights NYC'\n 0.800 affordable airfare to New York\n 0.117 apple harvest season guide\n 0.080 technology shares closed higher today\n -0.021 automobile maintenance guide\n\n'Apple stock'\n 0.401 technology shares closed higher today\n 0.374 apple harvest season guide\n 0.145 affordable airfare to New York\n 0.048 automobile maintenance guide\n\n" - } + "metadata": {}, + "outputs": [], + "source": [ + "from fastembed import TextEmbedding\n", + "import numpy as np\n", + "\n", + "model = TextEmbedding(model_name=\"sentence-transformers/all-MiniLM-L6-v2\")\n", + "\n", + "def cosine_similarity(a, b):\n", + " return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))\n", + "\n", + "pairs = [\n", + " (\"car repair\", \"automobile maintenance\"), # synonyms\n", + " (\"cheap flights to New York\", \"affordable airfare to NYC\"), # paraphrase\n", + " (\"cheap flights to New York\", \"best pizza in Chicago\"), # unrelated\n", + "]\n", + "\n", + "for query, document in pairs:\n", + " query_vec = list(model.embed([query]))[0]\n", + " doc_vec = list(model.embed([document]))[0]\n", + " score = cosine_similarity(query_vec, doc_vec)\n", + " print(f\"{score:.3f} | {query!r} vs {document!r}\")\n", + "\n", + "# Expected output:\n", + "# 0.733 | 'car repair' vs 'automobile maintenance'\n", + "# 0.821 | 'cheap flights to New York' vs 'affordable airfare to NYC'\n", + "# 0.332 | 'cheap flights to New York' vs 'best pizza in Chicago'" ] }, { "cell_type": "markdown", - "source": "Every query now finds its document, and each one was a total miss under keyword search.\n\nTwo things in that output are worth pausing on.\n\n**Negative scores exist.** `car repair` against `affordable airfare to New York` comes out slightly below zero. Cosine similarity ranges from -1 to 1 in principle, where -1 means the vectors point in opposite directions. With normalized text models these stay small and near zero, so treat a value like -0.03 as \"unrelated\" rather than \"opposite\".\n\n**The polysemy win is narrow.** `Apple stock` prefers the technology document over the fruit document, but only by about 0.03. The model got the sense right, and it was close. Hold on to that, because Section 4 is built on exactly this kind of narrow gap.", - "metadata": { - "id": "vs1L73iIgce1" - } - }, - { - "cell_type": "markdown", - "source": "### Where dense search struggles too\n\nThe four failure modes were framed as keyword problems, but word order is not solved by switching to embeddings. Compare a sentence with its meaning reversed, and with a genuine paraphrase.", - "metadata": { - "id": "WZvMWxJnfq3I" - } + "metadata": {}, + "source": [ + "**What to look for:**\n", + "\n", + "- The synonym and paraphrase pairs (0.733, 0.821) score high despite sharing almost no words.\n", + "- The unrelated pair (0.332) scores far lower, reflecting different meaning.\n", + "\n", + "**Your turn:** swap in a polysemy case. Score `\"apple stock\"` against both `\"shares of a tech company\"` and `\"a crisp red fruit\"`. Which comes out higher, and does it match the sense you meant?\n" + ] }, { "cell_type": "code", - "metadata": { - "id": "5CrWZaPfh2co", - "colab": { - "base_uri": "https://localhost:8080/" - }, - "outputId": "f6123b1e" - }, "execution_count": null, - "source": "reversed_pair = (\"dog bites man\", \"man bites dog\")\nparaphrase_pair = (\"dog bites man\", \"a canine attacked a person\")\n\nfor a, b in [reversed_pair, paraphrase_pair]:\n print(f\"{util.cos_sim(model.encode(a), model.encode(b)).item():.3f} | {a!r} vs {b!r}\")", - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": "0.907 | 'dog bites man' vs 'man bites dog'\n0.570 | 'dog bites man' vs 'a canine attacked a person'\n" - } + "metadata": {}, + "outputs": [], + "source": [ + "# Your turn: try the polysemy case here\n", + "query = \"apple stock\"\n", + "candidates = [\"shares of a tech company\", \"a crisp red fruit\"]\n", + "\n", + "query_vec = list(model.embed([query]))[0]\n", + "for candidate in candidates:\n", + " candidate_vec = list(model.embed([candidate]))[0]\n", + " score = cosine_similarity(query_vec, candidate_vec)\n", + " print(f\"{score:.3f} | {candidate!r}\")" ] }, { "cell_type": "markdown", - "source": "The reversed sentence scores higher than the correct paraphrase.\n\nThe model rates *opposite meaning, identical words* as more similar than *same meaning, different words*. Embeddings shifted the problem rather than removing it, and that is worth knowing before you trust similarity with anything important.", - "metadata": { - "id": "x84spDgOca8s" - } + "metadata": {}, + "source": [ + "### Distance Metric Comparison\n", + "\n", + "| Metric | Best for | Notes |\n", + "|--------|----------|-------|\n", + "| Cosine | Text similarity, NLP (Natural Language Processing) models | Robust to different vector magnitudes. Most common default. |\n", + "| Dot product | Vectors already normalized to unit length | Numerically identical to cosine similarity once vectors are unit length, same score, not a separate metric. |\n", + "| Euclidean (L2) | Image embeddings, spatial data | Sensitive to magnitude; works best with models trained for it. |\n", + "| Manhattan (L1) | Grid-like or count-based data | Sums absolute differences per dimension rather than squaring them first, making it less affected by extreme values in any single dimension. |\n", + "\n", + "**Cosine vs. dot product:** for vectors normalized to unit length, dot product produces the exact same ranking as cosine similarity, it's a cheaper way to compute the same result, not a different metric. That's why Qdrant normalizes vectors on upload and computes a \"Cosine\" collection as a dot product internally.\n" + ] }, { "cell_type": "markdown", - "source": "## 4. Why Similarity Alone Is Not Enough\n\nHere is the failure that costs real money. A user types an exact product code and wants that product.", - "metadata": { - "id": "KBtd488gnM8H" - } + "metadata": {}, + "source": [ + "## 6. Why Similarity Alone Is Not Enough\n", + "\n", + "Sections 1 and 2 showed keyword search failing on synonyms, paraphrasing, polysemy, and word order. It's tempting to read that as \"semantic search replaces keyword search.\" It doesn't, each is strong exactly where the other is weak, as the next two cases show. (Filtering by recency, permissions, or other payload values, and combining that with ranking signals, is a separate layer on top of similarity, later modules cover it once you have a collection to filter.)\n", + "\n", + "### Word Order and Negation Still Trip It Up\n", + "\n", + "Section 2 said keyword search can't tell \"dog bites man\" from \"man bites dog.\" You'd expect semantic search to fix that. It mostly doesn't:\n", + "\n", + "| Pair | Cosine similarity |\n", + "|------|-------------------|\n", + "| \"dog bites man\" vs \"man bites dog\" | 0.907 |\n", + "| \"safe for kids\" vs \"harmful to kids\" | 0.779 |\n", + "| \"dog bites man\" vs \"a canine attacked a person\" | 0.570 |\n", + "\n", + "The first two rows score high, even though each pair means something different: one flips who's doing the biting, the other flips safe into dangerous. The model mostly notices that the two sentences share almost all the same words, so it calls them similar, even though a person would read them as opposites right away.\n", + "\n", + "The third row is the sharpest version of the problem: \"a canine attacked a person\" is a genuine paraphrase of \"dog bites man,\" meaning the same thing in different words, yet it scores lower (0.570) than the reversed, opposite-meaning sentence (0.907). Shared words move the score more than shared meaning does.\n", + "\n", + "So: semantic search is great at synonyms and paraphrasing, but shaky on word order and negation. Don't rely on a similarity score alone anywhere it actually matters whether something is \"safe\" or \"not safe.\"\n" + ] }, { "cell_type": "code", - "metadata": { - "id": "QPqHtpfr8BON", - "colab": { - "base_uri": "https://localhost:8080/" - }, - "outputId": "f6455de4" - }, "execution_count": null, - "source": "skus = [\"SKU-48290\", \"SKU-48291\", \"SKU-48292\", \"SKU-48293\"]\ncatalog = [f\"{sku} replacement part, ships in 2 days\" for sku in skus]\n\nquery = model.encode(\"SKU-48291 issue\")\nsims = util.cos_sim(query, model.encode(catalog))[0]\n\nscored = sorted(zip(sims.tolist(), skus), reverse=True)\nfor score, sku in scored:\n marker = \" <-- the one they asked for\" if sku == \"SKU-48291\" else \"\"\n print(f\"{score:.3f} {sku}{marker}\")\n\nprint()\nprint(f\"spread between best and worst: {scored[0][0] - scored[-1][0]:.3f}\")", - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": "0.626 SKU-48290\n0.624 SKU-48291 <-- the one they asked for\n0.616 SKU-48293\n0.610 SKU-48292\n\nspread between best and worst: 0.016\n" - } + "metadata": {}, + "outputs": [], + "source": [ + "# Reproduce the table above\n", + "word_order_pairs = [\n", + " (\"dog bites man\", \"man bites dog\"),\n", + " (\"safe for kids\", \"harmful to kids\"),\n", + " (\"dog bites man\", \"a canine attacked a person\"),\n", + "]\n", + "\n", + "for a, b in word_order_pairs:\n", + " a_vec = list(model.embed([a]))[0]\n", + " b_vec = list(model.embed([b]))[0]\n", + " score = cosine_similarity(a_vec, b_vec)\n", + " print(f\"{score:.3f} | {a!r} vs {b!r}\")" ] }, { "cell_type": "markdown", - "source": "The requested SKU is not first, and look at the spread: all four codes land within a couple of hundredths of each other.\n\nThat is not a bug in the model. Those strings are nearly identical, so their vectors are nearly identical, and the model has no idea that the final digit is the only part that matters. Semantically they are the same thing. Commercially, three of the four are the wrong product.\n\nNo amount of model tuning fixes this, because the model is not wrong. What is missing is a hard constraint.", - "metadata": { - "id": "uHJKpbpeIeeN" - } + "metadata": {}, + "source": [ + "### Exact Matching: Where Keyword Search Wins\n", + "\n", + "Not every query needs semantic understanding. A query for an exact SKU (Stock Keeping Unit, the unique code a retailer assigns to one specific product), like \"SKU-48291,\" needs an exact match instead. Try it yourself: embed the query and three candidate SKUs, then compare their cosine scores.\n" + ] }, { "cell_type": "code", - "metadata": { - "id": "sXE1jFXe6kW9", - "colab": { - "base_uri": "https://localhost:8080/" - }, - "outputId": "d518e4e2" - }, "execution_count": null, - "source": "TARGET = \"SKU-48291\"\n\n# An exact constraint on a metadata field, not a similarity score\nfiltered = [(score, sku) for score, sku in scored if sku == TARGET]\n\nfor score, sku in filtered:\n print(f\"{score:.3f} {sku}\")\n\nprint()\nprint(\"In Qdrant that constraint is a payload filter, evaluated while the search runs:\")\nprint(' must=[FieldCondition(key=\"sku\", match=MatchValue(value=\"SKU-48291\"))]')\nprint(\"You will write exactly that in Module 2.\")", - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": "0.624 SKU-48291\n\nIn Qdrant that constraint is a payload filter, evaluated while the search runs:\n must=[FieldCondition(key=\"sku\", match=MatchValue(value=\"SKU-48291\"))]\nYou will write exactly that in Module 2.\n" - } + "metadata": {}, + "outputs": [], + "source": [ + "from fastembed import TextEmbedding\n", + "import numpy as np\n", + "\n", + "model = TextEmbedding(model_name=\"sentence-transformers/all-MiniLM-L6-v2\")\n", + "\n", + "def cosine_similarity(a, b):\n", + " return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))\n", + "\n", + "query = \"SKU-48291 issue\"\n", + "candidates = [\"SKU-48292\", \"SKU-48291\", \"SKU-48290\"] # wrong, correct, wrong\n", + "\n", + "query_vec = list(model.embed([query]))[0]\n", + "\n", + "for candidate in candidates:\n", + " candidate_vec = list(model.embed([candidate]))[0]\n", + " score = cosine_similarity(query_vec, candidate_vec)\n", + " print(f\"{score:.3f} | {candidate}\")\n", + "\n", + "# Real output:\n", + "# 0.730 | SKU-48292 (wrong product)\n", + "# 0.734 | SKU-48291 (correct product)\n", + "# 0.765 | SKU-48290 (wrong product, scores HIGHEST)" ] }, { "cell_type": "markdown", - "source": "### Key insight\n\nDense similarity finds the neighbourhood. Filters, exact matches, and payload constraints find the right point inside it. Production search needs both, which is why real systems are hybrid: dense retrieval for meaning, sparse retrieval for exact tokens, and filters constraining both.", - "metadata": { - "id": "4bt3X40UQlAj" - } + "metadata": {}, + "source": [ + "Semantic similarity here doesn't just drift toward the wrong SKU, it ranks the wrong one first. A keyword-style exact filter gets it right, trivially: restrict the results to points where the `sku` payload field equals `\"SKU-48291\"`, and only that one product comes back, no embedding, no similarity score, just an exact match. Module 2 builds a real filtered Qdrant query once there's a collection with payloads to filter.\n", + "\n", + "Keyword matching isn't obsolete, it's exactly the right tool when a query needs to hit one precise token. Dense similarity finds the general neighborhood of relevant results; exact keyword matching finds the right point within it. Neither replaces the other.\n" + ] }, { "cell_type": "markdown", - "source": "## Your turn\n\nOne exercise, then Module 2.\n\nSwap in your own polysemy case below. Score `\"apple stock\"` against both senses and see which wins, then try a word your own domain overloads: *charge*, *bank*, *lead*, *plant*. Does the model pick the sense you meant?", - "metadata": { - "id": "SPpd7YIH0Cyu" - } - }, - { - "cell_type": "code", - "metadata": { - "id": "pbOzJdwihqZy", - "colab": { - "base_uri": "https://localhost:8080/" - }, - "outputId": "693088a7" - }, - "execution_count": null, - "source": "candidates = [\n \"shares of a technology company\",\n \"a crisp red fruit\",\n]\n\nfor candidate in candidates:\n score = util.cos_sim(model.encode(\"apple stock\"), model.encode(candidate)).item()\n print(f\"{score:.3f} | 'apple stock' vs {candidate!r}\")", - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": "0.480 | 'apple stock' vs 'shares of a technology company'\n0.225 | 'apple stock' vs 'a crisp red fruit'\n" - } + "metadata": {}, + "source": [ + "## 7. When a System Needs Both\n", + "\n", + "The SKU example above is why some production systems run semantic and exact retrieval together rather than picking one: a single collection can serve queries that need meaning and queries that need one precise token. This isn't a universal requirement, plenty of systems only ever need one or the other, but when a use case needs both, that's called **hybrid search**, combining **dense** (semantic/vector) retrieval with **sparse** (keyword-style, e.g. BM25) retrieval.\n", + "\n", + "Hybrid search is the next module's topic.\n" ] }, { "cell_type": "markdown", - "source": "## What's next: Module 2\n\n- What a vector is, and why it has hundreds of dimensions\n- How similarity works under the hood, and when it fails\n- Your first Qdrant collection: points, payloads, and your first query\n\n[Continue to Module 2](https://qdrant.tech/course/beginners/module-2/)", - "metadata": { - "id": "QToEDnzI44rO" - } + "metadata": {}, + "source": [ + "## 8. References & Further Reading\n", + "\n", + "**Qdrant docs:**\n", + "\n", + "- [Qdrant Documentation Overview](https://qdrant.tech/documentation/overview/) \u2014 How Qdrant's vector search engine fits together: collections, points, payloads, and APIs.\n", + "- [Distance Metrics](https://qdrant.tech/course/essentials/day-1/distance-metrics/) \u2014 Cosine, dot product, and Euclidean distance compared, and how to pick the right one for your embedding model.\n", + "- [Filtering](https://qdrant.tech/documentation/search/filtering/) \u2014 Payload filter syntax, indexed fields, and combining filters with vector queries.\n", + "\n", + "**Go deeper:**\n", + "\n", + "- [Vector Embeddings Explained](https://qdrant.tech/articles/what-are-embeddings/)\n", + "- [What Is a Vector Database?](https://qdrant.tech/articles/what-is-a-vector-database/)\n", + "- [FastEmbed: Qdrant's Efficient Python Library for Embedding Generation](https://qdrant.tech/articles/fastembed/)\n", + "- [Fine-Tuning Sparse Embeddings for E-Commerce Search, Part 1: Why Sparse Embeddings Beat BM25](https://qdrant.tech/articles/sparse-embeddings-ecommerce-part-1/)\n", + "- [What Is RAG in AI?](https://qdrant.tech/articles/what-is-rag-in-ai/)\n", + "\n", + "**Definitions:**\n", + "\n", + "- [Cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity)\n", + "- [Dot product](https://en.wikipedia.org/wiki/Dot_product)\n", + "- [Euclidean distance](https://en.wikipedia.org/wiki/Euclidean_distance)\n", + "- [Manhattan (taxicab) distance](https://en.wikipedia.org/wiki/Taxicab_geometry)\n", + "\n", + "## What's Next: Module 2\n", + "\n", + "In the next module, we'll break down:\n", + "\n", + "- What is a vector, and why does it have hundreds to thousands of dimensions?\n", + "- How do dimensions actually represent meaning?\n", + "- How similarity really works under the hood, and when it fails.\n", + "- Your first Qdrant collection: points, payloads, and your first query.\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + }, + "colab": { + "provenance": [], + "name": "Module1.ipynb" } - ] -} + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file