Powerful disk-first full-text search without the infrastructure.
Explore, filter, and search structured data at any scale — from live application data to historical archives, and AI-ready datasets.
GitHub: https://github.com/flatseek/flatrun · Organization: https://github.com/flatseek
Part of the Flatseek ecosystem
Flatseek (Keyword Search) • Flatvec (Vector Search) • Flatask (RAG Runtime) • Flatrun (LLM Inference Runtime) • Flatweight (AI Model Storage) • Flattune (LLM Fine-Tuning) • Flatlens (Data Visualization)
Demo: flatlens.demo.flatseek.io · Docs: flatseek.io/docs · Sample Datasets: HuggingFace
Search a 14 GB index hosted on HuggingFace through a serverless API running with only 2 GB of memory.
# demo CLI
flatseek search https://huggingface.co/datasets/flatseek/public-dataset/resolve/main/6.3M-books.fsk \
"title:dune"
[reading remote index] ✓
[searching remote] ✓
Found: 1,018 matches
{'name':'Statisticity','author':['Yaron Glazer'],'genres':['Science Fiction'],'...':'...'}# demo API
curl -X 'GET' \
'https://api.demo.flatseek.io/6.3M-books/_search?q=dune&from=0&size=20&bucket=https%3A%2F%2Fhuggingface.co%2Fdatasets%2Fflatseek%2Fpublic-dataset' \
-H 'accept: application/json'
{"hits":{"total":1018,"hits":[{"_source":{"name":"Statisticity","author":["Yaron Glazer"],"...":"..."}}]},"took":1}
That query runs against 6.3M books dataset hosted on HuggingFace, and only pulls the byte ranges it needs to answer it. The same index also powers the Flatlens - live dashboard — try it with zero installation.
Need natural language instead of search syntax? Flatask is the AI companion for Flatseek. It turns any Flatseek index into a chat-ready knowledge base.
flatask chat https://huggingface.co/datasets/flatseek/public-dataset/resolve/main/6.3M-books.fsk
You> Recommend books similar to Dune.
Assistant> Foundation, Hyperion, Children of Time...
You> How many books were published after 2010?
Assistant> 482,391Flatask translates natural language into optimized Flatseek queries, retrieves only the relevant records, then lets any LLM generate accurate, grounded answers. Learn more: github.com/flatseek/flatask
Distributed search engines like Elasticsearch and OpenSearch are excellent at large-scale production systems—but they assume always-on infrastructure.
For many workloads, that means paying for 24/7 servers, storage, replicas, and operational overhead even when you're simply searching documents.
Whether you're powering a production API, exploring a dataset, preparing training data, building a RAG pipeline, or publishing a searchable archive, the infrastructure often costs more than the search itself.
Lightweight alternatives avoid the cluster, but they're usually limited to local files. If your data lives in S3, HuggingFace, or object storage, you still need to download it before you can search it, making remote search harder than it should be.
Flatseek uses one query language across two index formats, so the same search engine works for both live production data and portable datasets.
| Directory index | .fsk archive |
|
|---|---|---|
| Data | Mutable | Immutable |
| Best for | Production APIs, applications, internal analytics | AI & RAG datasets, training data, historical archives, public searchable datasets |
| Updates | Inserts, updates, deletes | Rebuild the archive or renew the license |
| Distribution | Multi-file index | Single portable file |
Both formats expose the same query language, embedded library, CLI, and REST API—switching between mutable production data and immutable archives doesn't require changing your application.
The .fsk format packages an entire index into a single portable file,
making it easy to publish on object storage, CDN, or platforms with
file-count limits while preserving the same query capabilities.
500K documents • Article schema • SSD • Compared against Elasticsearch
| Metric | Flatseek | Elasticsearch |
|---|---|---|
| Search p50 | 7.9ms | 16.1ms |
| Range query hits | 501,011 (exact) | 505,044 (approximate) |
| Build 500K rows | 216s | 113s |
Full comparison including tantivy, Typesense, Whoosh, ZincSearch: docs/benchmark.md or bench.flatseek.io
| Capability | Description |
|---|---|
| Full-text search | Trigram-based search with wildcard, phrase, and boolean queries — docs/search.md |
| Range queries | Exact filtering on numeric, date, and keyword fields — docs/search.md |
| Sorting | Single or multi-field sorting — docs/search.md |
| Aggregations | Terms, stats, cardinality, date histograms — docs/aggregate.md |
| Nested & array fields | Query nested objects and match values inside arrays — docs/search.md |
| Multi-index search | Query multiple index directories with glob patterns — docs/multiindex.md |
| Cross-lookup | Join two indexes on a shared key field — docs/cross-lookup.md |
| Remote indexes | Search HTTP-hosted indexes without downloading the entire dataset — docs/storage.md |
| Portable archives | Package complete indexes into a single .fsk file — docs/cmd.md |
| Distribution & licensing | Public, password-protected, time-limited, or renewable license-based archives — docs/distribution.md |
| Embedded library | Query directly from Python without a server — docs/python.md |
| REST API | Elasticsearch-compatible Search, Bulk, and CRUD APIs — docs/restapi.md |
| CLI commands | Full reference for all subcommands (build, search, pack, unpack, export, slice, ...) — docs/cmd.md |
| Parallel indexing | Multi-worker index builds for faster ingestion — docs/build.md |
| Export | Stream matching documents as JSONL or CSV — docs/export.md |
| Slice | Materialize query results as a new standalone index — docs/build.md |
| Compaction | Reclaim disk space after large delete operations — docs/cmd.md |
| Serve & dashboard | Self-hosted API server with embedded dashboard — docs/serving.md |
curl -fsSL flatseek.io/install.sh | shIncludes:
• CLI
• REST API
• Flatlens dashboard (http://localhost:8000/dashboard)
pip install flatseekCLI only. For the Flatlens dashboard:
git clone https://github.com/flatseek/flatlensgit clone https://github.com/flatseek/flatseek.git
cd flatseek
pip install -e .Requirements: Python ≥ 3.10, macOS / Linux / WSL.
# Generate 100K dummy data
flatseek generate -r 100000 -s article -f csv -o ./data.csv
# Build index
flatseek build ./data.csv -o ./data
# Query via CLI
flatseek search ./data "program:raydium AND amount:>1000000"
# Serve API + dashboard
flatseek serve -d ./data
# Pack index into single portable .fsk file
flatseek pack ./data -o ./data.fsk
# Query from portable file
flatseek search data.fsk "program:raydium AND amount:>1000000"
# Serve API + dashboard from portable file
flatseek serve data.fsk
# Query a .fsk archive directly from HTTP — no full download
flatseek search https://huggingface.co/datasets/owner/repo/resolve/main/data.fsk "program:raydium AND amount:>1000000"
# Serve a .fsk archive directly from HTTP — no full download
flatseek serve https://huggingface.co/datasets/owner/repo/resolve/main/data.fsk
# Export results
flatseek export ./data "program:raydium AND amount:>1000000" -f jsonl --out exported.jsonl
flatseek export data.fsk "program:raydium AND amount:>1000000" -f jsonl --out exported.jsonl
→ API: http://localhost:8000 → Dashboard: http://localhost:8000/dashboard
Build an index once, upload it to any HTTP-accessible storage, and query it from anywhere using the same embedded library, CLI, or REST API.
Flatseek reads only the byte ranges required to answer each query, so even large indexes can be searched remotely without downloading the entire file.
Supported providers:
- HuggingFace Datasets / Buckets
- S3-compatible storage (Amazon S3, MinIO, Cloudflare R2)
- Vercel Blob
- Any static HTTP server or CDN
Try it now: Use the live dashboard at
flatlens.demo.flatseek.io to explore
remote .fsk indexes directly from your browser—no installation required.
All datasets below are hosted on HuggingFace. The Flatseek API serving the dashboard is deployed on a free Vercel hobby account — and it searches a 14 GB index in under 5 seconds by fetching only the byte ranges it needs.
| Dataset | Index Size | Documents | Index File | Try in Flatlens |
|---|---|---|---|---|
| 6.3M Books | 14.1 GB | 6.3M Goodreads books | Download | Open → |
| 1.2M Movies | 2.1 GB | 1.2M TMDB movies | Download | Open → |
| 5M Wikipedia | 1.39 GB | 5M Wikipedia articles | Download | Open → |
| 1.2M Songs | 947 MB | 1.2M Spotify tracks | Download | Open → |
| 500K Startups | 600 MB | 500K Product Hunt launches (2013–2026) | Download | Open → |
| 800K Domains | 539 MB | 800K WHOIS domain registrations | Download | Open → |
| 500K Actors | 84.2 MB | 500K movie actors | Download | Open → |
| 271K Athletes | 75.6 MB | 271K Olympic athletes (1800–2000) | Download | Open → |
Flatseek supports four distribution models, depending on your distribution and access-control requirements.
| Model | Format | Index File | Encryption | Expiration | Access Renewal |
|---|---|---|---|---|---|
| Public | Folder | sample-articles | None | Never | — |
| Password-Protected | Folder | sample-encrypted | Per-file ChaCha20 | Never | Change the passphrase |
| Time-Limited | Single .fsk |
demo_enclosed_active | Full-file ChaCha20 | Fixed date | Must repack the archive |
| License-Based | Single .fsk |
demo_license | Section-level ChaCha20 | Renewable token | Issue a new token — no repack needed |
Key difference — time-limited vs. license-based: when a time-limited archive expires, you must repack it from existing non-fsk index to extend access. When a license-based archive expires, you issue a new HMAC token — the index itself stays unchanged.
The credentials below are public demo credentials for the samples above — not secrets.
| Model | Dashboard link | Credentials |
|---|---|---|
| Public folder | sample-articles → | None |
| Password-protected folder | sample-encrypted → | Passphrase: flatseek |
Time-limited .fsk |
demo_enclosed_active → | Passphrase: flatlens_demo_enclosed |
License-based .fsk |
demo_license → | Token: ZGVtby11c2VyfDE4MTQ3NDU2MDB8MHxtUkJaTkN4WUdLeWhrV2NUMGN3Wlo0MkhUc1IvelNpYXBzUGo4a2tZdVhzPQ== |
| Guide | Description |
|---|---|
| Quick Start | Install, index, query — in 5 minutes |
| Indexing | Formats, column types, parallel builds, encryption |
| Query Language | Full syntax reference |
| CLI Reference | All CLI commands |
| Command Reference | Detailed reference for every CLI subcommand |
| REST API | API endpoints |
| Search | Full-text, wildcard, range, boolean, nested/array queries |
| Aggregations | Terms, stats, date histogram, cardinality |
| Multi-Index | Wildcard search across multiple index directories |
| Cross-Lookup | Join two indexes on a shared key field |
| Python Library | Query, insert, upsert, update, delete, bulk from Python |
| Upsert & Write Ops | Insert, upsert, update, delete, bulk operations |
| Export | Stream matching documents as JSONL or CSV |
| Serve & Dashboard | Self-hosted API server with embedded dashboard |
| Remote Storage | HuggingFace, S3, Vercel Blob backends |
| Distribution | Public, password-protected, time-limited, or license-based archives |
| Schemas | Supported column types |
| Architecture | Structural and behavioral map |
| Internals | Deep technical breakdown |
| Tests | Test coverage matrix and gap analysis |
| Implementation Matrix | Feature checklist across CLI, library, and REST API (auto-generated) |
PRs welcome. Run tests:
pytest tests/ -v # all testsApache 2.0. See LICENSE.