Semantic Movie Discovery System

A semantic curation engine for film discovery, built on a private, million-title movie knowledge base.

Executive summary

Most movie systems answer: “What are other people likely to watch?”

This system answers: “What kind of movie experience is the user trying to find?”

It is a large-scale movie discovery and curation engine that helps people find films through meaning, mood, cultural context, quality signals, and personal intent — not only title, genre, or keyword matching.

At a simple level, it answers questions like:

• “Find me weird dystopian city movies that feel depressing and massive.”
• “Show me so-bad-it’s-good horror movies from the 80s.”
• “Create a Plex collection of obscure sci-fi movies with cult appeal.”
• “Find movies like The Hitchhiker’s Guide to the Galaxy, but more absurd and less action-oriented.”

The core insight: movie discovery is not just a database search problem. People often search with vague, emotional, cultural, or subjective language. Traditional systems struggle because they lean on genres, popularity charts, and collaborative filtering. This platform combines enriched metadata, semantic embeddings, lexical search, rating intelligence, popularity/obscurity controls, and (on the roadmap) an LLM-driven tuning layer that translates human intent into retrieval strategy.

The result is a system that can behave less like a search box and more like a knowledgeable film curator — especially once automatic tuning sits in front of the engine.

Strategic positioning: a semantic curation platform over a constructed movie knowledge base — not merely a search site, and not a traditional recommender.

The problem

Users search with intent, not metadata

People rarely arrive with clean filters. They arrive with feelings, references, eras, tones, and cultural categories:

A single fixed ranking algorithm fails because the correct balance of signals changes with intent.

Incumbent tools optimize the wrong objective

Gap: there is no widely available product that combines million-title coverage, semantic retrieval, intent-aware ranking, and personal library overlay in one self-hosted or API-first platform.

What the system is

Movie Index (internal project name) is a locally hosted movie intelligence platform:

• It builds and owns a private movie catalog (~1.2M+ TMDB identities; bulk snapshots can exceed 1.4M rows when importing large CSV archives).
• It enriches records from TMDB (and optional bulk sources), indexes them for hybrid search, and serves ranked results via HTTP API and web UI.
• It optionally overlays a user’s Plex library so discovery can mean “from the whole world” or “from what I already own.”

One-line definition:
A semantic curation engine that turns fuzzy human intent into tunable retrieval over a very large, normalized movie knowledge base.

What makes it different

Different question, different product

Tunable retrieval, not one algorithm

The engine exposes adjustable fusion weights and cutoff strategies so the same query infrastructure can serve opposite goals:

Key differentiator (product): an automatic curation layer (planned) that sets these knobs from natural language so casual users never see them.

Business opportunities

Each opportunity below reuses the same core: catalog + embeddings + hybrid search + ranking + (optional) library overlay.

Consumer movie discovery app

Product: Public or freemium web app — “describe the movie you want.”

Example queries:

• “Movies that feel like lonely neon cities at night.”
• “Absurd British sci-fi comedies.”
• “Forgotten 90s thrillers that are actually good.”
• “Bad movies that are fun, not just bad.”

Positioning: Competes with discovery and exploration (Letterboxd-adjacent browsing, film Twitter/list culture, niche cinephile search) — not with Netflix-style “what to stream tonight on our platform.”

Differentiator: Users do not need to know filters, genres, or metadata vocabulary. They describe vibe, era, tone, or cultural category; the system translates that into retrieval strategy.

Monetization paths: subscription, affiliate links (where legally appropriate), premium collections, API tier for power users.

Plex / Jellyfin collection generator

Product: Connect to a user’s media server; generate curated collections from natural language.

Example queries:

• “Build a cult sci-fi collection from my library.”
• “So-bad-it’s-good movies I already own.”
• “1980s creature-feature playlist.”
• “Hidden gems I forgot I had.”

Why this niche is strong:

• Users already maintain large personal libraries and care about organization.
• They are underserved by semantic discovery inside Plex/Jellyfin.
• No streaming of copyrighted content required — only metadata analysis and collection instructions returned to the local server.

Technical fit today: Plex sync marks in_library on catalog rows; search and collections APIs accept in_library: true filters. Jellyfin would be a parallel integration.

Monetization paths: one-time license, subscription plugin, homelab “pro” tier.

Movie metadata and semantic search API

Asset: A constructed, normalized, embedded catalog — not raw TMDB dumps.

Potential API customers:

• Indie app developers
• Recommendation startups
• Plex/Jellyfin plugin authors
• Film researchers and educators
• AI application builders
• Media catalog / metadata companies
• Hobbyists building local movie tools

API capabilities (existing or near-existing):

Positioning: A “semantic layer” on top of movie metadata — the hard data engineering and embedding work already done.

Monetization paths: usage-based API, tiered keys, enterprise license, white-label.

Precomputed embedding dataset

Problem: Generating embeddings for 1M+ movies is expensive, slow, and operationally painful (GPU batching, model versioning, index rebuilds).

Product: Licensed dataset bundles:

• Movie identity (TMDB ID, title, year, etc.)
• Plot/metadata embedding vectors
• (Roadmap) Historical/cultural context embeddings
• Similarity index metadata / version documentation
• Incremental update packages when the model or enrichment changes

Buyers: AI developers who want movie search or recommendations without building the pipeline.

Automated editorial and content marketing

Because the engine can surface clusters, outliers, and thematic slices, it can power:

• Listicles (“Weirdest low-budget 90s sci-fi”)
• Newsletter segments
• SEO landing pages
• Social content calendars

Examples:

• “Movies that accidentally became cult classics”
• “Dystopian city films before and after Blade Runner”
• “The best bad shark movies you’ve never heard of”

Monetization paths: ad-supported media property, B2B content tooling for publishers, lead gen for a consumer app.

Technical platform

This section covers architecture and implementation for readers who need credible technical depth — and for technical partners evaluating feasibility.

Architecture at a glance

flowchart TB
  subgraph external [External — ingestion only]
    TMDB_EXP[TMDB Daily ID Exports]
    TMDB_API[TMDB API v3]
    KAGGLE[Kaggle TMDB CSV snapshot — optional bulk]
    PLEX[Plex server — optional]
  end

  subgraph server [Single-server deployment — Docker Compose]
    API[FastAPI + Uvicorn — port 8080]
    WORKER[Enrichment worker — continuous]
    CLI[movie-index CLI / cron]

    subgraph data [PostgreSQL 16 + extensions]
      META[Constructed metadata — ~1M+ rows]
      FTS[Full-text search — tsvector + GIN]
      TRGM[Trigram title match — pg_trgm]
      VEC[Vector index — pgvector HNSW]
      QUEUE[Enrichment queue + phase tracking]
      COLL[Saved collections]
    end

    MODELS[Local model cache — Hugging Face weights]
    ARTIFACTS[Poster cache — local disk]
  end

  TMDB_EXP --> CLI
  TMDB_API --> WORKER
  KAGGLE --> CLI
  PLEX --> API
  CLI --> META
  WORKER --> META
  META --> FTS
  META --> VEC
  API --> FTS
  API --> VEC
  API --> TRGM
  MODELS --> VEC

Design principle: At query time, the system reads only PostgreSQL and local files — not TMDB, not cloud embedding APIs. External services are used during ingestion and sync only. That yields predictable latency, offline-capable search (once built), and no per-query SaaS inference bill.

Software stack

Deliberate non-choices (v1): No Elasticsearch, Pinecone, or managed vector DB — one database reduces operational cost for homelab and early commercial pilots.

Data sources and catalog construction

No single upstream provider ships a complete, query-ready movie database. The platform merges sources into one constructed catalog:

Scale targets:

• ~1.2 million movie identities from TMDB export workflow (project design target).
• ~1.42 million rows available in bundled Kaggle CSV snapshot (useful for bulk import and gap-fill).
• Enrichment at 4 TMDB requests/sec ≈ 3–4 days of continuous worker time for a full API detail pass over ~1.2M IDs (order-of-magnitude planning number).

Enrichment pipeline (automated):

1. Import IDs (export and/or Kaggle).
2. Queue movies for TMDB API detail fetch (priority: in-library first, then popularity).
3. After each batch: build search documents and embedding vectors (configurable ENRICHMENT_AUTO_INDEX).
4. Track per-movie enrichment phases: catalog spine → core metadata → poster media.

Stored metadata per movie (representative):

• Identity: TMDB ID, IMDb ID, titles, release date/year
• Text: overview, tagline, search document (derived)
• Facets: genres, keywords (JSON arrays)
• People: cast (top billed), crew (directors/writers)
• Franchise: collection id/name
• Signals: popularity, vote_average, vote_count, adult flag
• Media: poster paths, locally cached poster files
• Library: in_library boolean
• Provenance: detail_fetched_at, search_doc_built_at, enrichment phase status

Search document and embedding model

Search document (today): A single composed text block per movie used for both FTS indexing and embedding input:

• Title, original title, year, tagline, overview
• Genres, keywords (capped), top cast, directors
• Collection name

Embedding model (current default):

Models run entirely locally — weights cached under data/models/. No OpenAI/Cohere-style per-query embedding fees.

Hardware: Auto-detects CUDA → Apple MPS → CPU. Docker worker/API containers typically use CPU; native runs on Apple Silicon can use MPS for faster embedding batches.

Roadmap — second embedding space (“cultural / context”):
Planned separate representation for production history, reception, cult status, trivia, and “why this movie matters” — critical for queries like “so bad it’s good” where plot text is insufficient. Not yet implemented; today all semantic search uses the plot/metadata document embedding.

Hybrid retrieval pipeline

For each query, the engine:

1. Embeds the query with the same model (query-specific encoding per model family — e.g. Qwen uses a query prompt name).
2. Runs lexical branch:
   • PostgreSQL FTS rank on search_tsv (websearch_to_tsquery, English).
   • Trigram similarity on title (threshold > 0.2).
3. Runs semantic branch:
   • pgvector top-K by cosine distance on movie_embeddings, respecting SQL filters.
4. Fuses lexical + semantic candidate lists with Reciprocal Rank Fusion (RRF) — default k = 60.
5. Applies weighted score fusion across multiple signals (see below).
6. Optionally applies membership cutoff (how many results “belong” in the set).

sequenceDiagram
  participant User
  participant API
  participant Encoder
  participant PG as PostgreSQL

  User->>API: Natural language query + filters
  API->>Encoder: Embed query locally
  Encoder-->>API: Query vector (1024-d)
  par Lexical
    API->>PG: FTS + trigram title search
  and Semantic
    API->>PG: pgvector HNSW nearest neighbors
  end
  PG-->>API: Candidate pools (e.g. top 500 each)
  API->>API: RRF merge + weighted scoring + cutoff
  API-->>User: Ranked hits + explainable scores

Ranking dimensions

The API accepts a ScoringConfig with fusion weights (0–10 scale) and rating/cutoff settings:

Bayesian rating:
Raw TMDB vote_average is misleading for low vote_count. The system shrinks ratings toward a catalog mean (configurable prior, default 50 pseudo-votes) so that:

• “1.5 stars from 3 people” does not dominate “3.8 stars from 80,000 people.”

Reversing the quality signal (boost rating_low) surfaces titles that are credibly poorly rated at scale — a key ingredient for “so bad it’s good.”

Membership cutoff modes:

Elbow mode addresses: “Chupacabra movies” (tight cluster) vs “dreamlike loneliness” (broad semantic spread) without always returning exactly 100 or 200 titles.

Metadata filters

SQL-level constraints applied before or during retrieval:

API surface

Optional X-API-Key when API_KEY is set — suitable for LAN or partner pilots.

Example search request:

{
  "query": "depressing giant dystopian city",
  "limit": 50,
  "filters": { "year_min": 1970, "in_library": false },
  "scoring": {
    "weights": { "semantic": 2.0, "rrf": 1.0, "obscurity": 0.5 },
    "cutoff_mode": "elbow",
    "cutoff_on": "semantic"
  }
}

Plex integration

• Config: PLEX_BASE_URL, PLEX_TOKEN, optional PLEX_LIBRARY_NAME.
• Sync lists movie libraries, extracts TMDB/IMDb GUIDs, matches to catalog, sets in_library.
• Priority boost: titles in the user’s library are enriched and indexed first.
• Search UI defaults can restrict to library-only for “what should I watch from what I own?”

Jellyfin: Not implemented; the same overlay pattern applies.

How discovery works (end-to-end)

Example A — Thematic / mood query

User: “Depressing movies that take place in giant dystopian cities.”

Ideal retrieval plan (manual today; automatic via LLM later):

Example B — Canon / best-of query

User: “Best comedies of the 80s.”

Example C — “So bad it’s good”

User: “Funny-bad horror from the 80s.”

Example D — Hidden gems

User: “Forgotten films that are actually good but not famous.”

Conversational refinement (roadmap)

Multi-turn dialogue adjusts the same knobs:

1. “Too many movies with ‘city’ in the title” → reduce lexical/title weight, increase semantic.
2. “Darker and less mainstream” → increase obscurity, decrease notoriety, refine semantic query expansion.

The user steers like talking to a film expert — without operating a control panel.

Built today vs roadmap

Honest status framing for partners and investors. Screenshots in this post are from a TMDB-scale catalog (~1.2M titles) and intentionally omit personal-library overlays and media-server sync — those are optional deployment features, not required for semantic discovery.

Built and operational

In progress / partial

Roadmap (high value, not yet built)

Deployment and data ownership

Self-hosted first

• Typical deployment: one server (homelab Mac mini, NAS VM, small cloud VM).
• All data under MOVIE_INDEX_DATA_DIR (default ./data): Postgres files, exports, models, posters, backups.
• Moving hosts: copy repo + .env + entire data/ tree — no re-embedding required.

Why self-hosted matters commercially

Hosted SaaS (future)

A hosted tier is compatible with the architecture but is a go-to-market choice, not a technical requirement. TMDB attribution and API terms must be reflected in any public UI.

Competitive context

Moat hypothesis: The combination of million-title constructed catalog, hybrid retrieval, intent-aware multi-signal ranking, library overlay, and (when shipped) LLM tuning is harder to replicate than any single component alone.

Risks, dependencies, and compliance

Recommended near-term product

Strongest near-term wedge: a simple interface —

“Describe the kind of movie you want.”

→ curated list → conversational refinement.

Secondary wedge: Plex collection generator for the homelab community — passionate users, clear pain, willingness to pay for tooling.

Strategic summary

The engine is intentionally complex so the user experience can be simple. The business opportunity is to productize that translation layer — turning a powerful retrieval laboratory into a film discovery assistant that traditional search and recommendation stacks do not provide.

Appendix: query → strategy mapping