intelligent-search-platform

Codebase Structure

Repository Layout

intelligent-search-platform/
├── backend/
│   ├── extensions/                     # JS transform extensions (source of truth)
│   │   ├── catalog_mapper.js           # REST API → catalog schema mapper (run_order 0)
│   │   ├── version_score.js            # version/badge popularity boost (run_order 1)
│   │   └── merchandising_rules.js      # query-aware boost_queries stamping (run_order 2)
│   ├── src/isp/
│   │   ├── main.py                     # FastAPI app, lifespan, CORS, router registration
│   │   ├── config.py                   # Settings from env vars (.env)
│   │   ├── models.py                   # Pydantic models (request/response shapes)
│   │   ├── routers/
│   │   │   ├── indexes.py              # /v1/indexes/* — CRUD + ingest + rebuild
│   │   │   ├── search.py               # /v1/indexes/{id}/search
│   │   │   ├── catalog.py              # /v1/catalog — PLP product search + facets + suggest
│   │   │   ├── content.py              # /v1/content/{slug} — generic content API
│   │   │   └── extensions.py           # /v1/extensions — JS transform library management
│   │   └── services/
│   │       ├── database.py             # asyncpg pool + init_db() (table creation)
│   │       ├── index_service.py        # PostgreSQL CRUD for indexes/status/splits/discovery
│   │       ├── extension_service.py    # PostgreSQL CRUD for extensions + index assignments
│   │       ├── extension_runner.py     # V8 JS execution via PyMiniRacer
│   │       ├── typesense_client.py     # All Typesense operations
│   │       ├── embedding_service.py    # Voyage AI + two-tier cache (LRU + Redis)
│   │       ├── rest_connector.py       # Fetch + parse REST API data
│   │       └── sitemap_connector.py    # Crawl + parse sitemaps
│   ├── pyproject.toml                  # Dependencies (py-mini-racer, voyageai, asyncpg, ...)
│   ├── Dockerfile
│   └── .env                            # Environment variables
│
├── frontend/
│   └── src/
│       ├── pages/index.astro           # Entry point, mounts App component
│       ├── lib/
│       │   ├── api.ts                  # All fetch calls to backend API
│       │   ├── store.ts                # Svelte stores (appState, indexes, loading, error)
│       │   └── types.ts                # TypeScript interfaces
│       └── components/
│           ├── App.svelte              # Root: layout, routing, delete modal
│           ├── Header.svelte           # Top nav bar
│           ├── SourcesSidebar.svelte   # Left sidebar: source list + "⚙ Extensions" footer
│           ├── SourceContent.svelte    # Main panel: docs list + search + "⚙ Extensions" button
│           ├── RebuildPanel.svelte     # Rebuild modal: sitemap form OR curl paste + REST form + progress
│           ├── CreateWizard.svelte     # 4-step wizard to add a new source
│           ├── IndexDetail.svelte      # Tabbed detail view: Overview / Connector / Mapping / Fields / Documents / Search / API / Suggestions / Personalisation / Boost Rules
│           ├── FieldsTab.svelte        # Fields tab: table of Typesense fields with Searchable / Facet / Multi-value toggles
│           ├── ExtensionsPanel.svelte  # Global extension library (list/create/edit/delete)
│           └── IndexExtensionsPanel.svelte  # Per-index extension assignment overlay
│
├── docs/                               # Design documents (this folder)
├── docker-compose.yml
├── deploy-all.sh                       # Start everything in order
├── deploy-api.sh                       # Rebuild + restart backend
├── deploy-ui.sh                        # Build + restart frontend
└── deploy-db.sh                        # Start + wait for PostgreSQL

Backend: Key Files

main.py

FastAPI application entry point.

@asynccontextmanager
async def lifespan(app):
    await init_db()                    # create tables on startup
    _ensure_catalog_collection()       # create/verify catalog Typesense collection
    await _seed_extensions_from_files()# load *.js → DB → auto-assign to external indexes
    yield

app = FastAPI(lifespan=lifespan)
app.include_router(indexes.router,    prefix="/v1")
app.include_router(search.router,     prefix="/v1")
app.include_router(content.router,    prefix="/v1")
app.include_router(catalog.router,    prefix="/v1")
app.include_router(extensions.router, prefix="/v1")

_seed_extensions_from_files() scans settings.extensions_dir (default /app/extensions) for *.js files, upserts each using name as the merge key, and auto-assigns them based on header annotations:

Header annotation Effect
// @name … Display name (used as the upsert key)
// @description … One-line description
// @type transform Default. Per-item extension; function is transform(doc)
// @type pre_processor Full-response extension; function is pre_process(response). Never auto-assigned — must be manually assigned via the Extensions tab
// @assign external Auto-assign to catalog write-back indexes only (default for transform type)
// @assign all Auto-assign to every index
// @assign none Skip auto-assignment entirely
// @run_order N Execution order within the same stage (lower = first)

This makes the git repo the single source of truth for extension code — no manual DB management needed.

config.py — Settings

All configuration is read from environment variables or .env:

Setting Default Purpose
typesense_host typesense Typesense container hostname
typesense_port 8108 Typesense port
typesense_api_key local-dev-key-change-me Typesense auth key
database_url postgresql://...@isp-db:5432/isp asyncpg DSN
voyage_api_key "" Voyage AI key (empty = lexical only)
redis_url redis://isp-redis:6379/0 Embedding cache L2
embed_cache_ttl 86400 Redis TTL in seconds (24h)
embed_lru_size 2048 In-memory LRU slots per pod
extensions_dir /app/extensions Directory scanned for *.js files

models.py — Core Pydantic Models

Model Purpose
IndexMeta Full source metadata (id, name, connectors, external_*, doc_count, field_config)
IndexCreate Request body for creating an index (name, description)
RestConnectorConfig REST connector (url, method, headers, body, field_map, field shortcuts)
SitemapConnectorConfig Sitemap connector (url, max_pages, selectors, patterns)
IngestStatus Job status (state, total, indexed, error)
SearchRequest Search request (q, page, per_page, mode, filters: dict[str, list[str]])
DocumentItem Search result shape (id, title, content, url, source_type, indexed_at)
FieldUpdate PATCH body for field properties (facet, multi_value, searchable — all optional booleans)

RestConnectorConfig.field_map — optional source→target field renames applied before indexing. Setting a target to "" drops the field. Special target names title, content, url, id override the corresponding core field shortcuts.


Backend: Routers

routers/indexes.py — Endpoints

Method Path Action
GET /v1/indexes List all sources
POST /v1/indexes Create source + provision Typesense collection
GET /v1/indexes/{id} Get source metadata
DELETE /v1/indexes/{id} Delete source + Typesense data
POST /v1/indexes/discover Auto-import Typesense collections + auto-assign extensions
GET /v1/indexes/{id}/status Poll ingest job status
POST /v1/indexes/{id}/connectors/rest Save REST connector config (no ingest)
POST /v1/indexes/{id}/connectors/sitemap Replace sitemap connector config (no crawl); replaces existing sitemap connector in-place
POST /v1/indexes/{id}/ingest/rest Start REST ingest (background)
POST /v1/indexes/{id}/ingest/sitemap Start sitemap ingest (background)
POST /v1/indexes/{id}/rebuild Rebuild source (background); accepts RestConnectorConfig or SitemapConnectorConfig; 409 if already running
GET /v1/indexes/{id}/status Poll ingest/rebuild job state
GET /v1/indexes/{id}/documents List documents — includes vec (paginated)
GET /v1/indexes/{id}/documents/public List documents — vec excluded (external consumers)
PATCH /v1/indexes/{id}/documents/{doc_id} Update editorial fields without re-indexing
GET /v1/indexes/{id}/fields List all Typesense fields merged with stored field_config
PATCH /v1/indexes/{id}/fields/{field_name} Toggle facet / multi_value / searchable for a field; persists to field_config

Key internal functions:

Method Path Action
GET /v1/intent/tab Detect whether query is device or accessory
POST /v1/catalog Hybrid product search (PLP)
GET /v1/catalog/facets Manufacturer / category / price facets
GET /v1/suggest Typeahead autocomplete

URL in search results is read directly from metadata.url — set by the JS extension during rebuild. No URL base constants exist in Python.

routers/extensions.py — Extension Library

Method Path Action
GET /v1/extensions List all extensions
POST /v1/extensions Create a new extension
GET /v1/extensions/{id} Get one extension
PUT /v1/extensions/{id} Update name/description/code
DELETE /v1/extensions/{id} Delete (cascades index assignments)
GET /v1/indexes/{id}/extensions List extensions assigned to an index
POST /v1/indexes/{id}/extensions Assign extension to index
DELETE /v1/indexes/{id}/extensions/{ext_id} Unassign extension
Method Path Action
POST /v1/indexes/{id}/search Search index (hybrid/lexical/semantic) — includes vec
POST /v1/indexes/{id}/search/public Same search — vec excluded (external consumers)

Both routes share _run_search(exclude_vec) internally. Ranking is identical; the vector is used for candidate retrieval inside ISP and stripped before the response is serialised.

routers/content.py — Generic Content API

Method Path Action
GET /v1/content List all sources with their slugs
GET /v1/content/{slug} List documents for a source
GET /v1/content/{slug}/search Search within a source

Slugs are auto-derived from index names: "Smart Watches" → "smart-watches".


Backend: Services

services/database.py

Manages the asyncpg connection pool and init_db() which creates all five tables (indexes, ingest_status, deleted_splits, extensions, index_extensions) idempotently on startup.

services/index_service.py

PostgreSQL CRUD for the primary metadata tables:

_KNOWN_SPLITS maps Typesense collection names to named splits with display names, descriptions, and filter expressions. Add new catalog splits here when new external collections are deployed.

services/extension_service.py

PostgreSQL CRUD for the extension library and index assignments:

services/extension_runner.py

Executes JS extensions in V8 isolates via py-mini-racer. Two separate execution stages:

Stage 1 — pre-processor (full response):

Stage 2 — transform (per item):

Shared behaviour:

Inline pre-processor (connector config):

services/typesense_client.py

All Typesense API calls. Key functions:

All read functions accept exclude_vec=False (default). Pass True for public-facing calls to strip the vec field via Typesense exclude_fields: "vec".

services/infra/job_lock.py

Redis-backed distributed mutex preventing concurrent ingest/rebuild runs across multiple ISP instances:

services/embedding_service.py

Voyage AI voyage-3 embedding with a two-tier cache:

services/rest_connector.py

REST API data fetching:


Frontend: Key Files

lib/store.ts — Global State

appState: { selectedIndexId, view, detailTab }
indexes: IndexMeta[]
loading: boolean
error: string | null
view values: 'indexes' 'extensions'

lib/api.ts — Backend Client

All calls go to PUBLIC_API_URL (set at Astro build time). Key methods:

lib/types.ts — TypeScript Interfaces

Mirrors Pydantic models: IndexMeta, RestConnectorConfig, IngestStatus, DocumentItem, ExtensionMeta, IndexExtension, FieldEntry, FieldConfig.

FieldEntry — single field row returned by GET /fields:

interface FieldConfig  { facet: boolean; multi_value: boolean; searchable: boolean; }
interface FieldEntry extends FieldConfig {
  name: string; type: string; optional: boolean; system: boolean;
}

components/sources/FieldsTab.svelte — Fields Tab

Renders a table of all Typesense fields for the selected index with toggle controls.

App.svelte — Root Component

SourceContent.svelte — Main Panel

RebuildPanel.svelte — Rebuild UI

CreateWizard.svelte — 4-Step Wizard

ExtensionsPanel.svelte — Global Extension Library

IndexExtensionsPanel.svelte — Per-Index Assignment Overlay


Environment Variables (backend/.env)

TYPESENSE_HOST=typesense
TYPESENSE_PORT=8108
TYPESENSE_API_KEY=local-dev-key-change-me
DATABASE_URL=postgresql://isp:isp-dev-password@isp-db:5432/isp
VOYAGE_API_KEY=<your-voyage-key>
REDIS_URL=redis://isp-redis:6379/0
EMBED_CACHE_TTL=86400
EMBED_LRU_SIZE=2048
EXTENSIONS_DIR=/app/extensions
HOST=0.0.0.0
PORT=8001

References

Citations below are sourced from training knowledge (not live browsing). DOI and arXiv links are provided where known.

  1. Fowler, M. (2002). Patterns of Enterprise Application Architecture. Addison-Wesley. ISBN: 0-321-12742-0 — Defines the Service Layer, Repository, and Data Mapper patterns that structure ISP’s services/ and routers/ separation; each router delegates all I/O to domain services.

  2. Gamma, E., Helm, R., Johnson, R., & Vlissides, J. (1994). Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley. ISBN: 0-201-63361-2 — “Gang of Four” patterns; ISP uses the Singleton pattern (module-level _pool, _client globals) and the Strategy pattern (embedding provider selection by model id).

  3. Richardson, C. (2018). Microservices Patterns. Manning Publications. ISBN: 978-1-617-29454-1 — Microservice decomposition principles; ISP is a micro-monolith (all services in one process) but applies the same domain-service / infra-service boundary as microservice patterns recommend.

  4. Wynne, M., & Hellesøy, A. (2017). The Cucumber Book: Behaviour-Driven Development for Testers and Developers. Pragmatic Bookshelf — Background on background tasks and async pipelines; ISP’s BackgroundTasks ingest model follows the same separation-of-concerns principle.

  5. Merkel, D. (2014). “Docker: Lightweight Linux Containers for Consistent Development and Deployment.” Linux Journal, 2014(239). ISSN: 1075-3583 — Docker Compose enables the multi-service dev environment documented in backend/.env and docker-compose.yml.

  6. Dahl, R., & Schlueter Langford, I. (2013). Node.js in Action. Manning Publications. ISBN: 978-1-617-29036-9 — Background on V8 JavaScript engine; PyMiniRacer embeds V8 to execute JS extensions in sandboxed isolates without Node.js APIs.