Option A — Phase 13: Website Knowledge from Web (GPT-4o Mini Search)

This document is the plan for fetching store website knowledge from the web using the openai/gpt-4o-mini-search-preview model, storing it in an editable field and in the RAG pipeline so the chat AI can use it. A new Knowledge admin screen provides a text field (populated by “Fetch All Website Knowledge From Web”). The store’s website URL is derived from the existing shop_domain (shops table) — e.g. https://{shopDomain} — with no new Settings field required.

Prerequisites: Existing RAG pipeline (KnowledgeSource, KnowledgeDocument, KnowledgeChunk, Embedding), chat API with OpenRouter, Settings screen, and billing (CHAT_MARKUP_FACTOR, credits).

1. Objective

Website URL: Use the existing shop_domain column on the shops table (e.g. store-name.myshopify.com). Build the storefront URL as https://{shopDomain} when fetching website knowledge. No new field or Settings change required.
Knowledge screen: New admin screen “Knowledge” with:
- A text field (multiline) that holds the website knowledge content. The user can edit it after it is fetched.
- A “Fetch All Website Knowledge From Web” action (button/box) that:
  1. Reads shop_domain from the shop and builds the URL https://{shopDomain}.
  2. Calls openai/gpt-4o-mini-search-preview (via OpenRouter) with a prompt to gather all relevant information about that website URL.
  3. Populates the text field with the model’s response (and optionally saves it to the shop row).
Billing: The call to the search model uses the same CHAT_MARKUP_FACTOR logic as chat (log to openrouter_calls, deduct from credit balance with markup).
RAG storage: The final knowledge text (from the field, after fetch or after user edits) is ingested into the RAG tables: KnowledgeSource (e.g. type website), KnowledgeDocument(s), KnowledgeChunk(s), Embedding(s), so the chat agent can use it.
Optional: Store the full knowledge text in the shops table (e.g. website_knowledge or web_knowledge_raw) for persistence and so the Knowledge screen can load it on next visit.

2. Model and API

Model: openai/gpt-4o-mini-search-preview (OpenRouter).
Use case: Send a prompt such as “Fetch and summarize all relevant, public information about this website: [URL]. Include key pages, policies, FAQs, product or brand info, and anything a customer might ask about.” The URL is https://{shopDomain} from the shops table. The model uses its search/crawl capability to return a consolidated text.
API: Same OpenRouter chat/completions endpoint as existing chat; different model and a single “user” or “system” prompt. No embeddings endpoint for this step (embeddings are created later when ingesting into RAG).

3. Website URL (existing shop_domain)

Use the existing shop_domain column on the shops table (Prisma: shopDomain, mapped to shop_domain). This is the store’s Shopify domain (e.g. appifire-ai-chat-test-store.myshopify.com).
When fetching website knowledge, build the storefront URL as https://{shopDomain}. No new column or Settings field is required.

4. New Screen: Knowledge

Route: e.g. app/routes/app.knowledge.jsx (or app.web-knowledge.jsx).
Nav: Add a “Knowledge” link in the app nav (e.g. in app/routes/app.jsx).
Layout:
- Heading: “Knowledge” (or “Website Knowledge”).
- Text field (multiline): Large text area showing the current website knowledge content. Editable by the user. Load from: (a) shop’s stored website_knowledge (if we add it), or (b) latest KnowledgeDocument rawText for type website, or (c) in-memory only until “Save”/ingest. Plan: store in DB (shops or a single doc) so it persists and is loadable.
- Box/button: “Fetch All Website Knowledge From Web”
  - On click: call an action that (1) reads shopDomain for the shop and builds URL https://{shopDomain}; (2) if shop domain is missing, return error or toast; (3) calls OpenRouter with model openai/gpt-4o-mini-search-preview and prompt to fetch info for that URL; (4) applies CHAT_MARKUP_FACTOR and logs to openrouter_calls, deducts credits; (5) returns the model response and updates the text field (and optionally saves to shop).
- Save / Ingest: A “Save and add to RAG” (or “Save”) button that (1) saves the current text field content to the shop (if we have website_knowledge) and (2) runs ingestion into knowledge_sources (type website), knowledge_documents, knowledge_chunks, embeddings. So the chat can use it.

5. Database and RAG

shops table:
- Use existing shop_domain for the website URL; no new column for domain.
- website_knowledge (Text?, nullable) — optional; full fetched/edited knowledge text so the Knowledge screen can show and edit it, and we have a single source of truth before chunking. Migration only if this column is added.
RAG:
- One KnowledgeSource per shop with type website (e.g. name “Website Knowledge”).
- One KnowledgeDocument per “website” knowledge (e.g. externalRefId website), with rawText = the full knowledge text, source_url = https://{shopDomain} if desired.
- KnowledgeChunk + Embedding created by chunking rawText (reuse existing chunking, e.g. splitToTokenChunks). Hash check to skip re-embedding if unchanged.
Ingestion: New helper e.g. ingestWebsiteKnowledge(shopId, fullText) that upserts KnowledgeSource (type website), upserts KnowledgeDocument, deletes old chunks/embeddings, creates chunks, calls getEmbeddingsBatched, inserts embeddings. Same pattern as ingestCollection / ingestArticle.

6. Billing (CHAT_MARKUP_FACTOR)

When calling openai/gpt-4o-mini-search-preview for “Fetch All Website Knowledge”:
- Use the same CHAT_MARKUP_FACTOR as chat (from env).
- Log the request in openrouter_calls (shopId, endpoint chat, model, cost, chargedCost = cost * markup, purpose e.g. “Website Knowledge Fetch”).
- Deduct chargedCost (in cents, rounded) from the shop’s credit_balance_cents (paid plan). Free plan: optional no-op or block the action if no credits.
Ensures website-knowledge fetch is billed like chat and respects the same markup.

7. Implementation Checklist

DB: shops — Optionally add website_knowledge (Text?) for storing the full text; no new domain field (use existing shop_domain). Migration only if adding website_knowledge.
Knowledge screen — New route app.knowledge.jsx with loader (load shop’s shopDomain, website_knowledge or latest website doc rawText), and actions: (1) “Fetch” = build URL from shopDomain, call OpenRouter gpt-4o-mini-search-preview with that URL, apply markup, log and deduct, return response and save to field + shop; (2) “Save” / “Save and add to RAG” = save text to shop (if column exists) and call ingestWebsiteKnowledge(shopId, text).
Server: fetch website knowledge — New helper (e.g. app/lib/website-knowledge.server.js) that calls OpenRouter chat API with model openai/gpt-4o-mini-search-preview, prompt including website URL, returns response text; caller handles openrouter_calls and credit deduction with CHAT_MARKUP_FACTOR.
Ingestion — Add ingestWebsiteKnowledge(shopId, fullText) in ingestion.server.js: KnowledgeSource type website, one KnowledgeDocument, chunk rawText, create chunks and embeddings.
Nav — Add “Knowledge” link to app layout.

8. File Summary

Item	Action
`prisma/schema.prisma`	Optionally add `website_knowledge` (Text?) to Shop for full text storage. No new domain field — use existing `shop_domain`.
`prisma/migrations/`	Migration only if adding `website_knowledge`.
`app/routes/app.knowledge.jsx`	New: Knowledge screen — text field, “Fetch All Website Knowledge From Web” (uses `shopDomain` → `https://{shopDomain}`), “Save and add to RAG”; loader + actions.
`app/routes/app.jsx`	Add nav link “Knowledge”.
`app/lib/website-knowledge.server.js`	New: `fetchWebsiteKnowledgeFromWeb(websiteUrl)` — call OpenRouter with gpt-4o-mini-search-preview; return response text. Caller (action) logs to openrouter_calls and deducts credits with CHAT_MARKUP_FACTOR.
`app/lib/ingestion.server.js`	Add `ingestWebsiteKnowledge(shopId, fullText)` — upsert source (type `website`), document, chunks, embeddings.

9. Summary

Topic	Plan
Model	openai/gpt-4o-mini-search-preview (OpenRouter) to fetch/summarize website information from the store’s URL.
Website URL	Use existing shop_domain from `shops` table; build URL as `https://{shopDomain}`. No new Settings field.
Knowledge screen	Text field (editable) + “Fetch All Website Knowledge From Web” (populates field using model + shop domain URL) + Save / “Save and add to RAG”.
Billing	Same CHAT_MARKUP_FACTOR; log to openrouter_calls; deduct from credit balance.
Storage	RAG: KnowledgeSource (type `website`), KnowledgeDocument, KnowledgeChunk, Embedding. Optionally `shops.website_knowledge` for full text.

Next: optionally add website_knowledge to shops and migrate; implement Knowledge screen (using existing shop_domain for URL), website-knowledge server helper, and ingestion for type website.