Chat Guardrails Inspection

This document lists every place in the codebase that can prevent or restrict the chat from answering certain questions (e.g. “list down top python frameworks”).

---

1. Empty-context fallback (primary guardrail for off-topic questions)

File: app/lib/rag.server.js
Lines: 111–121

What it does: If the vector search returns zero chunks for the shop, the app never calls the LLM. It immediately returns a fixed message and does not call OpenRouter chat.

Relevant code:

// 8. Empty-context fallback — avoid an LLM call when no knowledge exists.
if (!chunks || chunks.length === 0) {
  // ...
  return {
    reply: "I don't have enough information to answer that. Please contact the store's support team.",
    sessionId: session.id,
  };
}

When this triggers:

• The shop has no ingested knowledge (no products, collections, articles, or website knowledge), so there are no rows in embeddings for that shop_id, and searchSimilarChunks returns an empty array.

When it does not trigger:

• As soon as the shop has any ingested chunks, vector search returns up to 5 chunks (the 5 nearest by embedding). So for “list down top python frameworks”, you typically get 5 unrelated store chunks, and the flow continues to the LLM. The block only happens when chunks.length === 0.

Conclusion: This is the guardrail that blocks any answer when the shop has no knowledge. It is the only place that short-circuits the flow without calling the LLM.

---

2. System prompt (LLM-level guardrail)

File: app/lib/prompt-builder.server.js
Lines: 3–5 (constant SYSTEM_PROMPT)

What it does: The system prompt is sent to the LLM and tells it how to behave. It can instruct the model to only use the provided store context and to say “I’m not sure” when the answer is not in that context.

Current behavior (implemented):

• In scope: store orders, catalog, policies, and industry education when relevant to the store’s niche (inferred from Store Context: product types, tags, collections, FAQ). Examples: “ski vs skateboarding” at a ski shop, benefits of wax, beginner gear guidance—answer with category knowledge, then guide toward what this store sells.
• Out of scope / refuse briefly: unrelated general trivia (e.g. “what is Yahoo” with no store tie), homework, coding, politics, medical/legal advice with no shopping link.
• See Store-only scope and Store support rules in app/lib/prompt-builder.server.js.

Conclusion: The primary LLM-level guardrail is the system prompt. It balances store grounding with helpful industry education—not a blanket refusal of all general-knowledge questions.

---

3. API / route layer

File: app/routes/api.chat.jsx

What was checked:

• No content filtering, no topic blocklist, no check for “python” or “frameworks.”
• Validation is only: shopDomain and message required, message.length <= 2000.
• No guardrail logic here that would prevent answering “list down top python frameworks.”

---

4. Vector search (no similarity threshold)

File: app/lib/vector-search.server.js

What it does: Returns the top topK (default 5) chunks by cosine similarity. There is no minimum similarity threshold; the code does not filter out low-similarity results.

So as long as the shop has at least one chunk, you get up to 5 chunks (even if they are unrelated to the query). The guardrail is not in the vector search layer.

---

5. Widget / client

File: extensions/appifire-chat/assets/chat-widget.js

What was checked:

• Sends message in the POST body to /api/chat. No client-side filtering or blocking of questions.

---

Summary: what blocks “list down top python frameworks”

Cause	Location	Result
No knowledge for shop (0 chunks)	app/lib/rag.server.js lines 111–121	Returns canned fallback with no LLM call.
Store-only system prompt	app/lib/prompt-builder.server.js SYSTEM_PROMPT	Refuses unrelated/general-knowledge questions and redirects to store-relevant support scope.

There are still no API-level keyword blocklists or vector similarity thresholds; behavior is primarily controlled by empty-context fallback + system prompt policy.