Chat Guardrails Inspection

This document lists every place in the codebase that can prevent or restrict the chat from answering certain questions (e.g. “list down top python frameworks”).

---

1. Empty-context fallback (primary guardrail for off-topic questions)

File: app/lib/rag.server.js
Lines: 111–121

What it does: If the vector search returns zero chunks for the shop, the app never calls the LLM. It immediately returns a fixed message and does not call OpenRouter chat.

Relevant code:

// 8. Empty-context fallback — avoid an LLM call when no knowledge exists.
if (!chunks || chunks.length === 0) {
  // ...
  return {
    reply: "I don't have enough information to answer that. Please contact the store's support team.",
    sessionId: session.id,
  };
}

When this triggers:

• The shop has no ingested knowledge (no products, collections, articles, or website knowledge), so there are no rows in embeddings for that shop_id, and searchSimilarChunks returns an empty array.

When it does not trigger:

• As soon as the shop has any ingested chunks, vector search returns up to 5 chunks (the 5 nearest by embedding). So for “list down top python frameworks”, you typically get 5 unrelated store chunks, and the flow continues to the LLM. The block only happens when chunks.length === 0.

Conclusion: This is the guardrail that blocks any answer when the shop has no knowledge. It is the only place that short-circuits the flow without calling the LLM.

---

2. System prompt (LLM-level guardrail)

File: app/lib/prompt-builder.server.js
Lines: 3–5 (constant SYSTEM_PROMPT)

What it does: The system prompt is sent to the LLM and tells it how to behave. It can instruct the model to only use the provided store context and to say “I’m not sure” when the answer is not in that context.

Current text in codebase (implemented):

const SYSTEM_PROMPT = `You are the AI customer support agent for this Shopify store.
...
Store-only scope (critical — highest priority):
- Answer only store/order-related support questions.
- If the question is general knowledge or unrelated to this store, do not answer from model memory.
- Reply briefly that you can help with this shop's products/orders and invite a store-related question.
- Do not include random external links for off-topic subjects.
...
`;

With this text, the model is now explicitly restricted to store support scope and should refuse unrelated questions (e.g. “what is yahoo.com”) with a short in-scope redirect.

Conclusion: The primary LLM-level guardrail is the system prompt in prompt-builder.server.js, and it is now configured for store-only support behavior.

---

3. API / route layer

File: app/routes/api.chat.jsx

What was checked:

• No content filtering, no topic blocklist, no check for “python” or “frameworks.”
• Validation is only: shopDomain and message required, message.length <= 2000.
• No guardrail logic here that would prevent answering “list down top python frameworks.”

---

4. Vector search (no similarity threshold)

File: app/lib/vector-search.server.js

What it does: Returns the top topK (default 5) chunks by cosine similarity. There is no minimum similarity threshold; the code does not filter out low-similarity results.

So as long as the shop has at least one chunk, you get up to 5 chunks (even if they are unrelated to the query). The guardrail is not in the vector search layer.

---

5. Widget / client

File: extensions/appifire-chat/assets/chat-widget.js

What was checked:

• Sends message in the POST body to /api/chat. No client-side filtering or blocking of questions.

---

Summary: what blocks “list down top python frameworks”

Cause	Location	Result
No knowledge for shop (0 chunks)	app/lib/rag.server.js lines 111–121	Returns canned fallback with no LLM call.
Store-only system prompt	app/lib/prompt-builder.server.js SYSTEM_PROMPT	Refuses unrelated/general-knowledge questions and redirects to store-relevant support scope.

There are still no API-level keyword blocklists or vector similarity thresholds; behavior is primarily controlled by empty-context fallback + system prompt policy.