Building the Foundation of the Trade Observations AI Knowledge Base

One of the long-term goals of Trade Observations is to evolve from an application that generates AI analysis into an application that learns from its own historical analyses.

This week marked the beginning of that transformation.

The Vision

Rather than asking an AI model to analyze a chart with no memory of the past, the objective is to create a persistent knowledge base that can answer questions such as:

• Have we seen this price action before?
• Which previous analyses most closely resemble the current chart?
• What did the AI recommend in those situations?
• How does the current market compare to historical Brooks-style patterns?

This is a classic Retrieval-Augmented Generation (RAG) architecture.

Installing pgvector

The first task was enabling vector search inside the existing PostgreSQL 15 database running on Amazon Linux 2023.

After installing and compiling pgvector 0.8.0, the extension was enabled:

CREATE EXTENSION IF NOT EXISTS vector;

With pgvector available, PostgreSQL became capable of storing OpenAI embedding vectors alongside traditional relational data.

Creating the RAG Schema

A dedicated rag schema was added to isolate the AI knowledge layer.

The first tables are:

• rag.ai_analysis_event
• rag.ai_analysis_chunk
• rag.ai_analysis_tag

These tables separate an analysis into logical reasoning components that can later be embedded and searched semantically.

The Existing AI Snapshot Data

One pleasant surprise was discovering that the existing tradeobs.ai_snapshots table already contained nearly everything required for a RAG pipeline:

• AI analysis text
• Structured JSON sections
• Chart image stored in S3
• Notebook references
• Tags
• GTO metadata
• Exit warning information

Instead of building a parser, the existing production data became the source for the knowledge base.

Backfilling the Knowledge Base

The migration inserted:

• 15,052 AI analysis events
• 59,323 reasoning chunks
• 53,473 searchable tags

Each analysis was divided into four semantic sections:

1. Context
2. Patterns
3. Probability
4. Entries / Avoidance

Those sections are now independent knowledge objects that can be embedded and searched.

Why Chunking Matters

Instead of embedding an entire analysis as one large document, each reasoning section becomes its own searchable concept.

For example:

• "Show me previous L2 sell setups."
• "Find analyses describing a bull spike into a trading range."
• "Find situations where breakout failure probability was HIGH."

This dramatically improves retrieval quality.

Charts Remain Connected

Every AI analysis retains a link to its corresponding chart artifact stored in Amazon S3.

That means future chatbot responses can retrieve not only similar reasoning, but also the original chart image that produced it.

The Next Phase

The next component is an embedding service that will:

1. Find chunks without embeddings.
2. Generate OpenAI embeddings.
3. Store vectors in PostgreSQL.
4. Enable semantic similarity search.

Once complete, the Trade Observations chatbot will be able to retrieve similar historical analyses before asking the language model to answer a question.

Looking Ahead

The architecture is intentionally reusable.

The same RAG engine will eventually power:

• Trade Observations — Price action, Brooks concepts, trade reviews
• MacroLabz — Macro regimes, economic data, investment research

Each application will contribute different knowledge while sharing the same retrieval engine.

Conclusion

This milestone represents a shift from generating AI analyses to building an AI system with memory.

Every five-minute market analysis now becomes a permanent piece of institutional knowledge that can be searched, compared, and reused.

Today the system contains over 59,000 reasoning chunks.

Tomorrow it will use them to help explain what the current market looks like—and why.