Context Graphs and World Models. How to Build One That Actually Learns.

I built a context graph that actually learns. Running in production right now. Let me show you how it works.

Most AI systems have the same problem. User rejects a recommendation, AI learns nothing. Next conversation starts from zero. That's not learning, that's just forgetting with a nice UI.

@akoratana put it perfectly:

"The honest answer is that this is structurally hard. Not 'scale up compute' hard—rethink your assumptions hard. Context graphs don't exist today because building them forces us to confront problems we've spent decades ignoring. Every organization pays a fragmentation tax: the cost of manually stitching together context that was never captured in the first place. Different functions use different tools, each with its own partial view of the same underlying reality. A context graph is infrastructure to stop paying that tax. But to build one, you first have to understand why the tax exists."

That fragmentation tax is exactly what we set out to eliminate.

The Foundation: Unified Data Layer

Before you can build a context graph that learns, you need data that's actually traversable. Most businesses run on anywhere from 15-40 different software apps. CRM, email, calendar, invoicing, job management, inventory. Each tool has its own schema, its own API, its own partial view of reality. Your dispatcher sees jobs. Your accountant sees invoices. Your sales team sees leads. Nobody sees the whole picture.

Liaison connects all of these into a single normalized data layer. Every customer, every job, every invoice, every email thread gets mapped into a unified entity graph. Same structure regardless of source system. When a sale closes in your CRM, that event flows through to the same data model as when a payment clears in your accounting software or a support ticket gets resolved. Same structure whether your tools are HubSpot, Salesforce, QuickBooks, Stripe, or any combination.

This isn't just an integration play. It's the prerequisite for everything else. You can't build intelligence on top of fragmented data. The AI would inherit the same blind spots your team already has. Structured, normalized, connected data is the foundation the context graph crawls.

Once you have that foundation, you can start building memory. Here's what we did.

The Rejection Loop

When someone rejects a recommendation we don't just log it. We generate a 1536-dimension vector embedding of the entire interaction context. The node they were looking at, what was recommended, the rejection category (wrong timing, already handled, not relevant, etc), and their free text reason if they gave one. All of that gets embedded together and stored with their role and tenant context.

The embedding captures semantic meaning not just keywords. So "this job was already assigned" and "we handled this yesterday" end up close together in vector space even though the words are different.

Querying Past Rejections

Next time our Liaison model is about to generate insights for a similar context? Before it even starts reasoning we query the rejection_events table using pgvector similarity search.

Here's what that looks like:

CREATE OR REPLACE FUNCTION match_rejection_events(
  query_embedding vector(1536),
  match_threshold float,
  match_count int,
  p_tenant_slug text
)
RETURNS TABLE (
  id uuid,
  rejection_category text,
  user_reason text,
  actor_role text,
  recommendation_data jsonb,
  similarity float
)
LANGUAGE plpgsql
AS $$
BEGIN
  RETURN QUERY
  SELECT
    r.id,
    r.rejection_category,
    r.user_reason,
    r.actor_role,
    r.recommendation_data,
    1 - (r.embedding <=> query_embedding) as similarity
  FROM rejection_events r
  WHERE r.tenant_slug = p_tenant_slug
    AND r.embedding IS NOT NULL
    AND 1 - (r.embedding <=> query_embedding) > match_threshold
  ORDER BY r.embedding <=> query_embedding
  LIMIT match_count;
END;
$$;

Cosine distance with a 0.70 threshold, limited to 5 results, filtered by tenant through row level security. The <=> operator is pgvector's cosine distance. We subtract from 1 to get similarity instead of distance.

On the application side:

const contextText = `${nodeType} ${nodeId}: ${nodeData?.label || ''}`;
const contextEmbedding = await generateEmbedding(contextText);

if (contextEmbedding) {
  const { data: rejections } = await supabase.rpc('match_rejection_events', {
    query_embedding: contextEmbedding,
    match_threshold: 0.70,
    match_count: 5,
    p_tenant_slug: tenantSlug
  });

  rejectionHistory = rejections || [];
}

Injecting Rejection Context

If we find matches, that rejection history gets injected directly into the system prompt before the model generates anything:

let rejectionContext = '';

if (rejectionHistory.length > 0) {
  rejectionContext = `
## REJECTION LEARNING
The following recommendations were recently rejected in similar contexts:
${rejectionHistory.map((r, i) =>
  `${i + 1}. Category: ${r.rejection_category} | Reason: "${r.user_reason || 'No reason given'}" | Role: ${r.actor_role}`
).join('\n')}

Avoid generating insights that match these rejection patterns.
`;
}

const enhancedSystemPrompt = baseSystemPrompt + rejectionContext;

The model sees its past mistakes before generating new suggestions. Simple concept but most systems don't do this.

Scoring for Rejection Risk

After the model generates insights, we score them against the rejection history:

if (insights.items && rejectionHistory.length > 0) {
  insights.items = insights.items.map(item => {
    const itemText = `${item.title} ${item.action} ${item.description}`.toLowerCase();

    const relatedRejections = rejectionHistory.filter(r => {
      const rejectionText = `${r.recommendation_data?.title || ''} ${r.recommendation_data?.action || ''}`.toLowerCase();

      const itemWords = itemText.split(/\s+/).filter(w => w.length > 3);
      const rejectionWords = rejectionText.split(/\s+/).filter(w => w.length > 3);
      const overlap = itemWords.filter(w => rejectionWords.includes(w)).length;

      return overlap >= 2 || r.recommendation_data?.actionType === item.actionType;
    });

    return {
      ...item,
      rejectionRisk: relatedRejections.length >= 2 ? 'high' :
                     relatedRejections.length === 1 ? 'medium' : 'low',
      precedentWarnings: relatedRejections.length > 0 ?
        relatedRejections.map(r => r.rejection_category) : null
    };
  });
}

Now the UI can surface rejection risk. High means we've been rejected for similar stuff before. Medium means proceed with caution. This gives users visibility into the learning.

The Hybrid Model

Some patterns aren't tenant specific. If three different businesses in the same vertical all reject priority recommendations on Monday mornings, that's industry knowledge not company knowledge. We run a pattern extraction job that looks across tenants for common rejection patterns. Anonymized completely. No PII, no tenant identifiers, just the pattern shape.

The rule is 2+ tenants must exhibit the same pattern before it becomes shared knowledge. One company rejecting something is their preference. Multiple companies rejecting the same thing is industry wisdom.

So now our model learns from your business through direct rejection history AND from patterns across similar businesses through anonymized industry patterns. Tenant data stays isolated but collective wisdom gets shared.

Agent Trajectories

We also track agent trajectories. Every request gets a session id and we log each step. Entity_access (what data got loaded), filter (what role-based filtering applied), aggregate (what got summarized), recommend (what insights got generated). When something goes wrong we can replay the entire decision path. Not just what the model said but what it looked at before saying it.

The Schema

The tables: agent_trajectories for the walk through entity space, rejection_events with vector embeddings for semantic search, decision_traces for accepted recommendations, industry_patterns for cross-tenant wisdom. All with RLS policies so tenants only see their own data except for the anonymized patterns.

Indexes matter here. HNSW on the embedding columns for approximate nearest neighbor search. Without that the similarity queries would be too slow for real-time use.

CREATE INDEX idx_rejection_embed ON rejection_events
  USING hnsw (embedding vector_cosine_ops);

What This Looks Like in Practice

This is what a world model looks like when it's actually running. Not a demo. Not a paper. Production code making real decisions and learning from real feedback loops.

The response now includes learning metadata:

{
  success: true,
  response: insights,
  learning: {
    rejectionHistoryCount: rejectionHistory.length,
    industryPatternsApplied: industryPatterns.length,
    learningActive: rejectionHistory.length > 0 || industryPatterns.length > 0,
    rejectionCategories: [...new Set(rejectionHistory.map(r => r.rejection_category))]
  }
}

You can see exactly what the model learned from before generating its response.

Building this at useliaison.com. Happy to talk architecture if you're solving similar problems. Reach out on X @iamcadec or book a demo to see the context graph in action.