Building Deep-Data-Spark: A journey from frustrated AI conversations to grounded, verifiable insights, through fun, experiments, hit & trials
5 min read
Picture this: You’re in a board meeting, confidently presenting insights from your AI assistant about company performance. The AI cited specific enrolment numbers, revenue figures, and departmental statistics. Everything sounded perfect… until someone cross-checked the data. most of that was real, but some wasn’t.
How would you know what to trust and what not?
This problem sparked me to go on a mission that has since consumed my days and nights for a good year. I decided to address AI hallucinations not by building a bigger model, but by building smarter context discovery tools.
The core insight: AI hallucinate less when it has access to real, accurate, and up-to-date context from your actual systems.
What I Built: Deep-Data-Spark
I architected a system that grounds every response in verifiable data. The solution isn’t revolutionary in concept, it’s revolutionary in execution.
Here’s the fundamental difference (using a fictious university) :
Traditional AI Approach (Hallucination-Prone):
User: “Which departments would be significantly impacted by the new international student visa policy changes?”
Traditional AI Response:
“Based on typical enrolment patterns, international students usually represent about 15-20% of graduate programs. The Engineering and Business schools would likely see the most impact, as they traditionally enroll 30-40% international students. Computer Science might see a 25% reduction in applications, and you should expect decreased revenue of approximately $2-3 million.”
Reality Check: ❌
Invented the 15-20% figure
Guessed which departments would be affected
Dangerous for strategic planning decisions
My Deep-Data-Spark Discovery Approach (Context-Grounded):
User: “Which departments would be significantly impacted by the new international student visa policy changes?”
Behind the scenes - Multi-Source Discovery in Action:
🔍 Discovery Phase (Parallel Execution):
1️⃣ SQL Server queries enrolment database:
→ International students by department
→ Visa status breakdown
→ Historical enrolment trends
2️⃣ Files Server retrieves policy documents:
→ New visa policy PDF (effective Fall 2026)
→ Affected visa categories (F-1, J-1 STEM extensions)
3️⃣ Entra ID Server analyses organisational structure:
→ Department faculty composition
→ Research program dependencies
→ International faculty ratios
4️⃣ MongoDB Server searches research database:
→ Active research projects with international students
→ Grant funding tied to international collaboration
5️⃣ Data Quality Server validates data completeness:
→ Visa status data: 98% complete
→ Department enrolment data: 100% complete
→ Financial data: 95% complete
⚙️ How It Works: The Discovery-First Architecture
I built six specialised “discovery servers” that act as intelligent translators between your data sources and AI. Each server runs independently and knows exactly how to query specific systems:
The magic happens in parallel: When you ask a complex question, multiple discovery servers work simultaneously, each querying their specialised data sources. The AI orchestrator then synthesizes verified information from all sources into a comprehensive, accurate response.
What’s Next: The Discovery Ecosystem Vision
Business Applications
Salesforce Server: “Show me our Q4 pipeline” → Real CRM data
Xero Server: “What were our expenses last month?” → Actual financial records
Dynamics 365 Server: Customer interactions, support tickets, sales data
Knowledge & Collaboration Systems
Confluence Server: Company wikis, documentation, processes
SharePoint Server: Document libraries, team sites
Slack/Teams Servers: Conversation history, decisions, shared files
Analytics & Intelligence Platforms
Tableau/Power BI Servers: Dashboard data, visualizations
Google Analytics Server: Website performance, user behavior
Mixed panel Server: Product analytics, engagement metrics
The end goal: An AI assistant that has perfect memory of your organisation, never guesses, and always provides traceable, verifiable insights.
🎯 The Technical Innovation: MCP Protocol
I built this on the Model Context Protocol (MCP) standard, which provides:
Standardised Interface: Every discovery tool speaks the same language
Parallel Execution: Multiple tools work simultaneously for speed
Composability: Tools can be chained for complex discoveries
Observability: Every query is logged and traceable
The Hidden Power: Context Preparation Before AI Processing
What makes Deep-Data-Spark Discovery truly powerful isn’t just what data we retrieve—it’s how we prepare that data before it ever reaches the language model. Most AI systems make a critical mistake: they dump raw, messy data directly into the LLM and hope for the best. We take a fundamentally different approach.
Our multi-stage context preparation pipeline ensures that the AI receives clean, validated, human-verified context rather than raw database dumps. Think of it as having a team of data analysts prepare a executive briefing before the CEO makes a decision—except this happens automatically, in seconds.
Stage 1: Intelligent Discovery & Retrieval
When a user asks a complex question like “Which departments would be significantly impacted by the new visa policy?”, our orchestrator doesn’t just query one database. It intelligently plans a discovery strategy:
Parallel data retrieval from multiple sources (student databases, policy documents, financial systems, research repositories)
Relationship mapping to understand how data points connect (students → departments → research projects → funding)
Temporal alignment ensuring all data represents the same time period
Completeness checks identifying what data exists and what’s missing
This discovery phase typically completes in 1-3 seconds, pulling data from 3-6 different sources simultaneously.
Stage 2: Data Quality Assessment & Cleansing
Raw data from enterprise systems is rarely clean. Before we trust it enough to base AI responses on it, our Data Quality Server (Port 8500) performs rigorous validation:
Automated Quality Checks:
Completeness scoring: “Visa status data is 98% complete (42 of 2,154 student records missing visa type)”
Consistency validation: Cross-references enrolment numbers across different systems to catch discrepancies
Freshness verification: Flags data older than defined thresholds (“Warning: Financial data last updated 3 weeks ago”)
Anomaly detection: Identifies statistical outliers that might indicate errors (“Computer Science enrolment showing 847 students—this is 34% higher than last semester. Verify?”)
Cleansing Operations:
Standardising formats (date formats, currency, naming conventions)
Deduplication across sources
Null value handling with explicit tracking
Data type validation and coercion
Quality Scoring Example:
Dataset: student_visa_analysis
├─ Completeness: 97.2% (2,112 of 2,154 records complete)
├─ Accuracy: 99.1% (cross-validated with Entra ID)
├─ Freshness: 100% (all data < 24 hours old)
├─ Consistency: 94.3% (minor discrepancies in 2 departments)
└─ Overall Confidence Score: 97.2%
This quality metadata gets passed along with the data, so the AI knows exactly how much to trust each piece of information.
Stage 3: Human-in-the-Loop Verification
For high-stakes queries, we don’t let data go directly from databases to AI responses. Instead, we implement human verification checkpoints:
Staged Verification Process:
Initial Discovery Results: System presents retrieved data to designated reviewers
Discrepancy Flagging: When data conflicts, humans make the call
Context Enrichment: Domain experts add critical context the data doesn’t capture
Sensitivity Review: Humans verify that responses won’t expose confidential data
Real-World Example:
When analysing the visa policy impact, our system:
Retrieved data from 6 sources
Flagged 3 data quality issues (2 resolved automatically, 1 required human review)
Asked department chair to verify unusual CS enrolment spike
Incorporated chair’s context: “We admitted an additional cohort for new AI program”
Got approval from compliance officer before including financial figures
This human verification took 8 minutes but ensured 100% accuracy and institutional compliance.
Stage 4: Context Assembly & Structuring
Now comes the sophisticated part: assembling verified data into optimally structured context for the LLM. We don’t just concatenate everything together—we build a hierarchical, semantically rich context package.
Building the future where AI doesn’t hallucinate—it discovers. 🔍✨
About this project: Deep-data-Spark represents 6 months of intensive research and development, testing, and refinement. The system is built on modern microservices architecture with a focus on security, scalability, and accuracy. All discovery tools respect existing permissions and access controls while providing AI with rich, contextual data for intelligent responses.