Building Deep-Data-Spark: A journey from frustrated AI conversations to grounded, verifiable insights, through fun, experiments, hit & trials
5 min read
Picture this: You’re in a board meeting, confidently presenting insights from your AI assistant about company performance. The AI cited specific enrolment numbers, revenue figures, and departmental statistics. Everything sounded perfect… until someone cross-checked the data. most of that was real, but some wasn’t.
How would you know what to trust and what not?
This problem sparked me to go on a mission that has since consumed my days and nights for a good year. I decided to address AI hallucinations not by building a bigger model, but by building smarter context discovery tools.
The core insight: AI hallucinate less when it has access to real, accurate, and up-to-date context from your actual systems.
What I Built: Deep-Data-Spark
I architected a system that grounds every response in verifiable data. The solution isn’t revolutionary in concept, it’s revolutionary in execution.
Here’s the fundamental difference (using a fictious university) :
Traditional AI Approach (Hallucination-Prone):
User: “Which departments would be significantly impacted by the new international student visa policy changes?”
Traditional AI Response:
“Based on typical enrolment patterns, international students usually represent about 15-20% of graduate programs. The Engineering and Business schools would likely see the most impact, as they traditionally enroll 30-40% international students. Computer Science might see a 25% reduction in applications, and you should expect decreased revenue of approximately $2-3 million.”
Reality Check: ❌
Invented the 15-20% figure
Guessed which departments would be affected
Dangerous for strategic planning decisions
My Deep-Data-Spark Discovery Approach (Context-Grounded):
User: “Which departments would be significantly impacted by the new international student visa policy changes?”
Behind the scenes - Multi-Source Discovery in Action:
🔍 Discovery Phase (Parallel Execution):
1️⃣ SQL Server queries enrolment database:
→ International students by department
→ Visa status breakdown
→ Historical enrolment trends
2️⃣ Files Server retrieves policy documents:
→ New visa policy PDF (effective Fall 2026)
→ Affected visa categories (F-1, J-1 STEM extensions)
3️⃣ Entra ID Server analyses organisational structure:
→ Department faculty composition
→ Research program dependencies
→ International faculty ratios
4️⃣ MongoDB Server searches research database:
→ Active research projects with international students
→ Grant funding tied to international collaboration
5️⃣ Data Quality Server validates data completeness:
→ Visa status data: 98% complete
→ Department enrolment data: 100% complete
→ Financial data: 95% complete
⚙️ How It Works: The Discovery-First Architecture
I built six specialised “discovery servers” that act as intelligent translators between your data sources and AI. Each server runs independently and knows exactly how to query specific systems:
The magic happens in parallel: When you ask a complex question, multiple discovery servers work simultaneously, each querying their specialised data sources. The AI orchestrator then synthesizes verified information from all sources into a comprehensive, accurate response.
What’s Next: The Discovery Ecosystem Vision
Business Applications
Salesforce Server: “Show me our Q4 pipeline” → Real CRM data
Xero Server: “What were our expenses last month?” → Actual financial records
Dynamics 365 Server: Customer interactions, support tickets, sales data
Knowledge & Collaboration Systems
Confluence Server: Company wikis, documentation, processes
SharePoint Server: Document libraries, team sites
Slack/Teams Servers: Conversation history, decisions, shared files
Analytics & Intelligence Platforms
Tableau/Power BI Servers: Dashboard data, visualizations
Google Analytics Server: Website performance, user behavior
Mixed panel Server: Product analytics, engagement metrics
The end goal: An AI assistant that has perfect memory of your organisation, never guesses, and always provides traceable, verifiable insights.
The Technical Innovation: MCP
I built this on the Model Context Protocol (MCP) standard, which provides:
Standardised Interface: Every discovery tool speaks the same language
Parallel Execution: Multiple tools work simultaneously for speed
Composability: Tools can be chained for complex discoveries
Observability: Every query is logged and traceable
The Hidden Power: Context Preparation Before AI Processing
What makes Deep-Data-Spark Discovery truly powerful isn’t just what data we retrieve—it’s how we prepare that data before it ever reaches the language model. Most AI systems make a critical mistake: they dump raw, messy data directly into the LLM and hope for the best. We take a fundamentally different approach.
Our multi-stage context preparation pipeline ensures that the AI receives clean, validated, human-verified context rather than raw database dumps. Think of it as having a team of data analysts prepare a executive briefing before the CEO makes a decision—except this happens automatically, in seconds.
Stage 1: Intelligent Discovery and Retrieval
When a user asks a complex question like “Which departments would be significantly impacted by the new visa policy?”, our orchestrator doesn’t just query one database. It intelligently plans a discovery strategy:
Parallel data retrieval from multiple sources (student databases, policy documents, financial systems, research repositories)
Relationship mapping to understand how data points connect (students → departments → research projects → funding)
Temporal alignment ensuring all data represents the same time period
Completeness checks identifying what data exists and what’s missing
This discovery phase typically completes in 1-3 seconds, pulling data from 3-6 different sources simultaneously.
Stage 2: Data Quality Assessment and Cleansing
Raw data from enterprise systems is rarely clean. Before we trust it enough to base AI responses on it, our Data Quality Server (Port 8500) performs rigorous validation:
Automated Quality Checks:
Completeness scoring: “Visa status data is 98% complete (42 of 2,154 student records missing visa type)”
Consistency validation: Cross-references enrolment numbers across different systems to catch discrepancies
Freshness verification: Flags data older than defined thresholds (“Warning: Financial data last updated 3 weeks ago”)
Anomaly detection: Identifies statistical outliers that might indicate errors (“Computer Science enrolment showing 847 students—this is 34% higher than last semester. Verify?”)
Cleansing Operations:
Standardising formats (date formats, currency, naming conventions)
Deduplication across sources
Null value handling with explicit tracking
Data type validation and coercion
Quality Scoring Example:
Dataset: student_visa_analysis
├─ Completeness: 97.2% (2,112 of 2,154 records complete)
├─ Accuracy: 99.1% (cross-validated with Entra ID)
├─ Freshness: 100% (all data < 24 hours old)
├─ Consistency: 94.3% (minor discrepancies in 2 departments)
└─ Overall Confidence Score: 97.2%
This quality metadata gets passed along with the data, so the AI knows exactly how much to trust each piece of information.
Stage 3: Human-in-the-Loop Verification
For high-stakes queries, we don’t let data go directly from databases to AI responses. Instead, we implement human verification checkpoints:
Staged Verification Process:
Initial Discovery Results: System presents retrieved data to designated reviewers
Discrepancy Flagging: When data conflicts, humans make the call
Context Enrichment: Domain experts add critical context the data doesn’t capture
Sensitivity Review: Humans verify that responses won’t expose confidential data
Real-World Example:
When analysing the visa policy impact, our system:
Retrieved data from 6 sources
Flagged 3 data quality issues (2 resolved automatically, 1 required human review)
Asked department chair to verify unusual CS enrolment spike
Incorporated chair’s context: “We admitted an additional cohort for new AI program”
Got approval from compliance officer before including financial figures
This human verification took 8 minutes but ensured 100% accuracy and institutional compliance.
Stage 4: Context Assembly and Structuring
Now comes the sophisticated part: assembling verified data into optimally structured context for the LLM. We don’t just concatenate everything together—we build a hierarchical, semantically rich context package.
Building the future where AI doesn’t hallucinate—it discovers. 🔍✨
About this project: Deep-data-Spark represents 6 months of intensive research and development, testing, and refinement. The system is built on modern microservices architecture with a focus on security, scalability, and accuracy. All discovery tools respect existing permissions and access controls while providing AI with rich, contextual data for intelligent responses.