When AI Builds It, Tests It, Deploys It, and Uses It, like a human — I Just Watch - Part 1

I am currently helping build DeepSkill.AI, a generative AI-powered training platform. But this time, I didn’t just code a few features, write some tests, and move on. Instead, I asked AI to build the system, then asked another AI to test it, and now I’ve got AI pretending to be a real user, interacting with the system in real time.

And me? I am just sitting here watching the whole thing unfold, reviewing logs, wondering if this counts as delegation or early retirement.

The Setup

The codebase was bootstrapped by AI dev tools (Claude and Cline). The system’s logic runs on Deepseek — a solid runtime LLM handling the real work. The “user” is an Anthropic LLM, with its own login, real account, and live interaction path. I’m the only human in the mix, playing referee.

Yes, the software is now building and testing itself. I’m just here to make sure no one breaks anything too creatively.

The Pain We All Know

You’ve been there.

Writing dozens of test cases for features that keep changing because the prompt changed. Green builds that pass all UI tests while the AI replies, “I’m not sure how to help you with that.” Regression testing that somehow causes new hallucinations. Trying to explain to your team why “hello” in the chat interface returned a stack trace.

That’s the reality of GenAI-based systems. Static mocks are useful, but limited — especially when the system evolves daily. So instead of pretending I know what users will say, I had the AI pretend to be the user too.

I literally asked the AI dev bot to write a test plan on how it would behave like a curious human. It logged in, chatted with the system, tested the flows, and gave feedback — live.

Why This Is Different

This isn’t your standard mocked Playwright script. This is real, unscripted LLM-to-LLM interaction — AI using AI.

The “user” tries different prompts, challenges the system’s logic, and checks whether the responses make sense. It acts more like a real user than any of my past automated tests ever could.

And surprisingly, it flagged usability gaps, broken flows, and logical inconsistencies — all without me telling it where to look.

Observations So Far

Found edge cases I wouldn’t have thought of Validated not just functionality, but overall experience Reusable for post-release confidence checks Not deterministic — which is fine, because neither are users

This is the new QA playground. Less about static coverage, more about continuous, intelligent probing.

Takeaways

This is not a replacement for all testing. I still write Playwright tests for critical flows and edge coverage. But for AI-native applications, the traditional QA mindset needs a reset.

You’re not just testing UI or endpoints anymore — you’re testing reasoning, tone, helpfulness, and adaptability.

Letting AI simulate user behavior in a live system might feel strange, but it’s surprisingly effective — especially when things change often (as they always do with generative systems).

Would I do it again? Absolutely.

It’s fast, repeatable, and gives me more confidence in how the system actually behaves when a real human lands on the page.

And if nothing else, it’s kind of fun watching two AIs talk to each other while you sip coffee and pretend to be productive.

What do you think? Have you tried using AI to test your AI system yet? Or are you still whispering “please work” to the console?

#GenAI #AIUserTesting #DeepSkillAI #SoftwareDevelopment #AIinProduction #AIvsAI #QA #PromptEngineering #FutureOfTestingI will be writing more parts, now have a lot to share with fellow AI enthusiasts, with a particular focus on business owners, innovators, and real world problem solvers - to help them get most out of Gen AI and bring value out to the world faster. Stay tuned!

The sample log:

AI User:

“message”: “Hi! I want to work on building my professional experience profile. I have been working as a software engineer for about 6 years now and want to document my key achievements.”

Deepskill.ai:

To start, I’d love to hear more about your experiences. For example:

1. What technical skills have you developed during this time? Beyond React/Node.js or AWS, are there any other areas like data science, machine learning, infrastructure deployment, or architecture design that you’ve worked on?

2. What are some of your most notable achievements? For instance, you mentioned the microservices migration—could you ela borate on that project or any others where you led initiatives, improved performance by a certain percentage, worked with lar ge teams, or adopted new practices like DevOps?

3. What kind of work environment do you thrive in? Do you enjoy working independently on technical challenges, collaborat ing with cross-functional teams, or leading and mentoring others?

Your insights will help us craft a profile that captures the impact of your work. Let’s dive into your experiences—I’m here t o guide you! “message”: “Thanks for those great questions! Let me start with my most impactful project. Last year I was tasked with fixing our main e-commerce platform that was constantly crashing during peak traffic. Our Black Friday sales were a disaster - the site went down for 4 hours and we lost about $2.3 million in revenue. My manager was under huge pressure and basically said “fix this or we are all in trouble”. I spent 3 weeks analyzing the bottlenecks, found the database queries were poorly optimized and the frontend was making way too many API calls. I redesigned the caching strategy, optimized 15 critical SQL queries, and implemented lazy loading. The result? This year Black Friday handled 300% more traffic with zero downtime. Sales hit $8.7 million - our best day ever!”

• Systems Optimization: Demonstrated expertise in identifying and resolving system bottlenecks, improving database perf ormance (query optimization), implementing efficient caching strategies, applying frontend performance techniques like lazy l oading.

So far in our conversation, you’ve shared a compelling project that showcases strong technical skills and measurable business impact. It’s great to hear about overcoming such a significant challenge, especially one with the potential revenue loss you mentioned earlier.

Now I’d love to learn more:

1. Could you share a bit about what specifically involved “reaching out” in this context? How did the technical findings tra nslate into actionable steps for your manager or other teams?
2. When you performed that deep technical analysis, was there a particular methodology or diagnostic tool (like profiling) t hat stood out in helping you pinpoint the issues?
3. Thinking back, if you were to reflect on this experience for your future job interviews or personal development, what wou ld be the top one or two takeaways or learnings that you feel best represent its value? “message”: “Great questions! For the technical analysis, I used New Relic for application monitoring and profiling, which showed me exactly where the bottlenecks were happening. I also used pgAdmin to analyze slow query logs from PostgreSQL. The “reaching out” part was actually presenting my findings to the CTO and product team in a crisis meeting. I had to explain technical concepts in business terms - like how the poorly indexed user_orders table was causing 8-second page loads that made customers abandon their carts. My biggest learning was that being a great developer is not just about coding - it is about communicating impact to non-technical stakeholders. I learned to always frame technical problems in terms of user experience and business metrics. Also, I realized the importance of proactive monitoring rather than reactive fixes. Now I always set up alerts and dashboards before problems happen.” How does this align with your broader career goals or the areas you want to highlight for upcoming roles?

Looking forward, I’d love to hear more about:

1. The impact: What was the direct, measurable outcome after implementing your recommendations or fixing the bottleneck? (e.g., percentage improvement in page load, reduced support tickets, positive user feedback).
2. The technical resolution: What was your specific recommendation or solution to address the poorly indexed user_order s table?
3. Your learnings: Beyond communication, what other key technical or soft skills did this experience solidify or teach yo u?

Your insights into bridging the gap between technical work and business value are incredibly valuable in a senior software en gineer role. “message”: “Actually, let me shift gears and tell you about a completely different experience that shaped me as a leader. Six months ago, our team grew from 3 to 8 developers when we acquired a startup. I was asked to onboard and mentor 5 new team members who had very different coding practices - some used PHP, others Python, and their code review process was basically non-existent. The integration was chaotic at first. I volunteered to create a mentorship program and spent my evenings creating coding standards documents, setting up pair programming sessions, and establishing a proper CI/CD pipeline. Two of the new developers were actually more senior than me in years of experience, which was intimidating. But I focused on knowledge sharing rather than hierarchy. After 3 months, our code quality improved dramatically - we went from 12 production bugs per week to just 2, and the team started collaborating much better. The CEO even mentioned in our quarterly meeting that our engineering velocity had increased by 60%. What I learned is that leadership isn’t about being the smartest person in the room, it’s about creating systems that help everyone succeed.”