Hey! I am Rana Usman Shahid, Senior SDET & AI Quality Engineer
Your AI fails silently — wrong answers, hallucinated facts, broken context. I catch what your error logs miss. For 6+ years I've built quality systems for production AI — LLM evaluation, RAG testing, agent quality, and performance under load — alongside high-stakes testing in FinTech and cybersecurity, where silent failures cost millions.
Quality systems for production AI — where silent failures cost millions.
Senior SDET with 6+ years specializing in production AI & LLM evaluation, RAG testing, agent quality, and prompt regression, alongside high-stakes testing in FinTech and cybersecurity. Deep expertise testing institutional investment management systems ($100M+ AUM), AI governance platforms (guardrails, PII filtering, observability, RBAC/ABAC), conversational AI agents, and semantic search validated to 94% relevance across 10,000+ queries. Proficient in Playwright, Appium, K6, Postman, REST Assured, JavaScript, Python, and AWS CloudWatch.
I've evaluated 1,200+ prompts and surfaced failure patterns to ML teams, lifted chatbot accuracy from 78% to 92%, and reduced critical production defects by 85%. My work isn't about writing more tests, it's about asking the questions that don't get asked until something goes wrong, and answering them before launch.
My Core Competencies
AI Quality & Evaluation
Automation & Frameworks
Testing Types
Languages & Tools
Methodologies
Documentation
Platforms
OS / Browsers
Where I've Worked
Senior Software Development Engineer in Test (SDET)
Kualimate
- Founded Kualimate, an AI quality engineering practice — the diagnostic layer between AI models and the users they ship to. Services span LLM evaluation, RAG testing, agent quality, prompt regression, and pre-launch AI audits for B2B and B2C teams.
- Leading the end-to-end load and performance program for a production LLM-powered virtual agent platform — concurrent-conversation simulation, latency degradation curves, and capacity planning.
- Building a K6-based load testing framework in JavaScript with AI-specific metrics, including Time To First Token (TTFT), error rate under sustained load, and tail latency at scale.
- Designing a Python evaluation pipeline that scores agent responses across five quality dimensions — accuracy, relevance, safety, instruction adherence, and conversational coherence — surfacing regressions before each release.
- Engagements span B2B (enterprise AI, FinTech, SaaS) and B2C (consumer AI, conversational agents) across North America, EU, UK, and Australia.
Senior Software Quality Assurance Engineer
CodingCops
- Lead QA strategy across four AI and consumer products in parallel, mentoring a team of 12 QA engineers and embedding shift-left, risk-based testing into Agile delivery.
- Designed validation frameworks for semantic search and chatbot systems; tested 1,200+ prompts and surfaced 80+ hallucination patterns to ML teams.
- Improved chatbot accuracy from 78% to 92% via structured prompt regression, and lifted semantic search relevance to 94% across 10,000+ queries.
- Validated chatbot access controls under RBAC and ABAC; exploratory and risk-based testing uncovered 120+ critical defects, including 15 high-severity security vulnerabilities.
- Architected Playwright and Appium automation, cutting regression cycles from 12 hours to 3 (75% reduction) and enabling twice-weekly releases at a 98% pre-release defect resolution rate.
Software Quality Assurance Engineer
Techverx
- QA owner for four enterprise platforms in regulated, high-stakes environments — most notably a $100M+ AUM investment management system (OMS/EMS/PMS) with a zero-critical-defect streak for 18 consecutive months.
- Prevented 5 critical financial calculation errors in portfolio valuation, trade execution, and interest accrual workflows using state-transition and boundary-value analysis.
- Executed 1,200+ test cases per quarter at 96% pass rate with risk-based prioritization across concurrent streams.
- Validated 30+ payment and financial API endpoints in Postman — data integrity, authentication, error handling, and performance.
Software Quality Assurance Engineer
LeapSofts
- QA delivery for 8 high-traffic eCommerce platforms generating $2M+ in monthly transactional revenue, owning checkout flows, inventory systems, and payment-gateway integrations.
- Caught 25+ critical payment integration defects pre-launch, safeguarding an estimated $200K+ in transaction revenue.
- Held consistent performance for 95% of active users through cross-browser and cross-device testing across 15+ environment combinations; usability findings tied to a 20% lift in customer satisfaction.
My Recent Projects
Hermetic AI
SMS Bot for Restaurant Lead Management
UBU
AI Based Influencer Marketing Platform
Roadway Construction Service
Construction Company Management System
Fittish.AI
AI Based Health Tracker App
mePrism Privacy
Personal Data Removal from Data Brokers
Perpetual Intelligence
Internal Organizational AI Chatbot
LightPoint Financial Technologies
Investment Management System
H2H Technologies
Debt Management SystemTrusted By Multiple Clients
"You are amazing @usmann !! Thank you for all of your hard work!! This will help me a lot!"
Kari Peters
Founder & CEO, UBU"Usman is a diligent tester who would be an asset on any engineering team. I've had the pleasure of working with him and watching his skillset develop over the course of a year, while he consistently went the extra mile for the projects we worked on. He is very thorough in testing workflows and picks up new concepts and use cases quickly. He also is a very adept at discovering potential bugs and troubleshooting issues. I'm excited to see where the future takes him."
Julian Taub
QA Manager, Lightpoint Financial Technologies"Usman has been an exceptional addition to the team. His problem solving abilities are great and he goes beyond his capacity to complete the tasks."
Saad Ajmal
Co-Founder & CGO, LeapSoftsLet's Work Together On a Project
Tell me about your AI system or product. I'll tell you where it's most likely to fail — and how to prove it won't.