How A-B Testing Your Tests Prevents False Confidence
from Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering Hosted by Fexingo
Most teams treat their test suite as a source of truth. But what if the tests themselves are misleading you? In episode 83 of Software Testing with Fexingo, Lucas and Luna explore the concept of 'testing your tests' through A-B-style experimentation. They walk through a real case from a mid-size fintech that ran a controlled experiment: one team relied on their existing 12,000-test suite, while the other added a layer of 'test oracle' checks on top. The result? The first suite missed 23% of injected defects. The second caught 92%. They discuss why test oracles degrade over time, how to set up a simple A-B framework using canary deployments and parallel test execution, and what it means for QA culture. If you've ever shipped with confidence only to see a production bug, this episode will make you rethink your test suite's authority. No fluff, no jargon – just a practical angle on test validation. #SoftwareTesting #QA #Automation #ABTesting #TestOracle #TestValidation #RegressionTesting #Fintech #CanaryDeployment #ParallelTesting #DefectDetection #FalseConfidence #TestSuite #QualityAssurance #FexingoBusiness #BusinessPodcast #Technology #EngineeringCulture Keep every episode free: buymeacoffee.com/fexingo
Embed this episode
No transcript yet.