Day 61 of sharing stats about my SaaS until I get 1000 users: My AI thinks it's perfect but my users are telling me it's actually 44 percent wrong

Post original

I've been staring at the similarity scores my ML model spits out and for a while I actually believed them. The math says we're hitting a 0.96 similarity on average which sounds like the leads are a perfect fit. But I finally started looking at the manual feedback loop I built and the reality is a lot more humbling. Only about 2 percent of users are even bothered to click the thumbs up or down on a lead. Out of those 77 pieces of feedback, the positive rate is sitting at 55.84 percent. That means nearly half the time the human on the other end is looking at what the AI called a perfect match and saying it's garbage. It's a weird spot to be in as a dev. The score distribution shows that 2,487 matches are sitting in the lower 0.7 bucket, which the model thinks is 'okay' but not great. I have almost nothing in the 0.9 bucket according to the internal logs, yet the average is still high. It feels like the model is grading its own homework and giving itself an A while the users are barely giving it a passing grade. Key stats: - 3254 total lead matches generated - 0.9614 average similarity score calculated by ML - 55.84 percent positive feedback rate from manual reviews - 2.37 percent feedback coverage across all matches - 2487 matches sitting in the 0.7 similarity bucket Current progress: 429 / 1000 users. Previous post: Day 60 — Day 60 of sharing stats about my SaaS until I get 1000 users: My AI found 3,600 leads but only 41 actually got a response   submitted by   /u/Less-Bite [link]   [comments]