Learned the hard way that giving real feedback in front of a crowd can backfire if the boss wasn't actually ready for it. What worked for me was keeping it short and sticking to facts, like "here's what slowed the team down last month" instead of going after her personally. I also saved the bigger stuff for a private chat after, which kept things from getting awkward in the room. Being honest in the moment is good, but reading the room matters too. If she asked in front of everyone, she probably wanted to look open to input, so I gave her something she could actually use without making her look bad.

What did your eval set look like before you rebuilt it? I had the same thing happen with a docs Q&A bot and it was chunking all along. My chunks were splitting tables in half so the model got "30 days from" and nothing else, then it guessed the rest. I spent two days rewriting instructions about not making things up before I actually read the retrieved text. Once I switched to splitting on headers and keeping rows together, made up answers mostly went away. My test set was like five questions I wrote from memory too, which of course all passed.