I am not sure if this could possibly apply to you or not, but reading your post made a connection for me with something that I have had to deal with in the past.
Often when my wife and I fought, hearing her re-tell the story to the couples counselor she would often cut bits out of it, so what in reality was a slow and gradual decline, with several bumps on the way down she will just describe the beginning and the end so that it sounds like "I did X and he just suddenly lost it out of nowhere". This I find incredibly frustrating because it is like she just doesn't see all of the bits in between that lead to that point & for her to paraphrase it as if it went from 0 to 100 for no reason comes across as unfair.
I find it hard not to think she is doing it on purpose, but over time I have learned that this may either be relating to her own issues she struggles with and/or something all human beings are prone to do which is called "confirmation bias". This is where you count the hits & ignore the misses.
I wonder if this could possibly be happening on either side in this situation. Either he is acutely aware of every time something happens & he is checking that off as "one of those times" and you might not be perceiving it at the same level he is so you discount it as nothing. So then when the two of you compare as you explained in your story, he says that you say that every time (he is counting what he thinks are hits) and you say that you have been great lately (you are not counting them because they weren't big enough for you). So while it feels unfair, it may just end up being a different perspective.
One thing I always try to remember is that being on the "inside" of something means you know all the facts. You know what lead up to it, how much it peaked, whether you think it was understandable that it happened & if you think it was an event. Looking at it from the outside, all you have to go on are external queues, its black or white, so from an outside perspective it either matches how it has come across before, or it didn't
Worth a thought