I think that would make sense, but I still don't know how the sketches, reports of strange jet stream patterns, and other such 'evidence' remain fairly consistent. If it were just a fictional craft, then I would have imagined a very large number of different designs would be out there, or that the lack of real evidence would have killed the belief long ago.
I guess that I would assume it was always just figment of science fiction. Such a radical aircraft design could not easily be covered up without leaving a paper trail behind. Even the F-117 wasn't that radical a design compared to a transonic spy plane.