For me a lot of OHMSS doesn't hold up well, but it does have some great action sequences - the ski chase especially - but yes, I would say it rates most highly because of the Tracy plot. In all the other movies Bond has adventures and saves the world a dozen times and it's usually fun to watch, but that's just him doing what he does. He's interesting and all, but he has no character arc.
Which is fine, he doesn't necessarily have to have one, but as we've said numerous times, a truly satisfying movie story is about the most important thing that ever happened to somebody. And of all the Bond films (up until the Daniel Craig Casino Royale reboot anyway, because a similar thing happens there) what happens to Bond in OHMSS is the most important thing that ever will happen to him. And they acknowledged that by referring to it in a few of the subsequent movies, too.
Also: Diana Rigg in her Avengers prime ( the REAL Avengers, not the comic book), I mean come on, goddam.
So that certainly sets OHMSS apart from most of the pack, but doesn't make it my personal fave. For that, I'll go with Living Daylights or Spy Who Loved Me.