Any sport has the potential to be difficult. It all depends on who you're competing against. Except for Ninja Warrior. Although in that case the competition would be the course itself.
Baseball is one of the hardest sports for anyone to just pick up and play. In high school all the lacrosse players would talk trash about baseball (lax bros and baseball players hating each other was a thing at my high school), until one time our coach had a few of them take some swings in batting practice. Not one of them came close to hitting the ball, and for two main reasons. 1, they didn't know how to properly swing a bat, which is much more complicated than it looks. 2, they were afraid of the ball. Who could blame them though, it was coming pretty fast... for batting practice
Besides, when the best players in the world have about a 35% success rate, that probably means it's not easy.
I love sports; playing much more than watching, but I do enjoy watching football and baseball, and sometimes college wrestling. It's more fun to watch a sport that you have a lot of experience playing, probably because you have a greater understanding and appreciation for the artistry of it, as Eddie was saying. You can marvel at the idiosyncrasies that the players have perfected over years of experience. People like watching people do impressive things, and that extends beyond sports.
I don't like watching soccer. It's a lot of back and forth, and in an hour and a half you're lucky if they score more than twice. But I've also never played it, so a lot of the little stuff blows right by me.