Yes, if it gets too far off sync it can get annoying. I once listened to a commentary not knowing they were watching the PAL version, and needless to say it got less relevant to the movie as time went on  Luckily you could hear their movie audio faintly, so I could pause as needed.

In my practical experience, I watch NTSC video and slow PAL commentaries down to .967 using Windows Media Player  (View > Enhancements > Play Speed Settings), which maintains the original pitch. Regardless, the aforementioned variables—including the occasional computer stutter and need to pause for short breaks—make it necessary to resync from time to time, so sync lines and direct references are helpful.

YMMV. I'm as obsessive about sync as you're likely to find. I have no significant troubles with DIF.