Speed is not the only factor. Stability is more important than raw speed.
I have customers with 20Mbps internet and thinks are solid. Others with Gigabit, and constantly complaining.
Can be anything from crappy WiFi (router, access points, or even just location of the router which can make a difference). The service may be throttling and causing slowdowns. There can also be all sorts of jitter or packet loss caused by a variety of factors.
A high latency is not a huge issue for stability. It just means that channel changes may be a little slow and vod may take a little longer to start. But once they start streaming, they are fine.
However if there is all sorts of jitter, it can kill your iptv experience.