It could be that your service isn't speedy enough, though most of those services are smart enough to seamlessly drop-down to lower resolution to account for the slower speed. I suppose what you're seeing could have something to do with your speed varying wildly, but I would also suspect that you're actually experiencing interruptions due to network congestion.
Comcast's Internet service is rated as peak speed. ("Up to 50 Mbps download speeds") The terms and conditions state explicitly, "Actual speeds vary and are not guaranteed." Bandwidth is a shared resource on the node (a node could have up to 3000 customers, I believe), so if a bunch of you are all streaming video at the same time, you're all going to experience slower service. Furthermore, Comcast's service includes a burst component, so while you're watching your video, if your neighbor comes along and starts downloading a big file, they're going to get the lion's share of the bandwidth for a little while. That's all part of the promises Comcast makes.
It shouldn't be surprising that they're designing their Internet service so that it doesn't reward customers for trying to replace their television service. But that's different from messing with your service because you switched down to basic television service. They just simply don't do that. There's really no reason to, given that simply through offering the burst service and through the excessive use provisions and such, they're already doing all they need to in that regard. (Also remember, right after you canceled the higher tiers of television service is when you started relying more heavily on your streaming service.)
However, if you're really concerned, boot up your computer on a Saturday mid-morning, on a warm day with bright sunshine; i.e., a time when you're very unlikely to be colliding with your neighbors. Visit http://speedtest.net
and do the test. You should see yourself getting decent download and upload number (though not necessarily the burst speed - maybe you already got your burst?)
If you're not getting decent speed, even with no competition on the pipe, then you're still not done testing. Think about what's between your computer and the spot where Comcast service enters your home. Comcast does not support your LAN, unless you pay them extra for it. And even then there are things that are your responsibility not theirs. So to really test your connection, you need to connect the computer you're testing with directly to the cable modem with a wire - no router, no wireless. If after all that, the Speed Test is still giving you bad numbers, you've got good data to start taking the issue to Comcast to get a resolution.