It could be affecting Youtube - well, the Adobe Flash player plugin that drives it!
It all depends on whether it's the downloading that's affected (broadband speed), or the viewing (processing speed). Which of these applies best?:
1) The videos take ages to download and the player position tab on the progress bar is always catching up with the downloaded position. Apart from this, the video and audio runs smoothly.
or
2) The downloads work quickly and the progress bar fills up nicely, but the video and audio is jerking or stopping / starting and the videos don't run smoothly.
Cheers,
Shaun
PS. Could you post the spec of your machine and what OS you're using?