Has this only happened recently, or is your connection always variable?
Your idea about holiday usage isn't silly. As you probably know, your bandwidth will probably be shared with up to 50 other users, so if everyone in your neighbourhood is at home, on the internet, this will slow you down.
Try getting up at 2.00am and seeing what it's like then! ;o)