You sure those figures are the right way round - upstream is always slower than downstream on ADSL?
The net gets busy during the day. Also, you are on a 50:1 contention ratio, i.e. you could be sharing your internet connection with up to 49 others. If they are all downloading at once, the connection will get very slow.