That happens if you fully utilise your upload speed. To combat this, use a program to limit your upload, and put the max upload just below what you can reach.
Say: you can reach up to 497kb, then set it to 495kb.
For downloading, you also need an upload connection, as every several bytes require you to send a verification byte. If your upload is full, the verification byte is being queued, making your internet download being queued too.