I think that you probably have a tick on "leave a copy of messages on server".
Open Outlook, click on Tools, Accounts. Select Mail, then left click on the name of the account, (if you have more than one). Click on properties on the right hand side. Click on advanced tab, and at the bottom you will see a box "leave a copy of message on the server". If it is ticked, then un-tick it. Click Apply, and OK.
That should correct your problem.