It usually takes several starts and/or resets to get it going. In the meantime sometimes only the disk activity light is on, sometimes only the "power on" light.
My advice reading this would be to change the PSU (so you were not ill advised)
Sometimes the on-board speaker gives one long and three short beeps
the beep code gives advice on what may be wrong (1 long 3short + memory or video display), the code depends on the BIOS manufacturer.
a memory problem certainly would starting problems try one stick at a time to try and eliminate.