Have a read of this review click here, which has both the HD5770 & GTS450 cards in its comparison charts, with several games tested. The performance summary at click here, has the HD5770 slightly ahead overall at all resolutions. G
So, it may come down to what motherboard you have, so tell us its make & model. E..g. if you have a Crossfire-capable board, then the HD5770 would make better sense. And if it's SLI-capable, the GTS 450 will be a better option.
Also tell us the make & model of your power supply (PSU), or its power wattage. I'm just being cautious to confirm that your PSU is up to the job of running either of those GPUs. G