The 180mbps maximum is a theoretical maximum and is normally taken with a pinch of salt. It sounds like you have an 802.11g router with a turbo mode. What is the make and model of the router?
If it is a turbo model, you need the matching card to benefit from any speed increase. 24mbps sounds about right for a 802.11g network (theoretical maximum is 54mbps).
As far as your internet connection goes, the 4mbps is also a maximum. However, your wireless router will pass on the speed from that connection. Therefore, if you are getting 24mbps on the router, then you will be benefitting from the 4mbps internet connection.