You don't say which type of Belkin modem router you have, but most are functionally similar.
In your router setup pages there is one page relating to WAN setup (ie connection to the internet). It is the page where you enter your username and password to connect to your ISP. On that page there are normally 2 entries such as Dial On Demand and Idle Time.
If you tick Dial On Demand the modem will connect every time someone tries to access the internet. Sometimes the alternative is auto start (ie the modem logs on when you switch it on) and sometimes manual start (ie YOU have to start it each time). Yours sounds like the latter, so whichever setting you have now, try changing it.
The Idle Time setting adjusts how long it will be with no internet activity before the modem automatically logs off. This was very common in the days of dial-up internet and there are still some pay-as-you-go broadband suppliers who charge by the minute as an option. Either set this number as high as you can (it is in usually in seconds, so 3600 represents one hour but yours may be different) or set it to 0, which normally means no time-out at all.
Then click Apply Changes, OK or whatever is needed to save the changes to the router and then OK your way out and log off.
If it still doesn't work to your satisfaction, adjust those settings until it does.