You access the router's Setup pages by entering its IP address (usually 192.168.2.1 for Belkin routers) into a web browser. You *DON'T* need to be connected to the internet to be able to do this.
It may be necessary initially to enter your network adapter's IP address manually if you cannot access the router. In the Network Connections folder, right click the Local Area Connection and select Properties. Highlight "Internet Protocol (TCP/IP)" and click the Properties button. Click "Use the following IP address and enter:
IP address - 192.168.2.2
Subnet mask - 255.255.255.0
Default Gateway - 192.168.2.1 (the router's IP address)
Preferred DNS Server - 192.168.2.1 (the router's IP address)
Click OK back through the boxes and then see if your web browser can access 192.168.2.1