Robot.txt Telling them where to go

  Proclaimer 12 Apr 10

I have a site (under construction) that is split into two. One part is a general website and the other is a Shopping Cart.

The General Site is here click here and the Cart (a Zen Cart) is here click here in a folder called 'store'.

The Cart could act as a stand alone website but I am not so happy with that so I have built the front end onto it.

The whole site is not seamless and because I am using a cart it never will be. However, if I submit the 'Website' front end to the Search Engines will the Robots automatically find the sub folder with the Shopping Cart in it?

I was thinking I should create a Robot.txt file that tells them to go and find the Cart and index that as part of the site too but I don't know if I need it or what to put into the file for the robots to understand what I want them to do.

  Proclaimer 12 Apr 10

although if a customer did hit the Zen-Cart first I wouldn't be too worried that they never seen the rest of the site. I am more concerned for the wee robots right now. :P

I do plan to link back to the Website from the Cart but I haven't really started editing the Cart yet, but thanks for pointing out that it will require links both ways.

I thought I might change the name of the Cart 'Home' button to something else and put my own Home button on the Cart that goes to the Website Homepage. Just need to work out what is a logical name for both 'Home' Buttons now.

  Ansolan 17 Apr 10


Great advice from forum member, websites are for people not robots. Incidentally, you can't use a robots.txt file to tell robots what to index, only where you don't want them to visit.

Even that is no guarantee of not having a URL indexed but you are in any event looking for the opposite. Would be worth submitting a .xml sitemap which includes the cart URLs click here No guarantee of indexing but can help with discovery.

Beyond that the point made about internal linking is important, as is the site itself. A minor observation on that, the stuffed keywords tag. Doesn't matter in an algorithmic sense, completely ignored due to previous abuse but if the site ever happened to come up for human review, the sort of silly thing that can tip decisions if someone's having a bad day.

A far more vital point, the content. Seems to apply in other places but as an example:

click here

click here

click here

Not much good I'm afraid, pot luck which might return in searches but in general, the longer standing and/or stronger site. You need to rewrite the descriptions or preferably add much more original useful content to each one. Otherwise a common scenario is that your pages will be indexed for afew weeks/months until Google get a grip on what they represent, then many will no longer appear in the index.

  Ansolan 17 Apr 10

Looks like the copyscape links are non functional, you can run the pages yourself anyway. The product content is all over the web.


This thread is now locked and can not be replied to.

Windows 10 Home vs Windows 10 Pro comparison: Here are the extra features you get in the Pro version

Why Formula-e needs to go faster

Found Studio reveals the making of the Unleashed, Uncut Peugeot 208 viral

How to use Apple Music in the UK: Complete guide to Apple Music's features

We use cookies to provide you with a better experience. If you continue to use this site, we'll assume you're happy with this. Alternatively, click here to find out how to manage these cookies

hide cookie message