Contact Forum Editor

Send an email to our Forum Editor:

PLEASE NOTE: Your name is used only to let the Forum Editor know who sent the message. Both your name and email address will not be used for any other purpose.

Web Design Help

It's free to register, to post a question or to start / join a discussion

View Forum Rules | Contact Forum Editor

 

Robot.txt Telling them where to go

Proclaimer

Likes # 0

Posted April 12, 2010 at 4:35AM

I have a site (under construction) that is split into two. One part is a general website and the other is a Shopping Cart.

The General Site is here click here and the Cart (a Zen Cart) is here click here in a folder called 'store'.

The Cart could act as a stand alone website but I am not so happy with that so I have built the front end onto it.

The whole site is not seamless and because I am using a cart it never will be. However, if I submit the 'Website' front end to the Search Engines will the Robots automatically find the sub folder with the Shopping Cart in it?

I was thinking I should create a Robot.txt file that tells them to go and find the Cart and index that as part of the site too but I don't know if I need it or what to put into the file for the robots to understand what I want them to do.

Like this post
Ansolan

Likes # 0

Posted April 17, 2010 at 11:03PM

Looks like the copyscape links are non functional, you can run the pages yourself anyway. The product content is all over the web.

Like this post
Ansolan

Likes # 0

Posted April 17, 2010 at 10:55PM

Hi

Great advice from forum member, websites are for people not robots. Incidentally, you can't use a robots.txt file to tell robots what to index, only where you don't want them to visit.

Even that is no guarantee of not having a URL indexed but you are in any event looking for the opposite. Would be worth submitting a .xml sitemap which includes the cart URLs click here No guarantee of indexing but can help with discovery.

Beyond that the point made about internal linking is important, as is the site itself. A minor observation on that, the stuffed keywords tag. Doesn't matter in an algorithmic sense, completely ignored due to previous abuse but if the site ever happened to come up for human review, the sort of silly thing that can tip decisions if someone's having a bad day.

A far more vital point, the content. Seems to apply in other places but as an example:

click here

click here

click here

Not much good I'm afraid, pot luck which might return in searches but in general, the longer standing and/or stronger site. You need to rewrite the descriptions or preferably add much more original useful content to each one. Otherwise a common scenario is that your pages will be indexed for afew weeks/months until Google get a grip on what they represent, then many will no longer appear in the index.

Like this post
Proclaimer

Likes # 0

Posted April 12, 2010 at 8:50AM

although if a customer did hit the Zen-Cart first I wouldn't be too worried that they never seen the rest of the site. I am more concerned for the wee robots right now. :P

I do plan to link back to the Website from the Cart but I haven't really started editing the Cart yet, but thanks for pointing out that it will require links both ways.

I thought I might change the name of the Cart 'Home' button to something else and put my own Home button on the Cart that goes to the Website Homepage. Just need to work out what is a logical name for both 'Home' Buttons now.

Like this post

Reply to this topic

This thread has been locked.



Check out PC Advisor's other tech forums

Send to a friend

Email this article to a friend or colleague:

PLEASE NOTE: Your name is used only to let the recipient know who sent the story. Both your name and the recipient's name and address will not be used for any other purpose.