We use cookies to provide you with a better experience. If you continue to use this site, we'll assume you're happy with this. Alternatively, click here to find out how to manage these cookies

hide cookie message
Contact Forum Editor

Send an email to our Forum Editor:


PLEASE NOTE: Your name is used only to let the Forum Editor know who sent the message. Both your name and email address will not be used for any other purpose.

Tech Helproom


It's free to register, to post a question or to start / join a discussion


 

Robot.txt Telling them where to go


Proclaimer

Likes # 0

I have a site (under construction) that is split into two. One part is a general website and the other is a Shopping Cart.

The General Site is here click here and the Cart (a Zen Cart) is here click here in a folder called 'store'.

The Cart could act as a stand alone website but I am not so happy with that so I have built the front end onto it.

The whole site is not seamless and because I am using a cart it never will be. However, if I submit the 'Website' front end to the Search Engines will the Robots automatically find the sub folder with the Shopping Cart in it?

I was thinking I should create a Robot.txt file that tells them to go and find the Cart and index that as part of the site too but I don't know if I need it or what to put into the file for the robots to understand what I want them to do.

Like this post
Proclaimer

Likes # 0

although if a customer did hit the Zen-Cart first I wouldn't be too worried that they never seen the rest of the site. I am more concerned for the wee robots right now. :P

I do plan to link back to the Website from the Cart but I haven't really started editing the Cart yet, but thanks for pointing out that it will require links both ways.

I thought I might change the name of the Cart 'Home' button to something else and put my own Home button on the Cart that goes to the Website Homepage. Just need to work out what is a logical name for both 'Home' Buttons now.

Like this post
Ansolan

Likes # 0

Hi

Great advice from forum member, websites are for people not robots. Incidentally, you can't use a robots.txt file to tell robots what to index, only where you don't want them to visit.

Even that is no guarantee of not having a URL indexed but you are in any event looking for the opposite. Would be worth submitting a .xml sitemap which includes the cart URLs click here No guarantee of indexing but can help with discovery.

Beyond that the point made about internal linking is important, as is the site itself. A minor observation on that, the stuffed keywords tag. Doesn't matter in an algorithmic sense, completely ignored due to previous abuse but if the site ever happened to come up for human review, the sort of silly thing that can tip decisions if someone's having a bad day.

A far more vital point, the content. Seems to apply in other places but as an example:

click here

click here

click here

Not much good I'm afraid, pot luck which might return in searches but in general, the longer standing and/or stronger site. You need to rewrite the descriptions or preferably add much more original useful content to each one. Otherwise a common scenario is that your pages will be indexed for afew weeks/months until Google get a grip on what they represent, then many will no longer appear in the index.

Like this post
Ansolan

Likes # 0

Looks like the copyscape links are non functional, you can run the pages yourself anyway. The product content is all over the web.

Like this post

Reply to this topic

This thread has been locked.



IDG UK Sites

Moto G2 (2014) vs Moto E comparison review: New Moto G is worth the extra cash

IDG UK Sites

Is Apple losing confidence in itself?

IDG UK Sites

Oculus Rift 'Crescent Bay' prototype hands-on: it's an amazing experience

IDG UK Sites

How (and where) to buy an iPhone 6 or iPhone 6 Plus in the UK. Plus: What to do if you pre-ordered...