*

Burton

  • ***
  • 128
Prevent Page Indexing
« on: February 16, 2015, 01:54:19 PM »
There are several pages that I do not want indexed, do not want showing in any search engine results.

What is the best method to do this?  using the robots.txt or adding meta tags onto the page?

For example the mysite/?page=stories page. How to prevent it being indexed?

<meta name="robots" content="noindex, nofollow, noarchive">

Where would I place this meta tag?  on a .tpl or a php <meta name="robots" content="noindex, nofollow, noarchive">

your help appreciated :)
« Last Edit: February 16, 2015, 02:03:33 PM by Burton »

*

Pharg

  • *****
  • 2,071
  • osDate Version: osDate Evo v1.0
Re: Prevent Page Indexing
« Reply #1 on: February 16, 2015, 08:37:00 PM »
Hi Burton,

That would be placed in the index_header.tpl but that would try to stop all the bots from indexing the entire site.

Best to use the robots.txt and put in each page you don't want indexed, not that all the bots follow those rules ;)
Regards,
Pharg ( Phill )

REMEMBER: ALWAYS BACKUP BEFORE YOU MAKE ANY CHANGES!!

osDateEvo v1.0 | PHP: 5.3.42 & PHP: 5.6.18 | MySQL: 5.5.35

*

Burton

  • ***
  • 128
Re: Prevent Page Indexing
« Reply #2 on: February 17, 2015, 01:32:37 AM »
Hi Phil,

appreciate your reply,

Yes I do not want to block entire site, just pages like, stories, article, news

This is my current robots.txt

Code: [Select]
User-agent: *
Disallow: /admin/
Disallow: /temp/
Disallow: /temp/cache/userimages/
Disallow: /temp/imageedits/
Disallow: /temp/tempimages/
Disallow: /temp/uservideos/
Disallow: /temp/userimages/
Disallow: /cronjobs/
Disallow: /chat/admin/
Disallow: /forum/admin/
Disallow: /temp/myconfigs/config.php

Sitemap: http://www.mydomain.com/sitemap.xml

How do I add individual pages?

Disallow: /?page=stories/

Disallow: /?page=articles/

Is that correct?

Phil do you also have Disallow for admin?  and any other pages?

*

Pharg

  • *****
  • 2,071
  • osDate Version: osDate Evo v1.0
Re: Prevent Page Indexing
« Reply #3 on: February 17, 2015, 01:45:36 AM »
Hi Burton,

Yes that's correct, some would say the best way to have this would be .
Code: [Select]
User-agent: *
Disallow:  admin/
Disallow:  temp/
Disallow:  temp/cache/userimages/
Disallow:  temp/imageedits/
Disallow:  temp/tempimages/
Disallow:  temp/uservideos/
Disallow:  temp/userimages/
Disallow:  cronjobs/
Disallow:  chat/admin/
Disallow:  forum/admin/
Disallow:  temp/myconfigs/config.php
Disallow:  index.php?page=stories
Disallow:  index.php?page=articles

Sitemap: http://www.mydomain.com/sitemap.xml
Regards,
Pharg ( Phill )

REMEMBER: ALWAYS BACKUP BEFORE YOU MAKE ANY CHANGES!!

osDateEvo v1.0 | PHP: 5.3.42 & PHP: 5.6.18 | MySQL: 5.5.35

*

Burton

  • ***
  • 128
Re: Prevent Page Indexing
« Reply #4 on: February 17, 2015, 02:08:41 AM »
Hi Phil,

Thank you for your quick helpful reply,

Do I also need to Disallow: Profile Search?  My Matches? Upgrade Membership? all other pages?

What do you do on your site Phil?

*

Pharg

  • *****
  • 2,071
  • osDate Version: osDate Evo v1.0
Re: Prevent Page Indexing
« Reply #5 on: February 17, 2015, 02:29:11 AM »
Hi Burton,

No, You don't need to add any of those pages.

You need to let the robots get some content, as the profiles change and that's good, especially for google as they
like content changing, Upgrade membership is up to you.

My site, I let them scan the stories, articles etc.... They can't scan my matches as that only work's logged in.
Regards,
Pharg ( Phill )

REMEMBER: ALWAYS BACKUP BEFORE YOU MAKE ANY CHANGES!!

osDateEvo v1.0 | PHP: 5.3.42 & PHP: 5.6.18 | MySQL: 5.5.35

*

Burton

  • ***
  • 128
Re: Prevent Page Indexing
« Reply #6 on: February 17, 2015, 02:46:40 AM »
Hi Phil,

Yes good info, want to feed those robots some content :)

Thank you :)