robots.txt file for osdate?
« on: August 04, 2014, 02:13:49 PM »
hi everyone, can someone help me im needing a robots txt file for osdate 2.65 but not sure what files to allow or exclude? thanks,

*

Pharg

  • *****
  • 2,071
  • osDate Version: osDate Evo v1.0
Re: robots.txt file for osdate?
« Reply #1 on: August 05, 2014, 12:29:30 AM »
Hi mnarkyp77,

Here is a basic robots.txt

Code: [Select]
User-agent: *
Disallow: /admin/
Disallow: /temp/
Disallow: /temp/cache/userimages/
Disallow: /temp/imageedits/
Disallow: /temp/tempimages/
Disallow: /temp/uservideos/
Disallow: /temp/userimages/
Disallow: /cronjobs/
Disallow: /chat/admin/
Disallow: /forum/admin/
Disallow: /temp/myconfigs/config.php

Sitemap: http://YOUR-DOMAIN-NAME/sitemap.xml

Sitemap: http://YOUR-DOMAIN-NAME/urllist.txt

Copy what you want and name the file robots.txt
Regards,
Pharg ( Phill )

REMEMBER: ALWAYS BACKUP BEFORE YOU MAKE ANY CHANGES!!

osDateEvo v1.0 | PHP: 5.3.42 & PHP: 5.6.18 | MySQL: 5.5.35

Re: robots.txt file for osdate?
« Reply #2 on: August 05, 2014, 12:51:43 AM »
hi mate thanks for that, im having a problem, when checking its sending back 403 error, its saying please correct the url, i redirect from http to https could this be the issue?
« Last Edit: August 05, 2014, 01:01:37 AM by mnarkyp77 »

*

Pharg

  • *****
  • 2,071
  • osDate Version: osDate Evo v1.0
Re: robots.txt file for osdate?
« Reply #3 on: August 05, 2014, 02:26:49 AM »
Hi mnarkyp77,

More than likely, you don't have sitemap setup in your domain so you need to remove the following lines

Sitemap: http://YOUR-DOMAIN-NAME/sitemap.xml

Sitemap: http://YOUR-DOMAIN-NAME/urllist.txt

These are only used in robots txt when you have a sitemap system setup
Regards,
Pharg ( Phill )

REMEMBER: ALWAYS BACKUP BEFORE YOU MAKE ANY CHANGES!!

osDateEvo v1.0 | PHP: 5.3.42 & PHP: 5.6.18 | MySQL: 5.5.35

Re: robots.txt file for osdate?
« Reply #4 on: August 05, 2014, 08:29:16 AM »
I have sitemap, and I removed the other line this is weird

*

Pharg

  • *****
  • 2,071
  • osDate Version: osDate Evo v1.0
Re: robots.txt file for osdate?
« Reply #5 on: August 05, 2014, 08:41:30 AM »
Just had a thought that I am betting that no htaccess has been setup for it either.

In your .htaccess file add these lines
Code: [Select]
RewriteRule ^robots.txt$ robots.txt$1

RewriteRule ^sitemap.xml$ sitemap.xml$1

RewriteRule ^urllist.txt$ urllist.txt$1

RewriteRule ^sitemap.txt$ sitemap.txt [L]


Regards,
Pharg ( Phill )

REMEMBER: ALWAYS BACKUP BEFORE YOU MAKE ANY CHANGES!!

osDateEvo v1.0 | PHP: 5.3.42 & PHP: 5.6.18 | MySQL: 5.5.35

*

Pharg

  • *****
  • 2,071
  • osDate Version: osDate Evo v1.0
Re: robots.txt file for osdate?
« Reply #6 on: August 07, 2014, 08:03:53 AM »
I have sitemap, and I removed the other line this is weird

Hi mnarkyp77,

Did you get the robots.txt working on your site and did you add the .htaccess code  so the rorbots.txt is not restricted or redirected ?
Regards,
Pharg ( Phill )

REMEMBER: ALWAYS BACKUP BEFORE YOU MAKE ANY CHANGES!!

osDateEvo v1.0 | PHP: 5.3.42 & PHP: 5.6.18 | MySQL: 5.5.35

Re: robots.txt file for osdate?
« Reply #7 on: August 07, 2014, 08:30:58 AM »
No still not working :(

*

Pharg

  • *****
  • 2,071
  • osDate Version: osDate Evo v1.0
Re: robots.txt file for osdate?
« Reply #8 on: August 07, 2014, 10:56:07 PM »
Remove both the Sitemaps from the robots.txt

Code: [Select]
Sitemap: http://YOUR-DOMAIN-NAME/sitemap.xml

Sitemap: http://YOUR-DOMAIN-NAME/urllist.txt

Then put in the .htaccess

Code: [Select]
RewriteRule ^robots.txt$ robots.txt$1

RewriteRule ^sitemap.xml$ sitemap.xml$1

RewriteRule ^urllist.txt$ urllist.txt$1

RewriteRule ^sitemap.txt$ sitemap.txt [L]

Make sure it's named robots.txt Not robots.text
Regards,
Pharg ( Phill )

REMEMBER: ALWAYS BACKUP BEFORE YOU MAKE ANY CHANGES!!

osDateEvo v1.0 | PHP: 5.3.42 & PHP: 5.6.18 | MySQL: 5.5.35

Re: robots.txt file for osdate?
« Reply #9 on: August 07, 2014, 11:02:37 PM »
Remove both the Sitemaps from the robots.txt

Code: [Select]
Sitemap: http://YOUR-DOMAIN-NAME/sitemap.xml

Sitemap: http://YOUR-DOMAIN-NAME/urllist.txt

Then put in the .htaccess

Code: [Select]
RewriteRule ^robots.txt$ robots.txt$1

RewriteRule ^sitemap.xml$ sitemap.xml$1

RewriteRule ^urllist.txt$ urllist.txt$1

RewriteRule ^sitemap.txt$ sitemap.txt [L]

Make sure it's named robots.txt Not robots.text

still gives error, no robots.txt found,

*

Pharg

  • *****
  • 2,071
  • osDate Version: osDate Evo v1.0
Re: robots.txt file for osdate?
« Reply #10 on: August 07, 2014, 11:09:10 PM »
Just went to your site and there doesn't seem to be a robots.txt uploaded as I got this error

Forbidden

You don't have permission to access /robots.txt on this server.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.
Regards,
Pharg ( Phill )

REMEMBER: ALWAYS BACKUP BEFORE YOU MAKE ANY CHANGES!!

osDateEvo v1.0 | PHP: 5.3.42 & PHP: 5.6.18 | MySQL: 5.5.35

Re: robots.txt file for osdate?
« Reply #11 on: August 07, 2014, 11:15:19 PM »
its deffo there, i think this is a maybe a problem because my siute redirect to https from http

*

Pharg

  • *****
  • 2,071
  • osDate Version: osDate Evo v1.0
Re: robots.txt file for osdate?
« Reply #12 on: August 07, 2014, 11:21:04 PM »
Yep that's a strange one alright  :o
Regards,
Pharg ( Phill )

REMEMBER: ALWAYS BACKUP BEFORE YOU MAKE ANY CHANGES!!

osDateEvo v1.0 | PHP: 5.3.42 & PHP: 5.6.18 | MySQL: 5.5.35

*

Burton

  • ***
  • 128
Re: robots.txt file for osdate?
« Reply #13 on: February 10, 2015, 06:33:56 AM »
Hi Phil,

I have just added the basic robots.txt from the first post.

Code: [Select]
User-agent: *
Disallow: /admin/
Disallow: /temp/
Disallow: /temp/cache/userimages/
Disallow: /temp/imageedits/
Disallow: /temp/tempimages/
Disallow: /temp/uservideos/
Disallow: /temp/userimages/
Disallow: /cronjobs/
Disallow: /chat/admin/
Disallow: /forum/admin/
Disallow: /temp/myconfigs/config.php

Sitemap: http://YOUR-DOMAIN-NAME/sitemap.xml

Sitemap: http://YOUR-DOMAIN-NAME/urllist.txt

2 questions:

I do not have a urllist.txt so do I remove Sitemap: http://YOUR-DOMAIN-NAME/urllist.txt or get a urllist.txt?

Do I need to add this to my .htaccess ?

Code: [Select]
RewriteRule ^robots.txt$ robots.txt$1

RewriteRule ^sitemap.xml$ sitemap.xml$1

RewriteRule ^urllist.txt$ urllist.txt$1

RewriteRule ^sitemap.txt$ sitemap.txt [L]


*

Pharg

  • *****
  • 2,071
  • osDate Version: osDate Evo v1.0
Re: robots.txt file for osdate?
« Reply #14 on: February 10, 2015, 07:45:07 AM »
Hi Burton,

If you don't have a urllist.txt or a Sitemap.xml then you don't need to add the
lines in the .htaccess
Regards,
Pharg ( Phill )

REMEMBER: ALWAYS BACKUP BEFORE YOU MAKE ANY CHANGES!!

osDateEvo v1.0 | PHP: 5.3.42 & PHP: 5.6.18 | MySQL: 5.5.35