• 1

Read this first!

We do not monitor these forums. The forum is provided to exchange information and experience with other users ONLY. Forum responses are not guaranteed.

However, please submit a ticket if you have an active subscription and wish to receive support. Our ticketing system is the only way of getting in touch with RSJoomla! and receiving the official RSJoomla! Customer Support.

For more information, the Support Policy is located here.

Thank you!

TOPIC: Robots Tag and RSSeo! Crawler

Robots Tag and RSSeo! Crawler 11 years 4 months ago #20548

  • jasonyohon
  • jasonyohon's Avatar
  • OFFLINE
  • Fresh Boarder
  • Posts: 2
  • Thank you received: 1
Does the the RSSeo crawler adhere to the robots tags when crawling? This would be really helpful when trying to drill down on tons of pages.
The administrator has disabled public write access.
The following user(s) said Thank You: albertdias

Robots Tag and RSSeo! Crawler 11 years 3 months ago #20999

In order to ignore links when crawling in RSSEO! you will have to go to RSSEO! > Settings > Crawler. Here in the "Ignore links when crawling" section you can type the links that you wish to be skipped when crawling and also use wildcards in order to cover a wider range of URLs that should be ignored by the Crawler.

www.rsjoomla.com/support/documentation/v...m-being-crawled.html
The administrator has disabled public write access.
  • 1

Read this first!

We do not monitor these forums. The forum is provided to exchange information and experience with other users ONLY. Forum responses are not guaranteed.

However, please submit a ticket if you have an active subscription and wish to receive support. Our ticketing system is the only way of getting in touch with RSJoomla! and receiving the official RSJoomla! Customer Support.

For more information, the Support Policy is located here.

Thank you!