C# (CSharp) Rock.UniversalSearch.Crawler.RobotsTxt Namespace

Nested Namespaces

Rock.UniversalSearch.Crawler.RobotsTxt.Helpers

Сlasses

Name Description
AccessRule
CrawlDelayRule
Line
Robots Provides functionality for parsing a robots.txt file's content and querying the rules and directives inside it.
Rule
Sitemap Represents a sitemap directive in a robots.txt file.