| Rock.UniversalSearch.Crawler.RobotsTxt.Helpers |
| Name | Description |
|---|---|
| AccessRule | |
| CrawlDelayRule | |
| Line | |
| Robots | Provides functionality for parsing a robots.txt file's content and querying the rules and directives inside it. |
| Rule | |
| Sitemap | Represents a sitemap directive in a robots.txt file. |