Crawl &Configuration
Crawl options:images,nofollow,robots.txt
Configure what the crawler follows and collects.
Crawl images
When enabled,the crawler fetches image URLs and records alt text,dimensions,and size. Use for image SEO audits.
Follow nofollow links
By default nofollow links are not followed. Enable in config if you want to include them in the crawl (e.g. for full coverage).
Respect robots.txt
KWT Spider always respects robots.txt by default. Disallowed paths are not crawled.
URL scope
Only URLs under the same domain (and subdomain,depending on settings) are crawled as internal;others are logged as external.