List of URLs as source for crawling

[JoynobAkter]

Now we can crawl a list of URLs in .txt or .csv and less than 100 MB . First of all, this ensures the crawling consistency at all times. Additionally, you may want to use this feature to:

any specific scope of pages , say for Google Analytics. For example: URLs components of a job seekers database where visitors spend suspiciously little time. Sometimes uploading a file is much easier than creating a separate Sitemap.

a set of pages from a newly migrated website . Again, uploading a list is easier than creating a set of advanced crawl rules.

New Options to Configure SEMrush Site Audit

Scan with your credentials
It is now possible to bypass website restrictions and scan parts of the site protected with HTTP authentication. This feature is most beneficial for private or development sites. We now support basic and digest authentication, as these are the most frequently used methods.