How to Enable Robots.txt file in your website
Robots.txt is the first thing a crawler visit in your website. Robots.txt tells search engine bot (spider or crawler) that which url needs to be crawled or which does not. Robots.txt Format Use the format below to create robots.txt file in notepad User-agent: * Allow: Sitemap: https://prashantchauhan.com/sitemap.xml Disallow: /dashboard/retrieve-password/ Disallow: /wp-admin “ * “ this …