Consumer-agent: [Needed, a number of for each team] The rule specifies the name of the automated customer called search engine crawler which the rule applies to. You've got entry to numerous absolutely free robots.txt generator tools online, most of which happen to be absolutely free. On the other hand, the https://seotoolstube.com/