robots.txt

1. 在publuc目录下
2. 通过<IP>/robots.txt访问 # Block all crawlers for / 防止爬虫爬取的地方 # Allow all crawlers 允许爬虫爬取的地方

# Block all crawlers for /accounts  
  
# Allow all crawlers  
User-agent: *  
Allow: /