You may not want certain pages of your site crawled because they might not be useful to users if found in a search engine's search results. If you do want to prevent search engines from crawling your pages, Google Search Console has a friendly robots.txt generator to help you create this file. Note that if your site uses subdomains and you wish to have certain pages not crawled on a particular subdomain, you'll have to create a separate robots.txt file for that subdomain. For more information on robots.txt, we suggest this Webmaster Help Center guide on using robots.txt files13.
Internet Marketing Inc. is one of the fastest growing full service Internet marketing agencies in the country with offices in San Diego, and Las Vegas. We specialize in providing results driven integrated online marketing solutions for medium-sized and enterprise brands across the globe. Companies come to us because our team of well-respected industry experts has the talent and creativity to provide your business with a more sophisticated data-driven approach to digital marketing strategy. IMI works with some clients through IMI Ventures, and their first product is VitaCup.
×