Basic SEO for Web Developers Part 2



  • 1. Robots Exclusion Standard

    • Robots.txt
    Location: https://example.com/robots.txt
    
    User-agent: googlebot
    Disallow: /example.html
    Sitemap: https://example.com/sitemap.xml
    
    ##More information at http://www.robotstxt.org/robotstxt.html
    
    • X-Robots
    Location: Sent in the HTTP headers
    
    X-Robots-Tag: noindex
    
    ##More information at http://noarchive.net/xrobots/
    
    • Meta Robots
    Location: In the html <head>
    
    <meta name="ROBOT NAME" content="ARGUMENTS" />
    
    ##More information athttp://www.robotstxt.org/meta.html
    
    • Robots Best Practices

      1. Only Meta Robots and X-Robots remove URLs from search results
      2. Don't block CSS or JavaScript files with robots.txt
    • Arguments can be:

      1. Nofollow (do not follow links)
      2. Noindex (do not index)
      3. Noarchive (do not archive)
      4. NoODP (Do not show Open Directory Project description)...Or combined (noindex, nofollow)
      5. If the robots <META> tag is not defined, the default is "INDEX,FOLLOW"

    2. Important User Agents

    • For robots.txt, robots meta tags, and X-Robots-Tag

      Googlebot (can be used as default for most Google crawlers)
      Googlebot-News
      Googlebot-Image
      AdsBot-Google
      Baiduspider
      FacebookExternalHit
      Slurp
      Mediapartners-Google (Mobile Adsense) or Mediapartners
      Googlebot-Mobile
      Googlebot-Video
      Bingbot
      Yandexbot
      Applebot
      Twitterbot
      Rogerbot
      

    3. Sitemap Syntax

    • XML Sitemaps

      <?xml version="1.0" encoding="UTF-8"?>
      <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
          <url>
              <loc>https://example.com/</loc> 
              <lastmod>2015-01-01</lastmod> 
              <changefreq>monthly</changefreq> 
              <priority>0.9</priority>
          </url> 
      </urlset>
      
    • Sitemap Index File

      <?xml version="1.0" encoding="UTF-8"?>
      <sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
          <sitemap>
              <loc>https://example.com/sitemap1.xml.gz</loc>
              <lastmod>2015-01-01T18:23:17+00:00</lastmod>
          </sitemap>
          <sitemap>
              <loc>https://example.com/sitemap2.xml.gz</loc>
              <lastmod>2015-01-01</lastmod>
          </sitemap>
      </sitemapindex>
      
    • Default Locations Can Be:

      https://example.com/sitemap.xml
      https://example.com/sitemap.xml.gz
      https://example.com/sitemap.gz
      
    • Other Common Sitemap Types:

      Mobile
      News
      Image
      Video
      

    4. Pagination

    • Use rel="next" andrel="prev" in the <head> sectionto indicate the relationship between paginated URLs

    • First Page - https://example.com/article

      <link rel="next" href="https://example.com/article?pg=2">
      
    • Second Page - https://example.com/article?pg=2

      <link rel="prev" href="https://example.com/article">
      <link rel="next" href="https://example.com/article?pg=3">
      
    • Final Page - https://example.com/article?pg=3

      <link rel="prev" href="https://example.com/article?pg=2">
      
      #More information at http://mz.cm/rel-next
      

    I'll post next part asap.
    Thank you all for your time in reading this!


  • Linux Help

    Thnx bro. can you link this part with previous part.



  • @root ur welcome, sure i'll link


  • Node.js

    Nice post... 😍



  • Thank you...



  • @fern & @rachitha ur welcome 😃


Log in to reply