Skip to content
The Go gophernasigolang.com

Respecting robots.txt & Rate Limits

Reading robots.txt

Section titled “Reading robots.txt”

robots.txt lives at the root of a domain (https://example.com/robots.txt) and lists which paths a given crawler (identified by User-agent) may or may not access, and sometimes a requested Crawl-delay:

User-agent: *
Disallow: /admin/
Disallow: /search
Crawl-delay: 5


User-agent: nasigolang-example-bot
Allow: /blog/
Disallow: /

A crawler should fetch and parse this before requesting anything else, and honor whichever block matches its own User-Agent (falling back to User-agent: *).

Parsing it in Go

Section titled “Parsing it in Go”

The temoto/robotstxt package is a well-established parser:

Terminal window
go get github.com/temoto/robotstxt
package main


import (
  "fmt"
  "net/http"


  "github.com/temoto/robotstxt"
)


func main() {
  resp, err := http.Get("https://example.com/robots.txt")
  if err != nil {
    panic(err)
  }
  defer resp.Body.Close()


  data, err := robotstxt.FromResponse(resp)
  if err != nil {
    panic(err)
  }


  group := data.FindGroup("nasigolang-example-bot")
  fmt.Println(group.Test("/blog/post-1")) // true or false
  fmt.Println(group.CrawlDelay)           // requested delay, if any
}

Build your crawler’s request loop so it calls group.Test(path) before every fetch, and sleeps for at least group.CrawlDelay between requests to that host.

Honoring rate-limit responses

Section titled “Honoring rate-limit responses”

Beyond robots.txt, watch the HTTP response itself:

resp, err := client.Do(req)
if err != nil {
  return err
}
defer resp.Body.Close()


if resp.StatusCode == http.StatusTooManyRequests {
  if retryAfter := resp.Header.Get("Retry-After"); retryAfter != "" {
    if secs, err := strconv.Atoi(retryAfter); err == nil {
      time.Sleep(time.Duration(secs) * time.Second)
    }
  }
  // retry after waiting
}

A 429 with Retry-After is the server explicitly telling you how long to wait — a well-behaved crawler always honors it rather than guessing its own backoff.

Per-host rate limiting

Section titled “Per-host rate limiting”

If your crawler ever touches more than one domain, track limits per host, not globally — golang.org/x/time/rate makes this easy:

import "golang.org/x/time/rate"


limiters := make(map[string]*rate.Limiter)


func limiterFor(host string) *rate.Limiter {
  if l, ok := limiters[host]; ok {
    return l
  }
  l := rate.NewLimiter(rate.Every(2*time.Second), 1) // 1 request every 2s
  limiters[host] = l
  return l
}


// before each request:
limiterFor(host).Wait(ctx)

What this deliberately does not cover

Section titled “What this deliberately does not cover”

This chapter — and this site — won’t cover rotating IPs, spoofing browser fingerprints, or otherwise trying to make a crawler indistinguishable from a real user in order to get around a site’s rate limits or bot protections. If robots.txt, rate limits, and Retry-After responses are telling you a site doesn’t want to be crawled the way you’re trying to crawl it, that’s the signal to stop, slow down further, or reach out to the site owner for an API — not to disguise the crawler.