Concurrent Crawling

Go’s goroutines make it tempting to fetch hundreds of URLs at once. Don’t. A bounded worker pool (introduced in Concurrency Patterns) with a small worker count is both more polite and, in practice, more reliable — servers are more likely to start throttling or blocking a crawler that suddenly opens 200 connections.

A polite, bounded crawler

package main

import (
  "fmt"
  "log"
  "net/http"
  "sync"
  "time"
)

type Result struct {
  URL    string
  Status int
  Err    error
}

func fetch(client *http.Client, url string) Result {
  req, _ := http.NewRequest("GET", url, nil)
  req.Header.Set("User-Agent", "nasigolang-example-bot/1.0 (+https://example.com/bot)")

  resp, err := client.Do(req)
  if err != nil {
    return Result{URL: url, Err: err}
  }
  defer resp.Body.Close()
  return Result{URL: url, Status: resp.StatusCode}
}

func crawl(urls []string, concurrency int, delay time.Duration) []Result {
  client := &http.Client{Timeout: 10 * time.Second}
  jobs := make(chan string)
  results := make([]Result, 0, len(urls))
  var mu sync.Mutex
  var wg sync.WaitGroup

  for i := 0; i < concurrency; i++ {
    wg.Add(1)
    go func() {
      defer wg.Done()
      for url := range jobs {
        res := fetch(client, url)
        mu.Lock()
        results = append(results, res)
        mu.Unlock()
        time.Sleep(delay) // pace requests from this worker
      }
    }()
  }

  go func() {
    for _, u := range urls {
      jobs <- u
    }
    close(jobs)
  }()

  wg.Wait()
  return results
}

func main() {
  urls := []string{
    "https://example.com/page1",
    "https://example.com/page2",
    "https://example.com/page3",
  }

  results := crawl(urls, 2, 500*time.Millisecond)
  for _, r := range results {
    if r.Err != nil {
      log.Printf("%s: error: %v", r.URL, r.Err)
      continue
    }
    fmt.Printf("%s: %d\n", r.URL, r.Status)
  }
}

A concurrency of 2 and a 500ms per-worker delay means this crawler makes roughly 4 requests per second at most — deliberately conservative.

Backing off on errors

Real crawls hit transient failures. Retry with exponential backoff instead of hammering the same URL immediately:

func fetchWithRetry(client *http.Client, url string, maxAttempts int) Result {
  var lastErr error
  for attempt := 0; attempt < maxAttempts; attempt++ {
    res := fetch(client, url)
    if res.Err == nil && res.Status < 500 {
      return res
    }
    lastErr = res.Err
    backoff := time.Duration(1<<attempt) * time.Second // 1s, 2s, 4s...
    time.Sleep(backoff)
  }
  return Result{URL: url, Err: lastErr}
}

A 429 Too Many Requests or 503 response is the server telling you to slow down — treat it as a signal, not an obstacle, and check for a Retry-After header (see the next chapter).