Ethics & Ground Rules

Go is a genuinely great language for fetching and processing data from the web — good HTTP client, easy concurrency, single-binary deployment. This section covers how to do that responsibly, because a scraper that ignores the rules below can get your IP banned, break a site for other users, or cross into legally risky territory.

Ground rules for this whole section

1. Check the site’s Terms of Service first. Many sites explicitly prohibit automated scraping; others provide an official API specifically so you don’t have to. If an API exists, use it — it’s more stable and won’t get you blocked.
2. Respect robots.txt. It’s the site owner’s explicit statement of what they’re comfortable being crawled. Covered in detail in Respecting robots.txt & Rate Limits.
3. Identify yourself. Send a real User-Agent with a contact URL or email, e.g. MyCrawler/1.0 (+https://example.com/bot). This is the opposite of trying to look like a regular browser — a good crawler is identifiable so a site operator can reach out if there’s a problem, instead of just silently blocking you.
4. Rate-limit yourself. One request every few seconds from a single crawler is invisible to a site’s infrastructure. Hundreds of concurrent requests can degrade service for real users — treat someone else’s server the way you’d want yours treated.
5. Only scrape data you’re allowed to store. Personal data, paywalled content, and copyrighted text all carry their own legal considerations independent of whether the scrape itself was “technically” possible.
6. Never try to defeat anti-bot/anti-abuse protections (CAPTCHAs, Cloudflare Turnstile/Challenge pages, rate-limit walls) on a site you don’t own. Those exist because the site operator decided automated access isn’t welcome there — bypassing them isn’t a scraping “technique”, it’s circumventing someone else’s explicit decision about their own infrastructure. If a site is protected this heavily, that’s a strong signal to look for an official API or partnership instead.

What this section actually teaches

With those rules in place, the rest of this section covers the genuinely useful, durable skills:

• Fetching and parsing HTML with net/http and goquery (Your First Scraper)
• Structuring concurrent crawls safely with worker pools (Concurrent Crawling)
• Parsing robots.txt, honoring Crawl-delay, and backing off on 429/Retry-After (Respecting robots.txt & Rate Limits)
• Getting the extracted data into a usable shape — CSV, JSON, or a database (Exporting Data)

Use case

The most common legitimate use case for a Go scraper is monitoring your own site or a small number of pages you have explicit permission to crawl — e.g. tracking price changes on a partner’s public product feed, or archiving your own company’s public documentation for offline search. When in doubt, ask the site owner for an API key or a data export instead of scraping.