Exporting Data

Once you’ve extracted structured data, getting it into a usable format is usually the easy part — but a few habits keep it clean and safe to re-run.

CSV

package main

import (
  "encoding/csv"
  "fmt"
  "os"
)

type Product struct {
  Name  string
  Price float64
}

func writeCSV(products []Product, path string) error {
  f, err := os.Create(path)
  if err != nil {
    return err
  }
  defer f.Close()

  w := csv.NewWriter(f)
  defer w.Flush()

  w.Write([]string{"name", "price"})
  for _, p := range products {
    w.Write([]string{p.Name, fmt.Sprintf("%.2f", p.Price)})
  }
  return w.Error()
}

JSON

func writeJSON(products []Product, path string) error {
  f, err := os.Create(path)
  if err != nil {
    return err
  }
  defer f.Close()

  enc := json.NewEncoder(f)
  enc.SetIndent("", "  ")
  return enc.Encode(products)
}

package main

import (
"encoding/json"
"fmt"
"os"
)

type Product struct {
Name  string  `json:"name"`
Price float64 `json:"price"`
}

func main() {
products := []Product{
	{Name: "Widget", Price: 9.99},
	{Name: "Gadget", Price: 19.99},
}
enc := json.NewEncoder(os.Stdout)
enc.SetIndent("", "  ")
enc.Encode(products)
}

Deduplicating before you write

Reuse the set pattern from Arrays, Slices & Maps to avoid duplicate rows across multiple crawl runs:

seen := make(map[string]struct{})
var unique []Product
for _, p := range products {
  if _, ok := seen[p.Name]; ok {
    continue
  }
  seen[p.Name] = struct{}{}
  unique = append(unique, p)
}

Writing to a database idempotently

For repeated crawls, an upsert avoids duplicate rows without a separate dedup pass:

_, err := db.Exec(`
  INSERT INTO products (name, price)
  VALUES ($1, $2)
  ON CONFLICT (name) DO UPDATE SET price = EXCLUDED.price
`, p.Name, p.Price)

This makes the whole crawl-and-store pipeline safe to re-run on a schedule (a cron job, a Cloudflare Worker on a Cron Trigger, etc.) without accumulating duplicates.

Keeping provenance

Store when and from where each row was collected — it makes debugging a stale or wrong value trivial later:

type Product struct {
  Name       string
  Price      float64
  SourceURL  string
  FetchedAt  time.Time
}