go-crawler
A web crawler in Go, written to learn systems, not to scrape anything.
50 pages, 1000 books, and the rules they left behind.
What it taught me
- 01
A retry loop with three attempts but only two delays. Index out of range on the final attempt.
When two numbers have to stay in step, derive one from the other instead of writing both down.
- 02
A nil check that could never fire.
strings.Fieldsnever returns nil, not even for an empty string, so the guard was dead code and the next line would have panicked.Check the property you need, not something you assume stands in for it. nil does not mean empty, it means never allocated.
- 03
A semaphore where failed requests kept their slot. Two failures froze the crawler forever. It did not crash, which made it harder to find.
A resource is returned on every exit path, errors included. Holding it on failure is not a penalty, it is a leak.