What is a User-Agent and why should I change it?
The User-Agent header is a string that identifies the client software making the HTTP request, including the browser name, version, and operating system.
Example User-Agent:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
This identifies a Chrome browser on Windows.
What servers do with User-Agent:
- Deliver appropriate content (mobile vs desktop)
- Provide compatible HTML/CSS/JavaScript for older browsers
- Return crawler-friendly content for search engines
- Track browser usage statistics
Why User-Agent matters for web scraping:
Many websites block requests with:
- Non-browser user agents
- Missing User-Agent headers entirely
- Outdated or suspicious user agent strings
Default library user agents reveal scraping:
- Python's
requests:python-requests/2.x.x - Node.js
axios:axios/1.x.x - Go's
net/http:Go-http-client/1.1
These are frequently blocked by anti-scraping systems.
Why changing User-Agent isn't enough:
Simply changing the User-Agent to mimic a browser is not sufficient:
- You must send other headers that real browsers include (
Accept,Accept-Language,Accept-Encoding) - Header order matters for some anti-bot systems
- Some systems check if the User-Agent matches the actual browser fingerprint from JavaScript execution
Best practices:
- Rotate between multiple recent browser User-Agents
- Ensure all headers are consistent with your chosen User-Agent
- Avoid outdated user agent strings (old browser versions)
- Use realistic combinations (don't claim to be Chrome but send Firefox-specific headers)
- Keep user agents updated with current browser versions
Advanced detection:
Some anti-bot systems:
- Check JavaScript execution environment against claimed User-Agent
- Validate that browser features match the claimed version
- Monitor for impossible combinations (e.g., Safari on Windows)