Should I include Accept-Encoding header for compression?

Yes, always include Accept-Encoding to enable compression. It reduces bandwidth and looks more like a real browser.

What it does:

Tells the server which compression formats your client supports:

Accept-Encoding: gzip, deflate, br

Accept-Encoding: gzip, deflate, br

Supported formats:

gzip:

Most common
Supported by all browsers
Good compression ratio

deflate:

Older compression format
Less common now
Include for compatibility

br (Brotli):

Modern, best compression
Supported by recent browsers
Saves bandwidth

Benefits:

Bandwidth savings:

Text content: 70-90% size reduction
HTML/CSS/JS: Often 80%+ reduction
Faster downloads

Appears legitimate:

All browsers send Accept-Encoding
Missing it is suspicious

Performance:

Smaller transfers = faster scraping
Lower bandwidth costs
Less data to parse (uncompressed on the fly)

Python example:

import requests

headers = {
    'Accept-Encoding': 'gzip, deflate, br'
}

# requests library handles decompression automatically
response = requests.get(url, headers=headers)
# response.text is already decompressed

import requests

headers = {
    'Accept-Encoding': 'gzip, deflate, br'
}

# requests library handles decompression automatically
response = requests.get(url, headers=headers)
# response.text is already decompressed

Node.js example:

const axios = require('axios');

const response = await axios.get(url, {
  headers: {
    'Accept-Encoding': 'gzip, deflate, br'
  }
});
// axios automatically decompresses

const axios = require('axios');

const response = await axios.get(url, {
  headers: {
    'Accept-Encoding': 'gzip, deflate, br'
  }
});
// axios automatically decompresses

Handling compressed responses:

Most HTTP libraries (requests, axios, fetch) automatically decompress responses. You don't need to do anything special:

# requests handles decompression automatically
response = requests.get(url, headers={'Accept-Encoding': 'gzip'})
html = response.text  # Already decompressed

# requests handles decompression automatically
response = requests.get(url, headers={'Accept-Encoding': 'gzip'})
html = response.text  # Already decompressed

When compression might not be used:

Binary files (images, videos) - already compressed
Very small responses - overhead not worth it
Server doesn't support compression
Some APIs explicitly disable it

Checking if compressed:

response = requests.get(url, headers={'Accept-Encoding': 'gzip'})
print(response.headers.get('Content-Encoding'))
# Output: 'gzip' if compressed

response = requests.get(url, headers={'Accept-Encoding': 'gzip'})
print(response.headers.get('Content-Encoding'))
# Output: 'gzip' if compressed

Mobile vs desktop:

Both support the same encodings:

Accept-Encoding: gzip, deflate, br

Accept-Encoding: gzip, deflate, br

Recommendation:

Always include Accept-Encoding: gzip, deflate, br in your headers. It's:

Expected by servers
Reduces bandwidth costs significantly
Handled automatically by HTTP libraries
Makes your scraper look more legitimate

Never send empty or no Accept-Encoding header.

Should I include Accept-Encoding header for compression?

Related Questions