How to scrape a website without getting blocked

Headers, pacing, IP rotation and geo-targeting: the habits that keep a scraper running, with code.

By DINPROXY Team · · 5 min read

On this page

Sites usually block scrapers for a few predictable reasons: too many requests from one address, headers that announce a script, or a pace no person could keep. Fix those, and most public pages become routine to collect. This guide covers the habits that keep a scraper running, and where DINPROXY residential proxies and the Web Unlocker fit in.

Send requests that look like a real browser

Every HTTP request carries headers that describe the client. Scripting libraries send short default headers that mark the traffic as automated before the page even loads. Send the headers a current browser would send instead.

  • User-Agent: copy the full string from an up-to-date desktop or mobile browser, and keep it the same for a whole session.
  • Accept and Accept-Language: match the language to the country you browse from, such as en-US for a US IP or de-DE for a German one.
  • Referer: when you follow a link, send the page you came from, as a browser does.
  • Cookies: keep the cookies a site sets and send them back on later requests in the same session.

Consistency matters as much as realism. A visitor with a German IP, a Japanese language header and a User-Agent that changes on every page stands out. Keep each identity's location, headers and cookies in step with each other.

Some pages build their content with JavaScript, so the HTML you download may not hold the data yet. Look for the JSON request the page makes and fetch that directly, or let a service render the page for you, as described in the last section.

Pace your requests and back off

Speed is the quickest way to stand out. A person reads a page before opening the next one, while a script can request hundreds of pages a second. Slow down to a pace the site can comfortably serve.

  • Space requests out with a short random delay, rather than a fixed interval.
  • Limit how many requests run against one site at once, and raise the limit gradually while watching the response codes.
  • When a site answers 429 or 503, wait before retrying and double the wait on each further attempt (exponential backoff). If the response has a Retry-After header, wait that long.
  • Cap the retries per URL and log the URLs you set aside, so you can look at them separately.

On DINPROXY, each account can keep up to 100 proxy connections open at once, residential and mobile together. Size your worker pool to fit within that, and close connections you no longer need.

Rotate IPs for pages, stay sticky for flows

Even a well-paced scraper concentrates a lot of traffic on one address if it only has one. Residential proxies spread your requests across IPs of real home connections, so each request reaches the site from an ordinary visitor's address.

With DINPROXY residential proxies, rotation is the default: a username with no session option gives every request a new IP. Use that for search results, listings and product pages, where each page stands on its own. An HTTPS connection keeps one IP while it stays open, so open a new connection for each request when you want a new IP every time.

Some jobs need the opposite. A login, a cart or a multi-step form expects one visitor from start to finish. For those, add a session ID and a duration to the username, such as @session=cart17@time=20. Every connection with that ID keeps the same IP for that many minutes, up to 120. Our guide to rotating vs sticky residential proxies covers when to use each in detail.

Get the right local page with geo-targeting

Many sites change prices, currency, language, search results and stock by location. Collect from the wrong place and you get data that looks right but describes another market. Choose the location of the exit IP to match the market you are studying.

DINPROXY residential proxies target by country, state and city. Add the options to the username: @country=us for the United States, @country=de@state=hesse for a German state, or @country=gb@state=england@city=london for one city. Countries are two-letter codes in lower case, and states and cities are written in lower case without spaces. The Select Location list in the dashboard's Connection setup shows the available locations and writes the values for you.

Here is one request through a US residential IP with browser-like headers. The reply from ipinfo.io/json shows the IP and location the site sees:

import { fetch, ProxyAgent } from "undici";
const token = Buffer.from("USERNAME@country=us:PASSWORD").toString("base64");const dispatcher = new ProxyAgent({ uri: "http://proxy.dinproxy.com:7777", token: `Basic ${token}` });
const response = await fetch("https://ipinfo.io/json", {  dispatcher,  headers: {    "User-Agent": "Mozilla/5.0 ...", // copy the full string from a current browser    "Accept-Language": "en-US,en;q=0.9",  },});console.log(await response.text());

Replace USERNAME with your connection username from Tools → Users, such as 1024~myuser, and PASSWORD with its password. Use http:// for HTTP and HTTPS sites and socks5h:// for SOCKS5, both on proxy.dinproxy.com port 7777. The residential code examples cover all eight languages, from Node.js to Ruby.

Hand pages behind bot checks to the Web Unlocker

Some pages sit behind CAPTCHAs or anti-bot systems that look at much more than your IP and headers. Instead of building browser automation and challenge handling yourself, send those pages to the DINPROXY Web Unlocker.

The Web Unlocker fetches the page for you. It routes the request, renders JavaScript, retries and works around anti-bot checks, then returns the HTML, a screenshot or both. Its product page lists the checks it handles, including reCAPTCHA, Cloudflare Turnstile and Challenge, AWS WAF, GeeTest, DataDome, PerimeterX and Imperva. Each successful request uses one result from your balance, and failed requests are free.

It also slots into an existing scraper with little change. In proxy mode, point your HTTP client or scraper at unblocker.dinproxy.com:7780 with the same proxy user, and each GET request comes back as the finished page. Add @country=us to the username to fetch from a country. For HTTPS pages, have your scraper trust the DINPROXY Web Unlocker certificate or skip the certificate check for this proxy, as the proxy mode guide shows.

A practical split is to send most pages through residential proxies and route only the protected ones to the Web Unlocker. Our comparison of the Web Unlocker and residential proxies helps you decide which pages go where.

Frequently asked questions

For independent pages such as search results and listings, yes: with no session option, every request gets a new residential IP. For logins and multi-step flows, use a sticky session ID instead.

Each account can keep up to 100 connections open at the same time, residential and mobile together.

Send them to the Web Unlocker. It renders JavaScript, retries and works around anti-bot checks, and only successful requests use a result.

Keep reading