How to Rotate Proxies for Web Scraping in 2026

Why websites block scrapers in the first place

To rotate well, it helps to know what you are rotating away from. Most anti-bot systems combine several signals:

• IP reputation. Datacenter ranges, known proxy subnets, and addresses with a history of abuse start with a low trust score.

• Request rate per IP. Too many requests in a short window from one address triggers rate limits, CAPTCHAs, or bans.

• Behavioral patterns. Hitting pages in perfect alphabetical order, at exact intervals, with no pauses, does not look like a person.

• Fingerprints. Headers, TLS handshake details, and browser properties are compared against what a real browser would send.

• Session consistency. A visitor who changes country or device halfway through a login looks suspicious.

Rotation addresses the first two directly. It helps a little with the third. It does nothing for fingerprints on its own. That is why “I added rotating proxies and I still get blocked” is such a common complaint.

What proxy rotation actually does

When you rotate, each request (or group of requests) leaves from a different IP address in the provider’s pool. No single address accumulates enough traffic to cross a rate limit, and a ban on one IP does not stop the job.

The trade-off is continuity. Many sites tie state to an IP: login sessions, carts, pagination tokens, rate-limit counters. Switch IPs at the wrong moment and you break that state. So the practical question is not whether to rotate, but where in your workflow it is safe to.

The three rotation strategies

1. New IP on every request

Each request goes out through a different IP.

Use it for: product pages, search result pages, public listings, any job where pages are independent and order does not matter.

Watch out for: anything stateful. A site that sets a cookie on request one and expects it on request two from the same IP will reject the second.

2. Timed rotation

The IP stays the same for a fixed window, then changes. For scheduled jobs, you can also simply start each run on a fresh IP.

Use it for: price and change monitoring on a cron schedule, recurring checks of the same site, jobs where you want some stability within a run but a fresh identity between runs.

Watch out for: window length. Too long and you recreate the single-IP problem. Too short and you add complexity for no gain. Match it to how long one logical task takes.

3. Sticky sessions

A session ID pins you to one IP for a set duration. Same session ID returns the same IP. A new session ID returns a new one.

Use it for: logins, carts and checkout flows, paginated results that depend on session state, multi-step QA and regression tests.

Watch out for: TTL. Set it slightly longer than the flow takes to finish, not much longer. A session that outlives its task is just a single IP with extra steps.

Quick decision table

A rule that covers most cases: if request B depends on request A, use a sticky session. If it does not, rotate every time.

Setting it up with SimplyNode

SimplyNode uses a single endpoint, ip.simplynode.io:9003, and the IP pool sits behind it. Rotation is controlled through your proxy credentials. For a sticky session, you add a session ID and TTL in seconds using the -session-{id}-ttl-{seconds} pattern. GEO targeting and other settings are configured in the proxy list generator in your dashboard.

Python with requests

python

import requests
import uuid

HOST = "ip.simplynode.io:9003"
USER = "your_username"
PASSWORD = "your_password"

def make_proxies(sticky=False, ttl=600):
    user = USER
    if sticky:
        user += f"-session-{uuid.uuid4().hex[:8]}-ttl-{ttl}"
    url = f"http://{user}:{PASSWORD}@{HOST}"
    return {"http": url, "https": url}

# Independent requests: a new IP each time
r = requests.get("https://example.com/product/1",
                 proxies=make_proxies(), timeout=30)

# Multi-step flow: one IP for 10 minutes
flow = make_proxies(sticky=True, ttl=600)
s = requests.Session()
s.get("https://example.com/login", proxies=flow, timeout=30)
s.get("https://example.com/account", proxies=flow, timeout=30)

Build the sticky proxy dict once per flow and reuse it. Calling make_proxies(sticky=True) for every request generates a new session ID each time, which defeats the purpose.

Playwright

For browser-based scraping, pass the proxy at launch:

python

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(proxy={
        "server": "http://ip.simplynode.io:9003",
        "username": "your_username-session-abc123-ttl-600",
        "password": "your_password",
    })
    page = browser.new_page()
    page.goto("https://example.com")
    browser.close()

One browser launch equals one proxy configuration here. If you want a new IP per page, either launch a new context with its own proxy or use rotating credentials without a session ID.

Scrapy

In Scrapy, set the proxy per request through meta:

python

import scrapy

class ProductSpider(scrapy.Spider):
    name = "products"

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                meta={"proxy": "http://your_username:your_password@ip.simplynode.io:9003"},
            )

For sticky flows, add the session suffix to the username and keep it constant across the requests of that flow.

Build retry logic that does not waste traffic

Rotating harder is not a fix for every failure. Match the response to the error:

• 429 Too Many Requests: you are going too fast. Back off first, then retry. Retrying instantly on a new IP just spends bandwidth.

• 403 Forbidden: the target rejected your fingerprint or IP. A new IP may help, but check your headers too.

• CAPTCHA page (often a 200 response): detect it by content, not status code. Slow down and randomize timing.

• Timeouts and connection errors: retry a small number of times on a fresh IP, then log and move on.

A simple pattern:

python

import time, random

def fetch(url, max_retries=3):
    for attempt in range(max_retries):
        r = requests.get(url, proxies=make_proxies(), timeout=30)
        if r.status_code == 200 and "captcha" not in r.text.lower():
            return r
        time.sleep((2 ** attempt) + random.uniform(0, 1))
    return None

Cap retries per URL and record the failures. A URL that fails three times on three different IPs usually has a problem that more rotation will not solve.

Pace your requests

Even with a huge pool, behavior counts. A few habits that help:

• Add randomized delays between requests instead of a fixed interval.

• Limit concurrency per target domain. Many parallel workers on one site can look like an attack regardless of how many IPs you use.

• Spread requests across pages and sections instead of crawling in strict sequence.

• Respect the load a site can reasonably handle, and check its terms and robots.txt for what is permitted.

Match geography to your data

Prices, availability, language, and even page structure change by country. If you scrape a US retailer through an IP in another region, you may get the wrong catalog or currency. When you need region-specific data, pick GEO targeting at country, region, or city level in the generator, and keep it consistent within a sticky session. A user who appears to jump countries mid-session is an easy flag.

Don’t forget the rest of your fingerprint

Rotating IPs while sending identical, unusual headers on every request is easy to spot. Send realistic headers, keep the User-Agent consistent with the rest of your request profile, and use a real browser (Playwright, for example) for sites that check the TLS handshake or run JavaScript challenges. IP rotation is one layer. It works best alongside the others.

Residential or mobile

Residential IPs come from real home ISPs. They fit most scraping, monitoring, and automation targets.

Mobile IPs come from carrier networks. They are worth using on platforms with aggressive detection where residential addresses still get flagged.

Start with residential. Test on the target that gives you trouble, watch the success rate, and switch to mobile only for the targets that need it. This keeps cost down while covering the hard cases.

Control your traffic spend

On a pay-per-GB model, wasted requests cost money. Some ways to keep usage efficient:

• Block images, fonts, and media in browser-based scraping when you only need text or data.

• Avoid retrying URLs that fail for reasons rotation cannot fix.

• Split traffic into separate proxy profiles per project or workflow, so you can see which jobs consume the most.

• Start small. Test on 1 GB before scaling a job, and look at the success rate first.

SimplyNode’s purchased traffic does not expire, so bursty or irregular schedules, such as monitoring jobs that spike in short windows, do not lose unused bandwidth at month end.

Common mistakes

• Rotating in the middle of a session. Breaks logins and carts. Use a sticky session.

• Setting TTL far longer than the task. You end up with one IP doing too much.

• Retrying instantly on a new IP. Back off first.

• Ignoring geo. Wrong-country IPs return wrong-market data.

• Too much concurrency on one domain. Pool size does not remove the need for pacing.

• Treating the proxy as the whole solution. Headers, timing, and browser fingerprint matter too.

• Not measuring. Without a per-target success rate, you are guessing.

FAQ

How often should I rotate proxies for web scraping?
It depends on the task. Rotate on every request for independent pages, on a timer for scheduled jobs, and hold one IP with a sticky session for multi-step flows.

What is the difference between rotating proxies and sticky sessions?
Rotating proxies give you a different IP per request. Sticky sessions keep the same IP for a set duration, so a multi-step flow does not switch addresses halfway.

Does rotating proxies prevent bans?
It lowers the risk but does not guarantee anything. Request pacing, realistic headers, and sensible retry logic also decide whether you get blocked.

Why am I still blocked with rotating proxies?
Common causes are requests that are too fast or too regular, an unrealistic header or TLS fingerprint, or a target that flags the IP type. Check pacing and fingerprint first, and try mobile IPs if residential keeps failing.

Should I use residential or mobile proxies for scraping?
Residential works for most targets. Mobile is the fallback for platforms with strict detection.

How do I keep the same IP for a whole login flow?
Use a sticky session with a fixed session ID and a TTL that covers the flow. The same session ID returns the same IP.

Does unused proxy traffic expire?
On SimplyNode, no. Purchased traffic has no bandwidth expiration.