Python Requests Proxy: Setup, Auth, Rotation & Async (2026 Guide)

How to Use a Proxy with Python Requests (2026 Guide)

Python's requests library takes a proxy in one line. Getting it to work reliably, with authentication, rotation, retries and thousands of requests per hour, takes more than that. Most tutorials stop at the one line. This one goes all the way to a production-ready script.

Every snippet below was written for Python 3.10+ and requests 2.32. Copy, paste, replace the credentials, run.

Quick answer

import requests proxies = { "http": "http://USERNAME:PASSWORD@PROXY_HOST:PORT", "https": "http://USERNAME:PASSWORD@PROXY_HOST:PORT", } r = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=15) print(r.json()) # {'origin': '<proxy IP, not yours>'}

Three rules that prevent 80% of problems:

1. Set both keys, http and https. If you only set http, every HTTPS request skips the proxy and goes out from your real IP.

2. Use http:// as the proxy scheme for both keys, unless your provider explicitly gives you a TLS proxy endpoint. The key says which target URLs use the proxy. The value says how to talk to the proxy.

3. Always pass timeout. Without it, a dead proxy can hang your script forever.

1. Install Requests and check your real IP

requests is not part of the Python standard library. Install it first:

pip install requests

Before adding a proxy, see what websites see now:

import requests r = requests.get("https://httpbin.org/ip", timeout=15) print(r.json()) # {'origin': '203.0.113.42'} <- your real IP

Keep this IP in mind. Every time you change the proxy setup, run the same check and make sure the IP changed.

2. How the proxies dictionary actually works

This is the part most guides get wrong, so it's worth one minute.

proxies = { "http": "http://user:pass@proxy_host:port", # used for http:// URLs "https": "http://user:pass@proxy_host:port", # used for https:// URLs }

• The key ("http", "https") is the scheme of the URL you request.

• The value is the address of the proxy and the protocol used to reach it.

For an https:// target, Requests sends a CONNECT request to the proxy, which opens a tunnel. TLS then runs end-to-end between you and the website, so the proxy can't read your traffic. That's why http:// is the right scheme for the https key in almost every case.

| You write | What happens | |---|---| | "https": "http://host:port" | Correct. HTTP connection to the proxy, TLS tunnel to the site. | | "https": "https://host:port" | Requests tries TLS to the proxy itself. Works only if the provider offers a TLS proxy port, otherwise you get SSLError or ProxyError. | | Only "http" key set | HTTPS traffic bypasses the proxy. Your real IP leaks. | | "https": "host:port" (no scheme) | Requests silently assumes http://. It works, but write the scheme anyway so nobody has to guess. |

You can also route by host. Requests checks keys from most to least specific:

proxies = { "https://api.example.com": "http://user:pass@us_proxy:port", # this host only "https": "http://user:pass@default_proxy:port", # all other HTTPS "all": "http://user:pass@fallback_proxy:port", # anything else }

3. Proxy authentication

Username and password in the proxy URL

This is the standard method, and the one that works:

proxies = { "http": "http://USERNAME:PASSWORD@PROXY_HOST:PORT", "https": "http://USERNAME:PASSWORD@PROXY_HOST:PORT", }

Don't use auth= for the proxy. You'll see this in several popular tutorials:

# Wrong: this sends credentials to the website, not to the proxy requests.get(url, proxies=proxies, auth=HTTPBasicAuth("user", "pass"))

The proxy never receives the credentials, and you get 407 Proxy Authentication Required. (requests.auth.HTTPProxyAuth exists, but it only works for plain http:// targets, because it can't add the header to the CONNECT request used for HTTPS.)

Special characters in passwords

If your password contains @, :, /, # or %, the URL breaks and the proxy sees a garbage username. Encode it:

from urllib.parse import quote user = "sn_user" password = "p@ss:w0rd#1" proxy_url = f"http://{quote(user, safe='')}:{quote(password, safe='')}@PROXY_HOST:PORT" proxies = {"http": proxy_url, "https": proxy_url}

Keep credentials out of your code

Hardcoded passwords end up on GitHub. Read them from the environment:

import os from urllib.parse import quote PROXY_USER = os.environ["PROXY_USER"] PROXY_PASS = os.environ["PROXY_PASS"] PROXY_HOST = os.environ["PROXY_HOST"] # e.g. the host:port from your dashboard PROXY_URL = f"http://{quote(PROXY_USER, safe='')}:{quote(PROXY_PASS, safe='')}@{PROXY_HOST}" PROXIES = {"http": PROXY_URL, "https": PROXY_URL}

IP whitelisting instead of passwords

Most residential providers, SimplyNode included, also let you whitelist your server's IP. Then you don't send credentials at all:

proxies = {"http": "http://PROXY_HOST:PORT", "https": "http://PROXY_HOST:PORT"}

Whitelisting is handy for servers with a fixed IP. Username and password is better for laptops, CI runners and anything behind a changing IP. We compared both in IP whitelisting vs username:password.

4. Use a Session (and avoid the env-var trap)

A Session reuses TCP connections and keeps cookies between requests. That's faster, and it's what you want for anything with a login:

import requests session = requests.Session() session.proxies.update(PROXIES) session.headers.update({"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36"}) r = session.get("https://httpbin.org/ip", timeout=15) print(r.json())

The trap: if HTTP_PROXY or HTTPS_PROXY is set in your environment (common on corporate machines, Docker images and CI), it overrides session.proxies. Your code looks right, but traffic goes through a different proxy, or none.

Two ways to fix it:

# Option 1: ignore environment proxy settings for this session session.trust_env = False # Option 2: pass proxies on each request; per-request values always win session.get(url, proxies=PROXIES, timeout=15)

Note that trust_env = False also stops Requests from reading .netrc and REQUESTS_CA_BUNDLE. Usually that's fine for scrapers.

5. Set a proxy with environment variables

If you don't want to touch the code at all, Requests picks up standard proxy variables automatically.

macOS / Linux:

export HTTP_PROXY="http://USERNAME:PASSWORD@PROXY_HOST:PORT" export HTTPS_PROXY="http://USERNAME:PASSWORD@PROXY_HOST:PORT" export NO_PROXY="localhost,127.0.0.1,.internal.company.com" python scraper.py

Windows PowerShell:

$env:HTTP_PROXY = "http://USERNAME:PASSWORD@PROXY_HOST:PORT" $env:HTTPS_PROXY = "http://USERNAME:PASSWORD@PROXY_HOST:PORT" python scraper.py

Windows cmd:

set HTTP_PROXY=http://USERNAME:PASSWORD@PROXY_HOST:PORT set HTTPS_PROXY=http://USERNAME:PASSWORD@PROXY_HOST:PORT python scraper.py

NO_PROXY lists hosts that should connect directly, like your own API or a local database.

To skip the proxy for a single request while env vars are set:

requests.get("http://localhost:8000/health", proxies={"http": None, "https": None}, timeout=5)

6. Use a SOCKS5 proxy with Requests

SOCKS support needs one extra package:

pip install "requests[socks]"

Then change the scheme:

proxies = { "http": "socks5h://USERNAME:PASSWORD@PROXY_HOST:PORT", "https": "socks5h://USERNAME:PASSWORD@PROXY_HOST:PORT", } r = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=15)

socks5 vs socks5h: with socks5://, your machine resolves the domain name and sends only the IP to the proxy. Your local DNS server sees every site you visit, and geo-targeted sites may get a DNS answer for your location instead of the proxy's. With socks5h://, the proxy resolves DNS. For scraping and privacy, use socks5h.

If you get InvalidSchema: Missing dependencies for SOCKS support, the pysocks package isn't installed in the environment your script runs in.

More on the protocol: What is a SOCKS5 proxy and how it works.

7. Rotating proxies in Python Requests

Rotation spreads requests across many IPs so no single IP hits a site often enough to get rate-limited. There are three ways to do it, and the right one depends on your task.

| Model | How it works | Best for | |---|---|---| | Rotating gateway | One endpoint. The provider assigns a new IP per connection. | Stateless scraping: SERPs, product pages, prices | | List of sticky endpoints | Many endpoints, each holding one IP for a set time | Logins, carts, multi-step flows, account management | | Your own pool | You rotate through IPs in code | Datacenter IP lists, mixed providers |

Model 1: Rotating gateway (simplest)

With a residential provider, you usually don't rotate in code at all. In the SimplyNode dashboard, set rotation to New IP on each request, copy the endpoint, and every new connection exits from a different residential IP:

import requests for i in range(5): r = requests.get("https://httpbin.org/ip", proxies=PROXIES, timeout=15) print(i, r.json()["origin"])

Note that this uses requests.get, not a Session. Connection reuse can defeat rotation. Most gateways pick the exit IP when a connection opens. A Session keeps the connection (and the HTTPS tunnel) alive, so several requests to the same site can leave from the same IP. If you need a new IP on every request, either use requests.get or add Connection: close:

session.get(url, headers={"Connection": "close"}, timeout=15)

Model 2: Sticky endpoints for logins and multi-step flows

Some tasks need the same IP for a while: log in, browse, add to cart, check out. If the IP changes mid-flow, the site sees a session hijack and logs you out.

In SimplyNode you set rotation to sticky or by time (from 90 seconds up to 6 hours) and export a proxy list. Each line in the list is a separate endpoint (a different port), and each endpoint keeps its own IP. So one endpoint = one identity:

import requests # proxies.txt, exported from the dashboard, one endpoint per line: # http://USERNAME:PASSWORD@PROXY_HOST:PORT_1 # http://USERNAME:PASSWORD@PROXY_HOST:PORT_2 with open("proxies.txt") as f: endpoints = [line.strip() for line in f if line.strip()] def session_for(account_id: int) -> requests.Session: """Pin each account to its own endpoint, so its IP stays stable.""" proxy = endpoints[account_id % len(endpoints)] s = requests.Session() s.trust_env = False s.proxies.update({"http": proxy, "https": proxy}) return s s = session_for(account_id=7) s.get("https://example.com/login", timeout=20) s.get("https://example.com/dashboard", timeout=20) # same IP as the login

Read more: Rotating vs sticky residential sessions.

Model 3: Rotate your own pool, with a cooldown

If you manage a list of IPs yourself, don't just call random.choice. A proxy that just got a 403 will get another one. Track failures and bench bad proxies for a while:

import random import time import requests class ProxyPool: def __init__(self, proxy_urls, cooldown=300): self.proxies = list(proxy_urls) self.cooldown = cooldown self.benched_until = {} # proxy -> unix time it can be used again def get(self): now = time.time() healthy = [p for p in self.proxies if self.benched_until.get(p, 0) <= now] return random.choice(healthy or self.proxies) def bench(self, proxy): self.benched_until[proxy] = time.time() + self.cooldown pool = ProxyPool(endpoints) def fetch(url): proxy = pool.get() try: r = requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=(5, 20)) except requests.exceptions.RequestException: pool.bench(proxy) raise if r.status_code in (403, 407, 429): pool.bench(proxy) return r

8. Timeouts, retries and backoff

Proxies add a network hop, and residential IPs are real home connections. Some requests will fail. Plan for it.

Use a tuple timeout

requests.get(url, proxies=PROXIES, timeout=(5, 30))

The first number is the connect timeout (time to reach the proxy). The second is the read timeout (time to wait for the site's response). A short connect timeout fails fast on a dead proxy. A longer read timeout lets slow pages load.

Automatic retries with HTTPAdapter

import requests from requests.adapters import HTTPAdapter from urllib3.util.retry import Retry retry = Retry( total=4, connect=3, backoff_factor=1, # waits 1s, 2s, 4s between retries status_forcelist=[429, 500, 502, 503, 504], allowed_methods=["GET", "HEAD"], respect_retry_after_header=True, ) session = requests.Session() session.trust_env = False session.proxies.update(PROXIES) session.mount("http://", HTTPAdapter(max_retries=retry)) session.mount("https://", HTTPAdapter(max_retries=retry)) r = session.get("https://httpbin.org/status/200", timeout=(5, 30))

This retries through the same proxy endpoint. With a rotating gateway, each retry on a new connection usually gets a new IP, which is exactly what you want after a 429.

Retry with a different proxy

When a site blocks one IP, retrying through the same IP is pointless. This loop switches the proxy on every failed attempt:

import random import time import requests RETRYABLE_STATUS = {403, 429, 500, 502, 503, 504} def fetch_with_rotation(url, proxy_list, attempts=4): last_error = None for attempt in range(attempts): proxy = random.choice(proxy_list) try: r = requests.get( url, proxies={"http": proxy, "https": proxy}, headers=HEADERS, timeout=(5, 30), ) if r.status_code not in RETRYABLE_STATUS: return r last_error = f"HTTP {r.status_code}" except (requests.exceptions.ConnectionError, requests.exceptions.Timeout) as e: last_error = repr(e) # ProxyError and SSLError are ConnectionError subclasses time.sleep(2 ** attempt + random.random()) # exponential backoff with jitter raise RuntimeError(f"{url} failed after {attempts} attempts: {last_error}")

9. Scale up: threads and async

One request at a time through a residential proxy runs at maybe one or two pages per second. Concurrency gets you to hundreds.

Threads with Requests

requests.Session is not guaranteed to be thread-safe. Give each thread its own:

import threading from concurrent.futures import ThreadPoolExecutor, as_completed import requests _local = threading.local() def get_session(): if not hasattr(_local, "session"): s = requests.Session() s.trust_env = False s.proxies.update(PROXIES) s.headers.update(HEADERS) _local.session = s return _local.session def fetch(url): r = get_session().get(url, timeout=(5, 30)) return url, r.status_code, len(r.content) urls = [f"https://httpbin.org/anything/{i}" for i in range(100)] with ThreadPoolExecutor(max_workers=20) as pool: futures = [pool.submit(fetch, u) for u in urls] for f in as_completed(futures): try: print(f.result()) except Exception as e: print("failed:", e)

SimplyNode allows unlimited concurrent sessions, so the limit is your machine and the target site's tolerance, not the proxy plan.

Async with httpx

httpx has a near-identical API to Requests and supports asyncio. Install it with pip install httpx (add httpx[socks] for SOCKS5). Since httpx 0.26 the argument is proxy= (singular):

import asyncio import httpx async def fetch(client, url, sem): async with sem: r = await client.get(url) return url, r.status_code async def main(urls): sem = asyncio.Semaphore(50) # max 50 requests in flight async with httpx.AsyncClient(proxy=PROXY_URL, headers=HEADERS, timeout=30) as client: results = await asyncio.gather(*(fetch(client, u, sem) for u in urls), return_exceptions=True) for res in results: print(res) asyncio.run(main([f"https://httpbin.org/anything/{i}" for i in range(200)]))

Async with aiohttp

import asyncio import aiohttp async def main(): auth = aiohttp.BasicAuth(PROXY_USER, PROXY_PASS) async with aiohttp.ClientSession() as session: async with session.get( "https://httpbin.org/ip", proxy=f"http://{PROXY_HOST}", proxy_auth=auth, timeout=aiohttp.ClientTimeout(total=30), ) as r: print(await r.json()) asyncio.run(main())

aiohttp supports HTTP proxies out of the box. For SOCKS5, add the aiohttp-socks package.

10. Geo-targeting: scrape as a user from a specific country

Prices, search results, ads and stock levels change by location. To see what a shopper in Germany sees, you need a German IP.

With SimplyNode, you pick the country, region, city or ISP in the dashboard (185+ countries, 700+ cities) and get endpoints for that location. A clean pattern is one endpoint per market:

import os import requests PROXIES_BY_COUNTRY = { "us": os.environ["PROXY_US"], "de": os.environ["PROXY_DE"], "jp": os.environ["PROXY_JP"], } def get_from(country, url): proxy = PROXIES_BY_COUNTRY[country] return requests.get(url, proxies={"http": proxy, "https": proxy}, headers=HEADERS, timeout=(5, 30)) info = get_from("de", "https://ipinfo.io/json").json() print(info["country"], info.get("city")) # DE Berlin

Run the ipinfo.io check once per endpoint before a big job. It takes a second and saves you from collecting a day of data from the wrong country. More in Geographic targeting in web scraping.

11. Proxies aren't enough: headers and fingerprints

A clean residential IP gets you through IP-based blocking. It doesn't hide that you're a Python script.

1. Change the User-Agent. By default Requests announces itself as python-requests/2.32.x. Many sites block that string outright.

HEADERS = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en-US,en;q=0.9", }

Match Accept-Language to your proxy's country. A German IP with en-US headers is a small inconsistency, but anti-bot systems add up small inconsistencies.

2. Know that Requests has a TLS fingerprint. Cloudflare, DataDome and Akamai look at how your client negotiates TLS (the JA3/JA4 fingerprint). Requests' fingerprint doesn't look like Chrome no matter what User-Agent you send. If you get 403s through a good residential proxy with good headers, this is the likely cause. curl_cffi keeps a Requests-style API and impersonates a real browser:

# pip install curl_cffi from curl_cffi import requests as cffi_requests r = cffi_requests.get( "https://example.com", impersonate="chrome", proxies=PROXIES, timeout=30, )

Deep dive: Antibot & fingerprinting: how detection works.

12. Troubleshooting: Python Requests proxy not working

| Error or symptom | Likely cause | Fix | |---|---|---| | Response shows your real IP | Only the http key is set, or env vars override session.proxies | Set both keys. Set session.trust_env = False. | | ProxyError: Tunnel connection failed: 407 Proxy Authentication Required | Wrong credentials, unencoded special characters, or credentials passed via auth= | Put credentials in the proxy URL and quote() them. Check that your IP is whitelisted if you use IP auth. See our 407 guide. | | ProxyError: Cannot connect to proxy / Max retries exceeded | Wrong host or port, firewall blocks outbound port, proxy is down | Test the same endpoint with curl -x (see cURL with proxy). Check the port in the dashboard. | | SSLError: wrong version number | https:// used as the proxy scheme for a plain HTTP proxy | Use http:// in the proxy URL. | | SSLError: CERTIFICATE_VERIFY_FAILED | Corporate MITM proxy, or outdated CA bundle | pip install -U certifi. Pass verify="/path/to/ca.pem". Use verify=False only for local debugging, never in production. | | InvalidSchema: Missing dependencies for SOCKS support | pysocks not installed | pip install "requests[socks]" in the same virtualenv. | | ConnectTimeout / ReadTimeout | Slow or offline exit IP | Use a tuple timeout like (5, 30) and retry through another IP. | | 403 Forbidden with a good residential IP | Headers or TLS fingerprint give away the bot | Send realistic headers. Try curl_cffi with impersonate. | | 429 Too Many Requests | Too many requests per IP | Rotate per request, add backoff, lower concurrency per domain. | | IP doesn't change on a rotating endpoint | Keep-alive reuses the connection | Use requests.get or the Connection: close header. |

13. Which proxy type should you use with Python?

| Type | Detection risk | Speed | Cost | Use it for | |---|---|---|---|---| | Free public proxies | Very high (most are already blacklisted) | Unpredictable | $0 | Learning only. Many log or alter your traffic. | | Datacenter | High on protected sites | Fastest | Low | Sites without anti-bot protection, internal tools | | Residential | Low | Fast (SimplyNode: <1s typical response) | From $2.25/GB | Most scraping: e-commerce, SERPs, travel, real estate | | Mobile (4G/5G) | Lowest | Good (<1.5s typical) | Higher | Social media, apps, the strictest anti-bot targets |

For most Python scraping projects, rotating residential proxies are the right default. They come from real home ISP connections, so sites can't block them without blocking real customers. Compare all options in Datacenter vs residential proxies.

14. Production-ready template

Everything above, in one file you can drop into a project:

"""scraper.py: Requests + residential proxies, with retries, rotation and threads.""" import os import random import threading import time from concurrent.futures import ThreadPoolExecutor, as_completed from urllib.parse import quote import requests from requests.adapters import HTTPAdapter from urllib3.util.retry import Retry PROXY_USER = os.environ["PROXY_USER"] PROXY_PASS = os.environ["PROXY_PASS"] PROXY_HOST = os.environ["PROXY_HOST"] # host:port of your rotating endpoint PROXY_URL = f"http://{quote(PROXY_USER, safe='')}:{quote(PROXY_PASS, safe='')}@{PROXY_HOST}" PROXIES = {"http": PROXY_URL, "https": PROXY_URL} HEADERS = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en-US,en;q=0.9", "Connection": "close", # new connection, and new IP, on every request } RETRY = Retry( total=3, backoff_factor=1, status_forcelist=[429, 500, 502, 503, 504], allowed_methods=["GET", "HEAD"], ) _local = threading.local() def get_session() -> requests.Session: if not hasattr(_local, "session"): s = requests.Session() s.trust_env = False s.proxies.update(PROXIES) s.headers.update(HEADERS) adapter = HTTPAdapter(max_retries=RETRY) s.mount("http://", adapter) s.mount("https://", adapter) _local.session = s return _local.session def fetch(url: str, attempts: int = 3) -> requests.Response: for attempt in range(attempts): try: r = get_session().get(url, timeout=(5, 30)) if r.status_code != 403: return r except (requests.exceptions.ConnectionError, requests.exceptions.Timeout): pass time.sleep(2 ** attempt + random.random()) raise RuntimeError(f"Giving up on {url}") def main(urls, workers=20): with ThreadPoolExecutor(max_workers=workers) as pool: futures = {pool.submit(fetch, u): u for u in urls} for f in as_completed(futures): url = futures[f] try: r = f.result() print(r.status_code, url) except Exception as e: print("ERROR", url, e) if __name__ == "__main__": main([f"https://httpbin.org/anything/{i}" for i in range(50)])

Run it:

export PROXY_USER="your_username" export PROXY_PASS="your_password" export PROXY_HOST="host:port from your SimplyNode dashboard" python scraper.py

Get residential proxies for your Python scripts

Everything in this guide works with any HTTP or SOCKS5 proxy. If you need one, SimplyNode gives you:

• 8M+ residential and 1.4M+ mobile IPs in 185+ countries, with city and ISP targeting

• HTTP, HTTPS and SOCKS5 on the same plan

• Rotation per request, by time (90s to 6h), or sticky

• Unlimited concurrent sessions, which matters once you add threads or async

• Pay-as-you-go from $2.25/GB, no subscription, and traffic that doesn't expire

Get residential proxies →

FAQ

How do I use a proxy with Python Requests?

Create a dictionary with http and https keys pointing to your proxy URL, for example {"http": "http://user:pass@host:port", "https": "http://user:pass@host:port"}, and pass it as proxies= to requests.get() or set it on session.proxies. Always add a timeout.

Why does Requests still show my real IP through the proxy?

Usually because only the http key is set, so HTTPS requests go direct. The other common cause is HTTP_PROXY/HTTPS_PROXY environment variables overriding session.proxies. Set both keys and use session.trust_env = False.

How do I pass a username and password to a proxy in Requests?

Put them in the proxy URL: http://username:password@host:port. URL-encode them with urllib.parse.quote(value, safe="") if they contain special characters. Don't use the auth= parameter: it authenticates you to the website, not to the proxy.

Should the https key use http:// or https://?

Use http:// unless your provider gives you a TLS-enabled proxy endpoint. The key is the scheme of the target site. The value's scheme is how you connect to the proxy. HTTPS traffic is still encrypted end-to-end through a CONNECT tunnel.

Does Python Requests support SOCKS5 proxies?

Yes. Install requests[socks] and use socks5:// or socks5h:// in the proxy URL. socks5h resolves DNS on the proxy side, which prevents DNS leaks and gives correct geo-located answers.

How do I rotate proxies in Python Requests?

The easiest way is a rotating residential gateway: one endpoint that assigns a new IP per connection. For your own list, pick a proxy per request, bench proxies that return 403/429 for a few minutes, and retry failed requests through a different proxy with exponential backoff.

How do I disable the proxy for one request?

Pass proxies={"http": None, "https": None} for that call, or add the host to the NO_PROXY environment variable.

Is session.proxies thread-safe?

requests.Session isn't guaranteed to be thread-safe. In multithreaded scrapers, create one session per thread with threading.local(), or switch to httpx.AsyncClient for async concurrency.

How do I set a proxy in httpx or aiohttp?

In httpx, use httpx.Client(proxy="http://user:pass@host:port") or httpx.AsyncClient(proxy=...). In aiohttp, pass proxy="http://host:port" and proxy_auth=aiohttp.BasicAuth(user, password) to each request.

Can I use free proxies with Python Requests?

Technically yes, practically no. Public free proxies are slow, mostly blacklisted, and can read or modify unencrypted traffic. They're fine for learning the syntax. For real data collection, use paid residential or mobile proxies.

Related reading

• How to rotate proxies for web scraping

• Mastering cURL with a proxy

• How to fix the 407 Proxy Authentication error

• Rotating vs sticky residential sessions

• Proxies for web scraping