Scrapers & Automation

Wire Proxio into Scrapy, Selenium, Puppeteer, and Playwright with minimal complete examples, each with proxy authentication and a verify step.

Scraping and browser-automation tools each take a proxy differently: some as a per-request setting, some as a launch flag paired with a separate authentication call. This guide covers Scrapy, Selenium, Puppeteer, and Playwright.

Targeting and rotation are already handled

For per-request geo or session control in any of these tools, encode it directly in the username. See Targeting & Username Syntax for the full grammar. You don't need extra rotation logic either: a plain username with no session segments already gets a new IP on every request.

Scrapy

import scrapy


class IpInfoSpider(scrapy.Spider):
    name = "ipinfo"
    start_urls = ["https://ipinfo.io/json"]

    def start_requests(self):
        proxy = "http://USERNAME:[email protected]:16666"
        for url in self.start_urls:
            yield scrapy.Request(url, meta={"proxy": proxy}, callback=self.parse)

    def parse(self, response):
        print(response.text)

Scrapy's built-in HttpProxyMiddleware is enabled by default and reads request.meta["proxy"] on every request, splitting the embedded USERNAME:PASSWORD into a Proxy-Authorization header for you, with nothing else to configure.

To set one proxy for an entire crawl without touching the spider, use the standard environment variables instead. HttpProxyMiddleware falls back to them whenever a request has no meta["proxy"]:

export HTTPS_PROXY="http://USERNAME:[email protected]:16666"
export HTTP_PROXY="http://USERNAME:[email protected]:16666"
scrapy crawl ipinfo

Verify: scrapy crawl ipinfo. The printed body is the JSON from ipinfo.io with the proxy's exit IP.

Selenium

Chrome's own --proxy-server launch flag has no field for a username or password. The browser just opens a login popup that Selenium can't fill in. selenium-wire injects the credentials for you:

from seleniumwire import webdriver

proxy_url = "http://USERNAME:[email protected]:16666"

options = {
    "proxy": {
        "http": proxy_url,
        "https": proxy_url,
        "no_proxy": "localhost,127.0.0.1",
    }
}

driver = webdriver.Chrome(seleniumwire_options=options)
driver.get("https://ipinfo.io/json")
print(driver.page_source)
driver.quit()

The clean alternative: IP authentication

If the machine running Selenium has a static IP, allowlist it instead (see IP Authentication) and drop back to a plain --proxy-server flag with no credentials at all.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--proxy-server=geo.proxio.cc:16666")

driver = webdriver.Chrome(options=options)
driver.get("https://ipinfo.io/json")
print(driver.page_source)
driver.quit()

Verify: either script. The printed page source is the ipinfo.io JSON body with the proxy's exit IP.

Puppeteer

import puppeteer from 'puppeteer'

const browser = await puppeteer.launch({
  args: ['--proxy-server=geo.proxio.cc:16666'],
})

const page = await browser.newPage()
await page.authenticate({ username: 'USERNAME', password: 'PASSWORD' })
await page.goto('https://ipinfo.io/json')
console.log(await page.evaluate(() => document.body.innerText))

await browser.close()

page.authenticate() supplies the credentials that --proxy-server alone can't carry, and it must be called before page.goto().

Verify: run the script. The logged text is the ipinfo.io JSON body.

Playwright

import { chromium } from 'playwright'

const browser = await chromium.launch({
  proxy: {
    server: 'http://geo.proxio.cc:16666',
    username: 'USERNAME',
    password: 'PASSWORD',
  },
})

const page = await browser.newPage()
await page.goto('https://ipinfo.io/json')
console.log(await page.textContent('body'))

await browser.close()

Playwright takes the proxy and its credentials together in launch(), with no separate authentication call needed, and the same shape works for Chromium, Firefox, and WebKit.