The 7 best web scraping Dedicated and Shared proxy providers. Shared vs. Dedicated Proxies
The 7 best web scraping Dedicated and Shared proxy providers. Shared vs. Dedicated Proxies
In the most simple terms, the choice between shared and dedicated proxies boils down to a simple question: What matters more to you, lowering the costs or raising the quality?
Shared proxies will always be cheaper than entirely private ones, as you’ll see in the provider list. While the price is undoubtedly an advantage, you’ll also have to deal with these drawbacks:
- You have higher chances of getting blocked due to other clients who use the same IPs as you. Sites like Amazon or Google are popular targets, so there’s a good chance that another user has already sent too many requests to them and got the IP blocked.
- The scraper will stand out more because the IP will generally be much more active due to the requests coming from multiple customers. Unusual activity often results in being sent to CAPTCHA pages or outright banned.
- You can expect lower speed since you’re sharing the server’s bandwidth with other users. Moreover, you won’t always have a steady pace since it goes down the more people use the proxy.
While shared proxies have some heavy disadvantages, it all depends on how serious you are about web scraping and the type of data you wish to collect. Popular scraping websites will block you more often, especially if you need large quantities of data. Small projects, however, or those that target less popular websites may not experience significant problems. So, shared proxies work well for smaller jobs and beginners.
Dedicated proxies are in many ways the opposite. They will always have higher prices, but the disadvantages below turn into advantages here:
- As only you have access to the IP, there’s no risk that the websites you target already associated the IP with a bot.
- As long as you make sure the web scraper doesn’t attract attention to itself and imitates regular visitors, it’s unlikely that you’ll get blocked.
- As long as the proxy isn’t a continent away from your location, you can expect good speeds and little to no fluctuations.
If shared proxies are a good way to gain web scraping experience, private IPs are much more likely to get you all the data you want. You’ll have higher operational costs but also much higher efficiency.
Apify proxy. Connection settings
To connect to the Apify Proxy, you use the. This means that you need to configure your HTTP client to use the proxy server atproxy.apify.com:8000and provide it with your Apify Proxy password and the other parameters described below.
The full connection string has the following format:
| Port | 8000 |
| Username | Specifies the proxy parameters such as groups,and location. Seebelow for details. Note : this is not your Apify username. |
| Password | Proxy password. Your password is displayed on the Proxy page in Apify Console. In Apify actors , it is passed as theAPIFY_PROXY_PASSWORDenvironment variable. See the environment variables docs for more details. |
WARNING: All usage of Apify Proxy with your password is charged towards your account. Do not share the password with untrusted parties or use it from insecure networks – the password is sent unencrypted due to the HTTP protocol's.
Username parameters
Theusernamefield enables you to pass parameters like,and country for your proxy connection.
For example, if you're usingand want to use thenew_job_123session using theSHADER
groups | Required | Set proxied requests to use servers from the selected groups: - groups-orautowhen using datacenter proxies.- groups-RESIDENTIALwhen using residential proxies.- groups-GOOGLE_SERPwhen using Google SERP proxies. |
|---|---|---|
session | Optional | If specified to The session string can only contain numbers (0-9), letters (a-z or A-Z), dot (.), underscore (_), a tilde (~). The maximum length is 50 characters. Session management may work differently for residential and SERP proxies. Check relevant documentations for more details. |
Web Scraper API documentation
Get to know how Web Scraper API works and integrate it into your app. Examples are provided in Curl, Javascript and Python.
Data scraping and parsing endpoint.
Query Parameters
| api_key (string, required) | Web Scraper API key | {"api_key":"0de32912321"} | default=null |
| country_code (str, optional) | Proxy country code (geolocation) | {"country_code":"fr"} | default=null, options - us, gb, de, fr, cn, jp. |
| render_js (bool, optional) | Render JS on page | {"render_js":"true"} | default=false |
| return_json (bool, optional) | Return HTML if flagged "false" | {"return_json": "false"} | default=true |
| headers (JSON, optional) | Custom headers | { "headers": {"user-agent": "Example user agent", "accept": "text/html,*/*"} } | default our headers |
| language (str, optional) | Language | {"language":"en-US"} | default=en-US, options - en-US, en-CA, es-ES, fr-CA. |
Returns
| Status code | Description | Example |
|---|---|---|
| 200 (Success) | Request successful. Returns JSON with headers and html fields | {"headers":{}, 'html':""} |
| 401 (Unauthorized) | API key is missing or wrong | {'error':'API key is missing or wrong'} |
| 422 (Unprocessable Entity) | Error in query parameters | {'error':'Wrong query'} |
| 504 (Timeout) | Site returned timeout after 3 attempts to reach it | {'error':'Timeout'} |
Country codes
If you want to define the geolocation for your request, you may set the country_code (string) parameter with one country code at the creation of the request.
Get to know how Web Scraper API works and integrate it into your app. Our documentation provides a comprehensive guide to help you understand the API's functionality and usage.
Examples
Here are some examples of how to use Web Scraper API in different programming languages:
Curl
curl -X GET \
https://api.example.com/scrape \
-H 'Content-Type: application/json' \
-d '{
"url": "https://www.example.com",
"country_code": "US"
}'
JavaScript
const axios = require('axios');
axios.get('https://api.example.com/scrape', {
params: {
url: 'https://www.example.com',
country_code: 'US'
}
})
.then(response => console.log(response.data))
.catch(error => console.error(error));
Python
import requests
response = requests.get('https://api.example.com/scrape', params={
'url': 'https://www.example.com',
'country_code': 'US'
})
print(response.json())
Geolocation
If you want to define the geolocation for your request, you may set the country_code (string) parameter with one country code at the creation of the request.
For example, to scrape a website with the country code "US", you can use the following request:
https://api.example.com/scrape?country_code=US
This will return the scraped data for the specified country.
Web scraping blog. What else can you do with web scraping?
Now that we scraped our blog and movie titles (if you did the tutorial), you can try to implement web scraping in more of a business-related setting. Our mission is to help you make better decisions and to make better decisions you need data.
Whatever you choose to do with web scraping, ParseHub can Help!
Check out our other blog posts on how you can use ParseHub to help grow your business. We’ve split our blog posts into different categories depending on what kind of information you're trying to extract and the purpose of your scraping.
Ecommerce website/ Competitor Analysis / Brand reputation
- How to Scrape Amazon Product Data: Names, Pricing, ASIN, etc.
- How to Scrape eBay Product Data: Product Details, Prices, Sellers and more.
- How to Scrape Walmart Product Data: Names, Pricing, Details, etc.
- How to Scrape Meta Titles and Meta Descriptions from any Website
- How to Scrape Amazon Reviews: at the step-by-step guide
- How to Scrape Etsy Product Data: Names, Pricing, Seller Information, etc.
- Scrape MercadoLibre Product Data: Names, Details, Prices, Reviews and More!
- How to Scrape Data from an Interactive Google Map or Store Locator
- How to Scrape Search Results from a List of Keywords
- How to Scrape Yellow Pages Data: Business Names, Addresses, Phone Numbers, Emails and more.
- How to Scrape Emails from any Website: Step-by-Step Guide
- Lead Generation: How to Drastically Improve your Process Going into the 2020s
- How to Scrape Twitter timelines: Tweets, Permalinks, Dates and more.
- How to Scrape Yahoo Finance Data: Stock Prices, Bids, Price Change and more.