Hi there!
So I was hoping programming might be able to help with this. I am trying to learn how to use Selenium for personal projects.
I have tried using normal, straight up selenium but it does not support authorized proxies…
I have tried SeleniumBase, but when I got it working, there were WebRTC leaks that I could not avoid.
I have tried using Undetected-chrome but that too I was unable to get authorized proxies working…
For proxies, I tried seleniumwire as well. That seems to use its own SSL certificates which unfortunately as far as I can tell is easy to detect.
Main purpose for me is web scraping and or simple login to X site, enter X info and close.
So my question is… What is the proper way to use Selenium for automating functions while avoiding detection?
Thank you in advance.
EDIT: I made this post because I am hitting a wall at every turn, and I feel like I might just be approaching this the wrong way, or maybe I’m just missing something crucial. I was hoping someone with more experience could explain to me what is the correct way of doing it without cobbling it together with duct tape.
Also if you are using the latest selenium (4.10 I think) you don’t need to point to the chromedriver executable, selenium will automatically download it if it can’t find the driver
Edit: more information about chrome arguments
Edit: info about driver download
Thank you. The problem with this is that it does not support authorized proxies though. You cannot pass user:pass through it.
My janky solution was to uae something like pproxy to relay the connection through my own peoxy server without user:pass. Has not been fully effective though.
I will look into doing this and using IP authorization instead of user:pass. Thank you for the help.
Also since Selenium just drives an actual browser the WebRTC and DNS leaks will be the browser’s responsibility not selenium.
As long as you can locate elements on a page your will be ok
You might need to install your own proxy on the selenium PC and then chain that proxy to the authenticated one.
Then configure the driver to use the local unauthenticated proxy
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !programming@programming.dev
Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person’s post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Rules
Follow the programming.dev instance rules
Keep content related to programming in some way
If you’re posting long videos try to add in some form of tldr for those who don’t want to watch videos
You can do it with plain selenium all you need to set the proxy in the browser options.
from selenium import webdriver
PROXY_WITH_PORT= “111.222.333.443:8080” chrome_options = webdriver.ChromeOptions() chrome_options.add_argument(f’–proxy-server={PROXY_WITH_PORT}')
chrome = webdriver.Chrome(chrome_options=chrome_options) chrome.get(“http://google.com”)
In general you can pass any command like argument to the browser using options. For chrome you can find all the proxy related options here: https://www.chromium.org/developers/design-documents/network-settings/
Also if you are using the latest selenium (4.10 I think) you don’t need to point to the chromedriver executable, selenium will automatically download it if it can’t find the driver
Edit: more information about chrome arguments Edit: info about driver download
Thank you. The problem with this is that it does not support authorized proxies though. You cannot pass user:pass through it. My janky solution was to uae something like pproxy to relay the connection through my own peoxy server without user:pass. Has not been fully effective though.
I will look into doing this and using IP authorization instead of user:pass. Thank you for the help.
How is vanilla selenium with WebRTC or DNS leaks?
Also since Selenium just drives an actual browser the WebRTC and DNS leaks will be the browser’s responsibility not selenium. As long as you can locate elements on a page your will be ok
You might need to install your own proxy on the selenium PC and then chain that proxy to the authenticated one. Then configure the driver to use the local unauthenticated proxy