Selenium web-driver exception in Deepnote

selenium error

Selenium is a popular tool for web scraping, but you can encounter various issues when running Selenium in different environments. One common issue in Deepnote is the WebDriverException related to the ChromeDriver. We will provide a solution to the Selenium WebDriverException in Deepnote and ensure your web scraping code runs smoothly.

Problem statement

WebDriverException: 
Message: Service /root/.wdm/drivers/chromedriver/linux64/111.0.5563/chromedriver unexpectedly exited. 
Status code was: 127

This error indicates that there is an issue with the ChromeDriver setup in Deepnote.

Solution

pip install selenium webdriver_manager lxml

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from IPython.display import display
import pandas as pd
from time import sleep

options = Options()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-gpu')
options.add_argument('--disable-extensions')
options.add_argument('--dns-prefetch-disable')

Add whatever URLs you want

url1 = {

'SPAB': '<https://www.morningstar.com/etfs/arcx/spab/portfolio>',

'BOND': '<https://www.morningstar.com/etfs/xnys/bond/portfolio>',

'GIBIX': '<https://www.morningstar.com/funds/xnas/gibix/portfolio>',

'WAPSX': '<https://www.morningstar.com/funds/xnas/wapsx/portfolio>',

'CTBRX': '<https://www.morningstar.com/funds/xnas/ctbrx/portfolio>',

'SPTS': '<https://www.morningstar.com/etfs/arcx/spts/portfolio>',

'SPTL': '<https://www.morningstar.com/etfs/arcx/sptl/portfolio>'

}

Making a pull request to scrape the data from the website.

for key, url in url1.items():

	with [webdriver.Chrome](<https://webdriver.chrome/>)(service=Service(ChromeDriverManager().install()), options=options) as driver:

		driver.get(url)

		sleep(8)

		html = [driver.page](<https://driver.page/>)_source

		tables = [pd.read](<https://pd.read/>)_html(html)

Example notebook

For more details, you can refer to an example notebook in Deepnote that demonstrates a functioning web scraping setup:

Example Web Scraper in Deepnote

Conclusion

By following these steps, you can resolve the Selenium WebDriverException and successfully run web scraping tasks in Deepnote.

If you encounter further issues, please contact our support. Happy scraping in Deepnote!