Superuser Christian Posted September 30, 2022 Superuser Share Posted September 30, 2022 A personal tool using Python's Scrapy framework to scrape Best Buy's product pages for RTX 3080 TIs and notify if available/not sold out. My first project using Python's Scrapy framework. I'm using this project personally for a couple friends of mine and I. Basically, it scrapes a products listing page from BestBuy that lists RTX 3080 TIs. It scans each product and if the c-button-disable class doesn't exist within each entry (indicating it is not sold out and available), it will email a list of users from the settings.py file. It keeps each ID tracked in SQLite to make sure users don't get emailed more than once. Requirements The Scrapy framework is required and may be installed with the following. python3 -m pip install scrapy Settings Settings are configured in the src/bestbuy_parser/bestbuy_parser/settings.py file. The following are defaults. # General Scrapy settings. BOT_NAME = 'bestbuy_parser' SPIDER_MODULES = ['bestbuy_parser.spiders'] NEWSPIDER_MODULE = 'bestbuy_parser.spiders' TELNETCONSOLE_ENABLED = False LOG_LEVEL = 'ERROR' # The User Agent used to crawl. USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36' # Obey robots.txt rules ROBOTSTXT_OBEY = True # Best Buy Parser-specific settings. # The email's subject to send. MAIL_SUBJECT = "RTX 3080 TI In Stock On Best Buy!" # Where the email is coming from. MAIN_FROM = "[email protected]" # The email body. MAIL_BODY = '<html><body><ul><li><a href="https://www.bestbuy.com{link}">{name}</a></li><li>{price}</li></ul></body></html> Running The Program You must change the working directory to src/bestbuy_parser/bestbuy_parser via cd. Afterwards, you may run the following. python3 parse.py This will run the program until a keyboard interrupt. Systemd Service A systemd service is included in the systemd/ directory. It is assuming you cloned the repository into /usr/src (you will need to change the systemd file if this is not correct). You may install the systemd service via the following command as root (or ran with sudo). sudo make install Credits @Christian GitHub Repository & Source Code Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.