I've been working on a scraping project in Scrapy over the last month, using Selenium as well. My Python skills are mediocre (mostly a Java/Kotlin dev).
Not only has it been a blast to try out, but also surprisingly easy to setup.
I now have around 11 domains being scraped 4 times a day through a well defined pipeline + ETL then pipes it to Firebase Firestore for consumption.
Not only has it been a blast to try out, but also surprisingly easy to setup.
I now have around 11 domains being scraped 4 times a day through a well defined pipeline + ETL then pipes it to Firebase Firestore for consumption.
Next step is to write the page on top of it.