| || |
Scraping the products from a e-commerce web site using web driver with java
Hi i want to scrap the products, price, images, description from a web site. now i am trying to get the page source in web driver using java, how could i use Jsoup and how to get the products, price, images, description. and i want to insert these details in to Db.. i need sample code for how to write the script in webdriver java...
I've done this before for scraping stock data from google and yahoo, for a little side project that correlated that judged if an article was good news/bad news based on the stock price jump.
Originally Posted by gvs048
I preferred to just use raw curl request, and grep against the data vs. automation via selenium. A lot of these data driven SOA websites also have backend apis, that when you call it using some parameter like output=json, or something along those lines, they'll give you straight json data that's easy to parse.
With selenium, you'll have to worry about the async nature of how the browser renders the page, then having to deal with the weird little intricacies.