[Solved] Scraping Data Table in websites

[shafiey]

Hello, first of all, I would like to thank you for your very useful and valuable training. I read your tutorials about Web Scraping in two different modes, but the table data in my favorite website is not scraped. The target site is https://scientometric.areeo.ac.ir/.
Thank you for checking these two files and solving my problem.
Also, is it possible to scrape the table in three different modes, according to the description on the screenshot (once in Scopus selection mode, once in WOS selection mode and once in Google Scholar selection mode)?
Thank you very much

 

[Rubayed Razib]

Greetings Shafiey,
Thanks a lot for your inquiry. The site you have provided is behaving dynamically as it is working with Javascript. Because of the javascript integration, it isn't easy to extract info from there. Even the table that you are seeing is not even available in the HTML at all. It pulls data from the server dynamically based on the data available in the javascript. Thats why the article from our website did not work in your case. You need to have a VBA code that will be able to dig into javascript and that is quite impossible to do. On thing you could is try to extract every single element of the site to the worksheet and see if they come in handy for you. Below i have given you a code running which you will get all the element name and their output value or content

Sub ExtractTabularDataToWorksheet()
    Dim driver As New Selenium.ChromeDriver
    Dim ws As Worksheet
    driver.Start "chrome"
    driver.Get "https://scientometric.areeo.ac.ir"
    ' Wait for the website to load (adjust the timeout as needed)
    driver.Wait 10000 ' Wait for 10 seconds
    ' Reference the worksheet where you want to display the content
    Set ws = ThisWorkbook.Sheets("Sheet3") ' Replace "Sheet1" with your sheet name
    Dim tbody As Selenium.WebElement
    Set tbody = driver.FindElementByClass("result-body")
    If Not tbody Is Nothing Then
        Dim trElements As Selenium.WebElements
        Set trElements = tbody.FindElementsByTag("tr")
        Dim row As Long
        row = 1 ' Start from row 1
        Dim trElement As Selenium.WebElement
        For Each trElement In trElements
            Dim tdElements As Selenium.WebElements
            Set tdElements = trElement.FindElementsByTag("td")            
            Dim col As Long
            col = 1 ' Start from column 1            
            ' Loop through <td> elements within <tr> and extract values
            Dim tdElement As Selenium.WebElement
            For Each tdElement In tdElements
                ws.Cells(row, col).Value = tdElement.Text
                col = col + 1 ' Move to the next column
            Next tdElement      
            row = row + 1 ' Move to the next row
        Next trElement
    End If
    driver.Quit
End Sub

After running this code you will see that the chart values from the website is now showing in the worksheet.

 

[shafiey]

Hello my dear
Thank you very very much for your guide
good luck