kotlinlang #skrape-it

Manuel Lorenzo

09/01/2020, 4:26 PM

I’ve got another question, this one might be a bit trickier or stupid 🙂

Manuel Lorenzo

09/01/2020, 4:26 PM

so I’ve forked the library and did a commit to my fork - is there an easy and fast way to have gradle pull that commit exactly from my GitHub fork?

Manuel Lorenzo

09/01/2020, 4:27 PM

just to check, because I want to test if what I did works before creating a PR

Christian Dräger

09/01/2020, 4:27 PM

I think I should add the docker thingy to the readme 🤔

Manuel Lorenzo

09/01/2020, 4:34 PM

yes, it works now 🙂

Manuel Lorenzo

09/01/2020, 8:01 PM

https://stackoverflow.com/q/63693300/257948 😄

Manuel Lorenzo

09/02/2020, 12:13 PM

hey guys, me here again

Manuel Lorenzo

09/02/2020, 1:10 PM

good news: I make skrape.it work on Android, the snapshot, so we can confirm it works without having to set the minSdkVersion to 26

Manuel Lorenzo

09/02/2020, 1:16 PM

how do you deal with cookies? for example if I try to parse https://www.huffpost.com/entry/patriot-prayer-portland_n_5f4ea8f0c5b69eb5c0358c26 the first thing I get is the GDPR cookie wall

Manuel Lorenzo

09/02/2020, 1:16 PM

in this case it’s all the huffpost articles

🙃 1

savrov

11/18/2020, 8:27 AM

Hi, is it possible to perform a click ("show more" button) during scrapping task? And what about xPath parsing? Thank you!

Christian Dräger

11/18/2020, 4:32 PM

Hey currently not. It should be technically possible somehow but since the library is first and foremost focusing on parsing html performing clicks or stuff like may or may come not as a feature in the future. Regarding XPath, it's also not supported currently but sounds like a suitable feature for the library that should be inplemented I can the near future .

danieeh

12/01/2020, 10:22 AM

hi there! 👋 I’m creating a web scraper for blog posts.. and one thing I found in common to extract the title, author, etc.. would be the <script type=“application/ld+json”> if this does not contain all the info wanted, I’ll provide alternatives for the different scenarios. could you point me to some examples on how to extract the

ld+json

script content? thanks in advance 😄

elroid

01/28/2021, 5:29 PM

Hello! For the most part I’m loving doing idiomatic scraping with Skrape{it}! However, I’m having trouble getting the js-rendered functionality working. The example at https://docs.skrape.it/docs/dsl/extract-client-side-rendered-data looks like it’s for a previous version, since

mode = Mode.DOM

is no longer available. According to the docs on Github, it looks like all I should need to do is pass

BrowserFetcher

to the skrape function as an argument, but that doesn’t seem to do the trick. I tried setting jsExecution to true, e.g. something like this:

Copy code

skrape(BrowserFetcher) {
    request { url = urlToScrape }
}
extract {
    htmlDocument(html = responseBody, baseUri = baseUri, jsExecution = true) {
        val t = title { findFirst { text } }
        i { "Got title:$t" }
    }
}

…which looks like it’s trying to do something, but I’m getting an error loading one of the included js files:

Copy code

java.io.IOException: Unable to download JavaScript from '<https://cdn>...Stub.js' (status 400).
	at com.gargoylesoftware.htmlunit.html.HtmlPage.loadJavaScriptFromUrl(HtmlPage.java:1075)

Not sure why it’s a 400, as the file in question seems to download fine with a browser or other http clients. Any assistance is much appreciated…

Christian Dräger

01/28/2021, 6:55 PM

Looks like the there is an script on the page that can't be loaded, which should not lead to an error like this. I will have a look at the BrowserFetcher and try to fix tomorrow or at latest on weekend

elroid

01/29/2021, 11:21 AM

Thanks Christian. DM me if you would like any further detail.

elroid

02/02/2021, 11:41 AM

@Christian Dräger have you had a chance to look at the issue? Is there anything else I can try such as overriding the http transport somehow?

elroid

02/02/2021, 6:25 PM

FWIW, after some investigation, the issue seems to lie in the Parser object. Specifically - the

toUriScheme()

method was causing a massive URL, that later when added as a

Referer

header had around 44kb of content - which was too large for the server to accept, hence the 400. Truncating it to 200 bytes meant there were no more errors, but unfortunately the generated dom content was incomplete, so at this point I don’t see a way of using Skrape.it for JS-based sites such as this one. If any fixes are made to make parsing this page viable with Skrape.it I will definitely try again!

Christian Dräger

03/23/2021, 8:33 AM

hey @elroid sorry for late response. had a lot stuff surrounding because of work and kid. i think your issue should be fixed now --> https://github.com/skrapeit/skrape.it/issues/134 will try to release soon. since the the hole project became an multi module project and i have to rearrange the release process

👍 1

samgarasx

04/15/2021, 6:22 PM

Hi, I'm trying to scrap a web and it's throwing an error like Exception in thread "main" java.lang.IllegalArgumentException: unexpected encodedPath:

Copy code

val extracted = skrape(HttpFetcher) {
    request {
        url = config.getString("url")
    }

    extract { 
        htmlDocument {
            h1 { findFirst { text } }
        } 
    }
}

Also, I don't see the option to ignoreContentType, since maybe it's the problem Fixed, error is caused by not adding a '/' to the end of the url

Christian Dräger

04/17/2021, 9:37 PM

hey @samgarasx, version 1.1.0 has just been released right now. “/” workaround is not needed anymore 🙂

👍 1

xexiz

09/13/2021, 11:46 AM

Hi!! First of all, thanks @Christian Dräger for that pretty library, it really makes skraping much more fun hehe 🙂 Although, I’ve been trying with a site and I’m always getting timeouts (even if I override the timeout param to 30sec). Is there any logs we can activate to have more info on what’s going on and why it always times-out? Here’s what I’m trying for now (I did manage to have a successful call a few times to make sure my selectors were ok.

Copy code

private suspend fun getTotalPages(): Int =
    withContext(<http://Dispatchers.IO|Dispatchers.IO>) {
        skrape(AsyncFetcher) {
            request {
                url = "<https://www.capfriendly.com/browse/active>"
            }
            response {
                htmlDocument {
                    div {
                        withClass = "pagination"
                        findFirst {
                            div {
                                withClass = "r"
                                val paginationText = findByIndex(1).text
                                paginationText.substringAfter(" of ").toInt()
                            }
                        }
                    }
                }
            }
        }
    }

• kotlin 1.5.30 • AGP 7.1.0-alpha11 • API 31 • skrapeit:1.1.5 Thanks

Roger Kreienbühl

08/08/2022, 2:53 PM

Hi I am new to skrape.it and try to extract content of a table. Is there a way to process every table by it self, so I get a List with all texts of a row in a List with all rows? e.g.:

List<List<String>>

One row looks like this:

Copy code

<tr role="row">
    <td class="grad" style="background:#067A25; width:50px">
        <div class="grad_outer">
            <span>4c</span>
        </div>
    </td>
    <td class="name">
        King Louie
    </td>
    <td>
        Dave
    </td>
    <td>
        11.03.2022
    </td>
    <td>
        12
    </td>
    <td>
        64.2
    </td>
</tr>

I also need to extract the background css attribute. How can I do this?

Kiet

04/15/2023, 6:45 AM

Hello, new to skrape-it. So I'm testing out using the BrowserFetcher. I was wondering how I can make it click a button on a webpage? For example in this webpage: https://mangasee123.com/manga/Onepunch-Man How would you make it click the "Show All Chapters" div, then get all URLs in the column after the fact?

Ayfri

12/28/2023, 7:12 AM

Is this library still maintained ?

Christian Dräger

12/28/2023, 8:07 AM

📢 🚨 Looking for Co-Maintainer(s), please contact me if you are interested in helping to maintain and evolve skrape{it} ❤️

👍 1

Ayfri

12/28/2023, 9:30 AM

Hello, I have a weird error when trying to scrap an URL, maybe this is interferring with the

Kord-Extension

library but I don't know what to do then (error in 🧵)

Ayfri

12/28/2023, 12:35 PM

Is there a way to find an element or get null if none element were found ?

Caleb Scott

06/03/2024, 2:01 AM

Hey, im trying to use skrapeit but im having some issues when attempting to scrape anything using the example on the docs.

Copy code

skrape(HttpFetcher) { // <-- pass any Fetcher, e.g. HttpFetcher, BrowserFetcher, ...
        request {
            url = "<https://example.com>"
            // ... request options goes here, e.g the most basic would be url
        }
        response {
            println(responseBody)
            // do stuff with the response like parsing the response body ...
        }
    }

error is here, im using a discord library as well, could that be causing the issue?

Copy code

7191 [DefaultDispatcher-worker-4] ERROR dev.kord.core.Kord - catching(java.lang.NoClassDefFoundError: io/ktor/client/features/HttpTimeout)
java.lang.NoClassDefFoundError: io/ktor/client/features/HttpTimeout

let me know if you need any other details to help :P

Jolan Rensen [JB]

02/02/2025, 11:22 PM

Hi! Cool library, just what I was looking for :) I have a JavaScript heavy university website I'd like to scrape. It works with selenium but selenium is very heavy and doesn't work on my home automation system, so I wanted to use something lighter like this :) However, even though BrowserFetcher supposedly executes js. My university website disagrees. I'm supposed to get a login screen but instead I get "Please enable JavaScript to continue using this application." Is there a setting I missed or something else I could try? My result: https://satyr.ugent.be/#/student

Copy code

<!--?xml version="1.0" encoding="UTF-8"?-->
<html lang="en"> 
 <head> 
  <link rel="preconnect" href="<https://fonts.gstatic.com|https://fonts.gstatic.com>" crossorigin=""> 
  <meta charset="utf-8"> 
  <title>
      SATYR
    </title> 
  <base href="/"> 
  <meta name="viewport" content="width=device-width, initial-scale=1"> 
  <meta name="apple-mobile-web-app-capable" content="yes"> 
  <meta name="apple-mobile-web-app-title" content="Satyr"> 
  <meta name="mobile-web-app-capable" content="yes"> 
  <link rel="icon" type="image/x-icon" href="favicon.ico"> 
  <link rel="manifest" href="manifest.webmanifest"> 
  <meta name="theme-color" content="#1976d2"> 
  <style type="text/css">
      @font-face{font-family:'Roboto';font-style:normal;font-weight:300;font-display:swap;src:url(<https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmSU5fCRc4AMP6lbBP.woff2|https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmSU5fCRc4AMP6lbBP.woff2>) format('woff2');unicode-range:U+0460-052F, U+1C80-1C8A, U+20B4, U+2DE0-2DFF, U+A640-A69F, U+FE2E-FE2F;}@font-face{font-family:'Roboto';font-style:normal;font-weight:300;font-display:swap;src:url(<https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmSU5fABc4AMP6lbBP.woff2|https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmSU5fABc4AMP6lbBP.woff2>) format('woff2');unicode-range:U+0301, U+0400-045F, U+0490-0491, U+04B0-04B1, U+2116;}@font-face{font-family:'Roboto';font-style:normal;font-weight:300;font-display:swap;src:url(<https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmSU5fCBc4AMP6lbBP.woff2|https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmSU5fCBc4AMP6lbBP.woff2>) format('woff2');unicode-range:U+1F00-1FFF;}@font-face{font-family:'Roboto';font-style:normal;font-weight:300;font-display:swap;src:url(<https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmSU5fBxc4AMP6lbBP.woff2|https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmSU5fBxc4AMP6lbBP.woff2>) format('woff2');unicode-range:U+0370-0377, U+037A-037F, U+0384-038A, U+038C, U+038E-03A1, U+03A3-03FF;}@font-face{font-family:'Roboto';font-style:normal;font-weight:300;font-display:swap;src:url(<https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmSU5fCxc4AMP6lbBP.woff2|https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmSU5fCxc4AMP6lbBP.woff2>) format('woff2');unicode-range:U+0102-0103, U+0110-0111, U+0128-0129, U+0168-0169, U+01A0-01A1, U+01AF-01B0, U+0300-0301, U+0303-0304, U+0308-0309, U+0323, U+0329, U+1EA0-1EF9, U+20AB;}@font-face{font-family:'Roboto';font-style:normal;font-weight:300;font-display:swap;src:url(<https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmSU5fChc4AMP6lbBP.woff2|https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmSU5fChc4AMP6lbBP.woff2>) format('woff2');unicode-range:U+0100-02BA, U+02BD-02C5, U+02C7-02CC, U+02CE-02D7, U+02DD-02FF, U+0304, U+0308, U+0329, U+1D00-1DBF, U+1E00-1E9F, U+1EF2-1EFF, U+2020, U+20A0-20AB, U+20AD-20C0, U+2113, U+2C60-2C7F, U+A720-A7FF;}@font-face{font-family:'Roboto';font-style:normal;font-weight:300;font-display:swap;src:url(<https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmSU5fBBc4AMP6lQ.woff2|https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmSU5fBBc4AMP6lQ.woff2>) format('woff2');unicode-range:U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+0304, U+0308, U+0329, U+2000-206F, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD;}@font-face{font-family:'Roboto';font-style:normal;font-weight:400;font-display:swap;src:url(<https://fonts.gstatic.com/s/roboto/v32/KFOmCnqEu92Fr1Mu72xKKTU1Kvnz.woff2|https://fonts.gstatic.com/s/roboto/v32/KFOmCnqEu92Fr1Mu72xKKTU1Kvnz.woff2>) format('woff2');unicode-range:U+0460-052F, U+1C80-1C8A, U+20B4, U+2DE0-2DFF, U+A640-A69F, U+FE2E-FE2F;}@font-face{font-family:'Roboto';font-style:normal;font-weight:400;font-display:swap;src:url(<https://fonts.gstatic.com/s/roboto/v32/KFOmCnqEu92Fr1Mu5mxKKTU1Kvnz.woff2|https://fonts.gstatic.com/s/roboto/v32/KFOmCnqEu92Fr1Mu5mxKKTU1Kvnz.woff2>) format('woff2');unicode-range:U+0301, U+0400-045F, U+0490-0491, U+04B0-04B1, U+2116;}@font-face{font-family:'Roboto';font-style:normal;font-weight:400;font-display:swap;src:url(<https://fonts.gstatic.com/s/roboto/v32/KFOmCnqEu92Fr1Mu7mxKKTU1Kvnz.woff2|https://fonts.gstatic.com/s/roboto/v32/KFOmCnqEu92Fr1Mu7mxKKTU1Kvnz.woff2>) format('woff2');unicode-range:U+1F00-1FFF;}@font-face{font-family:'Roboto';font-style:normal;font-weight:400;font-display:swap;src:url(<https://fonts.gstatic.com/s/roboto/v32/KFOmCnqEu92Fr1Mu4WxKKTU1Kvnz.woff2|https://fonts.gstatic.com/s/roboto/v32/KFOmCnqEu92Fr1Mu4WxKKTU1Kvnz.woff2>) format('woff2');unicode-range:U+0370-0377, U+037A-037F, U+0384-038A, U+038C, U+038E-03A1, U+03A3-03FF;}@font-face{font-family:'Roboto';font-style:normal;font-weight:400;font-display:swap;src:url(<https://fonts.gstatic.com/s/roboto/v32/KFOmCnqEu92Fr1Mu7WxKKTU1Kvnz.woff2|https://fonts.gstatic.com/s/roboto/v32/KFOmCnqEu92Fr1Mu7WxKKTU1Kvnz.woff2>) format('woff2');unicode-range:U+0102-0103, U+0110-0111, U+0128-0129, U+0168-0169, U+01A0-01A1, U+01AF-01B0, U+0300-0301, U+0303-0304, U+0308-0309, U+0323, U+0329, U+1EA0-1EF9, U+20AB;}@font-face{font-family:'Roboto';font-style:normal;font-weight:400;font-display:swap;src:url(<https://fonts.gstatic.com/s/roboto/v32/KFOmCnqEu92Fr1Mu7GxKKTU1Kvnz.woff2|https://fonts.gstatic.com/s/roboto/v32/KFOmCnqEu92Fr1Mu7GxKKTU1Kvnz.woff2>) format('woff2');unicode-range:U+0100-02BA, U+02BD-02C5, U+02C7-02CC, U+02CE-02D7, U+02DD-02FF, U+0304, U+0308, U+0329, U+1D00-1DBF, U+1E00-1E9F, U+1EF2-1EFF, U+2020, U+20A0-20AB, U+20AD-20C0, U+2113, U+2C60-2C7F, U+A720-A7FF;}@font-face{font-family:'Roboto';font-style:normal;font-weight:400;font-display:swap;src:url(<https://fonts.gstatic.com/s/roboto/v32/KFOmCnqEu92Fr1Mu4mxKKTU1Kg.woff2|https://fonts.gstatic.com/s/roboto/v32/KFOmCnqEu92Fr1Mu4mxKKTU1Kg.woff2>) format('woff2');unicode-range:U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+0304, U+0308, U+0329, U+2000-206F, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD;}@font-face{font-family:'Roboto';font-style:normal;font-weight:500;font-display:swap;src:url(<https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmEU9fCRc4AMP6lbBP.woff2|https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmEU9fCRc4AMP6lbBP.woff2>) format('woff2');unicode-range:U+0460-052F, U+1C80-1C8A, U+20B4, U+2DE0-2DFF, U+A640-A69F, U+FE2E-FE2F;}@font-face{font-family:'Roboto';font-style:normal;font-weight:500;font-display:swap;src:url(<https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmEU9fABc4AMP6lbBP.woff2|https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmEU9fABc4AMP6lbBP.woff2>) format('woff2');unicode-range:U+0301, U+0400-045F, U+0490-0491, U+04B0-04B1, U+2116;}@font-face{font-family:'Roboto';font-style:normal;font-weight:500;font-display:swap;src:url(<https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmEU9fCBc4AMP6lbBP.woff2|https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmEU9fCBc4AMP6lbBP.woff2>) format('woff2');unicode-range:U+1F00-1FFF;}@font-face{font-family:'Roboto';font-style:normal;font-weight:500;font-display:swap;src:url(<https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmEU9fBxc4AMP6lbBP.woff2|https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmEU9fBxc4AMP6lbBP.woff2>) format('woff2');unicode-range:U+0370-0377, U+037A-037F, U+0384-038A, U+038C, U+038E-03A1, U+03A3-03FF;}@font-face{font-family:'Roboto';font-style:normal;font-weight:500;font-display:swap;src:url(<https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmEU9fCxc4AMP6lbBP.woff2|https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmEU9fCxc4AMP6lbBP.woff2>) format('woff2');unicode-range:U+0102-0103, U+0110-0111, U+0128-0129, U+0168-0169, U+01A0-01A1, U+01AF-01B0, U+0300-0301, U+0303-0304, U+0308-0309, U+0323, U+0329, U+1EA0-1EF9, U+20AB;}@font-face{font-family:'Roboto';font-style:normal;font-weight:500;font-display:swap;src:url(<https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmEU9fChc4AMP6lbBP.woff2|https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmEU9fChc4AMP6lbBP.woff2>) format('woff2');unicode-range:U+0100-02BA, U+02BD-02C5, U+02C7-02CC, U+02CE-02D7, U+02DD-02FF, U+0304, U+0308, U+0329, U+1D00-1DBF, U+1E00-1E9F, U+1EF2-1EFF, U+2020, U+20A0-20AB, U+20AD-20C0, U+2113, U+2C60-2C7F, U+A720-A7FF;}@font-face{font-family:'Roboto';font-style:normal;font-weight:500;font-display:swap;src:url(<https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmEU9fBBc4AMP6lQ.woff2|https://fonts.gstatic.com/s/roboto/v32/KFOlCnqEu92Fr1MmEU9fBBc4AMP6lQ.woff2>) format('woff2');unicode-range:U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+0304, U+0308, U+0329, U+2000-206F, U+20AC, U+2122, U+2191, U+2193, U+2212, U+2215, U+FEFF, U+FFFD;}
    </style> 
  <style type="text/css">
      @font-face{font-family:'Material Icons';font-style:normal;font-weight:400;src:url(<https://fonts.gstatic.com/s/materialicons/v142/flUhRq6tzZclQEJ-Vdg-IuiaDsNcIhQ8tQ.woff2|https://fonts.gstatic.com/s/materialicons/v142/flUhRq6tzZclQEJ-Vdg-IuiaDsNcIhQ8tQ.woff2>) format('woff2');}.material-icons{font-family:'Material Icons';font-weight:normal;font-style:normal;font-size:24px;line-height:1;letter-spacing:normal;text-transform:none;display:inline-block;white-space:nowrap;word-wrap:normal;direction:ltr;-webkit-font-feature-settings:'liga';-webkit-font-smoothing:antialiased;}
    </style> 
  <link rel="apple-touch-icon" href="apple-touch-icon.png"> 
  <style>
      html,body{height:100%}body{margin:0;font-family:Roboto,Helvetica Neue,sans-serif}
    </style> 
  <link rel="stylesheet" href="styles.5eebbcd8f2f71c9f.css" media="print" onload="this.media='all'"> 
 </head> 
 <body class="mat-typography"> 
  <noscript>
    <link rel="stylesheet" href="styles.5eebbcd8f2f71c9f.css"> 
  </noscript> <app-root> 
  </app-root> 
  <noscript>
    Please enable JavaScript to continue using this application. 
  </noscript> 
  <script src="runtime.c9a776efee60aa35.js" type="module">
    </script> 
  <script src="polyfills.bf967bc038366b08.js" type="module">
    </script> 
  <script src="scripts.f98e52dbc72e3d18.js" defer>
    </script> 
  <script src="main.ae85e688225edaba.js" type="module">
    </script>  
 </body>
</html>