MMM-Scrapey - I made something that could be useful?

chris_d

@AndyHazz Thanks for this awesome module. I’m also using it to pull bus information and it’s working well. I wondered if there was a way of stripping formatting off the content it scrapes? The table I’m pulling from (https://bustimes.org/stops/250012526) includes hyperlinks in several of the columns, and I’d like to format them as plain text.

Any ideas how that could be achieved?
Thanks again for this, it’s totally made things amazing :-)

chris_d

I think I figured it out by adjusting the .js code to return .text rather than .innerHTML on line 88.
This could possibly be something that could be a module option. If I get some time I might consider taking this and doing some work on it, but like you I’m a bit of a ChatGPT coder :-), so I’m not even sure how to copy it!

It’s working great for me now though, so thank you again!

AndyHazz

@chris_d glad you’ve found it useful, and that’s a good idea for the module option - I actually had the same issue with the data I was scraping and was using some custom CSS to clean up the formatting, but a plain text option in the module is a better approach.

So, I’ve added a new ‘plainText’ option to the config. The default is still to use HTML in case anyone was making use of that (I doubt it but you never know), you’ll need to add ‘plainText: true’ to the module config to strip the HTML formatting.

fredric.nil

Hello.
Thanks for a great module.
I wonder how to decrese the distance between the lines.
I’ve tried adding table attributes to the table (as I can controll the table myself), but it doesn’t seems to work.
I also tried with custom.css to add the attributes style in it.
However, it appears that the module strips all styling from the presentation.

This is the code that I have used in custom.css, but these don’t seem to bite.

.MMM-Scrapey{
height:2px;
}

.MMM-Scrapey .td {
      padding: 2px;
}
.MMM-Scrapey .table {
      border-spacing: 2px;
}
.table {
      border-spacing: 2px;
}
.td {
      padding: 2px;
}

sdetweil

@fredric.nil don’t know the answer, but use the developers window elements tab

see second link in my signature below to learn how to use

fredric.nil

Thanks for the tip, but that’s where I was looking.
It seems that no matter what I put in for attributes, they don’t show up in the module.
I’m going crazy soon

sdetweil

@fredric.nil where are you putting them?

sdetweil

@fredric.nil

well there is a problem

.MMM-Scrapey .td {

the stuff before the {
is called the selector clause, it ‘selects’ the elements the styles after the { apply to

if the thing in the selector clause starts with a dot(.)
that means classname, class=
if it starts with a pound sign(#)
that means id, id=
otherwise no prefix means an html element name

table, tr, td, div, …

you have .td

fredric.nil

I put it in the custom.css.
It doesn’t seem to matter if I enter

.td , #td , td or .MMM-Scrapey .td , .MMM-Scrapey3 #td or .MMM-Scrapey td or even .td , #td or td before {

I can only say that this is not my cup of tea.

AndyHazz

@fredric.nil yes as @sdetweil says it needed to be .MMM-Scrapey td {} to have any effect.

Try this in your custom CSS, I think it’s the line height you want to adjust, not the padding:

.MMM-Scrapey td {
    line-height: 1;
}

MagicMirror default line height on the body element is 1.5 so I think everything inherits that unless overidden.