I don’t think there’s any other module out there which will let you scrape content from a webpage for display on MagicMirror, maybe there’s a good reason for that, but I went ahead and built one for myself, with quite a bit of help from ChatGPT (I wouldn’t consider myself a developer, just a tinkerer, so please excuse the code quality!).

I’ve made it quite flexible, so whilst my use case is to scrape live bus departure times from the stop outside my house, I’m sure there will be other use cases …

You specify a URL to scrape, a CSS selector to identify the table on the page, and then some options about what rows and columns you want to include.

I’m not sure if this is ready for public consumption, but if any devs would like to take a look and give me any advice, please do.

https://github.com/AndyHazz/MMM-Scrapey

Description:

Show alerts on your mirror when there’s a chance of seeing aurora over the UK, using data from https://aurorawatch.lancs.ac.uk/ - updates every 5 mins to show the current status.

Works well as a top_bar banner, that only appears when the criteria you set are met (e.g only show red alerts, at nighttime, when cloud cover is less than 20%). OpenWeather API key required for the daylight and cloudcover features.

Screenshots:

Download:

[ card:AndyHazz/MMM-AuroraWatchUK]
https://github.com/AndyHazz/MMM-AuroraWatchUK

@KristjanESPERANTO No problem, done!

@chris_d glad you’ve found it useful, and that’s a good idea for the module option - I actually had the same issue with the data I was scraping and was using some custom CSS to clean up the formatting, but a plain text option in the module is a better approach.

So, I’ve added a new ‘plainText’ option to the config. The default is still to use HTML in case anyone was making use of that (I doubt it but you never know), you’ll need to add ‘plainText: true’ to the module config to strip the HTML formatting.

@fredric.nil yes as @sdetweil says it needed to be .MMM-Scrapey td {} to have any effect.

Try this in your custom CSS, I think it’s the line height you want to adjust, not the padding:

.MMM-Scrapey td {
    line-height: 1;
}

MagicMirror default line height on the body element is 1.5 so I think everything inherits that unless overidden.

@fredric.nil yes as @sdetweil says it needed to be .MMM-Scrapey td {} to have any effect.

Try this in your custom CSS, I think it’s the line height you want to adjust, not the padding:

.MMM-Scrapey td {
    line-height: 1;
}

MagicMirror default line height on the body element is 1.5 so I think everything inherits that unless overidden.

@sdetweil yes, I have apiVersion: ‘2.5’ in my config - I didn’t change that recently though, I think I changed that perhaps a year ago when it stopped working previously … whatever the latest update did, I now need the /weather endpoint specified as well. All good though, it’s working fine now.

@chris_d glad you’ve found it useful, and that’s a good idea for the module option - I actually had the same issue with the data I was scraping and was using some custom CSS to clean up the formatting, but a plain text option in the module is a better approach.

So, I’ve added a new ‘plainText’ option to the config. The default is still to use HTML in case anyone was making use of that (I doubt it but you never know), you’ll need to add ‘plainText: true’ to the module config to strip the HTML formatting.

The same happened to me, my fix was a bit simpler but maybe only temporary …

It seems the default endpoint has changed - the documentation still says that the default for weatherEndpoint should be ‘/weather’, but I think in the latest update that changed to ‘/onecall’. For those of us with old api keys I think it’s still ok to use the /weather endpoint, you just need to specify that in the config now that it’s no longer the default.

weatherEndpoint: '/weather',

Description:

Show alerts on your mirror when there’s a chance of seeing aurora over the UK, using data from https://aurorawatch.lancs.ac.uk/ - updates every 5 mins to show the current status.

Works well as a top_bar banner, that only appears when the criteria you set are met (e.g only show red alerts, at nighttime, when cloud cover is less than 20%). OpenWeather API key required for the daylight and cloudcover features.

Screenshots:

Download:

[ card:AndyHazz/MMM-AuroraWatchUK]
https://github.com/AndyHazz/MMM-AuroraWatchUK

@KristjanESPERANTO No problem, done!

I don’t think there’s any other module out there which will let you scrape content from a webpage for display on MagicMirror, maybe there’s a good reason for that, but I went ahead and built one for myself, with quite a bit of help from ChatGPT (I wouldn’t consider myself a developer, just a tinkerer, so please excuse the code quality!).

I’ve made it quite flexible, so whilst my use case is to scrape live bus departure times from the stop outside my house, I’m sure there will be other use cases …

You specify a URL to scrape, a CSS selector to identify the table on the page, and then some options about what rows and columns you want to include.

I’m not sure if this is ready for public consumption, but if any devs would like to take a look and give me any advice, please do.

https://github.com/AndyHazz/MMM-Scrapey