I don’t think there’s any other module out there which will let you scrape content from a webpage for display on MagicMirror, maybe there’s a good reason for that, but I went ahead and built one for myself, with quite a bit of help from ChatGPT (I wouldn’t consider myself a developer, just a tinkerer, so please excuse the code quality!).

I’ve made it quite flexible, so whilst my use case is to scrape live bus departure times from the stop outside my house, I’m sure there will be other use cases …

You specify a URL to scrape, a CSS selector to identify the table on the page, and then some options about what rows and columns you want to include.

I’m not sure if this is ready for public consumption, but if any devs would like to take a look and give me any advice, please do.

https://github.com/AndyHazz/MMM-Scrapey

Description:

Show alerts on your mirror when there’s a chance of seeing aurora over the UK, using data from https://aurorawatch.lancs.ac.uk/ - updates every 5 mins to show the current status.

Works well as a top_bar banner, that only appears when the criteria you set are met (e.g only show red alerts, at nighttime, when cloud cover is less than 20%). OpenWeather API key required for the daylight and cloudcover features.

Screenshots:

Download:

[ card:AndyHazz/MMM-AuroraWatchUK]
https://github.com/AndyHazz/MMM-AuroraWatchUK

@mumblebaj for a bit more context I’m using MagicMirrorOS and it’s been great whilst it was working, but certainly seems to make dubugging harder so perhaps I’ll switch back to regular setup when I get a bit more time …

To clarify, I’m only seeing errors when trying to work around lack of access to logs in docker - if I just let the system run normally I’m not getting any errors, just an unexpected change in calendar behaviour.

docker compose up -d works if I enter /opt/mm/run directory first.
Not sure I need that compose up command though as the container is running ok already. docker exec -it mm bash works to get into the container, but then I can’t find any debug logs …

@KristjanESPERANTO I’m not sure but I think the errors referring to MMM-Remote-Control I posted were more to do with my failed attempts to get logs from the docker container, not the original issue of calendar display … but in case you’re onto something:

> node@magicmirroros:/opt/magic_mirror/modules/MMM-Remote-Control$ grep '"version":' package.json
  "version": "3.1.4",

@KristjanESPERANTO No problem, done!

Thanks, I’ve merged that logging improvement to MMM-Scrapey.

I suppose my query is simply whether anybody else can see the same calendar functionality issues as me?

• limitDays has changed to limit the number of calendar items, not the number of days worth of calendar items
• excludedEvents appears to do nothing any more, events matching excluded words have started showing when prior to 2.32.0 they were hidden

If it’s only affecting me, I’ll probably try rebuilding my setup from scratch.

@chris_d glad you’ve found it useful, and that’s a good idea for the module option - I actually had the same issue with the data I was scraping and was using some custom CSS to clean up the formatting, but a plain text option in the module is a better approach.

So, I’ve added a new ‘plainText’ option to the config. The default is still to use HTML in case anyone was making use of that (I doubt it but you never know), you’ll need to add ‘plainText: true’ to the module config to strip the HTML formatting.

said in 2.32.0 Calendar module limitDays and excludedEvents stopped working:

Previous suggestions to fix the excludedEvents still not working though.

@sdetweil hold up - it’s all working now! I don’t think I changed anything else, just an extra refresh. So, all your fixes are good. Sorry that took a while to figure out, and if I’d read @karsten13’s docker instructions properly I think that docker module override feature was probably the cause of earlier confusion.

@fredric.nil yes as @sdetweil says it needed to be .MMM-Scrapey td {} to have any effect.

Try this in your custom CSS, I think it’s the line height you want to adjust, not the padding:

.MMM-Scrapey td {
    line-height: 1;
}

MagicMirror default line height on the body element is 1.5 so I think everything inherits that unless overidden.