I don’t think there’s any other module out there which will let you scrape content from a webpage for display on MagicMirror, maybe there’s a good reason for that, but I went ahead and built one for myself, with quite a bit of help from ChatGPT (I wouldn’t consider myself a developer, just a tinkerer, so please excuse the code quality!).

I’ve made it quite flexible, so whilst my use case is to scrape live bus departure times from the stop outside my house, I’m sure there will be other use cases …

You specify a URL to scrape, a CSS selector to identify the table on the page, and then some options about what rows and columns you want to include.

I’m not sure if this is ready for public consumption, but if any devs would like to take a look and give me any advice, please do.

https://github.com/AndyHazz/MMM-Scrapey

Description:

Show alerts on your mirror when there’s a chance of seeing aurora over the UK, using data from https://aurorawatch.lancs.ac.uk/ - updates every 5 mins to show the current status.

Works well as a top_bar banner, that only appears when the criteria you set are met (e.g only show red alerts, at nighttime, when cloud cover is less than 20%). OpenWeather API key required for the daylight and cloudcover features.

Screenshots:

Download:

[ card:AndyHazz/MMM-AuroraWatchUK]
https://github.com/AndyHazz/MMM-AuroraWatchUK

@mumblebaj for a bit more context I’m using MagicMirrorOS and it’s been great whilst it was working, but certainly seems to make dubugging harder so perhaps I’ll switch back to regular setup when I get a bit more time …

To clarify, I’m only seeing errors when trying to work around lack of access to logs in docker - if I just let the system run normally I’m not getting any errors, just an unexpected change in calendar behaviour.

docker compose up -d works if I enter /opt/mm/run directory first.
Not sure I need that compose up command though as the container is running ok already. docker exec -it mm bash works to get into the container, but then I can’t find any debug logs …

@KristjanESPERANTO I’m not sure but I think the errors referring to MMM-Remote-Control I posted were more to do with my failed attempts to get logs from the docker container, not the original issue of calendar display … but in case you’re onto something:

> node@magicmirroros:/opt/magic_mirror/modules/MMM-Remote-Control$ grep '"version":' package.json
  "version": "3.1.4",

@KristjanESPERANTO No problem, done!

Thanks, I’ve merged that logging improvement to MMM-Scrapey.

I suppose my query is simply whether anybody else can see the same calendar functionality issues as me?

• limitDays has changed to limit the number of calendar items, not the number of days worth of calendar items
• excludedEvents appears to do nothing any more, events matching excluded words have started showing when prior to 2.32.0 they were hidden

If it’s only affecting me, I’ll probably try rebuilding my setup from scratch.

@chris_d glad you’ve found it useful, and that’s a good idea for the module option - I actually had the same issue with the data I was scraping and was using some custom CSS to clean up the formatting, but a plain text option in the module is a better approach.

So, I’ve added a new ‘plainText’ option to the config. The default is still to use HTML in case anyone was making use of that (I doubt it but you never know), you’ll need to add ‘plainText: true’ to the module config to strip the HTML formatting.

said in 2.32.0 Calendar module limitDays and excludedEvents stopped working:

Previous suggestions to fix the excludedEvents still not working though.

@sdetweil hold up - it’s all working now! I don’t think I changed anything else, just an extra refresh. So, all your fixes are good. Sorry that took a while to figure out, and if I’d read @karsten13’s docker instructions properly I think that docker module override feature was probably the cause of earlier confusion.

@fredric.nil yes as @sdetweil says it needed to be .MMM-Scrapey td {} to have any effect.

Try this in your custom CSS, I think it’s the line height you want to adjust, not the padding:

.MMM-Scrapey td {
    line-height: 1;
}

MagicMirror default line height on the body element is 1.5 so I think everything inherits that unless overidden.

said in 2.32.0 Calendar module limitDays and excludedEvents stopped working:

Previous suggestions to fix the excludedEvents still not working though.

@sdetweil hold up - it’s all working now! I don’t think I changed anything else, just an extra refresh. So, all your fixes are good. Sorry that took a while to figure out, and if I’d read @karsten13’s docker instructions properly I think that docker module override feature was probably the cause of earlier confusion.

@karsten13 aha, good to know, although I already started and now finished setting up from scratch again 🙄 no docker this time … I kept it on a spare sd card though so may switch back.

@sdetweil new code for limitDays reapplied on a fresh install, still seems fine. Previous suggestions to fix the excludedEvents still not working though.

@sdetweil no problem, that new code seems to work fine for me.

However, I don’t seem to have a great setup for testing - after making the changes, I can load up MM in my remote/laptop browser and immediately see the changes working, but if I try and refresh the electron browser on the raspberry pi running my mirror (using MMM-Remote-Control), it seems to take ages, crash, and then reverts the code back to the docker image default 😮‍💨

I can see this in the docker container logs so I reckon it’s reverting any changes I make there?

[entrypoint 18:36:04.283] [INFO]   copy default modules
[entrypoint 18:36:04.918] [INFO]   copy css files
> magicmirror@2.32.0 start
> node --run start:x11

@sdetweil I’ve no idea then, but if you were able to reproduce the issue I had, and then fix it on your end, I probably just messed something up somewhere and without any access to logs on my docker setup that may remain a mystery … I’ve just had another go at trying to figure out what that may be, no luck.

No rush at all but I guess if you’re happy with the fix it’ll show up in a new MM version at some point and I’ll update to that.

@sdetweil should I see a console message from loader.js about calendarfetcherutils.js being loaded?

I see

Load script: modules/default/calendar/calendar.js
Load script: modules/default/calendar/calendarutils.js

but no mention in the console of calendarfetcherutils.js

@sdetweil fair enough, complexity no doubt warranted!

Perhaps is there any reason why shouldEventBeExcluded function might not be getting called at all for me? I’m stumped why after making all these changes including console.log messages, nothing is appearing any different -

@sdetweil yep that’s what I’ve got, copying below in case I’m missing something …

I’ve also added your debug messages back in and changed Log.debug to console.log (as I still can’t get any debug logs from docker but I do get console.log messages from other modules) but nothing comes out in the console log - seems odd that all these changes have had zero effect, am I missing something that might be needed to get these changes to kick in?

        shouldEventBeExcluded (config, title) {
                let result = {  // CHANGED
                        excluded: false,
                        until: null
                };
                for (let f in config.excludedEvents) {
                        let filter = config.excludedEvents[f],
                                testTitle = title.toLowerCase(),
                                until = null,
                                useRegex = false,
                                regexFlags = "g";
                        if (filter instanceof Object) {
                                if (typeof filter.until !== "undefined") {
                                        until = filter.until;
                                }

                                if (typeof filter.regex !== "undefined") {
                                        useRegex = filter.regex;
                                }

                                // If additional advanced filtering is added in, this section
                                // must remain last as we overwrite the filter object with the
                                // filterBy string
                                if (filter.caseSensitive) {
                                        filter = filter.filterBy;
                                        testTitle = title;
                                } else if (useRegex) {
                                        filter = filter.filterBy;
                                        testTitle = title;
                                        regexFlags += "i";
                                } else {
                                        filter = filter.filterBy.toLowerCase();
                                }
                        } else {
                                filter = filter.toLowerCase();
                        }
                        console.log("should be excluded ", testTitle, filter, useRegex, regexFlags)
                        if (CalendarFetcherUtils.titleFilterApplies(testTitle, filter, useRegex, regexFlags)) {
                                if (until) {
                                        result.until = until;  // CHANGED
                                } else {
                                        console.log("event should be excluded = true,", testTitle )
                                        result.excluded = true;  // CHANGED
                                }
                                console.log("filter applies result =", result)
                                break;
                        }
                }
                console.log("filter applies returning =", result)
                return result;  // CHANGED
        },

p.s. just spotted comment on line 12 😅

TODO This seems like an overly complicated way to exclude events based on the title.

@sdetweil perhaps I’m missing something, but that’s not working for me - it’s made no difference, excluded events are still appearing … I’ve double checked and can’t spot anything amiss, I’ve made the 4 changes from ‘filter’ to ‘result’?

@sdetweil nice! Yes that’s fixed the limitDays issue for me …

Now, how about part 2 of my issue, the excludedEvents config? The first event showing for me is titled ‘Tutor time’ despite that string being excluded. I’ve tried other terms with/without spaces and nothing seems to work to exclude events any more?

@sdetweil yeah that’s what I was using limitDays for and it was working great for my needs until now - I have my kids school lessons synced to a google calendar, and setting limitDays to 1 used to show the full set of lessons for the day. When they get home from school (all events from the current day are in the past), it automatically showed all of the lessons for the following day. Best thing I’ve ever used MM for! Even highlights PE lessons so they know when to bring kit :)

For me now, limitDays: 1 just shows the single next event even when there are 6 more events on that day (and if that event is ‘Tutor time’, it displays when it should be hidden).

I can’t say exactly what version I was on prior to the update but it should have been whatever the previous version was - since I set this up with watchtower in docker I’ve not had to do any manual upgrades.

Here’s my anonamised calendar config - I have 5 different calendar modules running if that’s likely to be a factor, but this is one that has the config issues:

    {
        module: 'calendar',
        header: 'School day',
        position: 'bottom_right',
        config: {
            coloredSymbol: true,
            coloredText: true,
            showLocation: true,
            displaySymbol: false,
            hideTime: true,
            tableClass: 'medium',
            flipDateHeaderTitle: true,
            fade: false,
            timeFormat: 'absolute',
            dateFormat: 'ddd',
            fullDayEventDateFormat: 'ddd',
            urgency: '0',
            sliceMultiDayEvents: true,
            nextDaysRelative: false,
            limitDays: 1,
            fetchInterval: '600000',
            displayRepeatingCountTitle: true,
            wrapEvents: false,
            //maxTitleLength: 30,
            customEvents: [{ keyword: 'Physical Education', symbol: 'volleyball', color: 'Gold' }],
            excludedEvents: [
                'Tutor time'
            ],
            calendars: [
                {
                    symbol: 'graduation-cap',
                    url: 'https://calendar.google.com/calendar/ical/REDACTED/basic.ics'
                }
            ]
        }
    },