Calendar Module Error with Many Google Calendars

moorevineyard

I’m open to that, but this seems like a very clear pattern, and it’s been hours without the missing calendars getting synched.

Also, it seems clear that the problem is related to the number of calendars I am trying to synch. It is always the first 6 that get added and the failure begins on the 7th. Everything after that point gets an error.

If I comment out one of the early calendars in the config file, I get the next 6 to synch and again the remaining ones get errors. Is there a timer that I can set to make them attempt a re-synch sooner than the fetchInterval in order to test this?

moorevineyard

@karsten13

Is this all the relevant stuff?

> magicmirror@2.29.0 start
> DISPLAY="${DISPLAY:=:0}" ./node_modules/.bin/electron js/electron.js

[2024-10-03 17:27:24.356] [LOG]   Starting MagicMirror: v2.29.0

[2024-10-03 17:27:28.036] [INFO]  System information:
### SYSTEM:   manufacturer: ; model: ; virtual: false
### OS:       platform: linux; distro: Debian GNU/Linux; release: 12; arch: arm64; kernel: 6.6.31+rpt-rpi-v8
### VERSIONS: electron: 31.6.0; used node: 20.17.0; installed node: 22.9.0; npm: 10.8.3; pm2: 5.4.2
### OTHER:    timeZone: America/New_York; ELECTRON_ENABLE_GPU: undefined

karsten13

@moorevineyard

yes, thanks.

We had similar errors with older electron versions running with node v18, but you are using the newest mm release.

moorevineyard

@karsten13

In case it matters, I also get a lot of these:

[8477:1003/172727.247634:ERROR:gbm_wrapper.cc(74)] Failed to get fd for plane.: No such file or directory (2)
[8477:1003/172727.253029:ERROR:gbm_wrapper.cc(257)] Failed to export buffer to dma_buf: No such file or directory (2)

My searching on that makes it seem like it’s unrelated, but I’ve added the following into the mm.sh file and there’s no change:

export ELECTRON_DISABLE_GPU=1

moorevineyard

Well, I did some testing and it looks like @sdetweil may have been correct.

I have a total of 9 calendars that I’m trying to synch. I noticed that the logs show execution of a “calendarfetcher”.

[2024-10-03 18:17:18.637] [LOG]   Create new calendarfetcher for url: https:// ... .ics - Interval: 3600000

So it looks like the default interval is 3600000 (1 hr), but I thought the default fetchInterval was 300000 (5 min) based on the documentation. On top of that, I had a separate problem with the Pi that caused it to hang.

So that whole section of my logs looks like this…

[2024-10-03 18:17:18.637] [LOG]   Create new calendarfetcher for url: https://calendar.google.com/calendar/ical/first/basic.ics - Interval: 3600000
[2024-10-03 18:17:18.758] [LOG]   Create new calendarfetcher for url: https://calendar.google.com/calendar/ical/second/basic.ics - Interval: 3600000
[2024-10-03 18:17:18.763] [LOG]   Create new calendarfetcher for url: https://calendar.google.com/calendar/ical/third/basic.ics - Interval: 3600000
[2024-10-03 18:17:18.766] [LOG]   Create new calendarfetcher for url: https://calendar.google.com/calendar/ical/fourth/basic.ics - Interval: 3600000
[2024-10-03 18:17:18.770] [LOG]   Create new calendarfetcher for url: https://calendar.google.com/calendar/ical/fifth/basic.ics - Interval: 3600000
[2024-10-03 18:17:18.774] [LOG]   Create new calendarfetcher for url: https://calendar.google.com/calendar/ical/sixth/basic.ics - Interval: 3600000
[2024-10-03 18:17:18.777] [LOG]   Create new calendarfetcher for url: https://calendar.google.com/calendar/ical/seventh/basic.ics - Interval: 60000
[2024-10-03 18:17:18.781] [LOG]   Create new calendarfetcher for url: https://calendar.google.com/calendar/ical/eighth/basic.ics - Interval: 60000
[2024-10-03 18:17:18.785] [LOG]   Create new calendarfetcher for url: https://calendar.google.com/calendar/ical/ninth/basic.ics - Interval: 60000

I changed the fetchInterval for last three calendars to 1 min. At that point, those last three calendars are fetched, the screen gets updated, and the errors go away.

All seems good, right?

I did an additional test with the first 6 set to a 2 minute interval and the last three set to a 1 minute interval.

Again, the screen did not update properly for the first minute. At 1 minute, the last three calendars updated and the screen refreshed with all the info. After two minutes, all 9 calendars appeared to update without error and the screen stayed up-to-date.

So I think this is only a problem with the first run of the fetch function.

sdetweil

@moorevineyard 2 minute fetch is too frequent

it might not be enough time to get and parse

and DO you NEED to see the event you JUST added to the cal someplace else on the screen. when is the event? tomorrow, next week? not 2 minutes from now

moorevineyard

@sdetweil

I definitely agree that 2 minutes is too short for normal operations.

In my examples above, I was setting low times for troubleshooting purposes. I’m expecting to set the fetchInterval to about 24 hours once everything is set up the way I want. But this bug still causes lots of problems.

In particular:

For a new user that is frequently changing settings and styles, every change would require waiting a significant period of time before being able to check if it is acceptable. In my case, it means waiting a full day to see any calendar events again if I want to change the colors of one calendar.

Over time, events or calendars may be added by mistake. As far as I know, the normal way to refresh everything is to restart MagicMirror. Again, in this case, removing something that I do t want displayed means that I can’t see other events for a full day.

Is there another method of refreshing calendar data that would be less disruptive?

sdetweil

@moorevineyard you can set the fetch interval by url too

moorevineyard

I did some more testing and it appears that there may be some sort of rate limit with Google. I haven’t been able to find anything published, but slowing down the fetch process seems to fix the problem.

I’m not super great with JavaScript and don’t know anything about NodeJS, but this is what I did. In the calendar.js file, starting in line 121 and then down around line 158 I changed the following:

                this.config.calendars.forEach((calendar, index) => {  // added index
                        calendar.url = calendar.url.replace("webcal://", "http://");

                        const calendarConfig = {
                                maximumEntries: calendar.maximumEntries,
                                maximumNumberOfDays: calendar.maximumNumberOfDays,
                                pastDaysCount: calendar.pastDaysCount,
                                broadcastPastEvents: calendar.broadcastPastEvents,
                                selfSignedCert: calendar.selfSignedCert,
                                excludedEvents: calendar.excludedEvents,
                                fetchInterval: calendar.fetchInterval
                        };

                        if (typeof calendar.symbolClass === "undefined" || calendar.symbolClass === null) {
                                calendarConfig.symbolClass = "";
                        }
                        if (typeof calendar.titleClass === "undefined" || calendar.titleClass === null) {
                                calendarConfig.titleClass = "";
                        }
                        if (typeof calendar.timeClass === "undefined" || calendar.timeClass === null) {
                                calendarConfig.timeClass = "";
                        }

                        // we check user and password here for backwards compatibility with old configs
                        if (calendar.user && calendar.pass) {
                                Log.warn("Deprecation warning: Please update your calendar authentication configuration.");
                                Log.warn("https://docs.magicmirror.builders/modules/calendar.html#configuration-options");
                                calendar.auth = {
                                        user: calendar.user,
                                        pass: calendar.pass
                                };
                        }

                        /*
                         * tell helper to start a fetcher for this calendar
                         * fetcher till cycle
                         */
setTimeout(() => {             // added
                        this.addCalendar(calendar.url, calendar.auth, calendarConfig);
}, index * 5000);  // 5-second delay for each calendar
                });

This appears to add a 5 second delay to each of the fetches (one fetch occurs every 5 seconds during startup). I added a couple random calendars and now up to 11 seems to work without a problem.

Is it worth adding a configuration option to introduce the delay? Is Github the best place to make these kinds of suggestions?

sdetweil

@moorevineyard yes, github

you could create a pull request with the code solution
maybe just do it if the count is over 4 ?

11*5 = 55 seconds, almost a minute before some calendars display. Some would not like any delay