Calendar Module Error with Many Google Calendars

moorevineyard

Well, I did some testing and it looks like @sdetweil may have been correct.

I have a total of 9 calendars that I’m trying to synch. I noticed that the logs show execution of a “calendarfetcher”.

[2024-10-03 18:17:18.637] [LOG]   Create new calendarfetcher for url: https:// ... .ics - Interval: 3600000

So it looks like the default interval is 3600000 (1 hr), but I thought the default fetchInterval was 300000 (5 min) based on the documentation. On top of that, I had a separate problem with the Pi that caused it to hang.

So that whole section of my logs looks like this…

[2024-10-03 18:17:18.637] [LOG]   Create new calendarfetcher for url: https://calendar.google.com/calendar/ical/first/basic.ics - Interval: 3600000
[2024-10-03 18:17:18.758] [LOG]   Create new calendarfetcher for url: https://calendar.google.com/calendar/ical/second/basic.ics - Interval: 3600000
[2024-10-03 18:17:18.763] [LOG]   Create new calendarfetcher for url: https://calendar.google.com/calendar/ical/third/basic.ics - Interval: 3600000
[2024-10-03 18:17:18.766] [LOG]   Create new calendarfetcher for url: https://calendar.google.com/calendar/ical/fourth/basic.ics - Interval: 3600000
[2024-10-03 18:17:18.770] [LOG]   Create new calendarfetcher for url: https://calendar.google.com/calendar/ical/fifth/basic.ics - Interval: 3600000
[2024-10-03 18:17:18.774] [LOG]   Create new calendarfetcher for url: https://calendar.google.com/calendar/ical/sixth/basic.ics - Interval: 3600000
[2024-10-03 18:17:18.777] [LOG]   Create new calendarfetcher for url: https://calendar.google.com/calendar/ical/seventh/basic.ics - Interval: 60000
[2024-10-03 18:17:18.781] [LOG]   Create new calendarfetcher for url: https://calendar.google.com/calendar/ical/eighth/basic.ics - Interval: 60000
[2024-10-03 18:17:18.785] [LOG]   Create new calendarfetcher for url: https://calendar.google.com/calendar/ical/ninth/basic.ics - Interval: 60000

I changed the fetchInterval for last three calendars to 1 min. At that point, those last three calendars are fetched, the screen gets updated, and the errors go away.

All seems good, right?

I did an additional test with the first 6 set to a 2 minute interval and the last three set to a 1 minute interval.

Again, the screen did not update properly for the first minute. At 1 minute, the last three calendars updated and the screen refreshed with all the info. After two minutes, all 9 calendars appeared to update without error and the screen stayed up-to-date.

So I think this is only a problem with the first run of the fetch function.

sdetweil

@moorevineyard 2 minute fetch is too frequent

it might not be enough time to get and parse

and DO you NEED to see the event you JUST added to the cal someplace else on the screen. when is the event? tomorrow, next week? not 2 minutes from now

moorevineyard

@sdetweil

I definitely agree that 2 minutes is too short for normal operations.

In my examples above, I was setting low times for troubleshooting purposes. I’m expecting to set the fetchInterval to about 24 hours once everything is set up the way I want. But this bug still causes lots of problems.

In particular:

For a new user that is frequently changing settings and styles, every change would require waiting a significant period of time before being able to check if it is acceptable. In my case, it means waiting a full day to see any calendar events again if I want to change the colors of one calendar.

Over time, events or calendars may be added by mistake. As far as I know, the normal way to refresh everything is to restart MagicMirror. Again, in this case, removing something that I do t want displayed means that I can’t see other events for a full day.

Is there another method of refreshing calendar data that would be less disruptive?

sdetweil

@moorevineyard you can set the fetch interval by url too

moorevineyard

I did some more testing and it appears that there may be some sort of rate limit with Google. I haven’t been able to find anything published, but slowing down the fetch process seems to fix the problem.

I’m not super great with JavaScript and don’t know anything about NodeJS, but this is what I did. In the calendar.js file, starting in line 121 and then down around line 158 I changed the following:

                this.config.calendars.forEach((calendar, index) => {  // added index
                        calendar.url = calendar.url.replace("webcal://", "http://");

                        const calendarConfig = {
                                maximumEntries: calendar.maximumEntries,
                                maximumNumberOfDays: calendar.maximumNumberOfDays,
                                pastDaysCount: calendar.pastDaysCount,
                                broadcastPastEvents: calendar.broadcastPastEvents,
                                selfSignedCert: calendar.selfSignedCert,
                                excludedEvents: calendar.excludedEvents,
                                fetchInterval: calendar.fetchInterval
                        };

                        if (typeof calendar.symbolClass === "undefined" || calendar.symbolClass === null) {
                                calendarConfig.symbolClass = "";
                        }
                        if (typeof calendar.titleClass === "undefined" || calendar.titleClass === null) {
                                calendarConfig.titleClass = "";
                        }
                        if (typeof calendar.timeClass === "undefined" || calendar.timeClass === null) {
                                calendarConfig.timeClass = "";
                        }

                        // we check user and password here for backwards compatibility with old configs
                        if (calendar.user && calendar.pass) {
                                Log.warn("Deprecation warning: Please update your calendar authentication configuration.");
                                Log.warn("https://docs.magicmirror.builders/modules/calendar.html#configuration-options");
                                calendar.auth = {
                                        user: calendar.user,
                                        pass: calendar.pass
                                };
                        }

                        /*
                         * tell helper to start a fetcher for this calendar
                         * fetcher till cycle
                         */
setTimeout(() => {             // added
                        this.addCalendar(calendar.url, calendar.auth, calendarConfig);
}, index * 5000);  // 5-second delay for each calendar
                });

This appears to add a 5 second delay to each of the fetches (one fetch occurs every 5 seconds during startup). I added a couple random calendars and now up to 11 seems to work without a problem.

Is it worth adding a configuration option to introduce the delay? Is Github the best place to make these kinds of suggestions?

sdetweil

@moorevineyard yes, github

you could create a pull request with the code solution
maybe just do it if the count is over 4 ?

11*5 = 55 seconds, almost a minute before some calendars display. Some would not like any delay

sdetweil

@moorevineyard would you be willing to test a different fix?

due to some other problems, I added an increased timeout on the fetch function for any node helper using fetch .

I’m interested it if fixes your timeout problem (I set timeout to 30 seconds, but can be extended with an environment variable)…
pull request https://github.com/MagicMirrorOrg/MagicMirror/pull/3660

git branch to see what branch you are on // remember that name (with *)
git status to see if any changed files exist(I think yes on calendar)
git stash if any changed files

AND we need a newer version of node js
node -v
which n
if ( which n returns nothing do , sudo npm install n -g)
then
sudo n 20.18.1

to use that and not break your existing system

in the MagicMirror folder

git fetch origin pull/3660/head:test_timeout
git switch test_timeout
npm run install-mm

then run your system

to go back, in the MagicMirror folder

git checkout original branch name
git stash pop 
sudo n 20.9.0
npm run install-mm