MMM-CalendarEXT2 - Calendar Read Failing When Time Value Is Missing from ics file

edd189

@sdetweil Well, this is quickly becoming a bugabear!

The error is passed all the way from ical.js (found here: https://github.com/kewisch/ical.js) via a call through iCalExpander (found here: https://github.com/mifi/ical-expander) on node_helper.js line 137.

Line 1107 in time.js (sub-module of ical.js):

static fromDateTimeString(aValue, prop) {
if (aValue.length < 19) {
throw new Error(
‘invalid date-time value: "’ + aValue + ‘"’
);
}

This seems odd to me. I’m working with two different ics files and both of them present a problem for these four lines of code. But my computer, iPhone, and other MMM modules handle the ics file fine. Is this ics.js not used by other calendar modules? Did the accepted format for an ical file change at some point since CX2 was developed?

Acceptable format of an ics entry lists the time as an optional component: https://datatracker.ietf.org/doc/html/rfc5545#section-3.8.2

Time to dig more into their methods I suppose…

sdetweil

@edd189 if the time is not specified, the event is a full day event.

spec hasn’t changed, but the ics providers have changed their data, outside the spec.

MS in particular has started using the custom timezone feature, and mapping all the events to their custom(overlapping with IANA based standard timezones)

all this breaks a bunch of stuff…

only the calendar modules use the ics parser. ext2 had its own. MM default uses node-ical

edd189

@sdetweil Looks like the ics.js branch being used by CX2 had a report on this issue around a year ago with no fix. :(

https://github.com/kewisch/ical.js/issues/515

sdetweil

@edd189 yep, stuff gets old, people get tired, or move on… or it becomes too much work to keep it up to date

edd189

@sdetweil

from line 1142:
ICAL.Time.fromString = function fromString(aValue) {
if (aValue.length > 10) {
return ICAL.Time.fromDateTimeString(aValue);
} else {
return ICAL.Time.fromDateString(aValue);
}
};

Looks like time.js within (https://github.com/kewisch/ical.js) is already trying to do something different with the entry if the length is less than 10. But that count seems off to me – it should never make the call to line 1107 (fromDateTimeString) at all. It should be using line 1083 (fromDateString).

sdetweil

@edd189 you could

console.log("aValue='"+aValue+"'")

to check its contents (in the mm startup output)

edd189

@sdetweil

I tried to put

console.log(‘[CALEXT2] calendar: >> error on line 1111’);

within the time.js file, found embedded within the CX2 directory. It didn’t output anything. Will a sub-rountine embedded a few layers down still output to the console? Do I need to reinstall or re-link anything?

sdetweil

@edd189 generally yes, regardless of where it is…

no, no need to do any compile

sory, forgo this darned forum changes quotes not in a code block …

make sure they are the straight up and down ones
look back at my example

edd189

@sdetweil Doh, the error reporter already shows me the aValue.

aValue = 2022-01-01T::

By my count, that is 13 characters. Let me see if simply changing 10 to 13 in the above code fixes it.

edd189

@sdetweil Is this right?

ICAL.Time.fromString = function fromString(aValue) {
if (aValue.length > 10) {
return ICAL.Time.fromDateTimeString(aValue);
Log.log([CALEXT2] calendar: >> greater 10);
Console.log($avalue.length );
} else {
return ICAL.Time.fromDateString(aValue);
Log.log([CALEXT2] calendar: >> less 10);
Console.log($avalue.length );
}
};

sdetweil

@edd189 said in MMM-CalendarEXT2 - Calendar Read Failing When Time Value Is Missing from ics file:

Log.log([CALEXT2] calendar: >> greater 10);
Console.log($avalue.length );

no… those new statements are after the return… so will never be executed

Log.log and console.log are the same here …

edd189

@sdetweil

Thanks for your help. Not sure I’m finding the right place where the call is made to fromDateTimeString. I found a few more instances in a file under the build directory. I was previously only looking under the lib directory.

Not sure how I missed it, but I need to take a break for the evening. Pick it back up later.

edd189

@sdetweil

Look at file design.js here: https://github.com/kewisch/ical.js/tree/main/lib/ical

What does line 401 do? The call on line 403 is where I get the error. I’m not familiar enough with java to know what decorate means.

sdetweil

@edd189 sorry. no idea

edd189

@sdetweil

Ok, plan B – read the ICS files, save them locally, append the time to the DTSTART and DTEND fields.

Create a script to do this automatically every 24 hours.

Wish me luck! I’m in over my head, but its kinda fun.

Any advice? I am running this on linux, so figured I can just run a cronjob every 24 hours. What’s the best service to language to use to write such a simple script?

sdetweil

@edd189 people write in python, I do most of mine in bash shell.

MMRIZE

Sorry for late reply. I was outside in holidays. I’ll study what your issue is tomorrow.

edd189

@sdetweil

I went with an sh script. I run this every morning at 3am. Does the trick by fixing the calendar file (appends a made up time to the end where its missing) and saving locally. I had to point CX2 to a local file instead of the webcal link, but that is no problem.

The worst part was that the two calendars had slightly different formatting. The US Holidays had some sort of hidden return character at the end of the string, and it took me forever to figure out how awk should deal with that.

#!/bin/sh

#download files
curl “https://www.calendarlabs.com/ical-calendar/ics/76/US_Holidays.ics” --output us_holidays_raw.ics
curl “https://api.team-manager.gc.com/ics-calendar-documents/user/6e0678b2-2e99-44ce-9e3a-e32ac9ff6e78.ics?teamId=secret_id_string&token=secret_id_string” --output baseball_raw.ics

#clean files
awk ‘{if(($1 ~ /DTSTART/) && length($1)==16) {print $0"T050000Z"} else {print $0}}’ dirtdawgs_raw.ics > baseball_awk1.ics
awk ‘{if(($1 ~ /DTEND/) && length($1)==14) {print $0"T195959Z"} else {print $0}}’ dirtdawgs_awk1.ics > baseball_awk2.ics

awk ‘{if(($1 ~ /DTSTART/) && length($1)==17) {print substr($0,1,16)“T050000Z”} else {print $0}}’ us_holidays_raw.ics > us_holidays_awk1.ics
awk ‘{if(($1 ~ /DTEND/) && length($1)==15) {print substr($0,1,14)“T195959Z”} else {print $0}}’ us_holidays_awk1.ics > us_holidays_awk2.ics

#copy and delete temp files
/bin/cp us_holidays_awk2.ics us_holidays.ics
/bin/cp baseball_awk2.ics baseball.ics
/bin/cp us_holidays.ics /home/edd/MagicMirror/config/us_holidays.ics
/bin/cp dirtdawgs.ics /home/edd/MagicMirror/config/baseball.ics
rm us_holidays_awk1.ics
rm us_holidays_awk2.ics
rm baseball_awk1.ics
rm baseball_awk2.ics

exit

sdetweil

@edd189 nice work!

MMRIZE

Sorry for my late reply.

First, I’m apologize for neglecting management. I was so tired to maintain it.

Second,

DTSTART:20220101

is not a valid format by specification(RFC 5545). The default value of DTSTART MUST be date-time format({DATE + “T” + TIME) and, when DATE format is used, it SHOULD be described.

The valid format of this case would be;

DTSTART;VALUE=DATE:20220101

However, many ical providers don’t follow standard rules, and many ical parsers generally are parsing some non-standard formats. Unfortunately in this case, the parser that this module depends on, doesn’t accept this kind of non-standards.
The parser (kewisch/ical.js) was formerly maintained by Mozilla, so I thought this parser would be promising and better than node-ical of default MM’s caledar lays on. Well, somehow it would be better. Still ical.js is more popular, more times downloaded than node-ical.

But there is no perfect thing. I was tired to report/fix all the bugs which are not derived from me, so that’s why I gave up and delegated ical parsing to the other(default calendar/MMM-GoogleCalendar, …) in CX3.

Anyway, what I suggest is, how about using different ics for US holidays? Especially if only this calendar would be a problem.