@Chadeus Replying to myself, but wanted this for the community.
It was about fixing the rrule-alt file and I changed it from 1970 to 1900 per the above link. Specifically: /home/pi/MagicMirror/node_modules/rrule-alt/lib/rrule.js
Change: ORDINAL_BASE: new Date(1900, 0, 1), /* changed from 1970 to 1900 */
Also, I generated a fresh private url for my Google Calendar. Not sure if that plays into it or not, but it worked after the changes plus getting a new url.