A pity it does not work for you.
Strange that is only happens after a several hours
But mplayer is rather old, perhaps that is the primary issue.
Still, I tried it on my Pi 3b with 1Gb RAM and it runs well.
It is 3 to 5 seconds lagging.
Like the stream on my P4b with 8Gb.
It also does not seem very memory hungry.
Pi 3b is steaming now for over 2 hours, so I will need to look later again to see if it got messed up.
The Pi 4b is streaming for 2 weeks straight now, without issues.
To me it looks more like a camera issue, but if you say that other players have no issues …
Somehow mplayer does not like your rtsp stream, or it is a combination of factors.
I have found that there is a config file, where you can set options.
Try out some options (need to restart MagicMirror after every change).
For example try nocache first.
Then try cache=65536.
The option nosound is always good (can be set in the MMM-Mplayer options too, if not already done so).
The bandwidth option is just a shot in the dark.
$ vi ~/.mplayer/config
# Write your default config options here!
# No cache
nocache=yes
# Cache size in kBytes (4MB / 8MB / 16MB / 64MB)
# cache=4096
# cache=8192
# cache=16384
# cache=65536
# Do not play/encode sound
nosound=yes
# Bandwidth (10MB / 100MB / 1000MB)
# bandwidth=80000000
bandwidth=100000000
# bandwidth=1000000000
What also can be the case, is that the windowSize is not matching your stream resolution.
This can cause issues as it needs to recode this portion.
Try:
windowSize: { width: 960, height: 540 },
for a 1920x1080 (16:9) stream.
That is all I can think of.
Good luck.