Automatic checking of all MagicMirror² modules

KristjanESPERANTO

@MZ-BER said in Automatic checking of all MagicMirror² modules:

Great work @KristjanESPERANTO. Really, really like this. Is there a way to also publish the github stars and # of forks? I also would love if there is a way to “like” a module.

Thank you for your appreciation! :-)

karsten13 has already suggested the GitHub API and I have tested it, but if you make requests for 1000 modules you’ll got blocked quickly. I haven’t found another good approach yet.

@MMRIZE said in Automatic checking of all MagicMirror² modules:

I suggest you and the community expand package.json to regularise the details of the module. (recommended but optional)

Storing additional data in the package.json is an interesting idea! But I would like to avoid maintaining data redundantly. So before we tackle this, it would be good to have an overview of the data that is already used for the website and where it is taken from. Plus an additional list of data that is still needed, ideally with ideas on how to collect it. What’s not easy to implement could perhaps be addressed by extending the packge.json.

@mumblebaj said in Automatic checking of all MagicMirror² modules:

My MMM-SweepClock doesn’t have a package.json either as all work done in the MMM-SweepClock.js file. I can add one for completeness and conformity. I also notice I did not have an image. I usually name my images image-1.png etc. so the main image is generally the first one.

At the moment the script has no other source for keywords and the license. And without the license information no image will be taken. If you want an image to be displayed on the website, you would have to specify a free license in the package.json.

@sdetweil said in Automatic checking of all MagicMirror² modules:

my MMM-SleepWake module does not have any UI… so there is no photo…

There is no obligation for an image. But perhaps it would be nice to use an icon that symbolizes motion detection instead of a screenshot. Like this maybe: https://openclipart.org/detail/306403/motion-detection?

karsten13

@KristjanESPERANTO said in Automatic checking of all MagicMirror² modules:

karsten13 has already suggested the GitHub API and I have tested it, but if you make requests for 1000 modules you’ll got blocked quickly. I haven’t found another good approach yet.

2 ideas:

did you try to authenticate with a user before making the api requests? AFAIR we solved a similar problem at work with authentication …
if first idea doesn’t work an ugly solution is to work with e.g. a timer or sleep statement because these infos are not changed very often this could be a long running nightly job

KristjanESPERANTO

@karsten13 said in Automatic checking of all MagicMirror² modules:

did you try to authenticate with a user before making the api requests?

No, I didn’t. Thanks for the hint! I’ll try that! The documentation sounds promising:

Additionally, you can make more requests per hour when you are authenticated.

BKeyport

@KristjanESPERANTO You know, this is turning more and more into a NPM type project.

Perhaps it’s time to work on getting MM into the NPM ecosystem instead?

sdetweil

@BKeyport mm is already there but old

karsten13

@BKeyport

see https://github.com/MichMich/MagicMirror/issues/2876

sdetweil

@karsten13

https://www.npmjs.com/package/magicmirror

BKeyport

I’ve over simplified as I tend to do.

What I was thinking of was more along the lines of how Homebridge is doing things. Full on NPM integrations across the board. GUI installer/maintainer. bonus points for GUI configuration tool on their “plugins” etc. Can still get into the weeds if you want to mess with operations under the hood.

I honestly think it’s time, but I’ve got nowhere near the skill to do it.
MMM-Config on steroids.

The pieces are there - we’re just missing the core.

https://github.com/homebridge/homebridge-raspbian-image/wiki/Getting-Started#step-5-install-and-configure-plugins

Setting the core up would build in a natural module checking system and weed out the unmaintained stuff for new users, because new users won’t want to install stuff that has no GUI settings and/or unmaintained wouldn’t show in the official repository/search anymore.

karsten13

@BKeyport

sounds pretty much like what mmpm does

sdetweil

@karsten13 mmpm doesn’t do config editing
and to do config editing in a reliable way, we need programming standards too.

BKeyport

@sdetweil We already have some standards in place (node_helper.js, naming conventions, package.json, etc) - what’s one more (config-schema.json) for the programmer? Think along the lines of the schema.json file used in MMM-Config.

Programmer creates the GUI settings page, using API into a specific file. It’s up to the core to insert result into the main config file. if GUI file isn’t there, then resort to a web based editor.

https://github.com/homebridge/homebridge/wiki/Verified-Plugins describes the process in homebridge’s case. Provide a reward, and bam - We’ve got easier access to the project, making it more widespread, and it completely eclipses the automatic checking process here.

Reward for the programmer is rather simple. A badge set thusly - everywhere - in the built in plugin manager’s search, on websites, etc.

Example - No GUI settings page file (the dead module):

Example - GUI settings page file (my reworked module with more functionality):

and the resulting config (I believe they use JSON rather than JS):

sdetweil

@BKeyport I believe we use js because you can have comments, json does not support comments

BKeyport

@sdetweil It don’t matter how it’s done, honestly. JS, JSON - Same diff in my book… I’d just love to see it get simpler, and wrap up some of these projects into one.

sdetweil

@BKeyport what we really need is an AI page layout/designer tool. mushing all these disparate css together w the base design is getting beyond most people’s capabilities. I know it’s way beyond mine.

KristjanESPERANTO

Meanwhile, there have been some changes to the module list. E.g.

If you click on a maintainer’s name, you will get a list of all modules by that maintainer :smiley:
Dark mode switch
Style changes
Many module maintainers added screenshots and keywords and fixed listed issues
GitHub stars are displayed
New sort option: number of stars
Module tags are now clickable
Screenshots are now clickable
Optimized mobile view

And some more …

Jalibu

@KristjanESPERANTO are there plans to make this the official linked modules store on the project‘s page?
In my opinion it is already by far better than the existing one. I‘d also vote for making this the source of truth without dependency to the old page as a datasource

KristjanESPERANTO

@Jalibu wrote:

are there plans to make this the official linked modules store on the project‘s page?

I’m definitely open to the idea of turning it into an official part of the project and moving it to MagicMirrorOrg, for example.

However, I have not yet sought dialogue with the core developers in this regard. It would be very interesting to hear if the core developers are open to that idea. @karsten13 , @rejas and @sdetweil, what do you think?

I‘d also vote for making this the source of truth without dependency to the old page as a datasource

The current solution (the list in the wiki) works and does not cause any additional work for the core developers.

However, as soon as we automate the process of creating the new module webpage, a vandal could provoke ugly effects. That’s why a different approach to maintaining the list, one that provides more protection against (intentional and unintentional) vandalism and includes a simple quality check, would make sense.

My concrete suggestion would be to create a pull request template for this purpose.

If we decide that “my” list should become an official part of the MM project, I would spontaneously suggest these rough milestones in the following order:

moving the repository to MagicMirrorOrg.
finding a better way to maintain the source of truth (currently the wiki page).
replacing/redirect references to the old module list.
automating the creation of the module list website.

karsten13

@KristjanESPERANTO said in Automatic checking of all MagicMirror² modules:

@Jalibu wrote:

are there plans to make this the official linked modules store on the project‘s page?

I’m definitely open to the idea of turning it into an official part of the project and moving it to MagicMirrorOrg, for example.

However, I have not yet sought dialogue with the core developers in this regard. It it would be very interesting to hear if the core developers are open to that idea. @karsten13 , @rejas and @sdetweil, what do you think?

I‘d also vote for making this the source of truth without dependency to the old page as a datasource

The current solution (the list in the wiki) works and does not cause any additional work for the core developers.

However, as soon as we automate the process of creating the new module webpage, a vandal could provoke ugly effects. That’s why a different approach to maintaining the list, one that provides more protection against (intentional and unintentional) vandalism and includes a simple quality check, would make sense.

My concrete suggestion would be to create a pull request template for this purpose.

If we decide that “my” list should become an official part of the MM project, I would spontaneously suggest these rough milestones in the following order:

moving the repository to MagicMirrorOrg.

finding a better way to maintain the source of truth (currently the wiki page).

replacing/redirect references to the old module list.

automating the creation of the module list website.

I’m fine with this. The Wiki stuff should not stay the source of truth, this construction is horrible. Maybe we have to wait until we have full control over MagicMirrorOrg …

sdetweil

@KristjanESPERANTO @karsten13

I’m also ok with this style list becoming the current. but I think the order of task needs to change

show how to resolve the integrity problem, without constant intervention
show the automation
then move to org
then redirect links, and seo
and
show how we migrate the list as new requirements come along. is it you that becomes the maintainer?

mumblebaj

@KristjanESPERANTO My 20pence/cents worth, I vote to make this the official page. We need to move with the times and the old page is outdated.