Read the statement by Michael Teeuw here.
MMM-Scrapey - I made something that could be useful?
-
I don’t think there’s any other module out there which will let you scrape content from a webpage for display on MagicMirror, maybe there’s a good reason for that, but I went ahead and built one for myself, with quite a bit of help from ChatGPT (I wouldn’t consider myself a developer, just a tinkerer, so please excuse the code quality!).
I’ve made it quite flexible, so whilst my use case is to scrape live bus departure times from the stop outside my house, I’m sure there will be other use cases …
You specify a URL to scrape, a CSS selector to identify the table on the page, and then some options about what rows and columns you want to include.
I’m not sure if this is ready for public consumption, but if any devs would like to take a look and give me any advice, please do.
-
@AndyHazz Cool! Don’t forget to add it to the module list: https://github.com/MagicMirrorOrg/MagicMirror/wiki/3rd-party-modules 🙂
-
@KristjanESPERANTO No problem, done!
-
@AndyHazz Thanks for this awesome module. I’m also using it to pull bus information and it’s working well. I wondered if there was a way of stripping formatting off the content it scrapes? The table I’m pulling from (https://bustimes.org/stops/250012526) includes hyperlinks in several of the columns, and I’d like to format them as plain text.
Any ideas how that could be achieved?
Thanks again for this, it’s totally made things amazing :-) -
I think I figured it out by adjusting the .js code to return .text rather than .innerHTML on line 88.
This could possibly be something that could be a module option. If I get some time I might consider taking this and doing some work on it, but like you I’m a bit of a ChatGPT coder :-), so I’m not even sure how to copy it!It’s working great for me now though, so thank you again!
