module that displays a certain text from a website, the text on the website is updated daily

[ashishtank]

@Amoniak any error in dev console or PM2/terminal ?

[Amoniak]

@ashishtank i got it now, i will now try to parse the html response to extract the text, any good tips?

[ashishtank]

@Amoniak yes, I checked the response from watch tower in postman and they give data of last 2-3 days or may be from Sunday to this day, so you will need to select the right node. You can use class tabContent as identifier and today’s date for selection of the node. Also check which html parse is good enough, as HTML is very complex to parse you may want to choose right parser which gives speed and is lightweight. i.e. node-html-parser

Other thing you can do is once you have parsed text data from watch tower, you can assign it to poem’s format (title, content and name) and it will just work. later on you can change the UI and text property names if you want.

[sdetweil]

@ashishtank cheerio lib will parse html to json

[Amoniak]

well, I’ll get started then. Keep your fingers crossed.

[Amoniak]

i installed cheerio, but i don’t know how to use it.
do I have to put this now in my “MMM-PoemOfTheDay.js” ?

[sdetweil]

@Amoniak gotta go read the library usage

[lolo]

It gonna be harder than u think as your text is changing daily. Just follow the the pattern.
Every day this is changed( https://wol.jw.org/hr/wol/h/r19/lp-c).
new day new text.
Text is in :
div id=“dailyText”
but if you ignore reference in text as (Mat. 16:24, 25; Ivan 15:20) located in a class it’s gonna be more easier.
And with cheerio u can complete this.

[Amoniak]

so, this is what my NodeHelper.js looks like. like i thought, the screen is black.

const axios = require("axios");
const DetectLanguage = require("detectlanguage");

module.exports = NodeHelper.create({
        socketNotificationReceived: async function(noti, payload) {
                if (noti === "START") {
                        if (payload.updateInterval < 120000) {
                                payload.updateInterval = 120000;
                        }
                        const self = this;
                        (async function displayWatchtower () {
                                const poem = await getWatchtower(payload);
                                self.sendSocketNotification("UPDATE", poem);
                                setTimeout(displayPoem, payload.updateInterval);
                        })();
                }
        }
});

const cheerio = require("cheerio");
const axios = require("axios").default;

const fethHtml = async url => {
  try {
    const { data } = await axios.get(url);
    return data;
  } catch {
    console.error(`ERROR: An error occurred while trying to fetch the URL: ${url}`);
  }
};

const extracWatchtower = selector => {
  const theme = selector
    .find(".tabContent active")
    .find("[class='themeScrp'] > [class='b'])
    .text()
    .trim();

 const text = selector
    .find(".bodyTxt")
    .find("[class='section']")
    .text()
    .trim();

 return { theme,
          text
 };
};

const scrapWatchtower = async () => {
  const watchtowerUrl =
    "https://wol.jw.org/hr/wol/h/r19/lp-c";

  const html = await fethHtml(watchtowerUrl);

  const selector = cheerio.load(html);

  const searchResults = selector("body")
      .find("#dailyText"
  );

  const deals = searchResults.map((idx, el) => {
      const elementSelector = selector(el);
      return extractDeal(elementSelector)
    })
    .get();

  return Watchtower;

};

[ashishtank]

@Amoniak there are lots of missing pieces… add console.log with some message in each function and see how the flow works.