Reverse Lookup MMM-FRITZ-Box-Callmonitor - help needed

AxLed

Hi,
i am interested in extending MMM-FRITZ-Box-Callmonitor with a reverse lookup feature.

I check following tutorial:
https://scotch.io/tutorials/scraping-the-web-with-node-js
The git can be found at: https://github.com/scotch-io/node-web-scraper

I clone and installed it,

after this i modified it, so i can write parsed datas of http://www.dasoertliche.de/?form_name=search_inv&ph=[Phonenumber] in a output.json file.

This is what my server_mod.js looks like:

{
    "title": "1\n\nAutovermietung SIXT \n\n\t\t\t\t\n\t\t\t\t\tStreetsomething 1, 12345 City",
    "adresse": "Streetsomething 1, 12345 City",
    "anrufer": "Autovermietung SIXT"
}

Now i need some help, how to implement it into MMM-FRITZ-Box-Callmonitor, i have following problems:

I still have a static phonenumber for reverse lookup in server_mod.js

url = 'http://www.dasoertliche.de/?form_name=search_inv&ph=[phonennumber]';

To create a output.json file i have to start node server_mod.js (in Module directory) and have to call http://localhost:8081/scrape in my browser.

I think the first steps are made, maybe somebody can give me a hint how to move forward.

AxLED

pinsdorf

I like this idea! @paviro, the developer of MMM-FR´TZ-Box-Callmonitor is certainly the better person to explain, but the idea intrigues me and I had a look at the module’s code on GitHub.

The relevant code for name resolution is in node_helper.js, line 150:

var callInfo = {
    "time": moment(call.Date[0], "DD.MM.YY HH:mm"), 
    "caller": self.getName(call.Caller[0])
};

if (call.Name[0])
{
    callInfo.caller = call.Name[0];
}

When going through the list of callers, the modle calls getName(caller) which is responsible for resolving the caller. This method does in essence this (node_helper.js, line 33):

if (number_formatted in this.AddressBook) {
    return this.AddressBook[number_formatted];
} else {
    //Not in AdressBook return original number
    return number;
}

So it basically checks if the number is in the preloaded address book and if not it returns the original number.

I would put your reverse lookup here. In case the reverse lookup fails simply return the original number as is.

Try to condense the essence of the reverse lookup into a single method (potentially located in its own module), so that it does not require calling the localhost address in browser. Once this runs, I’d cache all numbers that were successfully resolved so that you don’t have to do the lookup again and again for the same number.

Sorry, it is hard to tell more as I know too little about your reverse lookup code.

pinsdorf

I’ve had a look at the node-web-scraper in the Git you reference above.

Line 7 of server.js binds the web scraping functionality to a HTTP Get endpoint http://localhost/scrape . It uses the Express framework to do so, see var app = require('express') and then app.get(...). You find a nice introduction to Express on the project’s website.

app.get('/scrape', function(req, res){
...
  request(url, function(error, response, html){
  ...
  })
})

You may want to extract the essential functionality and then put it into the module. This essential functionality is in the request call in lines 11-33. Maybe make a method getNameByLookup(caller) in node_helper.js of the MMM-Fritz-Box-Callmonitor. This should do the request to the online phone book page, use cheerio for scraping, and extract the relevant information.

Once you have this method, call it inside getName(caller)as described in my previous post. Because you are taking the request out of the callback of app.get you get rid of this hosting on localhost. The express hosting is great for testing scraping functionality, but not so beneficial for a MM module.

I think you’ve solved the tricky part already, i.e. finding the correct data source on the web and scraping the relevant information. Moving this into the module should be comparably easy.

You may want to fork the Git of MMM-Fritz-Box-Callmonitor, make a new branch (e.g. reverseLookup) and work there. This also allows you to channel back your change into the original module by sending a pull request (PR) to the module author. I’m happy to help if you tell me where your Git repo is.

AxLed

@pinsdorf
I am moving forward and getting closer to my target.
With your tips i got rid of the Express framework (calling http://localhost/scrape to start web scraping). I also found a way two show the results in the console and addition writing a file (for my personal debugging).

Right now i have to files:
package.json

{
  "name"         : "MMM-Alex2",
  "version"      : "0.0.1",
  "description"  : "Reverse Lookup Phonenumber.",
  "main"         : "server.js",
  "author"       : "Alex",
  "repository"   : {
    "type" : "git",
    "url"  : "https://github.com/"
  },
  "dependencies" : {
    "request"    : "latest",
    "cheerio"    : "latest"
  }
}

server.js

var request = require('request');
var cheerio = require('cheerio');
var moment = require('moment');
var fs      = require('fs');

var callnr = '0049xxxx';
var url = 'http://www.dasoertliche.de/?form_name=search_inv&ph=' + callnr;

title = '';
adresse = '';
anrufer = 'kein Treffer auf Örtliche.de für '+ callnr;

request(url, function (error, response, html) {
  if (!error) {
	var $ = cheerio.load(html);
	
	$('.left').filter(function(){
        var data = $(this);
        
		title = data.text().trim();
		//adresse = data.children().children().text().trim();
		adresse = data.children().eq(2).text().trim();
		})
		
	$('.name ').filter(function(){
        var data = $(this);
        anrufer = data.text().trim();
		})

    console.log('Anrufer: ' + anrufer);
	console.log('Adresse: ' + adresse);
    //console.log('Title: ' + title);
	console.log('Gesamt: ' + anrufer + ', ' + adresse + ' - '+ moment().format("DD.MM.YYYY HH:mm"));
    
	//Daten fortschreiben
	fs.appendFile('anrufe.txt', moment().format("DD.MM.YYYY HH:mm") + ' Gesamt: ' + anrufer + ', ' + adresse +'\n', function(err){
      console.log('File successfully written!');
    })
	
  }
  else {
    console.log("We’ve encountered an error: " + error);
  }
});

If i put them in a folder (under MagicMirror/modules) an do a npm install in that folder, it will install the depencies.

If in now run node server.js in MagicMirror/modules/FOLDERNAME i will get the result of a static phonenumber in the console and in a textfile in FOLDERNAME.

My next step will be the implementation in MMM-Fritz-Box-Callmonitor, as you suggested.

AxLED

AxLed

Hi,

i tried to implement Reverse lookup, so i extended the package.json file about the needed depencies and did a npm install afterwards, which worked fine.

Now i am struggeling in implementing aditional code in node_helper.js, as i dont know how to debug it. Some bugs are show in the console, as i start MM, but some not.

Is there a way to test node_helper.js by itself?
Or, how do you test new modules / module modifications?

I am looking forward to get some tipps.

Regards AxLED

pinsdorf

Hi @AxLed, great to see your progress!

There are a few remarks on testing that come to mind:

I personally like to develop the module on my Windows PC using VisualStudio Code and additional tooling from the Internet as I need it. For this you have to install the MagicMirror on your PC. @Mykle1 has a nice walkthrough document for doing so.
MM has a development mode. Start MM (with your to be tested module) with the command npm start dev. This starts a browser UI for MM and in addition shows you the loaded HTML, CSS, source code, etc. In the browser’s source window you can set breakpoints and once they get hit, the program execution pauses and you have time to inspect variables. Please note that you don’t have to stop and start MM all the time with the command above. Just change source code in your programming editor, save it, and reload the MM browser (F5). This works both on Raspberry Pi and Windows PC.
If you want to test part of your code in isolation, then you can simply write a small helper program debug.js which calls your test subject’s methods the way you need it. You find this approach in the debug.js of calendar module to test the calendar fetcher component. In this case you do not start MM, but just execute your helper program with node debug.js.
You can write a real (automated) test suite, but this may go a little beyond your current task. A test suite is helpful when you do test driven development and/or when you want to guarantee that a future change does not break expected behaviour.

pinsdorf

Oh, one idea for your module. The calendar has a dedicated calendar fetcher as mentioned in my previous post. This may be interesting for your reverse lookup as well. Other people may want to use other online phone books, e.g. for different countries, and by configuring the right reverse lookup fetcher they could do so. Maybe have a look at the code of the calendar fetcher and the node helper where it is being called. However, IMHO you should first get your module working as it is. This is a suggestion for a future extension.

AxLed

@pirnsdorf

Thanks for the information, so i started one of my MM-Dev-Environments (Ubuntu 16.04) and run MM with npm start dev, unfortunatelly i can not mark node_helper.js, as i only see the MODULNAME.js and the MODULNAME.css.

The part of code i want to debug is in node_helper.js, how can i do this?
Screenshot:

AxLED

pinsdorf

Hmm, this is weird. The original MMM-FRITZ-Box-Callmonitor comes with a node_helper.js. Are you sure you did not delete it by accident resp. forgot to copy it to your dev environment? Moreover package.json and the README.md are missing.

Are you familiar with git? I’d advice that you fork the MMM-FRITZ-Box-Callmonitor into your own GitHub account and make all your changes there. This would also allow me to read your latest code after each git commit. From this forked repository there is an easy way to get your improvement back into the original repository once you are done.

AxLed

Hi pinsdorf,
all mentioned files are in the folder MMM-FRITZ-Box-Callmonitor but on debugging, i only can choose from the ones above (see my screenshot).

I will follow your suggestion with git, if my code adaptation has passed the beta phase, as i still have some problems.