Reverse Lookup MMM-FRITZ-Box-Callmonitor - help needed

[pinsdorf]

I’ve had a look at the node-web-scraper in the Git you reference above.

Line 7 of server.js binds the web scraping functionality to a HTTP Get endpoint http://localhost/scrape . It uses the Express framework to do so, see var app = require('express') and then app.get(...). You find a nice introduction to Express on the project’s website.

app.get('/scrape', function(req, res){
...
  request(url, function(error, response, html){
  ...
  })
})

You may want to extract the essential functionality and then put it into the module. This essential functionality is in the request call in lines 11-33. Maybe make a method getNameByLookup(caller) in node_helper.js of the MMM-Fritz-Box-Callmonitor. This should do the request to the online phone book page, use cheerio for scraping, and extract the relevant information.

Once you have this method, call it inside getName(caller)as described in my previous post. Because you are taking the request out of the callback of app.get you get rid of this hosting on localhost. The express hosting is great for testing scraping functionality, but not so beneficial for a MM module.

I think you’ve solved the tricky part already, i.e. finding the correct data source on the web and scraping the relevant information. Moving this into the module should be comparably easy.

You may want to fork the Git of MMM-Fritz-Box-Callmonitor, make a new branch (e.g. reverseLookup) and work there. This also allows you to channel back your change into the original module by sending a pull request (PR) to the module author. I’m happy to help if you tell me where your Git repo is.

[AxLed]

@pinsdorf
I am moving forward and getting closer to my target.
With your tips i got rid of the Express framework (calling http://localhost/scrape to start web scraping). I also found a way two show the results in the console and addition writing a file (for my personal debugging).

Right now i have to files:
package.json

{
  "name"         : "MMM-Alex2",
  "version"      : "0.0.1",
  "description"  : "Reverse Lookup Phonenumber.",
  "main"         : "server.js",
  "author"       : "Alex",
  "repository"   : {
    "type" : "git",
    "url"  : "https://github.com/"
  },
  "dependencies" : {
    "request"    : "latest",
    "cheerio"    : "latest"
  }
}

server.js

var request = require('request');
var cheerio = require('cheerio');
var moment = require('moment');
var fs      = require('fs');

var callnr = '0049xxxx';
var url = 'http://www.dasoertliche.de/?form_name=search_inv&ph=' + callnr;

title = '';
adresse = '';
anrufer = 'kein Treffer auf Örtliche.de für '+ callnr;

request(url, function (error, response, html) {
  if (!error) {
	var $ = cheerio.load(html);
	
	$('.left').filter(function(){
        var data = $(this);
        
		title = data.text().trim();
		//adresse = data.children().children().text().trim();
		adresse = data.children().eq(2).text().trim();
		})
		
	$('.name ').filter(function(){
        var data = $(this);
        anrufer = data.text().trim();
		})

    console.log('Anrufer: ' + anrufer);
	console.log('Adresse: ' + adresse);
    //console.log('Title: ' + title);
	console.log('Gesamt: ' + anrufer + ', ' + adresse + ' - '+ moment().format("DD.MM.YYYY HH:mm"));
    
	//Daten fortschreiben
	fs.appendFile('anrufe.txt', moment().format("DD.MM.YYYY HH:mm") + ' Gesamt: ' + anrufer + ', ' + adresse +'\n', function(err){
      console.log('File successfully written!');
    })
	
  }
  else {
    console.log("We’ve encountered an error: " + error);
  }
});

If i put them in a folder (under MagicMirror/modules) an do a npm install in that folder, it will install the depencies.

If in now run node server.js in MagicMirror/modules/FOLDERNAME i will get the result of a static phonenumber in the console and in a textfile in FOLDERNAME.

My next step will be the implementation in MMM-Fritz-Box-Callmonitor, as you suggested.

AxLED

[AxLed]

Hi,

i tried to implement Reverse lookup, so i extended the package.json file about the needed depencies and did a npm install afterwards, which worked fine.

Now i am struggeling in implementing aditional code in node_helper.js, as i dont know how to debug it. Some bugs are show in the console, as i start MM, but some not.

Is there a way to test node_helper.js by itself?
Or, how do you test new modules / module modifications?

I am looking forward to get some tipps.

Regards AxLED

[pinsdorf]

Hi @AxLed, great to see your progress!

There are a few remarks on testing that come to mind:

• I personally like to develop the module on my Windows PC using VisualStudio Code and additional tooling from the Internet as I need it. For this you have to install the MagicMirror on your PC. @Mykle1 has a nice walkthrough document for doing so.
• MM has a development mode. Start MM (with your to be tested module) with the command npm start dev. This starts a browser UI for MM and in addition shows you the loaded HTML, CSS, source code, etc. In the browser’s source window you can set breakpoints and once they get hit, the program execution pauses and you have time to inspect variables. Please note that you don’t have to stop and start MM all the time with the command above. Just change source code in your programming editor, save it, and reload the MM browser (F5). This works both on Raspberry Pi and Windows PC.
• If you want to test part of your code in isolation, then you can simply write a small helper program debug.js which calls your test subject’s methods the way you need it. You find this approach in the debug.js of calendar module to test the calendar fetcher component. In this case you do not start MM, but just execute your helper program with node debug.js.
• You can write a real (automated) test suite, but this may go a little beyond your current task. A test suite is helpful when you do test driven development and/or when you want to guarantee that a future change does not break expected behaviour.

[pinsdorf]

Oh, one idea for your module. The calendar has a dedicated calendar fetcher as mentioned in my previous post. This may be interesting for your reverse lookup as well. Other people may want to use other online phone books, e.g. for different countries, and by configuring the right reverse lookup fetcher they could do so. Maybe have a look at the code of the calendar fetcher and the node helper where it is being called. However, IMHO you should first get your module working as it is. This is a suggestion for a future extension.

[AxLed]

@pirnsdorf

Thanks for the information, so i started one of my MM-Dev-Environments (Ubuntu 16.04) and run MM with npm start dev, unfortunatelly i can not mark node_helper.js, as i only see the MODULNAME.js and the MODULNAME.css.

The part of code i want to debug is in node_helper.js, how can i do this?
Screenshot:

AxLED

[pinsdorf]

Hmm, this is weird. The original MMM-FRITZ-Box-Callmonitor comes with a node_helper.js. Are you sure you did not delete it by accident resp. forgot to copy it to your dev environment? Moreover package.json and the README.md are missing.

Are you familiar with git? I’d advice that you fork the MMM-FRITZ-Box-Callmonitor into your own GitHub account and make all your changes there. This would also allow me to read your latest code after each git commit. From this forked repository there is an easy way to get your improvement back into the original repository once you are done.

[AxLed]

Hi pinsdorf,
all mentioned files are in the folder MMM-FRITZ-Box-Callmonitor but on debugging, i only can choose from the ones above (see my screenshot).

I will follow your suggestion with git, if my code adaptation has passed the beta phase, as i still have some problems.

[AxLed]

I almost made it, for debugging i went this way (as i dont know any better so far):

• Clientside: Browser with debugging feature
• Serverside (like node_helper.js): console.log on every important location in the code

My workaround (so far):
I cloned the function getName to getName2 and use Reverselookup of phonenumbers only in getName2.
I changed the call of getName to getName2 in following places:
Line 80-98:

//Incoming call
		monitor.on("inbound", function(call) {
			//If caller is not empty
			if (call.caller != "") {
				self.sendSocketNotification("call", self.getName2(call.caller));
			};
		});

		//Call accepted
		monitor.on("connected", function(call) {
			self.sendSocketNotification("connected", self.getName2(call.caller));
		});

		//Caller disconnected
		monitor.on("disconnected", function(call) {
			//send clear command to interface
			self.sendSocketNotification("disconnected", {"caller": self.getName2(call.caller), "duration": call.duration});
		});
console.log(this.name + " is waiting for incoming calls.");

Reasons why i did this workaround: loadCallList
This function is called initialy, when you start MM and everytime you get a call. I dont understand, why loadCallList goes through all calls from the fritzbox journal in the past (infinitely), no matter what configuration options of the module you choosed. This gives me an "Socket hang up" error on reverse lookup after 100 requests. Now you know why i did this workaround.

Regards

AxLED

[barnosch]

cool feature, i will definitely upgrade to that version, if it is running well
thumbsup
and now the but…
is this really necessary these days?
I don’t know any of my friends or our generation who really publicate their phonenumber in the public phonebook.
Mobile numbers especially not.
So it is only helpful for bigger companies, right?