help developing an outofmilk.com module

mchrdk

Hi,
I´m planning on migrating a windows based home-control-screen to magicmirror and so far i have found all the modules i need, except for a module for the website out of milk.

Since my old solution already has the info on how to get the data from the website, i dont think i would be a big problem.

The procedure right now is to call the mainsite https://outofmilk.com/ to get a session, and then call the /loginsite with post data containing credentials.
This request returns a logincookie that you can use to get a shoppinglist from the api.

im mostly a devops guy using powershell and other basic scripting languages, which makes the whole framework a big bite to take in at once.

Do you have any good pointers of modules i can look into that uses the same way of getting data?

i can code i already have in powershell if anyone is interrested.

gauravsacc

Reverse engineer
Looks like they have data exposed via APIs. See screenshot from chrome->devtools->network log
You just need to login and set the cookie and make the API calls.
0_1532615531053_Screen Shot 2018-07-26 at 10.28.39 AM.png

mchrdk

I have all of that done already, im using the data in another application, done in powershell, i just need a bit of beginners guidance on how to get a magic mirror module to access the data, as i have never done anything in js before.

mchrdk

#quick mockup in the entirely wrong language

$csrftoken = (($firstrequest.InputFields|where {$_.name -eq “csrfmiddlewaretoken”}).value)[0]
$username = “insertemailhere”
$password = “insertpasswordhere”
$listid = “find listhere”
$body = “csrfmiddlewaretoken=$csrftoken&login-email=$username&login-password=$password&btnLogin=Submit”
$headers = @{“Cookie”=“csrftoken=$csrftoken”; “Origin”=“https://www.outofmilk.com”; “Accept”=“application/json, text/javascript, /; q=0.01”; “Referer”=“https://www.outofmilk.com/ShoppingList.aspx?signin=1”; “X-CSRFToken”="$csrftoken"; “X-Requested-With”=“XMLHttpRequest”; “DNT”=“1”}
$secondrequest = Invoke-WebRequest -WebSession $Session -Uri “https://www.outofmilk.com/Login” -Method “POST” -Headers $headers -ContentType “application/x-www-form-urlencoded” -Body $Body
$body = “{"listID”:$listid,"listType":"shopping"}"
$thirdrequest = Invoke-WebRequest -WebSession $Session -Uri “https://www.outofmilk.com/Services/GenericService.asmx/GetItems” -Method “POST” -Headers $headers -ContentType “application/json; charset=UTF-8” -Body $body

@mchrdk I didn’t look inside deeply, but how about using ”curl”?

@mchrdk or you can handle request and response with ’express’ and ’http’ or ’request’ node modules.

mchrdk

okay, i see - i have been reading up on the http and request modules now.

it should be possible for me to figure out how to get it to output the data directly in the console, and then i can tackle the presentation part of the module later.

i have been looking at a lot of sourcecode for other mm modules and i cant seem to find any other projects that handle multiple requests like this directly, am i mistaken?

doubleT

Uh, interesting topic. I like API calls. Back when I started, my main issue was understanding the node_helper.js – and that it was neccessary.
I’m still struggling to make sense of your token complexity. Apart from that, your files would be something like this:

MMM-OutOfMilk.js:

Module.register("MMM-OutOfMilk",{
    defaults: {
        // your defaults
    },

    start: function() {
        var payload = "start"; // example
        this.sendSocketNotification("GetMyShoppingList", payload);
    },

    socketNotificationReceived: function(notification, shoppingList) {
        if (notification === "ShoppingListRecieved") {
            this.handeShoppingList(shoppingList);
        }
    },

    handleShoppingList: function(shoppingList) {
        // do something with your shopping list
    }
}

node_helper.js:

var NodeHelper = require("node_helper");
var request = require("request"); // needed? see below
var fs = require("fs"); // needed? see below
var timer = 0; // needed? see below "I added a timer"
var token;

module.exports = NodeHelper.create({
    start: function() {
    },
    socketNotificationReceived: function(notification, payload) {
        if (notification === "GetMyShoppingList") {
            // start the logic to recieve the shopping list
            // get token?
            // make call
            this.getShoppingList(payload);
        }
    },
    getToken: function(payload) {
        // however you get your token
        getShoppingList(payload);
    },
    getShoppingList: function(payload) {
        var self = this;
        var source = ; // your source URL
        if (token) {
        	request({
	            url: source,
	            json: true
	        }, function (error, response, body) {
	            if (!error && response.statusCode === 200) {
	                self.sendSocketNotification("ShoppingListRecieved", body);
	            }
	            else {
	            	// check error if it's a token issue
	            	// if so, get new token
	            	self.getToken(payload);
	            }
	        })
        }    
    }
}

From another project I know that sometimes a token can be used for a while. In that case I’d directly call the getShoppingList and only on an error that read “wrong token” or something like that, I’d call getToken which saves the token to a global variable and calls back getShoppingList to try again. That way you don’t have to get a new token for every call.

Alternatively?

    getShoppingList: function(token) {
        var self = this;
        var source = ; // your source URL
        var rawdata = fs.readFileSync(source);
        var history =  JSON.parse(rawdata);
        self.sendSocketNotification("ShoppingListRecieved", history);
    }

I’d also use a timer to request data on the first call and after that automatically only every full hour:
So, in the node_helper.js, in the socketNotificationReceived I don’t forward the payload to this.updateTimer(payload) instead of getToken or getShoppingList to do this:

    updateTimer: function(payload) {
        var self = this;
        var d = new Date();
        var h = d.getHours();
        var min = d.getMinutes();
        var sec = d.getSeconds();
        if (timer === 0) {
            timer ++; // prevent unnecessary timer by double calls
            this.getShoppingList(payload);

            if((min == '00') && (sec == '00')){
//              console.log(h + ":" + min + ":" + sec + " - update and wait 1 hour");
//              console.log("restart timer");
                setTimeout(() => {
                    this.clearTimer(payload);
                }, 60*1000*60); // 60 sec * 1000 ms = 1 min * 60 = 1 hour
            } else {
//              console.log(h + ":" + min + ":" + sec + " - update waits for " + (59-min) + " min and " + (60-sec) + "sec");
//              console.log("restart timer");
                setTimeout(() => {
                    this.clearTimer(payload);
                }, ((60-sec)*1000)+(60*1000*(59-min)));
            }
        }
        else {
//          console.log("timer already running, data displayed outside of timer run");
            this.getShoppingList(payload);
        }
    },

    clearTimer: function(payload) {
        timer --;
        this.updateTimer(payload);
    }

For one of my modules I couldn’t use fs.readFile... or request because of CORS issues. In that case I had to use a php proxy.
I’d strongly suggest trying to solve this with fs.readFile... or request but just for the sake of completeness and maybe to help you understand:

In order to use PHP within nodeJS via Child Process Dependency you have to have PHP installed and in your PATH!
You can read how I used it, here: https://forum.magicmirror.builders/topic/5830/call-api-no-cors-used-to-do-it-with-php-proxy/9