How to load a <script> src = " " </script> into my mirror?

nbrenn

I am trying to have my module load the d3 library to create a table for my csv data. But I am not sure how to type in the line:

src="d3.min.js?v=3.2.8"

(which is wrapped in script brackets), into my module. Right now, nothing shows up on my Mirror aside from the name of the module, datafeed. My code is as follows:

Module.register("datafeed",{
getScripts: function() {
 	return ["d3.min.js"]
 },

 // Define styles.
 getStyles: function() {
 	return ["datafeed_styles.css"];
 },
 

 // Override dom generator.
 getDom: function() {
     d3.text("JANUARY22TEST.csv", function(data) {
             var parsedCSV = d3.csvParseRows(data);

             var container = d3.select("body")
                 .append("table")

                 .selectAll("tr")
                     .data(parsedCSV).enter()
                     .append("tr")

                 .selectAll("td")
                     .data(function(d) { return d; }).enter()
                     .append("td")
                     .text(function(d) { return d; });
         });
 },
});

morozgrafix

Here is a snippet from Module Development documentation. Full details are here: https://github.com/MichMich/MagicMirror/tree/master/modules#getscripts

getScripts: function() {
    return [
        'script.js', // will try to load it from the vendor folder, otherwise it will load is from the module folder.
        'moment.js', // this file is available in the vendor folder, so it doesn't need to be available in the module folder.
        this.file('anotherfile.js'), // this file will be loaded straight from the module folder.
        'https://code.jquery.com/jquery-2.2.3.min.js',  // this file will be loaded from the jquery servers.
    ]
}

I would recommend saving it locally in your module directory and loading it from there, because mirror will stall if it can’t access externally hosted file.
Hope this helps.

nbrenn

Thanks for your response! I tried to change it to the method you described, but it is still isn’t showing anything. Looking at my code, do you know if there is something else keeping it from working?

I am getting an error that it failed to load resource, on the line of my csv. Do I have to specify the path?

morozgrafix

It seems that you are appending table element to the body. If you take a look at HelloWorld sample module you will see a basic structure for getDom method. It needs to return element that will be injected into HTML of the MM when system calls getDom method.

Very basic example would look like this:

getDom: function() {
	var wrapper = document.createElement("div");
	wrapper.innerHTML = "Hello World!";
	return wrapper;
}

When system calls getDom it will get wrapper which would have <div>Hello World!</div> HTML.
If you have a simple CSV file d3 may be an overkill for that, you can probably use a fairly simple javascript to get that done. Here is an example of how you can do it (I didn’t load CSV from external file in that example, I kind of threw it together quickly, but it seems to work): https://jsfiddle.net/morozgrafix/8oz1t9g4/1/

Module development documentation is also pretty good: https://github.com/MichMich/MagicMirror/tree/master/modules
In case your CSV data gets updated while mirror is running, you can also figure out how to periodically update information that you display by looking at other modules.

nbrenn

Thanks for the great response. Your example is very helpful.

I would like to update the CSV while the mirror is running, so I will have to look in to how to accomplsh that.

One question about loading the CSV, if I were to use your example. In your example, you are reading the CSV data from a text area. In my case, how would I read the CSV from a directory on my Raspberry Pi? It looks like I would have to access the file and then save it as the object “csvData”.

It also looks like I am receiving a “http://localhost:8080/home/pi/MagicMirror/modules/datafeed/JANUARY22TEST.csv Failed to load resource: the server responded with a status of 404 (Not Found)” error. The connection to port 8080 is refused (and my mirror is running on Port 8080 and all of my other modules are running fine). Do you know a workaround for this?

Thank you!

morozgrafix

Sorry I’m on mobile right now and it’s not easy to write code samples on the phone.

You will not be able to access your local file via URL like that. ~~You would need to read it from the file system.~~ In default compliments module there is a piece of code that you can modify to achieve that.

EDIT: If you place your JANUARY22TEST.csv file in the directory of your module named datafeed it should be available at this URL: http://localhost:8080/modules/datafeed/JANUARY22TEST.csv

Here is a snippet, but I would recommend looking through the whole module to get a better idea how it works:

	/* complimentFile(callback)
	 * Retrieve a file from the local filesystem
	 */
	complimentFile: function(callback) {
		var xobj = new XMLHttpRequest();
		xobj.overrideMimeType("application/json");
		xobj.open("GET", this.file(this.config.remoteFile), true);
		xobj.onreadystatechange = function () {
			if (xobj.readyState == 4 && xobj.status == "200") {
				callback(xobj.responseText);
			}
		};
		xobj.send(null);
	},

In this case it’s reading JSON file, but you would need to read plain text CSV. There are a few examples that are similar on Stack Overflow and in tutorials online. I will try to dig up an example when I’m back in front of my computer, but I’m sure you can easily search for them.

nbrenn

@morozgrafix Thanks for the response and the support!

This looks very promising solution and I’ll give it a go once I solve one nagging problem. I am receiving the connection refused error when I am connecting to the localhost: 8080:

pi@raspberrypi:~ $ wget --verbose localhost:8080
--2017-01-24 12:11:12--  http://localhost:8080/
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:8080... failed: Connection refused.
Connecting to localhost (localhost)|127.0.0.1|:8080... failed: Connection refused.
pi@raspberrypi:~ $

morozgrafix

@nbrenn is your mirror up and running with npm start or node serveronly? If webserver isn’t running on port 8080 you will get connection refused error when attempting to access it over HTTP.

nbrenn

@morozgrafix I start my mirror with npm start. And my Mirror config.js, I am using Port 8080.

morozgrafix

@nbrenn when your electron MM app shows up on the screen and while it is running, you should be able to do wget http://localhost:8080 without getting connection refused.

nbrenn

@morozgrafix Yep, when the mirror was running, and running the wget http://localhost:8080 command, I connected successfully.

So then when I open up a command terminal while the mirror is running, and run:

wget http://localhost:8080/home/pi/MagicMirror/modules/datafeed/JANUARY22TEST.csv

it says connected to 8080, but returns 404 Not Found.

morozgrafix

@nbrenn that path is incorrect. Try

wget http://localhost:8080/modules/datafeed/JANUARY22TEST.csv

nbrenn

@morozgrafix You got it! The path was the issue. Do you know why I can’t use the path the way I specified?

Now, I’ll go back and try your solution and see if I have any other problems!

Thanks again for your help. It’s much appreciated.

morozgrafix

@nbrenn glad you are getting closer to solving this.

Root directory that is served by webserver that is running on localhost:8080/ is mapped to MagicMirror directory. In most cases and in your case the full path of that is /home/pi/MagicMirror.
By default it is configured to serve you index.html.

So mapping looks kind of like this:

http://localhost:8080					---->	/home/pi/MagicMirror/
http://localhost:8080/index.html			---->	/home/pi/MagicMirror/index.html
http://localhost:8080/modules/module_name		---->	/home/pi/MagicMirror/modules/module_name/
http://localhost:8080/modules/module_name/foo.html	---->	/home/pi/MagicMirror/modules/module_name/foo.html

as you can see from above when you asking for

http://localhost:8080/home/pi/MagicMirror/modules/datafeed/JANUARY22TEST.csv

webserver will attempt to serve you something from

/home/pi/MagicMirror/home/pi/MagicMirror/modules/datafeed/JANUARY22TEST.csv

which doesn’t really exist and that’s the reason why you can’t use absolute file path in the URL. Which also brings up another point: because root directory that is served by webserver is mapped to /home/pi/MagicMirror, webserver can’t serve anything above the /home/pi/MagicMirror directory. (for example you won’t be able to access /home/pi/foo.html).

This is just a small portion of how it works and there are other rules that play role, but hopefully this info shines some light and gives you enough explanation on why your path didn’t work.

morozgrafix

I’ve moved this thread into Development category, since conversation is now mostly related to your module development.

Thanks.

nbrenn

@morozgrafix I am referring back to your jsfiddle example: jsfiddle example, and you created an object called csvData which you refer to in var data = ...

I was wondering how I would create a similar object, except with the file that I have in my directory. I couldn’t seem to find an HTML DOM function that would allow me to load the .csv file. I looked at the examples for the Compliments module, and they were in json. So, I saw that I might be able to use Papa Parse to convert the .csv to json and then follow the similar route as the Compliments.

My d3 approach loads the data from my csv onto my mirror, but the .css doesn’t seem to help it. It doesn’t appear in a table, and is very large (displaying in the upper left part of the mirror) even when I specified it to load in the middle.

For some context, my .csv data table contains two columns of time-series data for which I want to sum together. And then I just want to output a sentence like, “Your sum is XXX.”

Thanks again for the clarifications. I’m slowly wrapping my head around the interaction of the HTML and Javascript, and I think I get confused when I see code split into HTML and Javascript, when the modules are written with it all together in the HTML DOM format.

morozgrafix

It would be very difficult for me to guide you through building entire module without seeing the code and understanding what it is doing. That’s the reason why I’m only able to give you bits and pieces of information. One of the things that can help is for you to create a repo on GitHub with code that you got so far and then I can take a look and possibly offer some suggestions.

In your original post you mentioned that you wanted to display CSV data from the file in HTML table format which seems a bit different from your most recent requirement:

For some context, my .csv data table contains two columns of time-series data for which I want to sum together. And then I just want to output a sentence like, “Your sum is XXX.”

To achieve the above following very rough logic would need to be implemented:

CSV file is created/updated in /home/pi/MagicMirror/modules/datafeed/file.csv
When module is loaded JS creates a variable and reads contents of the CSV file into it (similar to what compliments module does)
CSV data in that variable is parsed, sum of the values in one of the columns calculated and assigned to another variable (sum).
DOM element is created and innerHTML of that element is updated with "Your sum is " + sum
At given config interval steps 1-4 are repeated to get fresh set of data.

Is this close to something that you are trying to do?

(Above steps aren’t accounting for any error handling etc.)
BTW I don’t think you need to convert your CSV to JSON, especially if this is a simple 2 column file. Commenting out or removing xobj.overrideMimeType("application/json"); from complimentFile example should be sufficient for CSV file reading.

Thanks.

morozgrafix

I also updated JSfiddle example with sum calculation logic: https://jsfiddle.net/morozgrafix/8oz1t9g4/2/

nbrenn

@morozgrafix Thanks for your outline. It is very, very helpful and appreciated. So I tried to take what you suggested to turn it into my code, and put it onto GitHub.

I created a GitHub repository that contains my .css, .csv, and datafeed.js. My areas of confusion (which you’ll probably see in the code) are around how I called in the remoteFile in my getDom. I am not sure if I did that correctly. https://github.com/brennn1/Magic-Mirror-Datafeed-Module

My .csv is coming in the format which you’ll see on the Github - so each data point is separated by a new line. So I changed where you had the line.split(",") to line.split("\n").

Do you see anything obvious that will cause me problems? I will go ahead and see what errors I get when I load it onto my mirror.

morozgrafix

@nbrenn perfect!!! i can now see your full code and have better idea what you are trying to achieve.

Your data file is not really a CSV (Comma Separated Values) file, just a text file with values on each line, so you were correct to remove that line.split(",") part. I already spotted a few things where your code will not work correctly. I will take a closer look at it later tonight and hopefully together we can get this resolved.