Is there a way to store secrets in the environment?

tillig · Aug 22, 2019, 10:28 PM

I’d like to put my config.js in a git repo, but I don’t want to store my API keys and such in there. I’ve tried to do something like:

var config = {
  modules: [
    {
      module: "currentweather",
      position: "top_right",
      config: {
        location: "My Location",
        locationID: "1234567", 
        appid: process.env.APIKEY
      }
    }
  ]
};

…but that always yields an error where the config.js is seen as invalid and everything defaults.

I’ve tried modifying it after the fact in a little code block after the var config ends but while I’ve updated the data and can console.log it, it seems to be too late for the change to have taken effect. The current weather still sees the placeholder.

var config = {
  modules: [
    {
      module: "currentweather",
      position: "top_right",
      config: {
        location: "My Location",
        locationID: "1234567", 
        appid: "OPENWEATHER_API_KEY"
      }
    }
  ]
};

if (!process.env.OPENWEATHER_API_KEY) {
  console.log("You must define the OPENWEATHER_API_KEY for weather support.");
} else {
  var owApiKey = process.env.OPENWEATHER_API_KEY;
  console.log("Updating modules that require the OPENWEATHER_API_KEY.");
  config.modules.forEach(function (mmModule) {
    if (mmModule.config && mmModule.config.appid == "OPENWEATHER_API_KEY") {
      console.log("- " + mmModule.module);
      mmModule.config.appid = owApiKey;
    }
  });
}

I feel like this is a n00b sort of Node.js question and I’m just missing something obvious.

tillig · Aug 23, 2019, 9:48 PM

I figured it out.

I am seeing this issue where, as it turns out, process environment variables are not actually carried over to the Electron app. Instead you have to request the variables from the remote process.

In the issue it recommends doing something like this:

const electron = window.require('electron');
const remote = electron.remote;
var x = remote.process.env["MY_VAR"];

In config.js it appears require is not defined, nor is window.require, at least when running in the Electron context.

Some of the challenge I’m seeing I think is that the same config.js file is getting used in a Node.js/server context and in an Electron app context. I sort of feel like the Electron app should be requesting the configuration from the /config endpoint of the server rather than doing its own JS parsing, but I’m gathering that’s not actually what’s happening here.

To get this to work, there’s some careful hackery where:

You need to enable the nodeIntegration feature on Electron. This is off for security reasons by default, however I’m not super concerned that someone on my local network will be hacking my Electron-based magic mirror app. Doing this allows window.require to bring in the electron module, which you need to read the remote process.
You need to have conditional logic that either uses the current process environment (for the server side) or uses electron.remote.process (for the client side).

If you do that, you can get environment variables to work. It will look something like this:

var remote = null;
if (typeof window !== "undefined") {
  remote = window.require("electron").remote;
}

var config = {
  address: "localhost",
  port: 8080,
  ipWhitelist: ["127.0.0.1", "::ffff:127.0.0.1", "::1"],
  language: "en",
  timeFormat: 24,
  units: "imperial",

  electronOptions: {
    webPreferences: {
      nodeIntegration: true
    }
  },

  modules: [
    {
      module: "currentweather",
      position: "top_right",
      config: {
        location: "Hillsboro",
        locationID: "5731371", //ID from http://bulk.openweathermap.org/sample/city.list.json.gz; unzip the gz file and find your city
        appid: "OPENWEATHER_API_KEY"
      }
    },
    {
      module: "weatherforecast",
      position: "top_right",
      header: "Weather Forecast",
      config: {
        location: "Hillsboro",
        locationID: "5731371", //ID from http://bulk.openweathermap.org/sample/city.list.json.gz; unzip the gz file and find your city
        appid: "OPENWEATHER_API_KEY"
      }
    }
  ]
};

var owApiKey = null;
if (typeof process !== "undefined" && process.env.OPENWEATHER_API_KEY)
{
  // process is undefined in the Electron app.
  owApiKey = process.env.OPENWEATHER_API_KEY;
}
if (remote && remote.process.env.OPENWEATHER_API_KEY)
{
  // remote is null if the Electron nodeIntegration value isn't set to true.
  owApiKey = remote.process.env.OPENWEATHER_API_KEY;
}

if (!owApiKey) {
  console.log("You must define the OPENWEATHER_API_KEY environment variable for weather support.");
} else {
  console.log("Updating modules that require the OPENWEATHER_API_KEY.");
  config.modules.forEach(function (mmModule) {
    if (mmModule.config && mmModule.config.appid == "OPENWEATHER_API_KEY") {
      console.log("- " + mmModule.module);
      mmModule.config.appid = owApiKey;
    }
  });
}

/*************** DO NOT EDIT THE LINE BELOW ***************/
if (typeof module !== "undefined") {
  module.exports = config;
}

I will probably refactor that into a little method in here to make it easier to use and clean up some of the redundancy, but that’s what you have to do.

A Former User · Aug 22, 2019, 11:08 PM

@tillig
maybe, this?
https://www.twilio.com/blog/2017/08/working-with-environment-variables-in-node-js.html

tillig · Aug 23, 2019, 8:42 PM

Well, maybe it’ll help to be more specific about what I have. I am able to read environment variables but it seems there’s an order of execution thing that I’m missing.

I have my config.js as such:


var config = {
  address: "localhost",
  port: 8080,
  ipWhitelist: ["127.0.0.1", "::ffff:127.0.0.1", "::1"],
  language: "en",
  timeFormat: 24,
  units: "imperial",

  modules: [
    {
      module: "compliments",
      position: "lower_third"
    },
    {
      module: "currentweather",
      position: "top_right",
      config: {
        location: "Hillsboro",
        locationID: "5731371",
        appid: "OPENWEATHER_API_KEY"
      }
    },
    {
      module: "weatherforecast",
      position: "top_right",
      header: "Weather Forecast",
      config: {
        location: "Hillsboro",
        locationID: "5731371", 
        appid: "OPENWEATHER_API_KEY"
      }
    }
  ]
};

// Replace the environment secrets - this doesn't seem to work if
// you just call it inline in the JSON.
if (!process.env.OPENWEATHER_API_KEY) {
  console.log("You must define the OPENWEATHER_API_KEY for weather support.");
} else {
  var owApiKey = process.env.OPENWEATHER_API_KEY;
  console.log("Updating modules that require the OPENWEATHER_API_KEY.");
  config.modules.forEach(function (mmModule) {
    if (mmModule.config && mmModule.config.appid == "OPENWEATHER_API_KEY") {
      console.log("- " + mmModule.module);
      mmModule.config.appid = owApiKey;
    }
  });
}
config.modules.forEach(function(mmModule) {
  console.log(mmModule.config);
});

/*************** DO NOT EDIT THE LINE BELOW ***************/
if (typeof module !== "undefined") {
  module.exports = config;
}

The goal is that I can read the environment variable OPENWEATHER_API_KEY and swap it in accordingly.

When I run the Magic Mirror, I do this:

OPENWEATHER_API_KEY=abcd1234
export OPENWEATHER_API_KEY
npm start

When I watch the logs, I can see that the appropriate weatherforecast and currentweather modules are picked up. Further, it appears the configuration is updated:

Updating modules that require the OPENWEATHER_API_KEY.
- currentweather
- weatherforecast
undefined
{ location: 'Hillsboro',
  locationID: '5731371',
  appid: 'abcd1234' }
{ location: 'Hillsboro',
  locationID: '5731371',
  appid: 'abcd1234' }

(The ‘undefined’ in there is because ‘compliments’ doesn’t have configuration.)

So it seems I’m configuring things right.

However, if I run this with npm start dev, I see the requests going out to OpenWeather aren’t using my environment key:

GET https://api.openweathermap.org/data/2.5/weather?id=5731371&units=imperial&lang=en&APPID=OPENWEATHER_API_KEY 401 (Unauthorized)

It’s still using the hardcoded placeholder that wasn’t updated. If I manually put my API key right into the JSON object, this works… but that’s what I’m trying to avoid by using the environment.

I did try doing something like this:

var config = {
  address: "localhost",
  port: 8080,
  ipWhitelist: ["127.0.0.1", "::ffff:127.0.0.1", "::1"],
  language: "en",
  timeFormat: 24,
  units: "imperial",

  modules: [
    {
      module: "weatherforecast",
      position: "top_right",
      header: "Weather Forecast",
      config: {
        location: "Hillsboro",
        locationID: "5731371", 
        appid: process.env.OPENWEATHER_API_KEY
      }
    }
  ]
};

However, using process.env right in the JSON object causes some sort of error that results in the configuration being seen as invalid.

I even temporarily modified the main app.js just after it parsed config to log all the module configurations and it was showing the correct value - my replaced value rather than the placeholder.

I’m just super stumped as to what’s up. Since I see the request in Chrome dev tools, does that mean it’s the client part that’s making the request? Is there a problem with how the configuration is passed to the client?

tillig · Aug 23, 2019, 8:44 PM

Viewing http://localhost:8080/config/ while it’s running, the configuration looks right to me with all the appid values replaced, no placeholders in sight.

tillig · Aug 23, 2019, 9:48 PM