MMM-AssistantMk2

[A Former User]

ADDED
Step-by-step guide about custom gaction to make a timer feature on wiki page.
https://github.com/eouia/MMM-AssistantMk2/wiki/Tip%3AMaking-Timer-with-gAction

[Davido]

@Sean : First of all, great Work from your side, my Assistant is working, but I have a question: Is it possible, that the answer box from google could stay a little bit longer onscreen? I also tried your custom.css customization, but it doesn’t work, changing it in MMM-AssistantMk2.css was successful, any suggestions?

@Lowis : you wrote that your Radio is running, i was also successful, but are you also able to stop it with the voice command? My Mic reacts on saying the Hotword, but then nothing happens. Only a reboot stops the Radio

[ejay-ibm]

@davido I think the option you are looking for regarding answer of assistant duration on screen is :
“screenDuration”: 2000 // here for 2 seconds

hope this helps

Ejay

[bachoo786]

@sean hello mate

i want to install google assistant on my rpi 3b+. i have connected a bluetooth speaker and was wondering if it would work? i have tried in the past with my rpi2b but couldnt just install assistant.

also is there a ste[ by step guide on how to install assistant?

many thanks god bless.

[A Former User]

@bachoo786 bt speaker will work. Personally I recommend.
And for instructions, read the github documents. And leave issues what trouble tou have. I’ll support.

[Lowis]

@davido could be that the radio is too loud . my running at 50% volume and it works.
Hotword is “webradio” for start and stop … next hotword is “next station”

At the moment i have problems with the sound config. Mic is USB MIC and Speaker HDMI from my monitor.
Assistant voice is distorted but radio is clean

Update: i deleted all ALSA settings in /etc/asound.conf and ~/.asoundrc and Voice works perfekt.
in raspi-config i do this setting > Force HDMI

[Davido]

@ejay-ibm :
i will try your setting, thx for your answer

@Lowis :
i tried it with 30% and the same result, also with the mic in front of my mouth, no action to stop … the mic onscreen reacts to my voice, but the input field leaves blank, and there is also no timeout for another input

[bachoo786]

@Sean I have installed assistantmk2 correctly and have also installed hotword. However I get a blank screen but when I say ok google everything works fine.

I have already been through the “electron-rebuild” step, this is what I did:

pi@raspberrypi:~/MagicMirror/modules/MMM-AssistantMk2 $ npm install --save-dev electron-rebuild
+ electron-rebuild@1.8.4
updated 1 package and audited 617 packages in 12.623s
found 0 vulnerabilities

pi@raspberrypi:~/MagicMirror/modules/MMM-AssistantMk2 $ ./node_modules/.bin/electron-rebuild
✔ Rebuild Complete

but still no joy.

and this is what I have in my config.js

{
    module: "MMM-AssistantMk2",
    position: "top_right",
    config: {
      record: {
        recordProgram : "arecord",
        device        : "plughw:1",
      },

      notifications: {
        ASSISTANT_ACTIVATED: "HOTWORD_PAUSE",
        ASSISTANT_DEACTIVATED: "HOTWORD_RESUME",
      },
    }
  },
  {
    module: "MMM-Hotword",
    config: {
      record: {
        recordProgram : "arecord",
        device        : "plughw:1",
      },
      autostart:true,
      onDetected: {
        notification: function (payload) {
          return "ASSISTANT_ACTIVATE"
        },
        payload: function (payload){
          return {
            profile: payload.hotword
          }
        }
      },
    },
  },

what I did notice that if I was to remove the MMM-Hotword from my config i.e. the last part of the code i.e.

{
    module: "MMM-Hotword",
    config: {
      record: {
        recordProgram : "arecord",
        device        : "plughw:1",
      },
      autostart:true,
      onDetected: {
        notification: function (payload) {
          return "ASSISTANT_ACTIVATE"
        },
        payload: function (payload){
          return {
            profile: payload.hotword
          }
        }
      },
    },
  },

the mirror starts working fine.

Also in the hotword git I have followed all the possible solutions and also the main one which is :

cd ~/MagicMirror/modules/MMM-Hotword/node_modules/snowboy
nano package.json ## Or open the file with any TEXT editor
Then, find this and replace From

“node-pre-gyp”: “^0.6.30”
To

“node-pre-gyp”: “^0.12.0”
Then retry npm install again in the module directory again.

I have no joy buddy. Please help me

[bachoo786]

Hi @Sean

So I managed to get it working and everything is fine. However I will be grateful if you could please help me with the following queries:

1). When i say jarvis or smart mirror, the assistantmk2 module takes 1 to 2 seconds to respond i.e. the ding bell I mean after which I can ask any question. How can I remove this lag and make it instant? I have seen other people dont have this issue.

2). How do I get the assistantmk2 to play youtube videos on my mirror by asking it to play something from youtube?

Thanks.

This is my config:

{
    module: "MMM-AssistantMk2",
    position: "top_right",
    config: {
      deviceLocation: {
                        coordinates: { // set the latitude and longitude of the device to get localized information like weather or time. (ref. mygeoposition.com)
                        latitude: 51.597442, // -90.0 - +90.0
                        longitude: -0.338839, // -180.0 - +180.0
          }
        },
      record: {
        recordProgram : "arecord",
        device        : "default",
      },
      play: { // Full values are in `FOR EXPERTS` section.
              playProgram: "mpg321", // recommended.
      },
      notifications: {
        ASSISTANT_ACTIVATED: "HOTWORD_PAUSE",
        ASSISTANT_DEACTIVATED: "HOTWORD_RESUME",
      },
    }
  },
  {
    module: "MMM-Hotword",
    config: {
      record: {
        recordProgram : "arecord",
        device        : "default",
      },
      autostart:true,
      onDetected: {
        notification: function (payload) {
          return "ASSISTANT_ACTIVATE"
        },
        payload: function (payload){
          return {
            profile: payload.hotword
          }
        }
      },
    },
  },

[Davido]

@bachoo786

to 1) which Hardware are you using?
to 2) i say the Hotword and then “play on youtube …” and it is working

[bachoo786]

@Davido I am using raspberry pi 3b+

[Davido]

@bachoo786 :

i am using a normal rPi 3B without + and the answering time is really fast … maybe some other Module which is interacting?

[bachoo786]

@Davido nope nothing is interacting with my assistantmk2 module.

[bachoo786]

@Davido can you share your config for both mmm-assistant and mmm-hotword please?

[Davido]

@bachoo786

I hope this will help

{
  module: "MMM-AssistantMk2",
  position: "top_right",
  config: {

    // --- ESSENTIALS / modifying for your environment might be needed.


    deviceLocation: {
      coordinates: { // set the latitude and longitude of the device to get localized information like weather or time. (ref. mygeoposition.com)
        latitude: 47.809490, // -90.0 - +90.0
        longitude: 13.055010, // -180.0 - +180.0
      },
    },

    defaultProfile: "default", // If you have several profiles and want to set one of them as default profile, describe here.

    profiles: {
      "default" : { // profile name.
        profileFile: "default.json", // profile file name.
        lang: "de-DE"
        //currently available (estimation, not all tested):
        //  de-DE, en-AU, en-CA, en-GB, en-US, en-IN
        // fr-CA, fr-FR, it-IT, ja-JP, es-ES, es-MX, ko-KR, pt-BR
        // https://developers.google.com/assistant/sdk/reference/rpc/languages
      },
      /* Add your other profiles here, if exists.
      "other_profile" : {
        profileFile: "other.json",
        lang: "de-DE"
      }
      */
    },

    record: { // Full values are in `FOR EXPERTS` section.
      recordProgram: "arecord",  // Defaults to "arecord" - also supports "rec" and "sox"
      device: null        // recording device (e.g.: "plughw:1")
    },

    play: { // Full values are in `FOR EXPERTS` section.
      playProgram: "mpg321", // recommended.
    },


    // --- OPTIONAL / not important but customizable for your usage


    responseVoice: true, // If available, Assistant will response with her voice.
    responseScreen: true, // If available, Assistant will response with some rendered HTML
    responseAlert: true, // If available, Assistant will response with Alert module of MM
    // Sometimes, any response might not be returned. responseAlert is useful for displaying error.

    screenZoom: "80%", // Adjust responseScreen to your mirror size.
    screenDuration: 2000, // milliseconds. How long responseScreen will be shown after speech.
    //If you set 0, Screen Output will be closed after Response speech finishes ASAP.

    youtubeAutoplay: true, //If set as true, found Youtube video will be played automatically.
    pauseOnYoutube:true, //If set as true, You cannot activate Assistant during youtube playing. Recommended for the performance (Because permanent hotword detecting might make performance lower)

    useWelcomeMessage: "", //Try "brief today" as this value. You can use this value to check module working when MM is starting.

    onIdle: {
      timer: 1000*60*30, // if you don't want to use this feature, just set timer as `0` or command as ""
      command: "HIDEMODULES"
    },

    onActivate: {
      timer: 0,
      command: "SHOWMODULES"
    },


    // --- FOR EXPERTS / For development, debug or more


    verbose:false, // You can get error or some logs when this value is set as true.

    startChime: "connection.mp3", // you can use `mp3` to play chime when your mic is ready. It should be playable with your `play.playProgram`
    noChimeOnSay: false, // When using the `ASSISTANT_SAY` trigger, you can prevent the chime from being played before your words

    auth: { // I believe you don't need to change this.
      keyFilePath: "./credentials.json"
    },

    record:  { // Full version
      sampleRate    : 16000,      // audio sample rate
      threshold     : 0.5,        // silence threshold (rec only)
      thresholdStart: null,       // silence threshold to start recording, overrides threshold (rec only)
      thresholdEnd  : null,       // silence threshold to end recording, overrides threshold (rec only)
      silence       : 1.0,        // seconds of silence before ending
      verbose       : false,      // log info to the console
      recordProgram : "arecord",  // Defaults to "arecord" - also supports "rec" and "sox"
      device        : null        // recording device (e.g.: "plughw:1")
    },

    play: { // Full version
      encodingOut: "MP3", //'MP3' or 'WAV' is available, but you might not need to modify this.
      sampleRateOut: 24000,
      playProgram: "mpg321", //Your prefer sound play program. By example, if you are running this on OSX, `afplay` could be available.
      playOption: [],
      // If you need additional options to use playProgram, describe here. (except filename)
      // e.g: ["-d", "", "-t", "100"]
    },

    useGactionCLI: false, // If set as true, you can update your gAction when MM is rebooted.
    projectId: "", // Google Assistant ProjectId (Required only when you use gAction.)
    deviceModelId: "", // It should be described in your config.json. In most of case, you don't need to this.
    deviceInstanceId: "", // It should be described in your config.json. In most of case, you don't need to this.

    action:{}, // You can catch your gAction command.

    transcriptionHook: {}, // You can catch transcription hook and be able to make your own `COMMAND` with this.
    //See the `transcriptionHook` section.

    command: {}, // You can make your own MM command for gAction and transcriptionHook
    //See the `command` section.
    record: {
        recordProgram : "arecord",  
        device        : "plughw:1",
      },

      notifications: {
        ASSISTANT_ACTIVATED: "HOTWORD_PAUSE",
        ASSISTANT_DEACTIVATED: "HOTWORD_RESUME",
      },
	
    notifications: { // You can redefine these notifications to communicate with specific modules.
      ASSISTANT_ACTIVATE: "ASSISTANT_ACTIVATE",
      ASSISTANT_DEACTIVATE: "ASSISTANT_CLEAR",
      ASSISTANT_ACTIVATED: "ASSISTANT_ACTIVATED",
      ASSISTANT_DEACTIVATED: "ASSISTANT_DEACTIVATED",
      ASSISTANT_ACTION: "ASSISTANT_ACTION",
      ASSISTANT_ACTIVATED: "HOTWORD_PAUSE",     
      ASSISTANT_DEACTIVATED: "HOTWORD_RESUME",      
      DEFAULT_HOOK_NOTIFICATION: "ASSISTANT_HOOK",
      TEXT_QUERY: "ASSISTANT_QUERY",
      SAY_TEXT: "ASSISTANT_SAY",
    }
  }
},
 {
    module: "MMM-Hotword",
    config: {
      snowboy: [
  {
    hotwords: "smartmirror", //this will be sent to other module for distinguishing which hotword is detected.
    file: "resources/models/smart_mirror.umdl",
    sensitivity: '0.5',
  },
  {
    hotwords: "snowboy",
    file: "resources/models/snowboy.umdl",
    sensitivity: '0.5',
  },
  {
    file: 'resources/models/jarvis.umdl',
    sensitivity: '0.8,0.80',
    hotwords: ['jarvis','jarvis'] //Kitt.ai changed their Jarvis UMDL, it has 2 models in one file. So weird.
    //anyway, you can give different name for each. ['jarvis_1', 'jarvis_2']. Even though I think this is useless.
  }
    ],
    record: {
      sampleRate    : 16000,      // audio sample rate
      threshold     : 0.5,        // silence threshold (rec only)
      thresholdStart: null,       // silence threshold to start recording, overrides threshold (rec only)
      thresholdEnd  : null,       // silence threshold to end recording, overrides threshold (rec only)
      silence       : 1.0,        // seconds of silence before ending
      verbose       : false,      // log info to the console. Use this when you want to check mic working or not.
      recordProgram : 'arecord',  // Defaults to 'arecord' - also supports 'rec' and 'sox'
      device        : null        // recording device (e.g.: 'plughw:1')
    },
    autostart: true,              // if 'false', this module will wait for 'HOTWORD_RESUME' notification to start hotwords detection at the beginning.
    autorestart: false,          // You can set this 'true' when you want this module to go back to listening mode automatically again after hotword is detected. But use this carefully when your other modules are using microphone or speaker.

    // customizable notification trigger
    notifications: {
      PAUSE: "HOTWORD_PAUSE",
      RESUME: "HOTWORD_RESUME",
      LISTENING : "HOTWORD_LISTENING",
      SLEEPING : "HOTWORD_SLEEPING",
      ERROR : "HOTWORD_ERROR",
    },
    onDetected: {
      notification: (payload) => {
        return "HOTWORD_DETECTED"
      },
      payload: (payload) => {
        return payload
      }
    },
	  record: {
        recordProgram : "arecord",  
        device        : "plughw:1",
      },
      autostart:true,
      onDetected: {
        notification: function (payload) {
          return "ASSISTANT_ACTIVATE"
        },
        payload: function (payload){
          return {
            profile: payload.hotword
          }
        }
      },
    },
  },
	

[bachoo786]

@Davido thank you

[A Former User]

UPDATED

[2.1.4] - 2019.03.26

• Added : Now you can use recipes. https://github.com/eouia/MMM-AssistantMk2/wiki/Usage#recipes

  recipe is an external js file containing definitions of command, transcriptionHook and action. Your configuration could be more shorter.

recipes:["hide_and_show_all_modules.js", "reboot.js", "screen_onoff.js", "shutdown.js"],

Feel free to request PR to share your recipes to others.

• Changed : onIdle, onDetected features are disabled by default.

[bryan_1]

@Sean
sorry for my bad English. I use the default recipes and set the language to German for all pattern (“spiegel reboot”, “monitor aus”, etc). Every thing works fine, only the “SHOWMODULES” doesn’t work. Although the hotword is correctly detected, but nothing happens. I set the pattern for SHOWMODULE to pattern: “alle module”,

This is also the same case if I use the OnIdle and OnActivate:
onIdle: {
timer: 10006020,
command: “HIDEMODULES”,
},
onActivate: {
timer: 0,
command: “SHOWMODULES”,
},

I can hide the modules, but after the hotword for SHOWMODULES being recognized the modules aren’t shown again. What do I wrong? Many thanks in advance.

[A Former User]

@bryan_1
Can you show me the recipe(command and transcriptionHook or gAction) about SHOWMODULES?

[bryan_1]

@Sean
thank you sean for your quick response. Actually I use the default hide_and_show_all_modules.js and change the pattern to german:
var recipe = {
transcriptionHook: {
“HIDE_ALL_MODULES”: {
pattern: “alle verstecken”,
command: “HIDEMODULES”
},
“SHOW_ALL_MODULES”: {
pattern: “alle module”,
command: “SHOWMODULES”
},
},
command: {
“HIDEMODULES”: {
moduleExec: {
module:()=>{
return []
},
exec: (module, params, key) => {
module.hide(1000, null, {lockString:“AMK2”})
}
}
},
“SHOWMODULES”: {
moduleExec: {
module:[],
exec: (module, params, key) => {
module.show(1000, null, {lockString:“AMK2”})
}
}
},
},
}

exports.recipe = recipe