MMM-AssistantMk2

Davido

@ejay-ibm :
i will try your setting, thx for your answer

@Lowis :
i tried it with 30% and the same result, also with the mic in front of my mouth, no action to stop … the mic onscreen reacts to my voice, but the input field leaves blank, and there is also no timeout for another input

bachoo786

@Sean I have installed assistantmk2 correctly and have also installed hotword. However I get a blank screen but when I say ok google everything works fine.

I have already been through the “electron-rebuild” step, this is what I did:

pi@raspberrypi:~/MagicMirror/modules/MMM-AssistantMk2 $ npm install --save-dev electron-rebuild
+ electron-rebuild@1.8.4
updated 1 package and audited 617 packages in 12.623s
found 0 vulnerabilities

pi@raspberrypi:~/MagicMirror/modules/MMM-AssistantMk2 $ ./node_modules/.bin/electron-rebuild
✔ Rebuild Complete

but still no joy.

and this is what I have in my config.js

{
    module: "MMM-AssistantMk2",
    position: "top_right",
    config: {
      record: {
        recordProgram : "arecord",
        device        : "plughw:1",
      },

      notifications: {
        ASSISTANT_ACTIVATED: "HOTWORD_PAUSE",
        ASSISTANT_DEACTIVATED: "HOTWORD_RESUME",
      },
    }
  },
  {
    module: "MMM-Hotword",
    config: {
      record: {
        recordProgram : "arecord",
        device        : "plughw:1",
      },
      autostart:true,
      onDetected: {
        notification: function (payload) {
          return "ASSISTANT_ACTIVATE"
        },
        payload: function (payload){
          return {
            profile: payload.hotword
          }
        }
      },
    },
  },

what I did notice that if I was to remove the MMM-Hotword from my config i.e. the last part of the code i.e.

{
    module: "MMM-Hotword",
    config: {
      record: {
        recordProgram : "arecord",
        device        : "plughw:1",
      },
      autostart:true,
      onDetected: {
        notification: function (payload) {
          return "ASSISTANT_ACTIVATE"
        },
        payload: function (payload){
          return {
            profile: payload.hotword
          }
        }
      },
    },
  },

the mirror starts working fine.

Also in the hotword git I have followed all the possible solutions and also the main one which is :

cd ~/MagicMirror/modules/MMM-Hotword/node_modules/snowboy
nano package.json ## Or open the file with any TEXT editor
Then, find this and replace From

“node-pre-gyp”: “^0.6.30”
To

“node-pre-gyp”: “^0.12.0”
Then retry npm install again in the module directory again.

I have no joy buddy. Please help me

bachoo786

Hi @Sean

So I managed to get it working and everything is fine. However I will be grateful if you could please help me with the following queries:

1). When i say jarvis or smart mirror, the assistantmk2 module takes 1 to 2 seconds to respond i.e. the ding bell I mean after which I can ask any question. How can I remove this lag and make it instant? I have seen other people dont have this issue.

2). How do I get the assistantmk2 to play youtube videos on my mirror by asking it to play something from youtube?

Thanks.

This is my config:

{
    module: "MMM-AssistantMk2",
    position: "top_right",
    config: {
      deviceLocation: {
                        coordinates: { // set the latitude and longitude of the device to get localized information like weather or time. (ref. mygeoposition.com)
                        latitude: 51.597442, // -90.0 - +90.0
                        longitude: -0.338839, // -180.0 - +180.0
          }
        },
      record: {
        recordProgram : "arecord",
        device        : "default",
      },
      play: { // Full values are in `FOR EXPERTS` section.
              playProgram: "mpg321", // recommended.
      },
      notifications: {
        ASSISTANT_ACTIVATED: "HOTWORD_PAUSE",
        ASSISTANT_DEACTIVATED: "HOTWORD_RESUME",
      },
    }
  },
  {
    module: "MMM-Hotword",
    config: {
      record: {
        recordProgram : "arecord",
        device        : "default",
      },
      autostart:true,
      onDetected: {
        notification: function (payload) {
          return "ASSISTANT_ACTIVATE"
        },
        payload: function (payload){
          return {
            profile: payload.hotword
          }
        }
      },
    },
  },

Davido

@bachoo786

to 1) which Hardware are you using?
to 2) i say the Hotword and then “play on youtube …” and it is working

bachoo786

@Davido I am using raspberry pi 3b+

Davido

@bachoo786 :

i am using a normal rPi 3B without + and the answering time is really fast … maybe some other Module which is interacting?

bachoo786

@Davido nope nothing is interacting with my assistantmk2 module.

bachoo786

@Davido can you share your config for both mmm-assistant and mmm-hotword please?

Davido

@bachoo786

I hope this will help

{
  module: "MMM-AssistantMk2",
  position: "top_right",
  config: {

    // --- ESSENTIALS / modifying for your environment might be needed.


    deviceLocation: {
      coordinates: { // set the latitude and longitude of the device to get localized information like weather or time. (ref. mygeoposition.com)
        latitude: 47.809490, // -90.0 - +90.0
        longitude: 13.055010, // -180.0 - +180.0
      },
    },

    defaultProfile: "default", // If you have several profiles and want to set one of them as default profile, describe here.

    profiles: {
      "default" : { // profile name.
        profileFile: "default.json", // profile file name.
        lang: "de-DE"
        //currently available (estimation, not all tested):
        //  de-DE, en-AU, en-CA, en-GB, en-US, en-IN
        // fr-CA, fr-FR, it-IT, ja-JP, es-ES, es-MX, ko-KR, pt-BR
        // https://developers.google.com/assistant/sdk/reference/rpc/languages
      },
      /* Add your other profiles here, if exists.
      "other_profile" : {
        profileFile: "other.json",
        lang: "de-DE"
      }
      */
    },

    record: { // Full values are in `FOR EXPERTS` section.
      recordProgram: "arecord",  // Defaults to "arecord" - also supports "rec" and "sox"
      device: null        // recording device (e.g.: "plughw:1")
    },

    play: { // Full values are in `FOR EXPERTS` section.
      playProgram: "mpg321", // recommended.
    },


    // --- OPTIONAL / not important but customizable for your usage


    responseVoice: true, // If available, Assistant will response with her voice.
    responseScreen: true, // If available, Assistant will response with some rendered HTML
    responseAlert: true, // If available, Assistant will response with Alert module of MM
    // Sometimes, any response might not be returned. responseAlert is useful for displaying error.

    screenZoom: "80%", // Adjust responseScreen to your mirror size.
    screenDuration: 2000, // milliseconds. How long responseScreen will be shown after speech.
    //If you set 0, Screen Output will be closed after Response speech finishes ASAP.

    youtubeAutoplay: true, //If set as true, found Youtube video will be played automatically.
    pauseOnYoutube:true, //If set as true, You cannot activate Assistant during youtube playing. Recommended for the performance (Because permanent hotword detecting might make performance lower)

    useWelcomeMessage: "", //Try "brief today" as this value. You can use this value to check module working when MM is starting.

    onIdle: {
      timer: 1000*60*30, // if you don't want to use this feature, just set timer as `0` or command as ""
      command: "HIDEMODULES"
    },

    onActivate: {
      timer: 0,
      command: "SHOWMODULES"
    },


    // --- FOR EXPERTS / For development, debug or more


    verbose:false, // You can get error or some logs when this value is set as true.

    startChime: "connection.mp3", // you can use `mp3` to play chime when your mic is ready. It should be playable with your `play.playProgram`
    noChimeOnSay: false, // When using the `ASSISTANT_SAY` trigger, you can prevent the chime from being played before your words

    auth: { // I believe you don't need to change this.
      keyFilePath: "./credentials.json"
    },

    record:  { // Full version
      sampleRate    : 16000,      // audio sample rate
      threshold     : 0.5,        // silence threshold (rec only)
      thresholdStart: null,       // silence threshold to start recording, overrides threshold (rec only)
      thresholdEnd  : null,       // silence threshold to end recording, overrides threshold (rec only)
      silence       : 1.0,        // seconds of silence before ending
      verbose       : false,      // log info to the console
      recordProgram : "arecord",  // Defaults to "arecord" - also supports "rec" and "sox"
      device        : null        // recording device (e.g.: "plughw:1")
    },

    play: { // Full version
      encodingOut: "MP3", //'MP3' or 'WAV' is available, but you might not need to modify this.
      sampleRateOut: 24000,
      playProgram: "mpg321", //Your prefer sound play program. By example, if you are running this on OSX, `afplay` could be available.
      playOption: [],
      // If you need additional options to use playProgram, describe here. (except filename)
      // e.g: ["-d", "", "-t", "100"]
    },

    useGactionCLI: false, // If set as true, you can update your gAction when MM is rebooted.
    projectId: "", // Google Assistant ProjectId (Required only when you use gAction.)
    deviceModelId: "", // It should be described in your config.json. In most of case, you don't need to this.
    deviceInstanceId: "", // It should be described in your config.json. In most of case, you don't need to this.

    action:{}, // You can catch your gAction command.

    transcriptionHook: {}, // You can catch transcription hook and be able to make your own `COMMAND` with this.
    //See the `transcriptionHook` section.

    command: {}, // You can make your own MM command for gAction and transcriptionHook
    //See the `command` section.
    record: {
        recordProgram : "arecord",  
        device        : "plughw:1",
      },

      notifications: {
        ASSISTANT_ACTIVATED: "HOTWORD_PAUSE",
        ASSISTANT_DEACTIVATED: "HOTWORD_RESUME",
      },
	
    notifications: { // You can redefine these notifications to communicate with specific modules.
      ASSISTANT_ACTIVATE: "ASSISTANT_ACTIVATE",
      ASSISTANT_DEACTIVATE: "ASSISTANT_CLEAR",
      ASSISTANT_ACTIVATED: "ASSISTANT_ACTIVATED",
      ASSISTANT_DEACTIVATED: "ASSISTANT_DEACTIVATED",
      ASSISTANT_ACTION: "ASSISTANT_ACTION",
      ASSISTANT_ACTIVATED: "HOTWORD_PAUSE",     
      ASSISTANT_DEACTIVATED: "HOTWORD_RESUME",      
      DEFAULT_HOOK_NOTIFICATION: "ASSISTANT_HOOK",
      TEXT_QUERY: "ASSISTANT_QUERY",
      SAY_TEXT: "ASSISTANT_SAY",
    }
  }
},
 {
    module: "MMM-Hotword",
    config: {
      snowboy: [
  {
    hotwords: "smartmirror", //this will be sent to other module for distinguishing which hotword is detected.
    file: "resources/models/smart_mirror.umdl",
    sensitivity: '0.5',
  },
  {
    hotwords: "snowboy",
    file: "resources/models/snowboy.umdl",
    sensitivity: '0.5',
  },
  {
    file: 'resources/models/jarvis.umdl',
    sensitivity: '0.8,0.80',
    hotwords: ['jarvis','jarvis'] //Kitt.ai changed their Jarvis UMDL, it has 2 models in one file. So weird.
    //anyway, you can give different name for each. ['jarvis_1', 'jarvis_2']. Even though I think this is useless.
  }
    ],
    record: {
      sampleRate    : 16000,      // audio sample rate
      threshold     : 0.5,        // silence threshold (rec only)
      thresholdStart: null,       // silence threshold to start recording, overrides threshold (rec only)
      thresholdEnd  : null,       // silence threshold to end recording, overrides threshold (rec only)
      silence       : 1.0,        // seconds of silence before ending
      verbose       : false,      // log info to the console. Use this when you want to check mic working or not.
      recordProgram : 'arecord',  // Defaults to 'arecord' - also supports 'rec' and 'sox'
      device        : null        // recording device (e.g.: 'plughw:1')
    },
    autostart: true,              // if 'false', this module will wait for 'HOTWORD_RESUME' notification to start hotwords detection at the beginning.
    autorestart: false,          // You can set this 'true' when you want this module to go back to listening mode automatically again after hotword is detected. But use this carefully when your other modules are using microphone or speaker.

    // customizable notification trigger
    notifications: {
      PAUSE: "HOTWORD_PAUSE",
      RESUME: "HOTWORD_RESUME",
      LISTENING : "HOTWORD_LISTENING",
      SLEEPING : "HOTWORD_SLEEPING",
      ERROR : "HOTWORD_ERROR",
    },
    onDetected: {
      notification: (payload) => {
        return "HOTWORD_DETECTED"
      },
      payload: (payload) => {
        return payload
      }
    },
	  record: {
        recordProgram : "arecord",  
        device        : "plughw:1",
      },
      autostart:true,
      onDetected: {
        notification: function (payload) {
          return "ASSISTANT_ACTIVATE"
        },
        payload: function (payload){
          return {
            profile: payload.hotword
          }
        }
      },
    },
  },

bachoo786

@Davido thank you