MagicMirror² v2.12.0 is available! For more information about this release, check out this topic.

MMM-AssistantMk2 v3 and MMM-Hotword with custom hotword not working



  • Hi
    first: MMM-AssistantMK2 is awesome thanks to the devs ( my wife and kids love it) I just upgraded both moduels to the last version --but I cant get it to work.
    Did remove both modules and installed it new, copied credentials and profiles

    platform:
    raspberrypi 4 , latest buster with all latest updateds

    ** steps to reproduce**
    .config.js

    /* Magic Mirror Config Sample
     *
     * By Michael Teeuw  http://michaelteeuw.nl
     * MIT Licensed.
     *
     * For more information how you can configurate this file
     * See https://github.com/MichMich/MagicMirror#configuration
     *
     */
    
    var config = {
    	address: "0.0.0.0", // Address to listen on, can be:
    	                      // - "localhost", "127.0.0.1", "::1" to listen on loopback interface
    	                      // - another specific IPv4/6 to listen on a specific interface
    	                      // - "", "0.0.0.0", "::" to listen on any interface
    	                      // Default, when address config is left out, is "localhost"
    	port: 8080,
    	ipWhitelist: [], // Set [] to allow all IP addresses
    	                                                       // or add a specific IPv4 of 192.168.1.5 :
    	                                                       // ["127.0.0.1", "::ffff:127.0.0.1", "::1", "::ffff:192.168.1.5"],
    	                                                       // or IPv4 range of 192.168.3.0 --> 192.168.3.15 use CIDR format :
    	                                                       // ["127.0.0.1", "::ffff:127.0.0.1", "::1", "::ffff:192.168.3.0/28"],
    
    	language: "de",
    	timeFormat: 24,
    	units: "metric",
    
    	modules: [
    		{
    			module: "alert",
    		},
    		{
    			module: "updatenotification",
    			position: "top_bar"
    		},
    		{
    			module: "clock",
    			position: "top_bar"
    		},
    
    		{
    			module: "compliments",
    			position: "lower_third"
    		},
    
    {
      module: "MMM-AssistantMk2",
      position: "top_left",
      config: {
        debug:true,
    	ui: "Classic", // or "Classic", "Classic2"
        assistantConfig: {
          projectId: "", // Required to use gaction.
          modelId: "", // (OPTIONAL for gaction)
          instanceId: "", // (OPTIONAL for gaction)
          latitude: 51.508530,
          longitude: -0.076132,
        },
        responseConfig: {
          useScreenOutput: true,
          useAudioOutput: true,
          useChime: true,
          timer: 5000,
          myMagicWord: false
        },
        micConfig: { // put there configuration generated by auto-installer
          recorder: "arecord",
          device: "plughw:1",
        },
        customActionConfig: {
          autoMakeAction: false,
          autoUpdateAction: false, // in RPI, gaction CLI might have some trouble.(current version should be 2.2.4, but for linux-arm, Google haven't updated) so leave this as false in RPI. I don't know it is solved or not.
          actionLocale: "en-US", // At this moment, multi-languages are not supported, sorry. Someday I'll work.
        },
        recipes: [ "with-MMM-Hotword.js", "with-MMM-TelegramBot.js" ],
        profiles: {
          "default": {
            profileFile: "default.json",
            lang: "de-DE"
          }
        },
      },
    },
    
    
    {
      module: "MMM-Hotword",
      config: {
    	recipes: ["with-AMk2v3_smart-mirror.js"],
        chimeOnFinish: null,
        mic: {
          recordProgram: "arecord",
          device: "plughw:1"
        },
        models: [
          {
            hotwords    : "computer",
            file        : "computer.umdl",
            sensitivity : "0.5",
          },
        ],
        commands: {
          "computer": {
            notificationExec: {
              notification: "ASSISTANT_ACTIVATE",
              payload: (detected, afterRecord) => {
                return {profile:"default"};
              }
            },
            restart:true,
            afterRecordLimit:0
          }
        }
      }
    },
    
    	]
    
    };
    
    /*************** DO NOT EDIT THE LINE BELOW ***************/
    if (typeof module !== "undefined") {module.exports = config;}
    

    result:
    magic mirror does start.
    I can speak “computer” and some words like “wetter in zürich” there is a beep and the mmm-assistantMK2 V3 logo starts to be animated and it looks like it expects some input but nothing happens

    expected result:
    after speaking "computer " and some words the google assistant starts responding

    log:

    [11:13:22.334] [LOG]    Server started ...
    [11:13:22.335] [LOG]    Connecting socket for: updatenotification
    [11:13:22.336] [LOG]    Connecting socket for: MMM-AssistantMk2
    [11:13:22.337] [LOG]    Connecting socket for: MMM-Hotword
    [11:13:22.338] [LOG]    [HOTWORD] MMM-Hotword starts
    [11:13:22.339] [LOG]    Sockets connected & modules started ...
    [11:13:22.620] [LOG]    Launching application.
    [11:13:25.748] [LOG]    [HOTWORD] Recipe is loaded: with-AMk2v3_smart-mirror.js
    [11:13:25.836] [LOG]    [AMK2] MMM-AssistantMk2 Version: 3.0.0
    [11:13:25.841] [LOG]    [AMK2] RECIPE_LOADED: with-MMM-Hotword.js
    [11:13:25.845] [LOG]    [AMK2] RECIPE_LOADED: with-MMM-TelegramBot.js
    [11:13:25.846] [LOG]    [AMK2] NO_ACTION_TO_MANAGE
    [11:13:25.871] [LOG]    [AMK2] AssistantMk2 v3 is initialized.
    [11:13:25.907] [LOG]    [HOTWORD] begins.
    [11:13:25.908] [LOG]    [HOTWORD] Detector starts listening.
    [11:13:25.941] [LOG]    [AMK2] tmp directory is now cleaned.
    [11:13:27.739] [LOG]    [HOTWORD] Detected: computer
    [11:13:27.741] [LOG]    [HOTWORD] stops.
    [11:13:27.747] [LOG]    [HOTWORD] Final Result: { detected: true, hotword: 'computer', file: null }
    [11:13:27.831] [LOG]    [AMK2] QUERY: { type: 'TEXT',
      profile: { profileFile: 'default.json', lang: 'de-DE' },
      key: null,
      lang: null,
      useScreenOutput: true,
      useAudioOutput: true,
      session: 1579428807774,
      status: 'standby',
      secretMode: false,
      sayMode: false }
    [11:13:28.816] [LOG]    [AMK2:AS] CONVERSATION:RESPONSE
    [11:13:28.891] [LOG]    [HOTWORD] begins.
    [11:13:28.892] [LOG]    [HOTWORD] Detector starts listening.
    [11:13:58.729] [LOG]    [AMK2:AS] CONVERSATION:RESPONSE
    [11:13:58.737] [LOG]    [AMK2:AS] CONVERSATION_ERROR : { Error: 14 UNAVAILABLE: Service unavailable.
        at Object.exports.createStatusError (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/common.js:91:15)
        at ClientDuplexStream._emitStatusIfDone (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client.js:233:26)
        at ClientDuplexStream._receiveStatus (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client.js:211:8)
        at Object.onReceiveStatus (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client_interceptors.js:1306:15)
        at InterceptingListener._callNext (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client_interceptors.js:568:42)
        at InterceptingListener.onReceiveStatus (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client_interceptors.js:618:8)
        at /home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client_interceptors.js:1123:18
      code: 14,
      metadata:
       Metadata { _internal_repr: { 'content-disposition': [Array] }, flags: 0 },
      details: 'Service unavailable.' }
    [11:13:58.741] [LOG]    [AMK2:AS] >> This error might happen when improper configuration or invalid Mic setup.
    [11:13:58.743] [LOG]    [AMK2] { session: 1579428807774,
      error: 'NO_RESPONSE',
      action: null,
      text: null,
      screen: null,
      audio: null,
      transcription: null,
      continue: false,
      lastQuery:
       { type: 'TEXT',
         profile: { profileFile: 'default.json', lang: 'de-DE' },
         key: null,
         lang: null,
         useScreenOutput: true,
         useAudioOutput: true,
         session: 1579428807774,
         status: 'standby',
         secretMode: false,
         sayMode: false } }
    [11:18:03.671] [LOG]    [HOTWORD] Detected: computer
    [11:18:03.673] [LOG]    [HOTWORD] stops.
    [11:18:03.675] [LOG]    [HOTWORD] Final Result: { detected: true, hotword: 'computer', file: null }
    [11:18:04.010] [LOG]    [AMK2] QUERY: { type: 'TEXT',
      profile: { profileFile: 'default.json', lang: 'de-DE' },
      key: null,
      lang: null,
      useScreenOutput: true,
      useAudioOutput: true,
      session: 1579429083751,
      status: 'standby',
      secretMode: false,
      sayMode: false }
    [11:18:04.592] [LOG]    [AMK2:AS] CONVERSATION:RESPONSE
    [11:18:04.921] [LOG]    [HOTWORD] begins.
    [11:18:04.922] [LOG]    [HOTWORD] Detector starts listening.
    [11:18:34.579] [LOG]    [AMK2:AS] CONVERSATION_ERROR : { Error: 14 UNAVAILABLE: Service unavailable.
        at Object.exports.createStatusError (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/common.js:91:15)
        at ClientDuplexStream._emitStatusIfDone (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client.js:233:26)
        at ClientDuplexStream._receiveStatus (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client.js:211:8)
        at Object.onReceiveStatus (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client_interceptors.js:1306:15)
        at InterceptingListener._callNext (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client_interceptors.js:568:42)
        at InterceptingListener.onReceiveStatus (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client_interceptors.js:618:8)
        at /home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client_interceptors.js:1123:18
      code: 14,
      metadata:
       Metadata { _internal_repr: { 'content-disposition': [Array] }, flags: 0 },
      details: 'Service unavailable.' }
    [11:18:34.584] [LOG]    [AMK2:AS] >> This error might happen when improper configuration or invalid Mic setup.
    [11:18:34.584] [LOG]    [AMK2] { session: 1579429083751,
      error: 'NO_RESPONSE',
      action: null,
      text: null,
      screen: null,
      audio: null,
      transcription: null,
      continue: false,
      lastQuery:
       { type: 'TEXT',
         profile: { profileFile: 'default.json', lang: 'de-DE' },
         key: null,
         lang: null,
         useScreenOutput: true,
         useAudioOutput: true,
         session: 1579429083751,
         status: 'standby',
         secretMode: false,
         sayMode: false } }
    
    

    Any help would be appreciated… AFAIK it must be a hotword config I messed up. after spending now 4 hours tweaking I reach out to the community ( pls make me and my kids happy 🙂 )



  • ah – progress but not solved (sometimes a hike helps)

    changed HOTWORD

    in config.js

    {
      module: "MMM-Hotword",
      config: {
    	recipes: ["with-AMk2v3_smart-mirror.js"],
        chimeOnFinish: null,
        mic: {
          recordProgram: "arecord",
          device: "plughw:1"
        },
    
      }
    },
    

    and in modules\MMM-Hotword\recipes\with-AMk2v3_smart-mirror.js

    var recipe = {
      models: [
        {
          hotwords    : "computer",
          file        : "computer.umdl",
          sensitivity : "0.5",
        },
      ],
      commands: {
        "computer": {
          notificationExec: {
            notification: "ASSISTANT_ACTIVATE",
            payload: (detected, afterRecord) => {
              var ret = {
                profile:"default",
                type: "MIC",
              }
              if (afterRecord) {
                ret.type = "WAVEFILE"
                ret.key = "modules/MMM-Hotword/" + afterRecord
              }
              return ret
            }
          },
          restart:false,
          afterRecordLimit: 7
        }
      }
    }
    
    exports.recipe = recipe // Don't remove this line.
    
    

    now it works with computer

    but now
    result in log

    [14:04:37.424] [LOG]    [HOTWORD] Detector starts listening.
    [14:04:38.714] [LOG]    [HOTWORD] Final Result: { detected: false }
    [14:04:40.875] [LOG]    [HOTWORD] begins.
    [14:04:40.971] [LOG]    [HOTWORD] Detector starts listening.
    [14:04:41.478] [LOG]    [HOTWORD] Final Result: { detected: false }
    [14:04:42.023] [LOG]    [HOTWORD] begins.
    [14:04:42.029] [LOG]    [HOTWORD] Detector starts listening.
    

    seems to be related to a loop issue https://github.com/eouia/MMM-Hotword/issues/120 and the devs are working on it …


  • Module Developer

    @davidoesch said in MMM-AssistantMk2 v3 and MMM-Hotword with custom hotword not working:

    [14:04:37.424] [LOG] [HOTWORD] Detector starts listening.
    [14:04:38.714] [LOG] [HOTWORD] Final Result: { detected: false }
    [14:04:40.875] [LOG] [HOTWORD] begins.
    [14:04:40.971] [LOG] [HOTWORD] Detector starts listening.
    [14:04:41.478] [LOG] [HOTWORD] Final Result: { detected: false }
    [14:04:42.023] [LOG] [HOTWORD] begins.
    [14:04:42.029] [LOG] [HOTWORD] Detector starts listening.

    It happens when your mic is not working properly. To diagnose, full log is needed. What kind of mic are you using?



  • Hmm

    the detector loop happens in one of ten cases-- expecially when I talk to fast

    The mic was working flawlessly with the older version of Hotword and AssistantMk2

    Nevertheless
    the mic used https://www.aliexpress.com/item/32783207436.html?spm=a2g0s.9042311.0.0.57274c4dlSRFlb

    arecord --list-devices
    **** List of CAPTURE Hardware Devices ****
    card 1: AK5371 [AK5371], device 0: USB Audio [USB Audio]
      Subdevices: 0/1
      Subdevice #0: subdevice #0
    
     lsusb
    Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
    Bus 001 Device 003: ID 0556:0002 Asahi Kasei Microsystems Co., Ltd
    Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
    Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
    
    

    Another side effect after the update of HOTWORD and mmm-AssistantMk2 v3 : MMM-PIR – can not activate the screen anymore …I discovered that

    dmesg
    [ 2718.210637] Out of memory: Kill process 1860 (electron) score 572 or sacrifice child
    [ 2718.210729] Killed process 1860 (electron) total-vm:604248kB, anon-rss:31300kB, file-rss:31676kB, shmem-rss:198652kB
    [ 2718.247983] oom_reaper: reaped process 1860 (electron), now anon-rss:0kB, file-rss:0kB, shmem-rss:198564kB
    
    

    and yes I rebuilt electron

    after a rollback to the old version everything works smoothly



  • @Sean
    here a larger the snippet of the session

    09.295] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.297] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.298] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.300] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.302] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.304] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.305] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.308] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.310] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.311] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.313] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.315] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.317] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.319] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.320] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.322] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.324] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.326] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.328] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.332] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.335] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.338] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.340] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.342] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.344] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.351] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.354] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
    [14:04:09.358] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 1198
    [14:04:09.361] [LOG]    [AMK2:AS] CONVERSATION_ALL_RESPONSES_RECEIVED
    [14:04:09.363] [LOG]    [AMK2:AS] CONVERSATION_END:COMPLETED
    [14:04:09.363] [LOG]    [AMK2:AS] CONVERSATION_PP:RESPONSE_AUDIO_MAKING
    [14:04:09.370] [LOG]    [AMK2:BW] RESPONSE_WAV_FILE_CREATED
    [14:04:09.371] [LOG]    [AMK2:AS] CONVERSATION_PP:RESPONSE_AUDIO_CREATED tmp/lastResponse.wav
    [14:04:10.499] [LOG]    [AMK2:BW] RESPONSE_WAV_FILE_CREATED
    [14:04:10.501] [LOG]    [AMK2:AS] CONVERSATION_PP:RESPONSE_AUDIO_CREATED tmp/lastResponse.wav
    [14:04:10.658] [LOG]    [AMK2] { session: null,
      error: null,
      action: { requestId: '5e23e8fb-0000-246c-a6f8-f4f5e8048f8c' },
      text: null,
      screen:
       { help: [ 'Und morgen?', 'Und nächste Woche?' ],
         trysay: 'Sag zum Beispiel …',
         links:
          [ 'http://www.weather.com/wx/today/?lat=51.51&lon=-0.08&locale=en_US&par=google' ],
         text:
          'Sonntag in London\nSonnig · Ndschlg.: 0 % · Feuchtigkeit: 68 ·\nweather.com\n7\n°C\n↑ 8°\n↓ -2°\nMo\n7° -1°\nDi\n6° 1°\nMi\n8° 4°\nDo\n8° 3°\nFr\n7° 4°',
         photos: [],
         path:
          '/home/pi/MagicMirror/modules/MMM-AssistantMk2/tmp/lastScreenOutput.html',
         uri: 'tmp/lastScreenOutput.html' },
      audio:
       { path:
          '/home/pi/MagicMirror/modules/MMM-AssistantMk2/tmp/lastResponse.wav',
         uri: 'tmp/lastResponse.wav' },
      transcription: { transcription: 'Wetter vor', done: true },
      continue: false,
      lastQuery:
       { type: 'TEXT',
         profile: { profileFile: 'default.json', lang: 'de-DE' },
         key: 'Wetter vor',
         lang: null,
         useScreenOutput: true,
         useAudioOutput: true,
         session: null,
         status: 'continue' } }
    [14:04:10.738] [LOG]    [AMK2] { session: null,
      error: null,
      action: { requestId: '5e23e6d7-0000-2ad6-87f4-089e082975cc' },
      text: null,
      screen:
       { help: [ 'Und morgen?', 'Und nächste Woche?' ],
         trysay: 'Sag zum Beispiel …',
         links:
          [ 'http://www.weather.com/wx/today/?lat=51.51&lon=-0.08&locale=en_US&par=google' ],
         text:
          'Sonntag in London\nSonnig · Ndschlg.: 0 % · Feuchtigkeit: 68 ·\nweather.com\n7\n°C\n↑ 8°\n↓ -2°\nMo\n7° -1°\nDi\n6° 1°\nMi\n8° 4°\nDo\n8° 3°\nFr\n7° 4°',
         photos: [],
         path:
          '/home/pi/MagicMirror/modules/MMM-AssistantMk2/tmp/lastScreenOutput.html',
         uri: 'tmp/lastScreenOutput.html' },
      audio:
       { path:
          '/home/pi/MagicMirror/modules/MMM-AssistantMk2/tmp/lastResponse.wav',
         uri: 'tmp/lastResponse.wav' },
      transcription: { transcription: 'Wetter vor', done: true },
      continue: false,
      lastQuery:
       { type: 'TEXT',
         profile: { profileFile: 'default.json', lang: 'de-DE' },
         key: 'Wetter vor',
         lang: null,
         useScreenOutput: true,
         useAudioOutput: true,
         session: null,
         status: 'think' } }
    [14:04:11.144] [LOG]    [HOTWORD] begins.
    [14:04:11.146] [LOG]    [HOTWORD] Detector starts listening.
    [14:04:11.496] [LOG]    [HOTWORD] Final Result: { detected: false }
    [14:04:11.718] [LOG]    [HOTWORD] begins.
    [14:04:11.724] [LOG]    [HOTWORD] Detector starts listening.
    [14:04:11.772] [LOG]    [HOTWORD] Final Result: { detected: false }
    

  • Module Developer

    @davidoesch said in MMM-AssistantMk2 v3 and MMM-Hotword with custom hotword not working:

    [14:04:09.371] [LOG] [AMK2:AS] CONVERSATION_PP:RESPONSE_AUDIO_CREATED tmp/lastResponse.wav
    [14:04:10.499] [LOG] [AMK2:BW] RESPONSE_WAV_FILE_CREATED
    [14:04:10.501] [LOG] [AMK2:AS] CONVERSATION_PP:RESPONSE_AUDIO_CREATED tmp/lastResponse.wav

    I don’t know why at this moment, but AMK2 seemed to be activated twice. That makes your issue. Can you send me whole log (from the start of MM)? (eouia0819@gmail.com)



  • This post is deleted!

  • Module Developer

    @davidoesch
    We found there could be potential memory issue, so we are trying to solve this issue by using smaller memory. I hope this could solve your issue also. Wait a while, sorry.


Log in to reply