MMM-AssistantMk2 v3 and MMM-Hotword with custom hotword not working

davidoesch · Jan 19, 2020, 10:22 AM

Hi
first: MMM-AssistantMK2 is awesome thanks to the devs ( my wife and kids love it) I just upgraded both moduels to the last version --but I cant get it to work.
Did remove both modules and installed it new, copied credentials and profiles

platform:
raspberrypi 4 , latest buster with all latest updateds

** steps to reproduce**
.config.js

/* Magic Mirror Config Sample
 *
 * By Michael Teeuw  http://michaelteeuw.nl
 * MIT Licensed.
 *
 * For more information how you can configurate this file
 * See https://github.com/MichMich/MagicMirror#configuration
 *
 */

var config = {
	address: "0.0.0.0", // Address to listen on, can be:
	                      // - "localhost", "127.0.0.1", "::1" to listen on loopback interface
	                      // - another specific IPv4/6 to listen on a specific interface
	                      // - "", "0.0.0.0", "::" to listen on any interface
	                      // Default, when address config is left out, is "localhost"
	port: 8080,
	ipWhitelist: [], // Set [] to allow all IP addresses
	                                                       // or add a specific IPv4 of 192.168.1.5 :
	                                                       // ["127.0.0.1", "::ffff:127.0.0.1", "::1", "::ffff:192.168.1.5"],
	                                                       // or IPv4 range of 192.168.3.0 --> 192.168.3.15 use CIDR format :
	                                                       // ["127.0.0.1", "::ffff:127.0.0.1", "::1", "::ffff:192.168.3.0/28"],

	language: "de",
	timeFormat: 24,
	units: "metric",

	modules: [
		{
			module: "alert",
		},
		{
			module: "updatenotification",
			position: "top_bar"
		},
		{
			module: "clock",
			position: "top_bar"
		},

		{
			module: "compliments",
			position: "lower_third"
		},

{
  module: "MMM-AssistantMk2",
  position: "top_left",
  config: {
    debug:true,
	ui: "Classic", // or "Classic", "Classic2"
    assistantConfig: {
      projectId: "", // Required to use gaction.
      modelId: "", // (OPTIONAL for gaction)
      instanceId: "", // (OPTIONAL for gaction)
      latitude: 51.508530,
      longitude: -0.076132,
    },
    responseConfig: {
      useScreenOutput: true,
      useAudioOutput: true,
      useChime: true,
      timer: 5000,
      myMagicWord: false
    },
    micConfig: { // put there configuration generated by auto-installer
      recorder: "arecord",
      device: "plughw:1",
    },
    customActionConfig: {
      autoMakeAction: false,
      autoUpdateAction: false, // in RPI, gaction CLI might have some trouble.(current version should be 2.2.4, but for linux-arm, Google haven't updated) so leave this as false in RPI. I don't know it is solved or not.
      actionLocale: "en-US", // At this moment, multi-languages are not supported, sorry. Someday I'll work.
    },
    recipes: [ "with-MMM-Hotword.js", "with-MMM-TelegramBot.js" ],
    profiles: {
      "default": {
        profileFile: "default.json",
        lang: "de-DE"
      }
    },
  },
},


{
  module: "MMM-Hotword",
  config: {
	recipes: ["with-AMk2v3_smart-mirror.js"],
    chimeOnFinish: null,
    mic: {
      recordProgram: "arecord",
      device: "plughw:1"
    },
    models: [
      {
        hotwords    : "computer",
        file        : "computer.umdl",
        sensitivity : "0.5",
      },
    ],
    commands: {
      "computer": {
        notificationExec: {
          notification: "ASSISTANT_ACTIVATE",
          payload: (detected, afterRecord) => {
            return {profile:"default"};
          }
        },
        restart:true,
        afterRecordLimit:0
      }
    }
  }
},

	]

};

/*************** DO NOT EDIT THE LINE BELOW ***************/
if (typeof module !== "undefined") {module.exports = config;}

result:
magic mirror does start.
I can speak “computer” and some words like “wetter in zürich” there is a beep and the mmm-assistantMK2 V3 logo starts to be animated and it looks like it expects some input but nothing happens

expected result:
after speaking "computer " and some words the google assistant starts responding

log:

[11:13:22.334] [LOG]    Server started ...
[11:13:22.335] [LOG]    Connecting socket for: updatenotification
[11:13:22.336] [LOG]    Connecting socket for: MMM-AssistantMk2
[11:13:22.337] [LOG]    Connecting socket for: MMM-Hotword
[11:13:22.338] [LOG]    [HOTWORD] MMM-Hotword starts
[11:13:22.339] [LOG]    Sockets connected & modules started ...
[11:13:22.620] [LOG]    Launching application.
[11:13:25.748] [LOG]    [HOTWORD] Recipe is loaded: with-AMk2v3_smart-mirror.js
[11:13:25.836] [LOG]    [AMK2] MMM-AssistantMk2 Version: 3.0.0
[11:13:25.841] [LOG]    [AMK2] RECIPE_LOADED: with-MMM-Hotword.js
[11:13:25.845] [LOG]    [AMK2] RECIPE_LOADED: with-MMM-TelegramBot.js
[11:13:25.846] [LOG]    [AMK2] NO_ACTION_TO_MANAGE
[11:13:25.871] [LOG]    [AMK2] AssistantMk2 v3 is initialized.
[11:13:25.907] [LOG]    [HOTWORD] begins.
[11:13:25.908] [LOG]    [HOTWORD] Detector starts listening.
[11:13:25.941] [LOG]    [AMK2] tmp directory is now cleaned.
[11:13:27.739] [LOG]    [HOTWORD] Detected: computer
[11:13:27.741] [LOG]    [HOTWORD] stops.
[11:13:27.747] [LOG]    [HOTWORD] Final Result: { detected: true, hotword: 'computer', file: null }
[11:13:27.831] [LOG]    [AMK2] QUERY: { type: 'TEXT',
  profile: { profileFile: 'default.json', lang: 'de-DE' },
  key: null,
  lang: null,
  useScreenOutput: true,
  useAudioOutput: true,
  session: 1579428807774,
  status: 'standby',
  secretMode: false,
  sayMode: false }
[11:13:28.816] [LOG]    [AMK2:AS] CONVERSATION:RESPONSE
[11:13:28.891] [LOG]    [HOTWORD] begins.
[11:13:28.892] [LOG]    [HOTWORD] Detector starts listening.
[11:13:58.729] [LOG]    [AMK2:AS] CONVERSATION:RESPONSE
[11:13:58.737] [LOG]    [AMK2:AS] CONVERSATION_ERROR : { Error: 14 UNAVAILABLE: Service unavailable.
    at Object.exports.createStatusError (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/common.js:91:15)
    at ClientDuplexStream._emitStatusIfDone (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client.js:233:26)
    at ClientDuplexStream._receiveStatus (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client.js:211:8)
    at Object.onReceiveStatus (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client_interceptors.js:1306:15)
    at InterceptingListener._callNext (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client_interceptors.js:568:42)
    at InterceptingListener.onReceiveStatus (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client_interceptors.js:618:8)
    at /home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client_interceptors.js:1123:18
  code: 14,
  metadata:
   Metadata { _internal_repr: { 'content-disposition': [Array] }, flags: 0 },
  details: 'Service unavailable.' }
[11:13:58.741] [LOG]    [AMK2:AS] >> This error might happen when improper configuration or invalid Mic setup.
[11:13:58.743] [LOG]    [AMK2] { session: 1579428807774,
  error: 'NO_RESPONSE',
  action: null,
  text: null,
  screen: null,
  audio: null,
  transcription: null,
  continue: false,
  lastQuery:
   { type: 'TEXT',
     profile: { profileFile: 'default.json', lang: 'de-DE' },
     key: null,
     lang: null,
     useScreenOutput: true,
     useAudioOutput: true,
     session: 1579428807774,
     status: 'standby',
     secretMode: false,
     sayMode: false } }
[11:18:03.671] [LOG]    [HOTWORD] Detected: computer
[11:18:03.673] [LOG]    [HOTWORD] stops.
[11:18:03.675] [LOG]    [HOTWORD] Final Result: { detected: true, hotword: 'computer', file: null }
[11:18:04.010] [LOG]    [AMK2] QUERY: { type: 'TEXT',
  profile: { profileFile: 'default.json', lang: 'de-DE' },
  key: null,
  lang: null,
  useScreenOutput: true,
  useAudioOutput: true,
  session: 1579429083751,
  status: 'standby',
  secretMode: false,
  sayMode: false }
[11:18:04.592] [LOG]    [AMK2:AS] CONVERSATION:RESPONSE
[11:18:04.921] [LOG]    [HOTWORD] begins.
[11:18:04.922] [LOG]    [HOTWORD] Detector starts listening.
[11:18:34.579] [LOG]    [AMK2:AS] CONVERSATION_ERROR : { Error: 14 UNAVAILABLE: Service unavailable.
    at Object.exports.createStatusError (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/common.js:91:15)
    at ClientDuplexStream._emitStatusIfDone (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client.js:233:26)
    at ClientDuplexStream._receiveStatus (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client.js:211:8)
    at Object.onReceiveStatus (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client_interceptors.js:1306:15)
    at InterceptingListener._callNext (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client_interceptors.js:568:42)
    at InterceptingListener.onReceiveStatus (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client_interceptors.js:618:8)
    at /home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client_interceptors.js:1123:18
  code: 14,
  metadata:
   Metadata { _internal_repr: { 'content-disposition': [Array] }, flags: 0 },
  details: 'Service unavailable.' }
[11:18:34.584] [LOG]    [AMK2:AS] >> This error might happen when improper configuration or invalid Mic setup.
[11:18:34.584] [LOG]    [AMK2] { session: 1579429083751,
  error: 'NO_RESPONSE',
  action: null,
  text: null,
  screen: null,
  audio: null,
  transcription: null,
  continue: false,
  lastQuery:
   { type: 'TEXT',
     profile: { profileFile: 'default.json', lang: 'de-DE' },
     key: null,
     lang: null,
     useScreenOutput: true,
     useAudioOutput: true,
     session: 1579429083751,
     status: 'standby',
     secretMode: false,
     sayMode: false } }

Any help would be appreciated… AFAIK it must be a hotword config I messed up. after spending now 4 hours tweaking I reach out to the community ( pls make me and my kids happy :-) )

davidoesch · Jan 19, 2020, 1:13 PM

ah – progress but not solved (sometimes a hike helps)

changed HOTWORD

in config.js

{
  module: "MMM-Hotword",
  config: {
	recipes: ["with-AMk2v3_smart-mirror.js"],
    chimeOnFinish: null,
    mic: {
      recordProgram: "arecord",
      device: "plughw:1"
    },

  }
},

and in modules\MMM-Hotword\recipes\with-AMk2v3_smart-mirror.js

var recipe = {
  models: [
    {
      hotwords    : "computer",
      file        : "computer.umdl",
      sensitivity : "0.5",
    },
  ],
  commands: {
    "computer": {
      notificationExec: {
        notification: "ASSISTANT_ACTIVATE",
        payload: (detected, afterRecord) => {
          var ret = {
            profile:"default",
            type: "MIC",
          }
          if (afterRecord) {
            ret.type = "WAVEFILE"
            ret.key = "modules/MMM-Hotword/" + afterRecord
          }
          return ret
        }
      },
      restart:false,
      afterRecordLimit: 7
    }
  }
}

exports.recipe = recipe // Don't remove this line.

now it works with computer

but now
result in log

[14:04:37.424] [LOG]    [HOTWORD] Detector starts listening.
[14:04:38.714] [LOG]    [HOTWORD] Final Result: { detected: false }
[14:04:40.875] [LOG]    [HOTWORD] begins.
[14:04:40.971] [LOG]    [HOTWORD] Detector starts listening.
[14:04:41.478] [LOG]    [HOTWORD] Final Result: { detected: false }
[14:04:42.023] [LOG]    [HOTWORD] begins.
[14:04:42.029] [LOG]    [HOTWORD] Detector starts listening.

seems to be related to a loop issue https://github.com/eouia/MMM-Hotword/issues/120 and the devs are working on it …

A Former User · Jan 19, 2020, 3:27 PM

@davidoesch said in MMM-AssistantMk2 v3 and MMM-Hotword with custom hotword not working:

[14:04:37.424] [LOG] [HOTWORD] Detector starts listening.
[14:04:38.714] [LOG] [HOTWORD] Final Result: { detected: false }
[14:04:40.875] [LOG] [HOTWORD] begins.
[14:04:40.971] [LOG] [HOTWORD] Detector starts listening.
[14:04:41.478] [LOG] [HOTWORD] Final Result: { detected: false }
[14:04:42.023] [LOG] [HOTWORD] begins.
[14:04:42.029] [LOG] [HOTWORD] Detector starts listening.

It happens when your mic is not working properly. To diagnose, full log is needed. What kind of mic are you using?

davidoesch · Jan 19, 2020, 5:09 PM

Hmm

the detector loop happens in one of ten cases-- expecially when I talk to fast

The mic was working flawlessly with the older version of Hotword and AssistantMk2

Nevertheless
the mic used https://www.aliexpress.com/item/32783207436.html?spm=a2g0s.9042311.0.0.57274c4dlSRFlb

arecord --list-devices
**** List of CAPTURE Hardware Devices ****
card 1: AK5371 [AK5371], device 0: USB Audio [USB Audio]
  Subdevices: 0/1
  Subdevice #0: subdevice #0

 lsusb
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 0556:0002 Asahi Kasei Microsystems Co., Ltd
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Another side effect after the update of HOTWORD and mmm-AssistantMk2 v3 : MMM-PIR – can not activate the screen anymore …I discovered that

dmesg
[ 2718.210637] Out of memory: Kill process 1860 (electron) score 572 or sacrifice child
[ 2718.210729] Killed process 1860 (electron) total-vm:604248kB, anon-rss:31300kB, file-rss:31676kB, shmem-rss:198652kB
[ 2718.247983] oom_reaper: reaped process 1860 (electron), now anon-rss:0kB, file-rss:0kB, shmem-rss:198564kB

and yes I rebuilt electron

after a rollback to the old version everything works smoothly

davidoesch · Jan 19, 2020, 6:00 PM

@Sean
here a larger the snippet of the session

09.295] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.297] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.298] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.300] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.302] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.304] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.305] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.308] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.310] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.311] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.313] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.315] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.317] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.319] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.320] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.322] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.324] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.326] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.328] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.332] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.335] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.338] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.340] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.342] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.344] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.351] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.354] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.358] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 1198
[14:04:09.361] [LOG]    [AMK2:AS] CONVERSATION_ALL_RESPONSES_RECEIVED
[14:04:09.363] [LOG]    [AMK2:AS] CONVERSATION_END:COMPLETED
[14:04:09.363] [LOG]    [AMK2:AS] CONVERSATION_PP:RESPONSE_AUDIO_MAKING
[14:04:09.370] [LOG]    [AMK2:BW] RESPONSE_WAV_FILE_CREATED
[14:04:09.371] [LOG]    [AMK2:AS] CONVERSATION_PP:RESPONSE_AUDIO_CREATED tmp/lastResponse.wav
[14:04:10.499] [LOG]    [AMK2:BW] RESPONSE_WAV_FILE_CREATED
[14:04:10.501] [LOG]    [AMK2:AS] CONVERSATION_PP:RESPONSE_AUDIO_CREATED tmp/lastResponse.wav
[14:04:10.658] [LOG]    [AMK2] { session: null,
  error: null,
  action: { requestId: '5e23e8fb-0000-246c-a6f8-f4f5e8048f8c' },
  text: null,
  screen:
   { help: [ 'Und morgen?', 'Und nächste Woche?' ],
     trysay: 'Sag zum Beispiel …',
     links:
      [ 'http://www.weather.com/wx/today/?lat=51.51&lon=-0.08&locale=en_US&par=google' ],
     text:
      'Sonntag in London\nSonnig · Ndschlg.: 0 % · Feuchtigkeit: 68 ·\nweather.com\n7\n°C\n↑ 8°\n↓ -2°\nMo\n7° -1°\nDi\n6° 1°\nMi\n8° 4°\nDo\n8° 3°\nFr\n7° 4°',
     photos: [],
     path:
      '/home/pi/MagicMirror/modules/MMM-AssistantMk2/tmp/lastScreenOutput.html',
     uri: 'tmp/lastScreenOutput.html' },
  audio:
   { path:
      '/home/pi/MagicMirror/modules/MMM-AssistantMk2/tmp/lastResponse.wav',
     uri: 'tmp/lastResponse.wav' },
  transcription: { transcription: 'Wetter vor', done: true },
  continue: false,
  lastQuery:
   { type: 'TEXT',
     profile: { profileFile: 'default.json', lang: 'de-DE' },
     key: 'Wetter vor',
     lang: null,
     useScreenOutput: true,
     useAudioOutput: true,
     session: null,
     status: 'continue' } }
[14:04:10.738] [LOG]    [AMK2] { session: null,
  error: null,
  action: { requestId: '5e23e6d7-0000-2ad6-87f4-089e082975cc' },
  text: null,
  screen:
   { help: [ 'Und morgen?', 'Und nächste Woche?' ],
     trysay: 'Sag zum Beispiel …',
     links:
      [ 'http://www.weather.com/wx/today/?lat=51.51&lon=-0.08&locale=en_US&par=google' ],
     text:
      'Sonntag in London\nSonnig · Ndschlg.: 0 % · Feuchtigkeit: 68 ·\nweather.com\n7\n°C\n↑ 8°\n↓ -2°\nMo\n7° -1°\nDi\n6° 1°\nMi\n8° 4°\nDo\n8° 3°\nFr\n7° 4°',
     photos: [],
     path:
      '/home/pi/MagicMirror/modules/MMM-AssistantMk2/tmp/lastScreenOutput.html',
     uri: 'tmp/lastScreenOutput.html' },
  audio:
   { path:
      '/home/pi/MagicMirror/modules/MMM-AssistantMk2/tmp/lastResponse.wav',
     uri: 'tmp/lastResponse.wav' },
  transcription: { transcription: 'Wetter vor', done: true },
  continue: false,
  lastQuery:
   { type: 'TEXT',
     profile: { profileFile: 'default.json', lang: 'de-DE' },
     key: 'Wetter vor',
     lang: null,
     useScreenOutput: true,
     useAudioOutput: true,
     session: null,
     status: 'think' } }
[14:04:11.144] [LOG]    [HOTWORD] begins.
[14:04:11.146] [LOG]    [HOTWORD] Detector starts listening.
[14:04:11.496] [LOG]    [HOTWORD] Final Result: { detected: false }
[14:04:11.718] [LOG]    [HOTWORD] begins.
[14:04:11.724] [LOG]    [HOTWORD] Detector starts listening.
[14:04:11.772] [LOG]    [HOTWORD] Final Result: { detected: false }

A Former User · Jan 20, 2020, 7:44 PM

@davidoesch said in MMM-AssistantMk2 v3 and MMM-Hotword with custom hotword not working:

[14:04:09.371] [LOG] [AMK2:AS] CONVERSATION_PP:RESPONSE_AUDIO_CREATED tmp/lastResponse.wav
[14:04:10.499] [LOG] [AMK2:BW] RESPONSE_WAV_FILE_CREATED
[14:04:10.501] [LOG] [AMK2:AS] CONVERSATION_PP:RESPONSE_AUDIO_CREATED tmp/lastResponse.wav

I don’t know why at this moment, but AMK2 seemed to be activated twice. That makes your issue. Can you send me whole log (from the start of MM)? (eouia0819@gmail.com)

Artech · Jan 21, 2020, 4:43 PM

This post is deleted!

A Former User · Jan 23, 2020, 7:58 AM

@davidoesch
We found there could be potential memory issue, so we are trying to solve this issue by using smaller memory. I hope this could solve your issue also. Wait a while, sorry.