MMM-AssistantMk2 v3 and MMM-Hotword with custom hotword not working

davidoesch

Hi
first: MMM-AssistantMK2 is awesome thanks to the devs ( my wife and kids love it) I just upgraded both moduels to the last version --but I cant get it to work.
Did remove both modules and installed it new, copied credentials and profiles

platform:
raspberrypi 4 , latest buster with all latest updateds

** steps to reproduce**
.config.js

/* Magic Mirror Config Sample
 *
 * By Michael Teeuw  http://michaelteeuw.nl
 * MIT Licensed.
 *
 * For more information how you can configurate this file
 * See https://github.com/MichMich/MagicMirror#configuration
 *
 */

var config = {
	address: "0.0.0.0", // Address to listen on, can be:
	                      // - "localhost", "127.0.0.1", "::1" to listen on loopback interface
	                      // - another specific IPv4/6 to listen on a specific interface
	                      // - "", "0.0.0.0", "::" to listen on any interface
	                      // Default, when address config is left out, is "localhost"
	port: 8080,
	ipWhitelist: [], // Set [] to allow all IP addresses
	                                                       // or add a specific IPv4 of 192.168.1.5 :
	                                                       // ["127.0.0.1", "::ffff:127.0.0.1", "::1", "::ffff:192.168.1.5"],
	                                                       // or IPv4 range of 192.168.3.0 --> 192.168.3.15 use CIDR format :
	                                                       // ["127.0.0.1", "::ffff:127.0.0.1", "::1", "::ffff:192.168.3.0/28"],

	language: "de",
	timeFormat: 24,
	units: "metric",

	modules: [
		{
			module: "alert",
		},
		{
			module: "updatenotification",
			position: "top_bar"
		},
		{
			module: "clock",
			position: "top_bar"
		},

		{
			module: "compliments",
			position: "lower_third"
		},

{
  module: "MMM-AssistantMk2",
  position: "top_left",
  config: {
    debug:true,
	ui: "Classic", // or "Classic", "Classic2"
    assistantConfig: {
      projectId: "", // Required to use gaction.
      modelId: "", // (OPTIONAL for gaction)
      instanceId: "", // (OPTIONAL for gaction)
      latitude: 51.508530,
      longitude: -0.076132,
    },
    responseConfig: {
      useScreenOutput: true,
      useAudioOutput: true,
      useChime: true,
      timer: 5000,
      myMagicWord: false
    },
    micConfig: { // put there configuration generated by auto-installer
      recorder: "arecord",
      device: "plughw:1",
    },
    customActionConfig: {
      autoMakeAction: false,
      autoUpdateAction: false, // in RPI, gaction CLI might have some trouble.(current version should be 2.2.4, but for linux-arm, Google haven't updated) so leave this as false in RPI. I don't know it is solved or not.
      actionLocale: "en-US", // At this moment, multi-languages are not supported, sorry. Someday I'll work.
    },
    recipes: [ "with-MMM-Hotword.js", "with-MMM-TelegramBot.js" ],
    profiles: {
      "default": {
        profileFile: "default.json",
        lang: "de-DE"
      }
    },
  },
},


{
  module: "MMM-Hotword",
  config: {
	recipes: ["with-AMk2v3_smart-mirror.js"],
    chimeOnFinish: null,
    mic: {
      recordProgram: "arecord",
      device: "plughw:1"
    },
    models: [
      {
        hotwords    : "computer",
        file        : "computer.umdl",
        sensitivity : "0.5",
      },
    ],
    commands: {
      "computer": {
        notificationExec: {
          notification: "ASSISTANT_ACTIVATE",
          payload: (detected, afterRecord) => {
            return {profile:"default"};
          }
        },
        restart:true,
        afterRecordLimit:0
      }
    }
  }
},

	]

};

/*************** DO NOT EDIT THE LINE BELOW ***************/
if (typeof module !== "undefined") {module.exports = config;}

result:
magic mirror does start.
I can speak “computer” and some words like “wetter in zürich” there is a beep and the mmm-assistantMK2 V3 logo starts to be animated and it looks like it expects some input but nothing happens

expected result:
after speaking "computer " and some words the google assistant starts responding

log:

[11:13:22.334] [LOG]    Server started ...
[11:13:22.335] [LOG]    Connecting socket for: updatenotification
[11:13:22.336] [LOG]    Connecting socket for: MMM-AssistantMk2
[11:13:22.337] [LOG]    Connecting socket for: MMM-Hotword
[11:13:22.338] [LOG]    [HOTWORD] MMM-Hotword starts
[11:13:22.339] [LOG]    Sockets connected & modules started ...
[11:13:22.620] [LOG]    Launching application.
[11:13:25.748] [LOG]    [HOTWORD] Recipe is loaded: with-AMk2v3_smart-mirror.js
[11:13:25.836] [LOG]    [AMK2] MMM-AssistantMk2 Version: 3.0.0
[11:13:25.841] [LOG]    [AMK2] RECIPE_LOADED: with-MMM-Hotword.js
[11:13:25.845] [LOG]    [AMK2] RECIPE_LOADED: with-MMM-TelegramBot.js
[11:13:25.846] [LOG]    [AMK2] NO_ACTION_TO_MANAGE
[11:13:25.871] [LOG]    [AMK2] AssistantMk2 v3 is initialized.
[11:13:25.907] [LOG]    [HOTWORD] begins.
[11:13:25.908] [LOG]    [HOTWORD] Detector starts listening.
[11:13:25.941] [LOG]    [AMK2] tmp directory is now cleaned.
[11:13:27.739] [LOG]    [HOTWORD] Detected: computer
[11:13:27.741] [LOG]    [HOTWORD] stops.
[11:13:27.747] [LOG]    [HOTWORD] Final Result: { detected: true, hotword: 'computer', file: null }
[11:13:27.831] [LOG]    [AMK2] QUERY: { type: 'TEXT',
  profile: { profileFile: 'default.json', lang: 'de-DE' },
  key: null,
  lang: null,
  useScreenOutput: true,
  useAudioOutput: true,
  session: 1579428807774,
  status: 'standby',
  secretMode: false,
  sayMode: false }
[11:13:28.816] [LOG]    [AMK2:AS] CONVERSATION:RESPONSE
[11:13:28.891] [LOG]    [HOTWORD] begins.
[11:13:28.892] [LOG]    [HOTWORD] Detector starts listening.
[11:13:58.729] [LOG]    [AMK2:AS] CONVERSATION:RESPONSE
[11:13:58.737] [LOG]    [AMK2:AS] CONVERSATION_ERROR : { Error: 14 UNAVAILABLE: Service unavailable.
    at Object.exports.createStatusError (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/common.js:91:15)
    at ClientDuplexStream._emitStatusIfDone (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client.js:233:26)
    at ClientDuplexStream._receiveStatus (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client.js:211:8)
    at Object.onReceiveStatus (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client_interceptors.js:1306:15)
    at InterceptingListener._callNext (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client_interceptors.js:568:42)
    at InterceptingListener.onReceiveStatus (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client_interceptors.js:618:8)
    at /home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client_interceptors.js:1123:18
  code: 14,
  metadata:
   Metadata { _internal_repr: { 'content-disposition': [Array] }, flags: 0 },
  details: 'Service unavailable.' }
[11:13:58.741] [LOG]    [AMK2:AS] >> This error might happen when improper configuration or invalid Mic setup.
[11:13:58.743] [LOG]    [AMK2] { session: 1579428807774,
  error: 'NO_RESPONSE',
  action: null,
  text: null,
  screen: null,
  audio: null,
  transcription: null,
  continue: false,
  lastQuery:
   { type: 'TEXT',
     profile: { profileFile: 'default.json', lang: 'de-DE' },
     key: null,
     lang: null,
     useScreenOutput: true,
     useAudioOutput: true,
     session: 1579428807774,
     status: 'standby',
     secretMode: false,
     sayMode: false } }
[11:18:03.671] [LOG]    [HOTWORD] Detected: computer
[11:18:03.673] [LOG]    [HOTWORD] stops.
[11:18:03.675] [LOG]    [HOTWORD] Final Result: { detected: true, hotword: 'computer', file: null }
[11:18:04.010] [LOG]    [AMK2] QUERY: { type: 'TEXT',
  profile: { profileFile: 'default.json', lang: 'de-DE' },
  key: null,
  lang: null,
  useScreenOutput: true,
  useAudioOutput: true,
  session: 1579429083751,
  status: 'standby',
  secretMode: false,
  sayMode: false }
[11:18:04.592] [LOG]    [AMK2:AS] CONVERSATION:RESPONSE
[11:18:04.921] [LOG]    [HOTWORD] begins.
[11:18:04.922] [LOG]    [HOTWORD] Detector starts listening.
[11:18:34.579] [LOG]    [AMK2:AS] CONVERSATION_ERROR : { Error: 14 UNAVAILABLE: Service unavailable.
    at Object.exports.createStatusError (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/common.js:91:15)
    at ClientDuplexStream._emitStatusIfDone (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client.js:233:26)
    at ClientDuplexStream._receiveStatus (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client.js:211:8)
    at Object.onReceiveStatus (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client_interceptors.js:1306:15)
    at InterceptingListener._callNext (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client_interceptors.js:568:42)
    at InterceptingListener.onReceiveStatus (/home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client_interceptors.js:618:8)
    at /home/pi/MagicMirror/modules/MMM-AssistantMk2/node_modules/grpc/src/client_interceptors.js:1123:18
  code: 14,
  metadata:
   Metadata { _internal_repr: { 'content-disposition': [Array] }, flags: 0 },
  details: 'Service unavailable.' }
[11:18:34.584] [LOG]    [AMK2:AS] >> This error might happen when improper configuration or invalid Mic setup.
[11:18:34.584] [LOG]    [AMK2] { session: 1579429083751,
  error: 'NO_RESPONSE',
  action: null,
  text: null,
  screen: null,
  audio: null,
  transcription: null,
  continue: false,
  lastQuery:
   { type: 'TEXT',
     profile: { profileFile: 'default.json', lang: 'de-DE' },
     key: null,
     lang: null,
     useScreenOutput: true,
     useAudioOutput: true,
     session: 1579429083751,
     status: 'standby',
     secretMode: false,
     sayMode: false } }

Any help would be appreciated… AFAIK it must be a hotword config I messed up. after spending now 4 hours tweaking I reach out to the community ( pls make me and my kids happy )

davidoesch

ah – progress but not solved (sometimes a hike helps)

changed HOTWORD

in config.js

{
  module: "MMM-Hotword",
  config: {
	recipes: ["with-AMk2v3_smart-mirror.js"],
    chimeOnFinish: null,
    mic: {
      recordProgram: "arecord",
      device: "plughw:1"
    },

  }
},

and in modules\MMM-Hotword\recipes\with-AMk2v3_smart-mirror.js

var recipe = {
  models: [
    {
      hotwords    : "computer",
      file        : "computer.umdl",
      sensitivity : "0.5",
    },
  ],
  commands: {
    "computer": {
      notificationExec: {
        notification: "ASSISTANT_ACTIVATE",
        payload: (detected, afterRecord) => {
          var ret = {
            profile:"default",
            type: "MIC",
          }
          if (afterRecord) {
            ret.type = "WAVEFILE"
            ret.key = "modules/MMM-Hotword/" + afterRecord
          }
          return ret
        }
      },
      restart:false,
      afterRecordLimit: 7
    }
  }
}

exports.recipe = recipe // Don't remove this line.

now it works with computer

but now
result in log

[14:04:37.424] [LOG]    [HOTWORD] Detector starts listening.
[14:04:38.714] [LOG]    [HOTWORD] Final Result: { detected: false }
[14:04:40.875] [LOG]    [HOTWORD] begins.
[14:04:40.971] [LOG]    [HOTWORD] Detector starts listening.
[14:04:41.478] [LOG]    [HOTWORD] Final Result: { detected: false }
[14:04:42.023] [LOG]    [HOTWORD] begins.
[14:04:42.029] [LOG]    [HOTWORD] Detector starts listening.

seems to be related to a loop issue https://github.com/eouia/MMM-Hotword/issues/120 and the devs are working on it …

@davidoesch said in MMM-AssistantMk2 v3 and MMM-Hotword with custom hotword not working:

[14:04:37.424] [LOG] [HOTWORD] Detector starts listening.
[14:04:38.714] [LOG] [HOTWORD] Final Result: { detected: false }
[14:04:40.875] [LOG] [HOTWORD] begins.
[14:04:40.971] [LOG] [HOTWORD] Detector starts listening.
[14:04:41.478] [LOG] [HOTWORD] Final Result: { detected: false }
[14:04:42.023] [LOG] [HOTWORD] begins.
[14:04:42.029] [LOG] [HOTWORD] Detector starts listening.

It happens when your mic is not working properly. To diagnose, full log is needed. What kind of mic are you using?

davidoesch

Hmm

the detector loop happens in one of ten cases-- expecially when I talk to fast

The mic was working flawlessly with the older version of Hotword and AssistantMk2

Nevertheless
the mic used https://www.aliexpress.com/item/32783207436.html?spm=a2g0s.9042311.0.0.57274c4dlSRFlb

arecord --list-devices
**** List of CAPTURE Hardware Devices ****
card 1: AK5371 [AK5371], device 0: USB Audio [USB Audio]
  Subdevices: 0/1
  Subdevice #0: subdevice #0

 lsusb
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 0556:0002 Asahi Kasei Microsystems Co., Ltd
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Another side effect after the update of HOTWORD and mmm-AssistantMk2 v3 : MMM-PIR – can not activate the screen anymore …I discovered that

dmesg
[ 2718.210637] Out of memory: Kill process 1860 (electron) score 572 or sacrifice child
[ 2718.210729] Killed process 1860 (electron) total-vm:604248kB, anon-rss:31300kB, file-rss:31676kB, shmem-rss:198652kB
[ 2718.247983] oom_reaper: reaped process 1860 (electron), now anon-rss:0kB, file-rss:0kB, shmem-rss:198564kB

and yes I rebuilt electron

after a rollback to the old version everything works smoothly

davidoesch

@Sean
here a larger the snippet of the session

09.295] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.297] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.298] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.300] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.302] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.304] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.305] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.308] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.310] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.311] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.313] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.315] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.317] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.319] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.320] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.322] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.324] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.326] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.328] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.332] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.335] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.338] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.340] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.342] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.344] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.351] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.354] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 4000
[14:04:09.358] [LOG]    [AMK2:AS] CONVERSATION:AUDIO 1198
[14:04:09.361] [LOG]    [AMK2:AS] CONVERSATION_ALL_RESPONSES_RECEIVED
[14:04:09.363] [LOG]    [AMK2:AS] CONVERSATION_END:COMPLETED
[14:04:09.363] [LOG]    [AMK2:AS] CONVERSATION_PP:RESPONSE_AUDIO_MAKING
[14:04:09.370] [LOG]    [AMK2:BW] RESPONSE_WAV_FILE_CREATED
[14:04:09.371] [LOG]    [AMK2:AS] CONVERSATION_PP:RESPONSE_AUDIO_CREATED tmp/lastResponse.wav
[14:04:10.499] [LOG]    [AMK2:BW] RESPONSE_WAV_FILE_CREATED
[14:04:10.501] [LOG]    [AMK2:AS] CONVERSATION_PP:RESPONSE_AUDIO_CREATED tmp/lastResponse.wav
[14:04:10.658] [LOG]    [AMK2] { session: null,
  error: null,
  action: { requestId: '5e23e8fb-0000-246c-a6f8-f4f5e8048f8c' },
  text: null,
  screen:
   { help: [ 'Und morgen?', 'Und nächste Woche?' ],
     trysay: 'Sag zum Beispiel …',
     links:
      [ 'http://www.weather.com/wx/today/?lat=51.51&lon=-0.08&locale=en_US&par=google' ],
     text:
      'Sonntag in London\nSonnig · Ndschlg.: 0 % · Feuchtigkeit: 68 ·\nweather.com\n7\n°C\n↑ 8°\n↓ -2°\nMo\n7° -1°\nDi\n6° 1°\nMi\n8° 4°\nDo\n8° 3°\nFr\n7° 4°',
     photos: [],
     path:
      '/home/pi/MagicMirror/modules/MMM-AssistantMk2/tmp/lastScreenOutput.html',
     uri: 'tmp/lastScreenOutput.html' },
  audio:
   { path:
      '/home/pi/MagicMirror/modules/MMM-AssistantMk2/tmp/lastResponse.wav',
     uri: 'tmp/lastResponse.wav' },
  transcription: { transcription: 'Wetter vor', done: true },
  continue: false,
  lastQuery:
   { type: 'TEXT',
     profile: { profileFile: 'default.json', lang: 'de-DE' },
     key: 'Wetter vor',
     lang: null,
     useScreenOutput: true,
     useAudioOutput: true,
     session: null,
     status: 'continue' } }
[14:04:10.738] [LOG]    [AMK2] { session: null,
  error: null,
  action: { requestId: '5e23e6d7-0000-2ad6-87f4-089e082975cc' },
  text: null,
  screen:
   { help: [ 'Und morgen?', 'Und nächste Woche?' ],
     trysay: 'Sag zum Beispiel …',
     links:
      [ 'http://www.weather.com/wx/today/?lat=51.51&lon=-0.08&locale=en_US&par=google' ],
     text:
      'Sonntag in London\nSonnig · Ndschlg.: 0 % · Feuchtigkeit: 68 ·\nweather.com\n7\n°C\n↑ 8°\n↓ -2°\nMo\n7° -1°\nDi\n6° 1°\nMi\n8° 4°\nDo\n8° 3°\nFr\n7° 4°',
     photos: [],
     path:
      '/home/pi/MagicMirror/modules/MMM-AssistantMk2/tmp/lastScreenOutput.html',
     uri: 'tmp/lastScreenOutput.html' },
  audio:
   { path:
      '/home/pi/MagicMirror/modules/MMM-AssistantMk2/tmp/lastResponse.wav',
     uri: 'tmp/lastResponse.wav' },
  transcription: { transcription: 'Wetter vor', done: true },
  continue: false,
  lastQuery:
   { type: 'TEXT',
     profile: { profileFile: 'default.json', lang: 'de-DE' },
     key: 'Wetter vor',
     lang: null,
     useScreenOutput: true,
     useAudioOutput: true,
     session: null,
     status: 'think' } }
[14:04:11.144] [LOG]    [HOTWORD] begins.
[14:04:11.146] [LOG]    [HOTWORD] Detector starts listening.
[14:04:11.496] [LOG]    [HOTWORD] Final Result: { detected: false }
[14:04:11.718] [LOG]    [HOTWORD] begins.
[14:04:11.724] [LOG]    [HOTWORD] Detector starts listening.
[14:04:11.772] [LOG]    [HOTWORD] Final Result: { detected: false }

@davidoesch said in MMM-AssistantMk2 v3 and MMM-Hotword with custom hotword not working:

[14:04:09.371] [LOG] [AMK2:AS] CONVERSATION_PP:RESPONSE_AUDIO_CREATED tmp/lastResponse.wav
[14:04:10.499] [LOG] [AMK2:BW] RESPONSE_WAV_FILE_CREATED
[14:04:10.501] [LOG] [AMK2:AS] CONVERSATION_PP:RESPONSE_AUDIO_CREATED tmp/lastResponse.wav

I don’t know why at this moment, but AMK2 seemed to be activated twice. That makes your issue. Can you send me whole log (from the start of MM)? (eouia0819@gmail.com)

Artech

This post is deleted!

@davidoesch
We found there could be potential memory issue, so we are trying to solve this issue by using smaller memory. I hope this could solve your issue also. Wait a while, sorry.

UNSOLVED MMM-AssistantMk2 v3 and MMM-Hotword with custom hotword not working