mmm assistant mk2

[dvbit]

A2D was not working at first launch.
npm rebuild in the directory did the trick.
Working now

[dvbit]

Hi
Curious it has been working for the first calls.
Now It answers on screen.
Not via speakers (but that could be for sure a problem with my audio setup)
The other thing is that the on screen google assistant prompt on top keeps spinning and no further interaction is possible.

Here is the debug log.
I can see the call is successfully created files inthe tmp directory and have scped them to my laptop and they are ok.
What changed in between was the tests are that I installed a2d .

While I investigate on my audio output to check if it works I post the log in debug mode
I see a suspicious “null” in volume in the log.
I tried a vlc lastResponse.mp3 and it was correctly output

[2020-06-01 12:15:47.539] [LOG]    [ASSISTANT:AS] CONVERSATION:SCREEN object
[2020-06-01 12:15:47.542] [LOG]    [ASSISTANT:AS] CONVERSATION:ACTION { requestId: '5ed37b16-0000-2f10-85f3-089e0827ce88' }
[2020-06-01 12:15:47.543] [LOG]    [ASSISTANT:AS] CONVERSATION:RESPONSE 
[2020-06-01 12:15:47.589] [LOG]    [ASSISTANT:AS] CONVERSATION:AUDIO 1600
[2020-06-01 12:15:47.592] [LOG]    [ASSISTANT:AS] CONVERSATION:AUDIO 1600
[2020-06-01 12:15:47.593] [LOG]    [ASSISTANT:AS] CONVERSATION:AUDIO 1600
[2020-06-01 12:15:47.594] [LOG]    [ASSISTANT:AS] CONVERSATION:AUDIO 1600
[2020-06-01 12:15:47.596] [LOG]    [ASSISTANT:AS] CONVERSATION:AUDIO 1600
[2020-06-01 12:15:47.597] [LOG]    [ASSISTANT:AS] CONVERSATION:AUDIO 1600
[2020-06-01 12:15:47.598] [LOG]    [ASSISTANT:AS] CONVERSATION:AUDIO 768
[2020-06-01 12:15:47.602] [LOG]    [ASSISTANT:AS] CONVERSATION_ALL_RESPONSES_RECEIVED
[2020-06-01 12:15:47.603] [LOG]    [ASSISTANT:AS] CONVERSATION_END:COMPLETED
[2020-06-01 12:15:47.603] [LOG]    [ASSISTANT:AS] CONVERSATION_PP:RESPONSE_AUDIO_PROCESSED
[2020-06-01 12:15:47.604] [LOG]    [MP3] MP3 FILE CREATED 10368 bytes
[2020-06-01 12:15:47.656] [LOG]    [ASSISTANT:SP] CONVERSATION:SCREENOUTPUT_CREATED
[2020-06-01 12:15:47.657] [LOG]    [ASSISTANT] ASSISTANT_RESULT { error: null,
  action: { requestId: '5ed37b16-0000-2f10-85f3-089e0827ce88' },
  text: null,
  screen:
   { text:
      'Bracciano RM / Ora locale\n12:15\nlunedì 1 giugno 2020\nOrario (Edited)',
     links: [],
     photos: [],
     path:
      '/home/pi/MagicMirror/modules/MMM-GoogleAssistant/tmp/lastScreenOutput.html',
     uri: 'tmp/lastScreenOutput.html' },
  audio:
   { path:
      '/home/pi/MagicMirror/modules/MMM-GoogleAssistant/tmp/lastResponse.mp3',
     uri: 'tmp/lastResponse.mp3' },
  transcription: { transcription: 'a che ora è', done: true },
  continue: false,
  volume: null,
  lastQuery:
   { type: 'MIC',
     key: null,
     lang: 'it-IT',
     useScreenOutput: true,
     useAudioOutput: true,
     status: 'standby',
     chime: true } }

[dvbit]

Update: I reinstalled from scratch (again for a2d I had to do a npm rebuild, you may want to take note)

Still the same issue of assistant hanging after query:
I do the query, hotword recognized, query recognized, answer fetched, answer saved in temp and displayed on screen bottom but no audio feedback and gogle spin icon going.
After refreshing the screen goes again in “steady state”

Now I am sure I have a sound output issue.
Meaning that I can play sounds on the mirror via vlc but somehow the plugin cannot.
I guess that somehow the plugin does not receive a feedback of “play ended” from the output.
What I also noticed is that if I refresh the browser by vnc the mirror will go again in a waiting status.
Will continue checking

[Chris]

@Bugsounet

Hello,

First of all, thank you for not stopping your work on this module which is of high quality.

I tried to install the new assistant. I followed the wiki. However, I have a problem.
Indeed, when the mirror is launched, I find myself with the following message:

[2020-06-01 14:04:54.256] [LOG]    [ASSISTANT][ERROR] token.json file not found 

However, I followed the WIKI well and applied the following command:

(env) pi@raspberrypi:~/MagicMirror/modules/MMM-GoogleAssistant $ ACCESSTOKEN=ya29.a0AfH6SMABBPj590X_itMQG616F8QpxddhPVuSXCE4qvZs6znHKbG2n5RggszFMdl0V5HP0lf956H9O7VHlbuGuMxSzkU3pMkdbpFxs485NwfXYl9T7hfLzB0EOPlRTwcifZW-8rIUlZlWXXXXXXXXXXXXXXXXXXX

Do you have any idea about this problem?

Thanking you in advance,

Regards,

[A Former User]

@dvbit try this:
launch it in root directly in RPI terminal (not ssh)

cd ~/MagicMirror
sudo ./node_modules/.bin/electron js/electron.js --no-sandbox

give me result

[A Former User]

@chris: token.json file must be located in ~/MagicMirror/modules/MMM-GoogleAssistant/token.json

[sdetweil]

@Bugsounet u can also run the fixsandbox script from my repo
https://github.com/sdetweil/MagicMirror_scripts

if they use my installer I fix it… otherwise they either need the --nosandbox or run my script

[A Former User]

sam, i suspect a bug in electron

chown root and chmod 4755 ?

[sdetweil]

@Bugsounet yes, what my script does…

not a bug, new feature

[A Former User]

yes but i don’t understand why … with pulseaudio there is no sound but with root it’s works …

There is a permission missing somewhere

[sdetweil]

@Bugsounet sorry, got no idea what the side effects are…

they wanted some additional security

are u launching a player as a background task, or playing direct to the device? if the latter then u get all the security crap

[A Former User]

sam:

in RPI4: it’s works

pi@Raspberry:~/MagicMirror/node_modules/electron/dist $ ls -la
total 107516
drwxr-xr-x 4 pi   pi     4096 mai   20 15:23 .
drwxr-xr-x 3 pi   pi     4096 juin   1 13:37 ..
-rw-r--r-- 1 pi   pi   145945 mai   20 15:23 chrome_100_percent.pak
-rw-r--r-- 1 pi   pi   209147 mai   20 15:23 chrome_200_percent.pak
-rwsr-xr-x 1 root pi  4204060 mai   20 15:23 chrome-sandbox

i have no idea if @dvbit it’s the same

Note: this Issue take me head since 6 month

[sdetweil]

@Bugsounet yes, added in V6 of electron (and chrome 45 I think?) we are running 6.1.7

[A Former User]

pi@Raspberry:~/MagicMirror/node_modules/electron/dist $ ./electron -v
v6.1.12

[sdetweil]

@Bugsounet weird we call for 6.1.7 or above

"electron": "^6.1.7"

[Chris]

@Bugsounet

Hello,
Indeed, I did not understand well in WIKI.

If it can be used for others, you must create a token.js file and insert the code:

ACCESSTOKEN=ya29.a0AfH6SMABBPj590X_itMQG616F8QpxddhPVuSXCE4qvZs6znHKbG2n5RggszFMdl0V5HP0lf956H9O7VHlbuGuMxSzkU3pMkdbpFxs485NwfXYl9T7hfLzB0EOPlRTwcifZW-8rIUlZlWXXXXXXXXXXXXXXXXXXX

Now it works perfectly,

Thank you so much,

Regards,

[A Former User]

but only in 6.x.x branch

actually electron is releasing with v9 ;)

[dvbit]

@dvbit said in mmm assistant mk2:

Update: I reinstalled from scratch (again for a2d I had to do a npm rebuild, you may want to take note)

Still the same issue of assistant hanging after query:
I do the query, hotword recognized, query recognized, answer fetched, answer saved in temp and displayed on screen bottom but no audio feedback and gogle spin icon going.
After refreshing the screen goes again in “steady state”

Now I am sure I have a sound output issue.
Meaning that I can play sounds on the mirror via vlc but somehow the plugin cannot.
I guess that somehow the plugin does not receive a feedback of “play ended” from the output.
What I also noticed is that if I refresh the browser by vnc the mirror will go again in a waiting status.
Will continue checking

could it be this: https://github.com/raspberrypi/firmware/issues/1243

Edit: Nope - vlc outputs sound

IMPORTANT:
The Install script for the audio output was succesfull just the first time.
on second install I skipped the output part as I could not get through indeed.
However the input was working just fine. recorded my voice and could hear it.

Posting here my asoundrc file

pcm.!default {
  type asym
  playback.pcm {
    type plug
    slave.pcm "output"
  }
  capture.pcm {
    type plug
    slave.pcm "input"
  }
}

pcm.output {
  type hw
  card 0
}

ctl.!default {
  type hw
  card 0

Input is : kinect microphones
Output : TV speakers (directly through TV speakers)

[sdetweil]

@Bugsounet it also has sandbox…

we just moved up to V6 from V3 on 2.10, some things newer don’t work on pi…

[A Former User]

@Bugsounet said in mmm assistant mk2:

pi@Raspberry:~/MagicMirror/node_modules/electron/dist $ ls -la
total 107516
drwxr-xr-x 4 pi   pi     4096 mai   20 15:23 .
drwxr-xr-x 3 pi   pi     4096 juin   1 13:37 ..
-rw-r--r-- 1 pi   pi   145945 mai   20 15:23 chrome_100_percent.pak
-rw-r--r-- 1 pi   pi   209147 mai   20 15:23 chrome_200_percent.pak
-rwsr-xr-x 1 root pi  4204060 mai   20 15:23 chrome-sandbox

@dvbit: can you give me your result please ?