Chatgpt+google STT + elevenlabs api

sdmydbr9

Added voice to the chatgpt model and made an interactive chatbot with voice input and voice output using google stt for transcription and eleven labs api for tts.
This is not final yet and I’m hoping someone will polish this, this is just a proof and concept and will post the complete code once it’s better. Here is the link to the video chatgpt

Todo: 1. Add Hotword detection. Right now depends on notification from the iOS shortcuts.
2. Add offline STT, google stt though more accurate takes up more time.
3. Optimisation of the process. Right now it takes time to complete a request

Rags

@sdmydbr9
Hey Sudhamoy, Its fantastic man. Congratulations.
Do you realise what you have achieved. Great going. Waiting for the finished module.

vijayrraut

@sdmydbr9 It’s amazing, last week I also tried using ChatGPT on Pi zero with wakeword and voice, to use like a smart speaker, also made it interactive using google stt.
Y’day tried to made it multilingual using google translate API. I will put this video soon.
https://youtu.be/b73XRWOFdz4

sdmydbr9

@Rags here is the detailed instructions of how I did it, it’s not polished but I tried by best to be thorough

repo here

soldatino

@sdmydbr9 Wooooow…that looks super interesting

I was able to install and setup everything…almost everything…I just have a problem with the microphone.
I can use the terminal:
To record
arecord --format=S16_LE --rate=16000 --file-type=wav /home/xxx/Desktop/out.wav

To Play
aplay /home/xxx/Desktop/out.wav

…all without problems.

But in MagicMirror, after I invoke the iOS shortcut, the GIF comes up saying “say something” but it’s not responding.

Which microphone are you using? Or what do I have to consider?

Thank you for your valuable feedback.

Best, Marco

sdmydbr9

@soldatino hi could you try this, go to MMM-Chat directory, there is a file called transcript.py, around line 34-36 you will find these

# start recording
frames = sd.rec(int(RATE * RECORD_SECONDS), samplerate=RATE, channels=CHANNELS, blocking=True)

Change it into


# start recording
print("say something...")
frames = sd.rec(int(RATE * RECORD_SECONDS), samplerate=RATE, channels=CHANNELS, blocking=True)

Console log is added to say something, now go to the end of the line and comment out the following line


# delete saved audio file
os.remove(WAVE_OUTPUT_FILENAME)

It should look like this now


# delete saved audio file
#os.remove(WAVE_OUTPUT_FILENAME)

And now run the script in the following way

Python3 transcript.py

It will print on the console “say something ……” try speaking on the mic and wait for it to complete, if everything goes well it will print the transcript on the console. If it does not print it in the console, you will get a wav file in the directory, try playing it and see if the audio is audible. If the audio is not clear you might have to change the rate according to which rate gives the best audio. Let me know if you have any more questions.

soldatino

@sdmydbr9 thank you very much for your answer. With this test I can confirm that my microphone works.
But unfortunately it doesn’t react as expected.

Here are the logs:

When I start MagicMirror I get the following error message:

ERROR! Could not find main module js file for MMM-11-TTS

And as soon as I run the shortcut, the GIF is displayed and nothing happens for 2 minutes.

[17.03.2023 14:44.26.411] [LOG]   say something
[17.03.2023 14:46.20.385] [LOG]   child process exited with code 0
[17.03.2023 14:46.20.394] [LOG]   Transcript extracted: say something...
[17.03.2023 14:46.20.401] [LOG]   Sent TRANSCRIPT_EXTRACTED notification
[17.03.2023 14:46.20.405] [LOG]   Sent OPENAI_REQUEST notification

Maybe it matters that my MagicMirror directory is /home/soldatino/MagicMirror/…and not /home/pi/MagicMirror/.
But I changed my path in this file:

/home/soldatino/MagicMirror/modules/MMM-Chat/transcript.py

And in the config:

			module: 'MMM-NotificationTrigger',
			config: {
                             .....
			 exec: (payload) => `python3 /home/soldatino/MagicMirror/modules/MMM-11-TTS/main.py "${payload.message}"`

Did I miss something?

Thank you.
Best, Marco

sdmydbr9

@soldatino okay, I think it’s because you are using a different username it’s breaking. I will try to add the config options to set the files path in the coming days, for now just check you nodehelper of MMM-Chat.js line 21 and let me know if that helps. You will have to set it to your correct absolute path. Cheers.