Snipek

Rating: 
5
Your rating: None Average: 5 (7 votes)

Voice assistant for Sailfish OS based on Snips software

Release 2.0 highlights:
- Support for Snips installation on the Sailfish OS phone. It gives truly offline voice assistant.
- Built-in skills: Date & Time, Call History
- Works on Jolla 1

Important Notice. Staring from January 31, 2020 Snips assistant is not longer publicly available. It means that Snipek further development will be very challenging and most likely not possible any more.

Snipek is a Sailfish OS app that provides remote microphone and speaker capability for Snips voice platform.

Snips is unique because it doesn’t require external cloud service. All processing (including voice) is entirely done locally, on a device where Snips is installed.

To make this app works you have to separately install Snips software on your Sailfish OS phone or on any another device in your local network (e.g. Raspberry Pi, Debian computer).

Snips installation

Two installation options are supported:

  • Local (Snips is installed on Sailfish OS phone together with Snipek)
  • Remote (Snips is installed on a different device in your local network)

When Snipek and Snips are running on the same device (on Sailfish OS phone), no network connectivity is required. Your phone can be in off-line mode and the voice assistant sill works. To install Snips on Sailfish OS follow this guide.

Keep in mind that Snips is not an open source software. The source code is not publicly available and the use of Snips is governed by Snips Terms of Use.

Snips does not provide binaries for x86 (32-bit) CPU architecture, therefore local installation on Jolla Tablet and any other non-ARM Sailfish OS device is not possible right now.

More information about Snips installation options are outlined here.

Snipek skills

Snipek app connects to Snips as an additional audio server. It sends audio stream captured from the microphone and plays audio data received from Snips. It also provides few built-in skills that enable phone control with the voice commands. Skills are the capabilities of voice assistant i.e. the things that assistant can do with a voice command.

Currently implemented skills are:

  • Date & Time (Reads current time or date)
  • Call History (Reads events from call history e.g. all missed calls)

Languages and translations

Snips software supports following languages and therefore only following languages Snipek can be translated to:

  • German
  • English
  • Spanish
  • French
  • Italian
  • Japanese
  • Portuguese (Brazil)

Currently only English language is provided in Snipek.

Bugs reporting

This is "work in progress" project in beta/experimental stage, so likely contains a number of bugs.

Any comments, issue reports are highly appreciated.

Bugs and feature requests can be reported on GitHub or just via email snipek@mkiol.net.

Snipek is a free application.

Screenshots: 
Application versions: 
AttachmentSizeDate
File harbour-snipek-2.0.1-1.armv7hl.rpm201.18 KB04/06/2020 - 10:10
File harbour-snipek-2.0.2-1.armv7hl.rpm395.39 KB11/06/2020 - 14:32
Changelog: 

version 2.0.2 (beta)
- Bug fixes and small improvements

version 2.0.1 (beta)
- Fixes for bugs discovered on the recent SFOS release

version 2.0.0 (beta)
- support for Snips installation on the Sailfish OS phone. It gives truly offline voice assistant.
- built-in skills: Date & Time, Call History
- works on Jolla 1

version 1.0.0
- support for multiple Snipek instances connected to single Snips system

Comments

objectifnul's picture

Snips downloaded successfully for local install (Xperia X).
snips_start.sh failed though:

Using directory: /home/nemo/.cache/harbour-snipek/harbour-snipek/snips.
Stopping Snips processes...
Check if all needed files are present in /home/nemo/.cache/harbour-snipek/harbour-snipek/snips directory.
Current installation version is 1.0.0 and this script version is 1.0.0.
Starting Snips...
~/.cache/harbour-snipek/harbour-snipek/snips ~
~
Check if all Snips processes are running.
Process mosquitto is running.
Process mosquitto stopped.
----
snipek.log below:

[D] :0 - Logging to file enabled
[D] :0 - MQTT init, state: MqttAgent::MqttState(MqttDisconnected)
[D] :0 - MQTT new URL: "tcp://127.1.0.0:1883"
[D] :0 - MQTT current URL: "tcp://127.1.0.0:1883"
[D] :0 - Staring MQTT thread
[D] :0 - New MQTT action: 1
[D] :0 - Connecting MQTT agent for URL: "tcp://127.1.0.0:1883"
[D] :0 - MQTT ID: "snipek-Cf85"
[D] :0 - MQTT connected changed: false
[D] :0 - Suspend listening requested
[D] :0 - Listening update, current: false desired: false
[W] :0 - Failed to connect MQTT, return code: -1
[D] :0 - Finishing MQTT thread
[D] :0 - MQTT connected changed: false
[D] :0 - Suspend listening requested
[D] :0 - Listening update, current: false desired: false
[D] :0 - Reconnecting attempt: 1
[D] :0 - MQTT init, state: MqttAgent::MqttState(MqttDisconnected)
[D] :0 - MQTT new URL: "tcp://127.1.0.0:1883"
[D] :0 - MQTT current URL: "tcp://127.1.0.0:1883"
[D] :0 - Staring MQTT thread
[D] :0 - New MQTT action: 1
[D] :0 - Connecting MQTT agent for URL: "tcp://127.1.0.0:1883"
[D] :0 - MQTT ID: "snipek-Cf85"
[D] :0 - MQTT connected changed: false
[D] :0 - Suspend listening requested
[D] :0 - Listening update, current: false desired: false
[W] :0 - Failed to connect MQTT, return code: -1
[D] :0 - Finishing MQTT thread
[D] :0 - MQTT connected changed: false
[D] :0 - Suspend listening requested
[D] :0 - Listening update, current: false desired: false
[D] :0 - Reconnecting attempt: 2
[D] :0 - MQTT init, state: MqttAgent::MqttState(MqttDisconnected)
[D] :0 - MQTT new URL: "tcp://127.1.0.0:1883"
[D] :0 - MQTT current URL: "tcp://127.1.0.0:1883"
[D] :0 - Staring MQTT thread
[D] :0 - New MQTT action: 1
[D] :0 - MQTT connected changed: false
[D] :0 - Suspend listening requested
[D] :0 - Listening update, current: false desired: false
[D] :0 - Connecting MQTT agent for URL: "tcp://127.1.0.0:1883"
[D] :0 - MQTT ID: "snipek-Cf85"
[W] :0 - Failed to connect MQTT, return code: -1
[D] :0 - Finishing MQTT thread
[D] :0 - MQTT connected changed: false
[D] :0 - Suspend listening requested
[D] :0 - Listening update, current: false desired: false
[D] :0 - Reconnecting attempt: 3
[D] :0 - MQTT init, state: MqttAgent::MqttState(MqttDisconnected)
[D] :0 - MQTT new URL: "tcp://127.1.0.0:1883"
[D] :0 - MQTT current URL: "tcp://127.1.0.0:1883"
[D] :0 - Staring MQTT thread
[D] :0 - New MQTT action: 1
[D] :0 - MQTT connected changed: false
[D] :0 - Suspend listening requested
[D] :0 - Listening update, current: false desired: false
[D] :0 - Connecting MQTT agent for URL: "tcp://127.1.0.0:1883"
[D] :0 - MQTT ID: "snipek-Cf85"
[W] :0 - Failed to connect MQTT, return code: -1
[D] :0 - MQTT connected changed: false
[D] :0 - Suspend listening requested
[D] :0 - Listening update, current: false desired: false
[D] :0 - Reconnect attempts limit reached
[D] :0 - Finishing MQTT thread
[D] :0 - Checking if Snips is installed...
[D] :0 - proces exit: 0
[D] :0 - Checking if Snips is running...
[D] :0 - proces exit: 1
[D] :0 - Starting Snips...
[D] :0 - proces exit: 1
[D] :0 - Starting Snips...
[D] :0 - proces exit: 1

mkiol's picture

Could you please do following tests:

#1

$ /usr/share/harbour-snipek/snips/snips_start.sh

$ ps -A | grep mosquitto

Does `mosquitto` process is running?

#2

$ /usr/share/harbour-snipek/snips/snips_start.sh -k

$ export LD_LIBRARY_PATH=~/.cache/harbour-snipek/harbour-snipek/snips

$ ~/.cache/harbour-snipek/harbour-snipek/snips/mosquitto

Do you see any error from mosquitto?

objectifnul's picture

#1 returns no output. ps doesn't list mosquitto.

#2:

[nemo@XperiaCed ~]$ /usr/share/harbour-snipek/snips/snips_start.sh -k
Using directory: /home/nemo/.cache/harbour-snipek/harbour-snipek/snips.
Stopping Snips processes...
snips-asr: no process found
snips-tts: no process found
snips-dialogue: no process found
snips-nlu: no process found
snips-hotword: no process found
snips-asr: no process found
mosquitto: no process found
Done. Snips successfuly stopped.
[nemo@XperiaCed ~]$ export LD_LIBRARY_PATH=~/.cache/harbour-snipek/harbour-snipek/snips
[nemo@XperiaCed ~]$  ~/.cache/harbour-snipek/harbour-snipek/snips/mosquitto
1591278990: mosquitto version 1.4.10 (build date Wed, 13 Feb 2019 00:45:38 +0000) starting
1591278990: Using default config.
1591278990: Opening ipv4 listen socket on port 1883.
1591278990: Opening ipv6 listen socket on port 1883.

objectifnul's picture

Something is weird in snipek.log:
[D] :0 - MQTT current URL: "tcp://127.1.0.0:1883"
Why not 127.0.0.1 instead?

mkiol's picture

This is a little embarrassing... but for some reason it works fine on all my SFOS devices.

Could please try this test package:
https://github.com/mkiol/Snipek/raw/master/binary/harbour-snipek-2.0.1-2...

objectifnul's picture

Tried on both Xperia X and Jolla phone. Same: couldn't get snips running (openrepos package or github package). So, there is something in my both device configurations that prevents snips from working, but I have no precise idea of what to be investigated.

mkiol's picture

Hmmm, let's reset the configuration to default.

Could you please:
- uninstall Snipek
- remove both: ~/.config/harbour-snipek and ~/.cache/harbour-snipek
- install Snipek (version from github)
- run script (as nemo): /usr/share/harbour-snipek/snips/snips_download.sh
- check if Snipek works

objectifnul's picture

Done. Same result, snips still not running after snipek configured "local" then started snips (within snipek) without success. I suspect my config is missing some hidden dependency (also tried with all patches unapplied).
Additional info: snips could be started (green strip in settings) provided my customized 'hosts' file (about 10,000 entries) is disabled and my wlan connection is turned off.
Even so, snipek still notifies 'connection problem'

mkiol's picture

I've prepared new test build with better logging.

Could you please install this package:

https://github.com/mkiol/Snipek/raw/master/binary/harbour-snipek-2.0.1-3...

After installing, please:

1. start Snipek and enable logging to file (propably is already enabled)

2. restart Snipek

3. Go to the settings page and set Local snips

4. close the app

5. send to me (snipek@mkiol.net) 3 log files that are in home dir (snipek.log, snips_start.log, snips_download.log)

objectifnul's picture

It's working now (https://github.com/mkiol/Snipek/blob/master/binary/harbour-snipek-2.0.1-...) with minimal built-in skills. How are we supposed to help building and sharing more skills? (despite Sonos' strategy obviously aiming to kill Snips)

mkiol's picture

> How are we supposed to help building and sharing more skills? (despite Sonos' strategy obviously aiming to kill Snips)

Acctually I don't know yet. I'm investigating how to migrate Snipek to different non-Snips "back-end" but didn't find any solution so far. With Snips, most likely further development is not possible any more.

 

objectifnul's picture

In other words, this is a dead project?

mkiol's picture

Snips is mostly dead. Snipek might be but currently it is in a "paused" state.

toxip's picture

Well, it was fun while it lasted but seems like this project will be dead along with snips. Loved this app but with the recent snips news it seems like the days of open snips for makers are over.

https://forum.snips.ai/t/important-message-regarding-the-snips-console/4145

mkiol's picture

I'm surprised and sad. I didn't notice it earlier... Indeed, this information makes Snipek continuation very difficult...

Ezreal's picture

nice!Sailfish is becoming smart!

jollex's picture

Wow that would be amazing!! Everything embeded

lutinotmalin's picture

I really don't understand how to use that app that seems wonderful. How do I connect to Snips ? What should I write for the IP, the port, the site ID ? Shall I create an account somewhere ?

mkiol's picture

To make this app works you have to have Snips server somewhere in your local network. The easiest way is to use Raspberry Pi because it is officially supported platform. The guide how to do it is here: https://docs.snips.ai/getting-started/quick-start-raspberry-pi (relatively easy but not trivial). Alternatively you can also install Snips on any Debian box.

My installation is based on Raspberry Pi 2 and it works great.

mkiol's picture

Just a info. I'm also investigating how to port Snips server to SFOS and provide standalone voice assistant that works without any network connectivity. Maybe it will be included in the next Snipek version... Right now server side (installed on RPi) is needed.

toxip's picture

I had trouble setting this up. My raspbi doesn't have any speakers or mic and this doesn't work either. What am I doing wrong?

mkiol's picture

Speaker and mic attached to a raspberrypi device are not needed because your phone (powered with Snipek) will provide both. Of course, you can use Snipek and raspberrypi's mic in the one system - they are not mutually exclusive. Snips can handle multiple audio sources.

Maybe your Snips installation is not properly setup? Did you follow Quick Start Raspberry Pi guide? What "sam status" command shows? Here is my output:

mkiol@rubi:~> sam status
Connected to device x.x.x.x
OS version ................... Raspbian GNU/Linux 9 (stretch)
Installed assistant .......... Test
Language ..................... en
Hotword ...................... hey_snips
ASR engine ................... snips
Status ....................... Live

Service status:
snips-analytics .............. 0.60.1 (not running)
snips-asr .................... 0.60.1 (running)
snips-audio-server ........... 0.60.1 (not running)
snips-dialogue ............... 0.60.1 (running)
snips-hotword ................ 0.60.1 (running)
snips-nlu .................... 0.60.1 (running)
snips-skill-server ........... 0.60.1 (running)
snips-tts .................... 0.60.1 (running)
toxip's picture

So the problem was my custom hotword. I couldn't get it to work at all but by switching to hey snips I could manage to get the thing respond. Also the Xperia X microphone didn't even work for the record (probably too much noise?) so I was using a headset mic for that. Still, even with the headset it didn't recognise the hotword. I would like to use a personal hotword as it'd be much cooler but that's a minor compromise.

Now the app is working great on my Xperia X but unfortunately the mic and speakers didn't work with Jolla 1 at all despite being connected. I would've wanted to repurpose my Jolla 1 as an always on station for the system but unfortunately couldn't do it because of this bug. Perhaps you're using some weird setup to get the mic data that it doesn't work for J1? Anyway, thanks for this app and introducing me to this wonderful platform. Had a great deal of fun playing around with it ^^

mkiol's picture

You are 100% right. Snipek doesn't work on Jolla 1. My fault, I've tested it only on Jolla C and XperiaX. The problem with Jolla 1 is a very slow audio processing. Accually, it works but there is huge delay ~20-30s. I'm investigating it and maybe will be able to fix it. Many thanks for detecting this bug.

toxip's picture

Need to wait until I get my Jolla C from other side of the world. Meanwhile I would like to ask, is there any chance to build the desktop app on Windows? I'm trying myself but I don't know how to. Some paho-mqtt things seem to give me an error. Any chance you could help me or crossbuild for windows? I'd love to get this running on my Windows PC which has my headset plugged in. Thanks!

mkiol's picture

Right, so currently on a desktop only Linux is supported because it was trival to port SFOS code. To be honest I don't have much experience with Windows and is is more important I don't have any Windows-running machine or VM. In theory I could install one but is it would be quite a big effort...

jollex's picture

Many thx! I discovered snips with your apps!

Historyscholar's picture

Good