SystemDataScope

rinigus's picture
Submitted by rinigus on Mon, 10/03/2016 - 23:24
Rating: 
5
Your rating: None Average: 5 (7 votes)

SystemDataScope is a GUI for visualization of collectd datasets. Together with collectd and RRDtool, it provides a system monitoring solution for Sailfish. Upgrade and this release notes are below. See also below for possible problem with SFOS 2.0.2 upgrade.

SystemDataScope reads the system performance data gathered by collectd. Graphs are generated using RRDtool and shown through provided GUI. If collectd is started by a GUI user through systemd, there is a support for starting/stopping the daemon.

Through process plugin of collectd, each program can be monitored with its RAM, CPU and disk access logged and shown on graphs through longer timespan.

The recorded statistics are kept with different time resolutions allowing to keep short-term (hours) as well as long-term data (year).

This release notes:

New feature: SystemDataScope can be used to generate report images that can be used for giving feedback on performance of your device or monitoring results on an app(s) that you selected in collectd configuration. By selecting "Report" in pulley menu, you would initiate generation of the graphs that will be stored, in Sailfish, under /home/nemo/Documents/SystemDataScope/TIMESTAMP. When the report graphs are all ready, a message will appear in SystemDataScope. With the file manager or Gallery app, select and send graphs that you want to share and delete the folder /home/nemo/Documents/SystemDataScope when not needed. This folder will be created if its missing the next time you would need a report. An example report graph is shown as one of the screenshots below.

Polishing user interface: time dialog has been redesigned.

Adding support for the changes in reporting by battery plugin. NB! After upgrade of SystemDataScope and collectd, start collectd, wait few minutes, and regenerate the graph definitions.

Note that the older format for battery plugin is not supported by generation script. I made the choice in dropping the support for older plugin data format to be able to focus on further development of the software. Note that all collectd plugins merged into upstream should have stable data reporting dataset naming and format. My effords are now mainly in polishing existing plugins and the SystemDataScope in general.

Default configuration

The collected statistics are organized in a tree. In the beginning, a list of graphs on a top level are shown. By pressing on a graph, a detailed information regarding that statistics is shown on a list, if it exists. At present, using such scheme, there are three levels used.

The default configuration for collectd provides data on:

  • CPU
    • CPU usage (system, user, nice, idle, ...)
    • Amount of time spent in CPU sleep
      • CPU sleep (time; suspend attempts; duration of an average suspend)
      • Used CPU frequencies
      • Used CPU idle states
  • Battery (charge, current, energy, power consumption, temperature, time to full/low, voltage)
  • Memory and storage
    • RAM usage (free, used, cached, buffered, ...)
    • Storage usage
      • root partition used and free
      • disk stats (I/O time, traffic, operations per second, ...)
  •  Network
    • Overall traffic
    • Traffic on cellular and WiFi interfaces
    • For each interface: traffic, accumulated traffic over displayed time period, errors, packets per second, packets dropped
  • Radio details:
    • Cellular and used internet radio (wifi or cellular) signal strength
    • Whether WiFi or Bluetooth are switched on
    • Bluetooth radio: connected, visible, enabled
  • Processes and Load
    • Number of processes in different states (running, stopped, zombies, ...)
    • Fork rate
    • Load
    • You could monitor the processes you are interested in. For each of them, you get data on CPU usage, RAM usage (rss, code size, virtual memory used, ...), number of running threads and processes, disk traffic induced by the process
  • Uptime

SystemDataScope is build to support multiple different log configurations. The configuration is setup through JSON file that can be autogenerated, specified by user, or downloaded from URL (local or remote). This should facilitate development of the graph generation scripts that are tailored for specific user setups.

Since some statistics are populated as you use the phone, such as used mobile network type, regenerate the configuration after a while again to see the corresponding radio signal strength.

The current implementation is developed on Linux Desktops and Sailfish OS mobile devices.

If something does not work, please send your bug reports via GitHub by opening an Issue (see GitHub page below). It is an open-source project and its an easiest way to contribute to it. If you have improvement suggestions, send them also via GitHub by opening a new issue. Developers are welcome to join.

Upgrade notes:

To get support for new collectd plugins, just regenerate graph configuration. Note that the new stats have to be supported by the generation script.

Upgrade notes for users of collectd versions 2016.07.17-6 and earlier:

For details, see collectd OpenRepos page ( https://openrepos.net/content/rinigus/collectd ), under upgrade notes. In short, for default collectd configuration users, please delete the old datasets and start with the new default configuration (will delete earlier data). For that, stop collectd and remove /home/nemo/.local/share/collectd ( in terminal: rm -rf /home/nemo/.local/share/collectd ) and start collectd again. While the data will be lost from earlier recordings, new databases will be initialized allowing you to record data for 1 year.

SFOS upgrade to 2.0.2 series

This may concern users who have been using collectd/SystemDataScope before upgrading SFOS to 2.0.2. On my device, the upgrade from 2.0.1 to 2.0.2.51 resulted in a change of a hostname from Jolla to Sailfish. As a result, collectd started to record under different folder with the graphs not updated in SystemDataScope. If you have the same problem, stop collectd, move your records to the new folder under .local/shared/collectd, delete old subfolder from where you moved the records, and regenerate the graph definitions in Settings. Start collectd and your old records will be updated as before.

Hopefully, Jolla will not be changing SFOS hostname default frequently. Otherwise, you could also set the hostname or specify folder in /etc/collectd.conf .

Homepage: https://github.com/rinigus/systemdatascope

License: MIT

Official TMO thread: http://talk.maemo.org/showthread.php?t=97756

Screenshots: 

Category:

Keywords:

Application versions: 
AttachmentSizeDate
systemdatascope-0.2.5-1.armv7hl.rpm108.14 KB24/07/2016 - 08:04
systemdatascope-0.2.5-1.i486.rpm113.85 KB24/07/2016 - 08:04
systemdatascope-0.2.7-1.armv7hl.rpm109.42 KB03/08/2016 - 10:25
systemdatascope-0.2.7-1.i486.rpm115.11 KB03/08/2016 - 10:25
systemdatascope-0.3.0-1.armv7hl.rpm110.65 KB31/08/2016 - 20:29
systemdatascope-0.3.0-1.i486.rpm116.52 KB31/08/2016 - 20:29
systemdatascope-0.4.0-1.armv7hl.rpm116.52 KB03/10/2016 - 23:24
systemdatascope-0.4.0-1.i486.rpm123.72 KB03/10/2016 - 23:24
Changelog: 

0.4.0

  • Add report generation
  • Add some support for changes in Sailfish Ambience
  • Support for merging statefs_battery into battery plugin
  • Redesign of Time dialog

 

0.3.0

  • Support for new variable name in cpusleep, cpuidle, and cpufreq plugins. NB! Regenrate graph definitions after upgrade and allowing collectd some time to make new databases.
  • Add progress bar as an indicator for graph generation
  • Add subpage title if the click on the graph opens new set of graphs
  • Small fixes

0.2.7

  • Support for cpufreq plugin
  • Support for cpuidle plugin
  • Support for suspend plugin

0.2.5

  • plugins covering WiFi, Bluetooth and cellular radios
  • reorganization of the graphs

For earlier versions, see github page.

Comments

kaulian's picture

kaulian

Tue, 10/04/2016 - 17:21

Hello,

Since the 0.4 battery graph does not work for my oneplusx port.

rinigus's picture

rinigus

Tue, 10/04/2016 - 18:20

Hi! Thanks for reporting. Battery plugin has been changed due to the merge with collectd upstream.

To fix it, let's see what you have in your configuration.

  1. Try to regenerate SystemDataScope graph definitions (Settings/Generate definitions button). If that wouldn't help, try options below.
  2. Check which plugin do you have enabled in /etc/collectd.conf . For that, run "grep LoadPlugin ~/collectd.conf | grep battery" . If its correct, you should see "LoadPlugin battery". If you are loading statefs_battery, disable it.
  3. You could also check if your collectd.conf corresponds to the one in my tree: https://github.com/rinigus/collectd/blob/sailfish/contrib/sailfish/colle... .
  4. After changes in collectd.conf, you would have to start and stop collectd (Settings in SystemDataScope).

Please let me know if it helped. I am sure we'll get the battery stats working for you. If you still have problems, please pastebin:

  1. collectd.conf from /etc
  2. Status of SystemDataScope (Status pulley, choose Copy to Clipboard, and paste it somewhere where I can read it)

Thank you for reporting the problem! There is some flux still expected in few plugins that I developed for collectd, but it will stabilize eventually when they will make it into upstream.

Looking forward to hear back from you,

rinigus

kaulian's picture

kaulian

Wed, 10/05/2016 - 14:21

Hi,
Regenerate SystemDataScope graph done and grah is now here

Thanks

rinigus's picture

rinigus

Wed, 10/05/2016 - 14:28

Excellent! Thank you for reporting back :)

erlen's picture

erlen

Sun, 09/25/2016 - 15:32

Since the last system update, it not working anymore. no graph printed.
All the systems are launched at the boot.
Do you have an idea how to fixe it ?

rinigus's picture

rinigus

Thu, 09/29/2016 - 19:48

Hi! Have you tried the proposed fix?

rinigus

rinigus's picture

rinigus

Sun, 09/25/2016 - 15:55

For me, the latest upgrade changed default hostname from Jolla to Sailfish. As a result, collectd started recording into /tmp/collectd/Sailfish directory leaving /tmp/collectd/Jolla intact. As a result, SystemDataScope (SDS) was plotting old data from Jolla subdir with the symptoms similar to what you describe. My issue is described in the end of openrepos SDS page, under "SFOS upgrade to 2.0.2 series". So, I suspect that you have been hit with the same issue.

To check if its true, see what directories do you have under /tmp/collectd. If you have 2, that is a problem. To resolve it,

  • stop collectd using SDS Settings
  • go with the file manager to .local/shared/collectd and either move old data from Jolla subdir to Sailfish or (if you don't care about old data) delete Jolla.
  • in SDS, regenerate plot definitions (under Settings, Button "Generate definitions"). This can be done with either collectd running or stopped.
  • start collectd using SDS Settings
  • wait a bit and your stats should be shown.

Please let me know if this was your problem and if it fixed it. If it is not, please paste data from Status (there is a pulldown menu in Status to copy it to clipboard). You could either paste it here or as a new issue in github.

erlen's picture

erlen

Thu, 09/29/2016 - 22:00

Thanks, it works for all excepted as before for the battery, where I have no data, but since I use the program (few months) I have the same problem.

rinigus's picture

rinigus

Thu, 09/29/2016 - 22:34

Good! OK, you are supposed to have battery data as well. So, its either something is wrong with the script that makes figures or maybe collectd is not configured properly.

Maybe we should try to fix battery as well. For that we could either use openrepos or, if you don't mind, maybe you could open an issue on github. The info that I would need:

* please paste data from Status. That's very easy to do on phone: Go to pulley menu, select status, Go to pulley menu again, select "Copy to clipboard". Then you could copy that to github issue, or send it to your pc via email, ...

* please send copy of your collectd conf. That's at /etc/collectd.conf . If you see that you have some other /etc/collectd.conf.rpm{something} you could look on the differences between them. Its possible that you don't have a battery plugin enabled, for example.

In respect to battery plugin, there will be changes in the next version. Namely, statefs extension has been incorporated into upstream and we get full functionality via collectd battery plugin (don't have to use separate statefs_battery). So, you may as well wait a bit and configure it when the next release will be out.

In general, please let me know if something does not work. Otherwise I cannot fix or help you out. On my phone, it all works and I expect that all should work on other devices as well (maybe with exception of suspend attempts due to possible differences in kernels).

cheers,

rinigus

gsalone's picture

gsalone

Wed, 08/03/2016 - 14:51

When trying to install this, I get an error:

cvs-1.11.23-1.1.4.armv7hl.rpm not found on medium 'https://releases.jolla.com/releases/latest/jolla/armv7hl/'

Any hint how to get this?

 

rinigus's picture

rinigus

Wed, 08/03/2016 - 15:53

No, not really. Sorry, for me it just installed it as a dependency. What's your device? Is it Jolla* or some ported one?

gsalone's picture

gsalone

Fri, 08/12/2016 - 00:58

First Jolla phone. Not ported one.
Having said that, it installed beautifully on Jolla C.

rinigus's picture

rinigus

Sun, 08/28/2016 - 15:55

Hi! I have updated rrdtool package with the reduced number of dependencies. You could try it again and I hope it would fix your problem. Please let me know if it still does not work.

rinigus's picture

rinigus

Fri, 08/12/2016 - 07:59

I am currently testing the reduction of the number of dependencies by the packages. It looks that rrdtool dependency on some standard packages induced rather large chain of additional package dependencies on sailfish. since I am working on getting the collectd plugins merged upstream, it would take some time before I get these changes published as a new release.

PamNor's picture

PamNor

Tue, 07/05/2016 - 11:26

SystemDataScope is now running and displaying data in Overview page.

When collectd is started two folders is created:

/tmp/collectd/jolla

/tmp/collectd/Jolla

 

To get it Work:

* Collectd folder while running: This must be set to /tmp/collectd/Jolla. Seems that this folder is hardcoded.

Python script has to be run against /tmp/collectd/Jolla

Unable to display data when collectd is stopped

Edit1:

Deleted folders:

/tmp/collect

~/.local/share/collectd

collectd folder wile running: /tmp/collectd/Jolla

collectd folder while stopped: ~/.local/share/collectd/Jolla

Start collectd: OK. Grid in Overview. Only one folder (Jola) in /tmp/collectd

Stop collectd and close SystemDataScope

Start SystemCollectDataScope: No grids in Overview

~/.local/share/collectd/Jolla -> full of folders and data.

/tmp/collectd -> empty.

Edit2
Collectd folder when stopped -> /home/nemo/.local/share/collectd/Jolla
Now working offline.
Thanks for great work and support.

rinigus's picture

rinigus

Tue, 07/05/2016 - 13:50

PamNor, thank you for reporting back and making it work! the feedback is very helpful and I'll try to make the app usage simpler in future.

rinigus's picture

rinigus

Tue, 07/05/2016 - 07:01

Reply to Edit1:

I have uploaded my Settings screeshot as an additional screenshot in app description. The difference that I see is a use of ~ in your settings as opposed to /home/nemo in mine. Please set

collectd folder while stopped: /home/nemo/.local/share/collectd/Jolla

I think this should fix it. Alternatively, you could delete folder name fully (do it only for 'while stopped'), close GUI and start it again. Then it should detect the correct folder. For auto-detection to work, there should be no space in folder name.

From your description, colletd is running and stop/starting as it should.

Please report back and tell if it helped. Thank you for your reports!

rinigus's picture

rinigus

Mon, 07/04/2016 - 22:59

Thank you for the  feedback!

This is strange that you get two directories under /tmp/collectd: jolla and Jolla. I have only Jolla and it maybe due to the hardware adaptation on Nexus 4. Which phone/tablet do you have?

Just a bit info for background. When you start collectd, systemd runs a small script that takes folder from /home/nemo/.local/share/collectd and syncs it to /tmp/collectd. As soon as you stop collectd, the same script syncs it back. NB! If your device crashes, you would loose the recorded data since the last stop. I have been trying to get systemd timers to sync data, but they seem to be very unreliable

In your case, for some reason, you have 2 subfolders in /tmp/collectd. One of them has a real data: /tmp/collectd/Jolla. What's in the other one, no clue (maybe you could post ls /tmp/collectd/jolla ?). To see data when you stop, go to Settings and set the corresponding folder to /home/nemo/.local/share/collectd/Jolla (same Jolla as in /tmp).

If you see graphs in Overview then you could navigate to more specific graphs by touching the graph. The graphs that can be navigated 'into' are marked with a small right arrow on the right of the graph.

It seems that configuration part is rather confusing and messy. If you have an idea how to improve it, please go to GitHub and open an issue. There it would be appropriate to discuss how to improve the GUI

PamNor's picture

PamNor

Mon, 07/04/2016 - 20:05

Running python script
Loading file:///home/nemo/myconf.json
No graphs in overview
Only pulley menus.
Status:
collectd: running
RRDtool: running

Running SystemDataScope from terminal give error like:
[W] unknown:189 -file:///usr/lib/qt5/qml/Sailfish/Silica/DialogHeader.qml:189: TypeError: Cannot read property 'BackIndicatorDown' of null

edit1
myconf.json 9.96kb
/tmp/collectd/Jolla
/home/nemo/.local/share/collectd/jolla

Have to start my laptop to link you downloadlink from dropbox. could takr some time.

hmm..

rinigus's picture

rinigus

Mon, 07/04/2016 - 20:12

Regarding the warning - I have seen it and it seems harmless (everything works with it)

rinigus's picture

rinigus

Mon, 07/04/2016 - 20:11

10kb is not much - mine with the default settings is towards 300k. So, something got undetected. I get 10k size, if I use a wrong directory. Did you specified as a root for collectd /tmp/collectd ? You should get a directory below that. On my Nexus 4, its

/tmp/collectd/Jolla

So, the script should be on my phone

systemdatascope-makeconfig /tmp/collectd/Jolla > myconf.json

If your phone sets different hostname, you would get something different under /tmp/collectd

rinigus's picture

rinigus

Mon, 07/04/2016 - 19:50

That's should be easier to fix.

After running a script, you should get myconf.json in your home dir: /home/nemo/myconf.json

That's if you redirected its output to that file. So, when you load, you should use:

file:///home/nemo/myconf.json

Thank you for testing, its of a great help!

EDIT: now I can see that you have put the correct path in URL. Can you send the json config by pastebin or some alternative?

EDIT2: or at least how large is it and whether its similar to https://github.com/rinigus/systemdatascope/blob/master/configs/sfos-n4.json

EDIT3: And what are you directories in Settings?

PamNor's picture

PamNor

Mon, 07/04/2016 - 16:25

Running Sailfishos 2.0.1.11 (arm).
rrdtool: installed
from terminal:
systemdatascope
..
...
[w] unknown:-1 - file:///usr/share/systemdatascope/qml/main.qml: file not found

[edit]
I cannot find qml folder. main.qml i in folder systemdatascope

rinigus's picture

rinigus

Mon, 07/04/2016 - 16:38

PamNor, please try the new version. It should have fixed the wrong location of QML (that's what happens when you develop for desktop & sailfish at the same time and don't want to include desktop QMLs into sailfish package) :)

Please let me know if it helped

PamNor's picture

PamNor

Mon, 07/04/2016 - 16:46

SystemDataScope is starting now.
I will continue testing .
Thanks.

rinigus's picture

rinigus

Mon, 07/04/2016 - 16:47

Thank you very much for reporting it!

rinigus's picture

rinigus

Mon, 07/04/2016 - 16:26

PamNor, excellent! It seems to be a bug due to packaging. Please uninstall SDS GUI and I'll update it to a new version. Just wait a bit...

rinigus's picture

rinigus

Mon, 07/04/2016 - 16:09

very strange - maybe something went wrong during install. QML files should be there. If you do

ls /usr/share/systemdatascope/qml

do you see the files? There should be main.qml, other QMLs from https://github.com/rinigus/systemdatascope/tree/master/qml, directory Platform.silica and a symbolic link Platform->Platform.silica .

If these are absent, maybe something happened during install. Try to reinstall it. Maybe download RPM manually and install with

pkcon install-local ....rpm

PamNor's picture

PamNor

Mon, 07/04/2016 - 15:26

Gui start loading for 30 sec and then quits. SystemDataScope process continue to run. Tried to start collectd from terminal in forehand,but no difference. chmod 666 /var/log/collectd.log, no difference.

rinigus's picture

rinigus

Mon, 07/04/2016 - 15:46

PamNor, sorry to hear about. Let's try to fix it.  In general, GUI should run without collectd started and is supposed to display you a help screen. So, let's keep collectd out of the way and fix GUI first.

Just few questions:

  • what platform do you run on - arm or i486?
  • do you have rrdtool installed. The graph generator starts it as "rrdtool" using QProcess. If you type in terminal a command "rrdtool" does it give you help text?
  • try to run systemdatascope in terminal and tell what error messages do you see. I might have to compile debug version, since quite a few messages are available only in debug, but let's see what will happen with your version.

As for collectd, it should be started via systemctl --user and, as far as I have seen, all its messages end up in system journal (journalctl as devel-su). But let's fix GUI startup first.