Interactive System Activity Grapher - Concept and Configuration

Contents

Summary

ISAG is tool for graphical presentation of performance data which is collected using sar/sadc utilities.

Document history

Date

Version

Author

Description

26/1/2003

0.41.0

D. Doubrava <linux_monitorvolnycz>

Actualized list of widgets/menus and variables

2/22/01

0.30.0

D. Doubrava <linux_monitorvolnycz>

New features an newly described variables

12/28/00

0.20.0

D. Doubrava <linux_monitorvolnycz>

New chapters: Internals and Roadmap

12/10/00

0.10.0

D. Doubrava <linux_monitorvolnycz>

Beginning of this document

Typographic convention

Font

Example

Description

Courier (fixed)

make all

File (program) or program variable

Courier (fixed)

draw()

Program function

Bold

sar_switch

Term with special meaning

Blue text

This can be...

Ideas or undefined processes

Data sources

Data sources are datafiles created by sar command in /var/log/sa directory. These files is examined using sar -f command and then is parsed into graphical representation. Default permissions for this directory are only for root's access.

Access to stored data files

If we need access to performance data, on their default location, for another users we have a problem. Good way is to make new user's group and add users, who needs this access, here.


How to do it:

The system have a isag user's group, and this this group has read access to this directory.

Default settings:

drwx------    2 root     root         1024 Nov 22 18:45 /var/log/sa

Needed permissions:

drwxr-x---    2 root     isag         1024 Dec 2  18:45 /var/log/sa

Users, who needs access to sar performance data, must be a members of isag group.

part of /etc/group file

dba:x:503:oracle,alice
isag:x:504:bob,charlie

And now bob and charlie has access to collected data.

Archiving old data files

If we have a old (for example) /var/log/sa/sa23 file, which was created in 23-rd of September, and now is 23-rd of October the new collected data will be appended to existent file. The result is: the file contains two pieces of unrelated data, and isag can make unexpected result/graph.

The best way is archiving performance data for some period, and rename data files to new name. This name can contain date and machine name. For example:

sa02.Dec.lenoch              sa10.Dec.lenoch  sa18.Dec.lenoch
sa04.Dec.brooks.seringas.fr  sa11.Dec.lenoch  sa19.Dec.lenoch
sa05.Dec.lenoch              sa12.Dec.lenoch  sa20.Dec.brooks.seringas.fr
sa07.Dec.lenoch              sa13.Dec.lenoch  sa20.Dec.lenoch
sa08.Dec.lenoch              sa16.Dec.lenoch  sa22.Dec.lenoch
sa09.Dec.lenoch              sa17.Dec.lenoch  sa23.Dec.lenoch

Sample of archiving script

Note: This is buggy. Consider what happens if subdirectory is contained there.

#!/bin/sh

CURRENT="sa`date +%d`"
SA_PATH=/var/log/sa
ARCH_PATH=/home/charlie/source/isag/sa
NEW_UID=isag
NEW_GID=isag

for F in `ls $SA_PATH` ; do
        if [ ! "$CURRENT" = "$F" ] ; then
                N="$F.`ls -l $SA_PATH/$F | awk '{print $6}'``date +%Y`.`uname -n`"
                mv $SA_PATH/$F $ARCH_PATH/$N
                chown $NEW_UID:$NEW_GID $ARCH_PATH/$N
        fi
done

Internal Architecture

The architecture is prepared for various data sources and various outputs.

PartProc/Var names
Functional
envelope
Input filter enginesag_if_
Internal processing enginesag_pf_
Output filter enginesag_of_

Window Parts

Screen Shot

Note: Next two pictures aren't actual, this is only for illustration not as full functionality description.

Menu

This is a main program menu, View pull down menu contains all supported graphs.

Data (Source) Chooser

Data chooser contains a menu, where is list of all data files.

Chart

Chart is biggest part of program's window. Here is a graph from selected data source.

Scale

Scale is used to limit of maximal y-value in selected graph. This helps to eliminate local maximum.

Note: Graph is redrawn after moving mouse pointer outside scale. On-line redrawing is CPU hungry process.

Widgets hierarchy

Widget path

Widget class

Purpose/Description

.

Main window


.menu

frame

Menu frame, relief raised

.menu.file

menubutton

[menu: Program]

.menu.file.m

menu

[menu: Program->...]

.menu.chart

menubutton

[menu: Chart]

.menu.chart.m

menu

[menu: Chart->...]

.menu.opt

menubutton

[menu: Options]

.menu.opt.m

menu

[menu: Options->...]

.menu.help

menubutton

[menu: Help]

.menu.help.m

menu

[menu: Help->...]

.file

frame

Frame for data file and selection menu

.file.lbl

label

Used data file, if no one selected label: (no file)

.file.menu

menubutton

Menu button for selection

.file.menu.m

menu

Pop-up menu with data files

.graph

frame

Frame for graph and scale control

.graph.scale

scale/slider

Scale control, maximal showed value, this can cut-off some peaks

.graph.canv

canvas

Space for chart



Internals

How data are processed and how graph is created. This chapter describes program's internal architecture. Program is designed as fully parametrized. Design avoids code duplicity as much as possible: "generic functions". Some most specific parts are different from generic parts. And all parameters are program variables. This looks as best way to future extensibility, without rewriting/redesigning program, and variability, for example using for different data sources such as sar on other systems.

How are parameters organized

The parameters are arrays indexed by sar_switch (this term is often used in next text). For example: sar -u shows CPU utilization like this

Linux 2.2.16-22 (lenoch)        12/28/2000

12:09:23 PM       CPU     %user     %nice   %system     %idle
12:09:33 PM       all      3.20      0.00      1.20     95.60
12:09:43 PM       all      6.60      0.20      5.00     88.20
12:09:53 PM       all     45.80      0.00     33.20     21.00
12:10:03 PM       all     23.20      0.00     10.20     66.60
12:10:13 PM       all     92.20      0.00      7.80      0.00
Average:          all     35.44      0.02     10.92     53.62

and all array variables with index u are related to CPU utilization. These parameters (indexed variables) contains for example data processing script (sag_if_cmd(_)), Graph labels (ylabel(_), xlabel(_)) or current limits for a graph (val_max(_)).

How program works

I think that this is best way how I can describe program architecture.

First step: Data file must be selected. This is done through Data Chooser as is shown on screen shot. If data file is correctly choose, then program variable sag_if_archive is set to its name. On program beginning/start this variable contains empty string.

When data file is chosen then set_file procedure is called. This procedure has one parameter, data_file_name. The global variable sag_if_archive is set to this name. Next are refreshed sag_if_cmd array, which contains "data transformation script".

Note: sag_if_cmd contains data file names and must be changed after every data source change!

After this change graph is redrawn using new data file. But only in case if some graph was drawn for previous data file. When no graph or no file was chosen before the Chart part remains empty.

Next step: If view or graph is selected using menu or keyboard shortcut then new_chart() procedure is called. Parameter is a related sar_switch for selected graph.

sar sets a actual scale limits and last_graph is set to appropriate value. Next is called generic function draw() with sar_switch as parameter. The draw() function contains common actions for all graphs:

Line processing: Because lines has a similar formats, then most graph data are created using generic function sag_if_proc_line_generic. Parameters are:

During line processing are checked maximal graph values, data time stamp is converted from 12 hours to 24 hours (gnuplot can't care this kind of time representation) if needed, and selected columns are written into temporary data file. We can write down all data columns but it contains redundant information.

Special cases of line processing

If some metrics (or sar switch) requres special processing/care then:

Actually only CPU has specific line processing, because this have instance name in data line. In future network activity will have specific function, there are also instance name (interface name).

List of program variables

NameDescriptionArray?/IndexSaved?
Tcl/Tk variable(s)
envTcl/Tk way how to reach program's environmentYes/env. var. nameno
General purpose variable(s)
gnuplot

Contains a path to executable gnuplot

no/-no
grep

Contains a path to executable grep

no/-no
gunzip

Contains a path to executable gunzip

no/-no
ident

Contains a path to executable ident

no/-no
sar

Contains a path to executable sar

no/-no
sed

Contains a path to executable sed

no/-no
sh

Contains a path to executable sh

no/-no
version   
Variable(s) designed for both (sag/isag) tools
sag_if_archive

Contains name of chosen sar data file

no/- 
sag_if_cmd

The dynamically set (When sag_if_archive is changed) array of commands. The result of paticular command is a parsable textual output

yes/prog_switch 
sag_gr_scale

Contains actual scale for left sided slider (widget .graph.scale or as picture)

  
sag_graph_ht

Default chart height Variable should be used for both planned tools sag/isag

no/-no
sag_graph_wd

Default chart width Variable should be used for both planned tools sag/isag

no/-no
sag_if_dd_last_time(0)   
sag_if_last_time

Contains last evaluated time (from data file), this value is used in case of 'LINUX RESTART' then are all counters set to zero in last valid time.

  
sag_if_reboot_flag

This variable is related to previous. When 'LINUX RESTART' is detected then is set to 1, and with new valid line counters are set to zero then to current value.

  
sag_opt_grid

Flag is set when a grid should be drawn

no/-yes
sag_opt_whole_day

Flag is set when whole day is shown on graph

 yes
sag_if_out_tokens

list of numbers, which tokens from input line is used for output. Using this value is set variable val_max (see function sag_if_proc_line_generic()).

yes/prog_switchno
sag_if_out_tokens2

This list is also used for output. But when secondary Y scale was designed for percentage scale, then usage of this values for scale was impossible. (see also function sag_if_proc_line_generic()).

yes/prog_switchno
Variable(s) designed only for isag tool
isag_cfg_file

Default placement and name of config file.

Note: This variable is set on the beggining of program. (see source variables.m4)

no/- 
isag_enResizing

The variable/flag which is used to prevent strange behaviour during window creation or resizing

no/-no
isag_gr_scale_old

Contains old scale, this is used for decision if scale was changed

  
has_negative

The flag which set if this metric can have negative values

yes/prog_switch 
last_graph

Contains sar_switch for graph/chart

  
needed_version

Contains recommended Tcl/Tk version (8.0), because this has a new features

  
patchlevel   
plotstr(a)   
prog_swtch

Contains the list of supported sar outputs which can be xformed to graph

No/- 
sar_data_mask   
sar_data_path   
isag_title

Contains a string "Sar Activity Grapher", each time when chart is changed, then main window title is changed too: $isag_title, $chart_name

  
val_cur(W)   
view_n(a)   
xlabel   
y2label(a)   
ylabel(a)   



Graphs

This chapter describes a graphs which are generated from sar data files. The screen shots are here. Each section contains sample from sar program related to switch and used data are in different colors.

I/O transfer rate

sar -b

01:02:28 PM tps rtps wtps bread/s bwrtn/s

...

Average: 57.77 51.64 6.13 119.17 32.71


Paging statistics

sar -B

01:02:28 PM pgpgin/s pgpgout/s

...

Average: 1676.61 24.78


Process creation

sar -c

01:02:28 PM proc/s

...

Average: 0.06


Memory and Swap usage

sar -r

01:02:28 PM kbmemfree kbmemused %memused kbmemshrd kbbuffers kbcached kbswpfree kbswpused %swpused

...

Average: 2904 141140 97.98 43890 14496 32080 526648 9600 1.79


Memory activities

This part needed a negative value feature. Metrics can be theoretically symeric.

sar -R

08:44:31 PM frmpg/s shmpg/s bufpg/s campg/s

08:44:35 PM -1.00 3.75 0.00 0.00

...

Average: 2.37 1.43 0.00 0.00

CPU utilization

sar -u

01:02:28 PM CPU %user %nice %system %idle

...

Average: all 31.61 0.03 20.73 47.63


inode status

sar -v

01:02:28 PM dentunusd file-sz %file-sz inode-sz %inode-sz super-sz %super-sz dquot-sz %dquot-sz rtsig-sz %rtsig-sz

...

Average: 16252 60 1.46 16362 99.86 10 3.90 25 2.44 0 0.00


System Switching

sar -w

01:02:28 PM cswch/s

...

Average: 352.83



System Swapping

sar -W

01:02:28 PM pswpin/s pswpout/s

...

Average: 1.85 6.05


Known Problems

Usage

isag [-p datafiles_path] [-c config_file] [-ght gr_height] [-gwd gr_width]
datafile path: path where are stored collected sar data files
config file: file where are stored some parameters, such as partial graphs scale, last showed graph and more (this depends on the version and list of values can be changed during development)
gr_height: Width of chart area, not whole window
gr_width: Height of chart area

Program Roadmap

Features and problems which will be solved in near future

Next part can be also named: Future Directions..

Planned features are.

And few metrics are not useful, such as memory and swap free.

The I/O transfer rate chart is currently without the tps metric, because this is a redundant information.

Small description: This is a system in pre-production phase. 16 Jan 2001 was working day, and peek before 3:00 o'clock is regular backup.

Last picture note: Sampling interval between 8:00 and 19:00 is different from other time and from weekend.

Porting notes

Generally: isag may vary in:

Final note: This list of planned features are not finished. Each useful idea can be added here. Please use the bug report form.

DD (C) 2000, 2001, 2002, 2003. Updated: 26 Jan 2003