I/O Stats Web Interface

From Wiki

Revision as of 07:20, 27 July 2011 by Admin (Talk | contribs)
Jump to: navigation, search


I/O Stats Overview


As you know most programs/scripts they read, compute and write data during their execution. Usually the processes of reading and writing data is quite simple. You open a file from the disk or you save a file to the disk. These are the so called Input/Output operations also known as I/O. For the kernel(the OS) however, the process is more complicated. Thus in order to simplify things the I/O statistics in Hive are exported from the kernel are collected at the program level.

NB: Note that there is difference between the actual Disk I/O(the I/O operations executed only on the local disk, raid, san, nas,
or other storage you are using) and the total I/O of a program. Each time a program executes a call to the system function read() 
the kernel collects the read bytes and stores them in the stats for this process. The situation is the same with writes.
The distinction comes because a program can read a single file from the disk and store it in memory once, then reread the same file 
from the memory multiple times. So the total I/O operations for a program are not all disk related I/O operations. This is why we 
collect separately the total and the disk I/O operations of the process.

Since Hive knows the ID of each process it runs, it can easily collect the I/O usage information from it. What it does is to check the I/O usage every 0.2 seconds while the process work and when the process finish store the collected data.

I/O Usage Data Collection

All I/O usage is gathered by a daemon called [Multistatsd]. This daemon is responsible for storing the statistical information for CPU time, Memory usage, I/O usage and process executions. It acts as a cache for the statistics because it stores the data in its memory until the first minute of the next hour. When it reaches the first minute it uploads the accumulated data into a PostgreSQL data base. The design of the software was done in this way because we wanted efficiency and performance.

All I/O operations are recorded in bytes. However since the values are really high when represented in just bytes they are all represented in MBytes under the Hive Web Interface for I/O Stats.

I/O Stats Web Interface

I/O Stats - Admin Portal Web Interface

This section will be available as soon as Memory statistics are available in the Central Portal.

I/O Stats - Local Web Interface

The I/O statistics interface can be reached via the 1H Local Interface for the server. The button 'I/O Stats' is located on the left side panel(just bellow the 'Memory stats' button).

NB: Note that you will not see the total I/O usage, only the I/O usage for chrooted programs executed through Hive.

Server Stats

Upon accessing the Interface, the the general server information will be loaded. This screen shows you the I/O usage of all users on the server.


Clicking on specific Date/Time will load a new screen with detailed statistics per user for the choosen hour.

User Stats =

As you can see the I/O interface is a bit different from the other interfaces for 1H Hive Statistics. The difference is that you have two separate statistics for the Server and for Users - Disk and Total I/O. You will be able to check in separate screens the disk only reads/writes and the total reads/writes for both Server and for User as shown below:


Upon choosing Disk and Memory I/O for user you will see a new screen similar to the one below, that will provide you with more detailed statistics per user:


You will see total and average read writes as well as total script executions for the users for the chosen period.

As a whole the 4 different statistics you will be able to check in the interface are:

Also script executions are show here for clarity and average values are calculated per user and for the whole server.


Similarly to other interfaces you can use the History button to check statistics for specific user for a defined period of time as well as check the I/O Statistics for specific date.

Personal tools