I/O Stats Web Interface

From Wiki

(Difference between revisions)
Jump to: navigation, search
(Functionality)
(I/O Stats Web Interface)
 
(9 intermediate revisions not shown)
Line 4: Line 4:
== Functionality ==
== Functionality ==
-
As you know most programs/scripts they read, compute and write data during their execution. Usually the processes of reading and writing data is quite simple. You open a file from the disk or you save a file to the disk. These are the so called Input/Output operations also known as I/O. For the kernel(the OS) however, the process is more complicated. Thus in order to simplify things the I/O statistics in Hive are exported from the kernel are collected at the program level.
+
Most programs/scripts read, compute and write data during their execution. The processes of reading and writing data is quite simple - for example, opening a file and saving it on the disk. These are the so called Input/Output operations also known as I/O. For the kernel (the OS) however, the process is more complicated. Thus in order to simplify things the I/O statistics in Hive are exported from the kernel and are collected at the program level.
-
  '''NB:''' Note that there is difference between the actual Disk I/O(the I/O operations executed only on the local disk, raid, san, nas, or other storage you are using) and the total I/O of a program. Each time a program executes a call to the system function read() the kernel collects the read bytes and stores them in the stats for this process. The situation is the same with writes. The distinction comes because a program can read a single file from the disk and store it in memory once, then reread the same file from the memory multiple times. So the total I/O operations for a program are not all disk related I/O operations. This is why we collect separately the total and the disk I/O operations of the process.
+
  '''NB:''' Note that there is a difference between the actual Disk I/O (the I/O operations executed only on the local disk, raid, san, nas,
 +
or other storage you are using) and the total I/O of a program. Each time a program executes a call to the system function read(),
 +
the kernel collects the read bytes and stores them in the stats for this process. It is the same with writes.
 +
The distinction comes because a program can read a single file from the disk and store it in memory once, then reread the same file  
 +
from the memory multiple times. So the total I/O operations for a program are not all disk related I/O operations but also memory related I/O operations. This is why we  
 +
collect separately the total (disk + memory operations) and the disk I/O operations of the process.
-
Since Hive knows the ID of each process it runs, it can easily collect the I/O usage information from it. What it does is to check the I/O usage every 0.2 seconds while the process work and when the process finish store the collected data.
+
Since Hive knows the ID of each process it runs, it can easily collect the I/O usage information from it. What it does is to check the I/O usage every 0.2 seconds while the process works and when the process finishes, store the collected data.
== I/O Usage Data Collection ==
== I/O Usage Data Collection ==
-
All I/O usage is gathered by a daemon called [Multistatsd]. This daemon is responsible for storing the statistical information for CPU time, Memory usage, I/O usage and process executions. It acts as a cache for the statistics because it stores the data in its memory until the first minute of the next hour. When it reaches the first minute it uploads the accumulated data into a PostgreSQL data base. The design of the software was done in this way because we wanted efficiency and performance.
+
All I/O usage is gathered by a daemon called [Multistatsd]. This daemon is responsible for storing the statistical information for CPU time, Memory usage, I/O usage and process executions. It acts as a cache for the statistics because it stores the data in its memory until the first minute of the next hour. When it reaches the first minute it uploads the accumulated data into a PostgreSQL data base. The software was designed in this way to increase efficiency and performance
All I/O operations are recorded in bytes. However since the values are really high when represented in just bytes they are all represented in MBytes under the Hive Web Interface for I/O Stats.
All I/O operations are recorded in bytes. However since the values are really high when represented in just bytes they are all represented in MBytes under the Hive Web Interface for I/O Stats.
Line 23: Line 28:
== I/O Stats - Local Web Interface ==  
== I/O Stats - Local Web Interface ==  
-
The I/O statistics interface can be reached via the 1H Local Interface for the server. The button 'I/O Stats' is located on the left side panel(just bellow the 'Memory stats' button).
+
The I/O statistics interface are accessible via the 1H Local Interface for the server. The button 'I/O Stats' is located on the left side panel(just bellow the 'Memory stats' button).
 +
 
 +
'''NB:''' Note that you will not see the total I/O usage, only the I/O usage for chrooted programs executed through Hive.
 +
 
 +
=== Server Stats ===
Upon accessing the Interface, the the general server information will be loaded. This screen shows you the I/O usage of all users on the server.
Upon accessing the Interface, the the general server information will be loaded. This screen shows you the I/O usage of all users on the server.
Line 29: Line 38:
[[File:Io1.png]]
[[File:Io1.png]]
-
'''NB:''' Note that you will not see the total I/O usage, only the I/O usage for chrooted programs executed through Hive.
+
Clicking on specific Date/Time will load a new screen with detailed statistics per user for the choosen hour.
 +
 
 +
=== User Stats ===
The I/O interface is a bit different from the other interfaces for 1H Hive Statistics. The difference is that you have two separate statistics for the Server and for Users - Disk and Total I/O. You will be able to check in separate screens the disk only reads/writes and the total reads/writes for both Server and for User as shown below:
The I/O interface is a bit different from the other interfaces for 1H Hive Statistics. The difference is that you have two separate statistics for the Server and for Users - Disk and Total I/O. You will be able to check in separate screens the disk only reads/writes and the total reads/writes for both Server and for User as shown below:
Line 35: Line 46:
[[File:Io2.png]]
[[File:Io2.png]]
-
Upon choosing Disk and Memory I/O for user you will see a new screen similar to the one below, that will provide you with more detailed statistics per user:
+
 
 +
Upon choosing Disk and Memory I/O for a user you will see a new screen similar to the one below, that will provide you with more detailed statistics per user:
[[File:Io3.png]]
[[File:Io3.png]]
-
As a whole the 4 different statistics you will be able to check in the interface are:
+
You will see total and average read writes as well as total script executions for the users for the chosen period.
 +
 
 +
 
 +
The 4 different statistics available in the interface are:
* Disk related reads (counted as bytes)
* Disk related reads (counted as bytes)
Line 45: Line 60:
* Total reads (counted as chars, one character is one byte)
* Total reads (counted as chars, one character is one byte)
* Total writes (counted as chars, one character is one byte)
* Total writes (counted as chars, one character is one byte)
 +
 +
Script executions are shown here for clarity and average values are calculated per user and for the whole server.
 +
 +
=== History ===
 +
 +
Similarly to other interfaces you can use the '''History''' button to check statistics for specific user for a defined period of time as well as check the I/O Statistics for specific date. For example, if you need the statistics for only one day in the past or the resource usage for a specific user for a 5-day time frame one week ago - this is where you can get the needed information.
 +
 +
There are 2 options available:
 +
 +
* Calendar - Here you can check statistics for a specific date. Alternatively, you can just click the Today button under Calendar and it will show the statistics that have been generated for the current day.
 +
 +
* Custom User - This is the functionality that will allow you to check user statistics for a specific time frame. All you need to do is enter the user cPanel username and choose the From and To dates.

Latest revision as of 07:45, 27 July 2011

Contents

I/O Stats Overview

Functionality

Most programs/scripts read, compute and write data during their execution. The processes of reading and writing data is quite simple - for example, opening a file and saving it on the disk. These are the so called Input/Output operations also known as I/O. For the kernel (the OS) however, the process is more complicated. Thus in order to simplify things the I/O statistics in Hive are exported from the kernel and are collected at the program level.

NB: Note that there is a difference between the actual Disk I/O (the I/O operations executed only on the local disk, raid, san, nas,
or other storage you are using) and the total I/O of a program. Each time a program executes a call to the system function read(), 
the kernel collects the read bytes and stores them in the stats for this process. It is the same with writes.
The distinction comes because a program can read a single file from the disk and store it in memory once, then reread the same file 
from the memory multiple times. So the total I/O operations for a program are not all disk related I/O operations but also memory related I/O operations. This is why we 
collect separately the total (disk + memory operations) and the disk I/O operations of the process.

Since Hive knows the ID of each process it runs, it can easily collect the I/O usage information from it. What it does is to check the I/O usage every 0.2 seconds while the process works and when the process finishes, store the collected data.

I/O Usage Data Collection

All I/O usage is gathered by a daemon called [Multistatsd]. This daemon is responsible for storing the statistical information for CPU time, Memory usage, I/O usage and process executions. It acts as a cache for the statistics because it stores the data in its memory until the first minute of the next hour. When it reaches the first minute it uploads the accumulated data into a PostgreSQL data base. The software was designed in this way to increase efficiency and performance

All I/O operations are recorded in bytes. However since the values are really high when represented in just bytes they are all represented in MBytes under the Hive Web Interface for I/O Stats.

I/O Stats Web Interface

I/O Stats - Admin Portal Web Interface

This section will be available as soon as Memory statistics are available in the Central Portal.

I/O Stats - Local Web Interface

The I/O statistics interface are accessible via the 1H Local Interface for the server. The button 'I/O Stats' is located on the left side panel(just bellow the 'Memory stats' button).

NB: Note that you will not see the total I/O usage, only the I/O usage for chrooted programs executed through Hive.

Server Stats

Upon accessing the Interface, the the general server information will be loaded. This screen shows you the I/O usage of all users on the server.

Io1.png

Clicking on specific Date/Time will load a new screen with detailed statistics per user for the choosen hour.

User Stats

The I/O interface is a bit different from the other interfaces for 1H Hive Statistics. The difference is that you have two separate statistics for the Server and for Users - Disk and Total I/O. You will be able to check in separate screens the disk only reads/writes and the total reads/writes for both Server and for User as shown below:

Io2.png


Upon choosing Disk and Memory I/O for a user you will see a new screen similar to the one below, that will provide you with more detailed statistics per user:

Io3.png

You will see total and average read writes as well as total script executions for the users for the chosen period.


The 4 different statistics available in the interface are:

Script executions are shown here for clarity and average values are calculated per user and for the whole server.

History

Similarly to other interfaces you can use the History button to check statistics for specific user for a defined period of time as well as check the I/O Statistics for specific date. For example, if you need the statistics for only one day in the past or the resource usage for a specific user for a 5-day time frame one week ago - this is where you can get the needed information.

There are 2 options available:

Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox