Memory Stats Web Interface

From Wiki

Jump to: navigation, search

Contents

Memory Stats Overview

Functionality

Each time a program is run it uses memory. The big problem with memory usage is that during the lifespan of a generic program the memory usage is not constant or simply increasing. It may rise to a certain amount and at some point drop to a small number. This makes it hard to rely on a simple check at the end of the program to collect the memory usage statistics for it.

That being said we decided that another solution is needed: a solution that will give us more accurate results of the memory usage for users on a server. Since Hive knows the ID of each process it runs, it can easily collect the memory usage information from it.

What Hive does is to check the memory usage every 200 milliseconds while the process works then save the average memory usage for this program. In short, if a program runs for 1.3 seconds(1300 milliseconds) the memory usage will be checked 7 times. The total amount of memory usage collected from those 7 checks will be divided by 7 to generate the Average memory usage that program had during its lifespan.

Memory Usage Data Collection

All memory usage is accumulated by a daemon called [Multistatsd]. This daemon is responsible for storing the statistical information for CPU time, Memory usage, I/O usage and process executions. It acts as a cache for the statistics because it stores the data in its memory until the first minute of the next hour. When it reaches the first minute it uploads the accumulated data into a PostgreSQL data base. The software was designed in this way to increase efficiency and performance.

All memory usage information is stored in bytes, but since the values are quite high the statistics in the HIVE web interface are presented in MBytes and GBytes.

Memory Stats Web Interface

Memory Stats - Admin Portal Web Interface

This section will be available as soon as Memory statistics are available in the Central Portal.

Memory Stats - Local Web Interface

The Memory statistics interface can be accessed via the HIVE Local interface - the button 'Memory Stats' on the left side panel(just bellow the 'CPU stats' button).

Server Stats

When you click on it, the first thing you see is the general server information. This screen shows you the memory usage of all users on the server. In this section you can see the total memory allocated for the hour and the average memory allocated per CPU second and per process.

Memstats1.png

NB: Note that you will not see the total memory usage, only the memory usage for chrooted programs executed through Hive.

The total allocated memory is the combined memory allocations of all programs running during that hour. For example if you had a total of 528 MB allocated memory for one hour and during that hour you had 14 applications running for 18 seconds they would have used an average of 29MB per second and 37MB per process. The average values are calculated as follows:

Clicking on specific Date/Time will load a new screen with detailed statistics per user for the choosen hour.

User Stats

Under the User Stats section you see the collected memory usage data per user. This way you can easily identify users with high memory usage.

Memstats2.png

Again the average values are calculated as follows:

By clicking on a specific user you will be redirected to a new page with statistics for the chosen user only.

History

Similarly to other interfaces you can use the History button to check statistics for specific user for a defined period of time as well as check the I/O Statistics for specific date. For example, if you need the statistics for only one day in the past or the resource usage for a specific user for a 5-day time frame one week ago - this is where you can get the needed information.

There are 2 options available:

Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox