• Introduction
• Local management
• Remote management
• Topology
• Topology analysis
• Turning off cores
• Idle states
• Energy
• Read energy
• Counters types
• CPU counters
• Clock frequency
• Change governor
• Change frequency
• Others
• Threads and processes
• Pinning
• Change priority
• CPU usage

---

• Configuration
• IBM Power8 sensors

Introduction

Mammut is structured as a set of modules, each of them managing a specific set of functionalities. In this page you will find instructions and tutorials on how to use the library. The tutorials can also be found in the corrisponding folder on the GitHub repository.

If you use Mammut for scientific purposes, please refer to the paper:

Mammut: High-level management of system knobs and sensors
Daniele De Sensi, Massimo Torquati, Marco Danelutto
SoftwareX, Volume 6, pp. 150 - 154

You can download it from here.

Local management

The following code snippet shows how to initialise the library:

#include <mammut/mammut.hpp>
using namespace mammut;
int main(int argc, char** argv){
    Mammut m;
}
				

Remote management

If you need to monitor a machine different from the one on which the code is running, you simply need to specify the address of that machine, as in the following snippet:

#define MAMMUT_REMOTE
#include <mammut/communicator-tcp.hpp>
#include <mammut/mammut.hpp>
using namespace mammut;
int main(int argc, char** argv){
    Mammut m(new CommunicatorTcp(remoteMachineAddress, remoteMachinePort));
}
				

In this case, a Mammut server should run on the remote machine. Please notice that this is the only modification required if you need to monitor a machine different from the local one. Once you instantiated Mammut, you can require handlers to the different modules. In the following we will describe each module separately, showing the main features provided and how to use them.

---

Topology

The topology module allows to analyse the topology of the machine. It mainly provides the following features:

• Analysis of the relationship between virtual cores, physical cores and CPUs.
• Turn on/off of cores.
• Management of the idle states (C-states).

The following terminology will be coherently used in documentation/function names.

• CPU: A system is composed by one or more CPU. Each of them is composed by one or more physical core.
• Physical Core: The basic computation unit of the CPU.
• Virtual core: When HyperThreading is present, more virtual cores (contextes) will be present on each physical core.

Topology analysis

The following code snippet shows how to get the list of CPUs, physical cores and virtual cores present on the machine:

					
#include <mammut/mammut.hpp>

#include <cassert>
#include <iostream>
#include <unistd.h>

using namespace mammut;
using namespace mammut::topology;
using namespace std;
int main(int argc, char** argv){
    Mammut m;
    Topology* topology = m.getInstanceTopology();
    
    vector<Cpu*> cpus = topology->getCpus();
    vector<PhysicalCores*> physicalCores = topology->getPhysicalCores();
    vector<VirtualCores*> virtualCores = topology->getVirtualCores();
}
				

You can also analyse these relationships at a finer grained level. For example, to only get the virtual cores on CPU 1, you can do:

					
#include <mammut/mammut.hpp>

#include <cassert>
#include <iostream>
#include <unistd.h>

using namespace mammut;
using namespace mammut::topology;
using namespace std;
int main(int argc, char** argv){
    Mammut m;
    Topology* topology = m.getInstanceTopology();
    Cpu* cpuOne = topology->getCpu(1);
    vector<VirtualCores*> virtualCores = cpuOne->getVirtualCores();
}
				

Similarly, you can retrieve the virtual cores associated to a physical core, the physical cores associated to a CPU and so on. You can also retrieve their identifiers, as assigned by the Operating System. Moreover, you can check which flags are present on your CPUs (e.g. sse, avx, etc...). You can find all the available member functions on the API documentation

Turning off cores

Once you have a VirtualCore object, you can turn off the corresponding virtual core, if such possibility is provided. The following code snippet shows how to turn off and then turn on again the virtual core with id 2.

					
#include <mammut/mammut.hpp>

#include <cassert>
#include <iostream>
#include <unistd.h>

using namespace mammut;
using namespace mammut::topology;
using namespace std;
int main(int argc, char** argv){
    Mammut m;
    Topology* topology = m.getInstanceTopology();
    VirtualCore* core = topology->getVirtualCore(2);
    if(!core){
        cout << "Core not present." << endl;
    }
    if(core->isHotPluggable()){
        core->hotUnplug();
        if(core->isHotPlugged()){
            cout << "Failed to offline the core." << endl;
        }else{
            cout << "Core offlined." << endl;
        }
        core->hotPlug();
        cout << "Core online again." << endl;
    }else{
        cout << "Core cannot be offlined." << endl;
    }
}
				

Idle states

The topology module also provides the possibility to analyse and modify the cores idle states. The following code snippet shows a full working example about the idle states management for the virtual core with identifier 0:

#include <mammut/mammut.hpp>

#include <cassert>
#include <iostream>
#include <unistd.h>

using namespace mammut;
using namespace mammut::topology;
using namespace std;

int main(int argc, char** argv){
    Mammut mammut;
    VirtualCore* vc = mammut.getInstanceTopology()->getVirtualCore(0);
    vector<VirtualCoreIdleLevel*> idleLevels = vc->getIdleLevels();
    if(idleLevels.size() == 0){
        cout << "No idle levels supported by Virtual Core 0." << endl;
    }else{
        cout << "The following idle levels are supported by Virtual Core 0:" << endl;
        for(size_t i = 0; i < idleLevels.size(); i++){
            VirtualCoreIdleLevel* level = idleLevels.at(i);
            cout << "[Idle Level: " << level->getLevelId() << "]";
            cout << "[Name: " << level->getName() << "]";
            cout << "[Desc: " << level->getDesc() << "]";
            cout << "[Consumed Power: " << level->getConsumedPower() << "]";
            cout << "[Exit latency: " << level->getExitLatency() << "]";
            cout << "[Absolute Time: " << level->getAbsoluteTime() << "]";
            cout << "[Absolute Count: " << level->getAbsoluteCount() << "]";
            cout << "[Enableable: " << level->isEnableable() << "]";
            cout << "[Enabled: " << level->isEnabled() << "]";
            cout << endl;
        }
    }
}
				

If you need, you can disable and then enable agian a specific idle level in the following way:

					
if(level->isEnableable() && level->isEnabled()){
    level->disable();
    // Do stuff
    level->enable();
}
				

---

Energy

The energy module allows to read the energy consumption of the machine. This feature is currently supported on the following architectures:

• Silvermont, Broadwell, Haswell, Ivy Bridge, Sandy Bridge, Skylake, Xeon Phi KNL. For these architectures, it is possible to read energy consumption of each CPU, of the cores on the CPU, of the DRAM controller and of the integrated graphic card. On some of these machines DRAM or integrated graphic counters may not be available. To use the energy counters, the msr module must be loaded. A sudo modprobe msr should be enough to let it work.
• IBM Power8 In this case the monitoring infrastructure needs to be configured. To do so, please refer to this manual.
• Odroid machines. For these architectures, it is possible to read the total energy consumption if a SmartPower is connected to the machine.
• Other machines with a power supply voltage between 3.0V and 5.25V. For these machines, it is possible to read the total energy consumption if a SmartPower is connected to the machine, similarly to the Odroid case.

Read energy

The following code snippet shows how to read energy consumption.

#include <mammut/mammut.hpp>

#include <cassert>
#include <iostream>
#include <unistd.h>

using namespace mammut;
using namespace mammut::energy;
using namespace std;

int main(int argc, char** argv){
    Mammut m;
    Energy* energy = m.getInstanceEnergy();
    Joules j;

    Counter* counter = energy->getCounter();
    if(!counter){
        cout << "Power counters not available on this machine." << endl;
        return -1;
    }

    counter->reset();
    sleep(2);
    j = counter->getJoules();
    cout << j << " joules consumed in the last 2 seconds." << endl;

    counter->reset();
    sleep(4);
    j = counter->getJoules();
    cout << j << " joules consumed in the last 4 seconds." << endl;
}
				

First of all, an handler to the energy module is obtained (line 13). Then, we require an energy counter (line 16). After that, we measure a code section (lines 22-24). Eventually, we reset the energy counter and we measure another code section (lines 27-29).

Counter types

Different type of energy counters are supported by Mammut.

• Plug counters: They count the power consumption of the entire machine, as seen at the power plug level.
• CPU conters: They count the power consumption of the CPUs and of individual components of the CPU.

To check the types of available counters, just call getCountersTypes() on the Energy object. This call will return a vector of avaiable counters types.
By default, the getCounter() call will return the most fine grained counter among those available. For example, if both Plug and CPU counters are availble, the getCounter() call will return the CPU counter.
It is also possible to explicitly require a specific counter type. To do that, you just need to specify the type of the counter. E.g., to get a Plug energy counter, you just need to replace the getCounter() call with a getCounter(COUNTER_PLUG) call. If a specific type of counter is not available on the monitored machine, the call will return a NULL value.

CPU counters

If CPU counters are available (this possibility is usually available on newer Intel machines), then it is possible to monitor the energy consumption of each CPU on the machine, of the set of cores on each CPU, of the uncore components of each CPU (e.g. integrated graphic card) or of the DRAMs. Depending on the specific type of architecture, some of these subcounters may not be available. The following code snippet shows how to check for the availability of these subcounters and how to read their energy consumption:

#include <mammut/mammut.hpp>

#include <cassert>
#include <iostream>
#include <unistd.h>

using namespace mammut;
using namespace mammut::energy;
using namespace std;

int main(int argc, char** argv){
    Mammut m;
    Energy* energy = m.getInstanceEnergy();

    /** Gets the energy counters (one per CPU). **/
    CounterCpus* counterCpus = (CounterCpus*) energy->getCounter(COUNTER_CPUS);
    if(!counterCpus){
        cout << "Cpu counters not present on this machine." << endl;
        return -1;
    }

    cout << "Sleeping 10 seconds." << endl;    
    sleep(10);

    cout << "Total Cpus Joules: " << counterCpus->getJoulesCpuAll() << " ";
    if(counterCpus->hasJoulesCores()){
        cout << "Total Cores Joules: " << counterCpus->getJoulesCoresAll() << " ";
    }
    if(counterCpus->hasJoulesDram()){
        cout << "Total Dram Joules: " << counterCpus->getJoulesDramAll() << " ";
    }
    if(counterCpus->hasJoulesGraphic()){
        cout << "Total Graphic Joules: " << counterCpus->getJoulesGraphicAll() << " ";
    }
    cout << endl;
}
				

If you want to read the energy consumption of the components of a specific CPU, just replace the getJoules*All() calls with getJoules*(cpuId) calls. For example, to get the energy consumption of the cores on the CPU with id 1, just call getJoulesCores(1). To further explore the relationship between the CPU id and the cores on that CPU, please read the manual for the topology module.

---

Clock frequency

This module provides the possibility to check the current frequency governor and the current frequency and to modify them. Usually, is not possible to change the frequency of the cores individually but only to change a frequency of a set of cores (typically all those on the same CPU). For this reason, Mammut has the concept of frequency domain. The following code snippet shows how to retrieve the frequency domains available on the monitored machine and the virtual cores associated to each domain:

#include <mammut/mammut.hpp>

#include <cassert>
#include <iostream>
#include <unistd.h>

using namespace mammut;
using namespace mammut::cpufreq;
using namespace mammut::topology;
using namespace std;

int main(int argc, char** argv){
    Mammut m;
    CpuFreq* frequency = m.getInstanceCpuFreq();

    vector<Domain*> domains = frequency->getDomains();
    cout << domains.size() << " frequency domains found" << endl;

    for(size_t i = 0; i < domains.size(); i++){
        Domain* domain = domains.at(i);
        vector<VirtualCore*> virtualCores = domain->getVirtualCores();
        cout << "Virtual cores on domain: " << domain->getId() << ":" << endl;
        for(size_t j = 0; j < virtualCores.size() ; j++){
            cout << virtualCores.at(j)->getVirtualCoreId() << ", ";
        }
        cout << endl;
    }
}

				

Change governor

Once you have a frequency domain, you can change its governor. A governor is a specific algorithm that regulates the frequency management. More information about governors can be found here. You can get a list of governors available on the monitored machine by using a getAvailableGovernors() call on the domain object. The current used governor on the domain can be retrieved by calling the getCurrentGovernor() call. To set a specific governor (e.g. userspace), you can use the setGovernor(GOVERNOR_USERSPACE) call.

Change frequency

If the current governor of the domain is userspace, then you can set the domain to a specific frequency by using the setFrequencyUserspace(f) call. By using the getAvailableFrequencies() member function you can get the list of the frequencies available on the machine (sorted from the lowest to the highest). The following code shows how to set all the domains to the lowest frequency:

#include <mammut/mammut.hpp>

#include <cassert>
#include <iostream>
#include <unistd.h>

using namespace mammut;
using namespace mammut::cpufreq;
using namespace std;

int main(int argc, char** argv){
    Mammut m;
    CpuFreq* frequency = m.getInstanceCpuFreq();
    vector<Domain*> domains = frequency->getDomains();
    for(size_t i = 0; i < domains.size(); i++){
        Domain* domain = domains.at(i);
        vector<Frequency> frequencies = domain->getAvailableFrequencies();

        /** Set the domain to the lowest frequency. **/
        if(domain->isGovernorAvailable(GOVERNOR_USERSPACE) && frequencies.size()){
            Frequency target = frequencies.at(0);
            cout << "Setting domain " << domain->getId() << " to frequency " << target << endl;
            domain->setGovernor(GOVERNOR_USERSPACE);
            domain->setFrequencyUserspace(target);
        }
    }
}

				

Others

Other calls are available, for example to get the current voltage of the domain, or to get the latency required to change the frequency. You can find the full list on the API documentation.

---

Threads and processes management

This module allows to manage the threads and processes currently running on the machine. An handler is associated to each process or thread. To get a process handler you should use the getProcessHandler(pid) call on the module. This handler must be explicitly released with the releaseProcessHandler() call. To get a thread handler, you should call the getThreadHandler(tid) on the ProcessHandler object. This handler must be released as well when not needed. For example, by using the following code you can get an handler to the thread with identifier 9999 in the process with process identifier 9900:

#include <mammut/mammut.hpp>

#include <cassert>
#include <iostream>
#include <unistd.h>

using namespace mammut;
using namespace mammut::task;
using namespace std;

int main(int argc, char** argv){
    Mammut m;
    TasksManager* pm = m.getInstanceTask();
    
    // 9900 is the pid of the process
    ProcessHandler* process = pm->getProcessHandler(9900); 
    // 9999 is the tid of the thread
    ThreadHandler* thread = process->getThreadHandler(9999);
    // Do something with the handler.
    process->releaseThreadHandler(thread);
    pm->releaseProcessHandler(process);
    return 0;
}

				

Pinning

Once you have an handler to a thread (or process), you can force it to run on a specific CPU, physical core or on a specific set of virtual cores, as shown in the following example:

#include <mammut/mammut.hpp>

#include <cassert>
#include <iostream>
#include <unistd.h>

using namespace mammut;
using namespace mammut::task;
using namespace mammut::topology;
using namespace std;

int main(int argc, char** argv){
    Mammut m;
    TasksManager* pm = m.getInstanceTask();
    Topology* topology = m.getInstanceTopology();
    
    // 9900 is the pid of the process
    ProcessHandler* process = pm->getProcessHandler(9900);
    // 9999 is the tid of the thread
    ThreadHandler* thread = process->getThreadHandler(9999);
    
    // Allows the thread to run on any core on CPU 0
    Cpu* cpu = topology->getCpu(0);
    if(cpu){
        thread->move(cpu);
    }
    
    // Allows the thread to run on the physical core with id 2
    PhysicalCore* pCore = topology->getPhysicalCore(2);
    if(pCore){
        thread->move(pCore);
    }
    
    // Allows the thread to run on the virtual cores with id 0 and 3
    vector<const VirtualCore*> vCores;
    VirtualCore* vCore = topology->getVirtualCore(0);
    if(vCore){
        vCores.push_back(vCore);
    }
    vCore = topology->getVirtualCore(3);
    if(vCore){
        vCores.push_back(vCore);
    }
    thread->move(vCores);
    
    process->releaseThreadHandler(thread);
    pm->releaseProcessHandler(process);
    return 0;
}
		
				

Please notice that the move call (as well as the other calls on process and thread handlers), may return false if the process or the thread does not exist anymore.

Change priority

Another feature provided by Mammut is the possibility to read and set the priority of a process or of a thread, by using the getPriority and setPriority calls on the thread or process handler. The values are platform dependent and are included in the range [MAMMUT_PROCESS_PRIORITY_MIN, MAMMUT_PROCESS_PRIORITY_MAX]. Higher values represent higher priorities. The following code shows how to set a thread to the higher priority:

#include <mammut/mammut.hpp>

#include <cassert>
#include <iostream>
#include <unistd.h>

using namespace mammut;
using namespace mammut::task;
using namespace std;

int main(int argc, char** argv){
    Mammut m;
    TasksManager* pm = m.getInstanceTask();
    
    // 9900 is the pid of the process
    ProcessHandler* process = pm->getProcessHandler(9900);
    // 9999 is the tid of the thread
    ThreadHandler* thread = process->getThreadHandler(9999);
    
    thread->setPriority(MAMMUT_PROCESS_PRIORITY_MAX);    

    process->releaseThreadHandler(thread);
    pm->releaseProcessHandler(process);
    return 0;
}
		
				

CPU usage

By using Mammut, you can also check what is the percentage of time that a process or a thread spent running on the CPU. You can do that by using the getCoreUsage call on the thread or process handler. You can reset that percentage by using the resetCoreUsage call.

---

Configuration

This section contains information on how to configure some Mammut functionalities on some specific architectures. If your architecture is not listed here, it is probably automatically supported by Mammut. If this is not the case, please contact me, providing information about the issue and your architecture.

IBM Power8 Sensors

To monitor Power8 sensors, you need to gather monitoring data from another machine (let's call it Monitor). We first explain how to monitor the Power8 from Monitor, then we show how to extend this in order to let Power8 being able to monitor itself.

1. First of all, you need to connect Monitor to the FSP of Power8. Check that you can reach the FSP from Monitor. The next steps must be executed on Monitor.
2. Download and install ipmitool.
3. Download and install Mammut.
4. Create a /tmp/amester folder.
5. Download and compile Amester. Suppose you cloned the Amester repository in /home/user/amester. After compiling it, enter the /home/user/amester/build/v7_0/linux/ folder. Then do a wget http://danieledesensi.github.io/mammut/download/amester.tcl. Then you need to edit the amester.tcl file, modifying the following fields:
   • fspaddr: You should set here the IP address of the FSP of your Power8 machine.
   • fsppass: The password you use to access the FSP of your Power8 machine.
   • p8host: This is the address of your Power8 machine (ATTENTION: address of the machine, not of its FSP).
6. Start the monitor service by running ./amester --nogui amester.tcl. This command must always be running in order to let the monitoring proceed. As soon as this command is stopped, you will not be able to perform monitoring.
7. You can then use Mammut to perform the energy monitoring, in the same way you would do for the other supported architectures.

To let Power8 be able to monitor itself, you need to perform the following additional steps on Power8:

1. Create a /tmp/amester folder. You need to mirror the /tmp/amester folder of the Monitor on the Power8 one (read only is ok). You can do that by using NSF. You can find here instructions on how to do that on Ubuntu. Check that every write you perform on /tmp/amester on Monitor is mirrored /tmp/amester of Power8 machine.
2. Clone mammut, install it and use it to perform the energy monitoring, in the same way you would do for the other supported architectures.