Why Manage Threads
Most non-real-time operating systems (OSs) are not very good at allocating threads for latency-sensitive applications. Modern OSs are typically configured to balance the load across various CPUs rather than reduce the latency of the application. When dealing with latency-sensitive applications, these strategies cause the latency to increase significantly.
The increase in latency happens because of non-uniform memory access (NUMA). NUMA servers have memory modules local to each CPU. Whilst this significantly reduces the latency to memory local to a CPU, it makes it much more expensive for remote CPUs to access that same memory. Take as an example a simple multi-threaded application with a reader, parser, and writer threads that receives data from a network card, and sends data back out again on the same card. In a Four-CPU NUMA machine where the OS just distributes threads across CPUs, the number of times the data is moved can be several times greater than in a NUMA machine that has been properly tuned:
The diagram above illustrates the tug-war between constraining resources to reduce latency without constraining them too much so latency is affected. The red rectangles on the left hand side represent various threads that may run on the system, including internal OS threads in an un-tuned system. The green rectangles on the right-hand side represent the same application configured to only use a single CPU. The same application running on the right-hand side will have significantly less latency than on the left hand side simply because the data has to be copied fewer times. However, the reduction in latency comes at a cost; instead of having 4 CPUs to use, the application with the configuration on the right-hand side can only use one CPU. This leads to the tug-war between CPU hops and CPU availability. This is where Pontus Vision’s ThreadManager™ helps tremendously.
ThreadManager™ uses a patent pending technology to optimize the way software threads run on hardware cores by trying to reduce the number of context switches (which add latency), trying to maintain applications that need to communicate with each other as close as possible. The results of the simulation can then become a runnable script that users can use to start their applications, or alternatively, a script that modifies the pinning strategies of running applications, or even RPM files (on Linux) that automatically configure the servers to run the threads in the most optimal way.
Hwloc, Numactl taskset migratepages, start & Thread Manager
One question that often arises is: why do I need ThreadManager™ if I already have all these OS commands to pin applications for me? The answer is simple: you need both because they are complementary, and not mutually exclusive.
These commands allow you to either figure out what the machine layout is (e.g. hwloc on Linux will show the level 1,2,3 cache relationships), or to actually pin the memory of a starting process (e.g. numactl, or migratepages on Linux, or start /node on Windows), or pin the execution of threads to a particular core (e.g. taskset on Linux, or start /affinity on Windows); however, they do not use the intelligence of how the applications are related to each other to make these choices. That’s where PontusVision Thread Manager’s model helps quickly find an optimal configuration. In fact, the simulation results will often be saved in templates that use these very same commands to actually implement the pinning strategies that the model created.
The Pontus Vision ThreadManager™ GUI is what users use to get optimized thread affinity strategies for a given server. The GUI is composed of three main windows, described in the following sections:
Thread Management Window
The Thread Management window allows users to enter information for the thread manager model, as well as view the model’s results. The input information is the target hardware platform, as well as the software that will run on that hardware. These are the main inputs to run the thread manager simulation, which is also shown in this screen.
Here are the main parts:
The toolbar at the top has several buttons that control the main functionality of the window:
- Save Button: saves a simulation into the PontusVision configuration tree.
- Save As Button: saves an existing simulation under a different name.
- Open Button: opens a previously saved simulation from the PontusVision configuration tree.
- Clear Button: clears out the page, erasing any unsaved simulation details.
- Run Simulation: takes the currently configured hardware and software details, and runs a simulation for (by default) 10 seconds.
- Simulation Time Text Box: amount of time (in seconds) that the simulation will run for (normally 10 seconds is sufficient for quite good results).
- Simulation Algorithm: algorithm used by the simulation; you can experiment with various algorithms, but the ‘Fit First’ usually comes up with a very good result.
Hardware and Software Trees
The hardware tree is used to drag and drop the hardware in to the canvas.
The software tree is used to drag and drop the components.
Right side panels
The right side Panel has three main accordion boxes.
The top part of the hardware accordion box has a canvas where users drag and drop servers to set the model’s target hardware platform.
The bottom part of the hardware accordion box has a properties grid. When users click on the properties grid you will see the properties associated with the hardware:
- id – The name of the server
- contextSwitchCosts – a number from 0 to 100 used by the model to penalize context switches
- physicalCpuList – a list of all the numa nodes where each core is located.
- latencyMatrix – a matrix showing the costs of moving data between various cores in the hardware simulation
- junkCores – a list of one or more cores that are sacrificed to run background OS tasks, and any non-latency critical applications.
The top part of the software accordion box has a canvas where users drag and drop pre-configured software components from the software tree.
The bottom part of the software accordion box has a properties grid, that shows properties for the currently selected software component. These are the main properties of a software component:
- Id – The name of the component
- threads – The list of the percentage utilizations for each of the threads (e.g. [10,100,0,0] means the app has 4 threads, with the first usng 10% CPU, the second 100%, and the remaining 2 threads with 0% CPU. Note that these entries are currently manually entered in the model, and should represent the workload of the system.
- allowedCores – a list of cores where this application is allowed to run. This should be used to simulate network drivers, where a NIC is physically attached to a specific CPU, or alternatively, for an app that has to do a lot of I/O with a particular NIC.
If no simulation has been run before, the grid will be blank. To see the simulation results, click on the run simulation button. That will show the solution grid.
solution grid has rows and columns.
Rows represents the core of the hardware.
Columns are the software components that you drop in the software canvas.
The save simulation window appears when the save button is pressed. This window allows users to save the thread management simulation results in to the PontusVision configuration tree. This allows the results of the model to be used to generate configuration scripts that can be used to start or modify currently running processes so their threads are pinned, or bound to the right cores.
This window has three main parts:
The configuration tree on the left hand side allows users to choose where to save the model results. Note that you can add branches to this tree by clicking on the ‘Configure’ button in the home screen. When you select a branch to save your information, two sub branches will appear, one where the model information is stored (e.g. all the drawing configurations, the hardware configuration, the software configurations), and one (underneath the first one), where the application-specific configuration will appear. The template types of these two branches are chosen in the ‘Template Options’ panel on the right hand side.
The template options panel on the right allows the user to choose two PontusVision templates to store the simulation results. The first template is used to store the simulation itself, whereas the second template is used to store specific simulation results. There is usually no need to change the first template’s default, so it is recommended to use the default value unless you are an advanced user. However, the second template, which is used to derive the application start up scripts, should be chosen depending on which operating system is used, and whether the template should be used to change a start-up script, or to generate a dynamic script that can be run to change a currently running process’s thread affinity.
As an example, if the ‘Thread-Mgr/apps/Linux-dynamic-cfg’ template is chosen, this will generate the following text to pin a running process’s threads:
/usr/bin/migratepages $PID “$OLD_NUMA” “1” /usr/bin/taskset -apc 12,13,47,44,15 $PID
This is generated by the following template:
/usr/bin/migratepages $PID “$OLD_NUMA” “^^NUMA_ZONE^^” /usr/bin/taskset -apc ^^CORE_LIST^^ $PID
In contrast, if the ‘Thread-Mgr/apps/Linux-start-cfg’ template is chosen, this will generate the following text to pin a newly created process to the correct threads:
/usr/bin/numactl –preferred 1 /usr/bin/taskset -c 12,13,47,44,15
This is generated from the following template:
/usr/bin/numactl –preferred ^^NUMA_ZONE^^ /usr/bin/taskset -c ^^CORE_LIST^^
In both cases, ^^NUMA_ZONE^^ and ^^CORE_LIST^^ are placeholders that are replaced by the tool with the simulation results for each individual application.
The name of these placeholders is configurable, but unless you are an advanced user, creating your own thread management templates, it is not advisable to change these. Here is a list of the configuration items, default values, and a brief description in case you are feeling adventurous:
|Config Items||Description||Default Template||Default Placeholder (within the Template)|
|Simulation (XML)||Simulation results, including the position of the graphs, scores, etc.||Thread-mgr/Simulation||^^XML_MODEL^^|
|Application (Core List)||List of cores for a specific application (one of the software components chosen by the user)||Thread-mgr/Apps/Linux-dynamic-app||^^CORE_LIST^^|
|Application (Name)||Name of the application (defaults to the name of the instance from the software components canvas)||^^ID^^|
|Application (Computer Name)||Name of computer where the application will run (derived from the hardware id)||^^AGENT^^|
|Application (Physical CPU List)||The physical CPU number; often used to pin memory as well as the threads.||^^NUMA_ZONE^^|
This window appears when the user clicks on the ‘Open Simulation’ toolbar button in the Thread Manager window. This allows the user to open a previously saved thread manager configuration. The following figure shows the Load Simulation window:
The following sections describe the different parts of this window:
Template Type Filter
The template type filter allows the user to filter previously saved simulations on the configuration tree.
The configuration tree allows users to locate previously saved thread manager configurations. The template type filter is used to locate any previously saved instances from the currently selected branch of the tree downwards. Any matching entries appear on the list of simulations on the right hand side.
List of Saved Simulations
The list of saved simulations allows the user to select the entries that match the template type filter types within the configuration tree branch chosen. Double-clicking on the instance will automatically open that simulation, and single clicking, will load the simulation name in the ‘Simulation Name’ text box.
ThreadManager™ works alongside existing techniques to manage threads, and enhances them by automating their implementation. It also significantly reduces testing cycles by automating a lot of the empirical application tests.