XLIO Configuration
You can control the behavior of XLIO by configuring:
The libxlio.conf file
XLIO configuration parameters, which are Linux OS environment variables
XLIO extra API
The installation process creates a default configuration file, /etc/libxlio.conf, in which you can define and change the following settings:
The target applications or processes to which the configured control settings apply. By default, XLIO control settings are applied to all applications.
The transport to be used for the created sockets.
The IP addresses and ports in which you want offload.
By default, the configuration file allows XLIO to offload everything except for the DNS server-side protocol (UDP, port 53) which will be handled by the OS.
In the libxlio.conf file:
You can define different XLIO control statements for different processes in a single configuration file. Control statements are always applied to the preceding target process statement in the configuration file.
Comments start with # and cause the entire line after it to be ignored.
Any beginning whitespace is skipped.
Any line that is empty is skipped.
It is recommended to add comments when making configuration changes.
The following sections describe configuration options in libxlio.conf. For a sample libxlio.conf file, see Example of XLIO Configuration below.
Configuring Target Application or Process
The target process statement specifies the process to which all control statements that appear between this statement and the next target process statement apply.
Each statement specifies a matching rule that all its sub-expressions must evaluate as true (logical and) to apply.
If not provided (default), the statement matches all programs.
The format of the target process statement is:
application-id <program-name|*> <user-defined-id|*>
Option |
Description |
<program-name|*> |
Define the program name (not including the path) to which the control statements appearing below this statement apply. Wildcards with the same semantics as "ls" are supported (* and ?). For example:
|
<user-defined-id|*> |
Specify the process ID to which the control statements appearing below this statement apply. Note
You must also set the XLIO_APPLICATION_ID environment variable to the same value as user-defined-id.
|
Configuring Socket Transport Control
Use socket control statements to specify when libxlio will offload AF_INET/SOCK_STREAM or AF_INET/SOCK_DATAGRAM sockets (currently SOCK_RAW is not supported).
Each control statement specifies a matching rule that all its sub-expressions must evaluate as true (logical and) to apply. Statements are evaluated in order of definition according to "first-match".
Socket control statements use the following format:
use <transport> <role> <address|*>:<port range|*>
Where:
Option |
Description |
transport |
Define the mode of transport:
The default is xlio. |
role |
Specify one of the following roles:
|
address |
You can specify the local address the server is bind to or the remote server address the client connects to. The syntax for address matching is: <IPv4 address>[/<prefix_length>]|*
|
port range |
Define the port range as:
Port range: 0-65536 |
Example of XLIO Configuration
To set the following:
Apply the rules to program tcp_lat with ID B1
Use XLIO by TCP clients connecting to machines that belong to subnet 192.168.1.*
Use OS when TCP server listens to port 5001 of any machine
In libxlio.conf, configure:
application-id tcp-lat B1
use xlio tcp_client 192.168
.1.0
/24
:*:*:*
use os tcp_server *:5001
use os udp_connect *:53
You must also set the XLIO parameter:
XLIO_APPLICATION_ID=B1
XLIO configuration is performed using environment variables. For the full list of XLIO parameters, please see libxlio README file.
XLIO parameters must be set prior to loading the application with XLIO. You can set the parameters in a system file, which can be run manually or automatically.
All the parameters have defaults that can be modified.
On default startup, the XLIO library prints the XLIO version information, as well as the configuration parameters being used and their values to stderr.
XLIO always logs the values of the following parameters, even when they are equal to the default value:
XLIO_TRACELEVEL
XLIO_LOG_FILE
For all other parameters, XLIO logs the parameter values only when they are not equal to the default value.
Beta Level Features Configuration Parameters
The following table lists configuration parameters and their possible values for new XLIO Beta level features. The parameters below are disabled by default.
These XLIO features are still experimental and subject to changes. They can help improve performance of multithread applications.
We recommend altering these parameters in a controlled environment until reaching the best performance tuning.
XLIO Configuration Parameter |
Description and Examples |
XLIO_RING_MIGRATION_RATIO_TX XLIO_RING_MIGRATION_RATIO_RX |
Ring migration ratio is used with the "ring per thread" logic in order to decide when it is beneficial to replace the socket's ring with the ring allocated for the current thread. Each XLIO_RING_MIGRATION_RATIO iteration (of accessing the ring), the current thread ID is checked to see whether the ring matches the current thread. If not, ring migration is considered. If the ring continues to be accessed from the same thread for a certain iteration, the socket is migrated to this thread ring. Use a value of -1 in order to disable migration. Default: -1 |
XLIO_RING_LIMIT_PER_INTERFACE |
Limits the number of rings that can be allocated per interface. For example, in ring allocation per socket logic, if the number of sockets using the same interface is larger than the limit, several sockets will share the same ring. Note
XLIO_RX_BUFS might need to be adjusted in order to have enough buffers for all rings in the system. Each ring consumes XLIO_RX_WRE buffers.
Use a value of 0 for an unlimited number of rings. Default: 0 (no limit) |
XLIO_RING_DEV_MEM_TX |
XLIO can use the on-device-memory to store the egress packet if it does not fit into the BF inline buffer. This improves application egress latency by reducing the PCI transactions. Using XLIO_RING_DEV_MEM_TX, enables the user to set the amount of the on-device-memory buffer allocated for each TX ring. The total size of the on-device-memory is limited to 256k for a single port HCA and to 128k for dual port HCA. Default value is 0 |
XLIO_TCP_CC_ALGO |
TCP congestion control algorithm. The default algorithm coming with LWIP is a variation of Reno/New-Reno. The new Cubic algorithm was adapted from FreeBsd implementation. Use value of 0 for LWIP algorithm. Use value of 1 for the Cubic algorithm. Use value of 2 in order to disable the congestion algorithm. Default: 0 (LWIP). |
XLIO can be loaded using Dynamically Loaded (DL) libraries. These libraries are not automatically loaded at program link time or start-up as with LD_PRELOAD. Instead, there is an API for opening a library, looking up symbols, handling errors, and closing the library.
The example below demonstrates how to load socket() function. Similarly, users should load all other network-related functions as declared in sock-redirect.h:
#include <stdlib.h>
#include <stdio.h>
#include <dlfcn.h>
#include <arpa/inet.h>
#include <sys/socket.h>
typedef int
(*socket_fptr_t) (int
__domain, int
__type, int
__protocol);
int
main(int
argc, const
char
** argv)
{
void
* lib_handle;
socket_fptr_t xlio_socket;
int
fd;
lib_handle = dlopen("libxlio.so"
, RTLD_LAZY);
if
(!lib_handle) {
printf("FAILED to load libxlio.so\n"
);
exit(1
);
}
xlio_socket = (socket_fptr_t)dlsym(lib_handle, "socket"
);
if
(xlio_socket == NULL) {
printf("FAILED to load socket()\n"
);
exit(1
);
}
fd = xlio_socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
if
(fd < 0
) {
printf("FAILED open socket()\n"
);
exit(1
);
}
printf("socket creation succeeded fd = %d\n"
, fd);
close(fd);
dlclose(lib_handle);
return
0
;
}
For more information, please refer to dlopen man page.
For a complete example that includes all the necessary functions, see sockperf’s xlio-redirect.h and xlio_socket-redirect.cpp files.