Memory Carveouts
Memory Allocation in Virtualized Configurations
In virtualized configurations the display and capture engines require that surfaces be allocated out of physically contiguous memory reserves.
In general, the virtual machines that own the respective engines must allocate memory carveouts to support this. The default memory carveout allocations are:
• 512 MB for single virtual machine (single-VM) configurations
• 128 MB for multiple virtual machine (multi-VM) configurations
The default carveout allocations provide for general use cases. In applications that require large surface allocations, the carveout allocation may have to be increased via the device tree. The carveout allocation is configured by this device node definition:
reserved-memory {
generic_carveout {
status = "okay";
size = <0 0x20000000>;
};
};
The constant 0x20000000 represents the carveout size in bytes. Change it to a value that is appropriate for your use case.
For assistance with changing the device node definition or with identifying the correct device tree file to update, contact your NVIDIA customer engineer.
Memory Allocation in Linux
These tables show the memory carveouts on the NVIDIA® DRIVE™ PX 2 AutoChauffeur (P2379) with native Linux and with Hypervisor and Linux.
Memory Allocation | NVIDIA® DRIVE™ PX 2 AutoChauffeur (P2379) with— |
Native Linux (MB) | Hypervisor & Linux (MB) |
System carveouts | | 344 | | 344 |
Sysinfo carveout | | n/a | | 2 |
Virtualization stack | | n/a | | 209 |
OS carveouts (generic) OS carveouts (ramoops) OS used memory Available memory Memory lost due to alignment Total Linux memory | n/a 2 180 7666 0 7848 | 7848 | 512 2 180 6940 3 7637 | 7637 |
Total memory allocation | | 8192 | | 8192 |
Carveout | Description | NVIDIA® DRIVE™ PX 2 AutoChauffeur (P2379) with— |
Native Linux | Hypervisor & Linux |
Size (MB) | Base Address | Size (MB) | Base Address |
SE | | 1 | EB200000 | | |
APR | | 10 | EB300000 | | |
SCE | | 1 | EBD00000 | | |
SPE | | 1 | 3004B000 | | |
APE | | 1 | EBF00000 | | |
TSECB | | 1 | EC100000 | | |
TSECA | | 1 | EC200000 | | |
WPR2 | | 1 | EC300000 | | |
WPR1 | | 1 | EC400000 | | |
NVDEC | | 1 | EC500000 | | |
VPR | | 186 | EC600000 | | |
MTS | | 128 | F8000000 | | |
ADSP | | 16 | D8300000 | | |
BPMP | | 1 | n/a | | |
RAMOOPS | | 2 | n/a | | |
TOS | | 10 | n/a | | |
Total | 362 | n/a | | n/a |
Memory Carveout Details
This table describes the individual memory carveouts.
Carveout name | Carveout size (KB) |
MemTotal | 7,619,896 |
MemFree | 5,791,508 |
MemAvailable | 6,250,400 |
Buffers | 27,852 |
Cached | 468,564 |
SwapCached | 0 |
Active | 630,444 |
Inactive | 355,084 |
Active(anon) | 489,240 |
Inactive(anon) | 45,604 |
Active(file) | 141,204 |
Inactive(file) | 309,480 |
Unevictable | 16 |
Mlocked | 16 |
SwapTotal | 0 |
SwapFree | 0 |
Dirty | 0 |
Writeback | 0 |
AnonPages | 489,108 |
Mapped | 179,856 |
Shmem | 45,752 |
Slab | 88,320 |
SReclaimable | 49,584 |
SUnreclaim | 38,736 |
KernelStack | 8,832 |
PageTables | 8,504 |
NFS_Unstable | 0 |
Bounce | 0 |
WritebackTmp | 0 |
CommitLimit | 3,809,948 |
Committed_AS | 3,125,276 |
VmallocTotal | 258,998,208 |
VmallocUsed | 0 |
VmallocChunk | 0 |
CmaTotal | 65,536 |
CmaFree | 63,380 |
To calculate memory carveouts
1. At the Hypervisor Debug Console, enter j.
Output similar to the following is displayed.
48740253|HV/c0: Total DRAM Size: 8192 MB
48742104|HV/c0: Total Carveout Size: 346 MB
48746228|HV/c0: Hypervisor:
48748773|HV/c0: Image Size: 320 KB
48751933|HV/c0: Page Tables Init Size: 7.93 MB
48756145|HV/c0: Page Tables Available: 7.87 MB
48760358|HV/c0: CPIO Actual Image: 5.12 MB
48764219|HV/c0: PCT Size: 37 KB
48767116|HV/c0: CPIO Max Image: 12 MB
48770538|HV/c0: Hypervisor Total: 32 MB
48774312|HV/c0: Guest 0 DRAM: 7637 MB
48777734|HV/c0: Physical Memory Growth Factor: 100%
48782386|HV/c0: Guest 1 (Server) DRAM: 4 MB
48786335|HV/c0: Physical Memory Growth Factor: 0%
48790811|HV/c0: Guest 2 (Server) DRAM: 128 MB
48794936|HV/c0: Physical Memory Growth Factor: 0%
48799412|HV/c0: Guest 3 (Server) DRAM: 4 MB
48803361|HV/c0: Physical Memory Growth Factor: 0%
48807837|HV/c0: Guest 4 (Server) DRAM: 8 MB
48811786|HV/c0: Physical Memory Growth Factor: 0%
48816262|HV/c0: Guest 5 (Server) DRAM: 8 MB
48820211|HV/c0: Physical Memory Growth Factor: 0%
48824687|HV/c0: Guest 6 (Server) DRAM: 4 MB
48828636|HV/c0: Physical Memory Growth Factor: 0%
48833112|HV/c0: Guest Total: 7637 MB
48836446|HV/c0: Server Total: 156 MB
48839959|HV/c0: IVC Queues Total Count 47
48843731|HV/c0: IVC Queue Total: 3.56 MB
48847593|HV/c0: Mempool (Server) 0: 4 MB
48851278|HV/c0: Mempool (Server) 1: 64 KB
48855052|HV/c0: Mempool (Server) 2: 4.93 MB
48859001|HV/c0: Mempool (Server) 4: 8 MB
48862687|HV/c0: Mempools Total Count 4
48866197|HV/c0: Mempool (Server) Total: 17 MB
48870498|HV/c0: Total Guest Info Page Size: 448 KB
48875062|HV/c0: Total Trace Buffer Size Size: 0 KB
48879801|HV/c0: Total Virtualization Consumption: 209 MB
System carveout size is reported as Total Carveout Size. Used virtualization stack memory is reported as Total Virtualization Consumption. DRAM usage for each guest is reported in the form Guest <guest_number> DRAM: <memory_used>.
Mempools are reported as Mempool <type> <mempool_number>: <memory_used>. Server mempools are calculated as part of total virtualization consumption as follows:
Total Memory = Total Carveout Size + Guest Total + Guest Mempools + Total Virtualization Consumption.
To check memory available to a guest OS
1. At the Linux rootfs prompt, enter the following command:
hexdump /proc/device-tree/memory@80000000/reg
Output similar to the following is displayed.
0000000 0000 0000 0080 0000 0000 0000 e07f 0000
0000010 0000 0100 0000 0000 0000 0100 405d 0000
0000020 ...
That output is in the format:
Start1 <byte7byte8 byte5byte6 byte3byte4 byte1byte2> Size1 <byte7byte8 byte5byte6 byte3byte4 byte1byte2>
Start2 <byte7byte8 byte5byte6 byte3byte4 byte1byte2> Size2 <byte7byte8 byte5byte6 byte3byte4 byte1byte2>
Actual start1 = 0xbyte8byte7byte6byte5byte4byte3byte2byte1
The memory reported here may be slightly less than that reported for Guest DRAM, because the kernel requires that size1 and size2 both be 2 MB aligned. (These entries are populated by QB/OSL.)
To determine memory consumed by guest OS carveouts
1. At the Linux rootfs prompt, enter the following command:
hexdump /proc/device-tree/reserved-memory/<carveout type>
Output is similar to the following:
hexdump /proc/device-tree/reserved-memory/generic_carveout
0000000 0000 0000 0020 0000
0000008
=> Size 512 MB (Decoding byte7byte8 byte5byte6 byte3byte4 byte1byte2)
2. Derive carveout type from the output of the following command:
dmesg | grep -i carveout
Output similar to the following is displayed.
[ 0.000000] Reserved memory: initialized node generic_carveout, compatible id nvidia,generic_carveout
[ 0.000000] Reserved memory: initialized node ramoops_carveout, compatible id nvidia,ramoops
[ 2.049113] tegra-carveouts tegra-carveouts: generic-0 :dma coherent mem declare 0x00000000dfe00000,536870912
[ 2.058912] tegra-carveouts tegra-carveouts: assigned reserved memory node generic_carveout
To determine CMA memory
• At the Linux rootfs prompt, enter the following command:
zcat /proc/config.gz | grep -i CMA
Output is similar to the following:
.....
CONFIG_CMA_SIZE_MBYTES=64
.....
To determine memory used by the Linux kernel
1. At the Linux rootfs prompt, enter the following command:
cat /proc/meminfo | grep -i MemTotal
The memory amount reported here is equal to the Memory available to the Linux guest, minus the memory consumed by OS carveouts, and minus CMA memory. What remains is the memory used by the Linux kernel for it's own use
The breakup of memory used by Linux kernel can be partially calculated from output from the following command:
dmesg | grep -i memory | grep -i available
Output is similar to the following:
[ 0.000000] Memory: 7036052K/7274496K available (12020K kernel code, 1767K rwdata, 5436K rodata, 1172K init, 2766K bss, 172908K reserved, 65536K cma-reserved)
This data is reported at particular instance of time during boot and can change later during subsequent stages.
Getting Allocation Type Details at Runtime
The SDK can display details about runtime use of individual allocation types.
To display details about individual allocation types
• Enter this command:
cat /sys/kernel/debug/nvmap/iovmm/allocations
The command displays output in this format, with one line per allocation type:
client process PID SIZE
BASE SIZE FLAGS REFS DUPES PINS KMAPS UMAPS SHARE UID
USER weston 3529 149504K
0 512K 101b0001 3 1 0 0 1 1 ffffffc06da61100
0 128K 10020001 2 1 0 0 0 1 ffffffc06da61200
. . . . . . . . . .
. . . . . . . . . .
The first four digits in the FLAGS field (bits 31:16) contain an allocation type ID, which identifies the allocation type. The table below describes the allocation type IDs.
Note: | Allocation type IDs are subject to change in new releases. |
Allocation type ID | Name |
0100 | Egl |
0200 | Camera |
0300 | Codecs |
0400 | Display |
0500 | Mm |
0501 | MmSecurity |
0600 | OpenMax |
0700 | Wsi |
0701 | Glsi |
0800 | Cuda |
0900 | Rm |
0901 | NvRmGpuSwizzler |
0a00 | System |
0b00 | Odm |
0c00 | Gralloc |
0d00 | OpenMaxCamera |
0f00 | Ddk2d |
0f01 | DdkVic |
1000 | Generic |
1001 | GenericInternal |
1002 | GenericImage |
1003 | Color |
1004 | Depth |
1005 | Stencil |
1006 | Accum |
1007 | ClipId |
1008 | Aux |
1009 | Gvo |
100a | Gvi |
100b | Ig |
100c | Texture |
100d | Video |
100e | Font |
100f | Cursor |
1010 | Dma |
1011 | Instance |
1012 | Primary |
1013 | Zcull |
1014 | Unused |
1015 | ShaderProgram |
1016 | OwnerRm |
1017 | Notifier |
1018 | Reserved |
1019 | RenderBuffer |
101a | RingBuffer |
101b | PushBuffer |
101c | Var |
101d | Vpipe |
101e | Semaphore |
101f | ZcullContext |
1020 | BufferObjectArray |
1021 | BufferObjectElementArray |
1022 | BufferObjectCopy |
1023 | BufferObjectPixel |
1024 | BufferObjectUniform |
1025 | BufferObjectFeedback |
1026 | BufferObjectShaderStorage |
1027 | BufferObjectOther |
1100 | Tvmr |
1200 | VideoEnc |
1300 | Tests |
1301 | NvMapTest |
1302 | NvRmGpuTests |
