.. _SD.Kernel.KernelCustomization:

.. include:: /content/swdocs.rsts

.. spelling::
   WDT
   sys
   tegra
   mce

Kernel Customization
!!!!!!!!!!!!!!!!!!!!

You can manually rebuild the kernel used for the |NVIDIA(r)| |Jetson(tm)| Linux. You must have Internet access for this.

.. _SD.Kernel.KernelCustomization-ObtainingTheKernelSourcesWithGit:

Obtaining the Kernel Sources with Git
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

Prerequisites
#############

- You have installed Git. You can install with the following command::

     $ sudo apt install git-core

- Your system has the default Git port 9418 open for outbound connections.

To sync the kernel sources
##########################

- Get the kernel source by running ``source_sync.sh``::

     $ ./source_sync.sh

Enter a tag name when the script prompts you. The proper tag name is specified in the *Release Notes*. This tag name syncs the sources to the source revision from which the release binary was built.

You can sync the sources to any other Linux tag if you need to do so. To see a list of the available release tags, enter this command in the kernel source repository::

   $ git tag -l tegra-l4t\*

Manually Downloading and Expanding Kernel Sources
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

As an alternative to syncing the kernel sources, you can download the kernel source files and then manually extract them. If possible, NVIDIA recommends that you sync with Git instead.

To manually download and expand the kernel sources
##################################################

#. In your browser, open the page https://developer.nvidia.com/embedded/downloads.

#. Locate and download the Jetson Linux source files for your release.

#. Extract the ``.tbz2`` file::

      $ tar -xjf public_sources.tbz2

#. Extract the kernel source file::

      $ cd Linux_for_Tegra/source/public
      $ tar –xjf kernel_src.tbz2

   This extracts the kernel source to the ``kernel/`` subdirectory.

.. _SD.Kernel.KernelCustomization-BuildingTheKernel:

Building the Kernel
@@@@@@@@@@@@@@@@@@@

Before you build the kernel you must install the Jetson Linux build utilities. Enter the command::

     $ sudo apt install build-essential bc

To build the kernel
###################

#. Sync or download and expand the kernel sources, and install the kernel build utilities, as described above.

#. If you are cross-compiling on a host system (i.e. anything other than the Jetson device you are building for), export the following environment variables::

      $ export CROSS_COMPILE_AARCH64_PATH=<toolchain-path>/aarch64--glibc--stable-2020.08-1

   See `The Jetson Linux Toolchain <AT.JetsonLinuxToolchain>`__ for information on how to download and build the reference toolchains.

#. Enter the command::

      $ mkdir kernel_out

   Where ``kernel_out`` is the directory where the compiled kernel is to be written.

#. Build the kernel::

     $ ./nvbuild.sh -o $PWD/kernel_out

#. Replace ``Linux_for_Tegra/rootfs/usr/lib/modules/$(uname -r)/kernel/drivers/gpu/nvgpu/nvgpu.ko`` with a copy of this file::

      $kernel_out/drivers/gpu/nvgpu/nvgpu.ko

#. Replace ``Linux_for_Tegra/kernel/Image`` with a copy of this file::

      $kernel_out/arch/arm64/boot/Image

   Optionally, archive the installed kernel modules::

      $ cd <modules_install_path>
      $ tar --owner root --group root -cjf kernel_supplements.tbz2 lib/modules

   You can use the installed modules to provide the contents of ``/lib/modules/<kernel_version>/`` on the target system. Before you run ``apply_binaries.sh``, make a copy of this archive to replace the one at this location on the target device::

      Linux_for_Tegra/kernel/kernel_supplements.tbz2

To sign and encrypt kernel, kernel-dtb, and initrd binary files
###############################################################

To support Secure Boot, the kernel, kernel-dtb, and initrd binary files must be signed and encrypted with keys to generate encrypted binary files and signature files.

See the topic :ref:`Secure Boot <SD.Security.SecureBoot>`.

.. todo::
   There used to be a reference to "SD.Security.SecureBoot-SigningAndEncryptingKernelKernelDtbInitrdAndExtlinuxConfFiles" here, but we gutted SecureBoot.rst and have a placeholder there now. Restore this reference when SecureBoot.rst is fleshed out.

Preparing to Build External Kernel Modules
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

These procedures describe how to prepare a set of kernel headers, or a kernel source tree to allow building an out-of-tree kernel module.

To prepare the Jetson Linux kernel headers on a Jetson system
#############################################################

On a Jetson system you can use the files in this directory to build out-of-tree modules without further action::

   /usr/src/linux-headers-$(uname -r)-ubuntu20.04_aarch64

or::

   /lib/modules/$(uname -r)/build/

To prepare the Jetson Linux kernel headers on a non-Jetson system
#################################################################

- Extract the archive and place it in a local directory with the commands::

     $ cd <local_src_dir>
     $ tar -xjf <top>/Linux_for_Tegra/kernel/kernel_headers.tbz2

  Where ``<local_src_dir>`` is a local directory of your choice.

  You can use the files in this directory to build out-of-tree modules::

     <local_src_dir>/linux-headers-$(uname -r)-linux_x86_64/

  To build out-of-tree modules, specify the kernel directory as::

     <local src dir>/linux-headers-$(uname -r)-linux_x86_64/

  .. todo:: Aren't the last two paragraphs duplicative? The commands are identical except for the spelling of the placeholder variable.

To use a manually built kernel source tree
##########################################

-  If you built the kernel from source, use the same source tree to build out-of-tree kernel modules. When building out-of-tree modules, use ``$TEGRA_KERNEL_OUT`` as the kernel directory.

To prepare a kernel source tree that is not built
#################################################

You can build out-of-tree modules that are compatible with a Linux kernel source tree without building the entire kernel image.

#. To extract the kernel source, set any required shell or environment variables, and create the ``.config`` file, follow the instructions provided in these topics:

   - `Obtaining the Kernel Sources with Git <#obtaining-the-kernel-sources-with-git>`__

   - `Manually Downloading and Expanding Kernel Sources <#manually-downloading-and-expanding-kernel-sources>`__

   - `Building the Kernel <#building-the-kernel>`__

#. Enter this command to prepare the kernel source tree for building out-of-tree kernel modules::

       $ make ARCH=arm64 O=$TEGRA_KERNEL_OUT -j<n> modules_prepare

    Where ``<n>`` is the number of parallel processes to be used. NVIDIA recommends using the number of CPUs in your system.

When building out-of-tree modules, point ``$TEGRA_KERNEL_OUT`` to the kernel directory.

To build external kernel modules
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

#. If you are cross-compiling the kernel, ensure that the ``CROSS_COMPILE`` environment variable is set as described in preceding sections.

  .. todo:: What if you're not? No instructions are given. If this is the way to do it, and not one of two alternatives, it should be described so.

#. Enter these commands to build an out-of-tree kernel module::

      $ cd <path_to_module_source>
      $ make ARCH=arm64 –C <kernel_directory> M=$(pwd)

   Enter these commands to strip unneeded symbols from the kernel module::

      $ <tool_chain_path>/aarch64-linux-gnu-strip -–strip-unneeded <path-of-kernel-module.ko>

Using the Jetson Linux Real-Time Kernel Package
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

   **Not Yet Supported in This Release** This section does not apply for this release. real-time kernel is still work in progress.

|NVIDIA(r)| |Jetson AGX Xavier(tm)| series processors offer the Early Access Release quality of the Jetson Linux Real-Time kernel support for r32 releases, starting with r32.5. This section explains how to install and remove Jetson Linux real-time kernel packages on your system through the NVIDIA repository.


Prerequisites
#############

- Upgrade all Jetson Linux packages to the same version of the point release as the Real-Time kernel.


To install the real-time kernel package
#######################################

#. Open the ``apt`` source configuration file as a superuser with a text editor. The file is at::

      /etc/apt/sources.list.d/nvidia-l4t-apt-source.list

#. Add a new entry for the ``rt-kernel`` repository::

      deb https://repo.download.nvidia.com/jetson/rt-kernel r<rel> main

   Where ``<rel>`` is the current release number, e.g ``32.6.1``.

#. Enter this command::

      $ sudo apt update

#. Install the real-time kernel packages::

      $ sudo apt install nvidia-l4t-rt-kernel nvidia-l4t-rt-kernel-headers

#. Reboot.

#. RT Latency Tuning
   Please apply these settings for Jetson AGX Xavier after RT kernel boot to get the best latency value.
   echo 100 > /sys/kernel/debug/tegra_mce/rt_window_us
   echo 20 > /sys/kernel/debug/tegra_mce/rt_fwd_progress_us
   echo 0x7f > /sys/kernel/debug/tegra_mce/rt_safe_mask


To remove the real-time kernel package
######################################

#. Enter this command::

      $ sudo apt remove nvidia-l4t-rt-kernel nvidia-l4t-rt-kernel-headers

#. Reboot.

Switching to a Different Kernel
###############################

Once you have installed the real-time kernel on a Jetson device, you may want to switch between the real-time kernel image and the original generic kernel image without re-installing or removing kernel packages.

You can do this by editing ``/boot/extlinux/extlinux.conf`` and setting the ``DEFAULT`` property to specify the kernel you want to boot. Set ``DEFAULT`` to ``real-time`` for the real-time kernel, or to ``primary`` for the generic
kernel::

   TIMEOUT 30
   DEFAULT real-time

.. todo::
   I gather that TIMEOUT is the setting before the one to be changed in the config file, but showing it implies that it has something to do with the kernel selection. If that's not true, we should not show it. Presumably there's only one DEFAULT setting in the file, so the reader won't need context to find it.

   I believe that during the wait for the timeout the user can boot a different kernel, perhaps by pressing any key to display a menu of available kernels. If that is so, then alternatively we could mention it and point out that TIMEOUT must be set to a non-zero value. It's an easier way to boot a different kernel if the user just wants to do so occasionally.