What can I help you with?
NVIDIA BlueField Platform Software Troubleshooting Guide
NVIDIA Docs Hub  NVIDIA Networking  BlueField DPUs / SuperNICs & DOCA  NVIDIA BlueField Platform Software Troubleshooting Guide  OpenSSL and PKA
Download PDF

On This Page

OpenSSL and PKA

Preface

OpenSSL library is used by applications to handle security communication like TLS/HTTPS along with certificate/key creation and management. BlueField enables high-performance computing by offloading requests to PKA hardware acceleration. The PKA library implements an OpenSSL engine to interact with the OpenSSL library and the PKA hardware.

The following are the layers of components from the application that interacts with the API/OpenSSL library down to the hardware: Application > OpenSSL Library > PKA Library > PKA driver > PKA hardware.

Command Cheat Sheet

Command

Description

lscpu

Linux command to display CPU information

lsmod

Linux command to list loaded kernel modules

pka_test_validation

BlueField test program to verify PKA library and hardware in multithreaded scenario

openssl

OpenSSL command to interact with OpenSSL library. Provides certificate/key creation, testing and verification

Logging and Counters

Logging is managed by applications using OpenSSL and PKA libraries, similar to Nginx or Apache web servers. You can find the log files in /var/log/nginx or /var/log/access_log.

OpenSSL offers APIs for tracing, which can be found at SSL_CTX_set_msg_callback. Applications should utilize these function calls to trace SSL/TLS protocols.

The PKA API functions communicate status and error information back to the OpenSSL library, which logs these messages if configured to do so. These messages are sent to stderr and depend on the application to redirect them to the application log.

Currently, counters are not exposed to users and are intended for development purposes only.

As an alternative to file logging, using packet capture tools like tcpdump can provide detailed information about the TLS handshake, including status and errors, which is often invaluable for debugging.

Debug Info Package

N/A

Scenarios

Common issues related to OpenSSL and PKA:

PKA Engine Loaded Incorrectly

Library Missing Error

The following indicates that the location for PKA engine shared library file is missing.

Copy
Copied!
            

            
$ openssl engine pka
20B095A0FFFF0000:error:12800067:DSO support routines:dlfcn_load:could not load the shared library:../crypto/dso/dso_dlfcn.c:118:filename(/usr/lib/aarch64-linux-gnu/engines-3/pka.so): /usr/lib/aarch64-linux-gnu/engines-3/pka.so: cannot open shared object file: No such file or directory


Version Mismatch Error

The following indicates that the libPKA.so.1 shared library cannot find a definition of function ASYNC_WAIT_CTX_get_fd().

Copy
Copied!
            

            
$ openssl speed -engine pka -async_jobs 8 rsa
Error configuring OpenSSL
281474842441520:error:25066067:DSO support routines:DLFCN_LOAD:could not load the shared library:dso_dlfcn.c:187:filename(/usr/lib64/openssl/engines/libpka.so): /lib64/libPKA.so.1: undefined symbol: ASYNC_WAIT_CTX_get_fd

This is part of the OpenSSL 3.0 async_jobs feature. OpenSSL's shared library should provide this function definition.

Insufficient PKA Resources

PKA hardware has limited PKA ring resources. Bluefield 2 has 32 rings, Bluefield 3 MB has 64 rings, and Bluefield 3 HB has 96. Each process invokes opensll and pka engine will take 4 rings(default) during the lifetime of the process. When pka resource is exhausted. The pka engine will not be loaded correctly.

Copy
Copied!
            

            
 $ openssl engine pka
(pka) PKA handle is invalid


Known OpenSSL Issue

The following message is benign and can be ignored:

Copy
Copied!
            

            
$ openssl engine pka
(pka) BlueField PKA engine support
2090E585FFFF0000:error:1280006A:DSO support routines:dlfcn_bind_func:could not bind to the requested symbol name:../crypto/dso/dso_dlfcn.c:188:symname(EVP_PKEY_base_id): /usr/lib/aarch64-linux-gnu/engines-3/pka.so: undefined symbol: EVP_PKEY_base_id
2090E585FFFF0000:error:1280006A:DSO support routines:DSO_bind_func:could not bind to the requested symbol name:../crypto/dso/dso_lib.c:176:

OpenSSL Utility Cannot Create Certificates

In rare cases, some crypto algorithms are not supported by the OpenSSL library due to compile time configuration. If you run into such an error, please report it to NVIDIA Enterprise Support.

Application Failure due to Error Returned by OpenSSL Function Calls

Example of this is the TLS/OpenSSL connection handshake. The Wireshark or tcpdump handshake will have the error code.

The following is an example response from the HTTPS/TLS server to TLS Client Hello request as captured by Wireshark:

Copy
Copied!
            

            
TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Handshake Failure)
    Content Type: Alert (21)
    Version: TLS 1.2 (0x0303)
    Length: 2
    Alert Message
        Level: Fatal (2)
        Description: Handshake Failure (40)

Possible TLS/HTTPS handshaking errors

  • Network related issue: interface provision, wiring, etc...

  • TLS errors: TLS versions between client and server, no matching cipher suites/supporting groups/signature algorithms/key management etc...

Please report this error with both information of both client and server-side information. Such as wget for HTTPS client, Apache webserver or google.com as HTTS server.

For all other issues, please follow the below steps to narrow down the errors and report it.

Diagnosing Issues with PKA Library

  1. Make sure CPU support AES algorithm:

    Copy
    Copied!
                
    
            
    $ lscpu
Architecture:           aarch64
  CPU op-mode(s):       32-bit, 64-bit
  Byte Order:           Little Endian
CPU(s):                 8
  On-line CPU(s) list:  0-7
Vendor ID:              ARM
  Model name:           Cortex-A78AE
    Model:              1
    Thread(s) per core: 1
    Core(s) per socket: 8
    Socket(s):          1
    Stepping:           r0p1
    BogoMIPS:           600.50
    Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp

  2. Check if PKA kernel module is loaded and devices(pka rings) are functioning.

    Copy
    Copied!
                
    
            
    $ lsmod | grep pka
mlxbf_pka             126976  0
 
$ ls /dev/pka/
0  10  12  14  16  18  2   21  23  25  27  29  30  32  34  36  38  4   41  43  45  47  49  50  52  54  56  58  6   61  63  8
1  11  13  15  17  19  20  22  24  26  28  3   31  33  35  37  39  40  42  44  46  48  5   51  53  55  57  59  60  62  7   9

  3. Run PKA validation test program:

    Copy
    Copied!
                
    
            
    $ /usr/bin/pka_test_validation -c 8 -r 4 -s 1
PKA system info
---------------
PKA API version: v1
Cache line size: 64
CPU count:       8
Ring count:      4
 
 
Running PKA inst: pka_test_validation
-----------------
Avail rings:      4
HW rings in use      :  000000000001111
 
Mode:            PKA_F_PROCESS_MODE_SINGLE(bit 1)
Sync:            PKA_F_SYNC_MODE_ENABLE(bit 8)
 
 
num worker threads: 8
Starting thread_idx=0 on cpu_num=0
Starting thread_idx=1 on cpu_num=1
Starting thread_idx=2 on cpu_num=2
Starting thread_idx=3 on cpu_num=3
Starting thread_idx=4 on cpu_num=4
Starting thread_idx=6 on cpu_num=6
Starting thread_idx=7 on cpu_num=7
Starting thread_idx=5 on cpu_num=5
thread_start_routine thread_idx=0 cpu_num=0 done
        tests_passed=60
        tests_failed=0
        total_tests=60 
thread_start_routine thread_idx=1 cpu_num=1 done
        tests_passed=60
        tests_failed=0
        total_tests=60 
thread_start_routine thread_idx=2 cpu_num=2 done
        tests_passed=60
        tests_failed=0
        total_tests=60 
thread_start_routine thread_idx=3 cpu_num=3 done
        tests_passed=6
        tests_failed=0
        total_tests=60 
thread_start_routine thread_idx=4 cpu_num=4 done
        tests_passed=60
        tests_failed=0
        total_tests=60 
thread_start_routine thread_idx=5 cpu_num=5 done
        tests_passed=60
        tests_failed=0
        total_tests=60 
thread_start_routine thread_idx=6 cpu_num=6 done
        tests_passed=60
        tests_failed=0
        total_tests=60 
thread_start_routine thread_idx=7 cpu_num=7 done
        tests_passed=60
        tests_failed=0
        total_tests=60 
validation tests passed!

Diagnosing Issues with OpenSSL

  1. Verify OpenSSl version

    Copy
    Copied!
                
    
            
    $ openssl version
OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)

  2. Verify if PKA is loaded by Openssl as default. News bfb would NOT have PKA loaded by default.

    Copy
    Copied!
                
    
            
    $ openssl engine
(dynamic) Dynamic engine loading support

  3. Check if OpenSSL can load PKA engine/library.

    Copy
    Copied!
                
    
            
    $ openssl engine pka
(pka) BlueField PKA engine support

    Info

    The printed errors are benign and can be ignored for now.

  4. Check if OpenSSL speed test can load PKA and offload requests to PKA engine in SYNC mode.

    Copy
    Copied!
                
    
            
    $ openssl speed -engine pka rsa
Engine "pka" set.
Doing 512 bits private rsa's for 10s: 131665 512 bits private RSA's in 9.97s
Doing 512 bits public rsa's for 10s: 556640 512 bits public RSA's in 10.00s
Doing 1024 bits private rsa's for 10s: 38463 1024 bits private RSA's in 9.99s
...
PKA_ENGINE: pka_modular_exp_crt failed, rc =-1504 Key length reaches PKA hardware limitation
RSA sign setup failure.  No RSA sign will be done.
2040969CFFFF0000:error:1C880004:Provider routines:rsa_sign:RSA lib:../providers/implementations/signature/rsa_sig.c:652:
RSA verify setup failure.  No RSA verify will be done.
version: 3.0.2
built on: Wed Jan 31 18:43:23 2024 UTC
options: bn(64,64)
compiler: gcc -fPIC -pthread -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/openssl-zqLXWk/openssl-3.0.2=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_TLS_SECURITY_LEVEL=2 -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
CPUINFO: OPENSSL_armcap=0xbf
                  sign    verify    sign/s verify/s
rsa  512 bits 0.000076s 0.000018s  13206.1  55664.0
rsa 1024 bits 0.000260s 0.000033s   3850.2  30368.3
rsa 2048 bits 0.001435s 0.000087s    696.9  11498.8
rsa 3072 bits 0.004575s 0.000173s    218.6   5764.5
rsa 4096 bits 0.010101s 0.000274s     99.0   3651.7

    Info

    PKA library currently supports only up to 4096 bits RSA.

  5. Check if OpenSSL speed test can load PKA and offload requests to PKA engine in ASYNC mode with async_jobs.

    Copy
    Copied!
                
    
            
    $ openssl speed -engine pka -async_jobs 8 dsa
Engine "pka" set.
Doing 512 bits sign dsa's for 10s: 152074 512 bits DSA signs in 8.05s
Doing 512 bits verify dsa's for 10s: 173404 512 bits DSA verify in 7.04s
Doing 1024 bits sign dsa's for 10s: 142377 1024 bits DSA signs in 8.04s
Doing 1024 bits verify dsa's for 10s: 148043 1024 bits DSA verify in 7.08s
Doing 2048 bits sign dsa's for 10s: 116583 2048 bits DSA signs in 7.44s
Doing 2048 bits verify dsa's for 10s: 68874 2048 bits DSA verify in 4.79s
version: 3.0.2
built on: Wed Jan 31 18:43:23 2024 UTC
options: bn(64,64)
compiler: gcc -fPIC -pthread -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/openssl-zqLXWk/openssl-3.0.2=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_TLS_SECURITY_LEVEL=2 -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
CPUINFO: OPENSSL_armcap=0xbf
                  sign    verify    sign/s verify/s
dsa  512 bits 0.000053s 0.000041s  18891.2  24631.2
dsa 1024 bits 0.000056s 0.000048s  17708.6  20910.0
dsa 2048 bits 0.000064s 0.000070s  15669.8  14378.7

    Note

    Due to an OpenSSL speed test issue with async_jobs and RSA, DSA algorithm is used here.

    Info

    PKA library currently support only up to 4096 bits RSA.
© Copyright 2024, NVIDIA. Last updated on Nov 12, 2024.
content here