Is this page helpful?

4.9. Error Log Management#

Error log management allows CUDA API errors to be reported in plain English, providing more information about the cause of an error than the API’s return value alone.

4.9.1. Background#

Traditionally, the only indication of a failed CUDA API call is a non-zero return code. Many CUDA error codes describe a category of failure without identifying which argument or requirement caused it. Error log management supplements those return codes with descriptive messages intended to help developers diagnose API usage errors.

4.9.2. Activation#

Set the CUDA_LOG_FILE environment variable to stdout, stderr, or a path to a writable file. CUDA writes each generated log message to the selected destination.

CUDA also stores recent messages in an internal buffer. Applications can use the error log management APIs to register a callback or dump this buffer even if CUDA_LOG_FILE was not set before the application started. An error-free execution might not generate any messages.

4.9.3. Output#

Messages written to CUDA_LOG_FILE or returned by the dump APIs use the following format:

[Time][TID][Source][Severity] Message

For example, calling cuLogsDumpToMemory with a null buffer can generate messages similar to the following:

[22:21:32.099][25642][CUDA][E] buffer cannot be NULL
[22:21:32.099][25642][CUDA][E] Returning 1 (CUDA_ERROR_INVALID_VALUE) from cuLogsDumpToMemory

Without error log management, the call returns CUDA_ERROR_INVALID_VALUE. Calling cuGetErrorString provides the generic description invalid argument, but not which argument was invalid.

4.9.4. API description#

Error log management is available through both the CUDA Driver and Runtime APIs. The Runtime interface operates on messages generated by both the Runtime and Driver APIs; both interfaces use the same underlying log buffer.

4.9.4.1. Callback registration#

The Driver API callback type and registration functions are:

typedef void (CUDA_CB *CUlogsCallback)(
    void *data, CUlogLevel logLevel, char *message, size_t length);

CUresult CUDAAPI cuLogsRegisterCallback(
    CUlogsCallback callbackFunc,
    void *userData,
    CUlogsCallbackHandle *callback_out);

CUresult CUDAAPI cuLogsUnregisterCallback(
    CUlogsCallbackHandle callback);

The corresponding Runtime API callback type and functions are:

typedef void (CUDART_CB *cudaLogsCallback_t)(
    void *data, cudaLogLevel logLevel, char *message, size_t length);

cudaError_t CUDARTAPI cudaLogsRegisterCallback(
    cudaLogsCallback_t callbackFunc,
    void *userData,
    cudaLogsCallbackHandle *callback_out);

cudaError_t CUDARTAPI cudaLogsUnregisterCallback(
    cudaLogsCallbackHandle callback);

The Driver callback reports errors as CU_LOG_LEVEL_ERROR and warnings as CU_LOG_LEVEL_WARNING. The corresponding Runtime values are cudaLogLevelError and cudaLogLevelWarning. Formatted output represents these levels as E and W, respectively.

CUDA passes userData to the callback without modification. The callback_out argument is optional, but an application must retain the returned handle if it intends to unregister the callback.

The callback receives the message text and its explicit byte length. The message does not include the timestamp, thread ID, source, or severity prefix added to formatted file and dump output. Applications should not rely on a particular callback thread or invocation timing unless the API reference explicitly guarantees that behavior.

4.9.4.2. Iterators and log dumps#

A log iterator identifies a position in the internal buffer. Calling cuLogsCurrent or cudaLogsCurrent sets an iterator to the current end of the buffer, so a later dump using that iterator returns only subsequently generated messages.

The Driver API functions are:

CUresult CUDAAPI cuLogsCurrent(
    CUlogIterator *iterator_out, unsigned int flags);

CUresult CUDAAPI cuLogsDumpToFile(
    CUlogIterator *iterator,
    const char *pathToFile,
    unsigned int flags);

CUresult CUDAAPI cuLogsDumpToMemory(
    CUlogIterator *iterator,
    char *buffer,
    size_t *size,
    unsigned int flags);

The corresponding Runtime API functions are:

cudaError_t CUDARTAPI cudaLogsCurrent(
    cudaLogIterator *iterator_out, unsigned int flags);

cudaError_t CUDARTAPI cudaLogsDumpToFile(
    cudaLogIterator *iterator,
    const char *pathToFile,
    unsigned int flags);

cudaError_t CUDARTAPI cudaLogsDumpToMemory(
    cudaLogIterator *iterator,
    char *buffer,
    size_t *size,
    unsigned int flags);

The flags argument to all current and dump functions is reserved for future use and must be zero.

The following Runtime API example marks the current end of the log and later dumps only messages generated after that point:

#include <cstdio>
#include <cuda_runtime_api.h>

cudaError_t printNewCudaLogs()
{
    cudaLogIterator iterator;
    cudaError_t status = cudaLogsCurrent(&iterator, 0);
    if (status != cudaSuccess) {
        return status;
    }

    int deviceCount;
    status = cudaGetDeviceCount(&deviceCount);
    cudaError_t deviceCountStatus = status;

    char buffer[25600];
    size_t size = sizeof(buffer);
    status = cudaLogsDumpToMemory(&iterator, buffer, &size, 0);
    if (status != cudaSuccess) {
        return status;
    }
    if (size != 0) {
        std::fwrite(buffer, 1, size, stderr);
    }
    return deviceCountStatus;
}

The intervening CUDA call still requires normal return-code handling. This example demonstrates how the iterator selects a window of log messages; it does not require deliberately making an invalid CUDA call. The fixed-size array uses the maximum documented dump capacity of 25,600 bytes.

4.9.4.3. Dump behavior#

If iterator is null, a dump requests all entries currently retained in the internal buffer. If an iterator is supplied, the dump begins at that position. After a successful dump, CUDA advances the supplied iterator to the current end of the log.

The internal buffer retains at most 100 entries. If requested messages have been overwritten and are no longer retained, CUDA adds a notice indicating that the output was truncated and returns the entries that remain available.

cuLogsDumpToMemory and cudaLogsDumpToMemory have these additional requirements:

  1. size initially specifies the capacity of buffer.

  2. CUDA null-terminates the buffer. Individual entries are separated by newline characters rather than null characters.

  3. A 25,600-byte output buffer is sufficient to hold all retained log data.

  4. On success, size contains the number of bytes written, excluding the terminating null character.

  5. If there are no messages to dump, the function succeeds and sets size to zero.

  6. If the buffer can hold some but not all requested messages, CUDA prepends a truncation notice, omits the oldest messages that do not fit, and retains the most recent message in the output.

  7. If the buffer cannot hold any messages, the function returns CUDA_ERROR_INVALID_VALUE or cudaErrorInvalidValue and sets size to zero.

4.9.5. Limitations and known issues#

  1. When the 100-entry limit is reached, CUDA replaces the oldest entries.

  2. Not all CUDA APIs generate descriptive messages yet. Coverage continues to expand.

  3. CUDA does not validate a CUDA_LOG_FILE destination until a message is generated.

  4. Log messages are not localized; CUDA provides them in US English.