> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# nemo_automodel.components.training.signal_handler

## Module Contents

### Classes

| Name                                                                                                      | Description                                                            |
| --------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| [`DistributedSignalHandler`](#nemo_automodel-components-training-signal_handler-DistributedSignalHandler) | Context manager to handle signals gracefully in a distributed setting. |

### Functions

| Name                                                                                    | Description                                                              |
| --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
| [`all_gather_item`](#nemo_automodel-components-training-signal_handler-all_gather_item) | Perform an all\_gather operation on a single Python object.              |
| [`get_device`](#nemo_automodel-components-training-signal_handler-get_device)           | Get the appropriate torch device based on the distributed backend.       |
| [`resolve_signal`](#nemo_automodel-components-training-signal_handler-resolve_signal)   | Resolve a user-provided signal specification to "signal.Signals" member. |

### Data

[`SignalLike`](#nemo_automodel-components-training-signal_handler-SignalLike)

### API

```python
class nemo_automodel.components.training.signal_handler.DistributedSignalHandler(
    sig: nemo_automodel.components.training.signal_handler.SignalLike | collections.abc.Sequence[nemo_automodel.components.training.signal_handler.SignalLike] = signal.SIGTERM,
    group: torch.distributed.ProcessGroup | None = None
)
```

Context manager to handle signals gracefully in a distributed setting.

Installs a signal handler upon entering the context that sets a flag
when the specified signal is received. The `signals_received` method
can be used to check if any rank received the signal (using all\_gather).
The original signal handler is restored upon exiting the context.

**Parameters:**

**`sig`** `SignalLike | Sequence[SignalLike]` — default: signal.SIGTERM

One or more signals to handle, each given as a signal number,
name (e.g. "SIGTERM"), or `signal.Signals` member. Accepts a
single value or a sequence. Defaults to signal.SIGTERM.

---

**`group`** `torch.distributed.ProcessGroup | None` — default: None

Process group whose ranks participate in signal propagation.
Defaults to the global process group.

---

**`original_handlers`** `= {}`

---

**`sig`** `Signals`

Backward-compatible accessor for the first configured signal.

---

```python
nemo_automodel.components.training.signal_handler.DistributedSignalHandler.__enter__() -> nemo_automodel.components.training.signal_handler.DistributedSignalHandler
```

Enters the signal-managed area.

**Returns:** `DistributedSignalHandler`

returns self.

```python
nemo_automodel.components.training.signal_handler.DistributedSignalHandler.__exit__(
    exc_type: type | None,
    exc_val: BaseException | None,
    exc_tb: types.TracebackType | None
) -> None
```

Release the signal handler and restore the original handler.

```python
nemo_automodel.components.training.signal_handler.DistributedSignalHandler.release() -> bool
```

Restore the original signal handler.

**Returns:** `bool`

True if the handler was released, False if it was already released.

```python
nemo_automodel.components.training.signal_handler.DistributedSignalHandler.signals_received() -> list[bool]
```

Check if any rank in the configured group received the signal.

Uses all\_gather to collect the signal status from all ranks.

**Returns:** `list[bool]`

A list of booleans, where each element indicates if the

```python
nemo_automodel.components.training.signal_handler.all_gather_item(
    item: typing.Any,
    dtype: torch.dtype,
    group: torch.distributed.ProcessGroup | None = None,
    async_op: bool = False,
    local_rank: int | None = None
) -> list[typing.Any]
```

Perform an all\_gather operation on a single Python object.

Converts the item to a tensor, performs all\_gather, and converts back to a list
of Python objects from all ranks.

**Parameters:**

**`item`** `Any`

The Python object to gather.

---

**`dtype`** `torch.dtype`

The torch dtype to use for the intermediate tensor.

---

**`group`** `Optional[torch.distributed.ProcessGroup]` — default: None

The process group to gather within
(defaults to the global group).

---

**`async_op`** `bool` — default: False

Whether the operation should be asynchronous.

---

**`local_rank`** `Optional[int]` — default: None

The local rank to determine the device.

---

**Returns:** `list[Any]`

list\[Any]: A list containing the gathered items (of type Any) from all ranks in the group.

```python
nemo_automodel.components.training.signal_handler.get_device(
    local_rank: int | None = None
) -> torch.device
```

Get the appropriate torch device based on the distributed backend.

**Parameters:**

**`local_rank`** `int | None` — default: None

The local rank, used to specify the CUDA device index for NCCL.
If None, uses the default CUDA device.

---

**Returns:** `torch.device`

The torch.device ('cuda' for NCCL, 'cpu' for Gloo).

**Raises:**

* `RuntimeError`: If the distributed backend is neither 'nccl' nor 'gloo'.

```python
nemo_automodel.components.training.signal_handler.resolve_signal(
    sig: nemo_automodel.components.training.signal_handler.SignalLike
) -> signal.Signals
```

Resolve a user-provided signal specification to "signal.Signals" member.

Accepts integers (e.g. "15"), "signal.Signals" members (e.g. "signal.SIGTERM")
and case-insensitive string names with or without the "SIG" prefix (e.g. "SIGTERM",
"sigusr1", "USR2"). String support allows the pre-emption signal to be configured from YAML.

**Parameters:**

**`sig`** `SignalLike`

The signal specification to resolve.

---

**Returns:** `signal.Signals`

The corresponding "signal.Signals" member.

**Raises:**

* `ValueError`: If the specification does not name a valid signal.
* `TypeError`: If sig is not an int, str, or "signal.Signals".

```python
nemo_automodel.components.training.signal_handler.SignalLike = int | str | signal.Signals
```