Nautobot Import

DPS Nautobot Integration - Current Use Cases and Future Considerations

Overview

This document describes the Nautobot import flow as it exists in this repo today, and separates that from ideas that are still future work.

Current implementation spans:

• Web UI import wizard
• CLI commands for generate/status/cancel/delete
• gRPC/HTTP API surfaces used by the UI and clients

It does not currently include saved default queries or import-specific RBAC.

Current API Surface

The current Nautobot import flow relies on these operations:

• GenerateEntities: starts an async fetch and returns a request_id
• GetGenerateEntitiesStatus: streams generated Entity messages for a request_id
• CancelGenerateEntitiesRequest: cancels a pending or running request
• DeleteGenerateEntitiesRequest: deletes a terminal request and its staged results
• UpsertEntities: imports selected entities into the main DPS entity store

Interface Scope

The interfaces do not all expose the same workflow:

• Web UI: fetch, review, select, optional edit, and import
• CLI: generate, status, cancel, and delete primitives
• API: the same low-level primitives used by clients

The guided review-and-import experience is currently a Web UI workflow, not a single dedicated first-class CLI or API workflow but can be accomplished in multiple manual steps.

Filter Semantics

Nautobot filter handling in the current implementation works as follows:

• Structured filters are locations, racks, device_types, and request_string
• Human-friendly values such as location names, rack names, and device type models may be resolved to Nautobot UUIDs before the device query runs
• Rack-name lookup is location-scoped when locations are available
• UUID values are passed through without lookup
• Some parameters in request_string are reserved for structured handling, while other parameters are passed through to the Nautobot devices endpoint

Persistence and Cleanup

Fetched entities are staged before final import:

• Async generate requests and fetched entities are stored in the generate_entities_* tables
• This staging data is separate from the main DPS entity store
• GetGenerateEntitiesStatus reads from staged request state and staged entities
• UpsertEntities writes selected entities into the main DPS entity store
• DeleteGenerateEntitiesRequest removes staged request data; it does not remove already imported entities

Capability 1: Nautobot-to-DPS Entity Fetch and Import

Use Case 1.1: Operator Queries Nautobot for Candidate Entities

Actor: Operator (via Web UI, CLI, or API)

Precondition: DPS is configured with Nautobot access.

Flow:

1. Operator submits Nautobot filters: locations, racks, device_types, and/or request_string.
2. DPS validates the request.
3. DPS creates an async GenerateEntities request and returns a request_id.
4. DPS resolves human-friendly filter values to Nautobot UUIDs where needed.
5. DPS queries Nautobot and stores generated entities in the generate_entities_* staging tables.
6. Client retrieves the generated entities by calling GetGenerateEntitiesStatus(request_id), which streams Entity messages until completion.
7. Web UI displays the fetched entities for review.

Postcondition: Candidate entities exist in the generate-entities staging tables and are available for review. They are not yet imported into the main DPS entity store.

Current validation rules:

• At least one filter is required
• Location-only is rejected unless the query string provides an allowed identifier such as rack=UUID
• Device-type-only is rejected
• Rack names require a location filter; rack UUIDs do not
• Query-string-only requests are allowed only for specific identifier-style filters

Error paths:

• Invalid filters -> validation error
• Nautobot unreachable or query failure -> request reaches terminal error state
• Missing request ID -> invalid argument
• Zero results -> successful request with no streamed entities

Concurrency note: The current implementation does not prevent multiple in-flight GenerateEntities requests per user or session. Each request gets its own request_id. The Web UI tracks a single lastRequestId and cleans up before re-fetch; CLI and API callers are responsible for managing their own request lifecycle.

Use Case 1.2: Operator Reviews and Selects Entities for Import

Actor: Operator (primarily via Web UI)

Precondition: Candidate entities were fetched via UC 1.1.

Flow:

1. Operator reviews the fetched entities.
2. Operator selects which entities to keep in the working set.
3. Operator optionally edits entities during the review step.
4. Operator confirms import.
5. UI uploads the selected entities through the existing UpsertEntities path.

Postcondition: Selected entities are upserted into DPS. They are no longer only staged import results; they are part of the DPS entity inventory and can be used to build a topology afterward.

Notes:

• The current repo does not have a dedicated ImportEntities(selectedEntityIDs) RPC.
• Selection and review are currently a Web UI workflow, not a first-class CLI workflow.
• “Optionally edits entities” refers to UI-level field adjustments before upsert; the specific editable fields and their validation are governed by the UpsertEntities schema, not this import flow.
• Re-importing the same entities is handled by UpsertEntities semantics (create-or-update); there is no special duplicate detection or merge logic in the import path.

Use Case 1.3: Operator Cancels or Clears a Pending Fetch

Actor: Operator (via Web UI, CLI, or API)

Precondition: A generate-entities request exists.

Flow:

1. Operator chooses to cancel a request.
2. DPS calls CancelGenerateEntitiesRequest(request_id).
3. If the request is pending or running, DPS cancels the stored context.
4. DPS marks the request status as cancelled.
5. If the operator also wants staged results removed, client calls DeleteGenerateEntitiesRequest(request_id) after the request is terminal.

Postcondition:

• After cancel: request is terminal and no more work should continue
• After delete: request row and staged entities are removed from the staging tables

Important distinction:

• CancelGenerateEntitiesRequest does not itself delete the staging row
• DeleteGenerateEntitiesRequest is a separate operation
• In the current Web UI, “Remove All” performs cancel first and then attempts delete

Use Case 1.4: Operator Re-fetches with New Filters

Actor: Operator

Precondition: A previous request may still exist.

Flow in the current Web UI:

1. Operator changes filters and clicks Fetch again.
2. UI attempts to cancel the previous request.
3. UI then attempts to delete the previous request and its staged entities.
4. UI clears the local working set.
5. UI starts a new GenerateEntities request.

Postcondition: The UI presents only the newly fetched candidate entities.

Scope note: Automatic cleanup on re-fetch is currently a Web UI behavior. CLI and direct API clients manage request lifecycle explicitly.

Capability 2 (Future): Automatic Import and Continuous Sync

Use Case 2.1: Operator Enables Auto-Import with Continuous Sync

Actor: Operator (via Web UI, CLI, or API)

Precondition: DPS is configured with Nautobot and Nautobot NATS access.

Flow: 0. Datacenter admin configured sync configurations at DPS deploy time through helm values and in DPS app settings at runtime

1. Operator submits Nautobot filters (locations, racks, device_types, and/or request_string).
2. DPS validates the request.
3. DPS saves the operator’s query filters to the database as a persistent sync configuration.
4. DPS queries Nautobot with the provided filters and automatically imports all matching entities into the DPS entity store (no staging/review step).
5. DPS subscribes to NATS for update and delete events for the imported entities.
6. DPS receives events on NATS event bus and re-queries Nautobot using the saved filters, including new devices.
7. On each periodic sync - period defined by datacenter admin in DPS app settings:
   • New devices matching the filters are automatically imported.
   • Updated devices are upserted in the entity store.
   • Deleted devices are handled according to the event.
8. DB table NautobotSyncLog rows are inserted to catalog what entities changed on sync for specific queries
9. If DPS cannot match a returned device to a known DPS entity, DPS publishes a notification on the message bus and/or includes it in DPS status publishing to alert that manual resolution is needed.

Postcondition: Entities matching the saved filters are kept in sync with Nautobot continuously. The operator is alerted to any unresolvable discrepancies.

Difference from Capability 1: Capability 1 is an operator-driven fetch-review-import workflow with explicit staging. Capability 2 skips the review/staging step and adds ongoing sync via NATS events and periodic polling.

Error paths:

• Invalid filters -> validation error
• Nautobot unreachable during initial import -> request reaches terminal error state
• Nautobot unreachable during periodic sync -> sync attempt fails; retried on next cron cycle
• NATS subscription failure -> logged and retried on next cron cycle; periodic polling serves as fallback
• Unresolvable entity mismatch -> notification published for manual resolution

Architecture and Sequences

UC 1.1 - Query Nautobot for Candidate Entities

                 Operator
                    |
        +-----------+-----------+
        |           |           |
      Web UI       CLI         API
        |           |           |
        +-----------+-----------+
                    |
                    v
          +------------------+
          |   DPS Gateway    |
          |  (gRPC / REST)   |
          +------------------+
                    |
          1. Validate filters
          2. GenerateEntities RPC
                    |
                    v
          +------------------+
          | Topology Manager |
          +------------------+
                    |
          3. Resolve names -> UUIDs
          4. Query Nautobot API
                    |
                    v
          +------------------+
          |  Nautobot Client |
          +------------------+
                    |
                    v
          +------------------+
          | Nautobot Instance|
          |   (REST API)     |
          +------------------+
                    |
          5. Store staged entities
                    |
                    v
          +------------------+
          |   Staging DB     |
          | (generate_entities_*) |
          +------------------+
                    |
          6. GetGenerateEntitiesStatus
          7. Stream Entity messages

Sequence:

sequenceDiagram
    participant Op as Operator
    participant UI as DPS UI, DPS Client
    participant GW as gRPC API
    participant TM as TopologyManager
    participant NC as NautobotClient
    participant NB as Nautobot
    participant DB as DB

    Op->>UI: Specify filters
    UI->>GW: GenerateEntities(filters)
    GW->>TM: Validate filters
    alt Invalid filters
        TM-->>UI: Validation error
    end
    TM->>NC: ListDevices(filters)
    NC->>NC: resolve names to UUIDs
    NC->>NB: Nautobot REST queries
    NB-->>NC: Device list
    NC-->>TM: []Device
    TM->>DB: Write request + candidate entities
    TM-->>UI: request_id
    UI->>GW: GetGenerateEntitiesStatus(request_id)
    GW-->>UI: stream Entity messages
    UI-->>Op: Display candidate entities

UC 1.2 - Review and Import Selected Entities

sequenceDiagram
    participant Op as Operator
    participant UI as Web UI
    participant GW as gRPC API
    participant TM as TopologyManager
    participant ES as Entity Store

    Op->>UI: Review fetched entities
    Op->>UI: Select subset and confirm import
    UI->>GW: UpsertEntities(stream Entity)
    GW->>TM: Validate and upsert entities
    TM->>ES: Upsert selected entities
    TM-->>UI: Success
    UI-->>Op: Import confirmed

Expanded sequence:

sequenceDiagram
    participant Op as Operator
    participant UI as Web UI
    participant GW as gRPC API
    participant TM as TopologyManager
    participant ES as Entity Store

    Op->>UI: Confirm import of selected entities
    UI->>GW: UpsertEntities(stream Entity)
    loop For each selected entity
        GW->>TM: Receive entity
        TM->>ES: Create or update entity
    end
    TM-->>UI: Empty success response
    UI-->>Op: Successfully imported N entities

UC 1.3 - Cancel and Optional Delete

sequenceDiagram
    participant UI as Interface
    participant GW as DPS Gateway
    participant TM as TopologyManager
    participant DB as StagingDB

    UI->>GW: CancelGenerateEntitiesRequest(request_id)
    GW->>TM: Lookup request
    alt Pending or running
        TM->>TM: Cancel stored context
    end
    TM->>DB: Set status = "cancelled"
    TM-->>UI: OK

    opt Remove staged results too
        UI->>GW: DeleteGenerateEntitiesRequest(request_id)
        GW->>DB: DELETE request + cascaded entities
        GW-->>UI: OK
    end

UC 1.4 - Re-fetch with New Filters

sequenceDiagram
    participant Op as Operator
    participant UI as Web UI
    participant GW as gRPC API

    Op->>UI: Adjust filters, click Fetch
    UI->>GW: CancelGenerateEntitiesRequest(previousRequestId)
    UI->>GW: DeleteGenerateEntitiesRequest(previousRequestId)
    UI->>GW: GenerateEntities(newFilters)
    UI->>GW: GetGenerateEntitiesStatus(newRequestId)

UC 2.1 - Auto-Import with Continuous Sync

sequenceDiagram
    participant Op as Operator
    participant UI as Interface
    participant GW as DPS Gateway
    participant TM as TopologyManager
    participant NC as NautobotClient
    participant NB as Nautobot
    participant ES as Entity Store
    participant DB as SyncConfigDB
    participant NATS as NATS

    Op->>UI: Submit filters with auto-import
    UI->>GW: AutoImportEntities(filters)
    GW->>TM: Validate filters
    alt Invalid filters
        TM-->>UI: Validation error
    end
    TM->>DB: Save sync configuration + filters
    TM->>NC: ListDevices(filters)
    NC->>NB: Nautobot REST queries
    NB-->>NC: Device list
    NC-->>TM: []Device
    TM->>ES: Upsert all matching entities
    TM->>NATS: Subscribe to update/delete events
    TM-->>UI: Sync configuration created
    UI-->>Op: Auto-import complete, sync active

    loop Periodic cron sync
        TM->>DB: Load saved filters
        TM->>NC: ListDevices(saved filters)
        NC->>NB: Nautobot REST queries
        NB-->>NC: Device list
        NC-->>TM: []Device
        alt New or updated devices
            TM->>ES: Upsert entities
        end
        alt Unresolvable mismatch
            TM->>NATS: Publish alert notification
        end
    end

    NATS-->>TM: Entity update/delete event
    alt Update event
        TM->>ES: Upsert updated entity
    end
    alt Delete event
        TM->>ES: Handle entity removal
    end

Request State Diagram

The async generate-entities request lifecycle is:

stateDiagram-v2
    [*] --> pending
    pending --> running
    running --> complete
    running --> error
    running --> cancelled

Notes:

• GenerateEntities creates the request in pending
• Worker execution moves it to running
• Successful generation moves it to complete
• Backend failures move it to error
• CancelGenerateEntitiesRequest moves a request to cancelled
• DeleteGenerateEntitiesRequest removes terminal requests from storage; it is not itself a request state
• In the current implementation, delete is rejected for pending or running requests

Future Considerations

Saved Default Queries

Ability to configure default query filters at a DPS server configuration level allowing users to only need to click Fetch. This would be provided in helm values by datacenter admin that deploys DPS.

Import-Specific Access Controls

Ability to scope entity generation from Nautobot to a specific access control for a user scope:generate