Coupled Manipulation#

A Franka gripper holding a deformable cable in the Newton viewer

The previous lesson used Newton IK to move a Franka end effector through target poses. Now we’ll use that same IK pattern for a task that needs more than one solver: picking up a deformable cable and moving it to a target location. This is where Newton’s coupled-solver framework shines, letting each subsystem use the solver that fits it best.

This lesson is adapted from the standalone Newton example newton/examples/multiphysics/example_franka_cable_ik_pick_place.py. We split the same idea into digestible pieces.

Tip

Short on time? Jump to the final example for the complete runnable script.

Note

This lesson downloads the Franka asset on first use. The newton[examples] extra in Getting Started installs the required GitPython dependency automatically.

In this lesson, we will:

  • Explain why a rigid arm and a deformable cable need different solvers.

  • Build one model that contains both subsystems and track their body, joint, and shape indices.

  • Configure a coupled solver with proxy coupling between the gripper and the cable.

  • Run a scripted IK pick-and-place sequence on the coupled scene.

Why Coupling?#

A rigid Franka arm and a deformable cable have different numerical needs. Newton’s coupled-solver framework lets each subsystem use a solver that fits it:

  • SolverMuJoCo handles the Franka articulation and its joint targets.

  • SolverVBD handles the cable, represented as a rod.

  • SolverCoupledProxy exposes selected gripper bodies as proxy bodies to the VBD side, so the cable can contact the gripper while the arm stays solved by MuJoCo.

The high-level control loop stays familiar: IK produces Franka joint targets, the gripper target changes through the task sequence, collision contacts are refreshed, and the coupled solver advances the full model.

Task Constants and Warp Kernels#

We start with the constants that describe the cable, the gripper poses, and the grip forces, taken from the standalone example.

from newton.solvers import SolverMuJoCo, SolverVBD
from newton.solvers.experimental.coupled import SolverCoupled, SolverCoupledProxy

FRANKA_Q = [
    -3.6802115e-03, 2.3901723e-02, 3.6804110e-03, -2.3683236e00,
    -1.2918962e-04, 2.3922248e00, 7.8549200e-01, 0.04, 0.04,
]

CABLE_CENTER = wp.vec3(0.5, 0.0, 0.256)
CABLE_LENGTH = 0.38
CABLE_CONTACT_KE = 1.0e4
CABLE_CONTACT_KD = 1.0e-5 * CABLE_CONTACT_KE

GRIPPER_DOWN = (1.0, 0.0, 0.0, 0.0)  # qx, qy, qz, qw: hand points straight down
GRIP_OPEN = 0.04
GRIP_CLOSE = 0.0
GRIP_HOLD = 0.0
GRIP_FORCE = 1500.0
GRIP_STIFFNESS = 1000.0

world_count = 1
payload_segments = 19
payload_radius = 0.005
surface_z = float(CABLE_CENTER[2]) - payload_radius

Two small Warp kernels write the gripper finger width and the per-frame IK task targets on the device, avoiding host-device round trips.

@wp.kernel
def set_gripper_q(joint_q: wp.array2d[float], finger_pos: wp.array[float], idx0: int, idx1: int):
    world_idx = wp.tid()
    joint_q[world_idx, idx0] = finger_pos[world_idx]
    joint_q[world_idx, idx1] = finger_pos[world_idx]


@wp.kernel
def set_task_targets(
    target_positions: wp.array[wp.vec3],
    target_rotations: wp.array[wp.vec4],
    finger_pos: wp.array[float],
    pos: wp.vec3,
    rot: wp.vec4,
    grip_width: float,
):
    world_idx = wp.tid()
    target_positions[world_idx] = pos
    target_rotations[world_idx] = rot
    finger_pos[world_idx] = grip_width

Build the Coupled Scene#

We build one model that contains both subsystems. As we add the Franka and then the cable, we record the ranges of body, joint, and shape indices each subsystem owns, so the coupled solver can hand each entry only the part of the model it’s responsible for.

def build_coupled_scene():
    builder = newton.ModelBuilder(gravity=-9.81)
    builder.rigid_gap = 0.01
    SolverMuJoCo.register_custom_attributes(builder)
    SolverVBD.register_custom_attributes(builder, dahl_defaults_enabled=False)

    franka_body_start = builder.body_count
    franka_joint_start = builder.joint_count
    franka_shape_start = builder.shape_count

    add_franka(builder, surface_z)
    # ... set arm/gripper gains, effort limits, and armature ...

    franka_bodies = list(range(franka_body_start, builder.body_count))
    # ... record franka_joints, franka_shapes, then add the VBD cable rod ...
    # ... record payload_bodies/joints/shapes and locate the gripper bodies ...

    builder.color()
    model = builder.finalize()
    return {"model": model, "franka_bodies": franka_bodies, ...}

The cable itself is a rod built from a straight line of points, with stretch and bend stiffness that make it behave like a springy cable rather than a rigid bar. The gripper bodies are found by label so the proxy coupling knows which bodies to share with the VBD side. The complete script below shows every line.

Preview of the coupled scene before the task runs: the MuJoCo-solved Franka arm above the VBD-solved cable, sharing one model.

Create the Coupled Solvers#

The coupled solver receives two model views. The MuJoCo view owns the Franka bodies and joints; the VBD view owns the cable. The proxy coupling sends the gripper bodies from the MuJoCo side into the VBD collision world so the cable feels the gripper.

solver = SolverCoupledProxy(
    model=model,
    entries=[
        SolverCoupled.Entry(
            name="mjc",
            solver=lambda view: SolverMuJoCo(model=view, solver="newton", integrator="implicitfast", ...),
            bodies=scene_data["franka_bodies"],
            joints=scene_data["franka_joints"],
        ),
        SolverCoupled.Entry(
            name="vbd",
            solver=lambda view: SolverVBD(model=view, iterations=20, ...),
            bodies=scene_data["payload_bodies"],
            joints=scene_data["payload_joints"],
        ),
    ],
    coupling=SolverCoupledProxy.Config(
        proxies=[
            SolverCoupledProxy.Proxy(
                source="mjc",
                destination="vbd",
                bodies=scene_data["gripper_bodies"],
                mode="lagged",
                ...
            )
        ],
        iterations=1,
    ),
)
solver.prepare_contacts(contacts)

Build the Franka IK System and Keyframes#

IK runs on a Franka-only model. Because the Franka is added to the coupled model first, its joint coordinates line up with the first coordinates of the full model, so we can copy IK results straight into the coupled model’s joint targets.

The task is described as a list of keyframes: approach the cable, descend, close the gripper, lift, move to the place location, descend, release, and retreat. Each keyframe carries a duration, a target position, an orientation, and a grip width.

def build_keyframes():
    cx, cy, cz = CABLE_CENTER[0], CABLE_CENTER[1], CABLE_CENTER[2]
    approach_z = cz + 0.20
    grasp_z = cz
    target_x, target_y = 0.4, 0.25
    qx, qy, qz, qw = GRIPPER_DOWN
    poses = np.array(
        [
            [1.0, cx, cy, approach_z, qx, qy, qz, qw, GRIP_OPEN],
            [0.5, cx, cy, grasp_z, qx, qy, qz, qw, GRIP_OPEN],
            [1.0, cx, cy, grasp_z, qx, qy, qz, qw, GRIP_CLOSE],
            [1.0, cx, cy, approach_z, qx, qy, qz, qw, GRIP_HOLD],
            [1.0, target_x, target_y, approach_z, qx, qy, qz, qw, GRIP_HOLD],
            [0.5, target_x, target_y, grasp_z, qx, qy, qz, qw, GRIP_HOLD],
            [0.5, target_x, target_y, grasp_z, qx, qy, qz, qw, GRIP_HOLD],
            [1.0, target_x, target_y, grasp_z, qx, qy, qz, qw, GRIP_OPEN],
            [0.5, target_x, target_y, approach_z, qx, qy, qz, qw, GRIP_OPEN],
        ],
        dtype=np.float32,
    )
    return poses[:, 1:], np.cumsum(poses[:, 0]), (target_x, target_y)

Run the Pick-and-Place#

Each rendered frame interpolates the next task-space target, solves IK, copies the solved Franka coordinates into the coupled model’s joint targets, refreshes contacts, and advances the coupled solver.

def simulate_coupled_frame():
    global state_0, state_1
    ik_solver.step(ik_joint_q, ik_joint_q, iterations=ik_iters)
    wp.launch(set_gripper_q, dim=world_count, inputs=[ik_joint_q, finger_pos_buf, finger_idx0, finger_idx1], device=device)
    wp.copy(dest=control_joint_target_q[:, :n_coords], src=ik_joint_q)

    for _ in range(sim_substeps):
        state_0.clear_forces()
        collision_pipeline.collide(state_0, contacts)
        solver.step(state_0, state_1, control, contacts, sim_dt)
        newton.eval_ik(model, state_1, state_1.joint_q, state_1.joint_qd)
        state_0, state_1 = state_1, state_0

The full coupled pick-and-place: IK drives the arm through the keyframes while the coupled solver lets the gripper grasp and carry the deformable cable.

Complete Script#

This is the full coupled pick-and-place task in one file. Before running it, open http://localhost:8080 in a browser tab and reload the tab when Viser starts. Download lesson4_coupled_manipulation.py, or save the script below as lesson4_coupled_manipulation.py and run it with the pinned Newton 1.5 command from Getting Started.

Show the complete runnable script
  1"""Newton Fundamentals - Lesson 4: Coupled Franka cable pick-and-place.
  2
  3Combines a MuJoCo-solved Franka, a VBD-solved cable, and proxy coupling.
  4Before running, open http://localhost:8080 in a browser tab and reload it when
  5the Viser server starts.
  6"""
  7
  8import time
  9
 10import numpy as np
 11import warp as wp
 12
 13import newton
 14import newton.utils
 15import newton.ik as ik
 16from newton.solvers import SolverMuJoCo, SolverVBD
 17from newton.solvers.experimental.coupled import SolverCoupled, SolverCoupledProxy
 18
 19wp.config.quiet = True
 20SolverMuJoCo.import_mujoco()
 21
 22
 23def make_viewer(name: str):
 24    """Open a Viser web viewer; it prints a URL (normally http://localhost:8080)."""
 25    return newton.viewer.ViewerViser(verbose=False)
 26
 27
 28# --- Task constants ---
 29FRANKA_Q = [
 30    -3.6802115e-03, 2.3901723e-02, 3.6804110e-03, -2.3683236e00,
 31    -1.2918962e-04, 2.3922248e00, 7.8549200e-01, 0.04, 0.04,
 32]
 33CABLE_CENTER = wp.vec3(0.5, 0.0, 0.256)
 34CABLE_LENGTH = 0.38
 35CABLE_CONTACT_KE = 1.0e4
 36CABLE_CONTACT_KD = 1.0e-5 * CABLE_CONTACT_KE
 37GRIPPER_DOWN = (1.0, 0.0, 0.0, 0.0)
 38GRIP_OPEN = 0.04
 39GRIP_CLOSE = 0.0
 40GRIP_HOLD = 0.0
 41GRIP_FORCE = 1500.0
 42GRIP_STIFFNESS = 1000.0
 43
 44world_count = 1
 45payload_segments = 19
 46payload_radius = 0.005
 47surface_z = float(CABLE_CENTER[2]) - payload_radius
 48
 49
 50# --- Warp kernels and helpers ---
 51@wp.kernel
 52def set_gripper_q(joint_q: wp.array2d[float], finger_pos: wp.array[float], idx0: int, idx1: int):
 53    world_idx = wp.tid()
 54    joint_q[world_idx, idx0] = finger_pos[world_idx]
 55    joint_q[world_idx, idx1] = finger_pos[world_idx]
 56
 57
 58@wp.kernel
 59def set_task_targets(
 60    target_positions: wp.array[wp.vec3],
 61    target_rotations: wp.array[wp.vec4],
 62    finger_pos: wp.array[float],
 63    pos: wp.vec3,
 64    rot: wp.vec4,
 65    grip_width: float,
 66):
 67    world_idx = wp.tid()
 68    target_positions[world_idx] = pos
 69    target_rotations[world_idx] = rot
 70    finger_pos[world_idx] = grip_width
 71
 72
 73def find_label_index(labels, suffix):
 74    for index, label in enumerate(labels):
 75        if label.endswith(suffix):
 76            return index
 77    raise ValueError(f"Could not find label ending in {suffix!r}")
 78
 79
 80def add_franka(builder, base_z):
 81    builder.add_urdf(
 82        newton.utils.download_asset("franka_emika_panda") / "urdf/fr3_franka_hand.urdf",
 83        xform=wp.transform(wp.vec3(0.0, 0.0, base_z), wp.quat_identity()),
 84        floating=False,
 85        enable_self_collisions=False,
 86        parse_visuals_as_colliders=False,
 87        force_show_colliders=False,
 88    )
 89    builder.joint_q[: len(FRANKA_Q)] = FRANKA_Q
 90    builder.joint_target_q[: len(FRANKA_Q)] = FRANKA_Q
 91
 92
 93def build_coupled_scene():
 94    builder = newton.ModelBuilder(gravity=-9.81)
 95    builder.rigid_gap = 0.01
 96    SolverMuJoCo.register_custom_attributes(builder)
 97    SolverVBD.register_custom_attributes(builder, dahl_defaults_enabled=False)
 98
 99    franka_body_start = builder.body_count
100    franka_joint_start = builder.joint_count
101    franka_shape_start = builder.shape_count
102
103    add_franka(builder, surface_z)
104
105    builder.joint_target_ke[:7] = [400.0] * 7
106    builder.joint_target_kd[:7] = [80.0] * 7
107    builder.joint_target_ke[7:9] = [GRIP_STIFFNESS, GRIP_STIFFNESS]
108    builder.joint_target_kd[7:9] = [100.0, 100.0]
109    builder.joint_effort_limit[:4] = [87.0] * 4
110    builder.joint_effort_limit[4:7] = [12.0] * 3
111    builder.joint_effort_limit[7:9] = [GRIP_FORCE, GRIP_FORCE]
112    builder.joint_armature[:7] = [1.0e-3] * 7
113    builder.joint_armature[7:9] = [0.0, 0.0]
114
115    franka_bodies = list(range(franka_body_start, builder.body_count))
116    franka_joints = list(range(franka_joint_start, builder.joint_count))
117    franka_shapes = list(range(franka_shape_start, builder.shape_count))
118
119    gravcomp = builder.custom_attributes["mujoco:gravcomp"]
120    if gravcomp.values is None:
121        gravcomp.values = {}
122    for body in franka_bodies:
123        gravcomp.values[body] = 1.0
124
125    payload_body_start = builder.body_count
126    payload_joint_start = builder.joint_count
127    payload_shape_start = builder.shape_count
128
129    cable_cfg = newton.ModelBuilder.ShapeConfig(
130        density=100.0, ke=CABLE_CONTACT_KE, kd=CABLE_CONTACT_KD, mu=1.0, margin=0.0, gap=0.01,
131    )
132    points, quats = newton.utils.create_straight_cable_points_and_quaternions(
133        start=CABLE_CENTER - wp.vec3(0.5 * CABLE_LENGTH, 0.0, 0.0),
134        direction=wp.vec3(1.0, 0.0, 0.0),
135        length=CABLE_LENGTH,
136        num_segments=payload_segments,
137        twist_total=0.0,
138    )
139    bend_stiffness = 5.0e-4
140    builder.add_rod(
141        positions=points,
142        quaternions=quats,
143        radius=payload_radius,
144        body_frame_origin="start",
145        cfg=cable_cfg,
146        stretch_stiffness=1.0e6,
147        stretch_damping=1.0e-1,
148        bend_stiffness=bend_stiffness,
149        bend_damping=2.0e-3 * bend_stiffness,
150        label="vbd_cable",
151    )
152
153    payload_bodies = list(range(payload_body_start, builder.body_count))
154    payload_joints = list(range(payload_joint_start, builder.joint_count))
155    payload_shapes = list(range(payload_shape_start, builder.shape_count))
156
157    gripper_bodies = [
158        body for body in franka_bodies
159        if "hand" in builder.body_label[body] or "finger" in builder.body_label[body]
160    ]
161    if not gripper_bodies:
162        raise RuntimeError("Could not locate Franka gripper bodies for proxy coupling")
163
164    plane_cfg = newton.ModelBuilder.ShapeConfig(
165        ke=CABLE_CONTACT_KE, kd=CABLE_CONTACT_KD, mu=1.0, margin=0.0, gap=0.01,
166    )
167    ground_shapes = [builder.add_ground_plane(height=surface_z, cfg=plane_cfg, label="cable_ground_plane")]
168
169    builder.color()
170    model = builder.finalize()
171    model.shape_material_ke.fill_(CABLE_CONTACT_KE)
172    model.shape_material_kd.fill_(CABLE_CONTACT_KD)
173    model.shape_material_mu.fill_(1.0)
174
175    return {
176        "model": model,
177        "franka_bodies": franka_bodies,
178        "franka_joints": franka_joints,
179        "franka_shapes": franka_shapes,
180        "payload_bodies": payload_bodies,
181        "payload_joints": payload_joints,
182        "payload_shapes": payload_shapes,
183        "gripper_bodies": gripper_bodies,
184        "ground_shapes": ground_shapes,
185    }
186
187
188def ground_shape_pairs(model, franka_shapes, payload_shapes, ground_shapes):
189    dynamic_shapes = set(franka_shapes) | set(payload_shapes)
190    ground_shapes = set(ground_shapes)
191    pairs = [
192        (int(a), int(b))
193        for a, b in model.shape_contact_pairs.numpy()
194        if ({int(a), int(b)} & dynamic_shapes) and ({int(a), int(b)} & ground_shapes)
195    ]
196    if not pairs:
197        raise RuntimeError("No robot- or cable-ground contact pairs were generated")
198    return wp.array(np.asarray(pairs, dtype=np.int32), dtype=wp.vec2i, device=model.device)
199
200
201def build_keyframes():
202    cx, cy, cz = CABLE_CENTER[0], CABLE_CENTER[1], CABLE_CENTER[2]
203    approach_z = cz + 0.20
204    grasp_z = cz
205    target_x, target_y = 0.4, 0.25
206    qx, qy, qz, qw = GRIPPER_DOWN
207    poses = np.array(
208        [
209            [1.0, cx, cy, approach_z, qx, qy, qz, qw, GRIP_OPEN],
210            [0.5, cx, cy, grasp_z, qx, qy, qz, qw, GRIP_OPEN],
211            [1.0, cx, cy, grasp_z, qx, qy, qz, qw, GRIP_CLOSE],
212            [1.0, cx, cy, approach_z, qx, qy, qz, qw, GRIP_HOLD],
213            [1.0, target_x, target_y, approach_z, qx, qy, qz, qw, GRIP_HOLD],
214            [0.5, target_x, target_y, grasp_z, qx, qy, qz, qw, GRIP_HOLD],
215            [0.5, target_x, target_y, grasp_z, qx, qy, qz, qw, GRIP_HOLD],
216            [1.0, target_x, target_y, grasp_z, qx, qy, qz, qw, GRIP_OPEN],
217            [0.5, target_x, target_y, approach_z, qx, qy, qz, qw, GRIP_OPEN],
218        ],
219        dtype=np.float32,
220    )
221    return poses[:, 1:], np.cumsum(poses[:, 0]), (target_x, target_y)
222
223
224# --- Build the scene and coupled solver ---
225scene_data = build_coupled_scene()
226model = scene_data["model"]
227device = model.device
228print(f"Model device: {device}")
229print(f"Bodies: {model.body_count}, Joints: {model.joint_count}, Shapes: {model.shape_count}")
230
231control = model.control()
232state_0 = model.state()
233state_1 = model.state()
234newton.eval_fk(model, model.joint_q, model.joint_qd, state_0)
235newton.eval_fk(model, model.joint_q, model.joint_qd, state_1)
236
237collision_pipeline = newton.CollisionPipeline(
238    model,
239    broad_phase="explicit",
240    shape_pairs_filtered=ground_shape_pairs(
241        model, scene_data["franka_shapes"], scene_data["payload_shapes"], scene_data["ground_shapes"]
242    ),
243)
244contacts = collision_pipeline.contacts()
245
246solver = SolverCoupledProxy(
247    model=model,
248    entries=[
249        SolverCoupled.Entry(
250            name="mjc",
251            solver=lambda view: SolverMuJoCo(
252                model=view, solver="newton", integrator="implicitfast", cone="elliptic",
253                iterations=100, ls_iterations=20, use_mujoco_contacts=False, njmax=256, nconmax=64,
254            ),
255            bodies=scene_data["franka_bodies"],
256            joints=scene_data["franka_joints"],
257        ),
258        SolverCoupled.Entry(
259            name="vbd",
260            solver=lambda view: SolverVBD(
261                model=view, iterations=20, rigid_avbd_beta=1.0e2,
262                rigid_contact_k_start=1.0e3, rigid_contact_history=False,
263            ),
264            bodies=scene_data["payload_bodies"],
265            joints=scene_data["payload_joints"],
266        ),
267    ],
268    coupling=SolverCoupledProxy.Config(
269        proxies=[
270            SolverCoupledProxy.Proxy(
271                source="mjc",
272                destination="vbd",
273                bodies=scene_data["gripper_bodies"],
274                mass_scale=1.0,
275                mode="lagged",
276                collision_pipeline=lambda proxy_model: newton.CollisionPipeline(proxy_model, broad_phase="explicit"),
277                collide_interval=1,
278            )
279        ],
280        iterations=1,
281    ),
282)
283solver.prepare_contacts(contacts)
284print("Coupled solver ready")
285
286# --- Build the IK system ---
287targets, key_times, place_target_xy = build_keyframes()
288
289ik_builder = newton.ModelBuilder(gravity=-9.81)
290add_franka(ik_builder, surface_z)
291ik_model = ik_builder.finalize(device=device)
292
293n_coords = ik_model.joint_coord_count
294ik_joint_q = wp.clone(model.joint_q.reshape((world_count, -1))[:, :n_coords])
295control_joint_target_q = control.joint_target_q.reshape((world_count, -1))
296finger_idx0 = n_coords - 2
297finger_idx1 = n_coords - 1
298finger_pos_buf = wp.full(world_count, GRIP_OPEN, dtype=float, device=device)
299hand_body = find_label_index(ik_model.body_label, "fr3_hand")
300
301target_pos = wp.vec3(*targets[0][:3].tolist())
302target_rot = wp.vec4(*targets[0][3:7].tolist())
303ik_target_positions = wp.array([target_pos] * world_count, dtype=wp.vec3, device=device)
304ik_target_rotations = wp.array([target_rot] * world_count, dtype=wp.vec4, device=device)
305
306pos_obj = ik.IKObjectivePosition(
307    link_index=hand_body, link_offset=wp.vec3(0.0, 0.0, 0.107), target_positions=ik_target_positions,
308)
309rot_obj = ik.IKObjectiveRotation(
310    link_index=hand_body, link_offset_rotation=wp.quat_identity(), target_rotations=ik_target_rotations,
311)
312joint_limit_lower = wp.clone(model.joint_limit_lower.reshape((world_count, -1))[:, :n_coords])
313joint_limit_upper = wp.clone(model.joint_limit_upper.reshape((world_count, -1))[:, :n_coords])
314joint_limits_obj = ik.IKObjectiveJointLimit(
315    joint_limit_lower=joint_limit_lower.flatten(), joint_limit_upper=joint_limit_upper.flatten(), weight=10.0,
316)
317ik_solver = ik.IKSolver(
318    model=ik_model, n_problems=world_count,
319    objectives=[pos_obj, rot_obj, joint_limits_obj],
320    lambda_initial=0.05, jacobian_mode=ik.IKJacobianType.ANALYTIC,
321)
322ik_iters = 24
323print(f"IK model bodies: {ik_model.body_count}, joint coordinates: {n_coords}")
324print(f"Task duration: {key_times[-1]:.1f} seconds")
325
326# --- Run the pick-and-place ---
327fps = 60
328frame_dt = 1.0 / fps
329sim_substeps = 10
330sim_dt = frame_dt / sim_substeps
331sim_time = 0.0
332
333
334def update_ik_targets(sim_time):
335    t = min(sim_time, float(key_times[-1]) - 1.0e-6)
336    interval = int(np.searchsorted(key_times, t))
337    t_start = key_times[interval - 1] if interval > 0 else 0.0
338    t_end = key_times[interval]
339    alpha = float(np.clip((t - t_start) / max(t_end - t_start, 1.0e-6), 0.0, 1.0))
340    cur = targets[interval]
341    prev = targets[interval - 1] if interval > 0 else cur
342    interp = (1.0 - alpha) * prev + alpha * cur
343    wp.launch(
344        set_task_targets,
345        dim=world_count,
346        inputs=[
347            ik_target_positions, ik_target_rotations, finger_pos_buf,
348            wp.vec3(*interp[:3].tolist()), wp.vec4(*interp[3:7].tolist()), float(interp[-1]),
349        ],
350        device=device,
351    )
352
353
354def simulate_coupled_frame():
355    global state_0, state_1
356    ik_solver.step(ik_joint_q, ik_joint_q, iterations=ik_iters)
357    wp.launch(set_gripper_q, dim=world_count, inputs=[ik_joint_q, finger_pos_buf, finger_idx0, finger_idx1], device=device)
358    wp.copy(dest=control_joint_target_q[:, :n_coords], src=ik_joint_q)
359    for _ in range(sim_substeps):
360        state_0.clear_forces()
361        collision_pipeline.collide(state_0, contacts)
362        solver.step(state_0, state_1, control, contacts, sim_dt)
363        newton.eval_ik(model, state_1, state_1.joint_q, state_1.joint_qd)
364        state_0, state_1 = state_1, state_0
365
366
367viewer = make_viewer("10_franka_cable_pick_place")
368viewer.set_model(model)
369try:
370    viewer.set_camera(wp.vec3(0.9, -1.4, 0.9), -22, 120)
371except Exception:
372    pass
373
374num_frames = int(np.ceil(float(key_times[-1]) * fps))
375print(f"Running {num_frames} frames (the first frame compiles kernels)...")
376for _ in range(num_frames):
377    update_ik_targets(sim_time)
378    simulate_coupled_frame()
379    viewer.begin_frame(sim_time)
380    viewer.log_state(state_0)
381    try:
382        viewer.log_contacts(contacts, state_0)
383    except Exception:
384        pass
385    viewer.end_frame()
386    sim_time += frame_dt
387
388print("Simulation finished. Reload the pre-opened viewer tab; press Ctrl+C to exit.")
389try:
390    while viewer.is_running():
391        time.sleep(0.1)
392except KeyboardInterrupt:
393    pass
394viewer.close()  # stops the viser server and frees the port

Key Takeaways#

You built a scene where two different solvers cooperate: SolverMuJoCo for the rigid Franka and SolverVBD for the deformable cable, joined by SolverCoupledProxy so the gripper can grasp the cable. The control loop reused the exact IK pattern from the previous lesson, driving a scripted keyframe sequence to pick up the cable and place it. This coupling framework is what lets Newton mix rigid, articulated, and deformable systems in a single simulation.