2.1 Up-scale interaction

We first address the "up-scale" interaction technique, a concept borrowed from graphical design tools that allow pixel-precise editing operations by zooming in on the manipulated object. It enables users to engage in pixel-level manipulation, increasing precision and accuracy for various tasks such as graphical design, designing electrical components, and medical imaging analysis. We transfer this technique into a virtual environment and tailor it to enable precise manipulation of a robot end effector.

Figure 1 demonstrates the up-scale interaction technique: a seemingly straight line at a lower magnification can, upon up-scaling, reveal itself to be a line with noise. This enhanced visualization afforded by up-scaling is instrumental in facilitating more nuanced and precise interaction, overcoming the limitations inherent in lower-scale representations.

On the left, a standard-scale depiction of a noise-free trace (yellow) is shown. This depiction is annotated with the text "normal scale". Meanwhile, the right side displays an enlarged view revealing noise within the trace on the heart, which is annotated by the text "zoomed in".

On the left, a standard-scale depiction of a noise-free trace (yellow) is shown. This depiction is annotated with the text "normal scale". Meanwhile, the right side displays an enlarged view revealing noise within the trace on the heart, which is annotated by the text "zoomed in".

Applying this technique to three-dimensional (3D) interfaces, the "pixel-precise" interaction ensures detailed viewing and precise manipulation of 3D objects. In 3D interfaces, the "pixel-precise" interaction involves enlarging objects in the environment to a desired size. The factor to which extent the object should be scaled, depends on the precision that is expected to be achieved, keeping in mind the constraints introduced by the robotic arm. For example, when a robotic arm has a maximal accuracy of one millimeter, scaling the object beyond millimeter precision, has no particular benefit, as the robotic arm simply cannot get this precise.

However, this approach proves beneficial when precise movements form the basis for interacting with physical models. Such interaction includes surgical scenarios involving anatomical models or assembling compact circuit boards with tiny components.

As seen in previous work [7, 8, 6], diverse methods have been explored to implement the up-scale interaction technique in virtual environments. Commonly used methods typically involve either duplicating a virtual object or adopting a specific viewpoint on objects as a proxy to facilitate interactions. The proxy can be scaled to make interaction easier. Interactions performed on the proxy are applied directly on the virtual object it represents [7, 8, 6]. Similarly to "pixel-precise" interaction, we aim to use the up-scaling technique to enhance the precision and accuracy of object manipulation by offering users a more detailed view of the physical models represented digitally. This behavior is backed by the work of Yu et al. [7], which found that up-scaling models in a virtual environment improved the accuracy of precise annotations. As a side note, our techniques share similarities with the approach presented by Mine et al. [9]. However, a distinctive feature of our methodology is that we apply scaling exclusively to the manipulated objects, departing from the broader scaling of the entire virtual environment proposed in their work.

2.2 Constraining interaction

Interacting with 3D objects in VR using a tracked interaction device, especially when precise manipulations are required, can be cumbersome because of the degrees of freedom and the lack of a physical reference frame. As haptic feedback is challenging and expensive in virtual reality [10], we address this by constraining the interaction of the user. We accomplish this by allowing users to lock the dimensions they are operating in. This means that the user can decide which axis, or combination of axes, they can lock on a specific value. These axes are the x-, y-, and z-axis and the rotational axes of the robotic arm. The techniques, illustrated in Figure 2, contribute to a more intuitive and controlled user experience in VR-based human-robot interaction. This design aligns with Bowman et al.’s guidelines [11], emphasizing the importance of minimizing degrees of freedom in 3D interaction techniques.

The dimension locking technique described while locking along the z-axis. The image shows a yellow xy-plane where the trace of the user is reflected on.

The dimension locking technique described while locking along the z-axis. The image shows a yellow xy-plane where the trace of the user is reflected on.

A drawn heart that shows how the trace of the user is reflected using path guidance.

A drawn heart that shows how the trace of the user is reflected using path guidance.

These two techniques, i.e., dimension locking and path guidance, form an inherent advantage of operating within a virtual environment. The main idea behind this is the utilization of motion retargeting, where the actual path is mapped onto the expected path. Figure 2(a) visually illustrates z-axis dimensional locking. A 3D path (green) is projected onto a plane, thus ignoring z-axis deviations. This method enables users to optimize their positioning or derive feedback by placing a tool on a surface.

In cases where dimension locking proves insufficient, several challenges arise. This scenario is exemplified in Figure 2(b), where a complex shape, such as a heart, is used. The expected trace, in yellow, follows the complex shape of the heart. Relying on the "dimension locking" technique will thus fall short, as the complex surface/contour is hard to approximate using ground planes. For this reason, we implemented a "path guidance" technique, which allows the trace in green to be projected on the surface/contour of the 3D model. Implementing such a technique has the advantage that the user does not have to worry about following the surface/contour exactly, and a more suitable or relaxed position can be adopted during the manipulation.

Furthermore, the simultaneous utilization of these techniques manifests their synergy. In the medical context, this could be used for precise straight-line incisions. The path guidance technique would be used to trace the surface/contour of the heart. The dimensional locking technique would be used to lock the y- and z-dimension, enabling one-dimensional tracing. This ensures accurate scalpel positioning, facilitating precise straight-line incisions on the heart.

2.3 Just-in-time post-editing

Figure 3 illustrates the concept of post-editing: after a line tracing task, the user can correct the path to ensure the robot will trace a straight line. Since the user executes all actions in a virtual environment, a delay can be applied before the robot executes the specified actions. Before the path is transferred to the robot, users can still make minor adjustments to provide corrections for potential deviating interactions. These corrections are possible using two approaches: (1) manual correction and (2) digital correction. The manual correction approach relates to manually correcting traces on the model. The digital correction approach utilizes noise filtering algorithms, e.g., Kalman filters, to automatically filter out noise introduced in the trace. After this step, the traces are ready to be communicated back to the robotic arm.

The post-editing process is depicted using a three-step example: (1) tracing the line, (2) evaluating noise in the traced line, and (3) smoothing out the noise. A cursor is added to the evaluation and smoothing traces. In the first scenario, the cursor in the image shows which noise we want to filter out, and in the second scenario, "pressing in" the noise to filter it.

The post-editing process is depicted using a three-step example: (1) tracing the line, (2) evaluating noise in the traced line, and (3) smoothing out the noise. A cursor is added to the evaluation and smoothing traces. In the first scenario, the cursor in the image shows which noise we want to filter out, and in the second scenario, "pressing in" the noise to filter it.

3.1 Tracking manipulation

Accurate tracking of the operators’ movement is fundamental, as it is important to define a correspondence between physical and virtual environments [11]. In our prototype for human-robot interaction, we employed an outside-in optical tracking system (OptiTrack) to monitor the movements of a physical tool held by the operator. The use of any tool with attached infrared reflective markers is possible, but it is advisable to utilize tools familiar to the operator, minimizing the need for additional training. Leveraging familiarity with known tools, especially those commonly used by operators, facilitates a seamless integration process. By 3D-printing the tracked tool, replicas can easily be made whilst also keeping the possibility of integrating the reflective markers. On top of this, 3D printing also ensures safety, as the VR headset will prevent the user from acknowledging the tool in real life, keeping the user from anticipating sharp edges.

Figure 4(b) illustrates a 3D-printed pen tool used for interaction in the virtual environment. Five reflective markers are strategically placed on the pen to minimize occlusion between markers while using the tool. Figure 4(a) depicts the OptiTrack setup used for tracking the pen, held in the surgeon’s hand.

An illustration of the OptiTrack setup, where camera icons define the OptiTrack cameras. The graphic design of the surgeon holds onto an example tool that contains reflective trackers.

An illustration of the OptiTrack setup, where camera icons define the OptiTrack cameras. The graphic design of the surgeon holds onto an example tool that contains reflective trackers.

3D-printed pen with five reflective markers attached to it.

3D-printed pen with five reflective markers attached to it.

3.2 Up-scaling manipulation

The tracked coordinates are transmitted to the Unity prototype, enabling scaled manipulation. The up-scaling algorithm, depicted in Figure 5, involves a user-set scaling factor and calculates the distance between two consecutive coordinates. However, the consecutive coordinates are 3D coordinates, meaning that scaling might become difficult. For this reason, the z-axis calculations are done independently of the x- and y-axis. Figure 5(b) illustrates the calculations along the z-axis (height). By doing this, the scaled distance along the z-axis can easily be calculated by subtracting the second z-coordinate from the first z-coordinate and multiplying it with the scaling factor. Figure 5(a) illustrates the calculation along the x- and y-axis. We take both axes independently whilst scaling them but combine them using the Pythagoras theorem to achieve the correct scaled distance.

The scaling along the x- and y-axis combined using the Pythagoras Theorem. The blue sections depict the scaled distances along both axes. The red section is the resulting combined scaled distance.

The scaling along the x- and y-axis combined using the Pythagoras Theorem. The blue sections depict the scaled distances along both axes. The red section is the resulting combined scaled distance.

The scaling along the z-axis. The red section depicts the up-scaled distance.

The scaling along the z-axis. The red section depicts the up-scaled distance.

Operating within an up-scaled environment necessitates down-scaling these manipulations before relaying them to the robotic arm. The down-scale process takes on the same approach. However, instead of multiplying with the user-defined scaling factor, the distances between consecutive points now should be divided by the scaling factor. When the computations are performed, the interaction cycle defined in AntHand is complete.