drawyer

design

~ GOALS ~ EVALUATION CRITERIA ~ DESIGN ~ CHOICES AND TRADEOFFS ~

GOALS

Well--maybe hopefully not utterly incomprehensible. Meet Drawyer, a Sawyer fitted with a fat Sharpie that can draw any image you feed it. Simple line drawing? Got it. A picture of a bowl of fruit? Easy. Maybe Drawyer’s drawings aren’t perfect, but you can grant a robot a little bit of artistic license, can’t you?

Welcome to a new era of art! An era fueled by algorithms: mapping muse to canvas, finding paths of lines, and executing via robot. You think art is meant only for humans to create? O’Keeffe it to yourself; that’s a load of Pollocks. It’s okay; we’ve Goya back. We’ll introduce you to Drawyer and robot-created art, one step at a time.

EVALUATION CRITERION

DESIGN

GOALS

Our system will take as input an unprocessed image of a line drawing on an arbitrarily oriented rectangular sheet of paper. We will then enable Sawyer to copy this drawing onto a canvas.

EVALUATION CRITERIA

We plan on evaluating our project with the following criteria:

Speed of process
Robustness to noise in input image and canvas placement
Quality of drawing (subjective)
Robustness of edge detection and path planning algorithms to complexity of line drawing (e.g., number of lines/intersections)

DESIGN

Hardware

Gripper Augmentation

We augmented Sawyer’s parallel grippers with a block of wood with parallel vertical grooves on opposing sides. By fixing a marker to this block of wood, we were able to ensure its rigidity in 6-DOF, thus ensuring a smooth drawing.

Image Processing

Paper Localization

We use the Canny edge detector to determine the contours corresponding to the paper in the original image.

We then warp the section of the image containing the paper to fill a rectangular area.

Conversion to Low-Resolution Binary Image

We next convert the preprocessed image to a low-resolution binary image, which indicates whether or not we would like to cover each pixel in Sawyer’s rendering. We first present our method for accomplishing this for line drawings, then an extension of this which operates on real-world images and art, when the paper detection step would be skipped.

For Line Drawings

We assume that the lines will be black on a white sheet of paper, and that they have a large but uniform width. Therefore, we can simply threshold the original image, then downsample the image to the desired resolution.

For Art/Real-World Images

We assume that edges will generally be thin. We therefore apply the Canny edge detector with an adaptive threshold based on the variance of pixel values in the original image, and an additional hyperparameter for art vs. real images.

Path Planning

From this low-resolution binary representation, we now need to find paths that Sawyer should trace over such that we will generally be tracing curves in the original drawing, rather than simply marking a sequence of disjoint points. This is accomplished by an algorithm described the ‘Implementation’ section of our website.

Path Execution

Calibration and Coordinate Mapping

We need to map the 2D points in the paths to 3D points in Baxter’s frame. Therefore, we manually move the marker to the top left and bottom right corners of the canvas, and use forward kinematics to determine the coordinates of these corners. We require that the axes’ of the canvas are aligned to Sawyer’s base frame, so computing the 3D coordinates of the 2D points is then straightforward.

Calibration and Coordinate Mapping

Using the MoveIt planner, we then have Sawyer move between the points specified by these paths, lifting up the marker when moving between the paths.

CHOICES AND TRADEOFFS

Rather than having the user move Sawyer’s arm to touch opposite corners of the canvas, it would have been more efficient to affix 3 AR tags to corners of the canvas. It would have then been possible to use computer vision to determine the orientation and location of the canvas. We did not do this, and instead fixed the canvas orientation, in order to keep the design simpler. This also allowed us to use only 2 points to calibrate Sawyer, instead of 3. Further, using CV techniques here may sacrifice some precision.
During calibration and drawing, we required that Sawyer’s gripper be oriented parallel to Sawyer’s base’s z-axis. This simplified our kinematics calculations and allowed us to use library functions with few modifications.
When having Sawyer trace paths, we assumed that the points were close enough together that the inverse kinematics solver would calculate straight lines in a plane. However, enforcing these constraints would have required that we write our own inverse kinematics solver.
We could have used the joint torques of Sawyer to calculate the force with which the marker was being pressed into the page. Instead, in order to make the above assumptions more likely, we had Sawyer draw lines points 1mm below the measured plane of the canvas. This was simpler, but less robust, and smashed the pen tip into the paper, rendering it useless for other purposes.
We limited the resolution of the pixel-covering binary image to limit the number of times we would need to call the inverse kinematics solver. This also added robustness to stray marks and noise.
The design of the pen holder is robust and durable. No compromising trade-offs were made here.

GOALS

EVALUATION CRITERIA

CHOICES AND TRADEOFFS

DESIGN