conclusions
For the most part, we were pleased with the results of the project. We successfully programmed Sawyer to recreate any image we fed it. The mechanical system was predictable, and our programs were robust enough to handle a variety of images with different characteristics. However, there are still many major areas for improvement.
MEETING DESIGN CRITERIA
Speed of Process
Due to our reliance on the inverse kinematics solver and the short distances between points, a simple image would take around 2 minutes to complete, which was very slow.
​
Robustness to Input Image Noise and Canvas Placement
​​​Empirically, our process handled input image noise extremely effectively. Inspection of the binary coverage image revealed relatively few artifacts on line drawings and real images, though some remained for art (call it artistic license!)
​
If the canvas was lowered after paper calibration, or was initially placed in a way that was not aligned to Sawyer, the drawing would be crooked or missing sections entirely.
​
Quality of Drawing (Subjective)
​​​Due to imperfections in the pen holder, time taken by the inverse kinematics solver, assumptions about the inverse kinematics behavior, and noise in paper calibration measurements, the resulting image would often appear as a collection of dots connected by thin lines, rather than a cohesive image.
​
However, the original shapes were almost always recognizable, barring significant canvas misalignment.
​
Robustness of Edge Detection and Path Planning Algorithms to Complexity of Line Drawing (e.g., number of lines/intersections)
​​​Regardless of the drawing on the paper or the background, we found that we were always able to use Canny edge detection to localize the paper in the input image.
​
However, our greedy path planning algorithm seemed to produce lower-quality results with increased input drawing complexity. This may partially be due to errors in Sawyer’s recreation becoming more egregious with increased input drawing detail. However, the number of separate paths output by the algorithm often seemed higher than necessary, and there might be discontinuities between connected segments of the input drawing (for example, if three lines met at a point). Fixing this would require modifications to our current algorithm.
DIFFICULTIES
Line Detection
We originally were planning to use the Canny Edge Detector to find the paths to follow. However, we noticed quickly that in our line drawings with a thick marker, Canny picked up two edges that corresponded to one line, such as in our squares drawing. Depending on the complexity of the image, it became difficult to know which two edges corresponded to one line. For images like this, we scrapped the Canny Edge Detector idea completely and implemented what we described in our Implementation page.
​
However, the Canny Edge Detector did prove to be useful for real-life images, where edges are thin and well-defined, like on the basket of apples.
Baxter to Sawyer
​​​We did most of our testing on Baxter due to scheduling conflicts. We suspected that moving our code to Sawyer would cause more problems, but it didn’t. It really consisted of changing the Baxter interface to the Sawyer one and switching the left arm to the right.
Pen Holder
Our pen-holder went through a variety of designs before it became a relevant design. Originally, in order to help out our robot artist, we fashioned a pen holder out of floral foam. The pen fits snugly inside the holder, and Baxter/Sawyer can hold it more easily. We assumed the foam was pliable enough to withstand any pressure from Sawyer’s grippers.
​
However, the foam proved to be problematic. Not only did it disintegrate into a green sand, but it also snapped easily. To prevent the snapping, we moved on to fashion a pen-holder out of wood. Wood is sturdier and more grippable. Or so we thought.
​
Wood’s firmness, however, meant fewer point of contact. When we had used the foam, the pads on Sawyer’s grippers sank in a little and the rest of the gripper was in contact with the pen-holder, creating a frictional force that prevented any rotation or swinging motion. However, we lost that squish factor in the wood, so the pen holder would swing, changing the z-coordinate of the tip of the pen. As the pen was dragged along the canvas, sometimes it would rotate so that the tip was no longer touching the surface.
​
In order to combat this additional problem (and preventing Stella from crying about using tape as a “super not robust solution, guys”), we used a table saw to cut notches in the wood of the pen holder. This prevented the grippers from slipping anywhere and kept the pen level and steady.
​
FLAWS, HACKS, AND IMPROVEMENTS
Calibration
In our current implementation, we designate our available canvas space by manually moving Sawyer’s gripper to the top-left and bottom-right corners of the canvas such that the pen is touching the canvas. This leaves two large spots on the canvas. While this is somewhat of a signature for Drawyer, it isn’t necessarily the most aesthetic. If we had the time, we would have liked to try to use AR tags to estimate the z-plane of the canvas--or even better, find the canvas even if it were an angled plane not parallel to the z-axis. We also could use CV to find the corners of the canvas to know what our workable canvas space.
​
Connect-the-Dots
Currently, our drawing technique is not very smooth. In reality, it is more of a game of finding some points on the desired lines and connecting them in small segments. We imagine that we could probably estimate each line as a mathematical function and have Sawyer draw the function, rather than small segments between points. This would result in a smoother image with fewer ink blots.
​
Style
There are a lot of ways to improve for style. One of the easiest would be to vary stroke thickness. We could get pressure feedback from Sawyer to manipulate how thin and thick each stroke would be. This way, rather than the uniform thick lines we currently achieve, we could achieve something more stylized. Furthermore, we could detect and add different colors. Even fancier, we could manipulate the threshold on the Canny Edge Detector to vary the level of detail in our image. An even bigger stretch goal would be to use this technique in conjunction with style transfer, where a neural net attempts to minimize a local ‘style’ loss while also preserving similarity between the input and styled images, allowing photos to be converted to various artists’ styles.
​
Below is a sample of what we played with when it came to level of detail. We played with the Canny threshold value on an already stylized painting of the Golden Gate Bridge. Changing the threshold allowed us to achieve varying levels of "abstractness."
