Non-Photorealistic Rendering Techniques for Artistic Robotic Painting

Abstract

In this paper, we present non-photorealistic rendering techniques that are applied together with a painting robot to realize artworks with original styles. Our robotic painting system is called Busker Robot and it has been considered of interest in recent art fairs and international exhibitions. It consists of a six degree-of-freedom collaborative robot and a series of image processing and path planning algorithms. In particular, here, two different rendering techniques are presented and a description of the experimental set-up is carried out. Finally, the experimental results are discussed by analyzing the elements that can account for the aesthetic appreciation of the artworks.

1. Introduction

Robotic painting is a challenging task that is motivated by an inner wish to discover novel forms of art and to experiment the technological advances to create something that can be aesthetically appreciated. Developing an automatic robotic painting system is hard, since the process comprises several different fields, including robotics, automation, image processing and art.

Busker Robot has been developed since 2016 at University of Trieste, Italy, in collaboration with University of Udine, Italy. The robotic system is composed of a six degree-of-freedom (DOF) robotic arm and a series of image processing and path planning algorithms that are capable of interpreting an input digital image into a real artwork. The aim of the work is, therefore, not to faithfully reproduce an image, as typical printers or plotters do, but to introduce an original contribution. The image processing is the result of the extensive implementation of non-photorealistic rendering (NPR) techniques with a trial and error procedure that is developed to the fulfillment of the artist. In each algorithm that we have implemented, several parameters can be controlled in order to modify the desired outcome. Moreover, these techniques are not fully deterministic, since random effects have been introduced, as explained in the paper, to obtain a different and non-repeatable result every time the algorithm is run.

The robotic machine that we have adopted for our purposes is a UR10 collaborative robot, produced by Universal Robots. Its collaborative features, such as force and speed limits as well as collision-detection systems, allow an artist to work side by side with the robotic arm. For example, the paint level, the type of brush and color can be changed or adjusted during the painting process, when needed.

Busker Robot, which name refers to street artists, has been previously presented in [1,2] and has been showcased for the first time in 2016 at the exhibition “Art, Science, Technology, Robotics” in Trieste, Italy. Then, it was shown at the SPS IPC Drives Italy 2017, at the “Algorithmic Arts and Robotics” exhibition during the international event Trieste Next 2017, at “Piccolo Teatro” in Milan, 2017, and, more recently, at the international festival “Robotics” in Trieste, 2018 (Figure 1) [3]. Furthermore, it took part to the 2018 International Robotic Art Competition (RobotArt), a context where 19 teams from all over the world competed and more than 100 artworks were created by painting robots; Busker Robot won an Honorable Mention [4].

With respect to previously published works [1,2], in this paper: (a) we present two novel non-photorealistic rendering techniques, cross-hatching and Random Splines, which are adopted for the artistic rendering of light and shades on an image, and (b) we employ non-diluted black Indian ink, instead of the previously adopted watercolor and gouache techniques.

The change from watercolor to ink was prompted by the idea to experiment the filling of areas with patterns of thin lines, in order to obtain different effects from those generated by water and pigments in the watercolor and gouache techniques. With respect to watercolor, in which the brush precision is not a big issue since the effects generated in the watered paper are uncontrollable and random, by handling the ink, the accuracy is much more important. Moreover, the height of the brush, the interval between two consecutive color refills and the planarity of the paper are much more critical.

With the ink painting technique several layers of thin lines and curves are applied to the paper, each one overlapped to the previous one, in order to produce dark effects in the shaded areas and lightness in the areas that are not covered by strokes. Experimental results show the feasibility of the proposed approach and good performances of the system architecture.

In the next section we provide a brief background including related works, in Section 3 we describe the architecture of the painting system, in Section 4 we present the non-photorealistic rendering techniques, and in Section 5 we include a description of the hardware set-up. The results are reported in Section 6, whereas Section 7 highlights the conclusion of this work and possible future improvements.

2. Related Work

Since the middle of the twentieth century, when the first programmable machines were adopted and what is known as “industrial robotics” started [5], research in this field has been focused in developing manipulators capable of replacing human labor in hazardous environments, which increases workplace safety and enhances production levels. By contrast, in recent years, the original industrial use has evolved and several artists have brought robotic machines from the factories to public exhibition spaces to develop novel artistic forms.

One of the earliest painting machines can be identified in the draughtsman automaton by Pierre Jaquet Droz, built in the 1760s [6]. In the last century, Jean Tinguely (1925–1991) can be considered one of the first artists that created painting machines to produce artworks with mostly random patterns [7]. Harold Cohen (1928–2016) devised AARON, a plotter capable of reproducing computer-generated artworks, considered pioneer in algorithmic and generative art [8].

In more recent years, several artists, engineers and computer scientists developed robots and machines skilled at drawing and painting. One example is given by the humanoid drawing human portraits by S. Calinon et al. [9], whereas G. Jean-Pierre and Z. Said developed the Artist Robot, a 6-DOF industrial robot programmed for drawing portraits using visual feedback [10]. Other examples are given by the works of C. Aguilar et al., who developed a system composed of an articulated painting arm and a machine-learning algorithm aimed at determining a series of brush strokes to be painted on the canvas [11]. An interesting example of robotic installation capable of drawing sketches of people is given by the machine developed by P. Tresset and F. F. Leymarie [12,13]. The system, based on visual feedback to iteratively augment and improve the sketch, draws using the equivalent of an artist’s stylistic signature. Furthermore, T. Lindemeier et al. presented e-David, a modified industrial robot that works with visual feedback and it is based on non-photorealistic rendering techniques [14,15]. The painting machine applies different layers of strokes on the canvas and produces beautiful artworks that mimic those produced by humans. Recently, the brush technique has been further improved by J. M. Gülzow et al. [16], who developed a method to analyze human brushstrokes and replicate them by means of the robotic painting system with impressive results.

More recently, examples of robots capable of producing artworks can be found in the work of X. Dong et al., who adopted a Scara robot to paint stylized portraits obtained with an algorithm based on a triangle coordinate system [17], of D. Berio et al., who adopted a collaborative Baxter robot to produce graffiti strokes [18], and by D. Song et al., who presented a 7-DOF robotic pen-drawing system, that is capable of creating pen art on arbitrary surfaces [19]. Furthermore, a robot artist performing cartoon style facial portrait painting using NPR techniques has been presented in [20].

Another application of robots in art, other than in pen drawing and brush painting, is that of spray painting, a task typically employed in industry [21,22,23]. Examples of robotic spray painting for artistic purposes can be found in the works of Viola Ago [24] and L. Scalera et al. [25], who adopted industrial robotic arms equipped with an airbrush, and in the work of A. S. Vempati et al., who developed PaintCopter, an autonomous UAV for spray painting on 3D surfaces [26].

Finally, other examples of artistically skilled robots are given by painting mobile platforms, such the ones by L. Moura, who developed Robotic Action Painter in 2006 [27], and by C.-L. Shih, who presented the trajectory planning and control of a mobile robot for drawing applications [28].

3. Busker Robot

Busker Robot is a system with amodular architecture, consisting of both hardware and software parameters (Figure 2). The hardware is composed of the painting machine, a 6-DOF robotic arm by Universal Robots, mounted in front of the painting support and equipped with a changeable brush. The software consists of the image processing and the path planning algorithms, that have been implemented in MATLAB^®. The image processing algorithms, i.e., the non-photorealistic rendering techniques, are applied to a digital input image that is processed in order to extract the contours, the details and the backgrounds that have to be reproduced on paper. Several different algorithms have been developed for Busker Robot [1,2]: it is the artists responsibility to choose which technique, or combination of techniques, has to be applied case by case.

The output of the artistic rendering is a structured list of points that identify the processed image. In particular, a MATLAB structure is adopted to handle the pixel coordinates of each single line or curve that composes the final image: each stroke is represented by an array that contains the coordinates of all the required way-points. These pixel coordinates are then scaled into the corresponding coordinates in millimeters on the painting canvas with a linear transformation. A trajectory planning is applied to these paths, in order to define the velocities profiles that the robot end-effector has to follow during the execution of the painting task. The trajectories are planned by adopting trapezoidal speed profiles for each motion: after the acceleration phase, the end-effector of the robot linearly moves with constant speed and circular blends through the way-points. After a stroke is completed, the robot tool z-coordinate is increased in order to lift the brush from the paper; it is then moved linearly at constant speed towards the first point of the next stroke but remaining at a higher z-coordinate; finally, it is lowered at paper level and the new stroke is started.

In the trajectory planning module, all the commands for the paint refill and the brush drying are as well defined. The output of this process is the list of commands for the robot, written in the specific UR Script Programming Language in a specific script file, which includes all the instructions needed for a specific layer of the artwork (Section 4). In particular, each command line corresponds to the motion of the robot end-effector through a specific way-point and it includes the Cartesian position coordinates in the operational space, the orientation of the tool in axis-angle representation, the velocity, the acceleration and the blend radius.

The execution of the painting task is then started and monitored from a remote computer connected to the robot controller via Ethernet. It is worth noting that the software is modular and, therefore, if any parameter regarding the painting set-up has to be changed, only the trajectory planning can be rerun, based on the same results of the artistic rendering module.

4. Non-Photorealistic Rendering Techniques

The two novel non-photorealistic rendering techniques implemented in this work are described in the following. These techniques can be adopted for the processing of the light and shades of an image in a manner that can enhance the aesthetic appreciation. Indeed, we would like to introduce a motor activity in the brush painting, so as to recall the gestures of a human artist in the robotic strokes. It has been demonstrated that the aesthetic experiences are enhanced in dynamic paintings that evoke a sense of movement and activate the brain area of visual motion during the observation [29,30].

Similarly to [1], the algorithms here presented are based on a thresholding process that is applied to a grey-scale version of the original image. For a given threshold $I_T$, i.e., the value of a specific grey intensity level (between 0 and 1), each pixel of the original image with intensity $I\left(P_i\right)$, is transformed into white if $I\left(P_i\right)\ge I_T$, or into black, otherwise. In this manner, the image can be decomposed into several layers, each one characterized by a different area to be painted, in a way similar to [31].

The algorithms presented in the following, cross-hatching and Random Splines, are applied to several layers of the same image. The paths resulting from the algorithms are then overlapped to obtain the final result. The NPR algorithms can be used for the processing of the areas with uniform grey intensity level of an image; furthermore, contours and details can also be added to the final result by adopting other techniques such as Canny Edge Detector, Hough Transform or Difference of Gaussians [1,2,32]. The Canny Edge Detector algorithm has been adopted in this work.

4.1. Cross-Hatching

The first NPR technique that we present in this paper is the cross-hatching. It consists of several layers of cross-hatch marks that allow to create nuanced differences in tone and light. The basic idea of hatching is to cover an area with parallel lines placed closely together: the areas where the hatching is placed will appear darker or in shadow, whereas the areas without hatching will appear illuminated. After the first set of lines, a second set of hatch marks can be placed on top, with a different orientation with respect to the first layer. Several sets of closely spaced lines can be drawn, corresponding to different thresholds. In this way, more saturated dark effects are obtained in the darkest side of the artwork. In the literature, examples of cross-hatching rendering can be found in [33,34,35].

In order to give the artwork greater vibrancy and to reproduce the gesture of an human painter, random effects have been implemented in our algorithm, so as to obtain small deviations (of the order of few millimiters) in the alignment and linearity of the single lines. These are generated by introducing a random perturbation in the longitudinal and transversal direction of a predefined number of points in the line, which becomes a poly-line. The points of the poly-line are then automatically blended to feed the robot with a smooth path. The maximum error for each of the two directions can be manually set before running the algorithm. The lines are finally filtered in order to remove the shortest segments which would otherwise be painted as dots, by the robot.

Two input images have been adopted for the evaluation of the algorithms: the Hydrodynamic Power Station and the Still Life with Tea Kettle, reported in Figure 3a,b. In this work, we used the Canny edge detector algorithm to define the contours for both the two images. Simulated results of the cross-hatching algorithms are reported in Figure 4 and Figure 5. In these images, the thickness of the lines is not representative of the real thickness of the brushstrokes, which is given by the z-coordinate of the brush. The threshold and orientation employed for each layer of the images are reported in

Table 1. Threshold and orientation for each layer of the Hydrodynamic Power Station, simulations with cross-hatching.

Layer	Figure 4a		Figure 4b
	Threshold	Angle	Threshold	Angle
1	0.20	80${}^{\circ }$	0.30	70${}^{\circ }$
2	0.30	30${}^{\circ }$	0.40	40${}^{\circ }$
3	0.35	45${}^{\circ }$	0.45	50${}^{\circ }$

and

Table 2. Threshold and orientation for each layer of the Still Life with Tea Kettle, simulations with cross-hatching.

Layer	Figure 5a		Figure 5b
	Threshold	Angle	Threshold	Angle
1	0.10	5${}^{\circ }$	0.25	85${}^{\circ }$
2	0.15	50${}^{\circ }$	0.30	80${}^{\circ }$
3	0.30	10${}^{\circ }$	0.35	50${}^{\circ }$
4	0.40	30${}^{\circ }$	0.40	10${}^{\circ }$
5	0.45	70${}^{\circ }$	0.45	5${}^{\circ }$

. From the figures it can be seen that, by adopting higher thresholds, the total covered area increases, the shades are better rendered and the light contrast is more marked.

Table 3. Software variables for the non-photorealistic rendering techniques.

NPR Technique	Software Variables
Cross-hatching	- grey threshold
	- distance between lines
	- angle of orientation
	- maximum error in longitudinal direction
	- maximum error in transversal direction
	- minimum length of lines
Random Splines	- grey threshold
	- number of random points
	- size of the box
	- survival rate parameter
	- minimum length of lines

reports the list of software parameters that can be manually tuned in the cross-hatching technique.

4.2. Random Splines

A different technique for the rendering of light and shades is Random Splines. Differently from other works in the field of NPR, where splines curves have been used to approximate the contours of the subjects to be painted, e.g., in [36], in this paper we adopt spline curves to fill areas characterized by uniform grey intensity level in a random manner. In this technique, a predefined number of points is randomly generated inside the area defined by a grey-scale threshold. For each point a survival rate is computed, which is proportional to the grey intensity of that point in the original image and to a constant defined by the user. A selection of the points is then applied in order to facilitate the survival of those positioned in the darkest side of the image.

A box centered in each point is generated and three random points are selected within the box boundaries. These points are interpolated with a spline curve, which is then sampled in order to save the sequence of way-points that the robot end-effector has to follow at constant linear speed during the painting task. The splines are sampled in the Cartesian space to obtain a good compromise between accuracy and number of points: a high number of points could, indeed, increase the computational effort of the robot controller and introduce vibrations in the tool if the distance between the points is smaller than the blend radius. A graphical example of the spline generation is shown in Figure 6.

The size of the box allows to control the total length of the splines, whereas the survival rate constant allows to concentrate the strokes in the darkest part of the original grey-scale image. Since the points are placed inside a thresholded layer but the box is independent from the layer contours, the resulting splines can be drawn even in parts of the image that do not belong to that layer. This results in lines that exceed the natural border of an object in the image. The resulting splines can be finally filtered to remove those with the shortest path. The software parameters that can be manually defined are listed in Table 3. Two examples of Random Splines for the Still Life with Tea Kettle are shown in Figure 7, obtained with the parameters reported in

Table 4. Threshold and box size (pixel) for each layer of the Still Life with Tea Kettle, simulations with Random Splines.

Layer	Figure 7a		Figure 7b
	Threshold	Box Size	Threshold	Box Size
1	0.15	30	0.25	30
2	0.20	20	0.30	25
3	0.20	30	0.35	20
4	0.30	60	0.40	15
5	0.40	40	0.45	10
6	0.40	50	0.50	10

.

5. The Painting Machine

The painting machine that we adopted for Busker Robot is a 6-DOF UR10 robot. It is an industrial collaborative robot, usually adopted for pick-and-place, material handling, assembling or other manufacturing processes in environments where a human-robot interaction is needed. It is provided with speed and torque limits as well as collision-detection systems, that allow a human to work safely side by side with the machine. However, the speed of the robot has to be kept low, since it affects the quality of the painted strokes.

The UR10 can handle 10 kg of payload and can count on 1300 mm of working radius, which allows to easily paint on a 450 mm × 850 mm surface. Furthermore, a repeatability of ±0.1 mm on the positioning of the end-effector is ensured by the manufacturer [37]. The six axes allow to paint on non-horizontal surfaces and to perform complex motions such as dipping the brush in the paint cup or scrape it in order to remove excess paint. The robotic arm is fixed on an aluminum frame and its flange is equipped with a custom support for the application of the brushes by means of 3D printed plastic supports. An automatic brush change system has been developed for the adoption of brushes with different size, and used, in a previous work [1], to paint strokes with varied thickness.

The target surface for the painting task is identified with respect to the base coordinate system of the robot (Figure 8). Since a parallelism error between the base of the robot and the target surface could affect the application of color on the paper, a calibration of the painting board is carried out before starting a new artwork. Furthermore, if the target surface is not perfectly planar, a planarity error arises and an approximate interpolating plane has to be defined by more than 3 points. This calibration is performed by measuring a set of n points $P_i={\left\{x_i{y}_i{z}_i\right\}}^T$, with $i=1,\dots ,n$, on the drawing surface, with respect to the robot reference frame ${\left\{000\right\}}^T$. After equipping the end-effector of the robot with a rigid and sharp tool with the same length of the brush, the tip of this is positioned so as to touch the surface and the corresponding pose of the robot is recorded. In order to calculate the approximating surface $Z=a_{0}X+a_{1}Y+a_2$, the following over-determined linear system has to be solved:

$$\left\{\begin{array}{c}{a}_0{x}_1+a_1{y}_1+a_2=z_1\\ \dots \\ a_0{x}_n+a_1{y}_n+a_2=z_n\end{array}\right.$$

(1)

By rewriting the system in matrix form $AH=B$, with

$$A=\left[\begin{array}{ccc}{x}_1& y_1& 1\\ \dots & \dots & 1\\ x_n& y_n& 1\end{array}\right]\mathrm{and}B=\left\{\begin{array}{c}{z}_1\\ \dots \\ z_n\end{array}\right\}$$

(2)

The coefficient vector $H={\left\{a_0{a}_1{a}_2\right\}}^T$ can be obtained as $H=A^{+}B$, where $A^{+}={\left(A^{T}A\right)}^{-1}{A}^T$ is the pseudo-inverse matrix of A that minimizes the squared sum of errors. The z-coordinate of the robot tool is then automatically updated via software to paint on the approximating surface.

Furthermore, the position of the paper, of the color cups and of the brush change repository is measured with respect to the robot reference frame, positioned in the central point of the robot base.

In this work, we adopted non-diluted black Indian ink provided to the robot in small plastic containers. After the dipping, the robot wipes off excessive paint from the brush on the edge of the ink cup.

A characterization of the brush strokes has been performed in order to estimate the influence of the z-coordinate of the brush (Figure 9), the traveled path and the robot speed on the stroke parameters: the black intensity and the thickness. This enables us to control the thickness of the strokes, to determine the traveled length after which the brush has to be dipped in the ink and to ensure a smooth deposition of the paint along the strokes.

To highlight the differences between watercolor and ink painting, some examples of stroke characterization are reported in Figure 10 and Figure 11 for a watercolor and an ink stroke, respectively. It can be seen that in the ink case the black intensity is much more constant along the stroke, with respect to the watercolor one, where the dilution of pigments leads to transparency effects.

An experimental characterization has been performed and a set of linear strokes, each 250 mm long, has been painted by the robot while varying the z-coordinate from $z_1=0$ mm to $z_3=-2$ mm with $\Delta z=1$ mm, and the maximum robot speed, from $v_1=0.1$ m/s to $v_3=0.3$ m/s with $\Delta v=0.1$ m/s. The maximum robot acceleration has been varied accordingly in order to maintain the same length of the constant velocity phase in the trapezoidal speed profile. Each stroke has been repeated three times and the results have been digitized and processed in order to obtain mean and standard deviation values. Figure 12 and Figure 13 report these results, where the effects of the brush z-coordinate and robot speed on black intensity and stroke thickness can be seen, respectively.

From Figure 12, it can be seen that the black intensity is almost constant along the strokes and it is not affected by the z-coordinate or by the robot speed. The black intensity shows a limited increase in the last part of the stroke. This could be due to the trapezoidal motion profile, which is characterized by a deceleration phase that can affect the paint deposition process. In this manner, a slower speed can cause an increase in the paint deposition per linear distance. On the other hand, a high painting speed can cause undesirable effects in the starting and end points of the stroke, due to a rough landing and detachment of the brush on the target surface, as well as the dry brush effect [1].

Conversely, the stroke thickness shows a decreasing trend along the stroke (Figure 13). The z-coordinate of the brush affects the stroke thickness, since the more the bristles are pressed against the paper, the larger the strokes are painted. The robot speed seems not to be relevant in the tested range of values. The values of the mean thickness along the whole strokes have been computed as well and the results are reported as box-plots, in Figure 14.

Busker Robot takes from one to three hours to complete one artwork, depending on the complexity of the subject, the number of layers and the speed of the robot. The total painting time can be optimized by finding the best compromise between quality of the strokes and speed of the robot, in order to minimize the brush travel time, while, at the same time, ensuring a smooth deposition of the paint along the strokes. A frame sequence of Busker Robot painting Hydrodynamic Power Station is reported in Figure 15.

6. Results

Experimental results of the robotic painting system are reported in Figure 16 and Figure 17, where the resulting artworks “Hydrodynamic Power Station” and “Still Life with Tea Kettle” are shown. The second subject has been realized with both the cross-hatching and the Random Splines techniques, to have a visual comparison between the results of the two NPR algorithms (Figure 17a,b).

The three artworks have been realized with the parameters reported in Table 1, Table 2 and Table 4, respectively. While the cross-hatching reproduces the effects of the light and shades in the image with a clean and ordered style, the Random Splines approach creates a configuration of lines that are closer to a sketch drawn by a human. The lines exceed the contours and the result is not accurate and tidy as in the cross-hatching case.

A precise calibration of the brush strokes is very challenging and, therefore, it can be seen that not all the strokes appear with the same thickness and color intensity. The interaction between brush bristles and paper, the ink deposition, the ripples of the paper and the amount of paint in the brush are not easy to be modeled and controlled, especially without feedback, and a lot of work can be done to improve the results. Nevertheless, the artworks produced by our automatic painting system are unique and can be considered an example of integration between robotics, image processing and art.

Even though a quantitative analysis of the artworks is not possible, since the qualitative comparison with the original image and the aesthetic appreciation is subjective, Busker Robot obtained a good feedback and has been considered of interest by the press and the public at several exhibitions.

7. Conclusions

In this paper we presented non-photorealistic rendering techniques for artistic robotic painting, that have been implemented for the processing of digital images into real artworks with Busker Robot, our robotic painting system. The system is capable of reproducing an image that has been previously processed through the application of artistic rendering algorithm. In particular, we presented two novel techniques, the cross-hatching and the Random Splines, that are used to process the light and shade of a painting. Experimental results shown the feasibility of the proposed approach and system architecture. In particular, two sample images have been tested with the new algorithms and the robotic system, with interesting results. Future works will include a further brush stroke characterization to further improve the quality and the repeatability of the lines. A visual feedback can as well be introduced to give the robotic system a consciousness of the current status of the painting, that can be used to correct possible errors or place the next stroke.