Runner Cutting Automation Report

By Mojtaba Ahmadi, Ph.D., Senior Production Automation Engineer | Published: October 13, 2022 | Updated: November 14, 2023

Overview

          Strawberry plants develop “runners”, also called “stolons”, for propagation in response to longer days and warmer temperatures. These runners are like stems but grow over the soil surface, which can extend out several inches from the crown, then take root in the soil and result in clone plants called “daughter plants” (Figure 1).

Figure 1. Morphology of strawberry plant (Credit: Cal Poly Strawberry Center).

          In fruit production fields, runners must be removed to minimize the plants' competition over limited resources, which helps to promote flower and fruit production. Most commonly grown strawberry varieties like Monterey, Fronteras, Cabrillo, and Portola produce lots of runners during the berry growing season, which means growers must bear a significant production cost to manage their fields in order to generate higher fruit yields. Figure 2 shows the labor intensity of conventional runner cutting.

Figure 2. Conventional runner cutting is a tedious, labor-intense operation in strawberry production (July 2020, Santa Maria).

          According to the 2021 Cost Study of Strawberry Production, runner removal is the second-most labor-intense operation in strawberry production. It is estimated to take 12.5 hours per acre per month during 10 months of the production season, which results in an approximate labor cost of $2,683 per acre. Given that California has 38,026 planted acres of strawberries as reported in 2022, the total cost of runner removal is over $100 million for the industry.

Problem

          Even though runner removal is essential for maintaining the healthy condition of fruit production fields, most growers are having a hard time performing it as frequently as needed due to labor shortages in the agricultural industry. In some cases, growers might have their field workers prioritize picking fruits and ignore runner cutting for some consecutive months. Figure 3 shows an extreme case of runner propagation in a fruit production field. An autonomous runner-cutting machine can help growers address this problem without sacrificing strawberry production.

Figure 3. An extreme example of runner propagation in a fruit production field (Credit: Dr. Gerald Holmes; July 2021, Santa Maria).

Proposed Solution

          Since 2020, the California Strawberry Commission (CSC) and the Cal Poly Strawberry Center (CPSC) have been investigating the potential use of automated runner cutters in the field. Our engineering team has developed a deep-learning framework to detect and identify runners from other strawberry plant organs by using RGB image data. In addition, the CPSC began a collaboration with Strio AI, a robotics start-up company based in Boston, MA, to develop an automated runner cutter. As part of their efforts, Strio AI built a robotic platform that was able to easily maneuver in strawberry fields, and with the use of a robotic arm and the fusion of RGB and depth data, it was able to successfully cut runners in the field.

Runner Detection

          Identifying runners can be achieved through a variety of methods. Initially, the CPSC tried traditional image processing techniques such as color segmentation, edge detection, and bandpass filter methods to identify runners by considering parameters like color, size, shape, and texture. However, the accuracy of each of these image-processing techniques depends on plant variety, time of year, and farm location. As a result, we decided to take a machine-learning approach to identify runners and differentiate them from stems. This approach utilized models trained on a wide range of image data from different field conditions and plant varieties. Due to the similarity between runners and plant stems as well as the crowded nature of strawberry canopies, common object detection networks like region-based convolutional neural networks (R-CNN) and YOLO were not chosen. Instead, we used a two-step framework for runner detection from RGB images. First, the segmentation network identified the runner pixels as a runner mask image, then an object detection network was applied to the created mask image to recognize and localize different instances of runners. With the combination of these two networks, we were able to achieve instance segmentation for runners, as shown in Figure 4.

Figure 4. Runner-detection deep-learning algorithm flowchart.

          To prepare the data for training and testing, runners in the images were labeled by using the semantic segmentation labeling tool of Amazon Web Service (AWS) SageMaker. Bounding boxes were placed around each runner’s label. Google Cloud Platform (GCP) AutoML Vision Object Detection was utilized to train an object detection model. This allowed us to predict runners and put bounding boxes around them for counting purposes.

Data Collection

          In order to provide the data needed to train and test our model, we collected image data from two strawberry fields in the Watsonville/Salinas area with the help of Strio AI's camera system and robotic platform as well as CPSC's tractor-mounted enclosed data collection platform (Figure 5). These data were collected at the end of July 2021 from three strawberry varieties (Monterey, Cabrillo, and Albion). The first two varieties were from the field in Salinas, and the last variety was from the field in Watsonville. All plants were conventionally planted in Fall 2020.

Figure 5. Data collection platform on a tractor (left) and Strio AI robotic platform (right) (July 2021, Watsonville and Salinas).

          During the data collection process, we used a color camera to capture still images at 5 frames per second with 1024×768 pixel resolution and set the exposure time as stated in Table 1. The exposure times were set differently based on travel speed to allow the camera more light absorption while avoiding blurriness on the images. Figure 6 shows examples of the images that were captured.

Table 1. Image data collection setup and data packages.

Cultivar	Location	Travel Speed	Exposure Time	Images and Labels
Monterey	Salinas	~1 mph	4000 µs	Download Data Package (773 MB)
Cabrillo	Salinas	~1 mph	4000 µs	Download Data Package (720 MB)
Albion	Watsonville	0.2 - 0.4 mph	5000 - 6000 µs	Download Data Package (615 MB)

Figure 6. Examples of collected image data: Monterey (left), Cabrillo (middle), Albion (right).

Results and Discussion

          In this study, for each cultivar, 80% of the collected data were randomly selected for training, and the remaining 20% were considered for testing. The results indicate that machine-learning methods are able to detect runners with high accuracy, as demonstrated in Table 2.

Table 2. Pixel-wise accuracy of runner detection.

Cultivar	Deep Learning Model	Number of Epochs	Intersection over Union (IOU) (%)	Boundary F1 Score (BF score) (%)
Monterey	ResNet-50	120	52	87
		150	52	87
	ResNet-101	120	52	87
		150	53	87
Cabrillo	ResNet-50	120	54	87
		150	55	88
	ResNet-101	120	56	89
		150	55	89
Albion	ResNet-50	120	72	95
		150	72	94
	ResNet-101	120	72	95
		150	72	95

          For Monterey, Cabrillo, and Albion, the best results for IoU and BF Score were 53% and %87%, 56% and 89%, and 72% and 95%, respectively. Even though the results between different trained models for each cultivar were similar to each other, the deep network was able to detect runners for the Albion variety with higher accuracy compared to the Monterey variety. This accuracy difference can be related to the different data collection and ambient conditions between the two varieties as indicated in Table 1.

          Based on this study, runner detection in strawberry fields can be performed by taking advantage of deep-learning models. However, in order to optimize the performance of these models, further studies are needed to improve data collection conditions and fine-tune deep-learning models.

Challenges

          The performance of a deep-learning framework to detect runners using RGB images depends on the quality of the acquired images. During the image collection process, the project faced several challenges that should be addressed for future improvements. These are listed below:

• Strawberry varieties: Each variety can be very different from others in terms of overall size, stem length, etc., which all can have an effect on data collection and model performance.
• Canopy size: As strawberries grow during the season, the canopy size also increases, which results in runners being partially or entirely covered by leaves.
• Strawberry bed configuration (two, three, or four rows): It is easier to capture runner images on two-row beds compared to four-row beds. On four-row beds, most of the runners are found in the middle of the bed. There are also more plants on the bed, which affects runner visibility especially as the canopy size grows.
• Field of view: It is important to capture images where the full length of a runner is clearly visible, so a wider field of view will help to capture more information in an image.

Strio AI

Figure 7. Strio AI's runner-cutting robot in operation          
(October 2021, Watsonville).

          Part of this project was to collaborate with the private sector on the development of an automated robotic runner cutter. Our team approached multiple robotics companies, and Strio AI, a Boston-based robotics start-up company, proposed a robotic solution for the laborious task of runner cutting.

          In December 2020, Dr. Ruijie He, CEO and co-founder of Strio AI, began visiting CPSC as well as strawberry fields in California and Florida to better understand runner cutting and its challenges. Dr. He and his team were able to develop the first prototype of a robotic runner cutter and test it in the strawberry fields in California and Florida. This robot was able to maneuver throughout the strawberry fields and perform runner-cutting tasks without causing any damage to the plants or fruits. This robot will need more improvements before it becomes commercially ready to use in strawberry fields. However, in March 2022, Strio AI joined Zoox, which is an Amazon subsidiary company that develops autonomous vehicles, and they are no longer active in the strawberry automation field.

References

1. Bolda, M., Dara, S. K., Fallon, J., Sanchez, M., Peterson, K. (2015). Strawberry Production Manual for Growers on the Central Coast. https://ucanr.edu/sites/santabarbaracounty-new/files/228579.pdf
2. Strand, L. L. (2008). Integrated pest management for strawberries (Vol. 3351). UCANR Publications.
3. Bolda, M. P., Murdock, J., Summer, D. A. (2021). Sample costs to produce and harvest strawberries: Central Coast region, Santa Cruz and Monterey Counties. University of California Agriculture and Natural Resources.
4. Darrow, G. M. (1929). Development of Runners and Runner Plants in the Strawberry. United States Dept. Agri. Technical Bulletin, (122), 1-28.
5. Lassen Canyon Nursery, Inc. (n.d.) Strawberry Varieties. Retrieved May 17, 2022, from https://www.lassencanyonnursery.com/strawberry-varieties/
6. UC Davis, Innovation and Technology Commercialization. (n.d.) Strawberry Licensing Program. Retrieved May 17, 2022, from https://itc.ucdavis.edu/strawberry-licensing-program/#cultivars

Related Materials

FIRA USA 2023 - Panel Discussion "Practical translation of R&D and technology to the California strawberry industry", Sept 19th, 2023
Invited Speaker: Mojtaba Ahmadi, Ph.D.,
Senior Production Automation Engineer, California Strawberry Commission/Cal Poly Strawberry Center

 

Cal Poly Strawberry Center - Automating the California Strawberry Industry
Speaker: John Lin, Ph.D.,
Director of Automation Engineering, California Strawberry Commission/Cal Poly Strawberry Center

Contacts

Mojtaba Ahmadi, Ph.D.

• California Strawberry Commission & Cal Poly Strawberry Center
• p: 831-254-3577
• mahmadi@calstrawberry.org
• mahmad3@calpoly.edu