Shape Based Deep Estimation of Future Plant Images

Abstract

Plants exhibit dynamic changes as they grow. For example, a new leaf may appear suddenly, and rotate and fold over time. Therefore, it is difficult to predict the growth of plants. For accurate growth predictions, it is important to predict the shapes, colors and textures of the leaves. The conventional methods simply use RGB images to predict the next plant at once. In this paper, we propose a novel deep network which is divided into two subnets of shape estimation and color reconstruction. Four gray time-series images are first aligned to a future target using a spatial transformer network (STN) for shape estimation. They are then fused using U-Net with two LSTMs to generate a future shape image. The color reconstruction subnet fuses the predicted shape with a RGB plant image to restore the color information. In addition, we use gray images with texture information for shape estimation instead of binary images with only simple information and RGB images with too much information. The proposed deep network can robustly generate future plant images for plant growth prediction. It is evaluated using our proprietary dataset as well as two public datasets for different types of plants. The experimental results demonstrate that our proposed network predicts the leaf shape more accurately and restores RGB. As a result, our method can create accurate future plant images.