Deep-Learning Methods of Cross-Modal Tasks for Conceptual Design of Product Shapes: A Review


 Conceptual design is the foundational stage of a design process that translates ill-defined design problems into low-fidelity design concepts and prototypes. While deep learning approaches are widely applied in later design stages for design automation, we see fewer attempts in conceptual design for three reasons: 1) the data in this stage exhibit multiple modalities: natural language, sketches, and 3D shapes, and these modalities are challenging to represent in deep learning methods; 2) it requires knowledge from a larger source of inspiration instead of focusing on a single design task; and 3) it requires translating designers' intent and feedback, and hence needs more interaction with humans, either designers or users. With recent advances in deep learning of cross-modal tasks (DLCMT) and the availability of large cross-modal datasets, we see opportunities to apply these learning-based methods to the conceptual design of product shapes. In this paper, we conduct a systematic review on the methods for DLCMT that involve three design modalities: natural language, sketches, and 3D shapes, which revealed 50 articles in the fields of computer graphics, computer vision, and engineering design. This review work identifies the key challenges and opportunities in applying DLCMT in the conceptual design of engineered products. The authors also propose a list of five research questions that point to future directions and call on the community to devote itself to principled research investigations that help translate knowledge from computer science to engineering design.