Title: Multi-Task Learning by a Top-Down Control Network. Abstract: As the range of tasks performed by a general vision system expands, executing multiple tasks accurately and efﬁciently in  a single network has become an important and still open problem. Recent computer vision approaches address this problem by branching networks, or by a channel-wise modulation of the network feature-maps with task speciﬁc vectors. We present a novel architecture that uses a dedicated top-down control network to modify the activation of all the units in the main recognition network in a manner that depends on the selected task, image content, and spatial location. We show the effectiveness of our scheme by achieving signiﬁcantly better results than alternative state-of-the-art approaches on four datasets. We further demonstrate our advantages in terms of task selectivity, scaling the number of tasks and interpretability. 
Code is supplied in the supplementary materials and will be publicly available.