The Visual Task Adaptation Benchmark
ml machine learning cnn imagenet pretraining finetuning fine-tuning google benchmark initialization supervised unsupervised bert artificial intelligence score
This paper presents a new benchmark for Visual Task Adaptation (i.e. BERT for images) and investigates several baseline methods for doing so.

Abstract:
Representation learning promises to unlock deep learning for the long tail of vision tasks without expansive labelled datasets. Yet, the absence of a unified yardstick to evaluate general visual representations hinders progress. Many sub-fields promise representations, but each has different evaluation protocols that are either too constrained (linear classification), limited in scope (ImageNet, CIFAR, Pascal-VOC), or only loosely related to representation quality (generation). We present the Visual Task Adaptation Benchmark (VTAB): a diverse, realistic, and challenging benchmark to evaluate representations. VTAB embodies one principle: good representations adapt to unseen tasks with few examples. We run a large VTAB study of popular algorithms, answering questions like: How effective are ImageNet representation on non-standard datasets? Are generative models competitive? Is self-supervision useful if one already has labels?

Authors: Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, Neil Houlsby

https://arxiv.org/abs/1910.04867
https://github.com/google-research/task_adaptation
 Hi there. Today we're looking at the visual task adaptation benchmark by a list of authors that's way too long to read out all from Google Brain. So what is this paper? This paper cares about a new benchmark that is abbreviated VTab and VTab is a benchmark for a task called visual task adaptation. So a benchmark, the meaning of a benchmark is it's kind of a number that you achieve with a model and whoever has the highest number is the best at this task. So the benchmark kind of standardizes how you evaluate models and the model is here. They do visual task adaptation. So what is visual task adaptation? So this is visual task adaptation. It's kind of illustrated in this figure. Imagine you have a bunch of what are called visual tasks and a visual task, and this is the right side here, a visual task is anything that can be solved from just visual input. So basically given a picture or many pictures and you ask kind of a question about it, if that question can be answered by just looking at the picture then that's called a visual task. For example in this data set you might be asked whether a picture contains a dog or a cat. In this data set you might be asked to outline where the objects are. So here the plane, you might be able to segment or you might be able to point out where buildings are in the images. Right here, here, there's no building here. So there's varieties of tasks that are possible. Or in the bottom domain you might be asked which one of the two red dots here is closer to the observer in 3D space. Or you might be asked in this picture please count the number of gray boxes. So there's a bunch of, all of these count as visual tasks. Now the setting that the authors imagine here is there are many of these visual tasks in the world for which there isn't much training data. Imagine something like this. These are aerial images so you kind of need a satellite or a plane to obtain them and then you need to label them. So all of this is isn't that cheap. Even more so in a for example medical domain where you have very expensive CT images of patients and then you need to obtain them and you need to convince the patients to release their data and someone needs to label it. So it's very costly to obtain lots of training data. Now what we want to do is we want to, for all of these tasks, we ideally want to build neural networks, deep neural networks because we know they're super accurate but they are only super accurate if you have lots of training data. So that conflicts with the fact that we might not have so much training data for these tasks. So the proposed solution here is what's called visual task adaptation and it's the following. Imagine you have lots and lots of what's called here upstream data. And upstream data, what they mean is data that is similar to the data here but not exactly the same but you have lots of it. And the example given is ImageNet. So imagine this here to be ImageNet. ImageNet is a data set with over a million images. All of them are labeled into one of a thousand classes and so you can build a very good model for ImageNet to predict the ImageNet class. And you can get very accurate, you have lots of data. Cool. So you build this model but now what you want to do is you want to use this what's here called an adaptation algorithm. And you want to use that model that you trained on ImageNet data and kind of change it just a bit. So you start from the model you have that works on ImageNet and with the few training data you have here on the right side and the author has actually standardized this in the benchmark to 1k samples. So you only have a thousand training samples compared to the millions that you potentially need. You have a thousand samples and you adapt your model to these tasks. So you train the model on ImageNet and you adapt it to predict whether or not there's a cat or a dog and you adapt it to segment these images and you adapt it to predict the depth of points. So you can consider this kind of as a pre-training thing. So you pre-train your model on ImageNet and then you adapt it to these others. That's what's called task adaptation. It's not exactly pre-training in the classic sense because pre-training in the classic sense means basically that you retain the same model but here it's a bit different. So in stage one you train a deep neural network on lots of training data. A deep neural network here this might be you know you have a bunch of layers layer layer layer layer layer and then here you have a thousand you classify into a thousand classes. This is your model. Then in stage two over here you adapt this model and what it ultimately means is you take for example this part here up until the second to last layer transfer it over put it here right bam bam bam bam bam you retain the weights you keep the weights but then you add just one or two new layers and classify your new tasks. This could be is it a cat or is it a dog? Then you train you can either elect to only train the green part here or you can train the whole thing. The second thing is called fine-tuning. The author is mostly elect to do fine-tuning in this work so you carry over the weights and you add a new head and then you train the entire thing with the 1000 samples that you have for this task and then you the kind of the goal is to get as good as possible on that one task where you only have a thousand samples. If your pre-training was good so if your stage one was good then you would expect that stage two would profit a lot from this pre-training which basically means that even though you only have a thousand samples you can reach accuracies that would usually only be possible with much more samples. That's the idea behind it. This is what's called visual task adaptation. The authors propose a benchmark for this. A benchmark for this part, for the adaptation algorithm. The adaptation algorithm they propose as a baseline is train on ImageNet and then fine-tune. That's an adaptation algorithm. They propose a score for this. If you come up with a better adaptation algorithm for example you could say no I'm going to train on YouTube data and then do fine-tune that and then maybe you'd reach better accuracies in these tasks over here and then your score would be higher. It's kind of a benchmark to compare adaptation algorithms. Here your benchmark score and this is conditioned on n, the number of samples that you have in the in the layer two tasks and this here is standardized to 1000 in their case. The score of an adaptation algorithm A is the following. It's the expectation over this is kind of an error measure and you can think of it basically as a test set classification error on the layer two tasks. Of that adaptation algorithm if given the data set of a layer two tasks of n samples and the layer two tasks here comes from a distribution of layer two tasks. What does it mean? This distribution of layer two tasks they imagine, they show this in this picture, they imagine the visual tasks like on this big landscape of visual tasks right here and what they ideally want to do is they want to sample a task here and this task corresponds to classifying these dog images and very close to it could be classifying bird images but then very far away could be a task of counting and depth estimation and so on. They imagine all the visual tasks have some kind of some sort of distribution. So what happens is you sample one of those visual tasks for each element in this expectation. You sample one of them, you build the data set with a thousand samples right you put it through your adaptation algorithms or your adaptation algorithm for example your pre-trained image net you adapt it to that task with a thousand samples and then you compute your error metric on that. Now if you do this over the whole distribution you get an expectation of this error metric in all the visual tasks and that will be your score. What does it mean in practice? I mean in practice you don't have this distribution right in practice you have a list so like list here is a list of tasks right there's this task this task this task this task there's whatever the pets task and then there is the aerial then there is the counting right you have a list of tasks and what is it like this stuff and this expectation ultimately right stage one train a model M stage two for each of these tasks adapt the model M or fine-tune your model M on these tasks then for each task get an error rate error rate one task two gives you error rate two tasks three gives you error rate three then jump simply one over n sum them up so take the take the average error rate of the of the of all of the tasks and that's your score that's kind of my first criticism of this thing like this this all just seems like super mathematized with like oh we imagine all of these tasks being in some distribution somewhere like that there is a distribution of tasks and we have an expectation over the distribution now like why just say here's a bunch of tasks right adapt your model to each one of them get the average error rate done that's your score that would have been first of all much easier and second of all they never actually care to characterize this distribution like if if they were to actually rigorously characterize this distribution of visual tasks I would agree that this formulation makes sense but all they say basically all they say is tasks that a human can solve from visual input alone and they give a bunch of examples of you know a good task would be the following right so label one one zero zero one right and you probably figured it out the task is is it a square or is it a triangle right that's a does a visual task in the classic sense human can solve it from visual input alone then the following task wouldn't be as easy labels one zero zero one so the task I had in mind was is there and spelling is the spelling of the shape over here does it contain an a so square contains an a circle doesn't line doesn't but triangle contains an a right so therefore this you kind of need world knowledge and you can't just solve it from visual input alone right especially not you can't generalize to new new shapes if you if you just from visually put so um they and they say appendix B they validate this right they validate that humans can solve it but I I actually disagree with this because just because humans can solve a task just from visual input doesn't mean that they don't use world knowledge in it like in this whatever pets example here right humans know how cats and dogs look anatomically right how they look from the side and from the back and so on even if they haven't seen it in a picture they they know how they behave and so on what is kind of realistic setting for a cat and a dog to be in so all of this it seems kind of a bit shady and the reason I'm saying this is if you make this distribution formulation you also you have to give a rigorous definition and because if a new task arrives now like one that's not in your list like never been before here in the world like new task arrives how do we know whether or not we should include it in the list or not right how do we know whether it's part of this distribution or not it just seems very very shaky so that being said they do give this list and this list has 19 tasks that's down here so there are 19 tasks their categorized as natural which means natural images these these yeah the examples here are pets flowers images house numbers and so on specialized images are for example images with that you special equipment for example medical images and then structured means where that's down here structured means that the model needs come to comprehend the structure of a scene so they give an example of object counting or 3d depth prediction I mean that's that's fair enough they have these 19 tasks but and they show kind of the tasks down here here's a list of tasks and kind of their baseline method on it but but why for me like the question is why exactly these tasks if they don't specify this distribution why these tasks and they don't really like they do some they do a lot of experimentation actually an investigation but what's kind of missing for me is to show that these tasks first of all are kind of internally consistent in that they're really visual tasks and second of all that they kind of cover this distribution or they represent this entire distribution that they're trying to model and it seems to me unclear why exactly these tasks why they left others out and included these ones in all fairness probably they simply took the ones that that they could get their hands on but still I feel that this is very shaky and that might that might lead to the benchmark not being adapted very widely but alright enough with the criticism let's go further in this so they do present this kind of baseline experiments and they they pre train always on image net and then they they they fine-tune on these layer two tasks and the way they pre train here is listed here for example so if they pre train a generative model it actually performs worse than if they just train from scratch for the layer two tasks on the thousand samples right self supervised is kind of a pre training method where if you have an image you do something like you rotate it to the right or to the left and then you ask a model some sort of a discriminator did it did I turn it to the right or to the left like zero is to the right left and one is to the right so you this is called self supervised you don't need labels for this right and it kind of works well semi supervised has some of the labels and supervised has is like image net with full labels and you kind of see unsurprisingly that the more information you have the the better you are going to be in all of these these kind of tasks interestingly the generative pre training works the worst worse than even from scratch training so that's kind of a sort of special what what I do really appreciate about this this investigation here is that they investigate a lot of variants of this of this benchmark and they come to the conclusion I think this encapsulated here one for example we find two models using 16 Google Cloud TPU hardware accelerators now that's expensive right but they say we conduct additional experiments to assess whether our result can be reproduced with a more basic hardware setup we evaluate on all the tasks using a single Nvidia P100 GPU with a thousand steps 64 images per mini batch right so they verify that you can do this benchmark you can take part in this benchmark even if you don't have much time or money or hardware right that's why for example they limit they limit the number of examples in the layer two tasks to a thousand they do investigate that this correlates with your performance if you were to include the full data sets of the layer two tasks so if you just include a thousand examples that correlates well they do investigate they do investigate whether you can put it on a single GPU they do investigate if you only run it for a thousand steps here you see this experiment you have to run it for a thousand steps basically and you're almost at the level if as if you were to run it for 50,000 steps so there's a lot of work to that goes into making sure that everybody can kind of participate in this benchmark and that I appreciate this a lot and there is actually code available so if you go to github and you just search for task adaptation actually I had it open before but I don't know so you go to github and you go to Google research and search for task adaptation to adaptation you'll you'll find it there is code that downloads all of the data sets for you prepares them and there is a script that runs your layer one model so you need to provide it a layer one model but then there is a script that that runs it on all of the different layer two tasks and at the end calculates your benchmark for you so that's pretty neat and I would encourage you if you have a good idea for a pre training or for a adaptation algorithm take part in the benchmark I suspect there will be a leaderboard kind of online leaderboard coming out at some point otherwise you simply can report the number in your papers and I hope you are going to be successful at that all right so that was it for me have lots of fun and bye bye
 Hi there. Today we're looking at the visual task adaptation benchmark by a
 list of authors that's way too long to read out all from Google Brain. So what
 is this paper? This paper cares about a new benchmark that is abbreviated VTab
 and VTab is a benchmark for a task called visual task adaptation. So a
 benchmark, the meaning of a benchmark is it's kind of a number that you achieve
 with a model and whoever has the highest number is the best at this task.
 So the benchmark kind of standardizes how you evaluate models and the
 model is here. They do visual task adaptation. So what is visual task
 adaptation? So this is visual task adaptation. It's kind of illustrated in
 this figure. Imagine you have a bunch of what are called visual tasks and a
 visual task, and this is the right side here, a visual task is anything that
 can be solved from just visual input. So basically given a picture or many
 pictures and you ask kind of a question about it, if that question can be
 answered by just looking at the picture then that's called a visual task. For
 example in this data set you might be asked whether a picture contains a dog
 or a cat. In this data set you might be asked to outline where the objects
 are. So here the plane, you might be able to segment or you might be able to point
 out where buildings are in the images. Right here, here, there's no building
 here. So there's varieties of tasks that are possible. Or in the
 bottom domain you might be asked which one of the two red dots here is closer
 to the observer in 3D space. Or you might be asked in this picture please count
 the number of gray boxes. So there's a bunch of, all of these count as visual
 tasks. Now the setting that the authors imagine here is there are many of these
 visual tasks in the world for which there isn't much training data. Imagine
 something like this. These are aerial images so you kind of need a satellite
 or a plane to obtain them and then you need to label them. So all of this is
 isn't that cheap. Even more so in a for example medical domain where you have
 very expensive CT images of patients and then you need to obtain them and you
 need to convince the patients to release their data and someone needs to label it.
 So it's very costly to obtain lots of training data. Now what we want to do is
 we want to, for all of these tasks, we ideally want to build neural networks,
 deep neural networks because we know they're super accurate but they are only
 super accurate if you have lots of training data. So that conflicts with the
 fact that we might not have so much training data for these tasks. So the
 proposed solution here is what's called visual task adaptation and it's the
 following. Imagine you have lots and lots of what's called here upstream data.
 And upstream data, what they mean is data that is similar to the data here but not
 exactly the same but you have lots of it. And the example given is ImageNet.
 So imagine this here to be ImageNet. ImageNet is a data set with over a
 million images. All of them are labeled into one of a thousand classes and so
 you can build a very good model for ImageNet to predict the ImageNet class.
 And you can get very accurate, you have lots of data. Cool. So you build
 this model but now what you want to do is you want to use this what's here
 called an adaptation algorithm. And you want to use that model that you trained
 on ImageNet data and kind of change it just a bit. So you start from the model
 you have that works on ImageNet and with the few training data you have here on
 the right side and the author has actually standardized this in the
 benchmark to 1k samples. So you only have a thousand training samples
 compared to the millions that you potentially need. You have a thousand
 samples and you adapt your model to these tasks. So you train the model
 on ImageNet and you adapt it to predict whether or not there's a cat or a dog
 and you adapt it to segment these images and you adapt it to predict the depth of
 points. So you can consider this kind of as a pre-training thing. So you pre-train
 your model on ImageNet and then you adapt it to these others. That's what's
 called task adaptation. It's not exactly pre-training in the classic sense
 because pre-training in the classic sense means basically that you
 retain the same model but here it's a bit different. So in stage one you
 train a deep neural network on lots of training data. A deep neural network
 here this might be you know you have a bunch of layers layer layer layer layer
 layer and then here you have a thousand you classify into a thousand classes.
 This is your model. Then in stage two over here you adapt this model
 and what it ultimately means is you take for example this part here up until the
 second to last layer transfer it over put it here right bam bam bam bam bam
 you retain the weights you keep the weights but then you add just one or two
 new layers and classify your new tasks. This could be is it a cat or is it a dog?
 Then you train you can either elect to only train the green part here or
 you can train the whole thing. The second thing is called fine-tuning.
 The author is mostly elect to do fine-tuning in this work so you carry
 over the weights and you add a new head and then you train the entire thing with
 the 1000 samples that you have for this task and then you the kind of the goal
 is to get as good as possible on that one task where you only have a thousand
 samples. If your pre-training was good so if your stage one was good then you
 would expect that stage two would profit a lot from this pre-training which
 basically means that even though you only have a thousand samples you can
 reach accuracies that would usually only be possible with much more samples.
 That's the idea behind it. This is what's called visual task
 adaptation. The authors propose a benchmark for this. A benchmark for
 this part, for the adaptation algorithm. The adaptation algorithm they
 propose as a baseline is train on ImageNet and then fine-tune. That's an
 adaptation algorithm. They propose a score for this. If you come up with a
 better adaptation algorithm for example you could say no I'm going to train
 on YouTube data and then do fine-tune that and then maybe you'd reach
 better accuracies in these tasks over here and then your
 score would be higher. It's kind of a benchmark to compare adaptation
 algorithms. Here your benchmark score and this is conditioned on n, the number of
 samples that you have in the in the layer two tasks and this here is
 standardized to 1000 in their case. The score of an adaptation algorithm A is
 the following. It's the expectation over this is kind of an error
 measure and you can think of it basically as a test set
 classification error on the layer two tasks. Of that adaptation algorithm if
 given the data set of a layer two tasks of n samples and the layer two tasks here
 comes from a distribution of layer two tasks. What does it mean? This
 distribution of layer two tasks they imagine, they show this in this picture,
 they imagine the visual tasks like on this big landscape of visual
 tasks right here and what they ideally want to do is they want to sample a
 task here and this task corresponds to classifying these dog images and very
 close to it could be classifying bird images but then very far away could be a
 task of counting and depth estimation and so on. They imagine all the visual
 tasks have some kind of some sort of distribution. So what happens is
 you sample one of those visual tasks for each element in this
 expectation. You sample one of them, you build the data set with a thousand
 samples right you put it through your adaptation algorithms or your
 adaptation algorithm for example your pre-trained image net you adapt it to
 that task with a thousand samples and then you compute your error metric on
 that. Now if you do this over the whole distribution you get an expectation of
 this error metric in all the visual tasks and that will be your score.
 What does it mean in practice? I mean in practice you don't have this
 distribution right in practice you have a list so like list here is a list of
 tasks right there's this task this task this task this task there's whatever the
 pets task and then there is the aerial then there is the counting right you
 have a list of tasks and what is it like this stuff and this expectation
 ultimately right stage one train a model M stage two for each of these tasks
 adapt the model M or fine-tune your model M on these tasks then for each
 task get an error rate error rate one task two gives you error rate two tasks
 three gives you error rate three then jump simply one over n sum them up so
 take the take the average error rate of the of the of all of the tasks and
 that's your score that's kind of my first criticism of this thing like this
 this all just seems like super mathematized with like oh we imagine all
 of these tasks being in some distribution somewhere like that there
 is a distribution of tasks and we have an expectation over the distribution
 now like why just say here's a bunch of tasks right adapt your model to each one
 of them get the average error rate done that's your score that would have been
 first of all much easier and second of all they never actually care to
 characterize this distribution like if if they were to actually rigorously
 characterize this distribution of visual tasks I would agree that this
 formulation makes sense but all they say basically all they say is tasks that a
 human can solve from visual input alone and they give a bunch of examples of
 you know a good task would be the following right so label one one zero
 zero one right and you probably figured it out the task is is it a square or is
 it a triangle right that's a does a visual task in the classic sense human
 can solve it from visual input alone then the following task wouldn't be as
 easy labels one zero zero one so the task I had in mind was is there and
 spelling is the spelling of the shape over here does it contain an a so square
 contains an a circle doesn't line doesn't but triangle contains an a right
 so therefore this you kind of need world knowledge and you can't just solve it
 from visual input alone right especially not you can't generalize to new new
 shapes if you if you just from visually put so um they and they say appendix B
 they validate this right they validate that humans can solve it but I I
 actually disagree with this because just because humans can solve a task just
 from visual input doesn't mean that they don't use world knowledge in it like in
 this whatever pets example here right humans know how cats and dogs look
 anatomically right how they look from the side and from the back and so on
 even if they haven't seen it in a picture they they know how they behave
 and so on what is kind of realistic setting for a cat and a dog to be in so
 all of this it seems kind of a bit shady and the reason I'm saying this is if
 you make this distribution formulation you also you have to give a rigorous
 definition and because if a new task arrives now like one that's not in your
 list like never been before here in the world like new task arrives how do we
 know whether or not we should include it in the list or not right how do we know
 whether it's part of this distribution or not it just seems very very shaky so
 that being said they do give this list and this list has 19 tasks that's down
 here so there are 19 tasks their categorized as natural which means
 natural images these these yeah the examples here are pets flowers images
 house numbers and so on specialized images are for example images with that
 you special equipment for example medical images and then structured means
 where that's down here structured means that the model needs come to comprehend
 the structure of a scene so they give an example of object counting or 3d depth
 prediction I mean that's that's fair enough they have these 19 tasks but and
 they show kind of the tasks down here here's a list of tasks and kind of their
 baseline method on it but but why for me like the question is why exactly these
 tasks if they don't specify this distribution why these tasks and they
 don't really like they do some they do a lot of experimentation actually an
 investigation but what's kind of missing for me is to show that these tasks first
 of all are kind of internally consistent in that they're really visual tasks and
 second of all that they kind of cover this distribution or they represent
 this entire distribution that they're trying to model and it seems to me
 unclear why exactly these tasks why they left others out and included these ones
 in all fairness probably they simply took the ones that that they could get
 their hands on but still I feel that this is very shaky and that might that
 might lead to the benchmark not being adapted very widely but alright enough
 with the criticism let's go further in this so they do present this kind of
 baseline experiments and they they pre train always on image net and then they
 they they fine-tune on these layer two tasks and the way they pre train here is
 listed here for example so if they pre train a generative model it actually
 performs worse than if they just train from scratch for the layer two tasks on
 the thousand samples right self supervised is kind of a pre training
 method where if you have an image you do something like you rotate it to the
 right or to the left and then you ask a model some sort of a discriminator did
 it did I turn it to the right or to the left like zero is to the right left and
 one is to the right so you this is called self supervised you don't need
 labels for this right and it kind of works well semi supervised has some of
 the labels and supervised has is like image net with full labels and you kind
 of see unsurprisingly that the more information you have the the better you
 are going to be in all of these these kind of tasks interestingly the
 generative pre training works the worst worse than even from scratch training so
 that's kind of a sort of special what what I do really appreciate about this
 this investigation here is that they investigate a lot of variants of this
 of this benchmark and they come to the conclusion I think this encapsulated
 here one for example we find two models using 16 Google Cloud TPU hardware
 accelerators now that's expensive right but they say we conduct additional
 experiments to assess whether our result can be reproduced with a more basic
 hardware setup we evaluate on all the tasks using a single Nvidia P100 GPU
 with a thousand steps 64 images per mini batch right so they verify that you can
 do this benchmark you can take part in this benchmark even if you don't have
 much time or money or hardware right that's why for example they limit they
 limit the number of examples in the layer two tasks to a thousand they do
 investigate that this correlates with your performance if you were to include
 the full data sets of the layer two tasks so if you just include a thousand
 examples that correlates well they do investigate they do investigate whether
 you can put it on a single GPU they do investigate if you only run it for a
 thousand steps here you see this experiment you have to run it for a
 thousand steps basically and you're almost at the level if as if you were to
 run it for 50,000 steps so there's a lot of work to that goes into making sure
 that everybody can kind of participate in this benchmark and that I appreciate
 this a lot and there is actually code available so if you go to github and
 you just search for task adaptation actually I had it open before but I don't
 know so you go to github and you go to Google research and search for task
 adaptation to adaptation you'll you'll find it there is code that downloads all
 of the data sets for you prepares them and there is a script that runs your
 layer one model so you need to provide it a layer one model but then there is
 a script that that runs it on all of the different layer two tasks and at the end
 calculates your benchmark for you so that's pretty neat and I would encourage
 you if you have a good idea for a pre training or for a adaptation algorithm
 take part in the benchmark I suspect there will be a leaderboard kind of
 online leaderboard coming out at some point otherwise you simply can report
 the number in your papers and I hope you are going to be successful at that all
 right so that was it for me have lots of fun and bye bye