Searching for better (Full) ImageNet ViT Baselines

<p>
<code>timm</code> 1.0.9 was just released. Included are a few new ImageNet-12k and ImageNet-12k -&gt; ImageNet-1k weights in my <a href="https://huggingface.co/collections/timm/searching-for-better-vit-baselines-663eb74f64f847d2f35a9c19">Searching for Better ViT Baselines</a> series. </p>
<div class="max-w-full overflow-auto">
<table>
<thead><tr>
<th>model</th>
<th>top1</th>
<th>top5</th>
<th>param_count</th>
<th>img_size</th>
</tr>
</thead><tbody><tr>
<td><a href="https://huggingface.co/timm/vit_mediumd_patch16_reg4_gap_384.sbb2_e200_in12k_ft_in1k">vit_mediumd_patch16_reg4_gap_384.sbb2_e200_in12k_ft_in1k</a></td>
<td>87.438</td>
<td>98.256</td>
<td>64.11</td>
<td>384</td>
</tr>
<tr>
<td><a href="https://huggingface.co/timm/vit_mediumd_patch16_reg4_gap_256.sbb2_e200_in12k_ft_in1k">vit_mediumd_patch16_reg4_gap_256.sbb2_e200_in12k_ft_in1k</a></td>
<td>86.608</td>
<td>97.934</td>
<td>64.11</td>
<td>256</td>
</tr>
<tr>
<td><a href="https://huggingface.co/timm/vit_betwixt_patch16_reg4_gap_384.sbb2_e200_in12k_ft_in1k">vit_betwixt_patch16_reg4_gap_384.sbb2_e200_in12k_ft_in1k</a></td>
<td>86.594</td>
<td>98.02</td>
<td>60.4</td>
<td>384</td>
</tr>
<tr>
<td><a href="https://huggingface.co/timm/vit_betwixt_patch16_reg4_gap_256.sbb2_e200_in12k_ft_in1k">vit_betwixt_patch16_reg4_gap_256.sbb2_e200_in12k_ft_in1k</a></td>
<td>85.734</td>
<td>97.61</td>
<td>60.4</td>
<td>256</td>
</tr>
</tbody>
</table>
</div>
<p>I'd like to highlight these models as they're on the pareto front for ImageNet-12k / ImageNet-22k models. It is interesting to look at models with comparable ImageNet-22k fine-tunes to see how competitive (near) vanilla ViTs are with other architectures. With optimized attention kernels enabled (default in <code>timm</code>), they are well ahead of Swin and holding up just fine relative to ConvNeXt, etc. </p>
<p>Something else worth pointing out, the <code>deit3</code> model weights are quite remarkable and underappreciated set of weights. The upper end of my <code>sbb</code> weights are matching <code>deit3</code> at equivalent compute -- it's also a great recipe. Though, one of my goals with <code>sbb</code> recipes was to allow easier fine-tuning. In opting for a less exotic augmentation scheme, sticking with AdamW, and sacrificing some top-1 (higher weight-decay), I feel that was achieved. Through several fine-tune trials I've found the sbb ViT weights to be easier to fit to other, especially smaller datasets (Oxford Pets, RESISC, etc) w/ short runs.</p>
<p>NOTE: all throughput measurements were done on an RTX 4090, AMP /w torch.compile() enabled, PyTorch 2.4, Cuda 12.4.</p>
<p><a href="https://cdn-uploads.huggingface.co/production/uploads/604a5184dca2c7ac7508b849/Cc_ERcZ5yRZCl9PjlOKSv.png" rel="nofollow"><img alt="image/png" src="https://cdn-uploads.huggingface.co/production/uploads/604a5184dca2c7ac7508b849/Cc_ERcZ5yRZCl9PjlOKSv.png"/></a></p>
<p><strong>Bold rows</strong>: Pareto frontier models</p>
<div class="max-w-full overflow-auto">
<table>
<thead><tr>
<th>model</th>
<th>img_size</th>
<th>samples_per_sec</th>
<th>top1</th>
<th>top5</th>
<th>param_count</th>
</tr>
</thead><tbody><tr>
<td><strong><a href="http://hf.co/timm/deit3_base_patch16_224.fb_in22k_ft_in1k" rel="nofollow">deit3_base_patch16_224.fb_in22k_ft_in1k</a></strong></td>
<td><strong>224</strong></td>
<td><strong>3326.85</strong></td>
<td><strong>85.73</strong></td>
<td><strong>97.75</strong></td>
<td><strong>86.59</strong></td>
</tr>
<tr>
<td><strong><a href="http://hf.co/timm/vit_betwixt_patch16_reg4_gap_256.sbb2_e200_in12k_ft_in1k" rel="nofollow">vit_betwixt_patch16_reg4_gap_256.sbb2_e200_in12k_ft_in1k</a></strong></td>
<td><strong>256</strong></td>
<td><strong>3302.28</strong></td>
<td><strong>85.73</strong></td>
<td><strong>97.61</strong></td>
<td><strong>60.40</strong></td>
</tr>
<tr>
<td><a href="http://hf.co/timm/vit_base_patch16_224.augreg2_in21k_ft_in1k" rel="nofollow">vit_base_patch16_224.augreg2_in21k_ft_in1k</a></td>
<td>224</td>
<td>3278.15</td>
<td>85.11</td>
<td>97.54</td>
<td>86.57</td>
</tr>
<tr>
<td><a href="http://hf.co/timm/vit_base_patch16_224.augreg_in21k_ft_in1k" rel="nofollow">vit_base_patch16_224.augreg_in21k_ft_in1k</a></td>
<td>224</td>
<td>3274.99</td>
<td>84.53</td>
<td>97.30</td>
<td>86.57</td>
</tr>
<tr>
<td><strong><a href="http://hf.co/timm/vit_mediumd_patch16_reg4_gap_256.sbb2_e200_in12k_ft_in1k" rel="nofollow">vit_mediumd_patch16_reg4_gap_256.sbb2_e200_in12k_ft_in1k</a></strong></td>
<td><strong>256</strong></td>
<td><strong>2761.64</strong></td>
<td><strong>86.60</strong></td>
<td><strong>97.94</strong></td>
<td><strong>64.11</strong></td>
</tr>
<tr>
<td><strong><a href="http://hf.co/timm/caformer_m36.sail_in22k_ft_in1k" rel="nofollow">caformer_m36.sail_in22k_ft_in1k</a></strong></td>
<td><strong>224</strong></td>
<td><strong>2345.11</strong></td>
<td><strong>86.61</strong></td>
<td><strong>98.04</strong></td>
<td><strong>56.20</strong></td>
</tr>
<tr>
<td><a href="http://hf.co/timm/convformer_m36.sail_in22k_ft_in1k" rel="nofollow">convformer_m36.sail_in22k_ft_in1k</a></td>
<td>224</td>
<td>2319.68</td>
<td>86.15</td>
<td>97.85</td>
<td>57.05</td>
</tr>
<tr>
<td><a href="http://hf.co/timm/swin_base_patch4_window7_224.ms_in22k_ft_in1k" rel="nofollow">swin_base_patch4_window7_224.ms_in22k_ft_in1k</a></td>
<td>224</td>
<td>2176.48</td>
<td>85.27</td>
<td>97.57</td>
<td>87.77</td>
</tr>
<tr>
<td><a href="http://hf.co/timm/regnety_160.sw_in12k_ft_in1k" rel="nofollow">regnety_160.sw_in12k_ft_in1k</a></td>
<td>224</td>
<td>2098.25</td>
<td>85.59</td>
<td>97.67</td>
<td>83.59</td>
</tr>
<tr>
<td><a href="http://hf.co/timm/coatnet_2_rw_224.sw_in12k_ft_in1k" rel="nofollow">coatnet_2_rw_224.sw_in12k_ft_in1k</a></td>
<td>224</td>
<td>1753.63</td>
<td>86.58</td>
<td>97.90</td>
<td>73.87</td>
</tr>
<tr>
<td><strong><a href="http://hf.co/timm/vit_betwixt_patch16_reg4_gap_384.sbb2_e200_in12k_ft_in1k" rel="nofollow">vit_betwixt_patch16_reg4_gap_384.sbb2_e200_in12k_ft_in1k</a></strong></td>
<td><strong>384</strong></td>
<td><strong>1467.64</strong></td>
<td><strong>86.60</strong></td>
<td><strong>98.02</strong></td>
<td><strong>60.60</strong></td>
</tr>
<tr>
<td><strong><a href="http://hf.co/timm/convnext_large.fb_in22k_ft_in1k" rel="nofollow">convnext_large.fb_in22k_ft_in1k</a></strong></td>
<td><strong>224</strong></td>
<td><strong>1457.60</strong></td>
<td><strong>86.61</strong></td>
<td><strong>98.04</strong></td>
<td><strong>197.77</strong></td>
</tr>
<tr>
<td><a href="http://hf.co/timm/convnext_small.in12k_ft_in1k_384" rel="nofollow">convnext_small.in12k_ft_in1k_384</a></td>
<td>384</td>
<td>1350.43</td>
<td>86.19</td>
<td>97.92</td>
<td>50.22</td>
</tr>
<tr>
<td><a href="http://hf.co/timm/seresnextaa101d_32x8d.sw_in12k_ft_in1k_288" rel="nofollow">seresnextaa101d_32x8d.sw_in12k_ft_in1k_288</a></td>
<td>288</td>
<td>1297.79</td>
<td>86.54</td>
<td>98.09</td>
<td>93.59</td>
</tr>
<tr>
<td><a href="http://hf.co/timm/regnety_160.sw_in12k_ft_in1k" rel="nofollow">regnety_160.sw_in12k_ft_in1k</a></td>
<td>288</td>
<td>1260.01</td>
<td>86.03</td>
<td>97.83</td>
<td>83.59</td>
</tr>
<tr>
<td><a href="http://hf.co/timm/swin_large_patch4_window7_224.ms_in22k_ft_in1k" rel="nofollow">swin_large_patch4_window7_224.ms_in22k_ft_in1k</a></td>
<td>224</td>
<td>1243.73</td>
<td>86.33</td>
<td>97.88</td>
<td>196.53</td>
</tr>
<tr>
<td><strong><a href="http://hf.co/timm/vit_mediumd_patch16_reg4_gap_384.sbb2_e200_in12k_ft_in1k" rel="nofollow">vit_mediumd_patch16_reg4_gap_384.sbb2_e200_in12k_ft_in1k</a></strong></td>
<td><strong>384</strong></td>
<td><strong>1214.59</strong></td>
<td><strong>87.44</strong></td>
<td><strong>98.26</strong></td>
<td><strong>64.27</strong></td>
</tr>
<tr>
<td><a href="http://hf.co/timm/deit3_base_patch16_384.fb_in22k_ft_in1k" rel="nofollow">deit3_base_patch16_384.fb_in22k_ft_in1k</a></td>
<td>384</td>
<td>1098.30</td>
<td>86.74</td>
<td>98.11</td>
<td>86.88</td>
</tr>
<tr>
<td><a href="http://hf.co/timm/deit3_large_patch16_224.fb_in22k_ft_in1k" rel="nofollow">deit3_large_patch16_224.fb_in22k_ft_in1k</a></td>
<td>224</td>
<td>1042.41</td>
<td>86.99</td>
<td>98.24</td>
<td>304.37</td>
</tr>
<tr>
<td><a href="http://hf.co/timm/vit_large_patch16_224.augreg_in21k_ft_in1k" rel="nofollow">vit_large_patch16_224.augreg_in21k_ft_in1k</a></td>
<td>224</td>
<td>1041.47</td>
<td>85.85</td>
<td>97.83</td>
<td>304.33</td>
</tr>
<tr>
<td><a href="http://hf.co/timm/seresnextaa101d_32x8d.sw_in12k_ft_in1k_288" rel="nofollow">seresnextaa101d_32x8d.sw_in12k_ft_in1k_288</a></td>
<td>320</td>
<td>1035.83</td>
<td>86.72</td>
<td>98.18</td>
<td>93.59</td>
</tr>
<tr>
<td><a href="http://hf.co/timm/convnext_xlarge.fb_in22k_ft_in1k" rel="nofollow">convnext_xlarge.fb_in22k_ft_in1k</a></td>
<td>224</td>
<td>921.30</td>
<td>86.97</td>
<td>98.20</td>
<td>350.20</td>
</tr>
<tr>
<td><strong><a href="http://hf.co/timm/convnext_large.fb_in22k_ft_in1k" rel="nofollow">convnext_large.fb_in22k_ft_in1k</a></strong></td>
<td><strong>288</strong></td>
<td><strong>881.61</strong></td>
<td><strong>87.01</strong></td>
<td><strong>98.21</strong></td>
<td><strong>197.77</strong></td>
</tr>
<tr>
<td><strong><a href="http://hf.co/timm/caformer_m36.sail_in22k_ft_in1k_384" rel="nofollow">caformer_m36.sail_in22k_ft_in1k_384</a></strong></td>
<td><strong>384</strong></td>
<td><strong>794.45</strong></td>
<td><strong>87.47</strong></td>
<td><strong>98.31</strong></td>
<td><strong>56.20</strong></td>
</tr>
<tr>
<td><a href="http://hf.co/timm/efficientnet_b5.sw_in12k_ft_in1k" rel="nofollow">efficientnet_b5.sw_in12k_ft_in1k</a></td>
<td>448</td>
<td>729.86</td>
<td>85.89</td>
<td>97.74</td>
<td>30.39</td>
</tr>
<tr>
<td><a href="http://hf.co/timm/convnext_xlarge.fb_in22k_ft_in1k" rel="nofollow">convnext_xlarge.fb_in22k_ft_in1k</a></td>
<td>288</td>
<td>559.14</td>
<td>87.37</td>
<td>98.33</td>
<td>350.20</td>
</tr>
<tr>
<td><a href="http://hf.co/timm/swin_base_patch4_window12_384.ms_in22k_ft_in1k" rel="nofollow">swin_base_patch4_window12_384.ms_in22k_ft_in1k</a></td>
<td>384</td>
<td>522.86</td>
<td>86.44</td>
<td>98.07</td>
<td>87.90</td>
</tr>
<tr>
<td><a href="http://hf.co/timm/convnext_large.fb_in22k_ft_in1k_384" rel="nofollow">convnext_large.fb_in22k_ft_in1k_384</a></td>
<td>384</td>
<td>500.83</td>
<td>87.46</td>
<td>98.38</td>
<td>197.77</td>
</tr>
<tr>
<td><strong><a href="http://hf.co/timm/maxxvitv2_rmlp_base_rw_384.sw_in12k_ft_in1k" rel="nofollow">maxxvitv2_rmlp_base_rw_384.sw_in12k_ft_in1k</a></strong></td>
<td><strong>384</strong></td>
<td><strong>456.17</strong></td>
<td><strong>87.48</strong></td>
<td><strong>98.37</strong></td>
<td><strong>116.09</strong></td>
</tr>
<tr>
<td><a href="http://hf.co/timm/coatnet_rmlp_2_rw_384.sw_in12k_ft_in1k" rel="nofollow">coatnet_rmlp_2_rw_384.sw_in12k_ft_in1k</a></td>
<td>384</td>
<td>404.42</td>
<td>87.40</td>
<td>98.31</td>
<td>73.88</td>
</tr>
<tr>
<td><a href="http://hf.co/timm/seresnextaa201d_32x8d.sw_in12k_ft_in1k_384" rel="nofollow">seresnextaa201d_32x8d.sw_in12k_ft_in1k_384</a></td>
<td>384</td>
<td>365.65</td>
<td>87.31</td>
<td>98.33</td>
<td>149.39</td>
</tr>
<tr>
<td><strong><a href="http://hf.co/timm/deit3_large_patch16_384.fb_in22k_ft_in1k" rel="nofollow">deit3_large_patch16_384.fb_in22k_ft_in1k</a></strong></td>
<td><strong>384</strong></td>
<td><strong>342.41</strong></td>
<td><strong>87.73</strong></td>
<td><strong>98.51</strong></td>
<td><strong>304.76</strong></td>
</tr>
<tr>
<td><a href="http://hf.co/timm/vit_large_patch16_384.augreg_in21k_ft_in1k" rel="nofollow">vit_large_patch16_384.augreg_in21k_ft_in1k</a></td>
<td>384</td>
<td>338.21</td>
<td>87.09</td>
<td>98.31</td>
<td>304.72</td>
</tr>
<tr>
<td><a href="http://hf.co/timm/swin_large_patch4_window12_384.ms_in22k_ft_in1k" rel="nofollow">swin_large_patch4_window12_384.ms_in22k_ft_in1k</a></td>
<td>384</td>
<td>315.38</td>
<td>87.14</td>
<td>98.23</td>
<td>196.74</td>
</tr>
<tr>
<td><a href="http://hf.co/timm/swinv2_base_window12to24_192to384.ms_in22k_ft_in1k" rel="nofollow">swinv2_base_window12to24_192to384.ms_in22k_ft_in1k</a></td>
<td>384</td>
<td>297.03</td>
<td>87.14</td>
<td>98.23</td>
<td>87.92</td>
</tr>
<tr>
<td><a href="http://hf.co/timm/swinv2_large_window12to24_192to384.ms_in22k_ft_in1k" rel="nofollow">swinv2_large_window12to24_192to384.ms_in22k_ft_in1k</a></td>
<td>384</td>
<td>186.30</td>
<td>87.47</td>
<td>98.26</td>
<td>196.74</td>
</tr>
</tbody>
</table>
</div>
