A Look at The First Place Solution of a Dermatology Classification Kaggle Competition

1.1 Inverted Residuals and Linear Bottlenecks (MobileNetV2)

MobileNetV2 [2] introduced a new type of neural network architectural building block often referred to as "MBConv". The two big innovations here are inverted residuals and linear bottlenecks.

First to understand inverted residuals, let's take a look at the basic residual block (also see my post on ResNet) shown in Listing 1. Two notable parts of the basic residual block are the fact that we reduce the number of channels to squeeze in the first layer, and then grow the number of channels to expand in the last layer. The squeeze operation is often known as a bottleneck since we have fewer channels. The intuition here is to reduce the number of channels so that the more expensive 3x3 convolution is cheaper. The other relevant part is that we have the residual "skip" connection where we add the input to the result of the transformations. Notice the residual connection connects the expanded parts x and m3.

def residual_block(x, squeeze=16, expand=64):
    # x has 64 channels in this example
    m1 = Conv2D(squeeze, (1,1), activation='relu')(x)
    m2 = Conv2D(squeeze, (3,3), activation='relu')(m1)
    m3 = Conv2D(expand, (1,1), activation='relu')(m2)
    return Add()([m3, x])

Listing 1: Example of a Basic Residual Block in Keras (adapted from source)

Next, let's look the changes in an inverted residual block shown in Listing 2. Here we "invert" the residual connection where we are making the residual connection between the bottleneck "squeezed" layers instead of "expanded" layers. Recall that we'll eventually be stacking these blocks, so there will still be alternations of squeezed ("bottlenecks") and expansion layers. The difference with Listing 1 is that we'll be making residual connections between the bottleneck layers instead of expansion layers.

def inverted_residual_block(x, expand=64, squeeze=16):
    # x has 16 channels in this example
    m1 = Conv2D(expand, (1,1), activation='relu')(x)
    m2 = DepthwiseConv2D((3,3), activation='relu')(m1)
    m3 = Conv2D(squeeze, (1,1), activation='relu')(m2)
    return Add()([m3, x])

Listing 2: Example of an inverted residual block with depthwise convolution in Keras (adapted from source)

The other thing to note is that the 3x3 convolution is now expensive if we do it on the expanded layer, so instead we'll use a depthwise convolution for efficiency. This reduces the number of parameters needed from \(h\cdot w \cdot d_i \cdot d_j \cdot k^2\) for a regular 3x3 convolution to \(h\cdot w \cdot d_i (k^2 + d_j)\) for a depthwise convolution where \(h, w\) are height and width, \(d_i, d_j\) are input/output channels, and \(k\) is the convolutional kernel size. With \(k=3\) this could potentially reduce the number of parameters needed by 8-9 times with only a small hit to accuracy.

def inverted_linear_residual_block(x, expand=64, squeeze=16):
    m1 = Conv2D(expand, (1,1), activation='relu')(x)
    m2 = DepthwiseConv2D((3,3),  activation='relu')(m1)
    m3 = Conv2D(squeeze, (1,1))(m2)
    return Add()([m3, x])

Listing 3: MBConv Block in Keras (adapted from source)

The last big thing thing that MBConv block changes was removing the non-linearity on the bottleneck layer as shown in Listing 3. A hypothesis the [2] proposes is that ReLU non-linearity on the inverted bottleneck hurts performance. The idea is that ReLU either is the identify function if the input is positive, or zero otherwise. In the case that the activation is positive, then it's simply a linear output so removing the non-linearity isn't a bit deal. On the other hand, if the activation is negative then ReLU actively discards information (e.g., zeroes the output). Generally for wide networks (i.e., lots of convolutional channels), this is not a problem because we can make up for information loss in the other channels. In the case of our squeezed bottleneck though, we have fewer layers so we lose a lot more information, hence hurt performance. The authors note that this effect is lessened with skip connections but still present. (Note: Not shown in the above code is that BatchNormalization is applied after every convolution layer (but before the activation).)

The resulting MobileNetV2 architecture is very memory efficient for mobile applications as the name suggests. Generally, the paper shows that MobileNetV2 uses less memory and computation with similar (sometimes better) performance on standard benchmarks. Details on the architecture can be found in [2].

1.2 Squeeze and Excitation Optimization

The Squeeze and Excitation (SE) block [3] is an optimization that can be added on to a convolutional layer that scales each channel's outputs by using a learned function of the average activation of each channel. The basic idea is shown in Figure 1 where from a convolution operation (\(F_{tr}\)), we branch off to calculate a scalar per channel ("squeeze" via \(F_{sq}\)), pass it through some layers ("excite" via \(F_{ex}\)), and then scale the original convolutional outputs using the SE block. This can be thought of as a self-attention mechanism on the channels.

Figure 1: Squeeze Excitation block with ratio=1 [ 3 ]

The main problem the SE block addresses is that each convolutional output pixel only looks at it's local receptive field (e.g. 3x3). A convolutional network only really considers global spatial information by stacking multiple layers, which seems inefficient. Instead, the hypothesis of the SE block is that you can model the global interdependencies between channels and allow each channel to increase their sensitivity improving learning.

Code for an SE block is shown in Listing 4. First, we do a GlobalAveragePool2D, which computes the mean for each channel. Then we pass it through two 1x1 convolutional layers with a ReLU and sigmoid activation respectively. The first convolutional layer can be thought of as "mixing" the averages across the channels, while the second one converts it to a value between 0 and 1. It's not clear whether more or less layers is better but [3] says that they wanted to limit the added model complexity while still having some generalization power.

def squeeze_excite(x, filters, ratio=4):
    # computes mean of each spatial dimensions (outputs a mean value for each channel)
    m1 = GlobalAveragePooling2D(keepdims=True)(x)
    m2 = Conv2D(filters // ratio, (1, 1), activation='relu')(m1)
    m3 = Conv2D(filters, (1, 1), activation='sigmoid')(m2)
    return Multiply(m3, x)

Listing 4: SqueezeExcite block in Keras (adapted from source)

Since the SE block only operates on channels as a whole, the added computational and memory requirements are modest. The largest contributors are usually the latter layers that have a lot of channels. In their experiments the parameters of a MobileNet network increased by roughly 12% but was able to improve the ImageNet top-1 error rate by about 3% [3]. Overall, it seems like a nice little optimization that improves performance across a wide variety of visual tasks.

1.3 EfficientNet

EfficientNet is a convolutional neural networks (ConvNet) architecture [4] (circa 2019) that rethinks the standard ConvNet architecture choices and proposes a new architecture family called EfficientNets. The first main idea is that ConvNets can be scaled to have more capacity in three broad network dimensions shown in Figure 2:

• Wider: In the context of ConvNets, this corresponds to more channels per layer (analogous to more neurons in a fully connected layer).

• Deeper: Corresponds to more convolutional layers.

• Higher Resolution: Corresponds to using higher resolution inputs (e.g. 560x560 vs. 224x224 images).

Figure 2: Model scaling figure from [ 4 ]: (a) base model, (b) increase width, (c) increase depth, (d) increase resolution.

The first insight [4] found is that, as expected, scaling the above network dimensions result in better ConvNet accuracy (as measured via top-1 ImageNet accuracy) but with diminishing returns. To standardize the evaluation, they normalize the scaling using FLOPS.

The next logical insight discussed in [4] is that balancing how all three scaling network dimensions is important to efficiently scale ConvNets. They propose a compound scaling method as:

The intuition here is that we want to be able to scale the network size appropriately for a given FLOP budget, and Equation 1, if satisfied, will approximately scale the network by \((\alpha \cdot \beta^2 \cdot \gamma^2)^\phi\). Thus, \(\phi\) is our user-specified scaling parameter while \(\alpha, \beta, \gamma\) are how we distribute the FLOPs to each scaling dimension and are found by a small grid search. The constraint \(\alpha \cdot \beta^2 \cdot \gamma^2 \approx 2\) (I believe) is arbitrary so that the FLOPS will increase by roughly \(2^\phi\). Additionally, it likely simplifies the grid search that we need to do.

A specific EfficientNet architecture is also proposed in [4] that defines a base architecture labelled "B0" shown in Figure 3 using the above MBConv MobileNetV2 block discussed above with the Squeeze and Excitation optimization added to each block. Overall the base B0 architecture is a typical ConvNet where in each layer the resolution decreases but channels increase.

Figure 3: EfficientNet-B0 baseline architecture [ 4 ]

From the B0 architecture, we can derive scaled architectures labelled B1-B7 by:

1. Fix \(\phi=1\) and assume two times more resources are available (see Equation 1), and do a small grid search to find \(\alpha, \beta, \gamma\), which were \(\alpha=1.2, \beta=1.1, \gamma=1.15\) (depth, width, resolution, respectively), which give roughly 1.92 according to Equation 1.

2. Scale up the B0 architecture approximately using Equation 1 with the constants described in Step 1 by increasing \(\phi\) (and round where appropriate). Dropout is increased roughly linearly as the architectures grow from B0 (0.2) to B7 (0.5).

Table 1 shows the flops, multipliers and dropout rate for each dimension.

For example, starting with B0, we have 0.39B FLOPs, going to B4 we have 4.2B flops, which yields \(\phi = 4.2 / 0.39 \approx 3.28\). This translates to scaling close to this value along the three dimensions with \(\alpha^{3.22} = 1.2^{3.22} \approx 1.8\), \(\beta^{3.53}=1.1^{3.53}\approx 1.4\), and \(\gamma^{3.78} = (1.15)^{3.78} \approx \frac{380}{224}\). We're not going for precision here, we just want a rough guideline of how to scale up the architecture. The nice thing about having this guideline is that we can create bigger ConvNets without having to do any additional architecture search.

1.4 Noisy Student

Noisy Student [5] is a semi-supervised approach to training a model that is useful even when you have abundant labelled data. This work is in the context of images where they show its efficacy on ImageNet and related benchmarks. The setup requires both labelled data and unlabelled data with a relatively simple algorithm (with some subtlety) and the following steps:

1. Train teacher model \(M^t\) with labelled images using a standard cross entropy loss.

2. Use the \(M^t\) (current teacher) to generate pseudo labels for the unlabelled data (filter and balance dataset as required)

3. Learn a student model \(M^{t+1}\) with equal or larger capacity on the labelled and unlabelled data with added noise.

4. Increment \(t\) (make the current student the new teacher) and repeat steps 2-3 as needed.

A few unintuitive points emphasized in bold. First, the student model uses an equal or larger model. This is different from other student/teacher paradigms where one is trying to distill the model knowledge into a smaller model. Here we're not trying to distill, we're trying to boost performance so we want a bigger model so it can learn from the bigger combined dataset. This seems to have a increase of 0.5-1.5% in top-1 ImageNet accuracy in their ablation study.

Second, the noise is implemented as randomized data augmentation plus dropout and stochastic depth. The added noise on the student seems to be around another 0.5% in top-1 ImageNet accuracy. Seems like a reasonable modification given that you typically want both of these things when training these types of networks.

Third, the iteration in step 4 also seemed important. Going from one iteration to three improved performance by 0.8% in top-1 ImageNet accuracy. It's not obvious to me that the performance would improve by iterating here but since the number of iterations is small, I can believe that it's possible.

Lastly, they discuss that they filter out pseudo labels that have low confidence by the teacher model, and then rebalance the unlabelled classes so the distribution is not so off (by repeating images). This also seems to improve performance a bit more modestly at 0-0.3% depending on the model.

The summary of the overall Noisy Student results are shown in Figure 4 where they conducted most of their experiments on EfficientNet. This figure only shows the non-iterative training (their headline result is within the iterative training). You can see that the Noisy Student dominates the vanilla EfficientNet results at the same number of model parameters and achieves SOTA (at the time of the paper). Note that the Noisy Student does have access to more unlabelled data than the EfficientNet, so perhaps it's not so surprising that it does better. In the context of this post, there are many versions of EfficientNet with Noisy Student training that are available for use as a pre-trained model.

Figure 4: Noisy Student training shows significant improvement over all model sizes. [ 5 ]

3.1 Dataset Creation and Data Preprocessing

The winning solution used a preprocessed dataset that one of his colleagues created [6]. This dataset was in fact used by many of the competing teams and arguably one of the most critical pieces of work (something that a huge amount of time is spent on in real world problems).

The first step in preprocessing was center cropping and resizing the images. Many of the JPEG images were really large and had different dimensions (e.g., 1053x1872, 4000x6000, etc.) totalling 32GB. After reducing them down to various standard sizes (e.g. 512x512, 768x768, 1024x1024) they were much more manageable to use, for example the 512x512 dataset was about 3GB for 2020 data.

Next, the preprocessed dataset also contained a "triple" stratified 5-fold validation dataset:

• Separate Patients: This stratification was to ensure that the same patient was not in both the train and validation set. This can happen when you have two skin lesion images from the same person, which is undesirable because the resulting diagnosis is likely highly correlated in these situations.

• Positive Class: This stratification was to ensure that the positive classes were distributed correctly across each fold. Due to the highly imbalanced problem of only having 1.76% positive classes, ensuring an even balance across folds was very important.

• Patient Representation: Some patients had only a few images while others had many. To have balanced folds, this stratification was to ensure that you have good representation of each across each fold as well.

Lastly, although the external data had a lot of additional images, many of them were in fact duplicates that should be removed. But this is harder than it looks because the images were not exact matches, for example they could be scaled and rotated, thus you cannot just compare the raw pixels. To have a clean validation set, you want to make sure you have a truly independent train and validation set. To solve this problem, the preprocessing in [6] used a pre-trained (EfficientNet) CNN to generate embeddings of each image, and then removed near duplicates (with manual inspection). Hundreds of duplicates were removed, making a much cleaner validation set.

3.2 Validation Strategy

The first place solution noted that one of the keys to winning was having a robust validation strategy, which was particularly important in this competition [6] (as well in the real world). As noted above, the original dataset had only a 1.76% positive rate over 33k training samples. That translates to around 580 positive samples, and barely over 100 samples when doing for a 5-fold cross validation. This naturally would lead to an unstable AUC (or pretty much any other metric you're going to use).

Beyond the training data provided, the test data that could be evaluated via the public leader board had only about 10k samples, 30% of which was used to evaluate AUC on the public leader board. If the distribution were similar in this test set, this would only leave about 50 or so positive test case samples. Thus, the public leader board evaluation was similarly unreliable and couldn't be used to robustly evaluate the model. This was clearly seen as the top 3 public leader ranks dropped significantly when evaluated on the private data set. The authors also mention that their cross validation scores (described below) were not correlated with the public leader board and that they basically ignored the leader board.

The winning solution instead utilized both the competition (2020) data and external data (2019) for training and validation. The 2019 data had 25k data points with a 17.85% positive rate, making it much more reliable when it was used for both training and validation.

The other key thing they did was to train on a multi-class problem instead of the binary target given by the competition. In the 2020 data, a detailed diagnosis column was given, while in the 2019 data, a higher-level multi-class label was given (vs. the binary label). As is typical in many problems, they leveraged some domain knowledge (using the descriptions from the competition) and mapped the 2020 detailed diagnosis to the 2019 labels shown in Figure 5. The main intuition of using a multi-class target is that it gives more information to the target when the lesion is benign (not cancerous).

Figure 5: Mapping from diagnosis to targets [ 1 ]

When evaluating the model the primary evaluation metric is the binary classification AUC of the combined 2019 and 2020 cross validation folds (the multi-class problem can easily be mapped back to a binary one). The cross validation AUC of the 2020 dataset was used as a secondary metric.

3.3 Architecture

The solution consisted of an ensemble of eighteen fine-tuned pre-trained ConvNets shown in Figure 6 that were combined using a simple average of ranks. Notice that the first 16 models are EfficientNet variants from B3 all the way to B7, while the last two are SE-ResNext101 and Nest101. For the EfficientNet variants, besides the model size, the models vary by the image input sizes (384, 448, 512, 576, 640, 768, 896) deriving from the next largest source image in the above described dataset (512, 768, 1024). The different models plus image sizes is an important source of diversity in the ensemble. Unfortunately, the authors didn't describe how they selected their ensemble except to say that diversity was important. Interestingly, the authors state [6] that the CNN backbone isn't all that important and they mostly just picked an off-the-shelf state-of-the-art model architecture (EfficientNet) where pre-trained models and code are readily available.

Figure 6: Model configurations for winning solution ensemble and their AUC scores [ 1 ]

The ensembles also varied based on their use of metadata with tuned learning rates and epochs for each configuration. The authors mention [6] that the metadata didn't seem to help much with their best single model not using metadata. They hypothesize that most of the useful information is already included in the image. However, they think it was useful in providing diversity in the ensemble (again being one of the most important parts of ensembling). Additionally, one of the models only used a reduced target with 4 classes (collapsing the "*" labels in Figure 5).

Another interesting part is how they incorporated the metadata with the images. Figure 7 shows the architecture with metadata. The metadata network is relatively simple with two fully connected layers whose output is concatenated with the CNN before the last classification layer. They use a pretty standard architecture with BatchNorm and dropout, but they do use the Swish activation.

Figure 7: Model architecture including metadata [ 1 ]

Lastly, I noticed a trick that I had not seen before in the last linear classification layer. They use five copies of the linear layer (shared params) each with a different dropout layer, which then are averaged together to generate the final output shown in Listing 5. My guess to its purpose is that it is trying to remove the randomness of dropout, which may be important since it's so high at a 0.5 dropout rate.

for dropout in enumerate(self.dropouts):
    if i == 0:
        out = self.myfc(dropout(x))
    else:
        out += self.myfc(dropout(x))
out /= len(self.dropouts)

Listing 5: Last layer of Winning Solution ( source )

3.4 Augmentation

Data augmentation is a common method to improve vision models and many libraries are readily available for standard transformations. The winning solution used the albumentations library that has a rich variety of image transformations that are performant and easily accessible.

The authors used a whole host of transformations where you can see the code here. I'll mention a few interesting points I found:

• The last transformation always resizes the image back to the target size.

• They use the compose() function to apply all transformations to each image, however (in most cases) each transformation will have some probability of activating or not.

• They'll also have some OneOf choices in there for blurring (Motion, Median, Gaussian, GaussNoise blurring) and distortion (OpticalDistortion, GridDistortion, ElasticTransform).

• Some of the transforms require more careful setting of parameters that depend on the domain. For example the Cutout transform, which blanks out a square region of the image, requires a bit more careful thinking to ensure that the region isn't too large. In this case, they used a single 37.5% of image sized square to cutout with 70% probability.

• The transformations are only on the training set. The validation set transforms are only used to preprocess the image for the model by doing a resize and normalization.

3.5 Prediction

On the prediction side there were a couple tricks that I thought are worth mentioning:

• Fold Averaging: The best model from each of the 5 validation folds is saved based on the validation dataset AUC. This means for every ensemble model (total 18), we have 5 trained models. The prediction for a single model is generated by averaging the 5 probabilities. That is, for each of the 5 trained models (identical configurations but different folds), compute the mean across the 5 models for each softmax output separately.

• Orientation Averaging: Due to the nature of the images, the solution (for each of the above fold models) averaged 8 different predictions per model, where each prediction was given a different orientation of the input image. This means for each model configuration you have 5 x 8 predictions, which are averaged by their probabilities.

  The different orientations are: original, horizontal flip, vertical flip, horizontal & vertical flip, diagonal "flip" (transpose), diagonal & horizontal flip, diagonal & vertical flip, diagonal & horizontal & vertical flip. For skin lesions the orientation probably doesn't matter at all, so computing the average over many different orientations probably smooths out any quirks the models has with a particular orientation. Intuitively, I would guess this increases the robustness of the model's prediction. See the source for the details on both of these tricks.

• Ensemble Construction: From each of the above 18 model configurations after averaging you have a single column of probabilities corresponding to the test set data. To generate the final prediction, we do a "rank average": convert all the probabilities in a column to ranks, normalize those relative ranks to between 0 and 1 (as a percentage), and finally compute a simple mean between all of the columns. This is probably more robust than computing a simple probability average because it does not overweight confident models that might output very high (or low) probability values.

3.6 Training Details

Here are the details for the training:

• Epochs: 15 for most models. I'm going to guess (because they didn't specify) that they picked a large enough number so that the AUC didn't continue to increase but also balanced with a manageable run-time. Since they save the best model in the fold (according to the validation AUC), as long as this number is big enough you're only losing run-time.

• Batch size: 64 for all models. They mentioned (I believe) that it was easier to just keep it all the same for each model than try to tune it, presumably because batch size wasn't expected to make much of a difference.

• Learning rate/schedule: Ranged from \(1e-5\) to \(3e-5\) with a cosine cycle, where the learning rate is tuned for each model (recall this is fine-tuning a pre-trained ImageNet model). There is also a single warm-up epoch at the beginning which is one tenth of the initial learning rate.

• Optimizer: Adam. Stated that using a standard strong optimizer was good enough.

• Hardware: Trained on V100 GPUs in mixed precision mode with up to 8 GPUs used in a data parallel (batch split across GPUs) manner.

4.1 Changes That Showed Improvement

Figure 8 shows the experiments that appears to have improved performance beyond just randomness. The figure shows (a) the mean best AUC across the three folds measured on the validation set which includes external data (AUC_mean), and (b) the similarly calculated AUC for the validation set with just the 2020 competition data (AUC_20_mean). Error bars indicate the standard deviation computed across the best AUC score across the three folds. Starting left to right, the changes from the baseline were:

• baseline: Baseline setup described above.

• no-pretraining: Did not used a pre-trained B3 model.

• no-augmentation: Removed data augmentation transforms.

• no-external-data: A run that only utilized training data from the 2020 competition data set (no previous years).

• warmup: Remove the first warmup epoch which starts at a tenth of the initial learning rate and gradually grows to the target LR by the end of the epoch.

• no-cosine: Did not use cosine learning schedule (which was implemented as one cycle across all epochs).

Figure 8: Experiments that showed improvement

As you can see, there are only a few things that are obviously beneficial particularly relating to the data. The biggest gain appears to be due to pre-training, which is around an 0.08 AUC gap from the baseline. This makes sense since we only have around 60K data points, so pre-training (even on an unrelated ImageNet dataset) would be useful. The other big drop seems to be whether or not we're using data augmentation. Although the AUC metric seems like it might be only a small drop, the AUC_20 shows quite a large one, indicating that this plays a significant role in the performance. Another important data related change was the use of external data. The AUC metric is not reported because the default computation from the script was not comparable (and I didn't feel like hacking it to make it comparable). Instead, we can see the AUC_20 metric which shows about a 0.03 AUC gap. These results really show the value of adding more data (in whatever form) leads to the most durable boosts to performance.

The two other more minor improvements come from the learning rate schedules with a warmup epoch and a cosine scheduling of learning rates. The warmup seemed the most significant with 0.02 AUC difference with the cosine scheduling showing very minor improvement with 0.01 AUC (possible that it's not significant). Although it could be noise, they seem like relatively harmless improvements, which are highly likely to give a small boost to performance.

4.2 Changes That Did Not Showed Improvement

Similar to the changes that showed improvement, I ran a bunch of other experiments shown in Figure 9 with the same metrics. For the most part, they did not show significant improvement over the baseline. Starting from left to right in Figure 9, the changes were:

• baseline: Baseline setup described above.

• +noisy-student: Use a pre-trained B3 model with Noisy Student training.

• effnet_b2: Use EfficientNet B2 model instead of B3.

• lr={1.5e-5, 2e-5}: Use a learning rate of 1.5e-5, 2e-5 (vs. 3e-5).

• +metadata: Utilize patient (non-image) metadata.

• dropout-layers={1, 3}: Utilize 1, 3 parallel dropout layers (vs 5) as described above in the Architecture section.

• img_size=448: Utilize image size of 448 (vs. 384).

• binary_labels: Utilize binary labels in training (as the competition expects) instead of 9 classes.

• tensorflow_impl: Utilize a tensorflow implementation of the pre-trained model.

• amp: Utilize Automated Mixed Precision (AMP) in PyTorch.

Figure 9: Experiments that did not showed improvement

I'll just comment on a few things that were surprising:

• "Fancy" things didn't seem to be that important. For example, Noisy Student or changing architecture to B2 didn't seem to do much. Similarly the dropout layers didn't change things much.

• Implementation details didn't seem to make a big difference like the Tensorflow implementation or using AMP.

• Not very sensitive to many hyperparameters like learning rate and image size. Although I suspect it could be because we're fine-tuning vs. doing direct training. The learning rate is so small (and we have a lot of epochs) so it may not make a big difference.

• One thing I did find surprising was that the binary labels didn't didn't make that much of a difference. Intuitively, it feels like better categories would encourage better learning but even if it did, it looks like it wasn't that significant. It could be because the different classes were all distinguishing the negative labels, never the difference between positives and negative. Similar to the previous section, this experiment only had an AUC_20 measure since the problem formulation was different.

• One of things that was borderline useful was the use of metadata, which I included in this section instead of the one above. The paper states that metadata didn't in general do much but was useful to make more diverse models. In my experiments, it's not significant but it's possible it does help, perhaps especially in smaller models/image sizes like B3/384x384. It's hard to draw conclusions though.

In general, as is the case in many of these situations, there are not many silver bullets. Most things do not significantly improve the performance of the problem in a real world scenario. Or at least they're not "first order" improvements that you would try on a first pass of a problem. If you're optimizing for a 0.1% improvement (e.g. AdWords) then you might want to spend more time with these "second order" improvements to hyper-optimize things. Although these experiments are not extensive, they probably point directionally to what you should care about: data and maybe better learning schedules.

4.3 Training Time

For most of my experiments, a typical run would take about 27 hours (about 37 mins / epoch) for the baseline setup. This seemed kind of unreasonably long for a small B3 setup, but I mostly ignored it because I was just queueing the jobs up and coming back to next day to see what happened. I even noticed that the GPU utilization was low but ignored it thinking it was a weird artifact of my training setup. At one point late in my experiments, I started playing around with AMP and PyTorch 2.0 compilation but didn't see a significant change in run-time.

Only after I had the idea to run the data augmentation experiment did I realize my issue: the data augmentation transforms were bottlenecking my batches! I had been using a single thread for the DataLoader (which I did to avoid an error initially, see notes below), which caused most of the run-time to be on my CPU. After increasing the number of workers to 6 (the number of cores on my CPU), I got the run-time down to less than 10 minutes per epoch with relatively high GPU utilization.

Most of my previous explorations have been on relatively smaller models (partly because I had a small 8GB 1070 up until last year) so I didn't have to think too much about run-time. But now that I'm running bigger jobs, savings 3-4x in run-time is pretty significant even though I typically leave it running for 24 hours between experiments (because I only work on these projects in the evening). I think I'll be looking more into the basics of optimizing run-time in the near future.