Efficient neural network architecture search

Consequently, numerous publications started to push the boundaries of accuracy, however at immense training costs. For tens of days, hundreds of GPUs evaluate thousands of candidates in order to discover a configuration that surpasses the accuracy of hand designed networks or of the results of other automated searches.

After designing our own type of search and analysing its drawbacks, we observed repetitive evaluations of similar candidates. Unless the intermediate results are wisely reused, architecture searches started from scratch have high chances of wasting compute time on similar or identical candidates. We propose to speed up the evaluation phase by replacing the slow training process with a fast and reliable prediction of the candidate’s performance.<br><br>In contrast to previously proposed approaches, our prediction is not only calibrated on the topological network information, but also on the characterization of the dataset-difficulty which allows us to re-tune the prediction for unseen datasets without any training.<br><br>We employ our predictor in image and text classification tasks, and emphasize oil the delivered speed-up in comparison with related work.