What Happens When ML Models Design ML Models?
As machine learning (ML) continues to advance, a very interesting frontier is emerging: ML models are actually designing new ML models. Moving from the creation of manually designed predictive algorithms to automatically creating them represents a huge accelerant for innovation in data science workflows. Let’s unpack how this works, and why it matters.
The process of building an ML model has a familiar lifecycle: define a problem and outcome; collect and prepare data; choose model type; train a model; evaluate; and lastly deploy. At almost every step along the way, there are human decisions – should I use a linear regression, a decision tree, a random forest, or a clustering approach?
So, what if an ML system made some of these decisions instead? That’s the basis for AutoML and Neural Architecture Search (NAS) – allowing models to automatically explore, tune, and create new model architectures.
AutoML frameworks allow users to automate the entire modeling pipeline to some extent, from featurization to hyperparameter tuning. Test many algorithms. The end-user builds and runs classification and/or regression models, each being run as an independent experiment and tracked as such. Tune hyperparameters. The end-user takes an existing hyperparameter tuning framework and selects desirable metrics including tree depth, learning rates, regularization strength, etc. Select the best performer. The end-user compares their models on several metrics including accuracy, f1, or mean squared error. What are you left with? A model/pipeline that was built with minimal coding and human interaction—done efficiently and robustly.
In addition to the selection of algorithms, Neural Architecture Search can enable ML to create its own new neural network architectures, including the types of layers, the connections between them, and the size of each layer. NAS frameworks are capable of exploring a search space of potential architectures (candidates), evaluating those candidates, and adapting suggestions based on past performance. This is similar to how humans may explore ideas—however, NAS is automated, massively parallel, and frequently humorously more creative than we could imagine. Architectures developed through Neural Architecture Search either equal or exceed the performance of networks developed by experts.
However there are some negative aspects to consider. Computational Expense. Testing many candidate models could be expensive in terms of computing resources. Interpretability. Automated pipelines could be opaque. Expertise in the Domain is still needed. Automation doesn't replace selection of metrics, feature engineering, or whether bias has been mitigated.
We are witnessing the emergence of automated AI designers in ML models designing ML models - where machines not only learn from data to learn, but also refine the architecture of how they learn. This automatically reduces the timeline of model building, creates pathways for innovation and ultimately imbeds repeatable patterns of best-practice while monitoring fairness, cost and interpretability within our evolved understanding of the intention of the architecture.
After all, just because you start letting the machine design the machine, does not mean you cannot have the smarter human involved - it just extends the horizon. Data scientists now can focus more time on framing the right questions and collecting the best data and deploying it ethically, while the intelligent systems optimize and iterate at scale.
Sources:
https://www.geeksforgeeks.org/machine-learning/machine-learning-models/
https://learn.microsoft.com/en-us/fabric/data-science/machine-learning-model