AutoML Tutorial: Hyperparameter Optimization, NAS, AutoKeras, TPOT (2026)
AutoML automates the end-to-end process of applying machine learning to real-world problems. It handles algorithm selection, hyperparameter tuning, feature engineering, and even neural architecture search.
By 2026, AutoML tools have matured significantly. AutoKeras provides accessible neural architecture search, TPOT optimizes pipelines with genetic programming, and HPO frameworks like Optuna and Ray Tune handle large-scale hyperparameter optimization.
Hyperparameter Optimization with Optuna
Optuna uses Tree-Structured Parzen Estimator for efficient hyperparameter search. It defines objectives with suggest_* API and prunes unpromising trials automatically.
I optimized an XGBoost model with 15 hyperparameters using Optuna. The study found a configuration improving RMSE by 12 percent in 200 trials, with automatic pruning cutting trial time by 40 percent.
import optuna\ndef objective(trial):\n params = {"lr": trial.suggest_float("lr", 1e-4, 0.1, log=True), "max_depth": trial.suggest_int("max_depth", 3, 12), "subsample": trial.suggest_float("subsample", 0.6, 1.0)}\n model = xgb.XGBRegressor(**params, n_estimators=500)\n return cross_val_score(model, X, y, cv=3).mean()\nstudy = optuna.create_study(direction="maximize")\nstudy.optimize(objective, n_trials=200)Neural Architecture Search with AutoKeras
AutoKeras automates neural network design using Bayesian optimization and network morphism. It searches over layer types, sizes, activations, and connectivity patterns.
AutoKeras found a CNN architecture for image classification that achieved 96 percent accuracy with 40 percent fewer parameters than a manually designed ResNet. The search completed in 4 hours on a single GPU.
import autokeras as ak\nclf = ak.ImageClassifier(max_trials=10, overwrite=True)\nclf.fit(x_train, y_train, epochs=50)\nmodel = clf.export_model()\nprint(clf.evaluate(x_test, y_test))Automated Pipeline with TPOT
TPOT uses genetic programming to optimize ML pipelines, including preprocessing, feature selection, and model choice. It evolves populations of pipelines through selection, crossover, and mutation.
TPOT found a pipeline combining PCA, polynomial features, and Gradient Boosting that outperformed our manual pipeline by 8 percent accuracy on a tabular dataset. The search ran for 24 hours but found solutions we never would have considered.
from tpot import TPOTClassifier\ntpot = TPOTClassifier(generations=5, population_size=50, cv=5, random_state=42, verbosity=2)\ntpot.fit(X_train, y_train)\nprint(tpot.score(X_test, y_test))\ntpot.export("tpot_pipeline.py")Automated Feature Engineering
AutoML extends to feature engineering through automated generation and selection. Featuretools automates relational feature creation, and TSFRESH extracts features from time series automatically.
Combining Featuretools with AutoML pipeline search is powerful: generate thousands of candidate features automatically, then let TPOT or Optuna select the best combination. This approach added 5 percent AUC for a fraud detection model.
from tsfresh import extract_features\nfrom tsfresh.feature_selection import select_features\n# Extract hundreds of time series features automatically\nextracted_features = extract_features(timeseries, column_id="id", column_sort="time")\nselected_features = select_features(extracted_features, y)Distributed HPO with Ray Tune
Ray Tune scales hyperparameter optimization across clusters using trial scheduling, checkpointing, and early stopping. It supports ASHA, HyperBand, and Bayesian optimization schedulers.
I used Ray Tune to tune a transformer model across 32 GPUs. The ASHA scheduler stopped poor trials early, reducing total search time from days to hours. Ray's distributed execution scaled seamlessly from laptop to cluster without code changes.
from ray import tune\nfrom ray.tune.schedulers import ASHAScheduler\nscheduler = ASHAScheduler(max_t=100, grace_period=10, reduction_factor=3)\ntuner = tune.Tuner(train_function, param_space=config, tune_config=tune.TuneConfig(scheduler=scheduler, num_samples=100))\nresults = tuner.fit()Meta-Learning and Model Selection
Meta-learning uses dataset characteristics to predict which algorithms and hyperparameters will perform best. Tools like Auto-sklearn use meta-features to warm-start the search, reducing optimization time.
Auto-sklearn's meta-learning component selects the top K configurations based on dataset similarity, reducing initial search space by 90 percent. Combined with Bayesian optimization and ensemble construction, it consistently outperforms default configurations across diverse datasets.
import autosklearn.classification\nautoml = autosklearn.classification.AutoSklearnClassifier(time_left_for_this_task=3600, per_run_time_limit=120)\nautoml.fit(X_train, y_train)\nprint(automl.leaderboard())Frequently Asked Questions
Is AutoML better than manual tuning?
AutoML systematically explores more configurations than manual tuning and often finds better solutions. However, domain expertise in feature engineering and problem framing remains essential.
How long does AutoML take?
From minutes (small datasets, simple search) to days (large datasets, neural architecture search). Set time limits and use early stopping for practical results within your schedule.
Can AutoML replace data scientists?
No. AutoML automates repetitive tuning but cannot replace domain expertise, problem formulation, data collection strategy, or business understanding.
Which AutoML tool should I use?
Optuna/Ray Tune for HPO, AutoKeras for neural architecture search, TPOT for automated pipeline discovery, Auto-sklearn for comprehensive automated ML on tabular data.
Originally published on Ayodhyyya. Last updated June 1, 2026.