Tutorial: Learn Autonomous Vehicles from Scratch (2026)
Autonomous vehicles (AVs) combine sensors, machine learning, planning algorithms, and control systems to navigate without human input. After developing perception pipelines for a self-driving shuttle project and working with LIDAR, camera, and radar fusion, I can say that autonomous driving is a solved problem in controlled environments but remains incredibly hard in unpredictable real-world conditions.
This tutorial covers the AV software stack: sensor hardware (LIDAR, cameras, radar, IMU), perception (object detection, segmentation, tracking), localization (GPS + SLAM, HD maps), path planning (A*, RRT, behavior trees), and vehicle control (PID, MPC). You will build a simulated autonomous driving system using open-source tools.
Sensor Suite and Data Fusion
Autonomous vehicles use multiple sensor types for redundancy: LIDAR (Light Detection and Ranging) provides 3D point clouds up to 200m with centimeter accuracy. Cameras provide semantic understanding (lane markings, traffic signs, traffic lights) at high resolution. Radar detects objects at long range (300m) in all weather conditions, providing velocity information directly via Doppler shift. IMU (Inertial Measurement Unit) provides orientation and acceleration at 100-200 Hz.
Sensor fusion — combining data from heterogeneous sensors — is done at three levels: raw data fusion (combine point clouds and pixels before processing), feature fusion (extract features separately then combine), or decision fusion (object detections from each sensor are combined). The Kalman filter and its variants (EKF, UKF) are the standard fusion algorithm, estimating the state (position, velocity, orientation) from multiple noisy measurements.
import numpy as np
from filterpy.kalman import KalmanFilter
def create_tracker():
kf = KalmanFilter(dim_x=6, dim_z=3) # x, y, z, vx, vy, vz
dt = 0.1
kf.F = np.array([
[1, 0, 0, dt, 0, 0],
[0, 1, 0, 0, dt, 0],
[0, 0, 1, 0, 0, dt],
[0, 0, 0, 1, 0, 0],
[0, 0, 0, 0, 1, 0],
[0, 0, 0, 0, 0, 1]
])
kf.H = np.eye(3, 6) # Measurement: position only
kf.P *= 10.0 # Initial uncertainty
return kf
def fuse_measurements(lidar_detections, radar_detections, camera_detections):
# Fusion via IOU tracking (simple approach)
fused = {}
for det in lidar_detections:
track_id = match_to_track(det, fused)
fused[track_id].lidar_update(det)
for det in radar_detections:
track_id = match_to_track(det, fused)
fused[track_id].radar_update(det)
return fusedPerception — Object Detection and Segmentation
The perception stack interprets sensor data into a world model. Object detection identifies and localizes vehicles, pedestrians, cyclists, and obstacles. 3D object detection from LIDAR (PointPillars, VoxelNet, CenterPoint) outputs oriented bounding boxes with x, y, z, width, length, height, and yaw. Semantic segmentation classifies every pixel (road, sidewalk, vehicle, vegetation).
Deep learning models for perception must run in real time — under 50ms inference latency. Use quantized models (TensorRT, ONNX Runtime) on embedded GPUs (NVIDIA Orin, Xavier). For redundancy, run multiple models simultaneously and vote on results.
import torch
import open3d as o3d
def lidar_object_detection(points_np, model, config):
# Convert numpy point cloud to voxel representation
voxels = voxelize(points_np, config.voxel_size)
with torch.no_grad():
detections = model(voxels)
results = []
for det in detections:
bbox = {
'x': det[0].item(),
'y': det[1].item(),
'z': det[2].item(),
'width': det[3].item(),
'length': det[4].item(),
'height': det[5].item(),
'yaw': det[6].item(),
'score': det[7].item(),
'class': int(det[8].item())
}
if bbox['score'] > config.confidence_threshold:
results.append(bbox)
return resultsLocalization and HD Mapping
An autonomous vehicle must know its position within centimeters. GPS provides ~3m accuracy (with RTK correction, <10cm). IMU dead-reckons between GPS updates. LIDAR-based localization matches real-time point clouds against a pre-built HD map using ICP (Iterative Closest Point) or NDT (Normal Distributions Transform). Particle filters (Monte Carlo Localization) handle multi-modal uncertainty.
HD maps contain lane geometry, traffic signs, speed limits, and semantic landmarks at centimeter precision. Building and maintaining HD maps is expensive — companies use survey vehicles that drive every road multiple times. OSM (OpenStreetMap) provides a free alternative for development and simulation.
import numpy as np
from ndt import NDT
def localize(scan_cloud, map_cloud, initial_pose):
ndt = NDT()
ndt.set_input_source(scan_cloud)
ndt.set_input_target(map_cloud)
# NDT alignment
transform = initial_pose
for i in range(30):
transform = ndt.iterate(transform)
# Particle filter for robustness
particles = initialize_particles(500, transform)
for obs in get_observations():
particles = prediction_step(particles, imu_data)
weights = update_step(particles, obs)
particles = resample(particles, weights)
return particles.mean(axis=0)Path Planning
Path planning operates at three levels: (1) Route planning — find a road-level path from A to B using A* or Dijkstra on a road graph. (2) Behavior planning — decide what the vehicle should do: follow lane, change lane, yield, stop, or merge, using finite state machines or behavior trees. (3) Motion planning — generate a smooth, collision-free trajectory using optimization (OMPL, Frenet-optimal planning).
The trajectory is a time-indexed sequence of (x, y, yaw, velocity) waypoints. Constraints include: kinematic (minimum turning radius, maximum acceleration), dynamic (max lateral acceleration for comfort), and safety (minimum distance to obstacles, speed limits).
import numpy as np
from scipy.optimize import minimize
def frenet_plan(start_state, goal_state, obstacles, road_center):
# Sample in Frenet coordinates (s: longitudinal, d: lateral)
s_samples = np.linspace(0, goal_state.s, 20)
d_samples = np.linspace(-2, 2, 11)
best_trajectory = None
best_cost = float('inf')
for d_target in d_samples:
for s_target in s_samples:
trajectory = calculate_polynomial(
start_state, {'s': s_target, 'd': d_target},
T=5.0 # 5 second horizon
)
cost = evaluate_cost(trajectory, obstacles)
if cost < best_cost:
best_cost = cost
best_trajectory = trajectory
return best_trajectory
def evaluate_cost(traj, obstacles):
lateral_cost = np.sum(traj.d**2)
jerk_cost = np.sum(np.diff(traj.d, 2)**2)
collision_cost = 0
for obs in obstacles:
if collision_detected(traj, obs):
collision_cost += 1000
return lateral_cost + jerk_cost + collision_costVehicle Control
Control systems translate the planned trajectory into steering, throttle, and brake commands. PID control is simple and effective for lateral (steering) and longitudinal (speed) control: the error between desired and actual position/speed is fed into proportional, integral, and derivative terms to produce the control output.
Model Predictive Control (MPC) is the state of the art: it solves a constrained optimization problem at each time step, considering the vehicle's dynamics model, and finds the optimal sequence of control inputs over a receding horizon (typically 1-3 seconds). MPC handles constraints (max steering angle, acceleration limits) naturally.
class MPCController:
def __init__(self, dt=0.1, horizon=10):
self.dt = dt
self.N = horizon
def solve(self, state, reference, obstacles):
# state: [x, y, yaw, v]
# reference: desired trajectory points
def objective(controls):
# controls: [steering_angle_0..N-1, acceleration_0..N-1]
steering = controls[:self.N]
acceleration = controls[self.N:]
total_cost = 0
sim_state = state.copy()
for t in range(self.N):
# Kinematic bicycle model
sim_state[2] += steering[t] * self.dt # yaw
sim_state[0] += sim_state[3] * np.cos(sim_state[2]) * self.dt
sim_state[1] += sim_state[3] * np.sin(sim_state[2]) * self.dt
sim_state[3] += acceleration[t] * self.dt
# Track reference
total_cost += (sim_state[0] - reference[t][0])**2
total_cost += (sim_state[1] - reference[t][1])**2
total_cost += steering[t]**2 * 0.1
return total_cost
result = minimize(objective, np.zeros(2 * self.N), method='SLSQP')
return result.x[:self.N], result.x[self.N:]Simulation and Testing
Autonomous vehicle software must be tested billions of miles in simulation before deployment on public roads. CARLA and AirSim are the leading open-source simulators: they provide photorealistic rendering, sensor simulation (camera, LIDAR, radar with configurable noise), traffic simulation, and scenario scripting. The development workflow: design scenario -> run in simulation -> analyze behavior -> improve -> run on closed track -> limited public deployment.
Scenario-based testing covers edge cases: cut-in maneuvers, pedestrians crossing unexpectedly, construction zones, adverse weather, and hardware failures. The ISO 21448 (SOTIF) standard guides testing of functions where the behavior is not fully specified by the requirements.
import carla
import random
client = carla.Client('localhost', 2000)
world = client.get_world()
# Create a test scenario
blueprint_library = world.get_blueprint_library()
vehicle = world.spawn_actor(
blueprint_library.filter('vehicle.*')[0],
carla.Transform(carla.Location(x=0, y=0, z=0.2))
)
# Add a pedestrian that crosses the road unexpectedly
pedestrian = world.spawn_actor(
blueprint_library.filter('walker.pedestrian')[0],
carla.Transform(carla.Location(x=50, y=-10, z=0))
)
pedestrian_control = carla.WalkerControl()
pedestrian_control.speed = 5
pedestrian_control.direction = carla.Vector3D(x=0, y=1, z=0)
pedestrian.apply_control(pedestrian_control)
# Run the simulation and log the ego vehicle's response
while True:
sensors = vehicle.get_sensors()
control = autonomous_stack.run(sensors)
vehicle.apply_control(control)
world.tick()Frequently Asked Questions
What level of autonomy are current vehicles at?
Most production vehicles are Level 2 (partial automation: adaptive cruise + lane centering). Mercedes Drive Pilot is Level 3 (conditional automation) approved in Germany, Nevada, and California. Waymo and Cruise operate Level 4 robo-taxis in limited geofenced areas. Level 5 (full automation) does not exist yet.
What hardware do I need to develop AV software?
Start with simulation (CARLA). For real hardware: a Linux workstation with an NVIDIA GPU (RTX 3080+), a LIDAR (Ouster or Velodyne), cameras (FLIR or Basler), and a CAN bus interface for vehicle control. An NVIDIA Jetson Orin is the standard embedded compute platform.
How does an AV handle sensor failure?
Redundancy is built in: if the primary LIDAR fails, the system degrades to camera+radar. The fault detection module monitors sensor health and triggers a minimal risk condition (MRC): pull over safely, reduce speed, or request remote assistance.
What is the biggest challenge in autonomous driving?
Handling edge cases — the long tail of rare situations the system has never seen. Simulation creates synthetic edge cases, but unknown unknowns (e.g., a mattress falling off a truck, a police officer directing traffic differently from signals) remain the hardest challenge.
Originally published on Ayodhyyya. Last updated June 1, 2026.