Tutorial: Learn AR VR Development from Scratch (2026)
Augmented and virtual reality are reshaping how we interact with digital content, blending virtual objects with the real world or immersing us entirely in digital environments. After shipping three AR applications and a VR training simulation for industrial safety, I have learned that the core challenge is not rendering — it is spatial understanding, user comfort, and interaction design.
This tutorial covers AR development with Unity AR Foundation, ARKit, ARCore, and standalone VR development. You will build a cross-platform AR app that detects surfaces and places virtual objects, then create a VR experience with locomotion and hand interactions.
Unity AR Foundation — Cross-Platform AR
Unity AR Foundation provides a unified API that works across ARKit (iOS) and ARCore (Android). You write your logic once; the underlying platform handles plane detection, point clouds, lighting estimation, and anchor tracking. The key components: AR Session (manages lifecycle), AR Plane Manager (detects surfaces), AR Raycast Manager (hit testing), and AR Anchor Manager (world-locked positions).
Placement works by raycasting against detected planes. When the user taps, you cast a ray from the camera through the screen point and check for plane hits. On hit, instantiate your object at the hit pose. Use estimated lighting to match virtual lighting to the real environment.
using UnityEngine;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;
public class PlacementManager : MonoBehaviour
{
public GameObject objectToPlace;
private ARRaycastManager raycastManager;
void Start() => raycastManager = GetComponent();
void Update()
{
if (Input.touchCount > 0 && Input.GetTouch(0).phase == TouchPhase.Began)
{
List hits = new List();
if (raycastManager.Raycast(Input.GetTouch(0).position, hits, TrackableType.PlaneWithinPolygon))
{
Pose pose = hits[0].pose;
Instantiate(objectToPlace, pose.position, pose.rotation);
}
}
}
} ARKit Specific Features
ARKit (Apple's AR framework) offers advanced capabilities beyond AR Foundation: people occlusion (people occlude virtual objects), motion capture (body tracking without external sensors), and LiDAR-based scene reconstruction on Pro devices. The depth API from LiDAR enables instant plane detection and real-time object scanning.
Face tracking uses the TrueDepth camera to map 52 blend shapes from the user's face. This powers AR filters, avatar puppeteering, and expressive interactions. ARKit's Collaboration Data enables multi-device shared AR experiences.
using UnityEngine.XR.ARKit;
ARFaceManager faceManager = GetComponent();
faceManager.facesChanged += OnFacesChanged;
void OnFacesChanged(ARFacesChangedEventArgs args)
{
foreach (ARFace face in args.added)
{
var blendShapes = face.faceMesh.blendShapes;
float eyebrowRaise = blendShapes[ARKitBlendShapeLocation.EyebrowUpperUpRight].value;
float jawOpen = blendShapes[ARKitBlendShapeLocation.JawOpen].value;
// Map blend shapes to avatar morph targets
}
} ARCore Depth and Cloud Anchors
ARCore (Google's AR platform) provides environmental understanding through its depth API, which creates a depth map of the scene using motion and stereo cues. This enables realistic physics — virtual objects bounce off real surfaces and occlude behind real objects. The Raw Depth API gives full-resolution depth images at 30 FPS.
Cloud Anchors allow multiple devices to share a common AR coordinate system. One host creates an anchor and uploads it; other devices resolve it. This is the foundation for multiplayer AR games and collaborative design reviews across Android and iOS.
using Google.XR.ARCoreExtensions;
ARAnchorManagerExtensions.CloudAnchorMode = CloudAnchorMode.Enabled;
// Host anchor
AnchorComponent hostedAnchor = anchorManager.HostCloudAnchor(anchor, 60);
string cloudAnchorId = hostedAnchor.cloudAnchorId;
// Resolve on other device
AnchorComponent resolvedAnchor = anchorManager.ResolveCloudAnchor(cloudAnchorId);
resolvedAnchor.anchorRequested += (args) => {
if (args.anchor != null)
Instantiate(sharedObject, args.anchor.transform);
};VR Locomotion and Comfort
VR locomotion is the hardest UX challenge in virtual reality. Teleportation is the most comfortable (no motion sickness) but breaks presence. Continuous locomotion is immersive but causes simulator sickness for many users. The middle ground: snap turning (15-45 degree increments), vignetting during movement (reduces peripheral visual flow), and arm-swing locomotion.
Use the XR Interaction Toolkit for locomotion: TeleportationProvider, ContinuousMoveProvider, and SnapTurnProvider, and ClimbProvider. Always give users a choice of locomotion method and let them adjust turning and movement speeds.
using UnityEngine.XR.Interaction.Toolkit;
public class LocomotionSetup : MonoBehaviour
{
public TeleportationProvider teleport;
public ContinuousMoveProvider continuous;
void Update()
{
if (Input.GetButtonDown("Teleport"))
{
continuous.enabled = false;
teleport.enabled = true;
}
}
}
// Enable vignette during movement
continuous.fadeOnMove = true;
continuous.vignetteEaseTime = 0.2f;Hand Tracking and Interactions
Hand tracking replaces controllers with natural gestures. Ultraleap (formerly Leap Motion) and camera-based solutions detect hand skeleton, finger positions, and gestures. Pinch to grab, swipe to scroll, point to indicate, and thumbs-up to confirm. The interaction pattern should feel as natural as possible with clear visual feedback.
For grab interactions, attach a collider to each hand palm and fingertip. When a palm collider overlaps with an object and the user makes a fist, parent the object to the hand. Release on open palm. Add haptic feedback (where available) via vibration.
public class HandGrab : MonoBehaviour
{
public XRHand hand;
private GameObject heldObject;
void OnTriggerEnter(Collider other)
{
if (other.CompareTag("Grabbable") && hand.IsPinching())
{
heldObject = other.gameObject;
heldObject.transform.SetParent(hand.transform);
heldObject.GetComponent().isKinematic = true;
}
}
void OnTriggerExit(Collider other)
{
if (heldObject != null)
{
heldObject.transform.SetParent(null);
heldObject.GetComponent().isKinematic = false;
heldObject = null;
}
}
} Performance Optimization for XR
XR demands 60 FPS (AR) or 72-90 FPS (VR). Every dropped frame causes discomfort. Key optimization targets: keep draw calls under 150 on mobile, use single-pass instanced rendering, bake lighting into lightmaps, use LOD groups, and limit transparent objects. The Unity Profiler with XR tracing is your essential tool.
Use occlusion culling aggressively, merge materials via texture atlasing, and disable shadows on mobile. On Quest devices, use foveated rendering (fixed or eye-tracked) to reduce pixel shader cost in peripheral vision.
// Single-pass instanced rendering (reduces draw calls 50%)
// Player Settings > XR Settings > Stereo Rendering Mode > Single Pass Instanced
// Foveated rendering on Quest
// OVRManager.foveatedRenderingLevel = OVRManager.FoveatedRenderingLevel.High;
// GPU instancing material
MaterialPropertyBlock block = new MaterialPropertyBlock();
for (int i = 0; i < instances.Length; i++)
{
block.SetColor("_Color", colors[i]);
renderers[i].SetPropertyBlock(block);
}Frequently Asked Questions
What is the difference between AR and VR development?
AR overlays digital content on the real world using the device camera (pass-through). VR replaces the real world entirely with a rendered environment viewed through a headset. AR has real-world lighting and surface constraints; VR has full control over the environment but must manage motion sickness.
Do I need a dedicated AR/VR device to develop?
For AR development, you can test with an iOS/Android phone camera. For VR, you need a headset. The Quest 2/3 is the most affordable entry point. Unity's Game View can simulate some VR functionality without a headset, but final testing requires one.
Which platform should I target first?
Start with AR Foundation targeting both iOS and Android. The market for AR is larger (phones are everywhere) and the development cycle is faster. Transition to VR when you have a specific use case that requires immersion.
How do I handle cross-platform differences?
Use Unity AR Foundation as the abstraction layer. Wrap platform-specific features (ARKit depth, ARCore Cloud Anchors) in conditional compilation (#if UNITY_IOS / #elif UNITY_ANDROID). Test on each platform regularly.
Originally published on Ayodhyyya. Last updated June 1, 2026.