AR study #4: Design guidance for AR - ARkit vs ARcore
Updated: Apr 21
For this week, I researched the design guidelines of the AR kit & ARcore. As I'd like to focus on AR product design for my next career, It's important to understand that how accessibilities can be different depends on different environments which is AR(Spatial based) product VS 2D screen-based products. Since I have been leading a design system at the NYU IT department, I was able to work on in-depth study design systems for digital products especially when it comes to responsive design. In addition to my working in the design system, I discovered what I need to more consider when I'm working on AR products in the future.
In many ways, you can sort of think of AR,as having a lot fo the same challenges as responsive design for the web in terms of different window sizes but it's even more complicated because now you have a responsive design for 3D spaces that are the user's actual living room. - Google I/o '18
I researched ARkit's design guideline though Apple's ARkit guideline
<9 Design principles>
#1: Creating an Engaging, Comfortable Experience
To create real-life like environment
Let people use the entire display (Avoid cluttering the screen with controls and information that diminish the immersive experience.)
Strive for convincing illusions when placing realistic objects. (Design detailed 3D assets, reflect environmental lighting conditions/camera grain/shadow, updates sciences 60 times per second to avoid AR object's jump)
Consider how virtual objects with reflective surfaces show the environment.(small or coarse reflective surfaces might help to maintain the illusion that an AR experience)
To enhance the interaction
Use audio and haptics to enhance the immersive experience. (A sound effect or bump sensation is a great way to confirm that interaction between virtual objects with physical surface or other virtual objects.)
Minimize text in the environment. If additional information or controls are necessary, consider displaying them in screen space. (Display only the information that people need for your app experience. It’s typically easy for people to find and view content in screen space because it remains stationary while the underlying AR environment moves with the device.)
Consider using indirect controls when you need to provide persistent controls. (Consider placing the controls so that people don’t have to adjust how they're holding the device to reach them. Also, consider using translucency in indirect control to help avoid blocking the underlying scene.)
To consider the user's physical experience of AR product
Anticipate that people will use your app in a wide variety of real-world environments.(Clearly communicate your app's requirements and expectations to people upfront to help them understand how their physical environment can affect their AR experience.)
Be mindful of people's comfort. (To help avoid causing user's fatigue, consider placing objects at a distance that reduces the need to move the device closer to the object.)
If your app encourages people to move, introduce motion gradually. (Give people time to adapt to the AR experience in your app and then progressively encourage movement.)
Be mindful of people's safety. (Considering user's physical surroundings which can be making rapid, sweeping, or expansive motions might be dangerous.)
#2: Using Coaching to Get People Started
Hide unnecessary app UI while people are using a coaching view.
If necessary, offer a custom coaching experience.
#3: Helping People Place Objects
Show people when to locate a surface and place an object. (After ARKit detects a surface, your app can display a custom visual indicator to show when object placement is possible. You can help people understand how the placed object will look in the environment by aligning your indicator with the plane of the detected surface.)
When people place an object, immediately integrate that object into the AR environment. (For example, if people place an object beyond the bounds of the detected surface, gently nudge the object back onto the surface.)
Consider guiding people toward offscreen virtual objects.
Avoid trying to precisely align objects with the edges of detected surfaces.
Incorporate plane classification information to inform object placement.
Be wary of potentially conflicting gestures.
#4:Designing Intuitive, Delightful Object Interactions
Let people use direct manipulation to interact with objects when possible. (It's more immersive when people can interact with onscreen 3D objects by touching them directly, but if people are moving around for using the app, indirect controls can be work better.)
Let people directly interact with virtual objects using standard, familiar gestures.
In general, keep interactions simple.
Respond to gestures within reasonable proximity of interactive virtual objects.
Support user-initiated object scaling when it makes sense in your app.
Strive for virtual object movement that’s consistent with the physics of your app’s AR environment. (Aim to keep moving objects attached to real-world surfaces and avoid causing objects to jump or vanish and reappear as people resize, rotate, or move them.)
Explore even more engaging methods of interaction. (Gestures aren't the only way for people to interact with virtual objects in AR. Your app can use other factors, like motion and proximity, to bring content to life.)
#5:Designing a Great Multiuser Experience
Consider enabling people's occlusion. When possible, let new participants enter a multiuser AR experience.
#6:Reacting to Real-World Objects (Developer guidance)
When a detected image first disappears, consider delaying the removal of virtual objects that are attached to it. (To help prevent virtual objects from flickering, consider waiting up to one second before fading them out or removing them.)
Limit the number of reference images in use at one time. (Image detection performance works best when ARKit looks for 100 or fewer distinct images in the real-world environment.)
Limit the number of reference images requiring an accurate position. (Updating the position of a reference image requires more resources. Use a tracked image when the image may move in the environment or 0when an attached animation or virtual object is small compared to the size of the image.
#7:Communicating with People
If you must display instructional text, use approachable terminology.
In a three-dimensional context, prefer 3D hints /3D instruction icon.
Make important text readable. (Use screen space to display text used for critical labels, annotations, and instructions. If you need to display text in 3D space, make sure the text faces people and that you use the same type size regardless of the distance between the text and the labeled object.)
If necessary, provide a way to get more information. (Design a visual indicator that fits with your app experience to show people that they can tap for more information.)
Consider using the system-provided coaching view to help people relocalize.
Minimize interruptions if your app supports both AR and non-AR experiences.
Allow people to cancel relocalization. (If people don’t position and orient their device near where it was before an interruption, relocalization continues indefinitely without success. If coaching people to resume their session isn’t successful, consider providing a reset button or other way to restart the AR experience.)
Indicate when the front-facing camera is unable to track a face for more than about half a second.
#9:Suggesting Ways to Resolve Problems
Let people reset the experience if it doesn’t meet their expectations. (Don't force people to wait for conditions to improve or struggle with object placement. Give them away to start over again and see if they have better results.)
Suggest possible fixes if problems occur. (If your app is notified of these problems, use straightforward, friendly language to offer suggestions for resolving them.)
- Presentation by Google I/O18
ARCore works in 2 ways
It tracks the position of your phone in the real world, and it builds an understanding of your surroundings by identifying walls and floors in the environment.
AR core's Key technologies :
Design principles category by
Physical environments (Design one experience for many different spaces - Give users a clear understanding of the amount of space they’ll need for your app.)
Flat surfaces without texture, such as a white desk
Environments with dim lighting
Extremely bright environments
Transparent or reflective surfaces like glass
Dynamic or moving surfaces, such as blades of grass or ripples in the water
Define size (Show users the ideal size and the perfect conditions for your experience.)
Responsive Playspace (Design the experience size to be responsive.)
Encourage movement (When users encounter their first AR experiences, they’ll often forget they can move around. Place a virtual object on the edge of the screen, encouraging them to move toward it.)
Let users know what movements will trigger the app.
Guide them through the types and range of movement possible.
Make easy transitions from one pose or movement to another.
Design for comfort.
Try not to require movement until it’s necessary.
Accessibility (Let the user move and rotate an object in case they can’t physically move around it, and to avoid making them fatigue)
Safety & Comfort
Keep the user safe
Don't make users walk backward
Avoid long play sessions
Take a break
Modeling (Remember, 3D objects can be viewed from all sides. Use complete objects, and be sure to render all surfaces, even those that a user might not immediately see, like the back surface of a curtain, or the bottom of a couch.)
Normal Maps (Ideal for giving objects a photorealistic appearance.)
Ambient Occlusion (A endering technique that controls the amount of shadow that falls on the surface of any object.)
Shadow Planes (A virtual surface that sits beneath your object. The plane only contains shadows, not the objects themselves).
Lighting (There’s a light estimation system built right into ARCore. It’ll let you render lighting conditions that adjust dynamically to the surrounding area.)
Presence (Use shadow, dynamic lighting, and physics to create object presence)
Plane discovery (Show users clearly how to scan for surfaces / Let users know about changes in system status in a timely and relevant way. Using visual highlighting or displaying text to help communicate that a surface has been successfully detected. / Communicate error states clearly and helps users recover quickly)
Optimal placement range (When you create your experience, consider the importance of depth. Think about how users will see your content, and whether or not they’ll be able to access it. Think of the scene in 3 distinct regions: downstage, upstage, and center stage.)
Placement (ARCore lets you drop any virtual object into a real-world space. Objects can be placed either automatically or manually.)
Anchoring (Anchoring is different than simply placing an object in the scene. Ordinary objects can be placed, moved around, and played with. On the other hand, an anchored object will stay locked in place until the user moves it.)
Selection (Let the user select a virtual object in order to identify, manipulate, and interact with it.)
Translation (Translation is the process of moving a virtual object along a surface, or moving it from one surface to another. / Encourage users to move a virtual object from one surface to another. / Add a boundary to show users how far they can move an object.)
Gestures & proximity (Consider the sizing of touch targets to allow for easy interactions.
Precise handling may prove to be challenging to users for objects that are small or distant.) *Two-finger gestures are commonly used to rotate or scale an object.
Initialization (Make a clear transition into AR)
Offscreen exploration (Use visual or audio cues to encourage offscreen exploration.)
Audio exploration (Use audio cues to enhance the user’s experience and encourage engagement.)
Haptic Feedback (Phone vibrations are frequently used to send feedback to users. However, haptic feedback isn’t recommended for AR experiences.In some cases, it can cause AR tracking to be lost.)
Depth collisions (Always consider your users’ spaces.)
Inside Object (Make an object blurry inside when you want to say, "You're not supposed to be here!")
Reset (Allow the user to reset the experience if they need to.)
Multiplayer Experience (A multiplayer experience lets different users share the same AR environment. An object that appears on one user’s device will appear to all users. Multiplayer experiences can require more hand-holding than single-user journeys. Guide your users through each step. Try to make the moment of connection as seamless as possible.)
Interface (Try to interrupt your AR world as little as you can. Get users into the experience, and then get out of the way. Persistent 2D overlays can also disrupt the user’s immersion. It’s a constant reminder that the world they’re looking at isn’t completely real.)
Onboarding & instructions (Provide an onboarding flow within the experience / Guide the user visually / Use familiar UI patterns)
Landscape & portrait (Provide support for both portrait and landscape modes. If this isn’t possible, select the one that’s best for your experience.)
Errors (You can communicate what went wrong, especially if it helps avoid that error in the future. Avoid blaming the user. Focus on getting the user to take the right action.)
Permissions (Clearly tell users why the app needs certain permissions.)
After the investigation of two different AR guidance, I could discover the significant considerations common in both AR kit / ARcore when I need to use them for my future AR product design.
1) The user's safety is first!
2) Avoid making users fatigue by adjusting the overall time of AR displaying, readable text, easy to manipulate Augmented object
3) Give users a clear guide at the onboarding process
4) Give users reset the experience
5) Use sound & haptic interaction to give users notification for transition, or changed status.
( But AR core not recommend haptic interaction due to possibility that it can make lost
6) Make the immersive experience by reflecting real-world environment such as lighting,