6D.ai SDK Release Notes

Release Notes
6D.ai beta SDK v0.22.1
Oct 1, 2019

Introduction

🚩🚩🚩

Happy Tuesday!

This minor release introduces support for the devices announced at the September Apple event: iPhone 11, iPhone 11 Pro, iPhone 11 Pro Max and 10.2" iPad.

Please note that the new iPads embed the 2016 A10 Fusion chip, which is significantly slower than the more recent chips found on the latest iPad Pro, Air and Mini. There are no changes on the neural network side; maps remain compatible with other clients running 0.22.x SDK across iOS and Android.

Changes

Sample Apps

  • Fixed crash in the Drawing sample of the Unity app when saving on iOS 13
  • Fixed mesh alignment issue in the SceneKit app when in portrait mode on iOS 13

Known Issues

The following issues will be addressed in future releases:

Meshing

  • Difficult environments (white walls, glass, mirrors, metallic or shiny surfaces) sometimes produce irregularities like mesh shards flying in mid-air.

Relocalization

  • Outdoor relocalization is currently impacted by the accuracy of GPS on the phone. In situations where the phone GPS varies greatly, relocalization may not occur properly.
  • Relocalization is sensitive to device orientation. E.g. relocalization may not work in portrait mode if the space was mapped in landscape mode.

Memory

  • Each SixDegreesSDK_Initialize() / SixDegreesSDK_Stop() cycle leaks between 2 and 3 megabytes.

Hardware Requirements

Please check our supported devices list for latest devices our SDK is compatible with.

Build Requirements

In order to build software on top of our SDK, please ensure your system matches the following specifications:

  • iPhone 11, iPhone 11 Pro, iPhone 11 Pro Max, iPhone XS Max, iPhone XS, iPhone XR, iPhone X, iPhone 8 Plus, iPhone 8, iPhone 7 Plus, iPhone 7
  • iPad Pro (All generations and sizes), iPad Mini (2019, 5th Gen), iPad Air (2019, 3rd Gen), iPad (2019, 7th Gen), iPad (2018, 6th Gen)
  • iOS 11.4 or newer
  • Xcode 10+
  • Unity3D 2019.2+
  • Unity Hub 2.0+

Please review the 6D.ai SDK Function Definitions and the Meshing API Guide for additional info on SDK functions and how to work with the mesh.


Release Notes
6D.ai beta SDK v0.22.0
Aug 27, 2019

Introduction

🚩🚩🚩

Happy Tuesday!

An incredible amount of behind-the-scenes work kept us busy over the past few months as we refined the technology and laid the groundwork for upcoming improvements and new features.

We are very excited to announce we are now opening sign-ups for our Android SDK private beta. Android SDK v0.22.0 is fully compatible with the iOS SDK v0.22.0 makes it easy to develop multiplayer cross-platform augmented reality experiences with the same persistence, occlusion and physics capabilities offered by 6D.ai.

We will invite a limited number of developers in the coming days and weeks, while we test on various hardware platforms. If your project is waiting on Android, please email details to eric@6d.ai.

Thanks to a new neural network, relocalization is now faster and more accurate, improving the experience of persistence and multiplayer. Mesh coverage also improved.

Please note that because the new neural network isn't backwards-compatible, 0.22.0 doesn't have access to maps created by apps running 0.21.0 and below.

Notes from the lab

Neural Networks

The backbone of 6D.ai's neural network infrastructure was entirely reworked to be faster and more modular, allowing us to run multiple networks efficiently, regardless of the host hardware platform and purpose of the networks.

This means that future updates will run semantic segmentation, contouring people and recognizing objects in the scene, seamlessly along with meshing and other applications, on iOS, Android and any other future device.

AR Cloud

With the release of beta SDK v0.21.0, we introduced our first AR Cloud-based Mesh Fusion, which changes the behavior of SaveToARCloud() by making it additive. In order to help developers with the concept, we wrote an in-depth guide to the AR Cloud's logic.

Changes

Relocalization

  • Successful relocalizations are now significantly more accurate and less sensitive to change in light conditions: the position of persisted objects and loaded mesh are now a lot more precise.
  • Small improvement in processing speed and memory footprint.

Meshing

  • Improvements in coverage: walls and other plain colored matte surfaces mesh more effectively.
  • Reduced GPU footprint by offloading depth fusion to the CPU.
  • Fixed occasional meshing hiccups when scanning large areas.

AR Cloud

  • Upload and download sizes increased by 20 to 30% for equivalent surfaces.
  • Maps are now compatible with the private beta Android SDK.
  • Like the previous release, the change in neural networks is non-backwards compatible, so maps from applications running 0.21.0 and below are invisible to 0.22.0 and vice-versa.

Known Issues

The following issues will be addressed in future releases:

Meshing

  • Difficult environments (white walls, glass, mirrors, metallic or shiny surfaces) sometimes produce irregularities like mesh shards flying in mid-air.

Relocalization

  • Outdoor relocalization is currently impacted by the accuracy of GPS on the phone. In situations where the phone GPS varies greatly, relocalization may not occur properly.
  • Relocalization is sensitive to device orientation. E.g. relocalization may not work in portrait mode if the space was mapped in landscape mode.

Memory

  • Each SixDegreesSDK_Initialize() / SixDegreesSDK_Stop() cycle leaks between 2 and 3 megabytes.

Hardware Requirements

Please check our supported devices list for latest devices our SDK is compatible with.

Build Requirements

In order to build software on top of our SDK, please ensure your system matches the following specifications:

  • iPhone 11, iPhone 11 Pro, iPhone 11 Pro Max, iPhone XS Max, iPhone XS, iPhone XR, iPhone X, iPhone 8 Plus, iPhone 8, iPhone 7 Plus, iPhone 7
  • iPad Pro (All generations and sizes), iPad Mini (2019, 5th Gen), iPad Air (2019, 3rd Gen), iPad (2019, 7th Gen), iPad (2018, 6th Gen)
  • iOS 11.4 or newer
  • Xcode 10+
  • Unity3D 2019.2+
  • Unity Hub 2.0+

Please review the 6D.ai SDK Function Definitions and the Meshing API Guide for additional info on SDK functions and how to work with the mesh.


Release Notes
6D.ai beta SDK v0.21.0
May 28, 2019

Introduction

Happy Tuesday!

After over three months focused on customer projects, we are excited to release a new update to the 6D.ai beta SDK, incorporating both new premium tier features and many improvements to the core technology available to all developers.

Please note that because this SDK introduces a new improved neural network that isn't backwards-compatible, it uses a new separate AR Cloud. Applications using SDK 0.20.x and lower will continue to use the old AR Cloud for loading and saving location map data.

Changes

Along with new premium tier features, this release contains many improvements and changes for all beta developers.

warning Changes are required to code using 6D.ai's Meshing API. Please refer to the Meshing section below.

iPad Support

  • 2019 models of iPad Air (3rd Gen) and iPad Mini (5th Gen) are now supported.

AR Cloud

  • Authentication is now enabled, and the SDK requires your app bundle to contain SixDegreesSDK.plist with the correct credentials in order to initialize. A new state "Disabled" has been added in case the credentials are incorrect.
  • Due to the new more efficient neural network, location relocalization map data is incompatible with previous versions of the SDK and now live in a new instance of the AR Cloud. The AR Cloud instance for versions 0.20.x and lower is still available but there will be no multiplayer and persistence possible between applications running 0.21.0 and above and 0.20.x and lower.
  • Saving is now additive, the SDK will only send to the AR Cloud location map data that was scanned during the session (and since the previous save), greatly reducing the bandwidth footprint of saving.
  • The AR Cloud now runs a process known as Mesh Fusion that straightens, aligns, and stitches location map data. This process corrects some of the tracking error accumulated by the SDK's underlying ARKit tracking and reduces distortions in maps, increasing the consistency of relocalization for persistence and multiplayer.
  • SixDegreesSDK_SaveToARCloud() and SixDegreesSDK_LoadFromARCloud() now return a timestamp to be used as a parameter in API calls to get status, cancel or retry AR Cloud requests.
  • SixDegreesSDK_GetSaveStatus() now takes a save timestamp as parameter and returns save state, save error, upload byte size, and upload progress percentage.
  • There is a new "Packaging" save state for the duration of the file operations previously considered part of "Uploading".
  • Failed saves can be retried with the new API call SixDegreesSDK_RetrySaveToARCloud(), e.g. following a network error.
  • SixDegreesSDK_GetLoadStatus() now takes a load timestamp as parameter and returns load state, load error, download byte size, and download progress percentage.

Meshing

  • Mesh radius has increased from 10m to 100m.
  • Most mesh discontinuities and seams have been fixed.
  • Mesh block size has increased to about 1.4 meters in order to eliminate application-side code that consolidate small blocks into chunks for physics performance. Sample code for Unity and SceneKit has been updated to include simpler mesh controllers.
  • Mesh API names have been updated for consistency. The mesh is known as "Block Mesh" for all beta developers, and is now accessed with SixDegreesSDK_GetBlockMeshInfo() and SixDegreesSDK_GetBlockMesh().
  • warning The vertex buffer has been reorganized to make it easier for mesh controllers to parse it and adapt to various numbers of vertex fields. From now on the vertex buffer contains all position coordinates { x, y, z } back-to-back in the first half of the buffer, and all vertex normal vectors { nx, ny, nz } back-to-back in the second half of the buffer. Sample code has been updated to reflect the change. Please refer to the Meshing API Guide on the dashboard for more information.
  • Due to the new neural network and performance optimizations, mesh updates should occur more frequently. Applications using the mesh for physics should consider throttling collision mesh updates. Updates for rendering and occlusions can be as frequent as possible.

Headers & Feature Tiers

  • SixDegreesSDK_advanced.h has been folded into SixDegreesSDK.h. We provide those API calls for expert developers who need specific ARKit features and recommend other developers use our standard API.
  • SixDegreesSDK_premium.h has been added to expose functionality open to our partners and premium tier customers. If your project needs those features, please contact sales@6d.ai.
  • warning Premium features are currently only available for customers working directly with 6D.ai on high profile custom projects, broadly defined as over $100k per annum and/or with exposure to many millions of end-users. If your project qualifies, please contact sales@6d.ai.
  • We are continuing to define and evolve our product, and plan to make some of these premium features available under a self-service model at a lower price point within the next year.

Unity Sample App

  • A bug in most recent versions of Unity 2018.3 creating a duplicate SixDegreesSDK.framework file in the generated Xcode project has been fixed.
  • The new recommended Unity version is 2018.3.14f1.
  • The drawing sample has been updated to use the Mirror library for local network peer-to-peer instead of the now-deprecated Unity Networking framework.
  • Cancel buttons have been added to the user interface to cancel SaveToARCloud() and LoadFromARCloud()
  • A debug toggle button has been added to the user interface to expose the new GetSaveStatus() and GetLoadStatus() values.
  • Collider mesh updates now happen every 10 mesh rendering updates in order to reduce their share of CPU usage.

SceneKit Sample App

  • Swift version was updated to 5.0.
  • Physics mesh updates now happen every 10 mesh rendering updates in order to reduce their share of CPU usage.

Known Issues

The following issues will be addressed in future releases:

Meshing

  • Difficult environments (white walls, metallic, reflective or transparent surfaces) sometimes produce irregularities like mesh shards flying in mid-air.

Relocalization

  • Outdoor relocalization is currently impacted by the accuracy of GPS on the phone. In situations where the phone GPS varies greatly, relocalization may not occur properly.

Memory

  • Each ​SixDegreesSDK_Initialize()​/​SixDegreesSDK_Stop()​ cycle leaks between 2 and 5 megabytes.

Hardware Requirements

Please check our supported devices list for latest devices our SDK is compatible with.

Build Requirements

In order to build software on top of our SDK, please ensure your system matches the following specifications:

  • iPhone 11, iPhone 11 Pro, iPhone 11 Pro Max, iPhone XS Max, iPhone XS, iPhone XR, iPhone X, iPhone 8 Plus, iPhone 8, iPhone 7 Plus, iPhone 7
  • iPad Pro (All generations and sizes), iPad Mini (2019, 5th Gen), iPad Air (2019, 3rd Gen), iPad (2019, 7th Gen), iPad (2018, 6th Gen)
  • iOS 11.4 or newer
  • Xcode 10+
  • Unity3D 2019.2+
  • Unity Hub 2.0+

Please review the 6D.ai SDK Function Definitions and the Meshing API Guide for additional info on SDK functions and how to work with the mesh.


Release Notes
6D.ai beta SDK v0.20.0
Feb 5, 2019

Introduction

Happy Tuesday!

More and more developers request the ability to start and stop the 6D.ai beta SDK within a larger app. It is now possible with the introduction of two API methods in this release: SixDegreesSDK_GetState()​ and ​SixDegreesSDK_Stop()​.

We also included a number of small improvements, described below.

Be sure to review Sample Apps and Documentation options on the 6D Developer Portal to learn more about how to install and use the sample code and 6D SDK.

Changes

New Features

  • SixDegreesSDK_GetState()​ returns the current state of the SDK: SixDegreesStateStopped​, ​SixDegreesStateInitializing​ and SixDegreesStateRunning​.
  • SixDegreesSDK_IsInitialized()​ is now deprecated and can be replaced with SixDegreesSDK_GetState() == SixDegreesStateRunning​.
  • SixDegreesSDK_Stop() ​interrupts the ARKit and 6D.ai sessions, cancels pending transactions with the AR Cloud, and releases the associated resources.

Bug Fixes

  • Fixed crashes occurring when calling ​SixDegreesSDK_Cancel()​ during certain transactions with the AR Cloud.

Improvements

  • Limited the visibility of method symbols in the static library in order to make it easier to link the 6D.ai framework concurrently with libraries like zlib.
  • Optimizations to the Metal pipeline.
  • Renamed the SceneKit sample app in line with the Unity sample app, added button to toggle the mesh visibility.
  • Cleanup of some Unity sample app warnings that appeared in 2018.3

Known Issues

The following issues require work under the hood and will be addressed in future releases:
  • Meshing:​ difficult environments (white walls, metallic, reflective or transparent surfaces) sometimes produce irregularities like mesh shards flying in mid-air.
  • Relocalization:​ when loading large multi-room locations from the AR Cloud, the relocalization phase can incur high resource consumption and framerate slowdown or freezes.
  • Memory: ​each ​SixDegreesSDK_Initialize()​/​SixDegreesSDK_Stop()​ cycle leaks between 2 and 4 megabytes.

Hardware Requirements

As of SDK version 0.20.0, the following iOS devices are supported:
Year
 
Supported iPhones
Identify your iPhone model
Supported iPads
Identify your iPad model
2018 iPhone XS
iPhone XS Max
iPhone XR
iPad Pro 12.9" (3rd Gen)
iPad Pro 11"
iPad 9.7" (6th Gen)
2017 iPhone X
iPhone 8
iPhone 8 Plus
iPad Pro 12.9" (2nd Gen)
iPad Pro 10.5"
2016 iPhone 7
iPhone 7 Plus
iPad Pro 9.7"
2015 iPad Pro 12.9" (1st Gen)
Not supported * (no plans to add support):
  • iPhone 6S (2015) and lower
  • iPad 5th Gen (2017) and lower
  • iPad Air family
  • iPad Mini family
* The SDK ​will not initialize​​ on those devices, even if they support ARKit. The API method SixDegreesSDK_IsDeviceSupported() can be used at runtime to detect if the device is supported by the SDK.

System Requirements to Build Sample Apps

  • iPhone XS Max, iPhone XS, iPhone XR, iPhone X, iPhone 8 Plus, iPhone 8, iPhone 7 Plus, iPhone 7
  • iPad Pro (All generations and sizes), iPad 2018 (6th Gen)
  • iOS 12 or iOS 11.4+
  • Xcode 10+
  • Unity3D 2018.2+

Please review the 6D.ai SDK Function Definitions and the Meshing API Guide for additional info on SDK functions and how to work with the mesh.


Release Notes
6D.ai beta SDK v0.19.2
Jan 3, 2019

Introduction

Happy New Year! 🥂

This release introduces a fix to a crash developers were experiencing initializing with Metal on iPhone 7, iPhone 7 Plus, iPad Pro (12.9" 1st and 2nd Gen, 9.7", 10.5") and iPad (6th Gen).

Additionally, we are releasing two new sample applications showcasing how to integrate additional ARKit features with the 6D.ai SDK.

Please refer to the previous 0.19.1 release notes and Unity upgrade guide on the beta developer portal for more information about the recent API changes.


Release Notes
6D.ai beta SDK v0.19.1
December 18, 2018

Introduction

Happy Holidays!

Accessing and configuring the underlying ARKit session to our iOS SDK has been a popular developer request from the beginning. We are excited to make it available to you today!

Because of the complexity of transforming ARKit coordinated to relocalized 6D coordinates, we are changing our 6D convention from Unity-style left-handed to ARKit-style right-handed. The sample code has been updated to reflect this change (adding conversion in Unity, taking it out in SceneKit). Your custom code written on top of our API, if any, might need updating.

This last release of 2018 also brings improvements in meshing accuracy and relocalization precision. On the sample app side, we now fully support up to 60 frames per second, greatly improving user immersion.

Be sure to review Sample Apps and Documentation options on the 6D Developer Portal to learn more about how to install and use the sample code and 6D SDK.

warning If you are upgrading from a previous version of our 6D Unity Package, please review our upgrade guide to v0.19.1

Changes

Changed API Methods

  • bool SixDegreesSDK_Initialize(void)
    now returns false if the SDK will not initialize. Details will be found in the logs (it usually is a configuration error).
  • bool SixDegreesSDK_InitializeWithEAGL(void* eaglContext)
    similarly now returns a bool.
  • int SixDegreesSDK_GetPose(float* poseDataOut, int bufferSize)
    now returns the pose in right-handed world coordinates (flipping the Z axis from the previously used Unity coordinate convention).
  • int SixDegreesSDK_GetMeshBlocks(int* blockBuffer, float* vertexBuffer, int* faceBuffer, int blockBufferSize, int vertexBufferSize, int faceBufferSize)
    similarly now returns the block coordinates, vertex coordinates and vertex normals in right-handed world coordinates.

New Expert API Methods

A new header in SixDegreesSDK.framework is now available: SixDegreesSDK_advanced.h

New API methods in this header are designed to provide configuration options and access to the underlying ARKit session, allowing options such as light estimation, plane detection, image detection, buffer access, etc. Because of potential issues in terms of performance, coordinate system complexities, etc. we only recommend those APIs for expert developers.

  • bool SixDegreesSDK_InitializeWithConfig(ARWorldTrackingConfiguration* configuration)
    allows enabling additional ARKit features. The SDK imposes constraints on the configuration, and InitializeWithConfig() will only return true if:
    • Autofocus is disabled
    • World alignment is ARWorldAlignmentGravity
    • The video format aspect ratio is 16:9
  • ARSession* SixDegreesSDK_GetARKitSession(void)
    returns the ARSession object managed by the 6D SDK. We recommend using that object to access values in the current ARFrame, but do not set a different delegate object or use start, pause and stop controls.
  • void SixDegreesSDK_GetARKitTransform(float* transformDataOut, int bufferSize)
    returns the coordinate transform from the native ARKit coordinate system to the 6D world coordinate system. This will be an identity matrix before relocalization. Use that transform to adapt coordinantes, vectors and transforms to the mesh and pose provided by 6D.

Right-handed Coordinates Conventions

Sample apps have been updated to reflect the change in coordinates conventions in the SDK methods. If you use a custom script in your project on top of our API, e.g. in Unity, make sure that you update it to reflect that change. If you use our stock classes, no change should be necessary.

To flip the Z axis in a transform such as a camera pose, use the following matrix Q:

1.0 0.0 0.0 0.0
0.0 1.0 0.0 0.0
0.0 0.0 -1.0 0.0
0.0 0.0 0.0 1.0

The same matrix Q is used to go from 6D to Unity and back. Please note that the math to change a pose P is:
Punity = Q6D→unity * P6D * Qunity→6D

You can avoid using matrix multiplications in your code by just flipping the sign of the right matrix indices; you'll notice this is what our sample code does:

m11 m21 -m31 x
m12 m22 -m32 y
-m13 -m23 m33 -z
0.0 0.0 0.0 1.0

Technology Improvements

Keep in mind that relocalization accuracy and success rate depend considerably on the computer-vision-friendliness of the environment: detailed, non-repetitive textures; stable and sufficient lighting; non-reflective materials; and absence of motion.

Meshing quality is also sensitive to those factors but tends to perform well in a larger variety of environments.

  • Relocalization is now more accurate, with a median error half the size it used to be. This also means that some false positives are eliminated, potentially increasing the average time to relocalization success.
  • Meshing is less noisy again, comparable to 0.15.x levels.
  • Background texture updates now run at 60 FPS.

Unity Package and Sample App

  • SDMesh.cs changed to use a mesh chunk prefab instead of programmatically adding components to an empty game object. This means you can just swap the prefab in your scenes if you wish to use different components. Mesh chunks have their own dedicated layer (User Layer 9). The mesh now updates at a fixed interval (0.02 sec by default).
  • Some prefab names changed to be more explicit. If the link breaks in your project, look for AR Background and AR Scene.
  • The AR Background now has its own dedicated layer (User Layer 8), separate from the UI layer (Builtin Layer 5). This was a popular request from developers using 3D UI. The Render Queue order of the background's material was also reduced to 1990.
  • The Sample App now targets a smooth 60 FPS instead of the Unity default 30 FPS.

Known Issues

  • Framerate drops when attempting relocalization in highly detailed environments, which is made more obvious by the application otherwise running at 60 FPS. Optimizations will address this in future releases.
  • ARKit tracking drift makes the previously scanned mesh look offset after a certain amount of motion. Continuous relocalization will address this in future releases.

Hardware Requirements

We are actively working on Android support and look forward to sharing exciting updates in a near future! For now, we are supporting the following iOS devices.
Year
 
Supported iPhones
Identify your iPhone model
Supported iPads
Identify your iPad model
2018 iPhone XS
iPhone XS Max
iPhone XR
iPad Pro 12.9" (3rd Gen)
iPad Pro 11"
iPad 9.7" (6th Gen)
2017 iPhone X
iPhone 8
iPhone 8 Plus
iPad Pro 12.9" (2nd Gen)
iPad Pro 10.5"
2016 iPhone 7
iPhone 7 Plus
iPad Pro 9.7"
2015 iPad Pro 12.9" (1st Gen)
Not supported * (no plans to add support):
  • iPhone 6S (2015) and lower
  • iPad 5th Gen (2017) and lower
  • iPad Air family
  • iPad Mini family
* The SDK ​will not initialize​​ on those devices, even if they support ARKit. The API method SixDegreesSDK_IsDeviceSupported() can be used at runtime to detect if the device is supported by the SDK.

System Requirements to Build Sample Apps

  • iPhone XS Max, iPhone XS, iPhone XR, iPhone X, iPhone 8 Plus, iPhone 8, iPhone 7 Plus, iPhone 7
  • iPad Pro (All generations and sizes), iPad 2018 (6th Gen)
  • iOS 12 or iOS 11.4+
  • Xcode 10+
  • Unity3D 2018.2+

Please review the 6D.ai SDK Function Definitions and the Meshing API Guide for additional info on SDK functions and how to work with the mesh.


Release Notes
6D.ai beta SDK v0.18.0
December 4, 2018

Introduction

Happy Tuesday! Today we are introducing SDK version beta 0.18.0, bringing Metal 2.0 support on iOS and making it the default graphics API. OpenGL ES 3.0 will still be supported for compatibility, for as long as it is supported on iOS. Since it was deprecated by Apple earlier this year, we recommend adopting Metal.

The sample apps (Unity and SceneKit) have been updated to run on Metal, but can still be configured to run on OpenGL ES with a simple configuration change. Upgrading from 0.17.0 will be easy and in most cases require no changes, e.g. our Unity C# scripts will auto-detect Metal or OpenGL ES and call the corresponding 6D.ai SDK initialization method.

Be sure to review Sample Apps and Documentation options on the 6D Developer Portal to learn more about how to install and use the sample code and 6D SDK.

Changes

New API methods for Metal

  • void SixDegreesSDK_Initialize(void)
    Initializes the 6D.ai SDK to run with Metal 2.0, the default and recommended graphics API for iOS.
  • void* SixDegreesSDK_GetBackgroundTexture(void)
    Returns the MTLTexture pointer of the camera background. Will return NULL if the SDK is not initialized or was initialized with OpenGL ES. Follow the snippets below to go from void* to MTLTexture.

    Swift:
    if let texPtr = SixDegreesSDK_GetBackgroundTexture() {
    let mtlTexture = Unmanaged<MTLTexture>.fromOpaque(texPtr).takeUnretainedValue()
    }
    Objective-C:
    void* texPtr = SixDegreesSDK_GetBackgroundTexture();
    if (texPtr != nil) {
    id<MTLTexture> mtlTexture = (__bridge id<MTLTexture>)texPtr;
    }

New API methods for OpenGL ES

  • void SixDegreesSDK_InitializeWithEAGL(void* eaglContext)
    Initializes the 6D.ai SDK to run with OpenGL ES 3.0, identical to the former API call void SixDegreesSDK_Initialize(void* eaglContext) from earlier versions.
  • int SixDegreesSDK_GetEAGLBackgroundTexture(void)
    Returns the OpenGL ES texture handle of the camera background, identical to the former API call int SixDegreesSDK_GetBackgroundTexture(void) from earlier versions. Will return 0 if the SDK is not initialized or was initialized with Metal.
warningIf you are upgrading from beta 0.17.0 and wish to continue using OpenGL ES, make sure your code calls those new API methods.

Switching between graphics APIs in the Sample apps

Unity:
Go to the Build Settings... (⇧⌘B) menu and click Player Settings.... In the panel that opens, find Settings for iOS > Other Settings. Ticking "Auto Graphics API" will select Metal; unticking it lets you manually choose between "Metal" and "OpenGLES3", whichever is on top will be the selected graphics API.
SceneKit:
Open the HexPitSample project settings, select the HexPitSample target, and in the General tab find Deployment Info > Main Interface. Selecting "Metal.storyboard" or "OpenGL.storyboard" will instantiate a different SixDegreesView implementation for the namesake graphics API.

Sample apps fixes

  • Fixed an issue in SDMesh.cs (Unity Sample App) and MeshController.swift (Swift/SceneKit Sample App) where part of the update code was inefficient and slowing down the app after a few minutes.
  • Fixed a crash in AutoNetworkManager.cs (Unity Sample App) when no action is attached to OnConnectionEvent.
  • Made the animation of multiplayer camera position updates in the Drawing Sample / Photon Drawing Sample of the Unity Sample App a lot smoother.

Hardware Requirements

We are actively working on Android support and look forward to sharing exciting updates in a near future! For now, we are supporting the following iOS devices.
Year
 
Supported iPhones
Identify your iPhone model
Supported iPads
Identify your iPad model
2018 iPhone XS
iPhone XS Max
iPhone XR
iPad Pro 12.9" (3rd Gen)
iPad Pro 11"
iPad 9.7" (6th Gen)
2017 iPhone X
iPhone 8
iPhone 8 Plus
iPad Pro 12.9" (2nd Gen)
iPad Pro 10.5"
2016 iPhone 7
iPhone 7 Plus
iPad Pro 9.7"
2015 iPad Pro 12.9" (1st Gen)
Not supported * (no plans to add support):
  • iPhone 6S (2015) and lower
  • iPad 5th Gen (2017) and lower
  • iPad Air family
  • iPad Mini family
* The SDK ​will not initialize​​ on those devices, even if they support ARKit. The API method SixDegreesSDK_IsDeviceSupported() can be used at runtime to detect if the device is supported by the SDK.

System Requirements to Build Sample Apps

  • iPhone XS Max, iPhone XS, iPhone XR, iPhone X, iPhone 8 Plus, iPhone 8, iPhone 7 Plus, iPhone 7
  • iPad Pro (All generations and sizes), iPad 2018 (6th Gen)
  • iOS 12 or iOS 11.4+
  • Xcode 10+
  • Unity3D 2018.2+

Please review the 6D.ai SDK Function Definitions and the Meshing API Guide for additional info on SDK functions and how to work with the mesh.


Release Notes
6D.ai beta SDK v0.17.0
November 13, 2018

Introduction

Happy Tuesday!

We are extremely proud to introduce SDK version 0.17.0 today, bringing much needed robustness and performance improvements to the mesh saving and loading features introduced by the experimental version 0.16.0.

Please note that the addition of scene mesh geometry and changes to location map data makes 0.17.0 persistence and multiplayer incompatible with previous versions. A lot of our work roadmap to production focuses on server-side data stewardship in order to prevent map overwriting, reduce bandwidth usage, and enable scalability and cross version compatibility.

Be sure to review Sample Apps and Documentation options on the 6D Developer Portal to learn more about how to install and use the sample code and 6D SDK.

Changes

Performance:
  • Improved framerate and mesh extension speed when scanning large meshes.
  • Reduced screen freezes when loading large meshes after successful relocalization.
  • Reduced sporadic screen freezes when building large meshes.

Relocalization:

  • Increased success rate of cross-model relocalization (e.g. iPhone X relocalizing on map made on iPhone 8).
  • Improved accuracy of relocalizations.

Meshing:

  • Reduced gaps between the loaded mesh after successful relocalizations and real-time mesh extensions.

Relocalization with saved mesh data:
Starting with SDK version 0.17.0, relocalization and loading of previously saved meshes is now possible. A few notes about how this system currently works so that developers understand how to use it. We still have many improvements to come around mesh merging, and will update release notes as functionality is updated.

  1. It is important to always start an app session by downloading any currently saved meshes via SixDegreesSDK_LoadFromARCloud(). The app will automatically attempt to relocalize across all saved maps that exist for your current location. This process will timeout after 30 secs, but you can stop it early with SixDegreesSDK_CancelLoad() if needed.
    1. After a successful relocalization, you will notice that any mesh you had been building locally is cleared from memory and the downloaded mesh corresponding to your current location is drawn. The SDK will immediately start to add to this mesh as you move around.
    2. Skip to step 3 if relocalization doesn’t succeed, map data is not available at your location, or if you just want to start from scratch.
  2. After you have built up some additional mesh data, use SixDegreesSDK_SaveToARCloud() to add your newly collected mesh to the stored maps in the 6D Reality Platform. Note that map data size increases linearly with usage, making file transfer more significant when calling SixDegreesSDK_LoadFromARCloud() / SixDegreesSDK_SaveToARCloud() - essentially each map you upload is saved, and the database of all saved maps is transferred back and forth with each upload/download action.
  3. warning During an app session, if you upload before you have downloaded or relocalized, your upload will be used to replace any existing stored maps for your location. This functionality is added to enable developers to easily clean out old saved maps during testing and will not function this way once we get to a general release candidate.

Hardware Requirements

We are actively working on Android support and look forward to sharing exciting updates in a near future! For now, we are supporting the following iOS devices.
Year
 
Supported iPhones
Identify your iPhone model
Supported iPads
Identify your iPad model
2018 iPhone XS
iPhone XS Max
iPhone XR
iPad Pro 12.9" (3rd Gen)
iPad Pro 11"
iPad 9.7" (6th Gen)
2017 iPhone X
iPhone 8
iPhone 8 Plus
iPad Pro 12.9" (2nd Gen)
iPad Pro 10.5"
2016 iPhone 7
iPhone 7 Plus
iPad Pro 9.7"
2015 iPad Pro 12.9" (1st Gen)
Not supported * (no plans to add support):
  • iPhone 6S (2015) and lower
  • iPad 5th Gen (2017) and lower
  • iPad Air family
  • iPad Mini family
* The SDK ​will not initialize​​ on those devices, even if they support ARKit. The API method SixDegreesSDK_IsDeviceSupported() can be used at runtime to detect if the device is supported by the SDK.

System Requirements to Build Sample Apps

  • iPhone XS Max, iPhone XS, iPhone XR, iPhone X, iPhone 8 Plus, iPhone 8, iPhone 7 Plus, iPhone 7
  • iPad Pro (All generations and sizes), iPad 2018 (6th Gen)
  • iOS 12 or iOS 11.4+
  • Xcode 10+
  • Unity3D 2018.2+

Please review the 6D.ai SDK Function Definitions and the Meshing API Guide for additional info on SDK functions and how to work with the mesh.


6D.ai beta SDK v0.16.0 (experimental)

Release Notes
6D.ai SDK v0.16.0
September 28, 2018

Introduction

After many weeks in the making, we are proud to introduce an experimental SDK version 0.16.0, with for the first time mesh persistence. Mesh persistence is the last basic building block of our technology and finally enables experiences combining multiplayer and meshing using the 6D.ai SDK.

Significant reworks were necessary to make this feature possible, so we are releasing this version as an experimental preview while still supporting version 0.15.1, released earlier this week. We expect to return to more frequent SDK releases moving forward, as we optimize performance and accuracy.

Please note that the addition of scene mesh geometry to location map data makes 0.16.0 persistence and multiplayer separate from 0.15.1.

In other news, finalists for the first round of the 6Demo-God Challenge have been selected, but it's not too late to submit new pitches for demos, as another round is in the works!

More information at 6D.ai Demo Challenge

Hardware Requirements

The following devices are supported by the 6D.ai SDK

  • iPhone 7 and newer
  • iPad Pro (all models)
  • iPad 2018 (6th gen).

The API method SixDegreesSDK_IsDeviceSupported() can be used at runtime to detect if the device is supported by the SDK.

Android support is work in progress, we hope to share exciting updates in a near future!

Changes

API Changes:

  • Introduced SixDegreesSDK_IsDeviceSupported(), in replacement of SixDegreesSDK_HasRealTimeMesh().
  • SixDegreesSDK_Initialize() will fail if the device is not supported.

Mesh Persistence:

  • Successful relocalizations will now also cause the mesh of the immediate surroundings to be loaded.
  • That imported mesh gets extended and updated as you move the device around, as if it had been generated from scratch by your device.

Known Issues

These issues will be addressed in upcoming releases.

Performance:

  • Framerate and mesh extension speed decrease when scanning very large meshes.
  • Screen freeze when loading large meshes after successful relocalization.

Relocalization:

  • Cross-model relocalization (e.g. iPhone X relocalizing on map made on iPhone 8) succeeds less often than in the previous version 0.15.1.
  • Relocalization struggles if light conditions change a lot (e.g. different times of the day).

Meshing:

  • Gaps appear in the loaded mesh after successful relocalization as it makes way for the real-time generated mesh.

AR Cloud:

  • Map data size increases linearly with usage, making file transfer more significant when calling SixDegreesSDK_LoadFromARCloud() / SixDegreesSDK_SaveToARCloud().

Release Notes
6D.ai beta SDK v0.15.2
November 6, 2018

Introduction

Happy Tuesday!

Today we introduce SDK version 0.15.2, now available for the latest iPad Pro Gen 3 devices announced by Apple last week.

Our recent efforts have been focused on improving the precision and reliability of relocalization in the 0.16.0 experimental build branch, as well as improving cross-device compatibility. Now that this is done, we are ironing out the SDK's stability so we can roll out a new SDK version soon!

Be sure to review Sample Apps and Documentation options on the 6D Developer Portal to learn more about how to install and use the sample code and 6D SDK.

Hardware Requirements

Android support is still work in progress, we hope to share exciting updates in a near future! For now, we are supporting the following iOS devices.
Chip
 
Supported iPhones
Identify your iPhone model
Supported iPads
Identify your iPad model
A12X iPad Pro - 3rd Generation
A12 iPhone XR, XS, XS Max
A11 iPhone X, 8, 8 Plus
A10X iPad Pro - 2nd Generation
A10 iPhone 7, 7 Plus iPad 2018 - 6th Generation
A9X iPad Pro - 1st Generation
Not supported (no plans to add support):
  • iPhone 6S and lower
  • iPad 5th Gen (2017) and lower
  • iPad air family
  • iPad mini family

System Requirements to Build Sample Apps

  • iPhone XS Max, iPhone XS, iPhone XR, iPhone X, iPhone 8 Plus, iPhone 8, iPhone 7 Plus, iPhone 7
  • iPad Pro (All generations and sizes), iPad 2018 (6th Gen)
  • iOS 12 or iOS 11.4+
  • Xcode 10+
  • Unity3D 2018.2+

Please review the 6D.ai SDK Function Definitions and the Meshing API Guide for additional info on SDK functions and how to work with the mesh.


Release Notes
6D.ai beta SDK v0.15.1
September 25, 2018

Introduction

Happy Tuesday!

Today we introduce SDK version 0.15.1, now available for iPhone XR, XS and XS Max devices.

Our efforts are focused towards our next major milestone, and large amounts of changes under the hood to support compelling mesh persistence. Stay tuned for more updates on our Slack channel!

In other news, finalists for the first round of the 6Demo-God Challenge have been selected, but it's not too late to submit new pitches for demos, as another round is in the works!

More information at 6D.ai 6demo-god-challenge

Hardware Requirements

Android support is still work in progress, we hope to share exciting updates in a near future! For now, we are supporting the following iOS devices.
Chip
 
Supported iPhones
Identify your iPhone model
Supported iPads
Identify your iPad model
A12 iPhone XS Max, XS, XR
A11 iPhone X, 8, 8 Plus
A10X iPad Pro (2nd Gen)
A10 iPhone 7, 7 Plus iPad 2018 (6th Gen)
A9X iPad Pro (1st Gen)
Not supported (no plans to add support):
  • iPhone 6S and lower
  • iPad 5th Gen (2017) and lower
  • iPad air family
  • iPad mini family

System Requirements to Build Sample Apps

  • iPhone XS Max, iPhone XS, iPhone XR, iPhone X, iPhone 8 Plus, iPhone 8, iPhone 7 Plus, iPhone 7
  • iPad Pro (All generations and sizes), iPad 2018 (6th Gen)
  • iOS 12 or iOS 11.4+
  • Xcode 10+
  • Unity3D 2018.2+

Please review the 6D.ai SDK Function Definitions and the Meshing API Guide for additional info on SDK functions and how to work with the mesh.


Release Notes
6D.ai beta SDK v0.15.0
August 28, 2018

Introduction

Happy Tuesday!

We have a number of cool new things in store for SDK version 0.15.0.

API:
  • LoadFromARCloud() will fail after 30 seconds if relocalization doesn't succeed.
  • New functions CancelSave() and CancelLoad() interrupt SaveToARCloud() and LoadFromARCloud(), respectively.
  • GetMeshBlocks() now returns vertex normals, see the Meshing API Guide.
Core Technology:
  • Large mesh gaps no longer appear when scanning the same surface twice.
  • Mesh triangles tend to appear at a greater distance from the camera.
Sample Apps:
  • SceneKit: Improved Mesh rendering and ported Unity's Mesh chunk logic to Swift.
  • Unity Sample App: project folders have been rearranged to separate 6D SDK files and 6D Examples (such as Sample Scenes).
  • Unity Sample App: New scene to show Photon integration with the Drawing Sample.
  • By popular demand, the SDK is now also available as a Unity Package.

Hardware Requirements

Android support is still work in progress, we hope to share exciting updates in a near future! For now, we are supporting the following iOS devices.
Chip
 
Supported iPhones
Identify your iPhone model
Supported iPads
Identify your iPad model
A11 iPhone X, 8, 8 Plus
A10X iPad Pro (2nd Gen)
A10 iPhone 7, 7 Plus iPad 2018 (6th Gen)
A9X iPad Pro (1st Gen)
Not supported (no plans to add support):
  • iPhone 6S and lower
  • iPad 5th Gen (2017) and lower
  • iPad air family
  • iPad mini family

System Requirements to Build Sample Apps

  • iPhone 7, iPhone 7 Plus, iPhone 8, iPhone 8 Plus, or iPhone X
  • iPad Pro (All generations and sizes), iPad 2018 (6th Gen)
  • iOS 11.4+ (iOS 12 beta 11 ok)
  • Xcode 9.4.1+ (10.0 beta 6 ok)
  • Unity3D 2018.2+

Please review the 6D.ai SDK Function Definitions and the Meshing API Guide for additional info on SDK functions and how to work with the mesh.


Release Notes
6D.ai beta SDK v0.14.0
August 20, 2018

Introduction

Happy Monday!

We are very pleased to introduce SDK version 0.14.0.

In this iPad-centric release, we expand full SDK support (including meshing) to first-generation iPad Pro powered by the A9X chip. Unfortunately, poor performance on iPhone 6S and 6S Plus powered by the similar A9 chip means these will remain unsupported.

Chip
 
Supported iPhones
Identify your iPhone model
Supported iPads
Identify your iPad model
A11 iPhone X, 8, 8 Plus
A10X iPad Pro (2nd Gen)
A10 iPhone 7, 7 Plus iPad 2018 (6th Gen)1
A9X iPad Pro (1st Gen)

1 A bug preventing the SDK from initializing on iPad 6th Gen (2018) running the iOS 12 beta has been fixed.

Changes

  • We now include an option in the Unity sample app for the video background to fill or fit the screen. Fill is now the default option, so you can enjoy letterbox-free meshing!
  • The SceneKit sample app sees some light cosmetic improvements.
  • An error that occured during some instances when switching between sample scenes with the Unity app has been fixed

System Requirements

  • iPhone 7, iPhone 7 Plus, iPhone 8, iPhone 8 Plus, or iPhone X
  • iPad Pro (All generations and sizes), iPad 2018 (6th Gen)
  • iOS 11.4+ (iOS 12 beta 5 ok)
  • Xcode 9.4.1+ (10.0 beta 5 ok)
  • Unity3D 2018.2+

API Changes

No changes to API from previous release

Please review the 6D.ai SDK Function Definitions and the Meshing API Guide for additional info on SDK functions and how to work with the mesh.


Release Notes
6D.ai beta SDK v0.13.0
August 14, 2018

Introduction

Happy Tuesday!

We are very pleased to introduce SDK version 0.13.0.

We have made some significant enhancements to our neural networks and meshing algorithms which resulted in speedups beyond what we expected. This means great news for many beta developers, as this update does something we didn't think was possible:

We're bringing full support, including real-time meshing, to the iPhone 7 family and the latest iPad generation (iPad Pro 12.9", iPad Pro 10.5", and iPad 6th Gen), all powered by the A10/A10X Fusion chip. According to Fiksu.com's iOS analytics, we now support over 50% of active iPhone devices.

The Apple website provides a handy guide to identify your iOS device. If you're not sure which iPhone or which iPad you have, follow one of the following links:

Note: the previously known issue of meshing slowing down and pausing after a couple of minutes of continuous meshing has been greatly improved, but we’re still seeing some glitches.

In addition, we're now maintaining a new sample app, built with SceneKit and Swift, thanks to our awesome beta developer Armands Lazdiņš.

System Requirements

  • iPhone 7, iPhone 7 Plus, iPhone 8, iPhone 8 Plus, or iPhone X
  • iPad Pro 12.9in (2nd Gen), iPad Pro 10.5in, iPad (6th Gen)
  • iOS 11.4+ (iOS 12 beta 5 ok)
  • Xcode 9.4.1+ (10.0 beta 5 ok)
  • Unity3D 2018.2+

API Changes

No changes to API from previous release

Please review the 6D.ai SDK Function Definitions and the Meshing API Guide for additional info on SDK functions and how to work with the mesh.


Release Notes
6D.ai beta SDK v0.12.0
August 7, 2018

Introduction

Happy Tuesday!

We are very pleased to introduce SDK version 0.12.0. This SDK update brings significant improvements to our meshing engine, memory utilization, and introduces multi-scene meshing, where meshes not in the current viewing area are segmented away to preserve system resources.

We’ve also simplified our Sample Apps, by combining them into a single Sample App! This means you’ll spend less time building and more time exploring what we’ve created. The sample app, and SDK now also support portrait mode!

System Requirements

  • iOS 11.4+ (iOS 12 beta 3 ok)
  • Xcode 9.4.1+ with ARKit 1.5+ support (10.0 beta 3 ok)
  • Unity3D 2018.2+
  • iPhone 8, iPhone 8 Plus, or iPhone X (these work with ALL sample scenes in the 6D.ai SDK Sample App)
  • iPhone 7 & 7 Plus (these will work with Basic Sample scene & Basic Drawing scene, but do not currently support meshing so will NOT work with the Meshing or Ball Pit scenes)

Changes

  • SDK Sample App
    • We’ve consolidated the 4 different Unity apps into one Unity app, adding in all of the individual scenes. This means you will only need to build the app once, in order to test out our functionality.

    • Sample App now supports Portrait mode
    • ⚠️ Now there are two different User Interfaces in the sample app. One for landscape mode, and one for portrait mode. We recommend making two different User Interfaces for your app as well.

  • Meshing

    We’ve added the first pieces to support multi-scene meshing. This means the mesh is automatically built from multiple spheres of dense data, approximately 10 meters in radius as you move around the world.

    As a result, we are able to free up memory for you to use by optimizing the underlying dense data usage for meshing, and only keep the mesh loaded that covers your immediate surroundings (about a 10m radius). This should allow for almost infinite continuous mesh generation. In an upcoming release, we will incorporate disk and AR Cloud for storage of unused spheres to persistently build the world in 3D; for now we are only temporarily storing these dense spheres in memory and discarding them once you move beyond the bounds of the current area you are walking in.

    For those who have tested the past few generations of our meshing SDK, you should notice that memory performance now remains constant, and several hundred more MB is available for your app, whereas previously large scale scanning caused the app to run out of memory. Note though that in this release we are seeing some frames-per-second (fps) performance slowdown as the mesh grows, even though memory use is stable. We have additional fps performance improvements in final testing now for a future SDK release.

    ⚠️ When remote meshes previously captured are unloaded (more than 10m behind you), they no longer occlude or provide physics. If you turn around and walk back, the meshes will regenerate.

Please review the 6D.ai SDK Function Definitions and the Meshing API Guide for additional info on SDK functions and how to work with the mesh.


Release Notes
6D.ai beta SDK v0.11.1
July 24, 2018

Introduction

Minor bugfixes to v0.11.0

System Requirements

  • iOS 11.4+ (iOS 12 beta 3 ok)
  • Xcode 9.4.1+ with ARKit 1.5+ support (10.0 beta 3 ok)
  • Unity3D 2018.2+
  • iPhone 8, iPhone 8 Plus, or iPhone X (these work with ALL sample apps)
    (iPhone 7 & 7 Plus will work with Basic Sample app & Basic Drawing app, but do not currently support meshing so will NOT work with the Meshing or Ball Pit apps)

Changes

  • Minor bug and documentation fixes
  • Modified SDK metrics for beta usage reporting

Please review release notes for v0.11.0 for core details on this release


Release Notes
6D.ai beta SDK v0.11.0
July 17, 2018

Introduction

We're beyond excited to introduce SDK version 0.11.0, with the first ever real-time, high-fidelity 3D reconstruction from monocular RGB capability in the industry!

We call that feature "meshing" as it outputs, several times a second, an updated 3D mesh of the environment around the user.

Just like relocalization enables multiplayer and persistence, meshing enables critical augmented reality features: occlusion, physics, real world interaction, and procedural augmentation.

System Requirements

  • iOS 11.4+ (iOS 12 beta 3 ok)
  • Xcode 9.4.1+ with ARKit 1.5+ support (10.0 beta 3 ok)
  • Unity3D 2018.2+
  • iPhone 8, iPhone 8 Plus, or iPhone X (these work with ALL sample apps)
    (iPhone 7 & 7 Plus will work with Basic Sample app & Basic Drawing app, but do not currently support meshing so will NOT work with the Meshing or Ball Pit apps)

Changes

Additions:

  • API methods are now prefixed by SixDegreesSDK_ instead of SixDegrees_SDK_ to be consistent with the name of the header file. Existing projects may see missing symbol errors after upgrading. We recommend overwriting the Unity files under Assets/Scripts/6DSDK in existing projects with those from the sample app.
  • Sample Apps are now four and described individually below, and now each focus on one particular aspect of the SDK.
  • Location IDs have been improved. You should no longer see "000000" when outside of Wi-Fi coverage, but a unique ID for the location.

Meshing

Meshing is the focus of this release, with the introduction of 4 new API calls:
  • bool SixDegreesSDK_HasRealTimeMesh()
  • int SixDegreesSDK_GetMeshBlockInfo(int* blockBufferSizeOut, int* vertexBufferSizeOut, int* indexBufferSizeOut)
  • float SixDegreesSDK_GetMeshBlockSize()
  • int SixDegreesSDK_GetMeshBlocks(int* blockBuffer, float* vertexBuffer, int* indexBuffer, int blockBufferSize, int vertexBufferSize, int indexBufferSize)

Please review to the 6D.ai SDK Function Definitions for additional info on functions mentioned in these notes.

Meshing is at the moment only generated on-the-fly, in the current AR session. For this reason, it is only available on the higher-end devices (​iPhone X, 8, 8 Plus​). In later SDK releases we'll include cloud-based meshing in order to allow other devices to use meshes generated from other devices.

SixDegreesSDK_HasRealTimeMesh() returns true if the current device can generate meshes on the fly.

Mesh information comes in three different buffers:

  • Block buffer (int)
  • Vertex buffer (float)
  • Face buffer (int)

Those buffers must be allocated on the app side, using the sizes suggested by SixDegreesSDK_GetMeshBlockInfo(), and passed as pointers to SixDegreesSDK_GetMeshBlocks() to be populated.

A block is a cube of 22.4cm by 22.4cm by 22.4cm (8.82in); each triangle of the mesh is fully contained inside a block. The size of that cube is returned by SixDegreesSDK_GetMeshBlockSize(), expressed in meters (consistent with the coordinate system).

Each block is described in the block buffer by 6 consecutive integers: [ x, y, z, vertexCount, faceCount, version ]. (x, y, z) represents the coordinates in space where the cube is, with (0, 0, 0) being the cube containing geometry between 0 and 0.224 on all 3 axes.

The vertex count and face count indicate how many vertices and triangles are contained in each block. The version value increments every time triangles in the block are updated.

Each vertex is described by 3 consecutive floats: [ x, y, z ].

Each face is made of 3 consecutive indices [ a, b, c ], referring to 3 vertices, starting with 0 for the first one.

The diagram above details a simple example block buffer, vertex buffer, and index buffer returned by SixDegreesSDK_GetMeshBlocks(). In this example, the mesh is a polygon spanning two blocks, described by 7 vertices (= 21 floats) and 3 faces (= 9 int indices).

The first block (yellow), at coordinates (0, 0, 0), has a square made of 2 faces (= 6 indices) and 4 vertices (= 12 floats). The second block (pink), at coordinates (1, 0, 0), has a triangle made of 1 face (= 3 indices) and 3 vertices (= 9 floats).

You can notice that some vertices are duplicated (vertex 1 and 4, for instance), because they are part of a different block.

The sample app makes one monolithic mesh spanning all blocks by only using vertices and indices; however it can be a good idea to use blocks to break it down into sub parts, to manage updates, or to optimize physics, or rendering.

Blocks also allow very efficient iterations through the mesh, for instance to find the Y value of the ground plane, without reading every single vertex.

Notes from the lab

Real-time meshing is a large pipeline involving neural networks for depth estimation, dense voxel map stitching, triangulation and decimation. It is extremely memory-intensive and we choose to prioritize feature availability over device support.

For this reason only the iPhone X, 8, 8 Plus are supported at the moment. Performance slowdowns can be experienced after the mesh reaches a certain size, and improvements are on the way to make the experience more consistent and allow larger areas to fit in the phone's memory.

Of course, an upcoming feature is cloud meshes, allowing devices without real-time meshing to benefit from meshes generated from other devices, and also allowing mesh extension.

In the sample apps, a MeshCollider is attached to the mesh in order to intercept raycasts and allow object collision. Mesh collider updates are very expensive and contribute to slow-downs that can be observed in the Ball Pit Sample App. They can be disabled by unchecking "Enable Mesh Collider" in the AR Mesh object.

Stay tuned for tips, tricks and examples of how to optimize mesh physics and interactions!


Release Notes
6D.ai beta SDK v0.10.0
June 26, 2018

Introduction

This week a number of small issues have been fixed in SDK version 0.10.0, ironing out the last few differences with vanilla ARKit performance.

Supported Devices

  • iPhone 7, 7 Plus, 8, 8 Plus, X

Changes

Additions:

  • We now officially support the iPhone 7 & iPhone 7 plus.
  • SixDegrees_SDK_GetVersion() is a new method and now the correct way of evaluating the SDK version at runtime.
  • SixDegrees_SDK_SaveToARCloud() and SixDegrees_SDK_LoadFromARCloud() now only request a GPS location if needed, speeding up the process in most cases.
  • SixDegreesLoadStatePositioning() and SixDegreesSaveStatePositioning() reflect when loading/saving is waiting on the GPS fix.
  • SixDegreesLoadErrorLocationNotAvailable() and SixDegreesSaveErrorLocationNotAvailable() reflect when loading/saving failed because the SDK couldn't use a WiFi SSID or a GPS position.
  • HTML API documentation has been added to the developer portal and is available in the Sample App bundle.

Fixes:

  • 1080p is now the preferred video format.
  • Sample app camera background & projection code is now more robust, fixed issue where the pose used was the previous frame's.
  • Sample app no longer dims.
  • Small performance improvements.

Notes from the lab

  • We recommend using ARKit 1.5 on iOS 11.4 at the moment, but our SDK also works fine with ARKit 2.0 beta as of iOS 12 beta 2.

Release Notes
6D.ai beta SDK v0.9.1
June 20, 2018

Introduction

Minor bugfix on v0.9.0

Supported Devices

  • iPhone 8, 8 Plus, X

Changes

Fixes:

  • Adjust URL location for where cloud assets are stored.

Please review release notes for v0.9.0 for additional details on this release


Release Notes
6D.ai beta SDK v0.9.0
June 19, 2018

Introduction

We are very excited to release this SDK version 0.9, which includes support for cellular connectivity, access to a basic content server for persistence, and a new 3D drawing sample app to showcase the capabilities of our platform.

Supported Devices

  • iPhone 8, 8 Plus, X

Changes

Additions:

  • We are moving away from relying on WiFi network SSID as a way to identify a location, and we are starting to use GPS coordinates, so locations can be differentiated when the device is not connected to a WiFi network.
  • A common request from developers wishing to test persistence was the capability to associate metadata to a location id, e.g. to keep a list of positions of objects. Typically, developers manage that cloud app data on their own servers, but for rapid prototyping we're opening access to our own sample content server. FileController.cs shows how to upload and download a simple CSV file associated to a developer API key and a location ID, that persists selected scene information.
  • We're introducing a richer sample app in which multiple users can finger paint the world in 3D together and permanently anchor their creations to the location. Thanks to our basic content server, no additional setup is needed. Just replace ​SixDegreesSDK.plist​ with your own credentials and ​voila! The previous sample scene with the rotating logos is still available. Please note that since we're using Unity's ​NetworkDiscovery​, real time multiplayer drawing is limited to Wi-Fi networks.
  • SixDegreesLoadStatePositioning() and SixDegreesSaveStatePositioning() reflect when loading/saving is waiting on the GPS fix.
  • SixDegreesLoadErrorLocationNotAvailable() and SixDegreesSaveErrorLocationNotAvailable() reflect when loading/saving failed because the SDK couldn't use a WiFi SSID or a GPS position.
  • HTML API documentation has been added to the developer portal and is available in the Sample App bundle.

Notes from the lab

  • We recommend using ARKit 1.5 on iOS 11.4 as we have found some instability with ARKit 2.0 beta as of iOS 12 beta.

Release Notes
6D.ai beta SDK v0.8.0
June 12, 2018

Introduction

SDK version 0.8 had a big focus on mapping improvements.

Supported Devices

  • iPhone 8, 8 Plus, X

Changes

  • A number of map-building tasks have been backgrounded to reduce their impact on UI framerate and app performance. This allows multiple devices to simultaneously contribute to the extension of the location's original map after their successful relocalization.
  • Until v0.7, the map would stop expanding after a call to LoadFromARCloud(). Now that the map never stops growing, players can share their contribution to the location's map, a chunk known as a "scene", with a call to SaveToARCloud(). Future players at that location will experience an easier, faster relocalization thanks to all the scenes making up the local map.

Notes from the lab

  • Meshing support is one or two months away, our CV team made huge progress tuning neural networks and optimizing routines to create a dense depth estimation and mesh updates, including decimation and outlier clean-up, multiples times per second.
  • We're working on Android support, better sample apps to showcase persistence and multiplayer, and an improved cloud infrastructure for online map improvement. We will also soon make it possible to identify locations without a WiFi SSID.
  • We recommend using ARKit 1.5 on iOS 11.4 as we have found some instability with ARKit 2.0 beta as of iOS 12 beta.

Release Notes
6D.ai beta SDK v0.7.0
May 15, 2018

Introduction

We are pleased to release this SDK version 0.7, which includes significant improvements in relocalization performance, multiplayer experience, and developer support.

Supported Devices

  • iPhone 8, 8 Plus, X

Changes

  • Relocalization is the most notable improvement, and has been made part of the LoadFromARCloud() process, which is now considered "done" only after relocalization succeeds. It is no longer necessary to call Relocalize() manually, and that call was removed from the API.
  • User sessions can now be persisted and matched with the new API call GetLocationId(). In the current implementation, it is a 6-character hash of the WiFi SSID, or an empty string if the coordinate system is only local to the device (i.e. if neither LoadFromARCloud() or SaveToARCloud() was called).
    • This enables cloud-based multiplayer, clients with the same location ID are at the same location and see the physical world in the same coordinate system. This also enables cloud-based persistence, sessions with the same location ID will anchor virtual objects at the same physical location.
  • We upgraded our cloud communication protocol to use TLS for privacy, which means that you can disable "App Transport Security Settings > Allow Arbitrary Loads" in your app's Info.plist.
  • We opened a developer portal at https://dashboard.6d.ai, where you'll find your API keys to use with the SDK. Those keys will be read from a file, SixDegreesSDK.plist, to be included in your app bundle.

Notes from the lab

  • Now that the new relocalizer is integrated into the SDK we are still working out the right balance between resource usage and relocalization accuracy. You'll notice a reduced framerate when scanning, as the map builds up in the background. We're working towards making that process as unobtrusive to the UX as possible.
  • Note that 0.7 location IDs are likely to be incompatible with future SDK releases, as we move away from the WiFi SSID as the location identifier.
  • In response to developer feedback, we are shifting more of our focus towards meshing.
  • We'll be attending AWE in Santa Clara from May 30 to June 1st - come visit our booth!