Contextual Awareness Features not working on a Android 12 device that meets requirements

[Chankeiro]

• Issue category: Real-time Mapping-Depth
• Device type & OS version: Android 12
• Host machine & OS version: Windows
• Issue Environment : On Device

Hi.

First of all, I would like to say that I think ARDK technology is amazing. Thank you very much for providing this kind of tool to the community. I believe and hope that great things are going to be done with it.

About my issue: I’ve been doing some LightShip testing with an Android 12 device that, according to the ARDK System Requirements, supports Contextual Awareness features ( Development and Runtime Requirements of ARDK — Niantic Lightship Augmented Reality Developer Kit release-2.5.2 documentation). It is a OnePlus Nord2 equipped with a Mali-G77 GPU, released one year ago approx (https://www.kimovil.com/en/where-to-buy-oneplus-nord-2)

Despite meeting ARDK requirements, the phone failed the following device support checks in all ARDK versions I’ve worked with:

• ARWorldTrackingConfigurationFactory.CheckDepthEstimationSupport().
• ARWorldTrackingConfigurationFactory.CheckMeshingSupport().

However, _ARConfigurationValidator.cs script states that “ARDK’s device support checks serve as recommendations, not a hard block. Devices are still able to try to run unsupported features.”. Therefore, I have tested the meshing functionality with several ARDK versions. These are the results:

• ARDK 1.1.0: Impossible to use. A few mesh chunks are rendered on the screen, but they aren’t accurate at all, and drift continuously. (Sorry, but I don’t have a video for this version. I can make one if it helps).

• ARDK 1.3.0 and ARDK 2.0.0: meshing is not very accurate, and sometimes it drifts a little. For example, an Android 11 device with an Adreno 650 GPU (it has a similar computational power) offers much more accurate and stable results. Still, the mesh is good enough for my use case, and the multiplayer experience is OK as well. I could work with that. You can see a scan with 2.0.0 in the following video:
  

• ARDK 2.1.0: from my point of view, there has been a regression. Accuracy is not as bad as with ARDK 1.1.0, but definitely worse than 2.0.0. The mesh also drifts noticeably. Impossible to use. Please, check two scan attempts in the following video:
  

The main issues I see here:

1. A relatively recent device, with a well-known GPU that meets the minimum requirements, cannot run contextual awareness features consistently.
2. The current minimum requirements stated in the documentation cannot be used as a guide as to whether or not a device can run ARDK’s contextual awareness functions. I think the only option to know for sure is for the user to install the app and wait for the device to fail at runtime.
3. There may be regressions between ARDK versions.

Related questions:

• I think the device checks in “_ARConfigurationValidator.cs” should always be used as a hard-block in case the app needs ARDK’s contextual awareness features, don’t you agree? On the one hand, there may be devices that fail the tests but can still run the features (like this OnePlus using versions 1.3.0 and 2.0.0). By using the device checks as a hard-block, we would be preventing those users from trying the app, which isn’t good. On the other hand, I think it is even worse to let them run the app and eventually have a bad experience (like the OnePlus in version 2.1.0).
• Could you provide a reliable list of capable devices (or device features) that the users could check before installing the app, or that we could display in our apps as a recommendation in case a device fails the checks (or even if the device doesn’t even support ARCore)?
• Are you working on trying to extend ARDK support to more GPUs? Do you know if these issues are occurring with other theoretically supported GPUs, or could this device just be an exception?
• Regarding the regressions between versions, well, I am just concerned about them. Any thoughts on that would be appreciated.

Thank you very much in advance for your support!!

[Jason_Rose1]

Hello Chankeiro,

I have some information for you regarding the issues you are experiencing:

1. The video you posted is showing tracking failing, which is why you see the mesh drifting. This means that Google’s ARCore tracking on the device is not correctly providing poses to ARDK in order for it to align the mesh. This typically happens on lower spec’d devices that struggle to keep up with the processing load.
2. ARDK 2.1.0 also brought with it the use of a newer version of ARCore as required by google so this may play into some of the performance differences you are seeing, but we cannot be sure at this time.
3. Currently there is no easy fix for this, but we are looking at a number of android specific optimisations for future ARDK releases which may improve this in the future.

Please let us know if you have any further questions we can answer for you.

[Chankeiro]

Hi @Jason_Rose1 ,

Thank you very much for the information. Some comments:

• “This tends to happen on low-spec devices that have difficulty keeping up with the processing load”: I don’t think this is the case with the OnePlus Nord2, as it meets Lightship’s contextual awareness requirements by quite a margin. In fact, I have been able to run contextual awareness functions on devices with much worse specs than the OnePlus. For example, the Xiaomi in the screenshot below is an old device with much lower specs than the OnePlus and doesn’t even meet the requirements (it has an Adreno 620 GPU), but you can get pretty accurate meshes with it.
  

  

• “We are looking at a number of android-specific optimizations for future versions of ARDK”: this is great news. In the meantime, could you share a list of capable devices (or device features) that we can use as a guide?

[Jason_Rose1]

Hello Chankeiro,

We don’t have a list of capable devices. All of the device features you can use are listed here: Android Devices and Contextual Awareness

[Chankeiro]

Hi. I’ve been told that contextual awareness features are not working on a Xiomi 11 Lite NE 5G (Android 12) which is equipped with an Adreno 642L https://www.kimovil.com/en/where-to-buy-xiaomi-11-lite-5g-ne This is the first time I’ve been notified of this problem for an Adreno device. Until now I had only received reports from Mali devices not working.

[Stacy_Niantic]

Hello Chankeiro,

Could you please provide some more details regarding how the features are not working? Are you noticing inconsistencies in the meshing like you had described in your previous posts, or is it something else entirely? If you have any screenshots or videos showing the issue, that would better help us troubleshoot.

Could you also please verify that you are using the latest version of ARDK (currently 2.3.1), and are you experiencing these issues on any devices besides the Xiaomi 11 Lite?

[Chankeiro]

Hi @Stacy_Niantic,

I’m sorry because I was not precise enough in my previous post. I will try to explain it better this time. I have released an app that is using ARDK 2.0.1. Currently, before using the contextual awareness features, I am running these two checks:

• ARWorldTrackingConfigurationFactory.CheckDepthEstimationSupport().
• ARWorldTrackingConfigurationFactory.CheckMeshingSupport().

If any of these checks fail on the device running the application, I inform the user that unfortunately they cannot continue. I do this because, in my experience, devices that fail these tests are very likely to experience behavior similar to that described in my first post (low accuracy and very noticeable drift, at least until ARDK 2.2.0). It could also be the case that the contextual awareness features run correctly on the device, but I cannot be sure about that. So I’d rather prevent the user from running the app even though, of course, that’s also a horrible user experience, since they’ve already downloaded it and are ready to use it. But I don’t think I have any other alternative right now.

Since I released the app, I have received several reports of users getting this “testing-failed” message. Until this week all of them were equipped with Mali GPUs (as the OnePlus Nord2 in my post), so I didn’t report it because I thought it was a problem specific to those GPUs, and I didn’t want to insist on something I already told you about. But this week I received a report from a user with this Xiaomi 11 Lite NE 5G, which should be capable enough as well, but this time with an Adreno GPU. That’s why I am reporting it to you, just in case it can help you to find the cause of this issue. I cannot provide you screenshots, because users can only see the message and close the app. They cannot run any AR experience.

However, I just tested ARDK 2.3.0 on the OnePlus Nord2, and I have good and bad news:

• Bad News: it still fails contextual awareness checks
• Good News: meshing is much more stable now. Accuracy is not as good as with a comparable Adreno device, although I think it might be good enough for my use case. But the best thing is that it doesn’t drift. So I guess there has been some improvement in the last ARDK version

Scan of a bed with OnePluse Nord2 (equipped with a Mali-G77), ARDK 2.3.0 :

Scan of the same bed with Adreno-650 device, ARDK 2.3.0 :

I cannot be sure if this improvement is general acrross all devices, or if I just was lucky with the OnePlus Nord2, so I think I’ll still keep the checks in my app for now, unless you give me some reason why I shouldn’t. Anyway, I hope it can be fixed in future ARDK versions.

Please, let me know if this answer your question, or if you need additional info.

[Stacy_Niantic]

Hello Chankeiro,

Thank you for all the details!

After looking into the Xiaomi device further, I noticed that it was not listed on the table of ARCore Supported Devices. ARDK depends on shared libraries like ARCore or ARKit, so even if a device’s hardware supports AR, it also needs to have those libraries installed and be able to run them. This would explain why they were getting a “testing-failed” message. Do you happen to know if the other devices that had failed in the past (with the Mali GPUs) are on this list?

I will continue consulting the team in case they have any other helpful information. Thank you for your patience with this.

[Chankeiro]

Hi Stacy,

I don’t think that list of ARCore Supported Devices is 100% up to date. I’ve seen devices that are not on that list being able to install Google Play Services for AR (for instance, the Xiaomi Redmi K30 5G from my second post).

Anyway, I enforce ARCore as required in the Android Manifest of the app Enable AR in your Android app  |  ARCore  |  Google Developers, so only ARCore compatible devices can install my app. Additionally, I check if ARCore is supported once the app is installed เปิดใช้ AR ในแอป Android  |  ARCore  |  Google Developers, and always before starting any AR experience. Therefore, only devices with Google Play Services for AR installed and up to date can run the AR experience. ARDK CapabilityChecker.CheckCapability() succeedes in all of them as well. There are also no problems running, for example, ARFoundation on them .

Summarizing, in all these cases the device is 100% capable to run the ARCore libraries, so that shouldn’t be an issue here.

[Stacy_Niantic]

Hello Chankeiro,

Thank you for your patience while we continue to investigate.

You are correct in that a device not being on the ARCore Supported Devices list does not necessarily mean it’s not supported. It’s a working list that is not comprehensive of all devices, and we are looking to improve this in the future.

That being said, even if contextual awareness features are not supported on a specific device, it should receive a warning (that performance may suffer) and allow the device to proceed, not act as a hard-block. I understand that you don’t allow users to proceed if their devices fail the ARWorldTrackingConfigurationFactory.CheckDepthEstimationSupport() and ARWorldTrackingConfigurationFactory.CheckMeshingSupport() checks, but these are separate than the devices getting the “testing-failed” message, correct? Have you implemented any other hard-blocks within your app? Also, was “testing-failed” the full error message that the users received?

I understand wanting users of your app to have the best experience possible, but have you tried removing any hard-blocks within your app to see if the “testing-failed” message still pops up? While this runs the risk of players having an unpleasant experience if their device doesn’t meet the basic requirements as you have noted, it may also prevent issues similar to what your users are experiencing.

I will update this thread again if I receive more information, but please understand that this will be delayed due to the holidays. Thank you for your patience.

[Chankeiro]

Hi Stacy,

• The first “hard block” is the device having ARCore installed and up to date. If the device fails this check the user receives a message that either their device does not support ARCore or that they need to update ARCore.
• Once the device has passed this first check, the ARWorldTrackingConfigurationFactory tests are run. If the device fails any of them, the user sees the “testing-failed” message, which actually looks like this:
  
  The “link” takes the user to a list of valid devices that I provide based on the feedback I am getting from other users. I would be great if you could provide a more reliable one.

I know that those contextual awareness checks are not a “hard-block” but, in my experience, failing those tests can mean not only poor performance, but the contextual awareness features not working properly (as in the videos in my first post). That’s the reason I still keep them.