How to spawn body tracking prefab on host and client and sync it?

Include the following details (edit as applicable):

  • Issue category: Multiplayer / Networking
  • Device type & OS version: iOS
  • Host machine & OS version: Mac
  • Xcode version: 15.4
  • ARDK version: ARDK 3
  • Unity version: 2022.3.30f1

Description of the issue:

Hi, I am new to multiplayer programming, may I ask for need some enlightenment.

what I am trying to do : Spawning AR human body tracking using AR Human Body Manager, to be shown on 2 devices. Host and Client.

Right now, I successfully spawned a prefab on the server side and sync it to the client. But I can’t manage to spawn and sync the body prefab from the client side.

forgive me if there are many logic flaws in the code, I’d be really happy if someone can help me.

Thank you

using System.Collections.Generic;
using UnityEngine;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;
using Unity.Netcode;

namespace UnityEngine.XR.ARFoundation.Samples
{
    public class HumanBodyTrackerShared : NetworkBehaviour
    {
        [SerializeField]
        [Tooltip("The Skeleton prefab to be controlled.")]
        GameObject m_SkeletonPrefab;

        [SerializeField]
        [Tooltip("The ARHumanBodyManager which will produce body tracking events.")]
        ARHumanBodyManager m_HumanBodyManager;

        public ARHumanBodyManager humanBodyManager
        {
            get { return m_HumanBodyManager; }
            set { m_HumanBodyManager = value; }
        }

        public GameObject skeletonPrefab
        {
            get { return m_SkeletonPrefab; }
            set { m_SkeletonPrefab = value; }
        }

        // Separate dictionaries for server and client prefabs
        Dictionary<TrackableId, BoneController> m_ServerSkeletonTracker = new Dictionary<TrackableId, BoneController>();
        Dictionary<TrackableId, BoneController> m_ClientSkeletonTracker = new Dictionary<TrackableId, BoneController>();
        Dictionary<TrackableId, BoneController> m_TrackedSkeletonsFromClients = new Dictionary<TrackableId, BoneController>();

        void OnEnable()
        {
            Debug.Assert(m_HumanBodyManager != null, "Human body manager is required.");
            m_HumanBodyManager.humanBodiesChanged += OnHumanBodiesChanged;
        }

        void OnDisable()
        {
            if (m_HumanBodyManager != null)
                m_HumanBodyManager.humanBodiesChanged -= OnHumanBodiesChanged;
        }

        void OnHumanBodiesChanged(ARHumanBodiesChangedEventArgs eventArgs)
        {
            // Server-side handling
            if (IsServer)
            {
                foreach (var humanBody in eventArgs.added)
                {
                    SpawnSkeletonServer(humanBody);
                }

                foreach (var humanBody in eventArgs.updated)
                {
                    if (m_ServerSkeletonTracker.TryGetValue(humanBody.trackableId, out BoneController boneController))
                    {
                        UpdateSkeletonPoseServer(humanBody, boneController);
                    }
                }

                foreach (var humanBody in eventArgs.removed)
                {
                    RemoveSkeleton(humanBody, m_ServerSkeletonTracker);
                }
            }

            // Client-side handling
            if (IsClient && !IsServer)
            {
                foreach (var humanBody in eventArgs.added)
                {
                    ClientSpawnSkeleton(humanBody);
                }

                foreach (var humanBody in eventArgs.updated)
                {
                    if (m_ClientSkeletonTracker.TryGetValue(humanBody.trackableId, out BoneController clientBoneController))
                    {
                        UpdateSkeletonPoseClient(humanBody, clientBoneController);
                    }
                }

                foreach (var humanBody in eventArgs.removed)
                {
                    RemoveSkeleton(humanBody, m_ClientSkeletonTracker);
                }
            }
        }

        // Server-side skeleton spawn
        void SpawnSkeletonServer(ARHumanBody humanBody)
        {
            if (!m_ServerSkeletonTracker.TryGetValue(humanBody.trackableId, out BoneController boneController))
            {
                var newSkeletonGO = Instantiate(m_SkeletonPrefab, humanBody.transform.position, humanBody.transform.rotation);
                boneController = newSkeletonGO.GetComponent<BoneController>();
                m_ServerSkeletonTracker.Add(humanBody.trackableId, boneController);

                NetworkObject networkObject = newSkeletonGO.GetComponent<NetworkObject>();
                if (networkObject != null && !networkObject.IsSpawned)
                {
                    networkObject.Spawn(true); // Spawn on server and make it visible to clients
                }

                boneController.InitializeSkeletonJoints();
                boneController.ApplyBodyPose2(GetJointPositions(humanBody), GetJointRotations(humanBody));

                SyncBodyPoseClientRpc(humanBody.trackableId.subId1, humanBody.trackableId.subId2, humanBody.transform.position, humanBody.transform.rotation, GetJointPositions(humanBody).ToArray(), GetJointRotations(humanBody).ToArray(), networkObject.NetworkObjectId);
            }
        }

        // Client-side skeleton spawn
        public void ClientSpawnSkeleton(ARHumanBody humanBody)
        {
            if (!m_ClientSkeletonTracker.ContainsKey(humanBody.trackableId))
            {
                // Instantiate the skeleton prefab locally on the client
                var newSkeletonGO = Instantiate(m_SkeletonPrefab, humanBody.transform.position, humanBody.transform.rotation);
                BoneController boneController = newSkeletonGO.GetComponent<BoneController>();
                boneController.InitializeSkeletonJoints();
                m_ClientSkeletonTracker.Add(humanBody.trackableId, boneController);

                // Apply the body pose locally on the client
                boneController.ApplyBodyPose2(GetJointPositions(humanBody), GetJointRotations(humanBody));

                // Notify the server about the local spawn, without spawning another object
                NotifyServerOfClientSpawnServerRpc(
                    humanBody.trackableId.subId1, humanBody.trackableId.subId2,
                    humanBody.transform.position, humanBody.transform.rotation,
                    GetJointPositions(humanBody).ToArray(), GetJointRotations(humanBody).ToArray());
            }
        }

        // Server-side pose update
        void UpdateSkeletonPoseServer(ARHumanBody humanBody, BoneController boneController)
        {
            boneController.transform.position = humanBody.transform.position;
            boneController.transform.rotation = humanBody.transform.rotation;
            boneController.ApplyBodyPose2(GetJointPositions(humanBody), GetJointRotations(humanBody));

            NetworkObject networkObject = boneController.GetComponent<NetworkObject>();
            if (networkObject != null)
            {
                // Sync updated body pose to all clients
                SyncBodyPoseClientRpc(humanBody.trackableId.subId1, humanBody.trackableId.subId2, humanBody.transform.position, humanBody.transform.rotation, GetJointPositions(humanBody).ToArray(), GetJointRotations(humanBody).ToArray(), networkObject.NetworkObjectId);
            }
        }

        // Client-side pose update
        void UpdateSkeletonPoseClient(ARHumanBody humanBody, BoneController boneController)
        {
            // Update the pose locally on the client
            boneController.transform.position = humanBody.transform.position;
            boneController.transform.rotation = humanBody.transform.rotation;
            boneController.ApplyBodyPose2(GetJointPositions(humanBody), GetJointRotations(humanBody));

            // Notify the server of the updated pose from the client
            NotifyServerOfClientPoseUpdateServerRpc(
                humanBody.trackableId.subId1,
                humanBody.trackableId.subId2,
                humanBody.transform.position,
                humanBody.transform.rotation,
                GetJointPositions(humanBody).ToArray(),
                GetJointRotations(humanBody).ToArray()
            );
        }

        // Remove skeletons
        void RemoveSkeleton(ARHumanBody humanBody, Dictionary<TrackableId, BoneController> tracker)
        {
            if (tracker.TryGetValue(humanBody.trackableId, out BoneController boneController))
            {
                Destroy(boneController.gameObject);
                tracker.Remove(humanBody.trackableId);
            }
        }

        // Sync and communication methods
        [ClientRpc]
        void SyncBodyPoseClientRpc(ulong subId1, ulong subId2, Vector3 rootPosition, Quaternion rootRotation, Vector3[] jointPositionsArray, Quaternion[] jointRotationsArray, ulong networkObjectId)
        {
            // Reconstruct the TrackableId from the two ulong values
            TrackableId bodyId = new TrackableId(subId1, subId2);

            // Sync the prefab on all clients
            if (NetworkManager.SpawnManager.SpawnedObjects.TryGetValue(networkObjectId, out NetworkObject networkObject))
            {
                BoneController boneController = networkObject.GetComponent<BoneController>();
                if (boneController != null)
                {
                    boneController.transform.position = rootPosition;
                    boneController.transform.rotation = rootRotation;
                    boneController.ApplyBodyPose2(new List<Vector3>(jointPositionsArray), new List<Quaternion>(jointRotationsArray));
                }
            }
        }

        [ServerRpc(RequireOwnership = false)]
        void NotifyServerOfClientSpawnServerRpc(ulong subId1, ulong subId2, Vector3 rootPosition, Quaternion rootRotation, Vector3[] jointPositionsArray, Quaternion[] jointRotationsArray)
        {
            // Create a new TrackableId from the client-provided IDs
            TrackableId bodyId = new TrackableId(subId1, subId2);

            if (!m_TrackedSkeletonsFromClients.ContainsKey(bodyId))
            {
                var newSkeletonGO = Instantiate(m_SkeletonPrefab, rootPosition, rootRotation);
                BoneController boneController = newSkeletonGO.GetComponent<BoneController>();
                boneController.InitializeSkeletonJoints();
                boneController.ApplyBodyPose2(new List<Vector3>(jointPositionsArray), new List<Quaternion>(jointRotationsArray));
                m_TrackedSkeletonsFromClients.Add(bodyId, boneController);

                NetworkObject networkObject = newSkeletonGO.GetComponent<NetworkObject>();
                if (networkObject != null && !networkObject.IsSpawned)
                {
                    networkObject.Spawn(true);
                }
            }
        }

        [ServerRpc(RequireOwnership = false)]
        void NotifyServerOfClientPoseUpdateServerRpc(ulong subId1, ulong subId2, Vector3 position, Quaternion rotation, Vector3[] jointPositions, Quaternion[] jointRotations)
        {
            TrackableId bodyId = new TrackableId(subId1, subId2);

            // Check if the server is tracking the skeleton from the client
            if (m_ServerSkeletonTracker.TryGetValue(bodyId, out BoneController boneController))
            {
                // Apply the updated pose on the server
                boneController.transform.position = position;
                boneController.transform.rotation = rotation;
                boneController.ApplyBodyPose2(new List<Vector3>(jointPositions), new List<Quaternion>(jointRotations));

                // Notify all clients (including the host) to update the pose
                SyncBodyPoseClientRpc(subId1, subId2, position, rotation, jointPositions, jointRotations, boneController.GetComponent<NetworkObject>().NetworkObjectId);
            }
        }

        // Helper functions
        private List<Vector3> GetJointPositions(ARHumanBody humanBody)
        {
            var positions = new List<Vector3>();
            foreach (var joint in humanBody.joints)
            {
                positions.Add(joint.localPose.position);
            }
            return positions;
        }

        private List<Quaternion> GetJointRotations(ARHumanBody humanBody)
        {
            var rotations = new List<Quaternion>();
            foreach (var joint in humanBody.joints)
            {
                rotations.Add(joint.localPose.rotation);
            }
            return rotations;
        }

        public void EnableHumanBodyTracking()
        {
            if (m_HumanBodyManager != null && !m_HumanBodyManager.enabled)
            {
                m_HumanBodyManager.enabled = true;
                Debug.Log("ARHumanBodyManager has been enabled.");
            }
        }

        public void DisableHumanBodyTracking()
        {
            if (m_HumanBodyManager != null && m_HumanBodyManager.enabled)
            {
                m_HumanBodyManager.enabled = false;
                Debug.Log("ARHumanBodyManager has been disabled.");
                ClearTrackedSkeletons();
            }
        }

        private void ClearTrackedSkeletons()
        {
            foreach (var skeleton in m_ServerSkeletonTracker.Values)
            {
                Destroy(skeleton.gameObject);
            }
            m_ServerSkeletonTracker.Clear();

            foreach (var skeleton in m_TrackedSkeletonsFromClients.Values)
            {
                Destroy(skeleton.gameObject);
            }
            m_TrackedSkeletonsFromClients.Clear();
        }
    }
}

Hello, and thanks for reaching out! For Unity-related problems such as this one, I would recommend reaching out to Unity’s forums here to ask your question. They could help you more with setting up multiplayer connections with Unity.

That being said, we can help you with any ADRK-related issues you might encounter further in your project. Alternatively, if you need assistance with any ARDK-related features now, I could assist you with those.