Main Idea
This is a game targeted at autistic children with difficulties understanding figurative language, though it could potentially be used by any child.
Neurodiversity is the term used to cover the range of differences in human brains regarded as part of normal variation in the human population(Armstrong 2015). Autism is a type of neurodivergence associated with differences in communication and sensory experience, though it can look different from person to person.
Literal thinking is a problem that some, but not all, autistic people struggle with (Vulchanova and Vulchanov 2022; Vicente et al. 2023). Idioms with their figurative and colourful language can be particularly challenging for some autistic children to pick up.
The game consists of a short introduction level with audio, text and an AR visual followed by 5 levels. Each level contains an explanation of a different idiom. Using AR facial recognition and a suitable visual the idiom is presented to the player in a playful manner. A text and audio explanation of the idiom is provided. The player is then asked to answer a multiple-choice question to show their understanding of the idiom. They can try to choose the correct answer as often as they want. Once they have the correct answer they are sent to the next level, or idiom, until all idioms have been presented. Once no idioms are remaining the game ends and exits.
Technology
This game was built in Unity editor version 2021.3.1f1 and using the following packages
API / framework / Technology / Package | Purpose |
| ARFoundation | A collection of MonoBehaviours and C# utilities for working with AR Subsystems. Enables the creation of multi-platform augmented reality (AR) apps with Unity |
| ARCore | Provides native Google ARCore integration for use with Unity’s multi-platform XR API. |
| ARKit Face Tracking | Support for face tracking on ARKit |
| ARKit | Provides native Apple ARKit integration for use with Unity’s multi-platform XR API. |
| OpenXR | OpenXR is an open, royalty-free standard developed by Khronos that aims to simplify AR/VR development by allowing developers to target a wide range of AR/VR devices. Use this plug-in to enable OpenXR in XR Plug-in Management. |
| TextMeshPro | TextMeshPro provides control over text formatting and layout with features like character, word, line and paragraph spacing. This was used for buttons and text elements on the canvas layer. |
| Timeline | Used to create game-play audio and text sequences and time the presentation of the question at the end of each level/idiom explanation. |
| Toolchain Linux x64 | Used to build the game on Linux |
| Visual Studio Code Editor | Integration for supporting Visual Studio Code editor. Adds support for generating csproj files for intellisense purposes and auto-discovery of installations, etc. |
| XR Plugin Management | Package that provides simple management of XR plug-ins. Manages and offers help with loading, initialization, settings, and build support for XR plug-ins. |
| Mobile phone testing was performed on the following mobile phones: |
|
Assets Used
Asset name and purpose | Link |
| Casual Game BGM #5 by B.G.M was used as the background audio music for the introduction and all levels. | Link |
| Puzzle stage & settings GUI Pack The orange suite is used as background images on question modal and answer images. The green suite was used for background images on text explanations. | Link |
| AR_Template_Zip.zip from the Unity learn tutorial “Getting started with AR” was used as the basis of each AR scene. | Link |
| 3D cake model from TurboSquid [3DS format] was used in the introduction. It is noted that this didn’t import well. Colours and textures had to be re-imported and updated manually | Link |
Basic face filter assets from “Create a basic face filter” tutorial on Unity were used for:
| Link |
| 2D Cat by Okta Chandra Ridzikia was used as the image for the cat’s particle systems in level 1 (“Raining cats and dogs” idiom) | Link |
| 2D Dog by Jamie Sale was used as the image for the dog’s particle systems in level 1 (“Raining cats and dogs” idiom) | Link |
| Fried egg by diqqiafrizal was used as a material for the AR face prefab on the second level (“Egg on their face” idiom) | Link |
| Back of bus image used as the material for the AR face prefab on the third level (“Face like the back of a bus” Idiom) I struggled to find a suitable image of the back of a bus. As such I ended up generating one using the Canva AI image generator tool. | Link |
| 2D ear designed by Ylivdesign was used as the material for the AR face prefab used on the fifth level (“I’m all ears” idiom) | Link |
Navigation Structure
Figure 1: Navigation flow of AR idiom game
Navigation Overview
The navigation structure of this game is extremely basic. As shown in Figure 1 above the flow is almost a straight line from start to finish.
The player starts with an introduction scene. This gives a basic audio and text explanation of what an idiom is, along with an example idiom visual and audio explanation.
This is followed by a series of 5 individual idioms. Each idiom is explained using both audio and text, an AR visual is presented for a few seconds and then the player is asked a question to show their understanding of the idiom. They get feedback text telling them if they have chosen the correct answer or not. If they are correct they progress to the next scene/idiom until none are left. If they choose incorrectly they can choose again until they choose correctly.
Navigation Description
Introduction:
The introduction scene explains what idioms are and gives an example, specifically explaining the idiom “a piece of cake”. Both audio and text are used to reinforce the information provided. A rotating 3D piece of cake is rendered in front of the user’s camera(see Figure 7). Finally, the user is invited to play a game, at which point the next scene is loaded.
Levels 1-5:
The player is then presented with a series of 5 idioms. Each level follows the same format. The idiom’s meaning is explained using both audio and text. The timing of the presentation of the background audio, the explanation audio, the text explanations and the question dialogue is managed on all levels by a timeline. Figure 2 below shows the timeline from Level 1. Each level’s timeline follows a similar format.
Figure 2: Timeline on Canvas used to schedule audio music, audio explanations, text explanations and presentation of the question interface for the “It’s raining cats and dogs” idiom – level 1.
An appropriate AR interaction is displayed which is detailed in the next section, and then the user is presented with a question and asked to choose the correct answer. The format for the QuestionDialog for each level is essentially the same ( see Figure 3 and Figure 4).
Figure 3: QuestionDialog from the “It’s raining cats and dogs” idiom – level 1
If the player clicks on the wrong button the red “Incorrect. Please try again” text is displayed (see Figure 5). The player is allowed to click on the answer buttons as many times as they like until they choose the correct answer. Once the correct button is tapped, the green “Correct!!” text is displayed and all of the buttons are disabled (see Figure 6). The green text is kept on screen for a short time and then the next scene is automatically loaded.
 Figure 4: Question dialogue on phone before answer selected |  Figure 5: Question dialogue when incorrect answer chosen |  Figure 6: Question dialogue when correct answer chosen |
The idioms and their AR interactions are as follows:
Idiom 1: “It’s raining cats and dogs”
The AR portion of this level shows 2D cats, 2D dogs and 3D water falling from the sky (see Figure 8). This was implemented using 3 overlapping particle systems attached to the AR face detection prefab used for this level.
Idiom 2: “Egg on your face”
As shown in Figure 9, this level renders an egg material across the player’s face, inviting playfulness as the idiom is explained.
Idiom 3: “Face like the back of a bus”
Figure 10 is a screenshot from this level, showing the face with an overlay of an image of the back of a bus.
Idiom 4: “Blue in the face”
For the AR portion of this level no image is necessary, instead, the material used for the face detection prefab is simply set to the colour blue (see Figure 11)
Idiom 5: “I’m all ears”
The AR face overlay for this final level is a mix of 3D goblin ears and a material overlay of a 2D ear, both attached to the face prefab for this level, as shown in Figure 12.
 Figure 7: AR “piece of cake” |  Figure 8: “Raining cats and dogs” |  Figure 9: “Egg on your face” |
 Figure 10: “Face like the back of a bus” |  Figure 11: “Blue in the face” |  Figure 12: “I’m all ears” |
Bibliography
Armstrong, T. (2015) ‘The Myth of the Normal Brain: Embracing Neurodiversity’, AMA Journal of Ethics, 17(4), 348–352, available: https://doi.org/10.1001/journalofethics.2015.17.4.msoc1-1504.
Bonnello, C. (2022) ‘Results and Analysis of the Autistic Not Weird 2022 Autism Survey – Autistic Not Weird’, available: https://autisticnotweird.com/autismsurvey/ [accessed 12 Jun 2023].
Buijsman, R., Begeer, S., and Scheeren, A.M. (2023) ‘“Autistic person” or “person with autism”? Person-first language preference in Dutch adults with autism and parents’, Autism, 27(3), 788–795, available: https://doi.org/10.1177/13623613221117914.
Davidson, J. and Henderson, V.L. (2010) ‘“Coming out” on the spectrum: autism, identity and disclosure’, Social & Cultural Geography, 11(2), 155–170, available: https://doi.org/10.1080/14649360903525240.
Vicente, A., Michel, C., and Petrolini, V. (2023) ‘Literalism in Autistic People: a Predictive Processing Proposal’, Review of Philosophy and Psychology, available: https://doi.org/10.1007/s13164-023-00704-x.
Vulchanova, M. and Vulchanov, V. (2022) ‘Rethinking Figurative Language in Autism: What Evidence Can We Use for Interventions?’, Frontiers in Communication, 7, available: https://www.frontiersin.org/articles/10.3389/fcomm.2022.910850 [accessed 5 Dec 2023].
