Virtual Reality IV: XR in Game, Gamification and Interactive media
by Hesam Sakian Mohamadi, Department of Multimedia, Tabriz Art University, Tabriz, Iran
Course Type: Design and Technology-focused
Keywords: Extended Reality, Unity3D, Mobile Sensors
| University | Department | Level | Credits | Length | Medium |
| Tabriz Art University | Multimedia | Undergrad | 3 | 16 weeks | In-person |
Course Description
Extended reality (XR) is an umbrella term that encompasses virtual reality (VR), augmented reality (AR), mixed reality (MR), and other immersive technologies that create or enhance digital experiences. XR has been widely used in game and gamification, which are the application of game elements and principles to non-game contexts, such as education, health, business, and social impact. In this course, students will learn the fundamentals of XR, game design, gamification, mobile sensors (accelerometer, gyroscope, GPS, TOF, compass, etc.) and how to use them to create engaging and meaningful experiences for various purposes and audiences. Students will also explore the current trends, challenges, and opportunities of XR in game and gamification, and its impact on society, culture, and ethics.
Course Prerequisites: Interest in XR, game design, gamification and interactive media. Basic programming skills, familiarity with 3D modeling software and Unity3D engine are recommended.
Course Objectives
- Explain the concepts, components, and types of XR, game design, and gamification
- Introduce, compare and contrast the advantages and disadvantages of different XR platforms, devices and SDKs
- Design and develop creative XR games and gamified applications using various tools and frameworks
- Evaluate and critique XR games and gamified applications based on usability, accessibility, aesthetics, and effectiveness
- Reflect on the role and responsibility of XR developers and users in game and gamification contexts
Weekly Outline
Week 1. Interaction and Immersion
Week 2. Mobile sensors (Gyroscope, Accelerometer, Compass, Magnetometer)
Week 3. Mobile sensors (ToF, GPS, Touch)
Week 4. Implementation of mobile sensors in Unity3D
Week 5. What is XR
Week 6. AR in game, gamification and art
- Defining AR and explaining how it works
- Providing examples of AR games, such as Pokemon Go, Ingress, and Harry Potter: Wizards Unite
- Discussing the benefits and challenges of AR games, such as engagement, immersion, accessibility, privacy, and safety
- Explaining how gamification can enhance learning outcomes and motivation by using game elements and mechanics in non-game contexts
- Providing examples of gamified learning experiences, such as Classcraft
- Discussing the benefits and challenges of gamification, such as feedback, rewards, competition, collaboration, and ethics
- Explaining how AR can be used for artistic expression and creativity
- Providing examples of AR art, such as Snapchat filters, Google Arts and Culture, and AR graffiti
- Discussing the benefits and challenges of AR art, such as interactivity, accessibility, authenticity, and ownership
Week 7 . Implementation of an AR project in Unity3D
- Introducing Unity3D as a cross-platform game engine and development tool
- Explaining the basic concepts and components of Unity3D, such as scenes, game objects, components, scripts, and prefabs
- Providing a step-by-step tutorial on how to create a simple AR project in Unity3D using the Vuforia SDK
- Explaining the features and functions of the Vuforia SDK, such as image targets, ground planes, virtual buttons, and area target
- Providing tips and best practices on how to design and test an AR project in Unity3D, such as lighting, performance, user interface, and user experience
Week 8. MR in game, gamification and art
- Defining MR and explaining how it works
- Providing examples of MR games, such as Minecraft Earth and Star Wars: Jedi Challenges
- Discussing the benefits and challenges of MR games, such as realism, immersion, interaction, compatibility, and cost
- Explaining how MR can be used for gamification and learning
- Providing examples of MR gamified learning experiences, such as Merge Cube
- Discussing the benefits and challenges of MR gamification and learning, such as engagement, collaboration, visualization, distraction, and accessibility
- Explaining how MR can be used for artistic expression and creativity
- Providing examples of MR art, such as HoloLens Art Gallery, Tilt Brush, and Graffiti Simulator
- Discussing the benefits and challenges of MR art, such as innovation, expression, interaction, and quality
Week 9. MR headset and glasses and how to use them
- Introducing MR headset and glasses as devices that enable MR experiences
- Explaining the differences and similarities between MR headset and glasses, such as field of view, resolution, tracking, input, etc.
- Providing examples of MR headset and glasses, such as Microsoft HoloLens, Magic Leap, Nreal, and Apple Vision Pro
- Explaining how to use MR headset and glasses, such as setup, calibration, navigation, and interaction
- Providing tips and best practices on how to use MR headset and glasses, such as comfort, hygiene, safety, and maintenance
Week 10. VR in game, gamification and art
- Defining VR and explaining how it works
- Providing examples of VR games, such as Beat Saber, Half-Life: Alyx, and Superhot VR
- Discussing the benefits and challenges of VR games, such as immersion, presence, interaction, motion sickness, and cost
- Explaining how VR can be used for gamification and learning
- Providing examples of VR gamified learning experiences, such as VRChat, and The Body VR
- Discussing the benefits and challenges of VR gamification and learning, such as engagement, empathy, simulation, isolation, and accessibility
- Explaining how VR can be used for artistic expression and creativity
- Providing examples of VR art, such as Quill, Medium, and Google Blocks
- Discussing the benefits and challenges of VR art, such as freedom, expression, collaboration, and quality.
Week 11. VR headsets and how to use them
- Introducing VR headsets as devices that enable VR experiences
- Explaining the differences and similarities between VR headsets, such as tethered, standalone, mobile, and hybrid
- Providing examples of VR headsets, such as Meta Quest, HTC Vive, PlayStation VR, Samsung Gear VR, and Play Station VR
- Explaining how to use VR headsets, such as setup, calibration, navigation, and interaction
- Providing tips and best practices on how to use VR headsets, such as comfort, hygiene, safety, and maintenance
Week 12. Implementation of a mobile VR project in Unity3D
- Introducing mobile VR as a type of VR that uses smartphones as the display and processing unit
- Explaining the advantages and disadvantages of mobile VR, such as portability, affordability, compatibility, and performance
- Providing a step-by-step tutorial on how to create a simple mobile VR project in Unity3D using mobile sensors
- Providing tips and best practices on how to design and test a mobile VR project in Unity3D, such as optimization, user interface, and user experience
Week 13. Game and narrative design workshop
- Introducing play and narrative design as the processes of creating the rules, mechanics, story, and characters of a game
- Explaining the importance and challenges of game and narrative design, such as engagement, immersion, emotion, and coherence
- Providing a framework and guidelines on how to design a game and a narrative, such as genre, theme, goal, conflict, and resolution
- Providing examples of game and narrative design, such as The Last of Us, Portal, and Journey
- Orienting the workshop towards where students can apply the principles of game design and narrative to create their own concepts and pitches
Week 14. Marketing and business considerations
Week 15. Free discussion on final projects
Week 16. Final projects evaluation
Viewings
- What are Virtual and Augmented Realities? | ColdFusion (YouTube)
- Fun with ARKit | Trekk (YouTube)
- Introducing ARCore | Google AR & VR (YouTube)
- Indoor Visual Positioning for enhanced location based AR | Blippar (YouTube)
- Large-scale Augmented Reality Pong | Futurism (YouTube)
- Microsoft HoloLens 2 Demo – Mobile World Congress | Microsoft BA TV (YouTube)
- Exposición de realidad aumentada en la Ciudad de México | Yunuene (YouTube)
- oneiros channel (YouTube)
- Anna dream brush (YouTube)
- Black Mirror (Netflix Show)
IDN Artifacts
- Pokémon GO (Mobile Game)
- INVASION! (VR animation)
- Half-Life: Alyx (VR Game)
- Unrecord (PC Game)
- Lumen in the Land of Nanite (Console Game – Proof of concept)
- Ori and the Blind Forest (PC/Console Game)
- MeetKa VR (Metaverse Platform)
Reading
- Emerging Trends in Digital Games (Book – Persian)
- Designing Immersive 3D Experiences: A Designer’s Guide to Creating Realistic 3D Experiences for Extended Reality (Book – English)
- The Gamer’s Brain: How Neuroscience and UX Can Impact Video Game Design (Book – English)
- Mobile sensors (Excerpt)
- Types of XR (Excerpt)
- Global XR revenue forecast, SuperData, MarketingCharts.com (Website)
- Augmented reality: A class of displays on the reality-virtuality continuum (Scientific Article)
IDE and IDN Authoring Tools
- LTS version of Unity3D 2020 or later (Engine)
- Visual Studio 2019 or later (IDE)
- JetBrains Rider 2020 or later (IDE)
- PTC Vuforia v10 or later (SDK)
- Unity Remote 5 (Tool)
Major Assignments (being assignments whose value is of 25% or more)
Mobile sensor project
- Platform: Unity3D, Android
- Purpose: Introduces students to the fundamental aspects of how mobile sensors work and how to use them using the Unity3D API.
- Requirements:
- Project Length: 5 – 20 minutes.
- Project Size: A scene with simple 3D objects which can be controlled (transformed) using mobile sensors
- Project Aesthetics: Simple UI, materials and textures. Emphasis on user experience.
- Coding Proficiency: Using Unity3D API to interact with mobile sensors. At least one game mechanic such as movement or rotation using data from mobile sensors.
- Evaluation:
- Interactivity:
- Engagement: Innovative and interactive aspects of the project that captures users’ attention.
- Purposeful interactions: How the interactive elements serve a meaningful purpose and contribute to the project’s overall vision.
- Usability standards: How the project upholds high standards of usability or even tailors usability in alignment with its artistic statement.
- User-friendly design: Intuitive interface that allows new players to navigate through the project without unnecessary obstacles or confusion.
- How the project aligns with the manifesto: How each interaction and design decision corresponds to and supports the underlying concepts outlined in the manifesto.
- Interactivity:
- Story and narrative: Narrative and story are not necessary for this assignment, however, it is recommended to have a general scenario to fulfill the overall goal of the project.
- Production values: This project is intended to be more technical and a testbed for students to learn about sensors and their usability. Therefore, this project is not going to be production ready.
AR & VR project
- Platform: Unity3D, Android
- Purpose: Introduces students how to implement an immersive experience using mobile devices.
- Requirements:
- Project Length: 10 – 20 minutes.
- Project Size: At least one scene with necessary accessories. At least one controllable character with necessary locomotion mechanisms in AR or VR.
- Project Aesthetics: 3D models and characters should be fully textured. A color palette should be employed. SFX must be included but VFX is optional.
- Coding Proficiency: Implementing binocular vision or augmentation using Unity 3D and SDKs such as Vuforia. Locomotion system for the player character and NPCs. Interaction system for objects in the scene.
- Evaluation:
- Interactivity:
- Engagement: Innovative and interactive aspects of the project that captures users’ attention.
- Purposeful interactions: How the interactive elements serve a meaningful purpose and contribute to the project’s overall vision.
- Usability standards: How the project upholds high standards of usability or even tailors usability in alignment with its artistic statement.
- User-friendly design: Intuitive interface that allows new players to navigate through the project without unnecessary obstacles or confusion.
- How the project aligns with the manifesto: How each interaction and design decision corresponds to and supports the underlying concepts outlined in the manifesto.
- Story and narrative: The project effectively utilizes story and narrative aspects, such as setting, plot, pacing, drama, characters, and dialogue to enhance engagement and immersion, fulfilling the objectives stated in the project’s artist statement. Furthermore, it incorporates immersive techniques and extended reality concepts to provide an enriched user experience.
- Interactivity:
- Production values: The work exhibits high quality and demonstrates polished refinement within its respective medium, encompassing aspects such as visuals, audio and interaction. The final product is meticulously crafted, surpassing basic requirements to present itself as a professional piece rather than an academic assignment. The associated marketing and presentation materials maintain consistency with the project’s core values and themes, ensuring a unified and professional appearance across all publishing platforms.
Final project
- Platform: Unity3D, Android/iOS/Desktop/Console
- Purpose: Students will put everything together with a focus on teamwork. This project can be a continuation of the previous project, but it should be refined.
- Requirements:
- Project Length: 10 – N/A minutes.
- Project Size: A vertical slice of a functional immersive narrative. At least one scene with necessary accessories. At least one controllable character with necessary locomotion mechanisms in AR or VR.
- Project Aesthetics: 3D models and characters should be fully textured. A color palette should be employed. SFX and VFX must be included. Character should be rigged and animated with appropriate motion imitation.
- Coding Proficiency: Implementing binocular vision or augmentation using Unity 3D and SDKs such as Vuforia. Locomotion system for the player character and NPCs. Interaction system for objects in the scene. Fully functional user interface. Code and performance optimization is essential.
- Evaluation:
- Interactivity:
- Engagement: Innovative and interactive aspects of the project that captures users’ attention.
- Purposeful interactions: How the interactive elements serve a meaningful purpose and contribute to the project’s overall vision.
- Usability standards: How the project upholds high standards of usability or even tailors usability in alignment with its artistic statement.
- User-friendly design: Intuitive interface that allows new players to navigate through the project without unnecessary obstacles or confusion.
- How the project aligns with the manifesto: How each interaction and design decision corresponds to and supports the underlying concepts outlined in the manifesto.
- Story and narrative: The project effectively utilizes story and narrative aspects, such as setting, plot, pacing, drama, characters, and dialogue to enhance engagement and immersion, fulfilling the objectives stated in the project’s artist statement. Furthermore, it incorporates immersive techniques and extended reality concepts to provide an enriched user experience.
- Interactivity:
- Production values: The work exhibits high quality and demonstrates polished refinement within its respective medium, encompassing aspects such as visuals, audio and interaction. The final product is meticulously crafted, surpassing basic requirements to present itself as a professional piece rather than an academic assignment. The associated marketing and presentation materials maintain consistency with the project’s core values and themes, ensuring a unified and professional appearance across all publishing platforms. Moreover, an emphasis on marketability strengthens the product’s competitive edge, further enhancing its overall appeal to the target audience.
Course Best Practices
- Course structure and contact hours:
- Allocate 2-4 hours per session, with at least 1 session per week.
- Ensure the course spans over 14 to 16 weeks.
- Dedicate approximately 40% of contact hours to lectures and presentations, with the remaining 60% divided among practical sessions and workshops.
- Session allocation:
- Practice: Allocate at least one full session per month for hands-on practice with XR tools and techniques. Implement group projects to encourage students to work collaboratively on XR design and development.
- Reflection: Integrate reflection phases during practical sessions and workshops, prompting students to analyze their progress and apply feedback from peers and instructors.
- Peer reviews: Organize peer-review activities biweekly, enabling students to evaluate each other’s work, exchange ideas, and learn from diverse perspectives.
- Workshops: Schedule workshops every three weeks. These should focus on specific XR topics such as user experience design, rapid prototyping, or advanced interaction techniques.
- Additional lecture resources:
- Provide suggested reading materials like articles, e-books, or video tutorials for offline self-study.
- Share real-world case studies related to XR in the gaming and media industry that effectively illustrate current trends and challenges.
- Encourage students to participate in online communities or forums dedicated to XR and interactive media discussions.
- Relationship between course material and assignments:
- Design assignments that integrate content from different lectures, helping students demonstrate their understanding of connections between various topics.
- Allow flexibility in assignment submissions; students should be able to choose specific aspects within a broader theme to align their interests and personalize their learning experience.
- Develop a final project where students apply cumulative knowledge gained throughout the course into a comprehensive XR game, gamification or interactive media experience.