WWDC 2024: Developer Innovations from The Platforms State of the Union

13 min readJun 13, 2024

On June 10, Apple hosted the Platforms State of the Union for developers, providing a deep dive into the technical advancements and tools unveiled at WWDC 2024. This event complemented the Keynote presentation by focusing on developer-centric updates and innovations. In our previous article, “WWDC 2024 Highlights: Apple’s Vision for the Future”, we explored the major announcements from the Keynote, including updates to Vision Pro, iOS 18, and macOS 15 Sequoia. Now, let’s turn our attention to the Platforms State of the Union, where Apple revealed a wealth of new features and capabilities aimed at enhancing the developer experience and expanding the potential of its platforms. From advanced AI integrations to powerful new APIs, let’s see what Apple has in store for developers this year.

Apple Intelligence: A Leap Forward in Personal AI

Introduction of Apple Intelligence

At the Platforms State of the Union during WWDC 2024, Apple unveiled its latest advancement in artificial intelligence: Apple Intelligence. This new personal intelligence system is designed to enhance the functionality and user experience across Apple devices by integrating sophisticated AI capabilities directly into the operating system.

Integration of Generative AI Models

Apple Intelligence incorporates state-of-the-art generative AI models for both language and image processing. These models enable a range of new functionalities, such as advanced text rewriting, summarization, and image creation, which are seamlessly integrated into the user experience. This allows for more intuitive interactions and personalized content generation, enhancing productivity and creativity for users.

On-Device Foundation Model

A key feature of Apple Intelligence is its on-device foundation model, which is built with a strong focus on privacy. By processing data locally on the device, Apple ensures that user information remains secure and private, reducing the reliance on cloud-based processing. This approach not only safeguards user privacy but also improves performance by minimizing data transfer delays.

New APIs for Developers

To enable developers to harness the power of Apple Intelligence, Apple introduced a suite of new APIs and features. These tools allow developers to integrate advanced AI capabilities into their applications, providing users with richer and more interactive experiences. With these APIs, developers can leverage Apple Intelligence to build smarter apps that can understand and respond to user needs more effectively, driving innovation in the app ecosystem.

Writing Tools, Genmoji, Image Playground, and Siri: Empowering Creativity and Interaction

System-Wide Writing Tools

Apple’s new system-wide writing tools, integrated with Apple Intelligence, are designed to improve text rewriting, proofreading, and summarization across all applications. This feature allows users to refine their writing with real-time AI assistance, ensuring clarity and precision in their communications. Whether drafting an email, composing a document, or sending a message, these tools make it easier to produce polished and professional content.

Genmoji: Personalized Emojis

Genmoji is a fun and innovative feature introduced to add a personal touch to digital communication. Users can create personalized emojis using their photos, transforming them into unique and expressive icons. This feature leverages generative AI to provide a highly customized and engaging way to enrich messages and social media interactions.

Image Playground API

The Image Playground API empowers developers to integrate dynamic and creative image generation into their applications. This tool allows users to create original images from preset prompts or descriptions, expanding the possibilities for visual content creation. By utilizing generative AI, Image Playground makes it easy to produce unique and engaging visuals, whether for social media, messaging, or creative projects.

Siri Updates

Siri has received substantial updates, significantly expanding its capabilities and improving user interactions. The virtual assistant now supports hundreds of new actions, making it more versatile and useful across various tasks. These new functions are powered by Apple Intelligence, enabling Siri to provide more comprehensive and context-aware responses.

App Intents Framework

The App Intents framework has been updated to allow for deeper integration with third-party apps. This framework enables developers to define specific actions that Siri can perform within their apps, making the virtual assistant more powerful and interactive. With the ability to invoke app menus and access displayed text, Siri offers a seamless and integrated user experience.

New Siri Capabilities

Siri’s new capabilities include advanced contextual understanding and the ability to handle more complex commands. Users can now stack commands or ask Siri to perform multi-step tasks, increasing its overall utility. These improvements make Siri a more intuitive and practical tool for managing daily activities and accessing information quickly and efficiently.

Machine Learning and AI Frameworks: Pushing the Boundaries of On-Device Intelligence

Built-In Machine Learning Frameworks

Apple’s built-in machine learning frameworks offer comprehensive support for natural language processing, sound analysis, speech understanding, and vision intelligence. These frameworks are designed to leverage Apple’s hardware capabilities, providing developers with the tools needed to integrate advanced AI features into their applications seamlessly.

New Swift API for Vision Framework

A significant addition is the new Swift API for the Vision framework. This API simplifies the integration of advanced image and video processing capabilities into apps, including features like image classification, object detection, and facial recognition. By using this API, developers can more easily implement robust vision-based functionalities within their Swift applications.

Core ML Tools for Model Optimization

Core ML continues to be a cornerstone for running AI models on Apple devices. The updated Core ML Tools provide various optimization techniques, ensuring that AI models run efficiently on-device. These tools support converting models from popular machine learning frameworks into Core ML format, including optimizations such as quantization and efficient key-value caching for large language models (LLMs).

Support for Advanced Models

Apple has added support for running advanced models like Whisper, Stable Diffusion, and Mistral directly on Apple devices. This integration allows developers to leverage cutting-edge AI capabilities, providing high performance and low latency. A practical demonstration showed substantial improvements in response times and memory savings, underscoring the benefits of these optimizations.

Advanced Control and Integration

For applications with heavy workloads, Apple provides advanced control over task execution to maintain overall performance. This includes using Metal Performance Shaders for sequencing machine learning tasks with other GPU workloads and the Accelerate framework’s BNNS Graph for managing real-time signal processing on the CPU.

Xcode 16 and Swift Assist

The latest version of Xcode, Xcode 16, brings numerous features designed to boost developer productivity and code quality. A significant addition is the integration of generative models for predictive code completion. This new engine, specifically trained for Swift and Apple SDKs, predicts the code you need by using project symbols and runs locally on your Mac, ensuring privacy and fast results even when offline.

Swift Assist

A standout feature in Xcode 16 is Swift Assist, a powerful tool that helps transform ideas into code using natural language. This feature leverages a larger model running in the cloud to assist with coding tasks, offering suggestions and creating prototypes based on user input. Swift Assist can answer coding questions, help experiment with new APIs, and generate code snippets that integrate seamlessly with your project.

For example, Swift Assist can help you quickly create a struct for a classic Mac collection, add images, and even play sounds with just a few simple prompts. This tool aims to streamline the coding process, making it easier to bring ideas to life with up-to-date and modern code suggestions that blend perfectly into existing projects.

Improved Developer Tools

In addition to Swift Assist, Xcode 16 includes other updates designed to enhance the developer experience. These features include a single view of backtraces, a “flame graph” for profiling data in Instruments, and expanded localization catalogs. These updates provide deeper insights into app performance and help developers create high-quality apps more efficiently.

The first beta of Xcode 16, including the new predictive completion for Apple silicon Macs, is available now, with Swift Assist set to be available later this year.

Swift Language and SwiftUI

Swift, Apple’s revolutionary programming language, celebrates its 10th birthday this year. Swift 6 introduces several new features aimed at enhancing code safety and developer productivity.

Swift 6

Swift 6 brings compile-time data-race safety, making concurrent programming easier by diagnosing data races at compile time. This feature is part of the new Swift 6 language mode, which developers can opt into to incrementally migrate their code. Swift 6 also includes improvements to concurrency, generics, and introduces “Embedded Swift” for targeting highly-constrained environments like operating system kernels and microcontrollers.

Swift Testing

A new testing framework, Swift Testing, has been introduced to simplify writing tests. Swift Testing features expressive APIs, cross-platform support, and integration with both Xcode and Visual Studio Code. It allows developers to write tests using a simple syntax and includes features like tagging for organizing tests and parameterizing tests for reuse. Swift Testing takes full advantage of Swift’s concurrency features, running tests in parallel to ensure efficient and comprehensive testing.

SwiftUI Improvements

SwiftUI, the framework for building user interfaces across Apple devices, continues to evolve with new features and customizations. This year, updates focus on previews, customizations, and interoperability. Xcode Previews now use a dynamic linking architecture, reducing the need for rebuilding projects when switching between previews and build-and-run sessions. New customization options include custom hover effects for visionOS, window behavior and styling for macOS, and a text renderer API for advanced visual effects and animations.

SwiftUI also improves interoperability with UIKit and AppKit, allowing developers to use built-in or custom UIGestureRecognizers in SwiftUI views and set up animations on UIKit or AppKit views driven by SwiftUI. Additionally, new features like custom containers, mesh gradients, and scrolling customizations make SwiftUI more versatile and powerful.

iOS and iPadOS Updates

Apple’s iOS and iPadOS updates were highlights of the Platforms State of the Union, bringing a suite of new customization and security features designed to improve user experience and app functionality.

Customizable Controls and New Controls API

With the introduction of customizable controls, users can now enjoy more flexibility in how they interact with their devices. The new Controls API allows developers to create and integrate controls that can be added to the Control Center or even appear on the Lock Screen. This improves user convenience by making frequently used functions more accessible.

Home Screen and Icon Customization

iOS 18 introduces extensive customization options for the Home Screen. Users can now choose between light, dark, or tinted versions of app icons, which will automatically adapt to the selected theme. This provides a consistent visual experience across the Home Screen while allowing for personalized aesthetics. Xcode supports these new icon variants, enabling developers to ensure their app icons look great in any appearance.

New Security Features: Passkeys

Security continues to be a priority for Apple with the introduction of new features aimed at protecting user data. Passkeys, which offer a more secure and phishing-resistant alternative to passwords, are now easier to implement with a new registration API. This API automatically creates passkeys for eligible users during their next sign-in, streamlining the transition to this safer authentication method.

Redesigned Tab Bar and Sidebar in iPadOS

iPadOS 18 brings a redesigned tab bar and sidebar, enhancing navigation and usability. The tab bar, which floats at the top of the app, provides quick access to favorite tabs and transforms into a sidebar for more in-depth interactions. This flexibility allows users to manage their workflows more efficiently, whether they are navigating through channels in Apple TV or exploring other apps.

These updates are designed to make the iOS and iPadOS experiences more intuitive and secure, reflecting Apple’s commitment to providing a seamless and user-friendly environment for both consumers and developers.

watchOS 11 and macOS Updates: Enhancing Integration and Usability

watchOS 11 Updates

Apple’s watchOS 11 brings significant enhancements that expand the capabilities and functionalities of the Apple Watch. With a focus on making information access more seamless, watchOS 11 integrates Live Activities, expanded widget functionalities, and Double Tap support.

Integration of Live Activities

Building on the popularity of Live Activities in iOS, watchOS 11 now supports this feature, allowing users to receive real-time updates directly on their watch. Developers who have already implemented Live Activities for the Dynamic Island on iOS will find it easy to extend these functionalities to watchOS, offering users a cohesive and consistent experience across their devices. With tools like Xcode 16, developers can preview how their Live Activities will appear on the watch, ensuring a polished user interface.

Expanded Capabilities for Widgets

Widgets on watchOS 11 are now more interactive and contextual. Using the same APIs as iOS and macOS, developers can create widgets with multiple interactive areas that perform actions and update states directly within the widget. The new WidgetGroup layout allows for detailed information display and interaction without needing to open the app. Additionally, specifying RelevantContexts ensures that widgets appear at the most useful times based on factors like time of day or location.

Double Tap Support

Double Tap, a feature introduced with Apple Watch Series 9, is now supported in watchOS 11, allowing users to quickly control apps with a simple hand gesture. Developers can use the handGestureShortcut modifier to integrate this functionality into their apps, widgets, or Live Activities, providing users with a more intuitive and hands-free interaction method.

macOS Updates

macOS continues to evolve with new features that boost productivity and integrate more deeply with Apple’s broader ecosystem. Key updates include the integration of Apple Intelligence, new productivity features, and a focus on gaming.

Integration of Apple Intelligence

macOS now supports Apple Intelligence features such as Writing Tools and Genmoji, allowing developers to incorporate powerful generative models into their apps. These tools leverage Apple’s foundation model to improve text and image creation capabilities, enabling more sophisticated and context-aware interactions within applications.

Productivity Features and New APIs

macOS introduces several new APIs designed to streamline app development and improve user experience. This includes easier window tiling for better multitasking, iPhone mirroring for seamless cross-device interactions, and significant improvements to MapKit for more detailed and interactive map experiences. The introduction of user space file system support allows for more flexible data management within apps.

Focus on Gaming

With the latest advancements in Apple silicon and Metal, macOS is becoming a powerful platform for gaming. The new Game Porting Toolkit 2 simplifies the process of bringing high-end games to Mac, iPad, and iPhone. This toolkit increases compatibility with advanced gaming features like AVX2 and ray tracing, and includes debugging tools for HLSL shaders, helping developers optimize their games for Apple’s hardware. The expanded human interface guidelines also ensure that games take full advantage of Apple devices’ unique capabilities, providing an immersive and high-quality gaming experience across the ecosystem.

visionOS 2: Advanced Spatial Computing Capabilities

visionOS 2 marks a significant leap forward in spatial computing, enhancing both the user and developer experiences. Here’s a detailed look at the key features and updates introduced in this release.

Advanced Volumetric Apps and Spatial Experiences

visionOS 2 introduces advanced volumetric apps, which leverage the new capabilities of the platform to create richer spatial experiences. Volumetric apps allow users to interact with 3D objects and scenes in a more immersive and intuitive way. These updates include:

Volume Resizing: Developers can now use the windowResizability scene modifier to set the size of their volumes, making it easier to adjust and optimize the user experience. Users can also resize volumes themselves, providing more flexibility in how they interact with the content.
Ornaments for Volumes: This feature allows UI elements, such as controls and additional information, to be affixed to volumes. Developers can place ornaments anywhere along the edge of the volume, offering new ways to design user interfaces.
Dynamic Movement and Scaling: Volumes can now be set to have either fixed or dynamic scales. This means that 3D objects can appear constant in size or change size as they move closer or further away from the user, mimicking real-world perspectives.

New Enterprise APIs for Specific Workflow Use Cases

visionOS 2 introduces a set of enterprise APIs aimed at specific workflow use cases, making the platform more versatile for professional environments. These APIs include:

Spatial Barcode Scanning: This API allows developers to create applications that can scan barcodes in a spatial environment, enhancing inventory management and other logistical tasks.
Low Latency External Camera Streaming: This feature supports real-time streaming from external cameras, which can be used for various applications, including remote assistance and monitoring.

Updates to Input and Scene Understanding Capabilities

visionOS 2 brings significant improvements to input and scene understanding capabilities, providing developers with more tools to create intuitive and interactive experiences. Key updates include:

Hand Tracking Improvements: Developers can now decide whether the user’s hands should appear in front of or behind the content, offering more creative control over the app experience.
Enhanced Scene Understanding: The platform can now detect planes in all orientations and anchor objects on surfaces more accurately. New Room Anchors consider the user’s surroundings on a per-room basis, and the Object Tracking API allows virtual content to be attached to physical objects for interactive experiences.

Introduction of TabletopKit for Collaborative Experiences

TabletopKit is a new framework designed to facilitate the development of collaborative experiences centered around a table. It handles the manipulation of cards and pieces, placement and layout, and the definition of game boards. Key features include:

Integration with GroupActivities, RealityKit, and SwiftUI: This ensures that developers can quickly set up collaborative experiences that work seamlessly with existing frameworks.
Support for Spatial Personas and SharePlay: This allows for social games and collaborative activities, enabling users to interact in shared virtual spaces.

Conclusion

The Platforms State of the Union at WWDC 2024 showcased Apple’s unwavering commitment to pushing the boundaries of technology and empowering developers with innovative tools. From the introduction of Apple Intelligence to the comprehensive updates across iOS, iPadOS, macOS, and visionOS, Apple has provided developers with a robust suite of capabilities designed to create richer, more interactive experiences.

The advancements in machine learning and AI frameworks, the powerful new features in Swift and SwiftUI, and the extensive customization options in iOS and iPadOS demonstrate Apple’s dedication to fostering a vibrant developer ecosystem. These updates not only enhance productivity but also open up new possibilities for app development, enabling developers to deliver high-quality, personalized experiences to users.

The focus on privacy and on-device processing underscores Apple’s commitment to user security, while the emphasis on gaming and spatial computing highlights the company’s vision for the future of immersive and interactive technologies.

How did you find WWDC 2024? What features are you most excited about? Share your thoughts and stay connected for more insights and discussions on the latest advancements from mobile development.