Apple is about to mainstream AI and most people won't know they are using AI and Gen AI. Read below for more details of what Apple Intelligence is and how it works.
Apple Intelligence Overview
Apple first introduced Apple Intelligence at the WWDC 2024 keynote video and then yesterday at the September 2024 event.
Below are some notes from that WWDC video and after that read how it actually works.
Intelligence that is most relevant for you.
They are using generative AI to understand and create language, text, images and more.
New AI powered writing tools embedded in all apps.
Create original images using AI inside apps. They offer unique styles of images. You can use your images to customize those creations.
You can use AI to search within your Apple devices (e.g. show my all the documents that John sent me last week).
You can correlate all the information in your devices to be able to ask Siri for specific information (for example, "the event change this evening, will that affect me taking my daughter to her school, play tonight?)
Most AI models run in the device, but there are times when they will use "remote models" to process that data.
When those remote models start processing your data, they will run in a new "Private Cloud Compute" and data (supposedly) will never be store in that server.
Siri will understand now more natural language, including if you make a mistake and correct it in the same request.
You'll have tools such as "rewrite", "response suggestions", "summarize" and more in all writing tools, both Apple's and third party.
Apple mail will now provide summaries (instead of the first few lines) as your preview in emails. Understanding context (and that means they are reading all your emails). Same with notifications, including chats (messages).
You can use Siri, using natural language to create events with context.
Siri will understand context from previous requests.
You can use Siri how to do things in your iPhone using natural language.
Siri will have "on screen awareness", meaning, if you are in a text, that was sent to you, you can say "add him as a new contact for me" and it will know who your talking about.
You can interact with Photos, asking for specific photos and edit them using Siri requests. You'll be able to even say "add this photo to my current document".
They will open up that API, called App Intents, to third party developers (to do what I said in 14 above).
You can create custom emojis using generative AI, including using your own photos from Photos app and creating custom emojis out of that.
Image Playground, a new app that allows you to create all kinds of images for all your apps using Gen AI. They are offering also an API for third party developers.
Image wand will allow you to create new generativen AI images in any app that integrates that tool based on sketches you created.
You can record and transcribe audio using AI, including summarizing.
ChatGPT will be accessible via Siri. Completely seamless.
In summary, AI with personal context is what Apple is calling "Apple Intelligence". And it will be everywhere in your iPhone (and other Apple devices).
What I'm seeing here is that AI is being normalized by Apple. It will just be "part of life" since it's embedded everywhere in the device, and not called out as AI, but as a new function that allows you to do something. It's going fully mainstream, but most people will have no clue they are using AI, including gen AI.
To me it seems that the future is going to be about finding a balance between convenience, helpful AI agents and privacy. More convenience, less privacy.
How Apple Intelligence Works
Apple Intelligence Foundational Model (On Device)
This model is an "on-device" LLM called "Foundation Model" that runs locally on Apple devices. The model is fine tuned for specializations using "adapters" for Summarization, Tone Adjustment, Friendly, Proofreading, Refining, Urgency, Mail Replies, Query handling and more.
The model was then made as small as possible through compression and optimized to get the shortest time from prompt to response as possible.
They then applied the same as above, but on a model focused on images.
But what if the device does not have enough power in it's on-device model to perform that request?
In comes...
Private Cloud Compute
This is a remote cloud services that will process those requests by the user on larger models stored in the cloud. Apple claims that this new Private Cloud Compute is specifically designed to run these user queries privately and securely.
It runs on a new OS called Private Cloud Compute OS. It allows for private cloud extensions to continually expand its capabilities and a full Machine Learning (ML) stack for processing the models. It DOES NOT ALLOW remote shelling and/or has any persistent storage. This means no one can log in externally to it and/or any data is kept in the server during that remote session with the device.
For each user request from the device, the Private Cloud Compute is called and is fully encrypted on both ends.
When a request is made, Apple Intelligence "Orchestration" decides how this request will be handled, either locally (on device) or remotely (via Private Cloud Compute). It then draws on the Semantic Index to get context for the request and uses the App Intents Toolbox to take actions on the device for the user.
New APIs for Apple Intelligence
APIs, also known as Application Programming Interfaces, are connectors that allow developers to harness the power of a platform, such as iOS, MacOS, etc. To harness the power of Apple Intelligence, and to use it beyond Apple apps, Apple is introducing various new APIs so that developers of third party apps can add Apple Intelligence features into their apps.
There are APIs that will allow developers to add functionality for Generative AI Emojis (called GenMojis), generative AI writing tools such as summarization, proofreading, etc, Image Playground API for creating new images with various styles, APIs for Siri to take actions based on user requests using Apple Intelligence and more.
App Intents Framework
App Intents is a framework that allows developers to expose their app's capabilities to Apple system services like Siri, Spotlight, and Shortcuts. With the new expanded capabilities of Apple Intelligence, developers can now use Apple Intelligence to allow users to control their apps using Siri. What's critical here, is that because Apple Intelligence understands "context", a user could make a request such as "Add a border around the photo I took yesterday of my dog and send it to my wife via text". In essence, it should find that photo, edit it in Photos app, add a border, save it and then create the text and send that photo attached to my wife, since she is in my contacts.
Context...context....context...
These are some of the current domains they will launch with. More to come.
Running Your Model Locally On Device
The Machine Learning Framework on the device has capabilities for natural language processing, sound, speech and vision. Developers can extend these models using Create ML by bringing in their additional data to train the models with. For example, a photographer may want to train the image model to understand their editing style. Apple will also allow developers to bring in their own models to run and train locally on the device.
AI for Developers
Xcode
Apple has added AI into their developer app, Xcode. Xcode will now offer full code completion and suggestions based on comments in your code. All of this runs locally on your Mac.
SwiftAssist
This is a model that runs in the cloud and works much like ChatGPT does, when you ask it to create a piece of code for you, but, this is working inside of Xcode, so it can create the code there and incorporate it into your own app.
I think Apple's announcements are great on two fronts.
First, they added a bunch of great new features to their platform that will improve the user experience with context, speed and additional capabilities.
Second, and more importantly, they are making AI ubiquitous in all their devices (and platforms). They are not even calling it out as AI from a user's point of view. They are just super smart tools that will enhance your experience in Apple devices in various new ways.
Welcome to the normalization and full acceptance (wether voluntarily or not) of AI in your life (at least if you own Apple devices).
Comments