You knew that Apple would not stay outside the car companies. Their latest plan is not to build a complete car – it requires skill that Apple does not yet have – but to be the caretaker and brain of a self-driving car. In other words, Siri for Everything in an autonomous or semi-autonomous car, and not just Siri / CarPlay for music and navigation that we have today.
Details can be found in Apple’s patent application US20200026288, “Guidance of Autonomous Vehicles in Destination Environments Using Intention Signals,” which was announced on Thursday. In other words, you tell Siri what you generally want to do – find a barbecue tent or the entrance to the mall closest to the food court – and Siri would find out what to do and where to park.
Some of the underlying algorithms based on back and forth with the (non) driver, plus gestures that hold and point the phone, seem pretty smart. But we doubt that Apple locks all rights on a smartphone that listens to the passengers, asks a follow-up question, gets more refined guidance and then tells the car where to go, or knows how to choose the best Asian merger restaurant 90 minutes away .
“A non-trivial challenge”
Apple’s patent application was originally filed on August 2, 2019 and was unlocked this week. Broadly speaking, Apple wants a patent for driving autonomous vehicles using a combination of voice commands, gestures (with your hands, the direction of your face or eyes, or tilting / pointing the phone) and touchscreen gestures. Apple says about his request:
This disclosure (patent application) generally relates to systems and algorithms for the use of different types of passenger signals for controlling the movement of autonomous or partially autonomous vehicles. “
… Given the multitude of choices that are usually available with regard to vehicle trajectories in real environments, the input of the occupants or guidance regarding the selection of vehicle trajectories (without traditional steering, braking, accelerating and the like) can be extremely valuable for the motion control components of such vehicles. Providing interfaces for such guidelines that are intuitive and easy to use, particularly in environments such as parking lots where detailed and / or accurate map data may not be available, can be a non-trivial challenge.
Apple’s plan to find a parking space for you.
The figure above is a flow chart, says Apple, which illustrates aspects of how the car, telephone and passenger can communicate via “intentional signals”. You know where you generally want to go (“the mall”) but you don’t know exactly which parking spaces are available. (Hashtag that one #firstworldproblem.) Apple and Siri will solve that.
The “navigation manager” starts with general ideas, such as parking at the main entrance of the mall. There the passenger and the telephone work together to find what you really intend to do, that is the store or the part of a store that you want to visit.
Scenarios for input via gesture or touch.
Park closest to Zumba Class
Another part of the application shows how the driver and car would handle. When you are near the ideal parking space, but there are two places available, for example, Siri can ask which and the driver / passenger can respond with a gesture of the telephone towards the desired location and the telephone knows the direction by its compass and accelerometer. Whether a camera in the car that follows the driver’s eye movements or guides the passenger’s eye movements, sees where the driver’s hand points, or even the direction in which the eyes move.
Some of the examples seem intended for Californians who don’t want to park the Tesla 150 feet from the entrance to the mall when there is AI that you can find the free parking just 100 meters away. That is important: it saves you unnecessary steps on your way to the Zumba class.
There are the usual “find me some coffee” concepts, or figuring out how to park near the garden center at Lowes (the search for bunches of bushes and small trees is an algorithm that I have personally used). You can also worry about how much power Apple has to send your purchases. We have not seen anything in Apple’s patent application that allows you to connect your own recommendation engine and preferences; your definition of “luxury Mexican” in California may not be El Torito, even if Apple wants to. (A problem that applies to all cars that recommend services.)
On the other hand, if you plan to park the car in parallel in an urban area, you may prefer to be on the side of the street where you have to cross (as a pedestrian) against heavy traffic, or it rains and you want the fastest line in the store. Assuming there is even one free parking space in the city, let alone two on the same block.
Are others now solving the problems Apple poses?
I have written in the past that when you drive to, for example, M&T Bank Stadium to watch the Baltimore Ravens play at The Bank (as the stadium is called), you really want directions to E Lot if it is closest to your seat , or you want Siri to negotiate the best price for parking within 10 minutes’ walk and ensure that there are private plots that are not managed by the stadium, and you want Siri to get you the best price, not just a spiff for Apple.
Apple assumes that many of these bits of drill-down information are unknown. But certainly, the parking lot of the mall and the parking lines are mapped, even if they are not yet on Google Maps (let alone Apple Maps). Drones or cameras that look down from the buildings of the mall can tell which exact parking spaces are free today when you enter. When cars have 5G telematics, everything is already in the main unit, accurate to the minute.
In general, Apple has many good ideas in its patent application. It remains to be seen who might dispute the application on the grounds that some of Apple’s concepts are deliberate but either obvious or already in use, generally under different conditions. As a passenger, who has not pointed to an open parking space while holding the phone in your hand and using it as part of your gesture? In-car cameras already follow the position of the driver in the seat, as well as the hand gestures (BMW is doing that now), and many follow the position of the driver’s head and even the direction of the eyes to measure distraction and vigilance.
This project could be Apple’s relapse when building its own car. (That and Apple CarPlay, which is enormously successful.) From 2014 to 2019, Apple’s Titan was roughly a land-based, autonomous, electrified vehicle project. Apple discovered that building a car is enormously complex, that there are legal obstacles to making it much more difficult than for phones or PCs, and you can’t build a world-class car factory in a few years. Apple also discovered that not everyone wants to run a contract factory for Apple, including BMW and Daimler, and if there was an agreement, the divorce court would have been closely monitored. Too many egos and everyone would want the last word.