The first time a programmer told me that he was planning to use a graphics card on a project, it was to borrow a little of its memory, because the PC he was building a networking application for, had run out. That was in 1987 and it serves only to show that there is little new in the world and that an envious eye has always been cast on specialist graphics processing engines. Now perhaps it is the turn of App Store developers to cast that envious eye over graphics power.
This is perhaps because Graphics is an interesting separate technology to the rest of computers, an area which went multi-threading, with parallel pipelines of instructions, like tiny super computers, at a time when general purpose processors were quietly drifting towards RISC designs, simplified by the free availability of what had once been expensive, super-fast, register memory. CPUs and GPUs don’t work the same and rarely talk to one another except to hand over jobs for each other to do.
During the 1980s we learned how to build cellular networks. In the 1990s we learned how to make them both digital and more reliable, using FEC techniques and anti interference modulation designs (CDMA). Since 2000 the journey has been about making them carry more data, and now in 2010 we want the power of the devices to be able to render high definition, 3D images so we can play games and watch video on them.
This last move is where Apple came in. It oozes graphics as a company. It is only successful, nay dominant, in graphical applications on its Macs. Playing and editing electronic documents and then video and music has been where it has held its ground during the survival years in 1994 through to 2004, when the company was under threat and had its smallest ever market share in PCs. It kept Quark Express loyal to its architecture, it launched Final Cut Pro and Final Cut Studio, and now has iLife and iWork and Aperture and any number of graphical software elements that secure it against the Adobe’s of this world, which attack the same space, but mostly on a PC.
So it is not surprising that at a point in history when we are most in need of graphics, that Apple has come to the fore. Apple devices ARE graphical. Video needs decoding and displaying, games need graphics rendering, even Touch screens need to show graphical icons and respond through geographical awareness. 3D will need shading and perspective calculations galore. Especially on the new generation of tablet style computers led off with the iPad.
And one of the core design issues for handsets, tablets and even netbooks, is how to get enough power in one place to do the job. There was some insight into this when AMD bought ATI Technologies in 2006. Graphics were coming together with processors ‘ on PCs at this time.
And as we said at the beginning of this piece, in traditional architectures, such as the PC, there has always been the desire to harness the graphics processor to carry out more jobs. In mobile there is a different imperative ‘ and that’s to do the same thing, but in the process save power.
It may have been a surprise to us that during the mid 2000s the trend became for companies like Qualcomm to try to bring baseband radio chips onto the same piece of silicon as the applications processor, while companies like Texas Instruments wanted to get out of baseband and build the entire remaining parts of a mobile phone, onto a single chip, an SoC.
When Apple changed the paradigm for smart phones it was to push them in ever more graphical directions, and that in turn has led to the same question being asked, should we put graphics cores into the same piece of silicon as ARM cores which handle applications.
The thing that made this highly desirable was the process whereby general purpose work can be offloaded to a GPU, when graphics are not needed. This allows us to cater for more exotic chip designs, including more and more GPUs on a chip, which in turn can either do graphics or something more mundane, changing from one second to the next.
But to do this each cellular chip which had a graphics core put on it would need special software to manage the communication and sharing of work between the two processors, and a special set of APIs to make all that power available in a C or C++ style language so that developers can get their hands on it.
Once that is done such diverse core designs can even share memory in the same way that multiprocessing was introduced on servers in the 1980s. So the opportunity to process jobs in parallel on a handset or tablet chip, appear to be just around the corner. It is this that has led to quite mistaken rumors that Apple might buy ARM.
The misunderstanding here is that because Apple bought PA Semi, a specialist in ARM cores, which in turn allowed Apple to create the A4 chip which the iPhone 4 and iPad are based on, in super fast time. But PA Semi did NOT own the ARM architecture, it owned expertise to move ARM chips a little further, and integrate ARM better with graphics, in this case not the Mali graphics cores from ARM, but with ‘ non-ARM graphics.
On the iPad and iPhone 4.0, Apple has relied on licensing specialist graphics cores from Imagination Technologies, a 20 year old UK specialist which got its start in life merging graphics onto PC architectures under the name VideoLogic, and it has now taken a page out of the ARM book ‘ and licensed its graphics know-how to chip makers in the form of designs that can be added to their chips. This includes Apple, Samsung, Intel (on its ATOM CE 4100), Freescale, Marvell, NXP, Renesas and Texas Instruments. Err that would be almost ALL the majors, barring Qualcomm.
These graphics cores have turned up so far in devices from Nokia, Asus, Fujitsu, NEC, Sony, Motorola, and even LG and HTC though not their mainstream cellular products.
These particular trends in cellular chips are suddenly about to go through a period of acceleration, because in June there was a new Version 1.1 of the embedded Open CL software toolkit which began to be distributed. OpenCL began (naturally) as an Apple project. It competes with Microsoft’s DirectX, which now has almost no support outside of Microsoft. It’s that same idea again, to harness Graphical Processing Units when they are NOT doing graphics, to help other general purpose jobs.
The overall concept is called GPGPU and it means General Processing on Graphical Processing Units. Apple gave it its start in 2007/8 and then made it open source. In this way it encouraged ATI and Nvidia to give up on their own ‘proprietary’ versions of GPGPU and adopt the Apple route, in the hope that it might win them favor and business in Apple products.
This move is about to bear fruit in the age of tablet architecture, where graphical elements will need to run faster and faster, and support a shift to 3D imaging on portable devices.
This idea has come of age and when you think about it, it works both ways. What is to stop a GPU supplier or core designer creating a GPU or a cluster of 3 or 4 cores, which do all the application processing for a phone or tablet, depending upon which cores need to be active at any point in time. Instead of having 4 ARM cores, why not have 2 and 2 GPU cores? All on one chip.
In June this year Imagination Technologies added OpenCL to its own multiprocessor design, the POWERVR SGX544MP multi-processor graphics IP core. Apple currently uses the POWERVR SGX545 chip on the iPad and iPhone 4. Imagination has been saying that theoretically its existing core suppliers can do this by releasing their own drivers and APIs. Now they can simply take them from Imagination.
It is widely expected that the next generation A4 chip will implement OpenCL using one of these cores. Apple has already implemented this last year on its Mac OSX Snow Leopard for the MAC, so already has some experience in how powerful a concept OpenCL is. It makes all GPU chips which process OpenGL graphics, capable of running application programs, which in turn makes it more desirable to put more of them into a design.
The new POWERVR SGX544 can be implemented as a 4-pipe single core, or in multiprocessor configurations of 2 to 16 cores creating 8 to 64 pipes. That will allow the type of floating point arithmetic used in early engineering workstations to be aimed at tablet designs, but within the power requirement of battery powered devices.
We already have considerable CPU parallelism in handsets, although this mostly means programs running in parallel, as opposed to parts of programs running in parallel. What this next step will give the world is CPUs being mixed up with GPUs on an equal footing. Developers will be able to harness any processing element on the device, and have parts of a single program work in parallel, which will give it blazing speed. GPUs are already designed to execute multiple threads in parallel, and now that power can be added to other applications and borrowed by other GPUs.
This is a cheap way of bringing the parallelism of something like the IBM Cell chip, which has 9 processor and powers the Sony PS3 gaming device, and putting it in a portable device. The gains are from running multiple parallel computer processes on the same chip, with less and less interconnection traffic between processing elements.
Qualcomm has, in the past, talked about devices which behave differently when plugged in, using more power and turning on dormant cores, so that a tablet might suddenly become a HD gaming graphics beast, when plugged in. It is this type of chip integration that might make that possible. Apple is likely to get to the front of this queue by owning a piece of Imagination Technologies, as does Intel.
This is not to say that Qualcomm isn’t already capable of producing cores in much the same way, it is. But until there is a guaranteed selling, iPad class, product, launched by one of its customers, there is not much point in a company like Qualcomm, the leader in cellular chips globally, making such a beast available. Qualcomm will offer its Snapdragon with and without cellular connectivity, it offers Gobi, which is only cellular activity, but it prefers to offer its chips with everything bundled in ‘ quite simply because it is harder to do and it can differentiate itself from other chip makers. But there is no point in going through this type of innovation unless a customer is asking for it. Apple is both customer and chip supplier, so it can co-ordinate the approach better.
Texas and Broadcom are in the same boat ‘ ready and willing to go through the next phase in battery powered SoC device development, but waiting for customers to emerge who know what they want. In the current era, customers rarely know what they want until they see Apple launch something, and then they copy it.
Imagination Technologies says its new OpenCL compliant chips offer the highest performance per square millimeter of silicon and the highest performance per mW (milliWatt of electric power), with the lowest system overhead. It also means that for every Imagination Core added to a design, it is potentially the loss of an ARM Core. Currently the average phone has 2.5 ARM cores.
This represents a considerable threat to graphics chip vendors such as Nvidia or ATI. They have both embraced the GPGPU approach, because it makes their chips more helpful in non-battery powered devices.
But in the long run it means that Nvidia and AMD’s ATI might find that multiple core types, some with in-built graphics acceleration, some without, can be customized at run time to offer power for highly graphical applications and they will each have to offer core designs to other chip makers, rather than chips, or offer their own SoCs (something we know they are trying) in order to retain handset customers. Either one might make a play at buying Imagination, but the chances are that Apple can veto any proposal of marriage to Imagination and will do so in order to retain market leadership. Right now it uses ATI designs in its larger Mac servers but it can switch away from that at any point.
Imagination has just brought out software stacks to make its new multiprocessing cores work with Windows Mobile, Linux, Android, MeeGo, WebOS, Symbian, Samsung’s Bada Nucleus, and a variety of other minor OS’s. Apple is always specifically omitted from announcements by Imagination, so IOS has not been mentioned. This step alone is likely to give portable devices enough graphical power to cope with 3D, shortly after it takes off on TVs.
This type of analysis just shows how focused Apple is in everything it does, and how it plans to hijack and remain ahead of all things graphical on devices which are portable.
To us the only real danger here is that ARM might make a play for Imagination Technologies, which is a public company. The two companies don’t like each other much, and are aware of the rivalry, but such a move might not upset Apple or any other Imagination customer except Intel. And again it would be particularly significant for Nvidia and AMD/ATI, and potentially fuels an Intel versus Arm based war as a long term industry trend.
Something which speaks against such an outcome is that ARM has its own highly respected Mali graphics engine ‘ but Apple didn’t choose it, and as far as we know it doesn’t quite yet support OpenCL, though it is committed to it. Mali has just 29 license wins, versus 267 for ARM 9, and only had three new licenses last quarter. So it has not exactly caught fire.
In our view an approach by Apple on either or these companies would ruin the grip of OpenCL at Imagination and ARM Cores at ARM, whereas another ‘arms dealer’ such as ARM might be able to broker such a deal without ruffling many feathers at licensees. Combined they would be stronger in our view. Such a move would perhaps lead to only Intel losing interest in Imagination. Intel has been mentioned as a potential suitor for Imagination as well in the past and is, like Apple, a shareholder which may give it some powers to block or counter bid in such an instance.
It was only this week that Microsoft signed a new licensing agreement for the ARM architecture, which has been interpreted as bringing ARM processing to a potential Microsoft driven tablet format. Microsoft already uses ARM designs for Windows Mobile products.
Expect the number of signings for both ARM and Imagination to act as pointers to the direction the cellular and portable chip industry heads in over the next crucial two years as tablets, and more graphical power takes a grip in portable devices.