In the last 10 years or so, I've worked on or witnessed a number of broken technology business processes. Most of the symptoms and causes appear over and over again, and it appears from the outside that the iPhone App Store process (if you look at it as a whole) seems like a colossal disaster in the making.
What is a technology business process? It a series of actions taking by various people in a business to perform a business function involving (or sometimes avoiding) some technological support.
A perfect example was the CheapTickets refund process I worked on in the late 90's. The process took 3 months to get refunds to people with unused tickets, and during that whole time no one actually knew the status of anything, the 5 people who worked on it were way overworked, and the company was losing every customer who ever needed a refund (plus those that they told). Eventually the costs of losing customers overcame the unwillingness to spend any money and they begrudgingly hired my employer (and thus me) to fix it (but only me, they were being cheap). After about 8 weeks or so I had built a new process and the software to make it work, trained the employees and it now took about 1 week with full status reporting.
Why did it get so screwed up? When a process (or company in this case) starts they don't plan on growth very well, so things sort of slide along for a while. Processes which are not immediately important often get overlooked. Then when it starts to be a pain, managers often start by making the existing employees work harder. If that doesn't work they raid other departments in the company (if politically possible) but that screws up other processes and generally creates stress in the new employees (how do I do this) and the old (I have to teach people AND do all this work). If the workers in the process complain management may point to budgets being limited or other "excuses" to keep things from bubbling up the chain. Spending money on tools (which might make people more productive) meets the same denial.
What often happens in these high pressure process environments is that the workers start to create their own "tools" to make their lives easier. A shared spreadsheet here, a couple databases there, and soon the process is being supported by unsupported tools and applications which although they seem to help the workers, actually makes the process even less effective. Management often approves of these "self-sufficient" initiatives as they seem to cost no money and make it look like progress is being made.
In the CheapTickets case, the poor workers had utilized spreadsheets, databases, the SABRE reservation system, boxes, files and post-it notes to try to make sense of the mess. Until I showed up no one had spent any time understanding what the process was, much less how it needed to work. By spending time with each employee, seeing where all the data came from and to, following the process through all its evil steps it was clear that it could be streamlined and simplified. Since I understood both the business and knew what technology could (and could not) do it was fairly easy to build something that worked for everyone involved.
Another big one I did was for Sabre's Corporate Air Pricing Group, same kind of kludgy manual process but on a larger scale, which cost Sabre millions a year in Airline penalties simply because they had no audit trail to prove if they were at fault in ticket errors or the Travel Agencies were. Yet despite the $ the IT people had no interest in such a "minor" process. So the CAP department hired us (and I wound up working with another guy on it from my employer). Same problem (no one cared, employees do the best they can, customers are pissed, and the company is losing $), and same result (orderly automated processes, full audit trail, better management, and tons of saved money).
So what do these two examples have to do with the iPhone App Store? Even though I am outside I do know a few folks inside, though mostly I can simply see the end result. Initially when they started the App Store team I know they raided other departments for personnel. They also seemed to predict the volume but apparently did not plan for it.
From what I can tell, the process isn't very automated, the reviewers are overworked, there is little supervision or approvals internally, and the lack of bug fixes on vital web apps used by iPhone Developers dragging on for months sure seems like a good sign no one is working on them (I still can't even log into the Apple Bug Reporter web app despite numerous emails to them, but I do get helpful automated responses telling me to file bug reports in the Bug Reporter!). Apple in the past has always been very supportive of its developers (I started as a Mac developer in late 1984) and I've always gotten good help from the Developer Tech Support group (where I worked for a bit in 1995). Apparently the App Store team is isolated from the rest of the employees, plus they are forbidden to talk to anyone even at Apple about what goes on there.
I'd say either the management of this group is incompetent or overwhelmed, or maybe they are neither but their management has hamstrung them with the usual excuses (you can't exceed your budget, sorry); either way the end result reeks of CheapTickets' refund process or Sabre's Air Pricing (Corporate was bad but the general group was even worse). For a company with $29 Billion in cash, complaining of budget troubles is rather silly. I imagine that Steve either doesn't care since it doesn't impact products directly, or with him being out he's not getting the full picture.
Business processes are everywhere, you interact with them everyday. Most companies big and small have terrible ones. Fixing them is not always expensive (Sabre's ROI was about 6 months) but it often scares management types since you do have to spend some money (try fixing your dead car with no money) and it means change (people hate change) and it might make them look bad (you had to hire someone to fix your processes, you must be incompetent).
Fixing stuff like this is actually fun, when they are willing to accept that it's necessary (like an alcoholic admitting it); generally the workers are happy someone is listening, the customers (if you get to interact with them) are happy the company cares, and then when its working you get all sorts of thank yous (unless you are incompetent and screw it up).
Right now I'd love to get inside Apple and take a look. Fixing a bad process is a process itself and requires understanding of a lot of areas, both business and technical, and being a patient interviewer. Plus you have to be a good storyteller and management soother.
Of course fixing the App Store would make me happy too. Maybe my damn apps might get approved in less than geologic time.
One can always hope. That is itself a process.
teju 05/10/2009 23:11
Technology has widely exploded. Its great.
Ben Smith 05/11/2009 03:25
Hope you don't take this too critically, but I find your writing style very hard to read (There are too many brackets!)
php 05/11/2009 04:14
Nice post!
@Ben I like the brackets actually. It makes sidenotes / things that are off the topic very easy to read & understand. You can always skip reading the contents in the brackets if you want.
not ben 05/11/2009 04:42
@php - there's so many of them though. if one were to skip them, they'd be left with only half the article and probably little understanding of the author's point.
it's annoying, but everyone has their own writing styles. if i want to keep reading his blog i suppose i can adapt.
Frank de Groot 05/11/2009 04:44
Thanks for a very well-argued article!
I have read about AppStore's problems before, but this is the best analysis so far. And the braces are totally OK.
george 05/11/2009 05:14
this article is a bloated mess.
Robby Slaughter 05/11/2009 05:31
There are whole companies (like mine, http://www.slaughterdevelopment.com/) dedicated to helping organizations identify problems with business process and addressing them.
However, it's hard to know if the App Store is a broken business process or a broken business model. Certainly, it takes too long to approve new applications and one would assume that Apple's staff is overworked and frustrated by the volume. However, the very idea of mandating a centralized clearinghouse seems counter productive. Hundreds of thousands of applications flourished on other mobile platforms such as the Palm and Pocket PC. There is no equivalent requirement for applications on the Mac. The App Store itself may be a broken model.
The biggest challenge of any business process assessment and redevelopment is the possibility that the analysis will reveal fundamental problems in the business process. No organization wants to hear that their approach does not make sense. A review of business processes may uncover the most difficult truth of all.
@robbyslaughter
Ed 05/11/2009 05:50
I really liked this post. As someone who is involved in this type of work on a daily basis - I enjoyed your stories about Cheaptickets and Sabre and can really relate to them. As for Apple, the mind boggles as to why they are not all over the app store and it's inherent process problems. One can only assume that this is growing pains and that they are all over this in the background and that some day soon things will get a lot smoother.
codist 05/11/2009 06:49
If Apple would admit it had troubles in their processes or model, that would be the first step. Generally people won't admit things aren't going well since it makes you look bad. Often it take extraordinary pain to get them to admit problems, just as in personal situations. Once you are willing to say it's not working, then getting help is possible.
Sounds exactly like an ad for Alcoholics Anonymous. SInce the App Store is booming I assume the problems are being ignored.
Marcus 05/11/2009 11:22
I'm really glad you identified the management and the overwork. It's amazing how this little detail, the simple overwork of the people at ground-level, is the one aspect that's ignored so very much. It's something I've experienced he wrong end of way too much, and it's a lot of stress. (Then again, it's not all that amazing... ;-)
HJ 05/11/2009 12:45
Liked this very much, and good writing style. Sort of made my day here :-)
Keep writing...it seems you have experience to share.
bobert 05/11/2009 14:09
Robby Slaughter said: "the very idea of mandating a centralized clearinghouse seems counter productive. Hundreds of thousands of applications flourished on other mobile platforms such as the Palm and Pocket PC. There is no equivalent requirement for applications on the Mac."
The FCC requires that any cell phone in range of a tower and with a working battery must be able to successfully call 911 - even if the bill hasn't been paid! If someone were to die because they couldn't reach 911 because they installed a malicious or poorly-written third-party app, the FCC would have the OS manufacturer on the hot seat real damn fast.
Also, the carriers (Verizon, ATT, etc.) are paranoid about software that could tap directly into the phone's radio system and hog the local cell or something.
So people like Palm and Microsoft and my employer spend a lot of time and effort and money locking down their OS's so apps just can't get at anything that would impair phone functionality or do illicit things with the radio.
Apple took an easier path, which I can't blame them for at all knowing how hard the alternative is. They only authorize "safe" OS routines for use by third-party apps. Then they funnel all apps through a vetting process, and reject apps that use unauthorized OS routines.
Apple's failure to recognize adapt to the success of the App Store and staff it adequately reminds me of the silliness at Radio Shack around the TRS-80 thirty years ago. A lot of their executives didn't expect it to sell well, and kept going to the press and saying things like "We expect to sell 15,000 units this year." That would be immediately followed in the article by "Tandy released sales figures indicating the TRS-80 Model I sold 175,000 units THIS MONTH." It was quite funny to see how such well-paid top execs were so completely out of touch with reality.
And as far as broken business processes...
Robert Townsend cites a management consulting firm who made big bucks in the 60s "computerizing" companies. They'd get called in and told, "We hear we can save lots of money by using computers." And the consultants would say, "Oh, yes, you can save perhaps 10% of your costs of doing business." The consultants would get a contract, and then say, "Oh, well, we'll have to modify some of your business practices in preparation for computerization." So they'd go around and ask people, "How can we do things cheaper and more efficiently," and collect the answers and present them as a report on how to prepare for "computerization." The company would adopt these practices, and tell the consultants, "They're all in place - we're ready for the computer." And the consultants would say, "Guess what? You just exceeded your savings goal by adopting those new business practices. Skip the computer." Then the consultants collected a big fat check.
EdMike 05/11/2009 17:22
"However, it's hard to know if the App Store is a broken business process or a broken business model."
I don't agree with this. It seems to me that, at least where business is defined as the process of making money, that the App Store is a perfectly viable business model. I don't know how much money it is making but it surely isn't bad given it would not be expensive to run. Also, it has huge indirect revenue generation in terms of increasing the demand for the iphone itself.
To me I think the original author hit the nail on the head. Apple needs to improve its business processes, ensure adequate staffing levels and keep collecting the pay cheque.
Doug Petrosky 05/11/2009 19:18
Maybe, but prove it to me!
I'm sorry but by what standard is this process a mess? When in the history of programing have you ever seen a company turn around any where near this many applications in this short of a period of time with only a handful of news worthy complaints that are normally resolved in less than days!
I mean seriously! If you were going to publish an application through any other company in any other way we are talking about a process that can easily take months and most app store apps hare turned around in days to weeks. Sure there are a small number that just don't yet fit into the model that get hung up for longer. And some get through and need to be pulled and a few even get bounced and then get approved. But 35,000 application (plus updates) in 6 months! Ignore the updates that have to be processed (which probably increases the number 4 fold) and if you have people working 7 days a week 24 hours a day you are approving an application every 5 minutes or so!
I'm not saying it can't be improved on, but I haven't seen any evidence that it is a "broken business plan" in any way.