So I’ve been as frustrated as anyone by the wide array of problems with Xcode 4, from frequent crashing to painfully long slowdowns while it does something important that it doesn’t bother to tell us about, etc etc.
I even tweeted a couple of times in frustration. Hopefully I kept from being too harsh unlike some I’ve seen (apologies if not). It seems easy for people (me included when in the midst of deep frustration and deadline pressure) to forget that there are good, hard-working people behind Xcode and the development tools. The ones I know certainly care a great deal about making these tools solid and excellent. I appreciate their efforts.
I’ve used pretty much every public developer tool from Apple for the Mac, with the exception of the Macintosh Programmer’s Workshop (is that what it was called?) that ran on the Lisa (I was a year too late for that one). And a ton of non-apple tools as well (Rascal, Consulair C, MacASM, MPW, Lightspeed Pascal, Lightspeed C, THINK Pascal, THINK C/C++, every version of CodeWarrior, SmalltalkAgents, Prototyper, ObjectMaster, C Master, etc etc). Even did a bit of early Objective C development with the precursors to todays Mac OS X tools on the first NeXT cube. In all that time I can’t quite remember a less stable set of of development tools than Xcode 4, with the possible exception of THNK C++ 1.0. On the one hand that’s troubling, on the other, it means things have been very good for us developers for a long long time.
One tweet I made was along the lines of “Apple should spend 1% of it’s cash hoard on QA for Xcode & Developer tools.” Other developers seemed to like that sentiment given the retweets and “+1” activity it got. Wasn’t the nicest comment I’ve made and this is the one that afterwards made me actually think about the problem of testing development tools.
Starting with, “ok, so say Tim Cook or someone in Apple management called me up today and took me up on that tweet but put me in charge of the effort.” Like, ” Ok big-mouth, here’s a Billion Dollar budget, all the authority you need, a great salary, and relocation, etc for your family, now get down here and lead the effort to fix this!” What would I do? (cue deep sinking feeling… 8-O). But ok, I’m a professional, I’ve been a professional developer, team lead and R&D department builder for software companies for a quarter century, what would I do?
Well, for starters, assume everyone on the development team is very very good and really cares about the project – everyone I’ve interacted with in the Xcode and developer tools team fits that description. So, it’s not likely to be a “weak programmers” problem. Process? Complexity? External forces? All candidates for sure.
Ok, let’s think about the situation:
Anyone who’s been in the software development industry for any moderate length of time knows the feeling of being forced to ship code you don’t think is ready because someone besides you sets the schedule (either Marketing, Sales or the realities of the buying season for your market). It’s not a happy feeling.
In this case the dev tools team is likely pushed/pulled by the hardware release schedule and are probably writing tools targeting hardware that’s half baked and buggy all on it’s own. Talk about a moving target! Don’t forget that there are multiple parallel hardware development streams all supported by this same toolset (multiple Macs, multiple iOS device, Apple TV), and multiple OS’s that are also running along depending on these same development tools (Mac OS, iOS, iOS for Apple TV) – and those are just the ones we know about! Who knows what else is going on in the secret labs? Rather mind boggling when I spell this out and try to draw a timeline of the various hardware and OS releases in the last 18 months.
Also critical to recognize that Xcode and related tools are a HUGE project. I don’t have direct knowledge, but it seems likely we’re talking multiple millions of lines of code. Millions. If you’ve never worked on a project that large, let me assure you it’s crazy big. Just finding the code that might be involved in a given bug is a real challenge. Never mind the edit-compile-test cycle time (!!! see below). “Just step through the code” means something rather different when there are that many lines of code involved. I don’t know how many engineers are on the team (or several teams I’d guess), but as you get more people involved things get exponentially more difficult to coordinate and there’s just too much code changing every week to keep up with the whole project. I’ve never personally been on or led teams larger than 15-20 engineers, but even at that size coordination becomes a challenge all by itself.
How huge? Well I don’t have access to the source for Xcode, but we do have access to llvm & clang so I grabbed those and did a sloccount on it:
SLOC Directory SLOC-by-Language (Sorted)
547769 tools cpp=453850,ansic=56962,objc=27864,python=6437,perl=1973,
374043 lib cpp=368997,ansic=3472,pascal=1542,asm=32
113405 test asm=110511,ml=1173,python=937,ansic=780,sh=4
67753 include cpp=60198,ansic=7555
36745 utils cpp=29453,python=4379,sh=1474,perl=876,ansic=255,
9178 unittests cpp=9178
9080 autoconf sh=9080
6185 examples cpp=4728,ml=1443,ansic=14
4032 bindings ml=2103,ansic=1929
2492 projects sh=2473,ansic=19
446 runtime ansic=446
0 cmake (none)
0 docs (none)
0 top_dir (none)
Totals grouped by language (dominant language first):
cpp: 926404 (79.10%)
asm: 110543 (9.44%)
ansic: 71432 (6.10%)
objc: 27864 (2.38%)
sh: 13127 (1.12%)
python: 11753 (1.00%)
ml: 4719 (0.40%)
perl: 2849 (0.24%)
pascal: 1665 (0.14%)
lisp: 651 (0.06%)
csh: 117 (0.01%)
exp: 4 (0.00%)
So without any of the UI (what we think of as Xcode), or any of the accompanying tools (IB, Instruments, etc) we’re already over a million lines of code. HUGE.
To get a sense of the edit-compile-test cycle I foolishly decided to run ‘make’ to see how long it would take to build llvm + clang. My i7 MBA fans are maxed and the build I started thirty minutes ago just finally finished. Thirty minutes despite an i7 chip and fast SSD. And that’s just the compiler pieces, no UI at all, nor the lldb debugger, or the standard libraries, etc etc.
Then we consider just how huge a leap Xcode 4 was from Xcode 3. It’s easy to think of it as a big stumble because of the problems, but instead think about how much the team added and or changed in the UI? Massive change – tons of new functionality, much of it rather amazing coming from Xcode 3 (really leveraging the new low level compilers). So, new compilers, debugger, new ObjC runtime enhancements, new technology underneath (really using gcd, blocks, etc etc), and the massive UI rewrite (and it’s a huge multi-app UI when you count all the pieces / tools involved), and new hardware and SDKs to target/support along the way. This was an incredibly ambitious project on a scale that very few of us Mac or iOS developers has experience with on a regular basis (I have for only one project in 25 years, and so have some of you other developers in the crowd I’m sure, but as a percentage…probably fewer than 10-20% (?) of iOS/Mac developers have dealt with million+ line code bases).
Ok. Now imagine how many bugs there are in the database against this codebase… Especially considering all the duplicates because of the limitations of Radar as it’s currently implemented (my post on that here). 8,810,881 is the first bug number listed in open radar in Jan 2011 and 10,969,283 is the most recent one March 1. That’s two million bug reports in 14 months, assuming they are consecutive (which seems safe to say), that’s just over five thousand bug reports every day. Let that sink in for a minute. 5,000 per day. wow. Now most of those are for OS X and iOS presumably, but still. The magnitude of these projects shouldn’t be underestimated. Talk about hellish bug scrubs…!
Ok. So we’ve been reminded of the scope of this project – HUGE! – but how are we going to stabilize it? Naive answer is, we’ll just hire a bunch of QA and maybe some more developers to help fix bugs! We have a BILLION DOLLARS, we can afford some more QA!
So, who do you imagine does QA on developer tools? … Yep, developers do. So, most readers will be familiar with the current iOS/Mac OS X developer market – insanely constrained would be putting it mildly. In 25 years I’ve seen some crazy stuff, but never offers of $10K for the email address of a qualified developer, payable if they agree to accept a job. That’s 10K to me just for referring someone who ended up getting hired. Wild.
Now, as a developer, how excited would you be to write test programs and work with beta developer tools to try and break them, writing up detailed and excellent bug reports for the issues you find… sound like a dream job? (if so, please contact @Jury asap and I’m sure he’ll put you in touch with the right people! :-)). I bet QA engineer is probably one of the least appealing Mac/iOS developer jobs most developers can think of.
So, how are we going to convince skilled developers to spend their days messing with flakey broken developer tools and trying to break them in reliable ways and then document this in detail? We’re going to have to pay them a ton of money – say double the usual bay area salary. Ouch. Even with that it still might be difficult to find people who’ll stick around. At some point they’ll have had all they can take. Huh, this is becoming harder than we thought. Ok, I got it. Unit tests, Automated testing, scripted testing of the UI using UI Automator (cool technology). Yah, so how excited are you to be writing all that test code? Thought so. Same problem, different angle.
So I got this far in my thinking and realized this is definitely a non-trivial problem. Which makes fixing Radar all the more important (see my post on that here).
I’d love to hear from an Apple Developer tools engineer or manager how the Xcode team does handle QA for developer tools. Now that I’ve tried to think it through, I’m amazed how good Xcode is, considering!
Anyway, that’s 1700 words of meandering to get down some of my thoughts relating to Xcode 4 and stability. And that’s following 1700 words on Radar. So I’ve procrastinated enough for one day. Maybe you have some suggestions to add in the comments? Go for it.
Thought I’d add some notes on things I’ve been observing via twitter about people having problems with Xcode and then having gotten those problems resolved. Consider these “tips & things to try if you’re having problems with Xcode SPOD or crashing on you regularly”.
- Multiple reports of deleting and re-installing. I myself found that a clean install made the difference between “unusable” and “nice!”. For 4.3.1 this was as simple as deleting the Xcode.app in /Applications and downloading a new one from MAS. Note: having another copy installed on another partition or USB stick or whatever can confuse the installer and it’ll update that one. The button in MAS should read “Install” if it’s not finding another copy to “Update”.
- Deleting derived objects folder (and/or the “build” folder from older projects) can help. Especially if you were using an older version of Xcode 4 which may have left less then pristine indexing (or whatever) files. Xcode Organizer window, Projects tab, … This has fixed the continually re-indexing problem that some have had.
- Look to see if your headers are including an individual file from a framework instead of the main framework header file (e.g., #import <QuartzCore/CoreAnimation.h> instead of #import <QuartzCore/QuartzCore.h> (99% sure this is what @Jury was talking about). And yah, I agree, this shouldn’t be a problem, but I’ve seen @Jury talk about how it might be and at least one developer then reported that changing this fixed his problems.
Any other tips that worked to make Xcode stable for you? Drop them in the comments and share the knowledge! Thanks.
P.S. I’ve recently posted a few Xcode 4 tips that you might find useful:
And I have at least one more I need to write up.
A dev who’s making his living using Xcode and who is experiencing problems with Xcode 4 hanging every hour replied to my suggestion to “Sampling Xcode during the hang might help you figure out what’s wrong, and would be good fodder for a radar” with:
Nope. Too busy to debug Xcode for Apple😦
While I can understand the sentiment (I don’t get paid to track down bugs in Xcode either!), I also think this is a bit short sighted and ignores a pretty important fact: Consider just how extensive the tools we get from Apple for free are. I mean, really: Instruments alone is pretty amazing, not to mention the OpenGL Profiler, etc etc. And Xcode 4, for all it’s rough edges, is a pretty amazingly powerful IDE. And before you chant “open source tools!”, have you actually used any? Like, say Eclipse? ugh. So, I think it’s worth considering if spending 1 hour a week to help improve the tools you use every day to make your living is a small price to pay for said tools or not. Even at my hourly rates, I think the answer is: this is a great deal. Give it a ponder and then do what feels right to you.