Let me preface this that the Unity guys have been pretty good about hearing the major qualms that their community faces and eventually gets around to promising to improve things. There are also a lot of issues that only come up on certain platforms or are a matter of personal preference or specific issues of the game you're working on.
That being said:
1) Poor source control integration and large team tools. As you mentioned, lots of non-diffable binary files. The editor ignores the read only flag for scene files. The actual files you have to check in isn't immediately obvious. Some of this is being rectified in 3.5 with SVN and P4 integration. There is also promises of text based scene formats. UPDATE: The text-based scene format is now available in the Pro version of Unity. See here.
2) Slow, programmer centric UI tools. Each widget is its own draw call, which has a lot of overhead on mobile platforms. There isn't the concept of things like panels with animations and all that other fancy stuff that makes a UI feel good without rolling it yourself. There is a promise of a new UI system on the roadmap (3.6?). There are some third party tools but they're not great.
3) Really rudimentary particle editing. They're promising a new curve-based system in 3.5, though. UPDATE: This curve-based system, called Shuriken, is now available. See here.
4) You can't nest prefabs. A small issue but when you're used to working with the prefab system and all the power it provides you, it can get frustrating. This has been promised but no specific date. UPDATE: You now can nest prefabs.
5) Next to impossible to get an iOS game under the over the air limit. The binary alone is like 8 megs in a best case scenario. This isn't something that's easily fixable.
6) Null reference exceptions crash platforms that don't allow JIT compiled code. On standalone or web versions, NREs are caught. It's still an uncaught exception, but at least the application will try to continue running. On iOS, it crashes the device. You can put it in debug mode and catch some types of exceptions, but performance suffers.
7) When working on a multiplatform game, whenever you switch build targets you have to reimport everything and that takes a long time. I've worked around this by actually just having multiple copies of the project on disk. Apparently there's an imported asset server coming in 3.5.