Comments on Entropy Overload: Delphi XE2 compiler performance

Think I just realized why a project of mine takes ...

2012-01-26T20:53:14.686Z

Think I just realized why a project of mine takes so long to compile. I'm making heavy use of namespaces in a D2009 project. I used them to avoid some unit name clashes with some existing code I was using.

Think I'll do a comparison and see how much difference it makes.

All platforms the compiler targets, yes. The cache...

2011-10-10T11:43:07.877+01:00

All platforms the compiler targets, yes. The caches only affect the front end, for things like symbol lookup and (particularly) unit name resolution.

Are these caches and the environment variable appl...

2011-10-10T05:31:49.886+01:00

Are these caches and the environment variable applicable to all platforms in XE2?

I specifically didn't use the term "re...

2011-10-06T11:17:02.869+01:00
I specifically didn't use the term "regression", because clearly a new problem introduced by a new feature is not a regression.

I disagree. Customers whose code, unchanged, compiles slower in version n+1 than version n, will say that there has been a performance regression in version n+1.

The problem being solved is performance. The regression is in the performance of compiling code - unchanged. The new feature is namespaces throughout the RTL to help in scoping code targeting multiple platforms (another new feature). But I don't think it's wrong to call the lack of performance in compiling unchanged code a regression, so it needs a fix.

I used the term "retrograde step", to characterise what you appear to be saying: that the new compiler - as a result of these new features - is now significantly slower in large/complex projects (if build times of the 2m LOC project were previously comparable to those I have experience of).

During development, yes, the compiler was slower on unchanged code. That was the whole point of fixing it.

i.e. a backward step in terms of productivity/performance

During development, yes. All sorts of products have ups and downs in performance while they are in development. That's why we try to fix these problems by release. And that's why I'm happy to say that the Delphi compiler is as fast as, or faster, than XE in almost all use cases.

This I am sure will be a matter of some concern for people with such large projects

Jolyon, I don't know how to make it any clearer to you that the performance regression is in the compiler that was not shipped. You are inventing problems based on a misunderstanding of the situation. Please clarify your understanding.

Assuming for a moment that an existing 2m LOC project could be successfully compiled "as-is" as a VCL application in XE2, would build times for that project be expected to be slower in XE2 vs XE or other earlier versions ?

I would expect 32-bit build times in XE2 to be faster than XE for many, if not most, large complex projects, and that has indeed been the feedback from customers.

The 64-bit back end is new, so hasn't had as much time to bed in. But then, XE didn't have a 64-bit compiler.

Ok, so "flabbergasted" would have been a...

2011-10-05T23:43:53.567+01:00
Ok, so "flabbergasted" would have been a better description than "speechless". :)

I specifically didn't use the term "regression", because clearly a new problem introduced by a new feature is not a regression.

I used the term "retrograde step", to characterise what you appear to be saying: that the new compiler - as a result of these new features - is now significantly slower in large/complex projects (if build times of the 2m LOC project were previously comparable to those I have experience of).

i.e. a backward step in terms of productivity/performance

This I am sure will be a matter of some concern for people with such large projects who in many cases will have no plans to migrate those projects to support anything other than the VCL yet - from what you are saying - might still be impacted by this.

Bottom line:

Assuming for a moment that an existing 2m LOC project could be successfully compiled "as-is" as a VCL application in XE2, would build times for that project be expected to be slower in XE2 vs XE or other earlier versions ?

Arnaud - it's not exactly the same, no. Some o...

2011-10-05T16:02:29.100+01:00
Arnaud - it's not exactly the same, no. Some of it is similar, but other bits are different.

Is it not exactly the same kind of optimization th...

2011-10-05T15:47:18.673+01:00
Is it not exactly the same kind of optimization that Andreas introduced in his http://andy.jgknet.de/ patches?

Jolyon, you are clearly not speechless. I am sayin...

2011-10-04T22:36:07.874+01:00
Jolyon, you are clearly not speechless. I am saying that the namespace lookup situation is entirely new in the XE2 compiler because the runtime library has changed, but the compiler was not heretofore designed for it. It's in the context of that, that compile times were reaching over two hours in certain projects (specifically, certain patterns of unit uses).

That is to say, it is not the compiler which has regressed (actually, in many if not most cases, it is faster than before); the code it is compiling stressed different parts of it, owing to pervasive use of namespaces where there were none before. Specifically, default namespace lookups (to maintain backward compatibility with code that doesn't explicitly use namespaces) turned unit uses from O(1) into O(n). This in turn caused overall project unit uses into O(m*n) instead of O(m). The explosion in unit name lookups turned into an explosion in file system lookups. The only proper fix for this (the file system lookups in particular) was a cache. But in building the cache, other opportunities could be taken.

Actually, I should clarify that my reaction is bas...

2011-10-04T20:30:41.431+01:00
Actually, I should clarify that my reaction is based on having worked on projects in excess of 2 million LOC where full build times of more than a few minutes was considered unusual.

Of course, if that particular 2m LOC project you reference always used to build in 3 hours, even in older compilers, then this isn't the retrograde step it might appear to be.

Some clarification on that point would be appreciated.

I am speechless. You seem to be saying that this ...

2011-10-04T20:24:56.952+01:00
I am speechless.

You seem to be saying that this caching behaviour is entirely new in the XE2 compiler, is that so ?

If so, and that build time of 3 hours on a 2 million line project is indicative of "normal" compiler performance without these new, fancy but broken, caches, then this represents a HUGE and worrying retrograde step in terms of compiler performance over previous versions.