ULS Workshop Wiki


KylesPositionStatement

Building Ultra Large Software Systems is quite different than building smaller-scale software systems. Most of my experience is in the design of systems that are ultra-large scale in terms of the number of users and the hit rate on the system. I've helped develop systems that have over 50 million users and experience over 50,000 hits per day.

Building software on this scale creates a number of unique problems:

(1) Off-the-shelf application servers are usually not designed with this kind of scalability in mind. Features that may scale up to dozens of servers break down when the number of servers begins to reach into the hundreds. For instance, administering 500 servers is quite a different job than administering 50 servers.

(2) Programming techniques that work well with artifacts numbering in the hundreds also break down when the number of artifacts begins climbing into the thousands or hundreds of thousands. For instance, using XSLT for language translation purposes works well when we are speaking about a few languages and a few hundred pages, but becomes difficult when the languages number in the dozens and we have thousands of pages. Changes become difficult to track down and maintain.

(3) Capabilities of servers and other middleware software such as caching also do not deal well with extremely large sets of artifacts and user volume. When we begin considering caches that are larger than a single address space new techniques for caching have to be considered.

(4) Development Tools that work for teams of a few dozen do not work for geographically distributed, extremely large teams that number in the hundreds. Existing approaches in breaking down work and customizing IDE's become interesting when the number of customizations are very large.