In pure voluntary open source, there generally will be only one open-source project in a given area. Innovation and invention are still possible, but the open-source philosophy is, as much as possible, to resist diversity and instead to press forward with a single solution with maximum effort. The few instances where there are two open-source projects trying to produce similar results are due to political issues: GNOME happened because of unhappiness with the license used by KDE, XEmacs came about because of a disagreement over how to add display support to Emacs, and Linux started up while AT&T was battling legally with Berkeley over BSD.
There are only so many open-source developers interested in a given area, and it is hard enough to create one community of them, let alone two. So if there is already an open-source project with goals similar to yours, then you should endeavor to join that project. If there is some very good reason why you cannot join with them--and it had better be an extremely compelling reason, such as they are using GPL and your business aim is to produce a proprietary version--then you should rethink whether you should be doing open source at all.
Sometimes diversity is what's needed. If you decide to start a project building something with the same basic intent as an existing open-source project, you need to understand that you will get fewer volunteers and that the differences must be clear and distinct. Over time, you might convince people that your differences are worthwhile and thereby attract more developers and users; and over the very long run you might be in a position to merge back with the original open-source project once the relative importance and attractiveness of the different design and implementation decisions become clear through use.
The story of XEmacs is illustrative here because it is perhaps the first instance of a commercial company working on free or open source as part of a business strategy, and it was also probably the first and most visible fork of an open code base.
In 1991, a company named Lucid, Inc.,1 was working on a product called Energize, which was an integrated development environment (IDE) for C and C++ on Unix. The design of this development environment involved very tight integration among the various tools: compilers, linkers, debuggers, graphers, and editors. The IDE needed a powerful editor to tie the whole thing together, and the Lucid designers chose Emacs. Moreover, the IDE was graphics based: It assumed a window system such as X rather than the character-oriented terminals of the day.
The then-current version of GNU Emacs from the Free Software Foundation was Emacs 18, which was a terminal-based editor relying on simple character placement for its visual display rather than on a graphics-based environment, which is standard today. At the time, there was another version of GNU Emacs called Epoch that had been developed at NCSA; this was a set of patches to Emacs 18 that gave it much better GUI support.
Also at that time, the next major release of Emacs, Emacs 19, was expected to integrate features from Epoch, but was long overdue. The Epoch maintainers themselves saw Epoch as an interim measure, awaiting the release of Emacs 19.
The developers at Lucid didn't want to tie themselves to the soon-to-be-obsolete Emacs 18 or to Epoch because those code bases were considered obsolete even by their maintainers. The Lucid developers wanted to use Emacs 19; the idea was that Energize would operate with the off-the-shelf version of Emacs 19, which most people would already have installed on their system anyway. In that way, Energize would make use, to some extent, of tools developers already had and were already using.
But because Emacs 19 wasn't done yet, Lucid decided to provide funding and resources to help get Emacs 19 finished. Many of the Lucid developers were fans of the Free Software Foundation and contributed to free software projects, so all their work on Emacs 19 was released under GPL, and they assigned copyright on all their work back to the Free Software Foundation. Lucid had no proprietary interest in Emacs per se--it was simply a tool that would enhance the value of their proprietary product, and they hoped there would be good PR associated with releasing the GUI-based Emacs when Emacs 19 had been delayed so long. Among the other work Lucid did for Emacs was to produce a user manual, which would be required for the proprietary product.
After Lucid went out of business in 1994 (for unrelated reasons), Lucid Emacs became XEmacs. To this day, XEmacs is about as popular as FSF Emacs because it still provides features and a design that many people find superior to the FSF's version.
As of fall 2003, the XEmacs project was looking at merging back into FSF Emacs, but there were still too many irreconcilable differences in the way the two projects viewed technical, programming, design, and organizational matters to make a merge possible. Although some commentators consider XEmacs to have been a divisive fork, others consider the competition between the two groups as positive for the rapid advancement of Emacs over the last decade.
If you decide to start a competing project it is very important that you establish good relations with the developers of the existing project. The last thing you want is to have them bad-mouthing your project on forums such as Slashdot. Also, you probably share many common goals with them and there are apt to be many subprojects you can work on together, such as establishing common file formats.
There will sometimes be cases where creating a competing open-source effort cannot be avoided. For example, when Sun released their StarOffice code as OpenOffice it overlapped with two existing open-source projects: Gnumerics and AbiWord. StarOffice was an existing proprietary office suite built to compete with Microsoft office. The lead developer for Gnumerics had been engaged in conversations with Sun about this overlap before OpenOffice was announced, and, as a result, he viewed the OpenOffice project in a good light and was receptive to working with Sun. However, no one on the AbiWord project was consulted before or after the OpenOffice announcement; worse, one of the AbiWord developers tried to start a discussion on one of the OpenOffice mailing lists about developing a common XML file format, but never received any replies from the StarOffice engineers. The result was that the AbiWord developers became suspicious of Sun's motives and did not work with Sun, and both efforts are the worse because of this.
One final point about creating a competing open-source project: Do not expect any people working on existing projects to switch to your project. Your outside contributors will come from people who have not yet gotten involved with either project, and you will be in direct competition with the other project for potential new developers. In the OpenOffice example, none of the Gnumerics or AbiWord developers switched over to work on OpenOffice.