My very first ZDNet column was about the pain being experienced by users upgrading to Red Hat 5.0, about a year and a half ago.
Version 5 featured a major upgrade of Linux's libc, which is easily the third most important and complex software component of a Linux system (after the kernel and "C" language compiler).
(And, yes, it's still a "Linux system," despite Richard Stallman's continuing moans to the contrary. His recently-stated points are nothing new, and neither are the explanations about why the term "GNU/Linux" is just plain dumb.)
The upgrade of libc in Red Hat 5.0 was a major overhaul of the library, going to a GNU-developed codebase more commonly referred to as glibc. Most importantly at the time, the upgrade broke compatibility with previous editions, meaning that software developed on that new version of Red Hat wouldn't run on earlier versions of Red Hat. But it also wouldn't run on other Linux distributions that hadn't yet taken the plunge into glibc. The result was a mess that lasted some months until the other distributions caught up.
I fear that with the recent release of Red Hat 7 we may be back in this mess again.
Lucky seven?
The Red Hat 7 package is certainly impressive, and comes with the first
non-WordPerfect keyboard template I've seen in a long time. But what's
most significant -- for the entire user and developer communities -- is
Red Hat 7's default use of new versions of the RPM package manager, the
gcc C compiler, and the aforementioned glibc.
The compiler is so new that it can't be used to compile the kernel on Red Hat 7 boxes -- you use a different (older) compiler, renamed kgcc, to do that. The use of this new compiler, and the new RPM (version 4) and new glibc (2.1.92), means that:
- Software compiled against the libc used by Red Hat 7 won't work on any other current Linux distribution, or on earlier versions of Red Hat.
- Software packaged on a Red Hat 7 system can't be installed on any earlier versions of Red Hat, or on other distributions that haven't upgraded to RPM release 4.
- Software compiled with the gcc compiler now default on Red Hat 7 will be incompatible with code produced by the current "standard" version of gcc used by most other distributions and by older versions of Red Hat.
On the surface, this is fair game. According to Erik Troan, vice president of product development at Red Hat, the company was caught between the need to innovate, meet deadlines, and provide a stable platform. "We were in a bind," Troan said. "We plan our releases well in advance, and we've delivered releases at regular intervals for many years." He added that a number of new glibc features, such as support for 16-bit languages and multiple hardware platforms, justified breaking some compatibility. "With Red Hat 7, we decided it was the right time to hurt forward compatibility in order to move to the next generation."
Fair enough. The company has gone to admirable pains to ensure that backwards compatibility (the ability to run older software and packages on current releases of Red Hat) has been maintained. Still, the Linux community takes Microsoft to task for its breaking of forwards compatibility in file formats as a way to strong-arm users to upgrade, and this situation is similar in some ways. The Red Hat upgrade isn't done primarily as a money maker as a Microsoft Office one is -- all this new Red Hat software is, after all, open source. Still there are certainly common frustrations. Users of perfectly good Linux systems will have to upgrade to get new Red Hat packages, unless they want to re-compile and re-assemble the code themselves.
Troan believes that Red Hat can use its position of leadership to help drive users into using more-advanced software. In the case of the move to glibc in Red Hat 5, Red Hat used a version of glibc (2.0.6) that wasn't a fully-cooked production release. "We didn't feel that the software would ever stabilize unless mainstream distributions used glibc and the benefits glibc 2.0 provided," he said.
Following Red Hat's lead, the other distributions eventually caught up, but the transitional period was especially painful for non-Red Hat users and software developers wanting to create software installable on all Linux systems.
Unfortunately, I suspect that the same mistakes are about to be repeated.
Version (out of) control
Red Hat 7's version of glibc hasn't been cleared for production use by the
glibc maintainers at the GNU project. The GNU web page on
glibc insists its most current stable release is 2.1.2 -- older than
the version already running in Red Hat's 6.2 package.
The fact that the GNU page doesn't appear to have been updated in a year makes one wonder whether it's the definitive source of glibc progress reports and stability information. Release information for glibc is all over the place. The GNU FTP site doesn't go past 2.1.91. Debian's "stable" version is 2.1.3 and its "unstable" one is at 2.1.94. And the freshmeat listing for glibc says 2.1.3 for production version and nothing for the development version.
In other words, glibc's maintainers haven't expressed confidence that anything newer than 2.1.3 will be stable, and you can't even find anything newer than 2.1.91 on the official glibc website. Yet something newer than both is exactly what's in today's shipping release of Red Hat.
The base code for 2.1.92 (and newer) releases of glibc are currently found only at the website of Cygnus, the influential development shop that's now part of Red Hat. Coincidentally (or maybe not), the main developer of glibc, Ulrich Drepper, has a cygnus.com email address. Drepper's glibc FAQ is totally different from the one on the GNU site that says the most recent stable version of glibc is 2.0.5c (!). To make matters worse, non-techie documentation on glibc is abysmal. While every patch and pre-patch for the kernel is well documented at sites such as Kernel Notes, to see what's changed in glibc you actually need to obtain and dissect its package, examining a file called "ChangeLog." (And good luck comprehending the ChangeLog if you're not a programmer.)
The situation with the gcc C compiler is almost as strange. The version in Red Hat is 2.96 which, according the to the GNU maintainers, does not exist. Indeed, the GNU gcc maintainers -- far more up-to-date than their glibc counterparts -- recently issued a warning about the gcc used in Red Hat 7.
Leadership or gamesmanship?
The inevitable question is, then: who's in control of glibc and gcc?
The GNU Project or Cygnus? The
Red Hat page on glibc
still says it's GNU. Yet the GNU website information is ancient and
its release recommendations are ignored.
And RPM, the package management system Red Hat invented that's widely used on other distributions, is also not immune from such games. While Red Hat 7 boasts the new (and also not forwards-compatible) release 4 of RPM, the (Red Hat maintained) RPM community site says the newest release is still only 3.0.4. RPM 4.0 is hidden away on the RPM website and there's no announcement that it's available, let alone ready for production use. It'll take weeks, maybe months, for the other RPM-using distributions to return to the same level.
Maybe this is also what Red Hat considers leadership at work, filling a void left where GNU appears to be dropping the ball on its maintenance of glibc, making its own non-standard changes to gcc, and silently synchronizing RPM development with that of its own distribution. The fallout experienced by users of other distributions, or by software developers wanting to use Red Hat to develop distribution-neutral products, is apparently irrelevant to the company.
. Such attitude is fair game for a competitive player in a marketplace of Linux vendors. However, I submit that Red Hat's actions, which needlessly throw parts of the community into disarray for a significant period of time, work against the company's long-term goals of portraying Linux as an attractive environment and a cohesive community. Furthermore, the inevitable scramble to catch up with Red Hat's new versions, or at least be compatible with them, diverts community resources and energy from the core task of making Linux -- all Linux, not just not just Red Hat's -- as good as it can be.
The cynical, including those who are wary of such "leadership," might wonder whether Red Hat is leveraging its ownership of Cygnus and its control of RPM development to forever have a leg up on other distributions on such core components as gcc, glibc and RPM. They might ask whether such actions indicate a desire by Red Hat to enhance its stature at the expense of other players in the Linux community, consequences be damned. It's not a critical problem, so long as all the code in question is open source. But it's still a serious nuisance that most certainly exacts a toll on the community at large, and it has the potential to cause real harm to newcomers not aware of these games.
In any case, there's got to be a better way to evolve the Linux world into better technologies than to use development-stage releases in production distributions. I vividly remember the problems that happened last time, and I hope the community will get through this transition with less grief this time around.
Errata
In last week's column, posted October 4, I incorrectly made an unjustified analysis based on the value of Apple stock. This comment did not consider a split of Apple stock that occurred in June 2000. The column was modified to remove this reference, and I believe the other points made in the column are still valid given the IDC report and other circumstances. However, I am truly sorry for the oversight and any inconvenience or upset this may have caused.
At the very least, perhaps it might be best to stick with the common wisdom that I've heard from many places: wait to deploy Red Hat until it comes out with 7.1. If we're very lucky, all this mess will have sorted itself out by then.
Are you willing to upgrade to Red Hat 7, complete with all its compatibility issues? Tell Evan in the TalkBack below or in the ZDNet Linux Forum. Or write to Evan directly at evan@starnix.com.
