Thursday, August 30, 2012

The deal with me and linux (or Why I don't use Linux unless someone is paying me)

I take quite a bit of flack for not being a Linux zealot. In fact I'm actually not particularly fond of any linux distributions out there and for some reason this invokes the ire of many a devotee who feels personally affronted by my lack of enthusiasm for using the operating system to which they feel a very deep and personal attachment. Here I endeavor to clarify my position and de-escalate the BS.

NOTE: If I sent you to this links it's probably because you're acting like a GNUb and I don't have time to keep re-typing the responses to the questions/challenges you pose. You can either read and learn or not. In the words of Dres from Black Sheep "...the choice is yourrrrrrrrssss...".

[update: 9/6/2012]: So, today Randal L. Schwartz found my post and shared it on Google+.  I saw more traffic in the past 3 hours than in the past 3 months. A few people have commented on my use of the term GNUb so I wanted to, here, clarify what I mean especially since more eyes are landing here and I genuinely do not endeavor to offend.

First I do not use GNUb to refer to all Linux, GNU/Linux or GNU enthusiasts. I have way too much respect for the likes of Linus, Simon Phipps, Alvaro Lopez (Cherokee Project), Thomas Hatch (Salt) and many others for that to be the case. The term is meant to apply to those who have very little experience with unix systems yet presume to know everything based on their (usually limited) exposure to GNU/Linux. These are the sorts of people that most of the hacker community can't stand but because their ire is pointed in the right direction (Redmond or Cupertino) no one challenges or corrects them.

The term GNUb is a bit callous and dismissive so I understand if you're offended. My hope, though, is that you realize I have no desire to disparage the GNU/Linux community as a whole, only those among you to whom the term truly applies.

Where to begin... 

I've been sitting on this post since early in June 2012 and have finally re-read and tweaked it enough times to think that maybe it's OK to post. Maybe I'll make this a living document and keep updating it over time (not likely but let's say maybe). The problem is that there is entirely too much ground to cover quickly and succinctly. One thing I should probably do is qualify what I'm about to say : 
    I've been doing some form of systems, software and/or security engineering since 1993 (yes I was in high school then). Bearing this in mind I have tremendous respect for people who choose to create software for the rest of us to use, especially those who do so without any expectation of monetary compensation (Free as in beer). Also, I admire those who fight to make sure that the rights of users of software are respected and upheld (Free as in freedom). 

I often use pretty strong language to express my dislike of or for various systems/tools. It is my hope that, despite that fact, you understand that I applaud all of the contributors who act with goodwill and in good faith to improve the landscape of Free and Open Source Software for all of us.

A (byte/8) about me (specifically my history with Unix-like systems)

Short version: I started programming in 1993 while in high school, mostly TI-BASIC (on my TI-81) and ASSEMBLY and BASIC on my 8088 at home. I was into the BBS scene and a buddy of mine had 'hacked' into a BSD/386 system and let me poke around a bit. I was smitten. Then in 1995 while a student at Northwestern University I got a job in the technology center (Vogelbach). The team I was on  was responsible for both servicing the technology needs of various departments across all supported platforms (Windows, Iris, DGUX, HP-UX, Novell Netware, Mac OS 7, and more) and running a helpdesk phone line for faculty and students seeking resolution on various internet service issues. I learned much.

A buddy I worked with, named Julian Koh, had lent me a copy of Slackware linux when I decided that I wanted to run a unix-like system at home. Fastforward a few months and I had about 8 computers at home with 4 ISDN lines and was up and running powered by a combination of Linux (slackware) and FreeBSD (thanks to another coworker, Daniel Johanson).

Since then I've been staunchly entrenched in the Unix world. I've supported just about every type of unix system you can name, at some point. I've been in small companies and large, always doing systems, software, or security engineering on unix (small 'u'). I've managed my own servers (actual hardware that I own) since 1995 and have used them in real world scenarios for various services and applications. One such service has been hosting the longest running US based mirror for LinuxMint since 2007. Giving back to the open source and free software communities is something about which I feel strongly.

The meat of the matter

The bottom line to me is this: using most linux systems is annoying to me. I know... that sounds mean ... it's okay... take a second.

Now, I don't mean the above to be offensive or provocative. It's simply a statement of fact. I don't hate your beloved OS of choice it's just annoying as hell to me and I choose not to use it. I'm not saying "Linux sucks!", just that I choose not to use it on any of my personal systems for the reasons listed below. 

I'm not dissing the kernel

I've stated many times in various posts that I think the linux kernel just awesome. I truly admire the kernel team and all of the work they have done. When I talk about linux systems being annoying to use, very little of that (if any) is the fault of the kernel.

Well there is NetFilter / IPTables

Ok IP Tables is a mind f**k that I wouldn't wish on my worst enemies. I have visions of a future where enough people realize that IP Tables was such a bad idea that it's synonymous with 'tried your best but completely screwed it up' and that get's used in sitcoms and vernacular:
 "Boy you really IP Table'd that one."
 "I know. I wish I were dead. Someone shoot me."
 "Heyyy..bud cheer up. Who knows next time you could totally PF it!"

OS not Distro

I like the cohesiveness of a system that's well put together. OS X, FreeBSD, OpenBSD, Solaris they all have one thing in common they are unix based operating systems (that I happened to run at one point on one of my desktop machines). Redhat, Debian, and Ubuntu are distributions where they take the kernel from one source the utilities from another source and cobble it all together. This isn't a fit of my pedantic nature shining through, there's actually a point here. 

It takes a considerable amount of energy to put together a system and get it working properly, there are many moving parts. If you control the kernel and the user land utilities you can make changes as you see fit. When (as in the case of Linux distributions) you get the kernel from X and the user land from Y (most often GNU) and paste them together you have to juggle pieces you don't control. You can patch things for your system and hope they get adopted upstream or you can fork and take over maintainer ship for the version that ships just for your system but either way it's a crapshoot and a bunch of work.

GNUbs say: ... but FreeBSD, NetBSD, and OpenBSD are all just distributions of BSD...

Yes and no. Yes because, let's face it, the D in BSD is, in fact, for "Distribution". All of the popular open source BSDs are based on on the 4.x BSD releases from the University of California at Berkley. No is where it get's interesting and sometimes too subtle for the GNUbs to grasp. Each of the BSD systems takes responsibility for the evolution of it's own kernel, user land tools, utilities, packaging system, etc. For example OpenBSD forked from the NetBSD project. When it happened they took the kernel and everything, modified it the way they wanted and it has since grown separately from NetBSD. The kernels share a common origin but are not compatible in the way you might think of Ubuntu being compatible with Debian.

Base System & Add on Packages

One thing I think most linux distributions get wrong is the whole idea of a base install or core system. Call it what you will but the idea is that I should know what's installed as part of my operating system versus what's an add on package. This concept appears to be lost on nearly every major linux distro.

To make matters worse, on some distributions it's unclear where certain files will be installed. For example, If I install the apache web server on distro-xyz. It may throw configs under /etc/httpd and actual content under /var/httpd/html. On the same distribution, if I install the lighttpd web server it might decide to install configs under /var/lighttpd/etc and html under /var/lighttpd/html. Stuff like that is pretty annoying and exceedingly commonplace in the GNU/Linux world.

File System Layout

On the BSD systems it's generally /usr/local/; on Solaris and HPUX it might be /opt/ on most linux distros it's /. WTF????

When I install a third party package it should be put someplace where I can easily identify whether a file is part of the operating system or not. On Debian, RHEL, Ubuntu, etc. when a third party application is installed it goes into the root of the system along with core utilities and tools like grep, awk, ed, vi, and such. Some things get thrown into  /var and /etc polluting the system. 

(Not) surprisingly the Linux distributions that take some pretty heavy cues from BSD systems have relatively sane ways to deal with this even if it's not set by default. Arch, Gentoo, Slackware and others fall into this category. Sometimes their solution is to just stay out of the way and let you run pkgsrc alongside their system package manager.

GNU/Linux User Land

Let's not forget that most Linux distributions are GNU/Linux. The GNU shell (BASH) is just annoying to me. I find it's conventions like declare and local to be useless bloat. The same goes for many GNU utilities like netstat which (re)appropriates -p to show process names despite the fact that on HPUX, Solaris, *BSD, and many other unix systems -p is used for protocol. In general the GNU user land utilities are pretty bloated and have odd options when compared to their non-gnu counterparts in both proprietary and free systems. Of course there are exceptions and, in some cases, justifications.

GNU grep is blazingly fast when compared to other implementations. Unfortunately if you were expecting it to respect the bounds of POSIX basic regular expressions (as is required by the standard) you'd be out of luck. Because it's so fast it doesn't need to discriminate between basic and extended regular expressions so be careful. 

Remember that GNU stands for GNU's Not Unix. In some very notable cases the FSF and GNU have fought hard to make sure that open standards are published and implemented in their tools. Often GNU tools were the first (if not only) utilities to actually be fully compliant with said standards (namely OpenGroup/POSIX). Where there has been room to implement outside the bounds of the standards they have in many cases chosen the most unconventional approaches. As someone who at one point supported 6 different flavors of unix at the same time working with the GNU versions of things like sed, awk, bc, make, tar, ifconfig, find, and roff was vexing to no end.

Add to the above, the fact that each distribution has it's own custom blend of GNU utilities plus some other random stuff they found somewhere to either supplement or substitute the parts from GNU.

With Ubuntu 12.04 Canonical has decided that it will no longer include the traceroute utility as part of the operating system. You can add it later if you decide that you want it but it doesn't come as part of the system (WTF?!?!?).  

GNUBs / GNUBes / GNUBies
Now that GNU/Linux is so prominent in the industry there is an issue of collateral damage springing up. That of GNU users being ignorant of things like standards and portability. I'm sick of seeing 'shell scripts' that are bash specific without noting as much. CLI extensions like pythonbrew and rvm while not trivial to implement could have been trivially portable to shells other than bash. It takes very little time to write portable code but now no one thinks about it because they are convinced that everyone is running GNU/Linux tools or should be. The same goes for other things like vi. note to all of the GNUBies out there: VIM != vi && GAWK != awk; and I say that as someone who uses VIM most of the time and loves gawk. 

I sit in IRC channels, browse, and watch forums where users ask UNIX questions and get GNU/Linux answers. In some cases this works out alright for that user sometimes they come back and state that something did not work and no one has any idea why. In either case I think it's bad for the unix community as a whole. 


Beyond the utilities from GNU, most linux systems come with far more turned on than I want or think is reasonable. In may cases just installing something (like a web server) results in the that same something starting automatically when you reboot the system. 

And come on colorls; really?!?!?

I need a system that just works

Randal Schwartz host of FLOSS weekly, international traveler, speaker, and world renowned Perl hacker get's flack from GNUbs and Open Source zealots for using a mac instead of some Linux distribution. In several talks he's mentioned that he needs a system that just works, he doesn't have time to keep tweaking his OS because he actually needs to get work done. I'm pretty much in the same boat. While some linux distributions offer a fare degree of customizability none of them come pre-built with the user land I like, a layout I can stomach, and a packaging system that doesn't make me want to pull my hair out. 

True I could cobble together my own system or endlessly tweak Arch, Gentoo, Funtoo, Slackware, DSL, etc but guess what.. FreeBSD, OpenBSD, DragonflyBSD and Mac OS X already work for me. Ok admittedly with OS X I have to go through the process of installing Homebrew and PKGSRC but that's trivial compared to just preparing to install Funtoo or Gentoo (which I've done a number of times).

Despite all of this

I'm still a fan of Linux; both the kernel and the GNU/Linux distributions. While I have no desire to use GNU/Linux on my personal systems I do work with Linux systems more often than any other single unix like OS except Mac OS X. Most of my client work is on Linux and I rarely ever suggest to a client that they should deploy anything other than linux for their applications. 

I know linux systems very well and over the past, decade and a half,  have come to appreciate them in the larger scheme of unix (like) systems. It's because of this that I support and contribute to projects like LinuxMint.  I wouldn't to be doing what I do today without linux in the picture. I just wish the GNUbs out there realized that diversity is good and portability is important. 


  1. I cannot agrees with your post more than I already do... you perfectly exposed all of my frustrations.

    I too come from freebsd, and nobody else seems to understand. Great write-up!

  2. Thanks for echoing my sentiments on this issue.

    1. Thank you for 1) letting me know I'm not a lone and 2) sharing my little corner of the inter-tubes with your audience.

  3. While I agree that IP tables suck, they are also used by that other popular operating system. And they were actually invented in UNIX land, long before Linux or Windows. And yes, the entire GNU/Linux software stack looks like it was written by thousands of different people, decades apart, with many attempts at improve old software. But I still think it's better than that other one. ;)

    1. I'm aware of a (presumably defunct) project to port Netfilter/IPTables to Solaris (though FSM knows why anyone would want to) but I do not know of any other OS using it.

      If by "that other one" you mean anything coming out of Redmond, we're in complete agreement to the point that it needn't even been stated. If you mean things coming from Cupertino, I'd say yes and no depending on a number of factors. If you mean any of the *BSDs then you're obviously crazy and should have the beard on your avatar cropped by at least 30% ;-)

  4. You bring up a lot of valid points, but this is really no fault of the disunity between *NIX systems. If you don't like something, fork it and make it your own — no one's forcing you to use GNU. It is nice that all GNU software has a standard for behaviour, not unlike the unix software that predated it. Yes, it's a tragedy that Ubuntu 12.04 removed the traceroute utility from their *desktop* install CD, but it's not the end of the world because you have to apt-get or yum install something you expected to be there.

    Otherwise I can agree with what's here entirely, with the notable exception of everything in your very sparse Bloat section:

    > Beyond the utilities from GNU, most linux systems
    > come with far more turned on than I want or think
    > is reasonable. In may cases just installing something
    > (like a web server) results in the that same something
    > starting automatically when you reboot the system.

    People who are installing webserver software would normally need it to start with the rest of the system. That's one of the points of a server. On the theme of different configurations in different directories, the various distros have various ways of dealing with bringing up services like httpd, and how to enable them for startup, and most webdevelopers don't want to have to fiddle with that. It also prevents silly things like launching apache from /etc/rc.local.

    > And come on colorls; really?!?!?

    Fact: Terminals made in the past 25 years support at least 16 colours. USE THEM. It makes little sense to let any system, terminal, graphical or otherwise, be forced to have the user pour over an 80x24 (or more often larger, since monitors have higher resolutions) splattering of characters for a few key words. Can you honestly tell me that you'd rather turn syntax highlighting off? It's not even a question of bloat! Grep with colour support is a whopping 156k binary.

    1. @jkltechinc:
      > People who are installing webserver software would
      > normally need it to start with the rest of the system.

      Yes, but here come my unix sensibilities, again: There are myriad reasons that (server) software may be installed and not all of them are related to the desire or need to have said software start automatically (on the next boot up of the machine). Further even if the user does want it started on the next boot up, there is a great set of circumstances where it's better to let the user configure exactly what they want/need before a service is started.

      re: Terminals and colors:
      Yes, many terminals support at least 16 colors. The way I use them should be up to me. I'm not saying that there shouldn't be colors in the terminal. I am saying that when I type 'ls' I expect to see a monochromatic response. I don't want to have to take into account the color scheme of grep and ls when I'm planning what colors I'll use in the terminal profile of the various machines I support.

      Syntax highlighting is a completely different issue, again I have no problem with colors in the terminal but they should be _my_ colors when/how _I_ decide that I want them. Take git, you can configure it to use colors (which I often do) but it's not on by default. Hell even if it were on by default, at least it's not a core utility of the operating system. RVM for instance has colors turned on by default, I don't even think you can turn them off without hacking the bash (shudder) code. That's ok, because you know that going in. If you don't like it, don't install it, or hack your way around it. I should not have to hack my way around core system utilities.

      Since you brought it up, though, I do a fair bit of my scripting and programming without syntax highlighting as much of it is done on FreeBSD systems where I do not bother to install vim, elvis, vile, emacs, etc. This leaves me with ed, (n)ex, (n)vi or a clever shell based do-while loop for text editing in many cases. On my PCBSD and OS X machines, I frequently use syntax highlighting in both emacs and vim.

    2. Colour vs not really is personal preference. I love lots of colours, but perhaps it's because I'm synesthetic, my mind just works so much better with colours matching categories. I see patterns faster, and spend less time searching the screen. I can understand not wanting it in embedded systems where every bit counts, but for regular desktops and servers, the "bloat" is tiny, and it's really easy to turn off the christmas tree lights.

  5. Former long term gentoo, and now long term Arch Linux user, so maybe I'm missing some of this. Yes vim != vi, and on arch I've aliased vim to vi (because I like it). I really can't do the debibuntu or centofedorpmis... Most of the distro's just don't work for me, largely because I like bleeding edge and having header files. I keep promising myself that some day I will seriously play with freebsd, but for now I've been happy with Arch.

  6. I understand the frustration for someone with experience when a total newcomer to Linux explain how perfect it is, because even if i only started in 2005, i stumble accross that every now and then. I still prefer Linux, although my adventures in BSD lands were too short for me to claim any real knowledge about them. However, i'm find something a bit confusing in your post, you claim that as Randal, you need a system that "just work", but don't really point any issue with that in Linux distributions, instead, your complains are more about support of unix standards, while i can understand such complain, it has very few to do with a system that work out of the box for users. Ubuntu (that i use) is not targetting unix hackers, but they can use it if they are willing to install the few missing tools. The real target of most Linux Distros those days are non-geek users, and last time i checked, installing a FREEBSD system was still way too complex for such user.

    (and that's not saying BSD system lack any merit, obviously)

  7. Awesome post. Succinctly written - I find that I have many of the very same frustrations with GNU/Linux as you do. I've worked in the web hosting industry doing support, development and systems engineering. After a while you learn the 'gotchas' of all of the various distributions but it really takes a toll when you're supporting various systems(hundreds or thousands). There is no unity between the operating systems and when a customer requests *BSD it's really a breath of fresh air. Keep on writing! You've got the perfect mixture of humor and informed writing that makes it a joy to read.

  8. I've used primarily FreeBSD for my personal usage, both desktop and server, for the last 10 years, and this post captures many of the things that I find irritating about using GNU or Linux in general.

    As you said, there are many things that are nice about Linux on a technical level, although the way it is put together is so annoying to use, among other complaints that you mostly addressed.

    As for GNU, I will say that I have a lot of respect for some of their applications, especially Emacs, even though I'm an XEmacs user.

    That said, I'm grateful that I am able to work with mostly FreeBSD systems at work, I never expected that, seriously.


  9. I agree with your opinion as if you read my mind.

    Beside everything, for me the most important thing is the fact that BSD is (or feel) more free than GPL. But that's just my opinion.

    BTW I like your blog keep it up.

    1. It depends on your perspective.

      I think of the BSD (style) license(s) as more free because I, as the developer, want to make sure that other developers can do what they want with my code.

      RMS, GNU and the FSF think of the GPL as more free (as in speech) because it specifically prevents the developer from doing whatever he wants with the code _unless_ he's going to contribute it back to the community. This 'protects' the freedoms of the users in that they do not have to worry about software not being supported or restricted to features only of interest to the original or derivative developer.

      I appreciate the position of GPL advocates and I'm glade that it exists. I just choose to use a different license and wish that more projects did as well. Likewise I wish that I didn't constantly get crap for _not_ using the GPL.

      Thanks for your kind words