Sunday, July 27, 2014

Sun NICS and pfSense 2.1.x (FreeBSD 8.x)

Opening Ramble

I’m doing more and more networking these days. I’ve got some Juniper hardware on order but I’m also futzing around with pfSense running on some old thin clients (builtin crypto hardware FTW!!). Because I don’t have copious free time, I ordered some prebuilt pfSense boxes from ebay. $70 and 4 days later I had a marginally well built box more than suitable for replacing any OpenWRT or DD-WRT box in my place, I have many.

Why get rid of *WRT?

First, I’m not getting rid of all of my *WRT boxes. I’m simply repurposing them in ways that make them less critical to my infrastructure. That said I’m doing so for a number of reasons, any of which would be sufficient grounds IMNSHO for dumping them altogether. 

  1. iptables — I’ve been managing firewalls since 1998 and my blood pressure rises every time I have to deal with that POC masquerading as a firewall solution.
  2. community & support — pfSense is actively maintained but I don’t actually care about that because I know how it’s built. I could build my own starting with NanoBSD. DD-WRT has packages out there for supported routers that haven’t been updated i many years. OpenWRT went through several bouts of inactivity and rapid development. I could learn the tool chain for getting the WRTs built and deployed or I could just use a system I already know and support myself if it ever comes to that
  3. hardware — The *WRTs run on a wide variety of cheap hardware that’s readily available.  Much of that hardware doesn’t have builtin crypto acceleration and isn’t very expandable (if at all). 
  4. commitment & knowledge — The pfSense guys own their project and it’s issues. While they work to keep things in line with FreeBSD they don’t abdicate responsibility for what they ship. They take it upon themselves to patch in new features, retrofit, and push changes upstream.

This really isn’t a comparison of one project versus another. There are places where one may be appropriate and not another. I’m not bashing the *WRTs. I just prefer to have a little linux running on my network as possible. D’oh!!! did I type that out loud?

The arrival

The boxes I’d ordered came with quad-port PCI Sun NICS (hme[0-3]) for a total of 5 ports including the builtin Ethernet port (vr0).  They were pre-installed with pfSense 2.1.4 configured for VGI/DVI output, vr0 set as WAN and hme0 set to LAN. 

I don’t know why it is that people buy used computing devices and use them without doing a full reinstallation to make sure that they have a clean base to start with. When I’d done a new installation I noticed that my Sun NICS were behaving oddly. They’d start up and be “connected” then after just a few seconds they’d show “unconnected”. The link light would intermittently show solid green light then go dark. Turns out this is a known issue and has a pretty simple fix.

If you can get to a shell on the device you simply need to set the media and mediaops flags like so: 

ifconfig hme0 media 100baseTX mediaopt full-duplex

Do this for each interface that’s flapping and you should be good.

Status…so far

So far, things are going smoothly. I’m not seeing any issues that were typical with the *WRT boxes I had. It’s only been two days so who knows? I may do an update after I’ve had time to gather some meaningful data. 

side notes:

My boxes are NeoWare CA22/E140. They include the Via C7 with PadLock Security Engine. These things tend to get warm. That’s the case even when run for their intended purpose.  

Wednesday, January 29, 2014

How to make a (real) network bridge with a Mac (mini) and OS X (Mavricks)

This is a quick post about setting up a simple layer 2 network bridge using a Mac (in this case a 2011 Mac Mini) running OS X (Mavricks). My goal was to setup the Mac, henceforth referred to as the mini, as an access point for my main network. I found no resources on line that covered doing this so I decided to write it up for those are interested.

What we’re not doing

I found many posts online that covered setting up “Internet Sharing” with a Mac. Some of those referred to this as ‘bridging’ which it is not. At least, in it’s simplest form it is little more than Network Address Translation (NAT) with packets being forwarded from one interface to another.  The problem with that approach is that only IP Traffic is passed and even that is adulterated such that clients on one side (the inside) of the link are not directly addressable by hosts on the other side (the outside) of the link.

It’s great for letting multiple machines browse the web but not good for having multiple machines  talk to each other whether they’re plugged in at the switch or connecting over Wifi and pulling from a common DHCP pool of addresses.

Configure Primary Interface

Primary is conceptual here. I’m taking about the interface that I’ll use to configure the host from over the network. The IP address for this interface will be the (main) address for this host. For me the primary interface is the ethernet port on the back of the Mini. Yours could be any of USB, BlueTooth, USB, FireWire, etc.

We want to make sure that the interface is properly setup on the network. For you that may mean DHCP configuration, manually IP-ing it,  or something else. I’m not going to spend time telling you how to do this. You’ll know it’s working when you can ping something external to the box (preferably external to the network).

In my case I set a static lease in my DHCP server so that every time the mac address for the Mini shows up it gets the same IP. The interface happens to be en0.

Reboot and make sure it works on start up.

Turn on Internet Sharing

Yes, I know what I said before. We’re using this to make sure that we can actually connect to the WIFI access point and get packets out on the network.

Go to System Preferences -> Sharing

On the left-hand side under Service click on Internet Sharing but make sure you do not click the ‘on’ checkbox next to it.

Share your connection from: should be set to Ethernet (at least for me).
To computers using: should have (only) Wi-Fi selected (the checkbox to the left). 

Click on  Wi-Fi Options and setup the SSID, password, and channel you plan to use in the end. You can use test values if you want, just remember to set them appropriately before you’re done with the final step.

When you’re sure everything is setup as you desire, click the checkbox next to Internet Sharing to enable it and click OK or Start if prompted by a subsequent dialog box to (re)start Internet Sharing.

Now configure a wireless client to connect to your Mac’s wireless network. Try to ping something to make sure it works.

Reboot and make sure it works on startup.

Bridge the interfaces

Internet Sharing is going to interfere with binding the two interfaces together into a bridge.  So the first thing we have to do is open a terminal window and type:

sudo launchctl unload -w\

That disables Internet Sharing. The actual command to create the bridge is remarkably simple. At the command line type in: 

sudo ifconfig bridge create

The output of this command depends on what other bridge devices may exist already.  If you get something like bridgeX where X is a number then that is your new bridge device and  you now only have to add interfaces to it:

sudo ifconfig bridge0 addm en0 addm en1 up

For me en0 is my ethernet port, en1 is the Wi-Fi device, and bridge0 is the name of the device returned by the previous command. Your device names/numbers may be different. Use ifconfig to help figure it out. 

Once you’re done with the above you need to re-enable Internet Sharing: 

sudo launchctl load -w\

Connect your devices and test to make sure that you can get a proper LAN IP address.

Making it last

You’ll want to run the command line steps every time the system starts. I do this with a cronjob for the root user.  I store it in ~root/bin/ (/var/root/bin) and invoke it with the line:

@reboot /var/root/bin/

The script contains:

#!/bin/sh -x 
#Purpose:   To create a layer 2 bridge between two interfaces allowing Ethernet
#           frames to pass between them. 

if1=${1:="en0"}  # If no primary interface is passed assume en0
if2=${2:="en1"}  # if no secondary interface is passed assume en1

ifconfig >> /tmp/ifconfig.out
echo "#############################################################################"
    #Helper funciton to insert "sudo" if succeeding command is invoked a non-root user
    case ${UID} in
        0) echo "" ;;
        *) echo "sudo " ;;

$(runas_root) launchctl unload -w /System/Library/LaunchDaemons/
bridge_if=$( $(runas_root) ifconfig bridge create)
$(runas_root) ifconfig ${bridge_if} addm ${if1} addm ${if2} up
$(runas_root) launchctl load -w /System/Library/LaunchDaemons/
echo "#############################################################################"
ifconfig >> /tmp/ifconfig.out

Reboot and make sure it all works on start up

Happy hacking...

Friday, March 22, 2013

Seriously forked up

[disclaimer] I'm an opinionated prick. I try to be pretty up-front about this fact. The site is "Not A (muther-fucking) Discussion dot Com" for a reason. [/disclaimer].

I should probably leave well enough alone. My "thing" is bitching and moaning about technology and trying, where I can, to make things a little better here and there. This post has absolutely nothing to do with that.

If you haven't heard about 'donglegate' from PyCon '13, check out the list of relevant links +wesley chun put together. I happened to be in the room when the comments were made. Luckily I didn't hear them. Also, over the course of the conference I met both +Adria Richards, who I genuinely like, and Mr-Hank, with whom I had a pleasant exchange.

People seem to want to pick sides here. Label one a victim the other a villain and the narrative writes itself. Doesn't matter who you put where, a compelling tale could be derived irrespective of who is placed into what role. I'm heartened by the show of support for Mr-Hank and his family and greatly disturbed by the name calling and threats leveled at Adria.

There are so many issues with what's transpired over the past few days that I don't know where to begin. One part of the problem is that we, in the US, have cultivated both a culture of victimhood and blind privilege. In the end we all lost. This is not a new realization, Amanda Blum pointed out as much so I'll try to avoid retreading that ground.

Daemon's advocate

If we accept Ms. Richard's account of the events as gospel then we should probably think critically about the conclusions she draws and the assertions the makes. 

The title of her blog entry is: Forking and Dongle Jokes Don't Belong At Tech Conferences. I'm not sure that's quite right. While at the conference I heard several jokes about sex and sexuality from men and women alike. It's an adult subject and we were (mostly) adults (legally anyway). There was a reference to being "more accommodating" than others from a member of one of the outreach/diversity groups there. A leer and some giggling followed. That particular statement happened to come from a female. Of course it would have been different if it had come from a guy (because of male privilege). We don't have gender parity and anyone who believes that we do is simply deluding themselves, but that's not my point. When you get adults together they tend to make adult jokes. Some are in poor taste. Hell, I sent a tweet that had the portmanteau: geekgasm. Either Mr. Chun or Mr. Martelli could have chosen to be offended and publicly shamed me, but they did not.

Based on what I've read/heard about the situation, Mr-Hank's statements could be best described as insensitive. If we assume that they were instead malicious there is still the question of how adults respond in the face of that which offends. It's terribly easy to be offended these days. How you handle the situation is what matters.

The Jimmy Choos on the other foot

A pretty easy and obvious question (given the above) is: Why didn't she just confront the guy(s) and say "That's not cool. Knock it off."?

I can't speak for Adria. I don't presume to know what she's thinking or what her motives are. I can, however, tell you from experience that many people of color are sensitized to being seen as "complaining victims". Being a "black female" it may well have been the case that direct confrontation was perceived as a less fruitful course of action. Maybe her concerns would have been discounted and shrugged off; maybe she would have been seen as a bitch who can't take a forking joke; maybe she would have gotten an immediate apology and then an eye roll and finger snap behind her back with spiteful resumption of the perceived slight. We don't know. Further we don't know whether she had concerns along those lines because her decision not to confront the offenders was not discussed in her post.

It's easy for an outsider to say, she should have done XY or Z. I can tell you first hand that confronting someone about (perceived) slights or inappropriate behavior is not always as straight forward as you may think. There was a speaker on stage and it was not likely that she could have made her point to them without consequently making a scene. How do you stop assholes from acting like assholes without being an asshole yourself? That's rhetorical and allegorical. I'm neither seeking an answer nor calling anyone involved an asshole.

The real fork wads here

I don't care who you think is right or wrong  between Adria and Mr-Hank. In my opinion they both did things that any one of us could have done and I won't go casting stones at people for being…uhmm… people.

Companies on the other hand are a different matter and both SendGrid and PlayHaven have landed on my sheet list (inside joke between myself, Adria and like 3 other people). 

PlayHaven used slimy PR double speak in their justification for terminating one of their employees. They took the opportunity to highlight that they are "dedicated to gender equality" as if the statements, allegedly made were in someway intended to be necessarily sexist or oppressive. With one stroke they managed to paint their (former) team member with the color of misogyny. Again, it's one thing to be insensitive and another thing entirely to be malicious. Congratulations PlayHaven on being reactive and behaving poorly in the face of adversity. 

SendGrid announced publicly that they terminated Adria. At least PlayHaven left the guys actual name out of their blog post. It would appear that they did as much to distance themselves from her and (hopefully) avoid suffering the ire of those responsible for DDoS them earlier this week. Basically it looks like they've given in to uncivil activism and it's hard to respect them after that.

In closing

This was tragic. There's plenty of blame to go around and it doesn't look like the worst of the storm has yet passed. Some want to protract this out to get to the "real issue". Others (like myself) want it to be over and done with so that we can minimize the adverse affects on the community as a whole. I hope that this one incident doesn't have any lasting affect on conferences of this type but I doubt that I'm fortunate enough for that to be the case. I wish Adria the best and hope that Mr-Hank can put this whole unfortunate mess behind himself.

Saturday, November 24, 2012

i3 my new WM of choice (so far)

I've been a relatively content user of KDE and XFCE for a while. Before that I'd used FVWM and AfterStep pretty extensively in addition to TWM, VUE, and MWM. A few years back (late '07) I had become obsessed with Lua as a programming language when I was trying to do somethings that AWK couldn't handle elegantly and for which other languages (Python, Ruby, Perl, etc) were just too big. Shortly thereafter I came across Awesome, a window manager that's tiny (originally a fork of DWM) and extensible through Lua. I have a friend who's all about his Haskell and is an ardent fan of xmonad. I never understood his zealotry until I'd found Awesome.

Honeymoon is over

Some time later (by mid 2010) I was working mostly from my Macbook Pro with only 4GB of RAM and I'd stopped using Awesome all together. All of my customizations were off in some back up somewhere on one of my file servers and I was focused on being productive with whatever UI constraints came with OS X (mostly).

The 7 year 24 month itch

The efficiency and elegance of a minimalistic tiled interface kept calling to me. I tried various 'tiled window managers' that sit on top of OS X and none of them really clicked for me (no pun intended). Most of them just added bloat to Aqua and didn't really bring to it the simplicity I'd enjoyed with Awesome. For a while (with Snow Leopard), I ran in an X only configuration. I couldn't get all of my Awesome configs to work the way they had been working previously (on FreeBSD) so I abandoned that effort and loaded up KDE. KDE was enough like Aqua that it's differences presented a problem for me. I'd been a long time KDE user, but had most recently used Aqua exclusively and on my Mac(s) in KDE my mind thought of only Aqua key bindings. I spent some time remapping keys and then gave up. Back to Aqua I went.

It's not you it's me

Sick of the poor memory performance I get on my Mac(s) with 8GB of RAM I decided to move most of my daily administration, systems engineering, and web development to another platform. I can do all of those things and more on a FreeBSD box with 4 GB and have no problems whatsoever. It just so happens that I have a 24GB quad core rig laying around begging to be put to use so I decided to use that instead.

With all of those resources I decided to run KDE with every option turned to maximum bloat. The performance was great. Unfortunately, KDE just doesn't seem to be my thing anymore. Just recently I was having a conversation with a former WebKit developer wherein we discussed my affinity for the KDE project. It pained me to realize that KDE and I just want different things.

Hot new thang

One day I'm digging around one of my boxes looking for my old Awesome configs and I decide to checkout DWM because a friend of mine has been tweaking the hell out of it himself for his own use. I came across a post on the Arch Wiki while doing a search for DWM configs and customizations which compared various tiling window managers. Much had changed in the world of tiling WMs since last I'd taken note. It was time to do some exploring. Anything built with Xlib was disqualified which meant DWM (the whole reason I'd come across this wiki page in the first place) was out. Being a big fan of python since 1999 I decided that Qtile must be investigated. First, though, I wanted to see just how extensible this i3 thing was. Obviously if you can't script it you don't have the power of something like Qtile, Ion, Awesome or xmonad. There was already a package for my BSD system so I installed it and off I was to play with a new toy.

Really pretty but a little annoying

I'd be lying if I told you that i3 is just as configurable as any of the other aforementioned window managers. With that said, i3 has a very basic and well thought out design that leverages existing system paradigms and mechanisms (sockets, pipes, etc) to achieve a level of configurability that goes deeper than most text based config systems.

I won't go into all of the details here but suffice it to say that I was very pleased with the system capabilities and immediately set about changing the defaults to suit my needs. By default ALT is used as a META/SUPER key; changed that do the "Windows Key". The docs talk about how they have vi-like key bindings. Unfortunately by default the keys are shifted by one to the right making h:j, j:k, k:l, and so forth. The reason for this was that META+h is used to indicate a horizontal split for a window container. META+v is used for Vertical. I changed it so that the '|' and '-' keys did horizontal and vertical splits respectively which freed up the h key to be used with the rest of the normal vi keys. I wanted an easy to switch between workspaces/tags so I mapped the brackets to switch the views and the braces to move active containers/windows from one view to another. Beyond the key mappings I replaced the status app (i3status) with my own i3status.awk script. I another script called mate.awk to persistently prevent my screensaver from kicking in when I don't want it to but also don't want to fully deactivate it. It's called mate (pronounced mah-teh) because it's similar in function to the program caffeine on OS X.

The full config is available on my fossil server. I tend to avoid using github but enough people expect stuff to be there that I reluctantly push things up from time to time. You may find it there if I get around to it.

In closing

i3 is a much more capable window manager than I'd suspected and suits novices and power users alike. I haven't yet gotten around to checking out Qtile because i3 is pretty nice. Whatever you can't get done with sockets, pipes, and I/O redirection you can achieve by hacking the source which is insanely well documented. My hat is off to the guys of the i3 project.

Tuesday, November 13, 2012

my monster .shrc lately

It's been while since I posted my monster .shrc file. There were a few bug fixes over time and some improvements that I had to make recently so I thought I'd post the latest version. You can find it on my fossil server at Occasionally I remember to push it up to github ( ). The file is now just about 400 ones long.

There are customizations for 3 advanced shells (KSH, ZSH (4.3.9+), MKSH) generics for a standard posix compliant shells (like ash, dash, posh, etc) and even stuff to make BASH more palatable when I'm stuck on a linux box.

I've included it below just for kicks with angry fruit-salad colors so you can get an idea of what's in it. If you actually want to download and make use of it, please do your self a favor and use one of the links above. Copying and pasting from here will most assuredly produce unexpected behavior. Some of the differences from when I posted about it before is that now my_pathadd() can handle paths with spaces in them. This was pretty key to me getting the titanium alias in place. I've compromised on my own box to allow the use of ls, sed, and head.  I am loathe to call external programs from shell scripts as they tend to add process overhead. In this case it's for one section where I need to create an alias that I use for doing mobile development with Appcelerator's Titanium framework. I left it in to illustrate for the masses how I get it done but most people can completely remove that whole subsection beginning with line #232 going through #252. The previous post did have a call to AWK to help resolve the currently employed shell. That has long since been modified to stay 'native'.

The over arching goal here is to give myself a shell that works the way I expect it to no mater the constraints of the environment I'm in. Since I work on many boxes and different operating systems, it's nice to know that I won't need to fiddle with too much just to get my work done. Hopefully others will derive some benefit from it being made public.

The core principles were:

  1. Keep it standard -- There are many rings that could have been done in a more elegant way if I'd been willing to compromise and use advanced shell features. This was rejected and everything, to the best of my knowledge, conforms to the 2008 opengroup/POSIX standard. That means it should work on every posix complaint shell out there.
  2. Don't needlessly call out to external programs -- There are parts of the code where it would have been easier to get something done by calling grep, sed, etc but I chose to keep it, as much as possible, shell code and not rely on external programs to do what the shell (as defined by the published POSIX specifications) can handle internally. For example in my_pathmatch() (beginning at line #56) I use a case statement instead of calling out to grep or sed.  Again, in my_pathadd() I use eval instead of calling sed or awk to create the string lists needed.
  3. Keep it performant -- Despite being 400 lines long it still performs pretty well… ok honestly it does well with zsh, ash, mksh, and just flys with KSH93. Unfortunately bash is dog slow and it's somewhat noticeable on my 5 and 7 year old machines when I do testing with bash. As I regularly avoid bash this isn't an issue for me. If you have enhancements/tweaks please let me know. I'd be happy to incorporate things that fall in line with #1 and #2. 

A few neat tricks:

  1. my_pathadd <PATH_VARIABLE_NAME> PATH1 PATH2 PATH3 -- will first check that the supplied paths exist and that they are not already in the path variable named. Once the checks are done named path variable gets the valid paths added. (e.g. my_pathadd PATH ~/.bin ~/.rvm/bin)
  2. At line 22 the simple (but possibly confusing) use of eval -- This is used to set a variable named for the PID of the currently running shell. This is used by the script to make sure that it doesn't loop over itself by setting $my_<PID> to the string "processed". You can use it in your scripts to (provided you know how to get at it with eval.  An example is at line #18 where we check to make sure the file hasn't been processed by this shell instance. 
  3. my_reload() -- This unsets the $my_<PID> variable and reload the script picking up new settings. 
  4. At line 91 the while loop -- The while loop is there because the for loop (commented out at line #92) ends up mangling the contents of the string-list in some shells. The only consistent way to get proper quoting/escaping of the elements of the list (that I could come up with) was to do a while loop calling shift and pulling ${1}. Not really a 'neat trick' just a workaround that seems to work (so far). 

  1 #Title: N/A
  2 #Author:    G. Clifford Williams
  3 #Purpose:   used as a .shrc or .profile this script initializes interactive
  4 #           shell sessions with the specified configuration determined by
  5 #           Operating System and/or shell implementation. In particular this
  6 #           script is for shells that implement the POSIX
  7 #           This Part
 10 #-----------------------------------------------------------------------------#
 11 #------------------------------------INIT-------------------------------------#
 12 #-----------------------------------------------------------------------------#
 13     #This section checks to see whether the file has been read already. If so
 14     #it exits (using the return command) with a nice message. It works by 
 15     #setting a variable named for the pid of the current shell ($$). If the 
 16     #variable is not empty the file has been processed before. This makes it 
 17     # more suitable to use one file for both the .profile and .shrc
 18 [ "$(eval "echo \${my_${$}}")" = "processed" ] && \
 19         { #echo "already read for my_${$}"
 20             return 1;}
 22 eval "my_${$}=processed" #set marker to indicate we've processed this file
 24 #-----------------------------------------------------------------------------#
 25 #-------------------------------RUN MODE CHECK--------------------------------#
 26 #-----------------------------------------------------------------------------#
 27 case "$-" in
 28     #This section checks for the 'interactive mode' flag in the '$-' variable
 29     #By checking the my_INTERACTIVE variable you can set chunks of code to 
 30     #be conditional on the manner in which the shell was invoked
 31     *i* )
 32             my_INTERACTIVE="yes"
 33             ;;
 34     * )
 35             my_INTERACTIVE="no"
 36             ;;
 37 esac
 40 #-----------------------------------------------------------------------------#
 41 #------------------------------------FUNCTIONS--------------------------------#
 42 #-----------------------------------------------------------------------------#
 43 my_lecho(){
 44     [ -n my_SILENT ] || echo "$(date +%H:%M:%S)|$@"
 45 }
 47 my_pathmatch_cleanup(){
 48     #clean up some sloppy global variables. I should really clean these up
 49     unset mPMatchDel
 50     unset mPMatchVar
 51     unset mPMatchString
 52     #zsh specific clean up
 53     [ "${reset_wordsplit:=no}" = "yes" ] && setopt noshwordsplit
 54 }
 56 my_pathmatch(){
 57     mPMatchVar=${1}        #Variable to check
 58     mPMatchString=${2:?}   #string to look for
 59     mPMatchDel=${3:-":"}   #optional delimiter (use ';' for LUA*)
 61     case ${mPMatchVar} in
 62         ${mPMatchString}|\
 63         ${mPMatchString}${mPMatchDel}*|\
 64         *${mPMatchDel}${mPMatchString}${mPMatchDel}*|\
 65         *${mPMatchDel}${mPMatchString})
 66             echo "rejecting (${mPMatchString}) because it's in there already"
 67             { my_pathmatch_cleanup; return 0 ;}
 68             ;;
 69     esac
 70     my_pathmatch_cleanup #clean up
 71     return 1 #return 1 if we've gotten this far (indicates no match above)
 72 }
 74 my_pathadd(){
 75     #This is a function to add path elements to a path variable. $1 is the 
 76     #name of the variable to append to (note: use PATHNAME, not $PATHNAME)
 77     my_PATHVAR=$1   #take the first parameter as PATHVAR
 78     shift           #remove $1 and shift the other params down
 79     case $my_PATHVAR in
 80         LUA*)
 81             #If we're modifying a Lua Path we set the separator to ';' instead
 82             #of the standard ':'
 83             my_OFS=";"
 84             ;;
 85         *)
 86             my_OFS=":"
 87             ;;
 88     esac
 90     [ -n "${1:?"USAGE: my_pathadd <VARIABLE> <PTH1> [<PTH2>] ..."}" ]
 91     while [ ${#} -gt 0 ] ; do
 92     #for PATH_add in ${@:?"USAGE: my_pathadd <VARIABLE> <PTH1> [<PTH2>]..."};do 
 93         PATH_add=${1}
 94         if [ -d "${PATH_add}" ] ; then #if the path is an existing dir
 95             if eval "[ -n \"\${$my_PATHVAR}\" ]" ; then #if the variable exists
 96                 #call my_pathmatch <VARIABLE> <PATH> <FS>. If we get 0 as a 
 97                 #return code (success) we know the PATH is already in 
 98                 #<VARIABLE>. If we get anything else we add PATH to <VARIABLE>
 99                 eval "my_pathmatch \"\${$my_PATHVAR}\" \"${PATH_add}\" \\${my_OFS}" ||\
100                 eval "$my_PATHVAR=\"\${$my_PATHVAR}${my_OFS}${PATH_add}\""
101             else
102                 #If PATHVAR doesn't already exist we initialize it to the first
103                 #PATH parameter passed ($2)
104                 eval "$my_PATHVAR=\"$PATH_add\""
105             fi
106         fi
107         shift
108     done
109     eval "export $my_PATHVAR"
110     unset my_PATHVAR
111     unset PATH_add
112 }
114 my_cleanpath(){
115     #function to set a very basic PATH
116     PATH=/bin:/usr/bin:/sbin:/usr/sbin
117 }
119 my_reload(){
120     #Quick function to unset my_<PID>; and reload ENV file. This is the way
121     #to refresh your environment since, by default, just sourcing it after
122     #initialization will not work.
123     unset my_${$}
124     #[ -n ${BASH_ENV} ] && . ${BASH_ENV:"$ENV"} || . ${ENV}
125     my_loadfile=${BASH_ENV:-"${ENV}"}
126     [ -n ${my_loadfile} ] && . ${my_loadfile}
127 }
129 my_getshell(){
130     #Biased test to figure out what shell we're running. We check for KSH first
131     #Some shells that are not true KSH (PDKSH|OKSH|MKSH) may lie here. If this
132     #test does not work for your needs please modify it. It would be nice if
133     #you sent your modification along so for the benefit of others.
134     if [ "${KSH_VERSION}X" != "X" ] ; then
135         echo "KSH_VERSION:${KSH_VERSION}"
136     elif [ "${ZSH_VERSION}X" != "X" ] ; then
137         echo "ZSH_VERSION:${ZSH_VERSION}"
138     elif [ "${BASH_VERSION}X" != "X" ] ; then
139         echo "BASH_VERSION:${BASH_VERSION}"
140     elif ([ "${.sh.version}X" = "Version M 1993-12-28 s+X" ]) 2>/dev/null; then
141         #The above is a test for the specific version of KSH that comes with
142         #OS X. Standard versions of KSH can be detected with the $KSH_VERSION}
143         #check at the top of this function. You will need to update the string
144         #if Apple ever gets around to upgrading their included KSH. 
145         #Note 1: The surrounding ()s are used to catch errors from shells that 
146         #complain about 'bad substitution'. 
147         #Note 2: This test is expensive so we only execute it if all else fails
148         echo "KSH_VERSION:${.sh.version}"
149     else
150         echo "UNKNOWN"
151     fi
152 }
154 #-----------------------------------------------------------------------------#
155 #-------------------Universal/Generic Settings--------------------------------#
156 #-----------------------------------------------------------------------------#
158 #------Get our SHELL-----#
159 [ -n "${my_SHELL}" ] || my_SHELL=$(my_getshell)
161 #-----Get our OS-----#
162 my_OS=$(uname)
164 #----Get our username----#
165 my_USERNAME=$(id -un 2> /dev/null || id -u)
167 #------Set EDITOR(s)-----#
168 if { which vim 2> /dev/null  1> /dev/null ;}; then
169     EDITOR=vim
170 else
171     EDITOR=vi
172 fi
176 TERM="xterm-256color"
177 export EDITOR
178 export FCEDIT
179 export HISTEDIT
180 export TERM
181 #------Set PAGER-----#
182 if { which less 2> /dev/null 1> /dev/null;}; then
183     PAGER=less
184 else
185     PAGER=more
186 fi
187 export PAGER
189 #------Useful bits of info-----#
190 my_FULLHOSTNAME=$(hostname)
191 my_HOST=${my_FULLHOSTNAME%%.*}
193 my_NEWLINE="
194 "
195 #The above two lines are a cheap hack to set a litteral new line character
196 #echo and print can't be used in a reliable fashion to do this across 
197 #environments. The above does not work with all shells, but it should not
198 #cause any problems. If you have a better (POSIX compliant) idea on how to
199 #do this, please let me know.
202 case ${my_OS:-unset} in
203     Darwin )
204         #-------OS X Specifics-------#
205         unset PATH
206         unset MANPATH
207         #added for HomeBrew (first priority)
208         my_pathadd PATH /usr/local/bin /usr/local/sbin
209         my_pathadd MANPATH /usr/local/man
211         #added for macports (third priority)
212         my_pathadd PATH /opt/local/bin /opt/local/sbin
213         my_pathadd MANPATH /opt/local/man
215         #regular PATH(s)
216         my_pathadd PATH /usr/bin /bin /usr/sbin /sbin /usr/local/bin
217         my_pathadd PATH /usr/local/sbin /usr/X11/bin
218         my_pathadd MANPATH /usr/man /usr/X11/man /usr/X11/share/man
220         #added for pkgsrc   (second priority)
221         my_pathadd PATH /usr/local/pkg/bin /usr/local/pkg/sbin
222         my_pathadd MANPATH /usr/local/pkg/man
224         #added for node.js  (homebrew install) 
225         my_pathadd NODE_PATH /usr/local/lib/node_modules
227         LC_CTYPE=en_US.UTF-8
228         export LC_CTYPE
229         OFFLINE=1
230         export OFFLINE
232         unset ti_path
233         #Titanium CLI support on OS X
234         for try_path in  \
235             "${HOME}/Library/Application Support/Titanium/mobilesdk/osx" \
236             "/Library/Application Support/Titanium/mobilesdk/osx" ; do
237             if [ ${#ti_path} -eq 0 ] ; then
238                 my_pathadd ti_path "${try_path}"
239             fi
240             unset try_path
241         done
242         if [ -n "${ti_path}" ] ; then
243             alias titanium="echo 'not yet poppin'"
244             unset ti_version
245             unalias titanium
246             ti_left="${ti_path%% *}"
247             ti_right="${ti_path##* }"
248             ti_version=$(ls -t "${ti_path}"|head -1)
249             alias titanium="$(echo ${ti_path}/${ti_version}/ |\
250                                 sed 's/ /\\ /g')"
251             unset ti_left
252             unset ti_right
253         fi
254         ;;
255     FreeBSD )
256         #-------FreeBSD Specifics-------#
257         BLOCKSIZE=K
258         export BLOCKSIZE
259         my_cleanpath
260         my_pathadd PATH /usr/local/bin /usr/local/sbin
261         ;;
262     Linux )
263         #-------Linux Specifics-------#
264         my_cleanpath
265         my_pathadd PATH /usr/local/bin /usr/local/sbin
266         #get rid of that annoying and stupid 'colorls' crap
267         alias ls='/bin/ls' #set it just in case it's not yet set
268         unalias ls #unset it now
269         #alias titanium=$HOME/.titanium/mobilesdk/linux/${ti_version}>/
270         ;;
271     CYGWIN_NT* )
272         #-------CygWin Specifics-------#
273         CYGWIN=tty
274         export CYGWIN
275         zstyle :compinstall filename '/cygdrive/c/.zshrc'
276         ;;
277 esac
279 #----------ALL SHELLS SETTINGS-----------#
280 HISTSIZE=2500
282 case ${my_SHELL:-unset} in
283     ZSH* )
284         #--------Z SHELL--------#
285         my_lecho "initializing ZSH"
286         # Lines configured by zsh-newuser-install
287         HISTFILE=~/.zsh_history
288         SAVEHIST=1000000
289         #don't overwrite history file
290         setopt appendhistory
291         #get rid of dupes in the history file
292         setopt hist_save_no_dups
293         #try good file locking for history file
294         setopt hist_fcntl_lock
295         #don't show dupes in history search
296         setopt hist_find_no_dups
297         #don't store the history/fc command in the history file
298         setopt hist_no_store
299         #write to the history file incrementally
300         setopt share_history
301         #allow short forms of various control structures
302         setopt short_loops
303         #list options on ambiguous completion
304         setopt autolist
305         bindkey -v
306         autoload -Uz compinit
307         compinit
308         # End of lines added by compinstall
309         PS1="[%n@%m:%/>${my_NEWLINE}%# "
310         ENV=${HOME}/.zshrc
311         export ENV
312         SHELL=$(which zsh) #Needed for screen on GNU systems (bleh)
313         export SHELL
314         [[ -s "$HOME/.rvm/scripts/rvm" ]] && . "$HOME/.rvm/scripts/rvm" # Load RVM into a shell session *as a function*
315         [[ -s "$HOME/.pythonbrew/etc/bashrc" ]] && source "$HOME/.pythonbrew/etc/bashrc"
316         ;;
317     BASH* )
318         #--------Bourne Again SHELL--------#
319         my_lecho "initializing BASH"
320         set -o vi #vi mode editing
321         set -b #immediate background job reporting
322         set -B #brace expansion
323         BASH_ENV=${HOME}/.bashrc
324         export BASH_ENV
325         #source the BASH_ENV if it's readable
326         [ -r ${BASH_ENV} ] && . ${BASH_ENV}
327         HISTFILE=${HOME}/.bash_history
328         HISTFILESIZE=100000
329         #Don't put Dupes in history file 
330         export HISTCONTROL=ignoredups
331         #Append to history file (don't overwrite)
332         shopt -s histappend
333         #update LINES & COLS when window size changes
334         shopt -s checkwinsize
335         #look for bash completion files
336         [ -f /etc/bash_completion ] && . /etc/bash_completion
337         [ -f /usr/local/etc/bash_completion ] &&\
338             . /usr/local/etc/bash_completion
339         [ -f /opt/local/etc/bash_completion ] &&\
340             . /opt/local/etc/bash_completion
341         PS1="[\u@\h:\w>\n\$ "
342         #[[ -s "/Users/gcw/.rvm/scripts/rvm" ]] && . "/Users/gcw/.rvm/scripts/rvm" # Load RVM into a shell session *as a function*
343         [[ -s "$HOME/.rvm/scripts/rvm" ]] && . "$HOME/.rvm/scripts/rvm" # Load RVM into a shell session *as a function*
344         [[ -s "$HOME/.pythonbrew/etc/bashrc" ]] && source "$HOME/.pythonbrew/etc/bashrc"
345         ;;
346     *MIRBSD\ KSH*)
347         set -o vi #vi mode
348         set -o vi-tabcomplete #tab completion 
349         ENV=${HOME}/.mkshrc
350         HISTFILE=${HOME}/.mksh_history
351         HISTFILESIZE=100000
352         PS1='$(whoami)@$(hostname -s):$(pwd)> '
353         case $(id -u) in
354             0 ) PS1="${PS1}${my_NEWLINE}:# ";;
355             * ) PS1="${PS1}${my_NEWLINE}:$ ";;
356         esac
357         ;;
358     KSH* )
359         #--------Korn SHELL--------#
360         my_lecho "initializing KSH (or something pretending to be it)"
361         set -o vi #vi mode 
362         set -o viraw #for real KSH
363         set -o bgnice #nice background processes
364         set -b #immediate background job reporting
365         ENV=${HOME}/.kshrc
366         export ENV
367         HISTFILE=${HOME}/.ksh_history
368         HISTFILESIZE=100000
369         PS1='$(whoami)@$(hostname -s):$(pwd)> '
370         case $(id -u) in
371             0 ) PS1="${PS1}${my_NEWLINE}# ";;
372             * ) PS1="${PS1}${my_NEWLINE}$ ";;
373         esac
374         ;;
375     * )
376         #--------GENERIC SHELL--------#
377         my_lecho "initializing unknown shell"
378         set -o vi
379         HISTFILE=${HOME}/.sh_history
380         HISTFILESIZE=100000
381         ENV=${HOME}/.shrc
382         export ENV
383         #PS1='$(whoami)@$(hostname -s):$(pwd)>'
384         PS1="$my_USERNAME@$my_HOST: "
385         case $(id -u) in
386             0 ) PS1="${PS1}${my_NEWLINE}# ";;
387             * ) PS1="${PS1}${my_NEWLINE}$ ";;
388         esac
389         ;;
390 esac
392 #-------After all is said and done-------#
393 my_pathadd PATH ~/bin ~/scripts ~/.bin
395 #-------Domain specific RC-------#
396 [ -r ${HOME}/.shrc_${my_DOMAIN} ]  && .  ${HOME}/.shrc_${my_DOMAIN}
397 #-------HOST specific RC-------#
398 [ -r ${HOME}/.shrc_${my_HOST} ]  && .  ${HOME}/.shrc_${my_HOST}
399 #-------LOCAL RC (always run if present)-------#
400 [ -r ${HOME}/.shrc_local ]  && .  ${HOME}/.shrc_local
402 #PATH=$PATH:$HOME/.rvm/bin # Add RVM to PATH for scripting

Thursday, August 30, 2012

The deal with me and linux (or Why I don't use Linux unless someone is paying me)

I take quite a bit of flack for not being a Linux zealot. In fact I'm actually not particularly fond of any linux distributions out there and for some reason this invokes the ire of many a devotee who feels personally affronted by my lack of enthusiasm for using the operating system to which they feel a very deep and personal attachment. Here I endeavor to clarify my position and de-escalate the BS.

NOTE: If I sent you to this links it's probably because you're acting like a GNUb and I don't have time to keep re-typing the responses to the questions/challenges you pose. You can either read and learn or not. In the words of Dres from Black Sheep "...the choice is yourrrrrrrrssss...".

[update: 9/6/2012]: So, today Randal L. Schwartz found my post and shared it on Google+.  I saw more traffic in the past 3 hours than in the past 3 months. A few people have commented on my use of the term GNUb so I wanted to, here, clarify what I mean especially since more eyes are landing here and I genuinely do not endeavor to offend.

First I do not use GNUb to refer to all Linux, GNU/Linux or GNU enthusiasts. I have way too much respect for the likes of Linus, Simon Phipps, Alvaro Lopez (Cherokee Project), Thomas Hatch (Salt) and many others for that to be the case. The term is meant to apply to those who have very little experience with unix systems yet presume to know everything based on their (usually limited) exposure to GNU/Linux. These are the sorts of people that most of the hacker community can't stand but because their ire is pointed in the right direction (Redmond or Cupertino) no one challenges or corrects them.

The term GNUb is a bit callous and dismissive so I understand if you're offended. My hope, though, is that you realize I have no desire to disparage the GNU/Linux community as a whole, only those among you to whom the term truly applies.

Where to begin... 

I've been sitting on this post since early in June 2012 and have finally re-read and tweaked it enough times to think that maybe it's OK to post. Maybe I'll make this a living document and keep updating it over time (not likely but let's say maybe). The problem is that there is entirely too much ground to cover quickly and succinctly. One thing I should probably do is qualify what I'm about to say : 
    I've been doing some form of systems, software and/or security engineering since 1993 (yes I was in high school then). Bearing this in mind I have tremendous respect for people who choose to create software for the rest of us to use, especially those who do so without any expectation of monetary compensation (Free as in beer). Also, I admire those who fight to make sure that the rights of users of software are respected and upheld (Free as in freedom). 

I often use pretty strong language to express my dislike of or for various systems/tools. It is my hope that, despite that fact, you understand that I applaud all of the contributors who act with goodwill and in good faith to improve the landscape of Free and Open Source Software for all of us.

A (byte/8) about me (specifically my history with Unix-like systems)

Short version: I started programming in 1993 while in high school, mostly TI-BASIC (on my TI-81) and ASSEMBLY and BASIC on my 8088 at home. I was into the BBS scene and a buddy of mine had 'hacked' into a BSD/386 system and let me poke around a bit. I was smitten. Then in 1995 while a student at Northwestern University I got a job in the technology center (Vogelbach). The team I was on  was responsible for both servicing the technology needs of various departments across all supported platforms (Windows, Iris, DGUX, HP-UX, Novell Netware, Mac OS 7, and more) and running a helpdesk phone line for faculty and students seeking resolution on various internet service issues. I learned much.

A buddy I worked with, named Julian Koh, had lent me a copy of Slackware linux when I decided that I wanted to run a unix-like system at home. Fastforward a few months and I had about 8 computers at home with 4 ISDN lines and was up and running powered by a combination of Linux (slackware) and FreeBSD (thanks to another coworker, Daniel Johanson).

Since then I've been staunchly entrenched in the Unix world. I've supported just about every type of unix system you can name, at some point. I've been in small companies and large, always doing systems, software, or security engineering on unix (small 'u'). I've managed my own servers (actual hardware that I own) since 1995 and have used them in real world scenarios for various services and applications. One such service has been hosting the longest running US based mirror for LinuxMint since 2007. Giving back to the open source and free software communities is something about which I feel strongly.

The meat of the matter

The bottom line to me is this: using most linux systems is annoying to me. I know... that sounds mean ... it's okay... take a second.

Now, I don't mean the above to be offensive or provocative. It's simply a statement of fact. I don't hate your beloved OS of choice it's just annoying as hell to me and I choose not to use it. I'm not saying "Linux sucks!", just that I choose not to use it on any of my personal systems for the reasons listed below. 

I'm not dissing the kernel

I've stated many times in various posts that I think the linux kernel just awesome. I truly admire the kernel team and all of the work they have done. When I talk about linux systems being annoying to use, very little of that (if any) is the fault of the kernel.

Well there is NetFilter / IPTables

Ok IP Tables is a mind f**k that I wouldn't wish on my worst enemies. I have visions of a future where enough people realize that IP Tables was such a bad idea that it's synonymous with 'tried your best but completely screwed it up' and that get's used in sitcoms and vernacular:
 "Boy you really IP Table'd that one."
 "I know. I wish I were dead. Someone shoot me."
 "Heyyy..bud cheer up. Who knows next time you could totally PF it!"

OS not Distro

I like the cohesiveness of a system that's well put together. OS X, FreeBSD, OpenBSD, Solaris they all have one thing in common they are unix based operating systems (that I happened to run at one point on one of my desktop machines). Redhat, Debian, and Ubuntu are distributions where they take the kernel from one source the utilities from another source and cobble it all together. This isn't a fit of my pedantic nature shining through, there's actually a point here. 

It takes a considerable amount of energy to put together a system and get it working properly, there are many moving parts. If you control the kernel and the user land utilities you can make changes as you see fit. When (as in the case of Linux distributions) you get the kernel from X and the user land from Y (most often GNU) and paste them together you have to juggle pieces you don't control. You can patch things for your system and hope they get adopted upstream or you can fork and take over maintainer ship for the version that ships just for your system but either way it's a crapshoot and a bunch of work.

GNUbs say: ... but FreeBSD, NetBSD, and OpenBSD are all just distributions of BSD...

Yes and no. Yes because, let's face it, the D in BSD is, in fact, for "Distribution". All of the popular open source BSDs are based on on the 4.x BSD releases from the University of California at Berkley. No is where it get's interesting and sometimes too subtle for the GNUbs to grasp. Each of the BSD systems takes responsibility for the evolution of it's own kernel, user land tools, utilities, packaging system, etc. For example OpenBSD forked from the NetBSD project. When it happened they took the kernel and everything, modified it the way they wanted and it has since grown separately from NetBSD. The kernels share a common origin but are not compatible in the way you might think of Ubuntu being compatible with Debian.

Base System & Add on Packages

One thing I think most linux distributions get wrong is the whole idea of a base install or core system. Call it what you will but the idea is that I should know what's installed as part of my operating system versus what's an add on package. This concept appears to be lost on nearly every major linux distro.

To make matters worse, on some distributions it's unclear where certain files will be installed. For example, If I install the apache web server on distro-xyz. It may throw configs under /etc/httpd and actual content under /var/httpd/html. On the same distribution, if I install the lighttpd web server it might decide to install configs under /var/lighttpd/etc and html under /var/lighttpd/html. Stuff like that is pretty annoying and exceedingly commonplace in the GNU/Linux world.

File System Layout

On the BSD systems it's generally /usr/local/; on Solaris and HPUX it might be /opt/ on most linux distros it's /. WTF????

When I install a third party package it should be put someplace where I can easily identify whether a file is part of the operating system or not. On Debian, RHEL, Ubuntu, etc. when a third party application is installed it goes into the root of the system along with core utilities and tools like grep, awk, ed, vi, and such. Some things get thrown into  /var and /etc polluting the system. 

(Not) surprisingly the Linux distributions that take some pretty heavy cues from BSD systems have relatively sane ways to deal with this even if it's not set by default. Arch, Gentoo, Slackware and others fall into this category. Sometimes their solution is to just stay out of the way and let you run pkgsrc alongside their system package manager.

GNU/Linux User Land

Let's not forget that most Linux distributions are GNU/Linux. The GNU shell (BASH) is just annoying to me. I find it's conventions like declare and local to be useless bloat. The same goes for many GNU utilities like netstat which (re)appropriates -p to show process names despite the fact that on HPUX, Solaris, *BSD, and many other unix systems -p is used for protocol. In general the GNU user land utilities are pretty bloated and have odd options when compared to their non-gnu counterparts in both proprietary and free systems. Of course there are exceptions and, in some cases, justifications.

GNU grep is blazingly fast when compared to other implementations. Unfortunately if you were expecting it to respect the bounds of POSIX basic regular expressions (as is required by the standard) you'd be out of luck. Because it's so fast it doesn't need to discriminate between basic and extended regular expressions so be careful. 

Remember that GNU stands for GNU's Not Unix. In some very notable cases the FSF and GNU have fought hard to make sure that open standards are published and implemented in their tools. Often GNU tools were the first (if not only) utilities to actually be fully compliant with said standards (namely OpenGroup/POSIX). Where there has been room to implement outside the bounds of the standards they have in many cases chosen the most unconventional approaches. As someone who at one point supported 6 different flavors of unix at the same time working with the GNU versions of things like sed, awk, bc, make, tar, ifconfig, find, and roff was vexing to no end.

Add to the above, the fact that each distribution has it's own custom blend of GNU utilities plus some other random stuff they found somewhere to either supplement or substitute the parts from GNU.

With Ubuntu 12.04 Canonical has decided that it will no longer include the traceroute utility as part of the operating system. You can add it later if you decide that you want it but it doesn't come as part of the system (WTF?!?!?).  

GNUBs / GNUBes / GNUBies
Now that GNU/Linux is so prominent in the industry there is an issue of collateral damage springing up. That of GNU users being ignorant of things like standards and portability. I'm sick of seeing 'shell scripts' that are bash specific without noting as much. CLI extensions like pythonbrew and rvm while not trivial to implement could have been trivially portable to shells other than bash. It takes very little time to write portable code but now no one thinks about it because they are convinced that everyone is running GNU/Linux tools or should be. The same goes for other things like vi. note to all of the GNUBies out there: VIM != vi && GAWK != awk; and I say that as someone who uses VIM most of the time and loves gawk. 

I sit in IRC channels, browse, and watch forums where users ask UNIX questions and get GNU/Linux answers. In some cases this works out alright for that user sometimes they come back and state that something did not work and no one has any idea why. In either case I think it's bad for the unix community as a whole. 


Beyond the utilities from GNU, most linux systems come with far more turned on than I want or think is reasonable. In may cases just installing something (like a web server) results in the that same something starting automatically when you reboot the system. 

And come on colorls; really?!?!?

I need a system that just works

Randal Schwartz host of FLOSS weekly, international traveler, speaker, and world renowned Perl hacker get's flack from GNUbs and Open Source zealots for using a mac instead of some Linux distribution. In several talks he's mentioned that he needs a system that just works, he doesn't have time to keep tweaking his OS because he actually needs to get work done. I'm pretty much in the same boat. While some linux distributions offer a fare degree of customizability none of them come pre-built with the user land I like, a layout I can stomach, and a packaging system that doesn't make me want to pull my hair out. 

True I could cobble together my own system or endlessly tweak Arch, Gentoo, Funtoo, Slackware, DSL, etc but guess what.. FreeBSD, OpenBSD, DragonflyBSD and Mac OS X already work for me. Ok admittedly with OS X I have to go through the process of installing Homebrew and PKGSRC but that's trivial compared to just preparing to install Funtoo or Gentoo (which I've done a number of times).

Despite all of this

I'm still a fan of Linux; both the kernel and the GNU/Linux distributions. While I have no desire to use GNU/Linux on my personal systems I do work with Linux systems more often than any other single unix like OS except Mac OS X. Most of my client work is on Linux and I rarely ever suggest to a client that they should deploy anything other than linux for their applications. 

I know linux systems very well and over the past, decade and a half,  have come to appreciate them in the larger scheme of unix (like) systems. It's because of this that I support and contribute to projects like LinuxMint.  I wouldn't to be doing what I do today without linux in the picture. I just wish the GNUbs out there realized that diversity is good and portability is important.