OpenWRT, rsync, and linux love

I use an rsync / hard link backup system of my own design (but similar in concept to this).  I have it providing 180 days of backups for numerous production machines spread around the internet, along with more permanent external backups provided by spideroak (referral link, but we both get free stuff if you sign up).  My internal backup system serves as my hot backups, so I want it available 24×7 via a remote file mount (sshfs) should I need it.

The machine I was running it on, though, was WAY overpowered and idles at around 70 watts – this was having a noticeable effect on our electricity bill.  So I put openwrt on my asus wl500gp v2 and have that now doing my backups. It’s silent, fanless, and combined with a good external USB drive has as much storage as you can afford. It also idles at 5 watts total, device and USB drive combined!

Notes:

  • The external drive is formatted for small files and inodes and a higher inode / block ratio. I just did “mkfs.ext -t small /dev/partition” from a full linux machine for the format – these options are a better fit the usage on an rsync / hard link backup system.
  • I installed the openwrt image with the 2.4 kernel because it seems it has better hardware support for this device – it works great.
  • I disabled the wlan and lan, leaving only the wan enabled with a static IP. I port forwarded an external port from my verizon router to allow ssh access from anywhere.
  • I had to install openssh via opkg because the dropbear ssh client doesn’t support outgoing key auth, or if it does it doesn’t support openssh-style keys.
  • I switched the default shell from ash to bash – just too many minor differences for me.
  • rsync is available via opkg. Install it.
  • cron is provided by busybox and has some minor differences in crontab syntax, I could not get @reboot jobs working.
  • USB storage is fairly easy to set up. I found, however, that the external device partitions were recognized at “/dev/discs/disc0/part1” instead of the more traditional “/dev/sda1” locations. No biggie, just odd.  You should read and implement the “start on boot” section.
  • You can see syslog output via the command “logread”
  • I needed to slow down the automount process via a “sleep” command  to allow the drive to spin up before mounting. Details here. Once I put in that delay, automount worked great.
  • “find” provided by busybox is way limited compared to gnu find, and I can’t seem to locate gnu find in the otherwise complete openwrt repos. Busybox find can’t search based on modification times nor link counts – both key to how I implemented my backup system. I reap backup directories via their modification times to expire old backups. I installed ruby and ruby-core (which contains the ruby stdlib that provides file / directory classes) and wrote my own little timed reaper. Source is below.

It’s working great so far – quiet, low-power and fast enough for me.

Stupidly simple timed directory reaper written in ruby

# expire_directories.rb. My backup directory names all look like "back-2012-01-27-04:44:05", 
# hence the regex along with the date check.
require 'find'
require 'fileutils'

days = ARGV[0]

Find.find('./') do |path|
  if FileTest.directory?(path)
    if path.scan('/').size == 1
      if path.match(/back/) && (File.stat(path).mtime < (Time.now - (60 * 60 * 24 * days.to_i)))
        puts "Removing: #{path}"
        FileUtils.rm_rf(path)
      end
    else
      Find.prune
    end
    #puts path
  end
end

invoked thusly:

cd /some/directory/that/contains/your/backup_directories && ruby ~/bin/expire_directories.rb 60

so pass it the number of days. Be sure you're in the proper directory before running this, it's doing an "rm -rf".

Kubuntu 11.10 on the Acer Aspire Timelinex AS4830tg-6450

Synopsis: The info below should tell you most of what you need to get kubuntu 11.10 working on the Acer Aspire Timelinex AS4830-tg 6450, which currently has an MSRP of $729 USD but can be found cheaper through some retailers. It’s working great! I don’t see why they wouldn’t work for a normal ubuntu 11.10 install.  These instructions should be valid for similarly kitted Acer Timelinex laptops too, like the AS3830TG-6424, the AS5830TG-6402 and other models with optimus switchable graphics.

Fear not! It is easier to get this machine working than the length of my instructions would imply.

Details

I’ve gotten Kubuntu 11.10 working with pretty much everything that I care about, including the full power-saving capabilities of the hardware. I get more than 7 hours on the battery. Notes:

  • Download and write kubuntu 11.10 to a usb stick, instructions elsewhere. I’m using the 64bit flavor and it’s wonderful.
  • HDD: Make some room on the hard drive for your linux install by shrinking the largest windows partition. I did this through windows, but whatever makes you happy. Leave the space unallocated, you’ll partition it through the kubuntu install process.
  • BIOS updates: Reboot, going into the BIOS by pressing f2 during the POST. Change the boot priority to use the USB device, or enable the “boot menu via f12” option that’s in there.While you’re in the BIOS, switch the graphics to “integrated.” This will power down the nvidia GPU and lead to some pretty major power savings.Yeah, this kinda sucks, but switchable graphics through ironhide is just not there yet. I was able to get it working (for some values of “working”) but it was VERY quirky to the point of being unusable.  Don’t get me wrong, the ironhide/bumblebee folks have done some great work and I’m looking forward to full optimus support in the near future.This BIOS change means if you want to use the nvidia GPU under windows that you’ll need to go into the BIOS during a reboot and change back to “switchable.” That’s a fine compromise as far as I’m concerned (98% of my time is in linux, I only use windows for games), and the fact that Acer makes this a BIOS option is great – my Asus 1215n didn’t have this option so the nvidia GPU was always on.
  • Install! Boot to your USB stick. Your wifi and ethernet cards should be recognized without a problem. You should probably leave the 18gb and 100mb recovery partitions alone so you can factory-reset the machine easily should you decide to sell it later on. I created a 35gb / partition, 5 gb swap and 240gb /home, but go with what you like. After the installation is complete, you have a few more linux-level tweaks to implement.
  • kernel boot options: edit /etc/default/grub, and change GRUB_CMDLINE_LINUX_DEFAULT to be:
    GRUB_CMDLINE_LINUX_DEFAULT=”quiet splash pcie_aspm=force acpi_osi=Linux i915.i915_enable_rc6=1″
    This will light up a bunch of hotkeys and power management features. Update grub after making this change, via “sudo update-grub”. Reboot. Thanks to this page for the boot options.
  • cpufreqd – install it. This will clock down your CPU according to different profiles and lead to major power savings.The default cpufreqd config needs tweaking, as it’s really not optimized well for the full capabilities of the linux kernel/i5 hardware combo. I really, really like the “ondemand” governor as it means a cooler running system that’ll use the full capabilities of your CPU when needed. I suggest changing every “policy=” in /etc/cpufreqd.conf to “ondemand” and every “minfreq=” and “maxfreq=” setting to 0% and 100% respectively.You can tweak cpufreqd.conf endlessly, but these minor changes make a HUGE difference with no noticeable impact on performance. Reboot the cpufreqd daemon after changing the config file, of course.
  • KDE changes:
    Effects:
    Go into “System Settings -> Desktop Effects -> Advanced” and switch the compositing type to “XRender”. This will give you access to a significant number of effects that work pretty well under the intel integrated gpu.Nepomuk: Go into “System Settings -> Desktop Search” and disable Nepomuk (boo!). You may not need to do this, but I’ve found it’s just too large a resource hog for me to keep it enabled. This is probably because I have a huge $HOME directory with a bajillion text files for my development projects along with three very large email accounts. Nepomuk just never seems to stop indexing and I definitely notice the load. Additionally, shutting down Nepomuk means that pulseaudio works better – I’m guessing because pulse isn’t getting CPU/io starved because of nepomuk’s endless indexing.

That’s it! You’ll have a blazingly fast linux laptop that’ll run over 7 hours on the battery and that weighs less than 5 lbs. I bought mine for $729 at newegg.com – this is a great value for a laptop that just oozes quality. If / when I start to feel performance constrained I plan on getting an intel 320 series SSD drive to replace the 5400rpm western digital it comes with.

What’s working

  • Most hotkeys
  • Powersaving: > 7 hours(!)
  • Sound, including speakers (some folks reported that as a problem). I have had sound lockups that’re fixed by just suspending / awakening the laptop – but since I disabled nepomuk they’ve mostly gone away.
  • Card reader
  • DVD drive
  • Networking – wifi and ethernet
  • Suspend
  • Multitouch on the pointing device.
  • Fan speed – others reported that as a problem, I’ve not seen it.
  • Dual boot
  • webcam (trying opening the /dev/video0 capture device via vlc)

What isn’t

  • Widi
  • Nvidia GPU acceleration under linux.
  • The windows key (probably just need to map it to something)
  • Hibernate – but who cares with working suspend and a pretty fast boot time?

What’s not been tested

  • USB power-off charging
  • Microphone – internal or external
  • HDMI out (probably won’t work, if it’s like other optimus devices)
  • VGA out (probably will work, if it’s like other optimus laptops)

** UPDATE **

  • VGA out works perfectly under the intel video drivers.

Figuring out what’s behind a listening process without a program / pid.

Say you’re being a good sysadmin and you’re checking out listeners on your machines:

root@deathstar:~# netstat -pant | grep LISTEN
tcp        0      0 127.0.0.1:873           0.0.0.0:*               LISTEN      3947/rsync      
tcp        0      0 0.0.0.0:8649            0.0.0.0:*               LISTEN      3826/gmond      
tcp        0      0 192.168.10.122:9102     0.0.0.0:*               LISTEN      4167/bacula-fd  
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      3229/portmap    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      3468/sshd       
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      3930/master     
tcp        0      0 0.0.0.0:44572           0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:34271           0.0.0.0:*               LISTEN      3247/rpc.statd  
tcp6       0      0 :::22                   :::*                    LISTEN      3468/sshd       

lolwut is listening on 44572 ?

 lsof -i -n -P | grep 44572

returns nothing! IT MUST BE A ROOTKIT!!eleventy!!!

Wait – maybe portmapper has assigned it to a kernel-level server?

root@deathstar:~# pmap_dump
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  44915  status
    100024    1   tcp  34271  status
    100021    1   tcp  44572  nlockmgr
    100021    3   tcp  44572  nlockmgr
    100021    4   tcp  44572  nlockmgr

Whew- it’s nlockmgr, part of nfs file locking.

So the moral of the story – just because netstat can’t determine a program or pid doesn’t mean you’re in trouble. Check portmapper’s assignments via pmap_dump first, as it might be innocuous.

Apparently netstat can be made aware of portmapper assignments, but it doesn’t appear it is aware of kernel level services – or – it just doesn’t work for me.

Bye bye ubuntu: hello debian!

So I have been increasingly annoyed with the direction ubuntu has been taking recently: a vitriolic take here.

In addition to some odd fiscal choices while trying to figure out how to pay their bills, they are going whole-hog into their own window manager that looks completely disinteresting to me: Unity. I mean, the idea of a fully accelerated desktop makes sense but I don’t trust ubuntu to build it. My experience under lucid and then maverick with Ubuntu One was less than stellar: it utterly failed. Their own software in their own environment. Busted. The window manager and ubuntu one issues are just one small part: generally, it seems like I derive very little from ubuntu but annoyance.

An example: I upgraded from lucid to maverick and flash performance (which was never stellar anyway) TANKED. A page with a flash embed would take 30 seconds at least to render, with or without a plugin crash. Unfortunately, as a web developer that works with video a lot, performant flash is pretty important. I have no idea why it broke during the lucid – > maverick upgrade: bad QA is the only reason I can come up with.

So I’m back on debian stable for my laptops and desktops and I couldn’t be happier. Everything feels much faster (it could be the switch from ecryptfs to dm-crypt but I expect there is more to it, I didn’t feel io bound), everything works and I have rock solid kde terminals. My ideals just jibe better with debian, it feels like I am home again.

Deploying Debian Squeeze on slicehost by upgrading from lenny

Don’t know why, but debian squeeze is not yet officially supported on my VPS vendor of choice – slicehost.com. This blog post got me started down a very easy path – upgrading a lenny slice to squeeze.

  1. Get a lenny slice. Log in.
  2. apt-get update && apt-get upgrade to start clean.
  3. apt-get install locales followed by dpkg-reconfigure locales to fix that annoying perl locale warning (optional, but SRSLY – why not?)
  4. Make sure your slice is using the latest slicehost kernel in the control panel.
  5. Edit /etc/apt/sources.list, replacing “lenny” with “squeeze”. The debian-volatile repo has been deprecated, so replace all references to it with “deb http://ftp.debian.org/debian squeeze-updates main contrib non-free
  6. apt-get update && apt-get dist-upgrade
  7. Watch lines scroll by, answer whatever questions you get asked.
  8. Reboot.
  9. ???
  10. Profit.

I think it took me longer to type those instructions than it did to complete the entire process.

On a side note – I’m done with ubuntu LTS on my servers. The crap they did meddling with sysvinit and a bunch of other minor annoyances have convinced me I should get back on debian stable. I’ll probably still use kubuntu for my desktops / laptops (and we’ll probably still stick with ubuntu at Berkman), but it’s debian stable where it’s my choice from now on.

How to extract uniq IPs from apache via grep, cut, and uniq

Say you’d like to find out the IP addresses of lines in your apache access.log (or any log file with a similar format, really) that contain “Googlebot”:

grep 'Googlebot' access.log | cut -d' ' -f1 | sort | uniq

which finds the lines via grep, uses cut to extract the first field (space delimited), sorts the IP addresses and then uniqifies them.

Dirt simple, stupidly powerful.