New Blog
Posted by Luke Cowell on September 30, 2011 at 12:14 PM
I've moved my blog to http://blog.lukecowell.com because this blog software is getting old and difficult to maintain.
How to fix failing time machine backups due to a corrupt disk image
Posted by Luke Cowell on April 02, 2010 at 05:05 PM
I do time machine backups through a USB disk attached to my Airport Extreme unit. Over time the disk image that I'm backing up to seems to become corrupt. Time Machine requires that if you're backing up to a disk image that it was unmounted cleanly or it pass a fsck before it will mount the image and use if for a backup.
The simplest solution is to start backing up to a new disk image, but this doesn't give you the same access to all your previous backups.
Here are the steps I took to fix the disk image, so I can continue backing up to the same image.
Mount the disk image in read write and do not mount the volume
hdiutil attach -noautofsck -nomount -readwrite location_of_your_image.sparsebundle
When you mount, fsck might be called on the image, but you want to kill that process.
Get the info about your attached disk image here:
hdiutil info
Call this fsck command and put your device number in place of the one I have here.
fsck_hfs -rfy /dev/disk1s2
-r means to rebuild that catalog btree (more options can be found in the fsck_hfs man page.
You may need to run this a couple of times before the image is marked as clean.
Static Pages in Rails
Posted by Luke Cowell on February 11, 2010 at 09:22 AM
I just read an interesting post on how to handle static pages in rails and I had a different approach that I wanted to share. There's lots of different ways to do this and each method has its own merit.
When I say static page, I mean any page that doesn't fit into your other controllers. This might be an about us or a contact page or anything else that might not require a dedicated controller to handle.
In routes.rb, I would set something like this up:
map.static '/static/:id', :controller => "static", :action => "show"
We can grab the name of the page from params[:id] in the controller. Feel free to change the name 'id' some something else if it gives you that wrong feeling to treat something as an id that isn't really an id.
Next, I define a controller called static.
class StaticController < ApplicationController
def show
short_name = params[:id]
begin
render short_name
rescue ActionView::MissingPageTemplate
render :file => "#{RAILS_ROOT}/public/404.html", :status => 404
end
end
end
So, what's happening here? We know we have an params[:id] or the route wouldn't match. We then call a begin block, so that we can rescue the exception. Which, in our case, is if someone passes an invalid filename through or the page hasn't yet been defined under app/views/static/pagename.html.erb. We just handle it as a 404.
What about if we want our static pages to live in a namespace without /static at the beginning ? ie. /about instead of /static/about
Define a regular expression that matches all the pages you want to support.
STATIC_PAGES = %w(about contact fun_facts)
STATIC_REGEXP = %r{#{STATIC_PAGES.join("|")}}
Then, in your routes file use the requirements parameter to match only these pages.
map.static ':id', :controller => "static", :action => "show", :requirements => {:id => STATIC_REGEXP}
This is just another approach and there are lots of ways to do this. I was inspired by Yehuda Katz to write this as he recently wrote a post about giving back to the community. Also, be sure to check out his posts about rails 3.
Fix for inDesign CS3 crashes
Posted by Luke Cowell on October 01, 2009 at 12:17 PM
If inDesign is crashing when you quit it, launch it, when you activate fonts or at any other time, you should consider removing or replacing the following file:
$HOME/Library/Preferences/Adobe\ InDesign/Version\ 5.0/InDesign\ Defaults
It appears to contain font information among other things. You care about this because if you were to have activated a corrupt font, or a font with a duplicate postscript name, your problems may continue even after you resolve the font issue because that font will still be cached in here.
eg.
... @Semibold @MyriadPro-Semibold @Myriad Pro Semibold @Semibold @Myriad Pro Semibold 8@Version 2.007;PS 002.000;Core 1.0.38;makeotf.lib1.7.9032 @Semibold Italic @MyriadPro-SemiboldIt @Myriad Pro Semibold Italic @Semibold Italic ...
iPhone recovery mode vs. DFU mode
Posted by Luke Cowell on September 26, 2009 at 06:46 AM
DFU mode and recovery mode are different. Here's how to get tot either one.
To put your phone in DFU mode:
- Turn your iPhone off
- press and hold the home and sleep/wake button for 10 seconds
- release the sleep wake button and hold the home button for 10 more seconds
To put your phone in recovery mode follow these steps:
- Disconnect the USB cable from the iPhone
- Turn your iPhone off
- Press and hold the home button while connecting the USB cable to your device.
- iTunes should now display a message to indicate you are in recovery mode
Installing rmagick on OS X
Posted by Luke Cowell on June 01, 2009 at 09:59 PM
For some reason the gem tries to build against a PPC library, which on my intel machine, doesn't not work.
sudo su -
ARCHFLAGS='-arch i386' gem install rmagick
Note: you might need to fix the quotes.
Linotype FontExplorer Settings via command line
Posted by Luke Cowell on May 25, 2009 at 08:31 PM
Finding setting keys for Linotype Font Explorer
There's lots more and they're easy to figure out.
defaults read com.linotype.FontExplorerX > a
Now make your setting change in Font Explorer - it works best to quit at this point.
defaults read com.linotype.FontExplorerX > a
diff a b
The output of the diff will tell you what keys changed.
This output from diff:
fontRequestsAutoActivate = 1;
Becomes:
defaults write com.linotype.FontExplorerX fontRequestsAutoActivate 1
A list of keys I've found useful
Here are a list of linotype keys that I've found useful in configuring workstations settings.
If a font from a different path with the same name is imported, remove the old one and re-add
defaults write com.linotype.FontExplorerX sameFont_importingAdjustments 1
defaults write com.linotype.FontExplorerX sameFont_useGlobalImportingAdjustments 1
Remove and re-add a font if imported from the same path
defaults write com.linotype.FontExplorerX samePath_importingAdjustments
defaults write com.linotype.FontExplorerX samePath_useGlobalImportingAdjustments
When a duplicate font is requested deactivate the current font and activate the requested font without warning
defaults write com.linotype.FontExplorerX hideActivationWarning 1
defaults write com.linotype.FontExplorerX hideActivationWarningOption 1
When activating a font activate the entire suitcase
defaults write com.linotype.FontExplorerX alwaysActivateSuitcase 1
Deactivate Fonts when linotype quits
defaults write com.linotype.FontExplorerX deactivateFontsFromSessionOnQuit 1
Autoactivate fonts when possible
defaults write com.linotype.FontExplorerX fontRequestsAutoActivate 1
Contributing to the Community
Posted by Luke Cowell on April 23, 2009 at 11:13 PM
I'm proud to announce I made my first real contribution to the ruby / ROR community. I added a current quiz feature to the ruby quiz site and added an RSS feed to the site for all quizzes and solutions.
Announcement http://www.ruby-forum.com/topic/183984#809279
Ruby Quiz RSS Feed: http://rubyquiz.strd6.com/quizzes.rs
Ruby Quiz Site: http://rubyquiz.strd6.com/
The Lie
Posted by Luke Cowell on April 16, 2009 at 10:23 PM
Blueprint CSS
Posted by Luke Cowell on April 16, 2009 at 10:14 PM
Blueprint CSS does a bunch of different things, but this is what really interests me: “A CSS reset that eliminates the discrepancies across browsers.”
This means that many of those annoying quirks (ie your layout is broken in Firefox or more likely IE) may be a thing of the past.
Hooray!
Well that's Iterating
Posted by Luke Cowell on March 20, 2009 at 03:43 PM
A great page explaining different iterators available in ruby.
Read about it here.
Compromising Window Positions
Posted by Luke Cowell on February 25, 2009 at 09:52 AM
I recently had a reader write in about a problem with their Mail program. Here's what they had to say:
Dear Luke Cowell,
I am a big fan of your blog, where you explain how to make beautiful christmas trees out of incomprehensible code that outputs a hilarious parody version of "The Twelve Days of Christmas" that I don't really get.
Perhaps you can do a blog about why my Mac Mail program doesn't save my window size settings. When I close the program, and re-open it later, the window is small, and the "preview" pane, which I like to have hidden, takes up half the window.
Sincerely,
Murray Inkster
Vancouver, BC
Dear Murray,
Thank you for writing in. Apple mail stores most of its preferences in a file called com.apple.mail.plist. This is located in your home directory, under Library/Preferences. Make a backup of this file.
Quit Mail and open terminal and paste the following lines in: defaults delete com.apple.mail ActiveViewers defaults delete com.apple.mail ActiveSingleViewers
These will delete any information Mail.app has about window position and will hopefully allow you to put your windows in all sorts of compromising positions again.
If that doesn't work you could also try deleting the com.apple.mail.plist file I mentioned above. Be warned that you'll need to set up your mail rules and accounts all over again.
Hope this helps,
Luke Cowell
www.lukecowell.com
Ruby Quiz #187
Posted by Luke Cowell on December 22, 2008 at 11:57 AM
I saw this solution on Ruby Quiz and I thought it was brilliant. I thought I'd explain how the original author came up with this beautiful looking tree to generate some code.
### M E R R Y M A T Z M A S ###
"/|"\
"/wl)e";eval(\
"A | A"\
"A | A"\
\
"InHhL"\
",Z,%w{-"\
",Y&/LH,L*"\
"cCnd,Hh&/d,"\
"YCB/Hh,Y&YHh,"\
"L&xHh,L*v*nHh,*"\
"&ghHh,n&nHh,H*nHh"\
",*q*v*nHh,Hw*qv*Hh}"\
"Iq&n*L,Z,[,n&qK,n&qK,"\
"@TwC,LH/&ng,gLBbL@K,@Th"\
"/**,YC/k*d,Hh/*QdL@K,@FCB"\
"/,cQqq&ng,=/CcL@K,@F&v*,RBb"\
"y,g*mL!@K,@S&x,mQ/LhQqL,dBm=&"\
"ng@K,@S*v*n,dBckL,Q-Hy=&ng@K,@E"\
"&ghH,d&/L,Q-gqCbb&ng@K,@N&n*,/Qng"\
"*L,LH*==&ng@K,@T*n,H/yL,Q-cQHch&ng@"\
"K,@Eq*v*n,&CL,=&=&ng@K,@Tw*qv*,mCnk*y"\
"L,=QHch&ng@,]IDAJJA2)J*Qch,dC,U&UI=/&nH"\
",@\nOn,Hh*,#{nHhL[&]},dQy,CY,MQHzmQL,my,"\
"H/B*,qCv*,gQv*,HC,m*:\n@I&JdCwnHCD2),dC,UjU"\
"I=/&nH,@,,,#{q&n*L[j]}\n@I*ndI=/&nHDD&,ZZ,A)"\
"?,@,,"\
",G@,:"\
",@,,,"\
"Gnd,Q"\
"@)I=/"\
"&nH,@,n*w,v*/L&Cn,CY,RBby!\n@I*nd"\
"I".tr(' ID/VHLYGBqCA&|*UQJ=Z@K,',
'+;(r/tsfAulo1i8e|a.p=", '))
Here's the original code: http://pastie.org/344030.txt
What it outputs is something like this:
On the first day of Matzmas my true love gave to me: A new version of Ruby! -snip- On the twelveth day of Matzmas my true love gave to me: Twelve monkeys patching Eleven ios piping Ten trys a-catching Nine ranges stepping Eight dirs a-globbing Seven ducks a-typing Six marshals dumping Five Ruby gems! Four calling procs Three forked threads Two string gsubs And a new version of Ruby!
Well everything up to the 'eval' is fluf, so we can ignore that. If we paste everything inside the eval without calling the eval into irb, we get:
"1+8+11++8++1;nths = %w{- first second third fourth fifth sixth seventh eighth ninth tenth eleventh twelveth};lines = [ nil, nil, \"Two string gsubs\", \"Three forked threads\", \"Four calling procs\", \"Five Ruby gems!\", \"Six marshals dumping\", \"Seven ducks a-typing\", \"Eight dirs a-globbing\", \"Nine ranges stepping\", \"Ten trys a-catching\", \"Eleven ios piping\", \"Twelve monkeys patching\" ];(1..12).each do |i|;print \"\nOn the #{nths[i]} day of Matzmas my true love gave to me:\n\";i.downto(2) do |j|;print \" #{lines[j]}\n\";end;print((i == 1)? \" A\" : \" And a\");print \" new version of Ruby!\n\";end;"
So we see that the tr function generates the above code and turns it into some ruby code that can be passed to eval. We can also ignore everything up to nths.
tr(' ID/VHLYGBqCA&|*UQJ=Z@K,',
'+;(r/tsfAulo1i8e|a.p=", ')
The code roughly cleans up to this:
nths = %w{- first second third fourth fifth sixth seventh eighth ninth tenth eleventh twelveth};
lines = [ nil, nil, "Two string gsubs", "Three forked threads", "Four calling procs", "Five Ruby gems!", "Six marshals dumping", "Seven ducks a-typing", "Eight dirs a-globbing", "Nine ranges stepping", "Ten trys a-catching", "Eleven ios piping", "Twelve monkeys patching" ];
(1..12).each do |i|;
print "\nOn the \#{nths[i]} day of Matzmas my true love gave to me:\n";
i.downto(2) do |j|;
print " \#{lines[j]}\n";
end;
print((i == 1)? " A" : " And a");
print " new version of Ruby!\n";
end;
So, if we reverse the 'tr' command we can encode similar blocks of code. Here's an example:
"asdasd"
eval("DAJJA0)"\
"J*Qch,dC,UnUI,=/"\
"&nH,@I,cQn,cCBnH,H"\
"C,Hh*,nBmb*/,\#{n}\n@"\
"I*ndI"\
.tr(' ID/VHLYGBqCA&|*UQJ=Z@K,'\
,'+;(r/tsfAulo1i8e|a.p=", '))
Ta-da! It's a lumpier, uglier version of the tree that outputs some pretty unimpressive code.
Powerpoint 2008 Greek Fonts
Posted by Luke Cowell on October 22, 2008 at 02:02 PM
I had a problem with one of my clients powerpoint installation today. They had recently updated to Office 2008 and since then, powerpoint wouldn't display fonts correctly. In particular, arial (regular) wouldn't display correctly, no matter what size, but as soon as I changed it to the bold or italics version, it worked.
The fix was pretty surprising. It turns out that Powerpoint is somehow dependent on the 'symbol' font being present. The solution is so bizarre that I did a number of test to ensure this was the fix. Seems like <cough>bad programming<cough>.
So, if you're having problems with powerpoint 2008 and fonts. Make sure Symbol.dfont is enabled on your system.
MacBookMacBookProSU1.2.pkg - New Software Update for Macbook (Pro) 5.1
Posted by Luke Cowell on October 15, 2008 at 09:43 PM
143MB uncompressed. For those of you who are used to right clicking on a package and choosing 'show package contents' you may have notices that since leopard came around you can't do that any more. To convert the package back from a flatfile package to the previous version do this:
pkgutil —expand /Volumes/MacBook\,\ MacBook\ Pro\ Software\ Update\,\ 1.2/MacBookMacBookProSU1.2.pkg destination
Now you can root around, see the install scripts and view the files that were in the package.
Apple says this about the update: The update improves compatibility with external displays and includes a variety of software fixes. This update is recommended for all users of these systems.
One things I noticed in this update is that there's a lot of files relating to bluetooth being updated. This is interesting because apple only talked about graphics updates. So, if you're getting one of these new macbooks make sure the bluetooth is working correctly.
/usr/libexec/cups/backend/bluetooth
/usr/sbin/blued
/System/Library/Extensions/IOBluetoothFamily.kext
/System/Library/Extensions/IOBluetoothHIDDriver.kext
iCal has unexpectedly quit
Posted by Luke Cowell on October 15, 2008 at 12:57 PM
If you get the message the iCal has unexpectedly quit. Deleting the following from your home directory may help you:
Library/Sync Services
Application Support/Sync Services
Sometimes you'll have an application that will register itself as part of sync services, but it's busted, so it will crash iCal when the sync happens.
Find Empty iCal Calendars
Posted by Luke Cowell on September 27, 2008 at 07:58 AM
I recently tried a piece of software called 'Contactizer'. It didn't do what I wanted it to, so I uninstalled it. During its stay on my computer it created about 10 additional iCal calendars, and put a few events in one or two of them. So, I needed a way to determine if a calendar was empty before I deleted it.
Here's a little applescript that will tell you the number of events in each calendar.
tell application "iCal"
repeat with mycal in every calendar
get name of mycal
get count of every event in mycal
end repeat
end tell
What features should a web site include ?
Posted by Luke Cowell on September 24, 2008 at 07:37 AM
Someone I know recently had their web site redone. Apparently, their site was vulnerable to a security vulnerability and they were told by the original 'developer' (I'll call them design-corp) that it would need a complete rebuild. Hmm, I'd suggest that it's a better idea to patch the vulnerability, rebuilding the entire site sounds a little bit like a cash grab or that they don't know how to fix it. My confidence in design-corp was waning.
This situation got my curiosity and so I thought I'd check out design-corp's site. They actually had an impressive client list. Lots of local, but very high profile clients. They had a extensive portfolio divided into < $10000 sites, > $10000 sites and Ecommerce and Database sites (which I can only guess are the most expense). Design-corp isn't cheap, I'll say that much, so it made me think that it's not necessarily about what you're selling, but how you sell it. That is to invest a lot of time in helping the client understand the value of what they're getting.
So, here's the punch-line, in form of a list of features that Design-corp's web sites can include.
- Web strategy
- Training
- Project management
- Web design/creative
- Programming
- Database development
OK, some of these aren't funny, but a couple are.
Programming - my question is: what web site's don't require some programming ? How far can you go in the business without it ?
Project Management - read: the project was big enough that we had to charge for an additional service. We all do this, don't we ? It's called getting the job done, so you get paid.
Come on, they're suppose to be web experts! Should they list that they know basic computer operations such turning a computer off and on, or that they are experienced with Microsoft Word ? To me that stuff is all par for the course in the web world.
I should be thankful, at least they aren't driving the prices of web development down.
Blackberry portal where art thou ?
Posted by Luke Cowell on September 22, 2008 at 04:19 PM
Most of the people I know, who have anything more than a basic phone, have bought iPhones. Some people have chosen to stick with the 'venerable' Blackberry. Configuring email on blackberry's can be a pain in the ass sometimes. I know there's built in email activation on the blackberry, but that requires you to have access to the unit. Some of the people I support live thousands of kilometers away and in different time zones, so that doesn't always work.
In theory, if there were a web page I could change the settings at, I could avoid lengthy support calls and configure their email myself. For some reason blackberry doesn't want you to know this, but each company that offers blackberries has their own portal on the blackberry site. In fact, the service providers (Rogers, ATT, Verizon) don't seem to really want you to know this either because it's proven to be almost impossible to find this sort of info on their sites.
OK, so here's the trick - use google. Let's say I want to find the Rogers portal, craft a search like this:
rogers site:blackberry.com inurl:brand
Obviously you substitute the company you're looking for with rogers.
site: just tells google to limit the search to blackberry.com.
inurl: tell google to ensure that the word brand is in the url.
Here's a few providers:
http://bis.na.blackberry.com/html?brand=rogers
http://bis.na.blackberry.com/html?brand=telus
http://bis.na.blackberry.com/html?brand=mycingular
http://bis.na.blackberry.com/html?brand=vzw <—Verizon
I just don't understand why this has to be some complicated....
SVN Setup for Rails Apps
Posted by Luke Cowell on September 22, 2008 at 04:08 PM
A lot of machine specific information doesn't need to go into the repository for your rails app. The following is taken from http://tuples.us/rails-guide/ which is largely taken from the ugly rails wiki.
svn remove log/* svn commit -m 'removing all log files from subversion' svn propset svn:ignore "*.log" log/ svn update log/ svn commit -m 'Ignoring all files in /log/ ending in .log' svn remove tmp/* svn commit -m 'removing all tmp files from subversion' svn propset svn:ignore "*" tmp/ svn update tmp/ svn commit -m 'Ignoring all files in /tmp' svn move config/database.yml config/database.yml.example svn commit -m 'make database.yml template' svn propset svn:ignore "database.yml" config/ svn update config/ svn commit -m 'ignoring database.yml'
It may also be a good idea to add:
svn remove db/*.sqlite3 svn propset svn:ignore "*.sqlite3" db/ svn update db svn commit -m ' removing sqlite dbs from svn'
Stop websites from draining your battery
Posted by Luke Cowell on July 02, 2008 at 07:18 AM
Sites that incorporate flash are often draining your battery. You can disable flash from automatically loading using safari stand.
- Install Safari Stand: http://hetima.com/safari/stand-e.html
- Go under the Stand menu
- Choose SafariStand Setting...
- click on advanced
- check load Plug-in manually in the flash section
- restart safari
Unless you're one of those people who browse with your activity monitor open, you're probably not aware that many sites are sucking your battery dry. Many sites incorporate flash, in particular to display annoying advertisements. These little flash tidbits are often poorly written and will make your CPU work really hard, which will drain your battery. Now when your browser encounters flash, you can click the box where the flash would have been to load it.
You can also add exceptions to the site alteration area. For example, you might want to let flash automatically load in if you were at youtube.com.
Installing produCtion based mysql gem on OS X
Posted by Luke Cowell on June 28, 2008 at 07:01 AM
The default mysql database adapter that ships with OS X is not optimized for produCtion use (read: not written in C).
Installing this gem on OS X can be a bit tricky because mysql isn't where the gem installer expects it to be. You attempt to install the gem and the compile fails because it can't find libraries etc. If you've installed mysql via the installer available on mysql's website (vs. ports) these instructions should work for you.
sudo gem install mysql -- --with-mysql-dir=/usr/local/mysql --with-mysql-include=/usr/local/mysql/include/ --with-mysql-lib=/usr/local/mysql/lib/ --with-mysql-config=/usr/local/mysql/bin/mysql_config
I was doing some heavy data crunching and switching to the production gem has definitely decreased cpu usage.
I should also credit this site: http://weblog.imapenguin.com/articles/2006/02/01/mac-os-x-10-4-ruby-mysql-adaptor
I'm not sure if this is where this originated.
10.5.3 hang on startup
Posted by Luke Cowell on May 31, 2008 at 08:56 PM
Symptom: after installing 10.5.3 your computer will not get past the initial grey startup screen.
Verify that directory services is causing the hang by booting into verbose mode. (Apple-V) You should see repeated messages about directory services crashing.
Fix:
-boot into single user mode (Apple-S)
-mount your drive in read-write mode
mount -uw /
-boot into single user mode and remove (delete or rename) this folder:
rm -ri /Library/Preferences/DirectoryServices
When you reboot these files will be recreated with the defaults.
NOTE: this will delete any configuration you make in directory utility.
A simple and tedious fix to your Mail.app rule issues
Posted by Luke Cowell on December 20, 2007 at 08:48 AM
Some people have been having problems with the Mail.app rules in Leopard. Mail filtering rules somehow don't seem to be working the way they should.
A little sleuthing let me to a file in ~/Library/Mail called MessageRules.plist. This file contains all the rules you've created as well as the apple rules (eg. junkmail rules and 'news from apple' rules). What I noticed was that this file also contained very old rules that I had created, but were not visible in the rules list in Mail.app.
My hunch was that the file had become corrupt. So I removed it as well as MessageRules.plist.backup and restarted mail. I made some new rules and it worked! I actually think this issue may have started before I upgraded to leopard, so it's worth a shot in Tiger as well.
In Summary, remove these files:
~/Library/Mail/MessageRules.plist ~/Library/Mail/MessageRules.plist.backup
Wait, that's not tedious! You said this would be tedious! OK, as promised here's the tedious part:
Go and remake all your rules in apple mail again.
Ta-Da!
Create Packages for OSX using snapshots
Posted by Luke Cowell on December 14, 2007 at 09:17 AM
If your an administer for an OS X network, you probably know how much time can be saved by creating a package (.pkg) installation versus manually installing the software on each workstation. Many pieces of software use old or proprietary installers vs. packages, so building a package can take a little patience. That was the wind up, here's the pitch.
Building packages just got a lot easier with the snapshot feature in the newest version of packagemaker. Start the snapshot recording by choosing 'Add Snapshot Package...', run your installation procedure of whatever software you want to make into a package, and when you stop it will show you what files have changed.

How is this magic done ? Spotlight keeps track of which files to reindex by adding hooks to all the low level write functions on the disk. You update or create a file and the OS knows that it needs to be reindexed. There's now a public framework called fsvents that keeps track of all file operations. So, all packagemaker needs to do is hook into that for the duration of the install.
(This is also how timemachine knows which files to back up on your computer without scanning your whole disk)
Check outArs technica for a more detailed explanation about how fseventsd. http://arstechnica.com/reviews/os/mac-os-x-10-5.ars/7
Once the snapshot is generated you have a chance to choose which files you want to include in the snapshot. Remember that files besides the installer will change during the period of the install. For example logs will be updated, but you definitely don't want to include those.

That's it. Much easier than manually collecting those files.
Handling bulk email in Qmail using multiple queues
Posted by Luke Cowell on August 13, 2007 at 10:23 PM
I have a webhosting company, Talesa. One of our clients has a mailing list set up. (This is an opt-in mailing list.) Over time it's grown to have thousands of recipients and they send out messages several times a day. The majority of the list is delivered in about 5 minutes; way to go qmail.
The problem with all of this is that any message sent out while qmail is doing its mailing-list duty takes about 5 minutes to get through. No big deal right ? Maybe not, but why should a mailing list clog up the queue for other messages, when it's not a big deal if the mailing list messages take 20 minutes to go out?
What can I do to solve this ? First, I change /var/qmail/control/concurrencyremote to 80. The list goes out faster, but regular emails are still getting held up. On further investigation, it looks like what's happening is there are a few major 'free' email providers out there who are holding connections open with us. This quickly takes up all our available qmail-remote processes. Once all 80 of your qmail-remote slots are taken up qmail waits for them to timeout.
qmail has no notion of message priority. Even if you mark the message as bulk, qmail will show no bias and will treat all messages equally. So, what do we do ? Set up another server ? Not if we don't need to.
The solution is in making a separate qmail installations. The idea being that each installation will have its own queue and a glut of messages in one queue, won't affect the other.
In FreeBSD I add
QMAIL_PREFIX=/usr/qmail-bulk
to the Makefile. Or if you're doing this by hand, edit the conf-qmail file. I chose /usr/qmail-bulk to differentiate it from the /usr/qmail installation. Doesn't qmail live in /var/qmail ? Good point, but when you're using FreeBSD, the /var you set up when you installed the system can often seem plenty big at the time, but can quickly become inadequate.
Now that you've got your new qmail installation, there's a few things left to do.
- update your tcpserver script to listen on an alternate IP. If you don't have an extra IP available, you could have it listen on an alternate port and have the primary qmail installation forward mail to that using smtproutes.
- update the /var/qmail/rc script. Calling /var/qmail-bulk/rc stop will not only stop your bulk qmail instance, but it will also stop your main one. So, the rc script needs to stop and start both qmail instances.
I manage the mailing lists through ezmlm. Why ? because I'm a masochist. At any rate, if domain.com had a mailing list. It would be handled through lists.domain.com. All I need to do is add lists.domain.com to:
/var/qmail-bulk/control/
locals
rcpthosts
and I need to update the mx records to point to the alternate IP.
Common problems
- when you're updating your tcpserver file. Usually, it's set to listen on all IPs using '0' as the host. You'll need to change that to a specific IP before you'll be able to start your alternate qmail instance.
- make sure that you can contact all the server's IPs internally and externally. For some reason the default netmask (255.0.0.0) meant that I could reach that IP externally, but I couldn't communicate with it from on the server.
- remove the virtualdomains config from the main qmail installation otherwise if your mailing list commands are issued through smtp on the main server the messages will be put in the wrong queue.
- if you're using ezmlm, you may need to recompile it to use the alternate qmail installation.
Other recommendations
- set your concurrencyremote to a nice low number like 20. No point in slowing down the whole email server.
- check to see that things are working as they should by doing tail -f /var/log/mailog | grep status
This should produce output like:
Aug 14 07:15:59 server qmail: 1187100959.346850 status: local 2/20 remote 3/80
Aug 14 07:15:59 server qmail: 1187100959.349790 status: local 1/20 remote 20/20
we can see that the main queue is at 3/80 and the bulk queue is at 20/20. Just where we want it.
Questions and Comments are always welcome.
Luke
How to have quicksilver index your login items
Posted by Luke Cowell on July 17, 2007 at 08:03 AM
I use mozy backup. It’s a great product, but I think development is still a little behind on the mac. Long story short, the application ends up taking about 500MB of resident RAM over time. The mozy status app that lives in your menubar is actually embedded inside the main mozy application.
/Applications/Mozy.app/Contents/Resources/Mozy Status.appThis means that if I kill the process, it’s going to be tedious to relaunch it.
Or is it??
I know that this is called as a startup item and so here’s how to get mozy to index your statup items.
Click on catalog and create a defaults reader under custom in quicksilver. Any defaults reader in quicksilver can parse a defaults file (a plist) and return certain results.
I know that the file I’m looking for is loginwindow.plist, so I enter loginwindow without the .plist as the Bundle ID. Similarly if you wanted to index your sidebar items, the file com.apple.sidebaritems.plist would be entered as com.apple.sidebaritems.
Now I open loginwindow.plist in the property list editor. Many of these files are now binary, so opening them in ‘vi’ is no longer an option.
If you’re following along with me you’ll see the parent item is Root. Skip that.- The next item is AutoLaunchedApplicationDictionary. Add that to the first line in your quicksilver keys. Now press option-enter to start a new line.
- Enter ’*’. This tells quicksilver to parse every item under AutoLaunchedApplicationDictionary.
- Press option-enter again and enter Path. This is the final key and this tell quicksilver that we want to know what is in that key. Because this is a key that represents a path we indicate that the type is a ‘path’. Your settings should look like this:

And if you did everything right, you should have all your login items under contents (after a reload of course).
Safari 3.0 for Mac and PC
Posted by Luke Cowell on June 11, 2007 at 12:59 PM
The new beta of safari is out and its available for mac and PC.
http://www.apple.com/safari/
The new interface is almost identical to the 2.0 interface we became familiar with. There are a few interesting tweaks that aren't immediately obvious.
- You can rearrange tabs.
- You can drag a tab out to put it in its own window.
- Searching now highlights your search results as Firefox has for the last couple of years.
- Spellcheck ? I don't remember it having this functionality before.
- You can resize text entry boxes.
and more...
Why make Safari for Windows ?
Apple does a great job of designing software and hardware that is easy to use. Those of us who use macs know this already, but many windows users don't and live with obtuse key combinations and UIs that are unnecessarily complicated. Releasing Safari for Windows is yet another avenue for a windows user to make the realization that: "Hey, I like what these apple guys are doing".
Apple's market-share is growing, I just hope that Apple doesn't become the next Microsoft.
Get real about startup optimization
Posted by Luke Cowell on May 17, 2007 at 07:08 PM
This will be a useful read for you if you are looking to speed up the time between when you start the login process and when your key startup items are loaded.
There are a plethora of tutorials out there on how to improve your login startup times in OS X. Some suggest adding delays between startup items. Some suggest using an applescript to do your launching.
Time to usefulness
It is possible to measure the amount of time that has passed until all your login items have loaded, but let's get real, many of the startup items could be started at any point within the first few minutes logging in. Unfortunately, when your login items (from your account preferences) are executed they are all started at once. This means that most of the programs aren't ready until all the programs are ready. We should then be measuring how long it takes before you can use your key startup items, rather than timing how long it takes until all startup items are launched.
The problem with the applescript solution
I've seen some very elegant applescript solutions. For example:
tell application "Firefox" to launch
tell application "Apple Help" to launch
Unfortunately, when you use an applescript in this way, the applescript will scan through your applications folder to find the apps you've told it to launch. That means lots and lots of unneeded disk activity, which is your primary bottleneck on startup.
Shell script
I recommend writing a shell script. This code sucks, but it works. I've added a start delay aka. startup as this is called with the other startup items and I've added delays aka. between between the items to stagger the launch process.
#!/bin/sh
startup=15
between=1
sleep $startup
open /Applications/smcFanControl.app/
sleep $between
open /Applications/Spirited\ Away.app/
sleep $between
open /Applications/Mozy.app/Contents/Resources/Mozy\ Status.app/
sleep $between
open /Applications/Microsoft\ AutoUpdate.app/Contents/MacOS/Microsoft\ AU\ Daemon.app/
sleep $between
open /Applications/Palm/Transport\ Monitor/
sleep $between
open /Applications/iTunes.app/Contents/Resources/iTunesHelper.app/
You may need to wrap this in an applescript ala:
do shell script "mystartupscript.sh"
add this to your login items.
Pros
-I can start using quicksilver much more quickly now.
-My computer is more responsive in the first couple minutes after login.
Cons
-How often do I actually restart my computer ? Rarely, so the benefit is minimal.
-More difficult to maintain than using standard login items.
I'd also recommend that you check out Lingon for speeding up your startup in general.
http://lingon.sourceforge.net/
Update: Avi Flax has some comments about startup optimization
Quicksilver Compound Commands with Delays
Posted by Luke Cowell on May 10, 2007 at 07:50 AM
I been using quicksilver for a while and it's a great tool. One thing that disappointed me was that when building triggers I could figure out how to couple several commands together. I found this article which does a great explanation of it.
You need advanced features enabled etc. I won't re-explain what the previous author did a great job of explaining, but I will explain how to overcome one limitation they mentioned.
How do I order and add delays between commands ?
-Enable the Terminal Commands module
-For the sake of a simple illustration I'm going to use large type commands. Create and save 2 large type commands using the encapsulate [^+Enter] and save commands.
-Enter the following:
.sleep 5 [tab]
run command in shell [enter]
[^+Enter] (to encapsulate command)
Save command to file [enter]
Let's try it out. Use the comma trick to group your 3 commands together and then choose to run. Neet - delay.
Extending the concepts
-you can use the comma trick to build triggers with multiple commands coupled together.
-you can change the delay between commands by modifying the number after sleep.
-you can couple as many commands together as you want using this technique, adding delays when required.
Secure Notes with Keychain Access
Posted by Luke Cowell on April 30, 2007 at 06:51 PM
How many of you have a sticky note or a text file on your computer with sensitive information in it ? We sometimes forget that it would be possible to have our computers stolen. Often times the information on our computers is of more value that the computer itself.
Keep your information safe using keychain access (located under /Applications/Utilities).
Keychain will unlock the login keychain on your computer using your account password. If your passwords match then you want have to enter your keychain password on login. This is important to note because it means that anything that resides in the login keychain is not really protected if someone can gain access to your account.
- Open Keychain Access.
- Create a new keychain if you don't already have one. (File->New Keychain).
- Create a new secure note in that keychain (File->New Secure Note Item).
- Give your note a name and enter your secure information.
Don't forget to lock your keychain (Apple-L) before closing the app.
How to find the DHCP assigned address of a new device
Posted by Luke Cowell on April 23, 2007 at 06:05 PM
Most devices that can talk TCP/IP have screens attached to them. I was setting up a IP->Serial device by quatech and it conveniently uses DHCP. It turned on and got link, but what's its IP ?
Most devices with an Ethernet interface will have the MAC address written somewhere on the outside of the device. Find the mac address and you're half way there.
MAC addresses are 6 sets of numbers. The first 3 sets are assigned uniquely to a manufacturer and the last 3 are for that manufacturer to distribute.
- A MAC address is always unique, an IP is not.
- A MAC address does not change for an interface, an IP can.
I found the MAC address of my device on the case: 00:0b:28:00:1d:8d
Now I ping the broadcast of my subnet:
eg. ping -c 1 192.168.2.255
And finally I check my now populated ARP cache:
eg. arp -an | grep -i 00:0b:28:00:1d:8d
Yielding:
? (192.168.2.224) at 00:0b:28:00:1d:8d on fxp0 [ethernet]
You can always look up MAC addresses and vendors here:
http://coffer.com/mac_find/
Soekris Solution
Posted by Luke Cowell on April 22, 2007 at 02:05 PM
Previously, I had written about what you can do with a soekris. I did get everything working as it should and it was pretty interesting. I'd unplug the unit and watch that it would flawlessly start up again. This is important because there's no screen hooked up to these and if the filesystem were to become dirty, login requires a null modem cable, which not everyone keeps handy.
Goals:
- can be safely unplugged at any time
- can receive commands from web or sshd
I thought that I had achieved my goals and all I had to do was put a web server on there. Lighttp seemed like the best bet, being pretty compact, but I was quickly realizing that 64MB wasn't going to be enough space to fit everything into.
What I've opted to do is have the / filesystem still load up with the kernel into memory in rw mode and have /usr mount off the card in ro mode. Lock and temp files are written to /tmp + /var under /.
The only pain in the butt about this is that if you want to make a change to the system you must connect a null modem cable and do the following.
shutdown -t now
umount /dev/wd0a
mount -rw /dev/wd0a
This isn't a major problem as there are rarely going to be changes that need to be made. I the next iteration of this project I am going to divide the card up into several partitions.
wd0a rw / 25MB kernel is read of this partition. Completely volatile.
wd0e ro /usr 200MB all our userland stuff
wd0f rw /data 10MB special
/data would be mounted after startup by an rc script. This is important because /data could be dirty and could impede startup of basic services. /data would also have a rc.local script on it that would be called on startup that would copy files or config, restart services etc.
I've also been working a little with Axis cameras and have noticed that they use an embedded Linux system. I think they use uClinux.
Soekris Basics
Posted by Luke Cowell on April 10, 2007 at 10:24 PM
What do you call a 486 with 64MB of RAM that’s roughly the size of a small book ? A Soekris 4501. These units are equipped with 3 ethernet ports, a mini PCI slot, a full size PCI slot, a Compact Flash slot and 2 serial ports. The second serial port would need to be wired up manually.
It’s been interesting figuring out the best way to configure this. The simplest route is to just treat the CF card as a HDD. This works well with a couple of exceptions:- Powering the machine off without properly shutting it down can lead to a dirty disk, which could mean that you’d need to intervene with a console the next time it starts up.
- I haven’t read anything conclusive, but it’s been said that continuous use of a CF card will drastically shorten its life.
- Not fast.
The other method is to have the system boot into a ram disk. There are actually 2 ram disks that are created. One is part of the kernel and we would equip it to have a basic set of tools to prepare the second ram disk. The second ram disk is mounted at /usr – nothing too interesting there.
I’ve used FreeBSD for years, but this is the first time I’ve spend any serious amount of time using OpenBSD. I’m really impressed with OpenBSD. It’s so minimalistic, which makes it a perfect candidate for an embedded system, such as this.
The documentation here was very helpful and I’ll let it cover most of the details:
http://256.com/gray/docs/soekris_openbsd_diskless/
- Kernel is read in off the CF card and basic ram disk is created. At this point we are still independent of the CF card, but we can’t really do anything yet.
- The rc.diskless script creates the second ram disk (/usr), mounts the CF card and copies /mnt/usr off the CF into /usr (on the ram disk).
- Unmounts the CF card.
This means that if I were to unplug the soekris, unless it’s booting up, it will never be a dirty disk.
I used OpenBSD 4 (vs 3.x), so I had to do some minor adaptations. I also really sped up the development process by using a couple of VMs to be the host/build OpenBSD system and one to be the ‘soekris’.
Blog Software
Posted by Luke Cowell on April 10, 2007 at 08:00 AM
I'm using a little something called simplelog for my site. I decided to use it based on the following criteria:
Built on Ruby on Rails
A Web development platform I'm very interested in.
- Simple to use
Good built in theme
I have no design ability.
Can link directly to posts
Many of my posts will be for the benefit of my clients and peers and I need to be able to link directly
Tags
I will be posting on a wide variety of subject, that in combination with my bad memory means that I might be making the same posts twice. The same posts twice.
I'm sure that there's other great blog tools out there. What are you using ? What are you staying away from ?
Got Blog ?
Posted by Luke Cowell on April 09, 2007 at 09:29 AM
I too can now join the masses - I have a blog. It's yet to be seen whether this is something that is regularly updated.
The reason that I'm doing this is that I'm finding that there is more and more need to:
- Document - even if only for my own benefit, how I accomplished something.
- Give back to the community by doing so.
And so to my audience of one, I would like to say "Keep checking back for exciting updates and 'how-tos'".
Luke