Bitcetera TechBlog

wemakeit.ch

Thu, 03 May 2012 18:43:41 GMT

It’s been rather quiet on this blog lately. The reason is the latest Ruby on Rails project I’m working on: wemakeit.ch, a crowdfunding web application which focuses on culture and the arts. The wish list of features is still pretty long, but we’re very pleased with how it looks and behaves at this point. Take a look, you might like it as well and maybe you even find something worth some of your bucks on wemakeit.ch.

(Sven Schwyn)

sendgrowl

Tue, 22 Nov 2011 21:10:19 GMT

This is a followup to the sendmail via Growl article a while ago.

growlnotify with Callback

Growl 1.3 has switched to the new GNTP (Growl Notification Transport Protocol) and it’s growlnotify companion now features limited support for “notification clicked” callbacks. Let’s see what we can do with this.

The old sendmail-via-growl script displayed messages piped to sendmail as Growl notifications. This is great for system messages from Cron which are usually short and text only. It’s not so useful for larger or HTML messages since Growl notifications are very limited when it comes to available space and styling options. Wouldn’t it be nice to just click on a Growl notification and then see a preview of the message?

Bye sendmail-via-growl, Hello sendgrowl

The new sendgrowl script does just that. Download sendgrowl, gunzip and save it as /usr/local/bin/sendgrowl, make it executable and then wire it up with sendgrowl --on. The original sendmail executables are renamed and symlinks to sendgrowl are created instead. To revert these changes, just call sendgrowl --off.

When a message hits sendmail now, it triggers a Growl notification. Click it within 10 seconds to see the entire message in a Quick Look preview window. The 10 seconds are hardcoded into growlnotify to keep it from hanging forever. However, if you miss this time frame, you can manually open the preview with sendgrowl -s {ID}. The ID is displayed in the Growl notification.

The sendgrowl script filters “X-” mail headers only, however, it’s certainly possible to chain in more sed commands to make the Growl notification more readable. Please add your improvements as comments.

MockSMTP

You need a more comprehensive mail preview solution to develop webapps which send multipart messages? MockSMTP may be what you are looking for.

(Sven Schwyn)

Diaspora*

Sun, 20 Nov 2011 16:37:18 GMT

If you know Diaspora* already and just want to know how to become part of it, simply skip the first two sections.

Social Networking

Linking with other people on the net is great, particularly if you have friends in far away places or enjoy a more nomadic lifestyle yourself. And it’s cheap to stay in touch and communicate since Facebook® and alike are free, or …?

Wrong! Facebook® will soon operate more than 100’000 servers and someone has to pay for all this. The mere quantity of equipment indicates, that Facebook® not only serves it’s users. A lot of juice goes into further processing of all this data in order to pour it into products interesting to enterprise customers.

In other words, your data is being sold. Your pictures, your posts and your links to other people are being sold. In fact, your entire digital self is being sold. And all this is perfectly legal since you have given Facebook® a “non-exclusive, transferable, sub-licensable, royalty-free, worldwide license to use any IP content” by agreeing to it’s terms. There are no limits as to what for.

But Facebook® doesn’t stop there, it aggregates data on people who are not active on Facebook®, it even creates shadow profiles for those who don’t even have a proper account. And unless you disable the well hidden automatic face recognition, machines will take over where you are lazy and interlink all faces on your uploaded pictures.

Some of this activity is illegal in many countries since it deals with personal data of people who have never agreed to the Facebook® terms. There will be lawsuits in the future, but meanwhile the data monster will continue to grow fat.

Big Brother Inc is watching you! And me, whether we agree or not.

Social Networking Reloaded

Roughly one year ago a group of developers began work on a crowdfunded next generation social network project called Diaspora*. The main difference: Unlike Facebook® your data is not stored in one centralized database owned by one giant enterprise but on an independent Diaspora* server called “pod”. These Diaspora* pods can communicate between each other and thus link friends and exchange posts across all pods of the Diaspora* network. You are free to choose any pod operator or even run your own pod.

If you ever change your mind, you can export all your data (which is utterly impossible on Facebook®). And soon it will be possible to take this export and recreate your profile on another pod.

How to Become Part of Diaspora*

But I Have All my Friends on Facebook® already

Diaspora* is still young and doesn’t offer nearly as many features as Facebook® yet. (As a side note: Diaspora* introduced ‘aspects’ later copied by Google+ as ‘circles’ and Facebook® as ‘lists’.) Furthermore, all your friends are on Facebook® which seems to make the change of social network unthinkable.

No problem, Diaspora* plays nice with other services such as Facebook® or Twitter®. If you want, it can forward all your posts on Diaspora* to Facebook® and it also offers a tool to ease the transition of your friend links. You won’t have to hit no big reset button and start from scratch, the idea is to convince one friend at a time and thus slowly move towards the day when Facebook® becomes obsolete.

If you run a server, you might consider to install your own Diaspora* pod.

Everybody else is better off with a public Diaspora* pod. All pods are equal, but you might prefer certain operators based on who runs the pod or it’s physical and thus legal location.

Choose any public Diaspora* pod.
Create an account.
Link your account to your Facebook® account.
Check out the tutorials and/or FAQ to learn the easy to use basics.

To connect with other people, sign in to Diaspora* and simply look for them in the top search field. However, keep the following in mind: Since users are spread on many independent Diaspora* pods, you have to search for the fully qualified username which resembles an email address. (Mine is svoop@diaspora.bitcetera.com for instance.) Fetching usernames from other pods may take a moment, so if your search doesn’t instantly yield any results, just wait a minute and try again.

(Sven Schwyn)

Changed Overlay URLs

Thu, 27 Oct 2011 15:25:21 GMT

Gentoo ebuild overlays managed through layman have been slightly overhauled recently. We’ve chosen to update our overlays before the old specification is deprecated. If you use any of our overlays, edit the file /etc/layman/layman.cfg and replace the URL as follows:

  OLD:
    http://github.com/svoop/bitcetera-overlay/raw/master/layman.xml
  NEW:
    http://github.com/svoop/bitcetera-overlay/raw/master/repositories.xml

(Sven Schwyn)

Mail Followup Marker Sanitation

Tue, 18 Oct 2011 14:19:16 GMT

First of all, I love languages. I’m fluent in four and a half of them and have no doubts that the variety of languages is a vital part of our human culture. However …

Once upon a time

Well before WWW saw the day of light, people had already communicated on the early internet by use of email and Usenet. They had agreed on the use of “Re:” in the subject to denote “this is a followup to your recent mail”. For Usenet, this was formally codified 1983 in RFC 850 and later for email in RFC 2822.

Unfortunately, some modern mail clients (most notably Outlook) don’t care. They interpret “Re” as short for “response” and thus translate it to other locales. A german Outlook user therefore prefixes “AW” and I’m sure many more such prefixes exist in other languages – a clear breach of standards.

Threads

To make things worse, if two email clients with different locales pingpong a message, the subject line sprawls on every response: “Hello”, “Re: Hello”, “AW: Re: Hello”, “Re: AW: Re: Hello” and so forth. Since modern email clients group messages by the “Re”-stripped subject, they will fail to do so in the above case if they stick to the standards. What a mess!

Postfix’ header_checks to the Rescue

Postfix offers an easy way to rewrite subject lines by use of regular expressions. Please note that my examples rely on PCRE (Perl compatible regular expressions), so make sure your Postfix is compiled with PCRE support. If postconf -m | grep pcre doesn’t return anything, your Postfix does not yet do so.

Now create the file /etc/postfix/header_checks with the following content:

/^Subject:s*((Fwd|WG):s*)+(.*)$/          REPLACE Subject: Fwd: $3
/^Subject:s*((Re|AW|Antwort):s*)+(.*)$/   REPLACE Subject: Re: $3

These two lines replace and collapse the german translations of “Re” and “Fwd” with their standard complient counterparts. You may add other languages to the list such as the danish “Svar”. Please drop a comment below since I’d like to compile a more comprehensive list of translations out there.

Now we’re ready to apply these checks. Edit the file /etc/postfix/main.cf to contain:

header_checks = pcre:/etc/postfix/header_checks

Finally, reload or restart Postfix and send yourself an email with a subject such as “AW: Re: AW: AW: Hello World” for testing.

(Sven Schwyn)

Lion Fullscreen for TextMate

Wed, 17 Aug 2011 09:32:32 GMT

While some changes in early Mac OS X Lion versions reduce overall usability (e.g. I’d love a more exposéish Mission Control), the introduction of the systemwide fullscreen mode comes in very handy in everyday use. Wouldn’t it be great to put TextMate into fullscreen mode as well?

But wait, TextMate development has not been particularly agile in recent years and who wants to wait for TextMate 2 in order to see this feature. You could compile your own brew of TextMate from source to get fullscreen, tutorials for this are available in the depths of the interweb. But there’s a much simpler solution.

Thanks to enormego, installing a tiny plugin for TextMate is all it takes. Browse to the EGOTextMateFullScreen page on Github, hit the download button, unzip the file and double-click on the plugin. Voilà!

(Sven Schwyn)

NoSQL on Mac OS X

Tue, 15 Feb 2011 17:56:38 GMT

NoSQL databases such as MongoDB and Redis have become quite popular especially in the Ruby and Rails ecosystem. If you’re developing on a Mac though, installation can be quite a hassle unless you rely on some package manager. The easiest solution is probably Homebrew, a quite mature installer written in Ruby. However, it lacks many feautures of a real package manager such as versions, slotting and a clean uninstall (which includes dependencies).

This is why I prefer Gentoo Prefix which is powered by Gentoo’s excellent Portage source package manager. Check out my introductory blog post to get a feel for Gentoo Prefix.

MongoDB

MongoDB is not yet part of the official Gentoo Prefix packages repository as of now, but you can still install it from our overlay. Follow the instructions to subscribe to it.

MongoDB defaults to the Spidermonkey JS engine, however, due to some issues with packaging and speed, I prefer Google’s V8 engine – at least for now:

echo "dev-db/mongodb v8" >>$EPREFIX/etc/portage/package.use
echo "dev-lang/v8 **" >>$EPREFIX/etc/portage/package.keywords
emerge -av mongodb

The following aliases come in handy for your daily work with MongoDB:

alias mongo-start="mongod --fork --dbpath \$EPREFIX/var/lib/mongodb --logpath \$EPREFIX/var/log/mongodb.log"
alias mongo-stop="killall -SIGTERM mongod 2>/dev/null"
alias mongo-status="killall -0 mongod 2>/dev/null; if [ \$? -eq 0 ]; then echo 'started'; else echo 'stopped'; fi"

Let’s go ahead and try:

mongo-start
mongo

Redis

The installation for Redis is the usual one-liner:

emerge -av redis

The convenience aliases:

alias redis-start="redis-server \$EPREFIX/etc/redis.conf"
alias redis-stop="killall -SIGTERM redis-server 2>/dev/null"
alias redis-status="killall -0 redis-server 2>/dev/null; if [ \$? -eq 0 ]; then echo 'started'; else echo 'stopped'; fi"

That’s it! Give it a try:

redis-start
redis-cli

(Sven Schwyn)

Sendmail via Growl

Sun, 05 Dec 2010 16:20:35 GMT

This blog article is out of date, check out sendgrowl instead.

If you have never seen the shell prompt of your Mac, then this post is not for you. However, if you are used to work with more than just the eyecandy parts of Mac OS X, you probably have come across the following problem at some point: Some asynchronous system tools such as Cron or Portage from Gentoo Prefix pipe their output to sendmail and hope someone will receive it. Yet, that won’t happen with the current default settings.

Postfix or sSMTP

There are several ways to fix this. You could configure the Postfix MTA bundled with Mac OS X to relay the messages to your SMTP host. Unless you know quite a bit about Postfix already, you should however cancel your plans for the rest of the day. Or go for some frontend GUI application if you don’t mind it to mess around with your configuration.

You could also install sSMTP as a really simple replacement MTA which is a breeze to configure. Unfortunately, it doesn’t do SASL (unless you build from Andrew Wansink’s fork bfg branch). And without SASL, delivery will quite silently fail if the recipient address relies on greylisting to reduce SPAM.

Sendmail via Growl

Time to take one step back: Is it really necessary to deliver these messages? If this were a server, the answer would be an unconditional “yes”, but we are talking about a Mac OS X workstation here, so there shouldn’t be any root cronjobs and the Mac shouldn’t be on all day and night – for our planet’s sake. Why not just display a Growl notification?

It is pretty easy to turn sendmail messages piped to sendmail into Growl notifications, so I’ve written a little script for this purpose. Download sendmail-via-growl, move it to /usr/local/bin and make it executable. As always, you are using this script at your own risk.

To enable Growl notifications:

sendmail-via-growl --on

To test Growl notifications:

echo “Let’s test this!” | sendmail

The Growl notification strips all mail headers beginning with “X-”. If you want to read the complete message, look for it in /tmp/sendmail-via-growl where old messages are kept for 30 days.

And you can of course revert back to the original sendmail:

sendmail-via-growl --off

(Sven Schwyn)

Cloud Backup Tue, 30 Nov 2010 17:38:18 GMT Backups are important. Most people have learned that, some the hard way, and keep backups on external drives. This is a good idea, but it’s not enough. Physical desasters like fire or thieves in the house still have the potential to carve your name into the gravestone of deceased digital lives. Cloud backups to the rescue! Requirements The storage provider should be fast, unlimited and affordable. Furthermore, you want to keep at least some control as to where the data is actually being stored. (Jurisdiction in say US and EU differ significantly when it comes to data privacy.) As of now, the best match is Amazon S3. The cloud backup software on the other hand has to encrypt the data before it is sent to the cloud. Moreover, subsequent backups should only store the bits that have changed meanwhile. And of course the software should be easy to use and as failsafe as possible. Mac OS X It’s been a fruitless task to find a suitable cloud backup software on Mac OS X ever since cloud storage appeared in the sky. But fear not, a relatively new kid on the block fills this gap amazingly well: Arq by Haystack Software. Arq mimics Apple’s TimeMachine when it comes to ease of use, yet it’s still possible to tweak important parameters such as the physical storage region or the redundancy class (more reduntant = more expensive). All you have to do is create an Amazon S3 account, then launch Arq and answer the few question it initially asks. If you are on a high bandwith link, you can configure Arq to do hourly or daily backups. Nomadic users on the other hand may prefer to kick off backups manually from the Arq menubar item. The overall volume (and thus cost) of cloud storage can be limited. Restoring files is as easy as dragging them out of the Arq window and even if the Mac you backup from gets lost in a sparkling blaze, it’s very easy to restore the files to any other Mac – provided you remember your encryption password or have a hardcopy of it safely tucked away in a safe place. Arq is not freeware, but it’s very actively maintained and worth every cent. Linux Servers Failing backup systems offered by hosting providers are not unheard of and disasters may not stop at the doors of a server farm. Cloud backups therefore make sense for servers as well, at least for very essential data such as /etc or database dumps. The list of available backup software is quite large but shrinks considerably if you are looking for a console tool with support for Amazon S3. A mature solution is duplicity by Ben Escoto and Kenneth Loafman, essentially a Python script which relies on rsync, GnuPG (for encryption), boto (for Amazon S3 connectivity) and NcFTP as an alternative storage backend. The following wrapper script is an example for how to call duplicity as a daily cronjob. Check out the man page of duplicity for details. You should create a new set of GPG keys per server solely for use with duplicity as the passphrase for this GPG key is present in the wrapper script. Also make sure you set restrictive permissions on the wrapper script. To test your setup, use the --dry-run option which is handed through to duplicity. #!/bin/bash export AWS_URL='s3+http://myserver.mydomain.com' # Amazon S3 bucket export AWS_ACCESS_KEY_ID='' # Amazon S3 key ID export AWS_SECRET_ACCESS_KEY='' # Amazon S3 access key export GPG_ENCRYPT_KEY='root@myserver.mydomain.com' # GPG key for encryption export GPG_SIGN_KEY='' # GPG key ID (hex) for signing export PASSPHRASE='' # GPG key passphrase duplicity \ --full-if-older-than 1M \ --include /etc \ --include /root \ --exclude '**' \ --exclude-other-filesystems \ --encrypt-key ${GPG_ENCRYPT_KEY} \ --sign-key ${GPG_SIGN_KEY} \ / \ ${AWS_URL} \ --s3-european-buckets \ --s3-use-new-style \ --verbosity warning \ $(if [ "$1" == '--dry-run' ]; then echo '--dry-run'; fi) \ $(if [ "$1" != '--dry-run' ]; then echo '--no-print-statistics'; fi) \ \ && \ \ duplicity remove-all-but-n-full \ 12 \ ${AWS_URL} \ --s3-european-buckets \ --s3-use-new-style \ --verbosity error \ $(if [ "$1" != '--dry-run' ]; then echo '--force'; fi) \ \ && \ \ duplicity cleanup \ --extra-clean \ ${AWS_URL} \ --s3-european-buckets \ --s3-use-new-style \ --verbosity error \ $(if [ "$1" != '--dry-run' ]; then echo '--force'; fi) \ >/dev/null This is just an example which works for me, you use it at your own risk. Other I haven’t investigated other use cases such as Linux workstations or Windows. Feel free to share your killer apps by submitting a comment. (Sven Schwyn) RSS Tips for Mail.app Mon, 17 May 2010 16:02:38 GMT Why Mail.app? While Mail.app certainly ain’t the best RSS reader out there, having both email and RSS feeds under one roof certainly has it’s advantages. This feature has been around for a while but only recently have some nasty bugs been resolved which caused read and deleted articles to re-appear at random. RSS Folders If you’re like me subscriber of a whole range of RSS feeds, you might want to group feeds at some point. Wouldn’t it be nice to have all say Ruby related RSS articles in one folder? This feature is available in Mail.app, however, the context menu in the sidebar doesn’t show it and thus many don’t know about it. To create a RSS folder, click the “Mailbox -> New Mailbox” menu item and then select “RSS” in the location dropdown. The folder appears in the sidebar and you can move feeds there. Make sure you enable the Mailbox column in order to see from which RSS feed a particular article originates. Feeds with Media Content Some feeds include media content, most notably pod- or screencast feeds. As of version 4.2, Mail.app attempts to download all linked media content of all RSS articles being fetched which may result in several GB of traffic, takes a long while and slowly eats up your memory. Once you run out of physical memory, the Mac starts to scratch to disk which pretty much stalls the computer. There’s no easy cure for this, however, a bug has been filed already and Apple seems to be working on it. Meanwhile, you will have to rely e.g. on iTunes to subscribe to the pod- or screencast directly. Feed URL Have you ever tried to find out from which URL a RSS feed viewed in Mail.app is being fetched? There is no easy way to display this information in Mail.app itself, but you can open a terminal and type pubsub list to get a list of all RSS feeds subscribed by either Mail.app, Safari or another application using the facilities of Mac OS X. (Sven Schwyn)