Samat Says Samat Jain's personal weblog

A couple years ago, I setup my first gigabit Ethernet network. I wanted to test just how fast it could go with the equipment I gave it (that is, the NICs, cabling, and switches it operated on). Gigabit Ethernet, theoretically, can operate at 1000 Mbit/sec. This translates to 125 MiB/sec, units your OS typically displays when doing downloads (1000/8). How close is your network setup to that maximum? Copying files between PCs, while being a very “real world” test, will be limited by how fast your disks can read or write. A specialized tool is needed.

While many system benchmark suites include network testing tools, most are not easily separate from their suites, and are not easy to install and use.

Enter NetStrain. It’s a very simple C application for Linux and MacOS X designed to stress network connections. Unfortunately, it’s not included in most Linux distributions or MacOS X, so you need to download and compile it yourself.

After compiling, use is simple. One machine acts as a server, and another machine acts as a client. Start the server first with:

This starts a server using IPv4 networking on port 9999 (use a different port if you know this is in use; remember to pick one above 1024 if you’re not running as root). On your client machine, start the client connect to the server (assumed to be running on IP 192.168.1.2 and port 9999):

NetStrain will then try to send as much over your network connection as it can as long as the client is running. NetStrain is very spartan, so there are not a lot of options. In addition to sending, you may want to test receiving, as well simultaneously sending and receiving. Check NetStrain’s README for details.

Most likely, you will not get anything near 125 MiB/sec—but hopefully, you’ll get better speeds than a normal 100 Mbit connection to make everything worthwhile.

What if you want to make things faster (without buying newer, better hardware)? There are many parameters you can tune on your operating system’s networking stack. However, in most modern operating systems, most of them are already set, or are automatically configured (e.g. TCP window scaling). The one major tunable is something called MTU (Maximum Transmission Unit).

Data is transferred over Ethernet in packets; the MTU defines the size of those packets. A larger packet size means fewer packets are needed to send the same amount of data, reducing the amount of processing that needs to be done by your computer, switches, and routers. Your computer’s NIC, switches, and routers need to support large-size MTUs, a feature often advertised as “Ethernet jumbo frames.” Jeff Atwood wrote an article on the promise and perils of jumbo frames that you may want to read if you’re interested.

I’m usually a luddite when it comes to the latest Internet fads. I technically did not start blogging until 2003. I didn’t create a Flickr account or a Facebook account until 2006. I never bothered with MySpace. I turned samat.org into my OpenID in 2008. Given those things, I still hate YouTube (and all web video in general), and have yet to create a podcast or upload a video. I usually don’t think lolcats are funny, either.

Joining in the past year’s latest fad, I’ve started microblogging. Also known as “twittering,” microblogging revolves around the publication of little 140-character notes. The idea is that you share via these little notes news, thoughts, ideas, or whatever you happen to be doing at the moment. These notes are also known as “twits,” “dents,” etc.

Believe it or not, you’ve probably been doing a form of microblogging for a while. If you use an IM service and set “Away” messages, you’re microblogging. If you set your status on Facebook or LinkedIn, you’re microblogging as well. The currently accepted notion of microblogging is, started by the start-up company Twitter, a little different. Instead of messages being available to a select group of friends, your messages are global. Anyone in the world can read and respond to what you’re doing (that if, of course, if you have something interesting to say). Microblogging, Twitter-style, could be considered a type of global instant messaging.

Twitter, however, is a closed service. Your posts, lists of friends, etc live in a silo owned and controlled by them, and it’s difficult to extract data from that silo. They dictate how and when you’ll use their service, most evidenced by the frequent downtimes (it’s been so bad they’ve started a new meme, “the fail whale”). They’re also, unfortunately, a company out to make out to profit, and at this point, it’s not clear how they will do that—what if they disappear tomorrow?

Because of these and many other reasons, I’ve eschewed using Twitter and gone with Identi.ca instead. In it’s most simple description, it is an open-source Twitter clone, oriented around a new openly-developed standard for microblogging. You can download the software that runs Identi.ca (called Laconica) and run it yourself. Your data is also available in open formats: you can easily take your posts and friends lists with you. Best of all, you can still interact with other open microblogging sites in a large, distributed network, hopefully making reliability problems things of the past.

I’ve been microblogging since the beginning of the year. Most of my entries are about the same topics as this blog—Linux, open-source software, etc. I notice that I also tend to write a lot of things about New York City. If you care about any of these things, please subscribe to me on Identi.ca. If you use Twitter, you can look read my cross-postings on my Twitter account too.

Something that has bothered me forever is that applications that use GNU aspell for spell checking kept marking my name as a misspelling (I’m looking at you, KMail). Most front-end applications don’t provide a way for you to add your own custom words.

Apparently, creating your own personal dictionary is ridiculous easy with aspell.

If your language is English, create a file in your home directory called ”.aspell.en.pws”:

The first line is a required header. Every subsequent line is a word you want to add to your dictionary. I can’t believe I’ve let this sit for so long. Because it’s a nice text file, syncing this file between machines to take your dictionary with you is trivially easy.

Drupal-powered websites can be put into an “offline mode.” This is much better than most alternatives (such as taking the web server offline), especially for search engines, as the message and HTTP status codes given to users and robots alike will tell them to patiently come back later.

I’ve found that putting the site into offline mode makes database backups go much faster on heavily trafficked sites (which is obvious). However, for a particular site I was working with, this needed to be done in an automated manner, and on a dedicated database server that did not have access to the Drupal installation.

Most people take their Drupal sites offline through Drupal’s web-based administration interface. They can also be put offline through the Drupal Shell. Neither were suitable for me: the former cannot be automated easily, and the latter requires access to the Drupal installation. Fortunately, Drupal sites can easily be taken offline by setting things in the database, which can easily be done via bash scripts and the command-line MySQL client.

Given your database user is my_db_user, password my_password, and database my_drupal_db, the backup script would look something similar to:

Update: The original version of this article had some problems on some setups with the variables table being cached. I added another SQL statement to make sure this cache is flushed so the site actually reflects its configuration.

Update: This method really doesn’t work that well, and the more I think about it, there isn’t a way to get around writing something that interacts with Drupal. I’m working on a script that will be more fool-proof.

Python provides a neat way for functions to return multiple arguments via “tuple unpacking”. For example:

The same can be done in PHP relatively easily via the list construct:

SpamAssassin is one of the most-used spam filtering systems in use today. Unfortunately, because there are so many different ways SpamAssassin can be used, SpamAssassin remains subject to many performance problems. Fortunately, there are several speed-ups and optimizations that can be applied to most SpamAssassin installations to speed up its rule processing, especially on Debian and Ubuntu GNU/Linux-based systems. These instructions can be adopted to other operating systems as well.

This article does not discuss configuring your mail filtering system (i.e. procmail, maildrop). This depends completely on your setup, and more than likely there are plenty of other articles that describe the best way to setup what you want.

This past Ubucon’s key signing party was my first key signing party. One thing I noticed–signing keys after a key signing party is a boring repetitive task. Summarized from the Ubuntu wiki entry on typical key signing post-party protocol:

1. Retrieve all public keys of key signing party participants, using gpg –-recv-key
2. Compare the hardcopy fingerprint from the keysigning party to the fingerprint of the retrieved public keys, using gpg –-fingerprint
3. Sign the key, using gpg –-sign Send the signed key back, either by
   • E-mail: export the key, then e-mail it to the key owner, using gpg –-export -a | mail -s “Your signed key” user@example.com
   • Key server: send the key to a public keyserver, using gpg –send-keys

This is incredibly monotonous—and people have to wonder why Web of Trust-based encryption is not more popular?

The Debian signing-party package provides the utility caff to automate some of this. It’s not very friendly to “desktop” users, however:

• it’s a CLI application
• it requires a local MTA (/usr/sbin/sendmail in particular), or an “open” SMTP server, with no support for authenticated SMTP or SMTP/SSL
• the configuration file syntax is Perl and confusing; there are also few examples on the Internet

You could add authenticated SMTP or SMTP/SSL support to the script, but having to know how to hack Perl definitely disqualifies caffe from being a desktop-friendly application.

So, I hacked together my own key signing party script in Python that would send signed keys back to people via KMail. To use it, create a text file listing all key IDs you wish to sign, one per line. Pipe the contents of this list into the script:

The script will download each key, ask you to verify the fingerprint, and then sign it. It then will open a KMail composer window, pre-filled with the key owner’s e-mail address, a friendly template message (customizable in the script), and attached key. Review each e-mail to make sure it is kosher, and click send. Other than continuing to be a CLI program, I think this is much friendlier–the only manual work done is the creation of list of keys to sign, comparing fingerprints (this could be automated, but it seems in the spirit of the Web of Trust-based systems not to), and clicking send in a familiar desktop e-mail client.

Now for some notes…

It uses the DCOP automation features of KDE’s Kmail to send messages. You could similarly use Evolution and D-Bus, but I don’t use Evolution so I can’t contribute that bit of functionality. Mozilla’s Thunderbird unfortunately does not yet support any kind of automation features (as far as I know, anyway), so you’re completely out of luck if you use it.

DCOP with Python is a complete, utter, pain. The easy way to drag-and-drop boiler-plate code with kdcop did not work, as it appears the APIs have changed. A problem with KDE/Python dcopext’s module and multiple identically-named-functions sealed the deal for me and I gave up trying to use DCOP with Python, and instead settled for a hack of using the shell instead. I’m looking forward the one Linux desktop IPC protocol to rule them all, D-Bus, to debut in KDE4.

My script does not provide all the functionality of caffe. It, for example, does not encrypt the messages and their keys back to their owners. There doesn’t appear to be an easy way to do this with KMail and DCOP, so it’s a feature that will have to wait.

So, you’ve gotten your shiny new EVDO datacard working under Linux (if not, see High-speed cellular wireless modems (e.g. EVDO, HSPDA) in Ubuntu GNU/Linux 6.10) and you want to now setup the actual Internet connection?

In this article I document how I setup Sprint’s Mobile Broadband service with ppp in Ubuntu GNU/Linux 6.10.

Note: If you are running Ubuntu 7.04 or greater, this article is no longer relevant. Your EVDO modem should be detected and run at a higher speed automatically.

I’ve been raving about cellular wireless modems/data cards for a while now. While they’ve been available for a long while, they’ve finally become practical with networks such as EVDO and HSPDA that offer broadband-like speeds. I personally own a Novatel Merlin S720 that I use with Sprint’s Mobile Broadband service.

Most of these datacards are easy to get running in Linux–I actually setup mine in Linux faster than I did in Microsoft Windows. However, due to some shortcomings in the kernel used by Ubuntu GNU/Linux 6.10, you cannot take advantage of the speeds that these modern wireless networks offer.

This article talks about some of the problems of the often-used usbserial driver, and how to use the better-performing airprime driver instead.

I’m a T-Mobile Hotspot subscriber, but I cannot say I’m particularly happy with it. Reliability is in general pretty good, but there have been a few times a certain hotspot has been flaky, and these tend to be the times I needed access the most. It’s also a pain to have to go somewhere to get Internet access, especially when, for example, I don’t like Starbuck’s coffee. I rather have the Internet come to me.

Enter EVDO. It’s a 3rd generation cellular technology that allows for broadband-like speeds, typically almost everywhere you have a cellular phone signal. There are different speeds depending on what network is available in a particular location:

• 1xRTT, allowing for 144 Kbps/144 Kbps download/upload speeds
• EVDO 1x Rev 0, allowing for 2.45 Mbps/150 Kbps
• EVDO 1x Rev A, allowing for 3.1 Mbps/1.8 Mbps speeds.

All three types of networks are available can be found in the United States, and a typical provider’s access plan lets you roam between them anywhere in the country for free.

Access comes through a provider-specific modem (i.e. you cannot use one provider’s modem with another provider). These usually are PCMCIA cards, reminiscent of the 802.11b network cards people used before WiFi was built-into notebook computers. Connection to a provider usually is provided through PPP software. Most the modems available on the market today are a little oddball: they expose a USB controller, which then exposes a USB serial interface which controls a virtual modem. Yes, it’s strange, especially when these devices aren’t actually modems (there is no MOdulation or DEModulation taking place, the devices are more “network bridges”), but thankfully it allows these devices to easily work with alternative operating systems like Linux and MacOS X.

In the USA, there are essentially three major EVDO providers: Sprint, Verizon Wireless, and Alltell, with Sprint and Verizon having the largest networks by far. What differentiates the Sprint and Verizon, I think, is pricing and policies. If you do not want to sign a contract, both providers cost the same. If you want to sign a contract for 2 yrs, you only get a discount rate with Verizon if you’ve a qualifying voice plan—Sprint has no such limitation to get a discounted rate.

Verizon does a bit of questionable marketing: they advertise their service as “unlimited,” but they pull a trick often used in contract writing and specifically define “unlimited” as 5 GB/month. If you go over this limit, you’re breaking Verizon’s terms of service. Verizon often cancels subscribers accounts, and assumes you are a criminal, downloading illegal music or software. An article in the Washington Post, Bandwidth Bandit, discusses about one subscriber’s woes. Their terms of service disallows many popular Internet applications as well, such as VoIP, video conferencing, or any online gaming. Sprint’s terms of service are more vague and do not explicitly disallow these things, but reports from their subscribers say that they don’t have unreasonably low bandwidth limits nor have draconian policy enforcement assuming you guilty until proven innocent.

This wouldn’t be a good summary without me discussing what new bleeding-edge technology was right around the corner. EVDO Rev B, allowing for at least 4.9 Mbps/1.8 Mbps speeds, has been deployed in a few places in Asia, but given how backward North America tends to be in technology adoption, won’t be in the United States anytime soon. WiMAX, a 4th generation cellular technology allowing for speeds of at least 10 Mbps, will probably take the place of EVDO. Sprint is the only major provider dedicated to building a WiMAX network, with plans to begin deployment at the end of 2007.

Some external links with good information:

• EVDOforums.com: Forum dedicated to EVDO. Sponsored by an online store Booster-Antenna.com, which specializes in EVDO technologies, offers much better prices than provider’s retail stores. I maintain a thread about EVDO on Linux, with community-provided configurations and vignettes.
• EVDO on Wikipedia
• Comparison of wireless data standards on Wikipedia