Archive for the '*nix' Category

Adding Docs for iPython on Mac OSX Leopard

I have been playing with iPython as part of my exploration into Python. A friend tipped me off that iPython is a little more sophisticated than the standard Python interactive prompt.

One of the features of iPython is an interactive help guide for the language. It’s pretty sweet, however, on the standard Python Mac build, it doesn’t include all the documentation. This means some of the help guide is also missing parts. In order to add the HTML docs for Python 2.5, I ended up needing to compile the docs. This required me to install a few items, which are listed below.

Essentially, the docs need to be compiled as HTML from LaTeX. I had to install LaTeX for Mac, LaTeX2HTML and Darwin Ports (aka Mac Ports). This wasn’t too difficult, but I had a few missteps along the way. The directions are mostly for my own benefit, but if you find them useful or need a correction please let me know.

Goal: Create a link to the Python Docs for use with iPython.

  • Install Darwin Ports
  • Install LaTeX for Mac
  • Intall LaTeX2HTML
  • Download Python 2.5
  • Copy compiled HTML docs to Standard Location
  • Create env variable PYTHONDOCS for iPython help reference

Notes: I am running Mac OSX 10.5.4 with Python 2.5.1 and iPython 0.10. The $ is simply my prompt in Terminal.

Installing Darwin ports is very fast and easy, just visit http://darwinports.com/ and grab the latest version. Once you get it downloaded, it’s only 1.4 megs, just click and install. It’s a very quick step.

For LaTeX, you have two choices that are outlined in this guide. I chose the simple method, which results in downloading a fairly large file (1.2 gigs) but I didn’t want to spend my day debugging LaTeX install issues, after all this is to help me learning Python!

After downloading LaTeX, it’s a basic package installer. This is the version I used.

The next requirement is the LaTeX2HTML program. It converts the doc files to HTML that will be used by iPython. You can read more about it here. I found the installation instructions were not exactly what I needed. I had to add the /opt/local/bin to my Bash path in order to install LaTeX2HTML. I did this permanently by altering .bash_profile, but for the purpose of this instructions, I have done this temporarily.

# adding to Path
$ PATH = $PATH:/opt/local/bin
$ cd  /opt/local/bin
$ sudo mkdir -p portslocation/dports/latex2html/
$ cd portslocation/dports/latex2html/
# install latex2html
$ sudo port install latex2html

This process takes a little while, so grab a cup of coffee while it’s installing. After it finishes, you’re all set to compile the doc files for Python. First, you need to download the entire Python package. Take a look at the releases here. Here is the specific process I followed.

$ cd ~; mkdir tmp; cd tmp
$ curl -O http://www.python.org/ftp/python/2.5/Python-2.5.tgz
$ tar -zxf Python-2.5.tgz
$ cd Python-2.5/Doc
$ make

This will compile the docs. Specifically, we are interested in the HTML version it creates via LaTeX2HTML. Once it’s complete, you need to place the HTML files somewhere more permanent.

$ sudo cp -r html/ /usr/share/doc/python-2.5-html

Now it’s time to set PYTHONDOCS to the location we just moved the HTML files too. You can do this a few ways, the best method is adding it to the .bash_profile.

$ vi ~/.bash_profile
## Add the following
# export PYTHONDOCS=/usr/share/doc/python-2.5-html
# Save the file and exit
$ source ~/.bash_profile

You’re now ready to start using the doc files with iPython. For convenience, I have attached the compiled files here. Have fun!

Blog Optimization – Enabling Mod Deflate

Getting started with Apache Mod Deflate was fairly simple. Like mod concat from the first piece in the Blog Optimization series, it’s just a matter of compiling and installing mod deflate. Mod deflate comes with Apache 2.2, the first step is locating it.

prompt:$ find / -name mod_deflate.c
# change directory (cd) to the location of mod_deflate.c
prompt:$ apxs -i -a -c mod_deflate.c
prompt:$ apachectl configtest
# make sure the syntax is okay
prompt:$ apachectl graceful # restart the server

Now that mod_deflate is enabled, it’s time to configure it. I couldn’t find much information during the research phase on configuring the module. I knew from previous experience that I wanted to gzip, or deflate, all static files. Eventually, I found this post on configuring mod_deflate with cPanel. It was helpful, I modified it slightly to include a few other file types

<Location />
    AddOutputFilterByType DEFLATE text/html text/plain text/xml application/xml application/xhtml+xml text/javascript text/css application/x-javascript application/x-shockwave-flash
</Location>

The Apache 2.2 docs indicate the AddOutputFilterByType directive is deprecated and mod_filter should be used instead, I haven’t had time to play around with mod filter and the Apache 2.2 docs urge caution since it’s not fully tested. If anyone has experience with mod filter, please share in the comments section below.

Mod deflate has done wonders to reduce my bandwidth costs and site performance. On average each request uses 25% less bandwidth than without mod_deflate. Obviously images don’t benefit from gzip compression, but all JavaScript, CSS, HTML and flash files are benefiting by being about 50% smaller in size. For instance, JQuery is about 30.3kb on the server. When it’s delivered using mod_deflate it’s 16kb according to Firebug.

Have a look at some before and after numbers from my server logs.

Day Number
of
visits
Pages Hits Bandwidth
26 Dec 761 3966 12131 242.29 MB
31 Dec 761 4161 12809 196.94 MB

Do you have any blog optimization techniques you are using on your site? Check out my related posts on Blog Bloat and using Mod concat if you interested in optimizing your blog or website.

Blog Optimization – Getting Started with Mod Concat

Following my post on Blog Bloat, I have finally gotten around to installing mod_concat. I asked my hosting provider ServInt to handle the upgrade to Apache 2.2.x, they did it in about 20 mins – thanks guys, you rock!

The actual installation of mod_concat was fairly simple thanks to the included instructions, though it lacked a few of the finer points, like needing to install the module into the Apache modules directory and modifying httpd.conf to include mod_concat, this is easily handled by adding the -i and -a flag. First grab the project out of SVN, then you can compile it.

prompt:$ svn checkout http://modconcat.googlecode.com/svn/trunk/ modconcat-read-only
prompt:$ cd modconcat-read-only/mod_concat
prompt:$ apxs -i -a -c mod_concat.c 
#Make sure the config file is working okay
prompt:$ apachectl configtest
#Then it's safe to restart and use the concat module
prompt:$ apachectl -k graceful

That’s all it takes to get the module working with Apache 2.2.x. I setup a super simple test here to ensure mod_concat is working.

Optimizing WordPress K2 using mod_concat

Getting this Apache module working with WordPress and K2 wasn’t quite as simple as I was hoping. I ended up having to do more customization work than I desired. The issue with K2 is that many of the included JavaScript files, for handling the rolling archives and live search, have a .php extension that includes a gzip directive header. Additionally, the K2 files are nested so deep that mod_concat can’t handle it.

To deal with these short comings, I sourced the K2 JavaScript files and moved them to a root directory. Currently, the files to be concatenated are hard coded into the plugin. Perhaps a future revision will resolve this, however, I am more interested in testing out mod_jsmin and mod_deflate before I get around to this enhancement.

Here is the plugin code I wrote in order to use mod_concat with K2.

<?php
/*
Plugin Name: WP Mod Concat
Description: This plugin leverage the Apache 2.2.x module mod_concat to combine a list of files into a single file
Version: 0.1
Author: Gregory Tomlinson
Author URI: http://gregorytomlinson.com/encoded/
*/
 
remove_action('wp_head', 'wp_print_scripts');
add_action( 'wp_head', 'wp_mod_concat_optimzie' );
 
function wp_mod_concat_optimzie(  $handles = false  ) {
        $base_url = "http://gregorytomlinson.com/js/??";
        $script_files = array("jquery_1_2_6.js", "k2.functions.js", "k2.livesearch.js",  "k2.rollingarchives.js",  "k2.slider.js", "k2.trimmer.js", "swfobject.js" );
 
        echo '<script type="text/javascript" src="'. $base_url . join(",", $script_files) .  '"></script>';
}
?>

By using remove_action(’wp_head’, ‘wp_print_scripts’);, I can stop WordPress from loading JavaScript files. I then replace the head script files with the concatenated version leveraging mod_concat, which produces a script tag that looks like the following.

<script type="text/javascript" src="http://gregorytomlinson.com/js/??jquery_1_2_6.js,k2.functions.js,k2.livesearch.js,k2.rollingarchives.js,k2.slider.js,k2.trimmer.js,swfobject.js"></script>

You can learn more about mod_concat here.

The other drawback to mod_concat is it’s incapable of handling query parameters. I would like to be able to append a ?ver=1.1 in order to handle busting the browser cache, but the current version isn’t able to do this, maybe Ian can fix this :)

That’s all it takes to use mod_concat. It’s probably not the ideal method to use with WordPress and K2, you may consider using PHP Speedy. Dave Artz runs down a list of other concatenation methods in this blog post.

Installing SVN on Red Hat Enterprise Linux AS release 4

This couldn’t be easier, all I did was run the following, once I figured out what flavor of Red Hat the IT team had installed on my machine.

prompt:$ rpm -i http://the.earth.li/pub/subversion/summersoft.fay.ar.us/pub/subversion/latest/rhel-4/bin/subversion-1.3.2-1.rhel4.i386.rpm

You can find SVN for your operating system here. Many thanks to Wijaya for helping me quickly determine my flavor of Red Hat in the blog post here.

Open or Create a file in Terminal to Coda

I have recently started using Coda by Panic (the people who make Transmit). I have to say I really like it. I have been a die hard BBEdit fan for years and was hesitant to make the switch. However, a co-worker was talking up Coda and mentioned that it had auto-complete – a feature that has been sorely lacking from BBEdit.

The switch turned out to be fairly painless. One feature that I really missed from BBEdit was the ability to create or open files from the command line simply by invoking the following

$: bbedit -c myFileToOpen

Sadly, Coda didn’t appear to have the same functionality. I contacted the Coda team, they liked the idea and planned to look into it, but I am way too impatient to wait. In the meantime, I decided to write my own little bash script that I named ‘coda’

#! /bin/bash 
if [ "$1" = "" ]; then
	echo "Please specify a file to open or create"
	exit 0
else
	for ARG in $*
		do
    	            touch -a $ARG && open -a Coda $ARG 
		done
	exit 0
fi

By using touch, I am able to create a file if it doesn’t exist. I pass the ‘a’ argument, but not the ‘m’ argument so the timestamp doesn’t change for already existing files. After touch works its magic, the open command opens the file in Coda. I am running a for loop over the list of arguments in order to allow a list of files or a wildcard as well as a single file to be passed into the script. Sweetness! I threw this little script in my bin folder and set it to executable using chmod +x. Using the script looks like this

$: coda myFileToOpenOrCreate
$: coda *.txt

Installing Git on Mac OSX 10.5

I was able to find an excellent tutorial for installing Git on a Mac over at Tim Dysinger’s blog, here. I highly recommend it. It worked like a charm for me. The one item I did change was grabbing a slightly more recent tarball than Tim has in his instructions. I grabbed git-1.5.6.4.tar.bz2 instead of git-1.5.5.tar.bz2. Otherwise, it works great. More on Git and it’s integration with SVN and Eclipse coming soon, stay tuned.

The one caveat, you must have Xcode (Developer Tool Box for Mac) installed in order to compile Git from source. You can find the Xcode DMG here, you will need an Apple Id – don’t worry it’s free.

A Robust FFMPEG Install for CentOS 5

A little while back, I posted a basic install of ffmpeg for CentOS 5. After working with that build over the last month, I found I needed to expand it to include many different video codecs. Because of licensing restrictions, binaries of the build cannot be distributed. However, instructions for a similar build can! I spent several days (or more) researching the packages I would need and I must have combed dozens of blogs to find this information. The build I made strips sound from videos [See update at the end of this post for more information on this subject], this decreases the final file size. If you need sound, you will not want to follow these steps exactly. I also stripped ffserver and ffplay from my build.

These packages were executed in the following order. I cannot say for certain this exact order must be followed, it was the order in which ffmpeg threw the errors. This tutorial assumes you have wget, bzip2, tar and subversion installed on your system. If you do not, please find and install the lastest versions of these utilities for CentOS 5. Additionally, I have allowed all packages to install into their default directories, typically this is /usr/local/[bin | lib]. If at any point during the process of a ‘make’ you run into errors, be sure and run ‘make clean’ before running ‘make’ again. You will either need root access of su access to install ffmpeg. The marker ‘codec:$’ is my prompt, it is merely to indicate separate commands.

Getting Started

codec:$
codec:$ mkdir -p ./tmp/ffmpeg-packages
codec:$ cd ./tmp/ffmpeg-packages

Installing a52

codec:$ wget http://liba52.sourceforge.net/files/a52dec-0.7.4.tar.gz
codec:$ tar -zxf a52dec-0.7.4.tar.gz
codec:$ cd a52dec-0.7.4
codec:$ ./configure --enable-shared=PKGS
codec:$ make && make install
codec:$ cd ..

Installing FAAD2

codec:$ wget http://downloads.sourceforge.net/faac/faad2-2.6.1.tar.gz
codec:$ tar zxf faad2-2.6.1.tar.gz
codec:$ cd faad2
codec:$ autoreconf -vif
codec:$ ./configure --disable-drm --disable-mpeg4ip
codec:$ make && make install
codec:$ cd ..

Installing FAAC

codec:$ wget http://downloads.sourceforge.net/faac/faac-1.26.tar.gz
codec:$ tar zxfv faac-1.26.tar.gz
codec:$ cd faac
codec:$ ./bootstrap
codec:$ ./configure --disable-mp4v2
codec:$ make && make install
codec:$ cd ..

Installing LAME

codec:$ wget \
http://superb-east.dl.sourceforge.net/sourceforge/lame/\
lame-3.98b8.tar.gz
codec:$ tar zxfv lame-3.98b8.tar.gz
codec:$ cd lame-3.98b8
codec:$ ./configure
codec:$ make && make install
codec:$ cd ..

Installing yasm

YASM is a modular assembler, it is required by the x264 package.

codec:$ wget \
http://www.tortall.net/projects/yasm/releases/yasm-0.7.0.tar.gz
codec:$ tar zfvx yasm-0.7.0.tar.gz
codec:$ cd yasm-0.7.0
codec:$ ./configure
codec:$ make && make install
codec:$ cd ..

Installing x264
The x264 package is under git revision control, which is much like CVS or SVN. Thankfully, they provide daily tarballs. I grabbed this one:

codec:$ wget \
ftp://ftp.videolan.org:21//pub/videolan/x264/snapshots/\
x264-snapshot-20080513-2245.tar.bz2

I just did a little exploring via an FTP program to find the snapshot I was after. If you are feeling adventurous, download git and try checking out the latest version from their repository.

codec:$ bzip2 -d x264-snapshot-20080513-2245.tar.bz2
codec:$ tar xfv x264-snapshot-20080513-2245.tar
codec:$ cd x264-snapshot-20080513-2245
codec:$ ./configure --enable-mp4-output --enable-shared --enable-pthread
codec:$ make && make install
codec:$ cd ..

Installing Xvid

codec:$ wget http://downloads.xvid.org/downloads/xvidcore-1.1.3.tar.gz
codec:$ tar zxfv xvidcore-1.1.3.tar.gz
codec:$ cd xvidcore-1.1.3/build/generic
codec:$ ./configure
codec:$ make && make install
codec:$ cd ../../..

Installing libraw1394

codec:$ wget http://www.linux1394.org/dl/libraw1394-1.3.0.tar.gz
codec:$ tar zxfv libraw1394-1.3.0.tar.gz
codec:$ cd libraw1394-1.3.0
codec:$ ./configure
codec:$ make dev
codec:$ make && make install
codec:$ cd ..

Installing libdc1394

This project requires libraw1394, you must build it first. This project is a little confusing there is both a libdc1394 and a libdc1394-2. I have only installed the former. Visit Sourceforge here. I grabbed the tarball here:

codec:$ wget \
http://superb-west.dl.sourceforge.net/sourceforge/libdc1394/\
libdc1394-1.2.2.tar.gz
codec:$ tar zxfv libdc1394-1.2.2.tar.gz
codec:$ cd libdc1394-1.2.2
codec:$ ./configure
codec:$ make && make install
codec:$ cd ..

Installing ffmpeg

For FFMPEG, you will need to get the latest out of SVN. FFMPEG doesn’t ever make releases. To do so, run:

codec:$ svn checkout svn://svn.mplayerhq.hu/ffmpeg/trunk ffmpeg
codec:$ cd ffmpeg
codec:$ ./configure --enable-gpl  --enable-postproc --enable-nonfree --enable-postproc --enable-libfaad --enable-swscale --enable-avfilter --enable-pthreads --enable-libxvid --enable-libx264 --enable-libmp3lame --enable-libdc1394 --enable-liba52 --enable-libfaac --disable-ffserver --disable-ffplay
codec:$ make
codec:$ make install

It usually takes at least 5 mins for ‘make’ to run, be sure you have a good file before attempting to install. You will need to be sure and run the ldconfig setting if you have to re-compile, otherwise ffmpeg will throw an error that it cannot find library files. Here is a good resource for using ffmpeg.

Post-Installation Actions

After the install in complete, you may need to add the /usr/local/lib directory to your ld.so.config file. Do the following:

codec:$ cd /etc/
codec:$ cd ld.so.conf.d
codec:$ vi ffmpeg.conf

You need to add ‘/usr/local/lib’ [sans-quotes] to this file and save it. Press Esc, :, x to do so in vi. Afterwords you need to run the following from the commandline

codec:$ ldconfig

That’s it, you now have a fairly robust build of ffmpeg that can take almost anything you can throw at it. Sadly, licensing restrictions prevent binaries from being made of the above steps. Happy (video) encoding!

Update: This build doesn’t actually strip sound from videos, I am doing that with a flag when I run ffmpeg, however, it should be noted that all the audio codecs you may find in many common video formats may not be supported in this build of ffmpeg, which is why I am choosing to strip audio entirely from videos.