AnalogClock






Showing posts with label howto. Show all posts
Showing posts with label howto. Show all posts

Friday, August 6, 2010

Bit Flipping in Python

In Python you can access bits directly with the &, |, ^, ~, <<, >> operators on integer objects.

An example bit transformation might be one that moves all of the set bits in a 12 bit number to the left and all of the unset bits in the same 12 bit number to the right.

For example the 12 bit number 101101000001 would be transformed into the 12 bit number 111110000000 using that transformation.

This example bit manipulation can be concisely written in Python as:

n1 = int("101101000001", 2)
bc = sum(1 for i in range(12) if n1 & (1 << i))
n2 = int(("1" * bc) + ("0" * (12 - bc)))

In Python 2.6 and later this example can more concisely be written as:

n1 = 0b101101000001
bc = sum(1 for i in range(12) if n1 & (1 << i))
n2 = sum(1 << i for i in range(11, (11 - bc), -1))

The generic 12 bit manipulation can be more re-usably written as:

# The 12 bits
b12 = tuple(1 << i for i in range(11, -1, -1))

# There are only 13 solutions
result_numbers = tuple(sum(b12[i] for i in range(c)) for c in range(13))
result_strings = tuple(("1" * i) + ("0" * (12 - i)) for i in range(13))

original_number = int("101101000001", 2)
bit_count = sum(1 for i in range(12) if original_number & b12[i])
transformed_number = result_numbers[bit_count]
print result_strings[bit_count]

The Python 2.6 and later version is:

# The 12 bits
b12 = tuple(1 << i for i in range(11, -1, -1))

# There are only 13 solutions
result_numbers = tuple(sum(b12[i] for i in range(c)) for c in range(13))
result_strings = tuple(bin(i) for i in result_numbers)

original_number = 0b101101000001
bit_count = sum(1 for i in range(12) if original_number & b12[i])
# Updated 2011-01-08
# in python 2.7 and later the above can be further optimized
# using range(original_number.bit_length) in place of range(12)
transformed_number = result_numbers[bit_count]
print result_strings[bit_count]

There is no need to define a function call or set of function calls. Bit manipulation easily lends itself to mapping list indexes to result sets that can be prebuilt to aid in readability and execution speed.

Wednesday, February 17, 2010

python no overhead debugger on demand

One of the best ad-hock troubleshooting ways to figure out what caused an exception post mortem.

Run:

python -i your_script.py

After the exception you will be left at the python interpreter prompt staring at your exception.

Type:

import pdb
pdb.pm()

And you will be put into the context of the stack of the last exception. This means you can print/examine the local variables at the point of failure after the failure has occurred without having to change a line of code or import pdb in advance.

Another method that requires a little bit of preparation is to put the import pdb and pdb.set_trace() in a signal handler trap. *Moved the import of pdb out of the signal handler as imports in functions are deprecated. That way you can do a kill -SIGNAL PID or if the signal you trap is INT you can just Ctrl-C to get dropped into the debugger at any point. The signal handler technique is good for testing release candidates and released code because the signal handler doesn't create any runtime overhead.

Signal handler example. Debugger starts with Ctrl-C:

import pdb
import signal


def int_handler(signal, frame):
  pdb.set_trace()


if __name__ == "__main__":
  signal.signal(signal.SIGINT, int_handler)

Put that at the top of your script and you can start debugging your script at any point by typing Ctrl-C. Resume programe execution by typing exit at the Pdb prompt.

Thursday, April 2, 2009

Step One: Insert Foot in Mouth

This is a follow up to my Your Fancy Language is Burdensome and Dangerous post.

Python and Ruby don't have the APL array/vector/matrix syntax built in to the core language. But a little nudge from someone made me do the little bit of research I should have done before I wrote that post. And I rediscovered that I did not originate the idea that these languages need array/matrix math syntax thought.

There is a gem for Ruby called NArray and an egg for Python called Numpy that not only add array syntax but also coherent coercion rules and the ability to apply functions across arrays without calling map. I'm glad I found it because they are easy as pie to use and do exactly what I wanted and a heck of a lot more.

Check out this Python example:

from numpy import *
a = array([1,2,3])
b = array([4,5,6])
print a+b #equals array([5,7,9])
print (a+b)/4
print (a+b)%6

Outputs
[5,7,9]
[1,1,2]
[5,1,3]

As you can see Numpy also replicates the integer type bug/feature which is very C like but is being dropped in Python 3000 in favor of coercing integers to float on divide. You can typecast the divisor to float to get floating point division and float results.

Tuesday, June 17, 2008

ffmpeg with h.264 3gp/amr xvid lame/mp3 a52/ac3 dts aac

I was going through the source for mythweb to find a good spot to insert code to setup and return VLC VOD links instead of simple http streams.  I followed the code for flash video .flv on the fly conversion and streaming with http range based seeking.  I was surpised to find a very simple and elegant ffmpeg based  CGI script doing all the work.  In /var/www/mythweb/modules/stream/handler.pl I found the following:

elsif ($ENV{'REQUEST_URI'} =~ /\.flv$/i) {
# Print the movie
$ffmpeg_pid = open(DATA,
"$ffmpeg -y -i ".shell_escape($filename)
.' -s '.shell_escape("${width}x$height")
.' -r 24 -f flv -ac 2 -ar 11025
.' -ab '.shell_escape("${abitrate}k")
.' -b '.shell_escape("${vbitrate}k")
.' /dev/stdout 2>/dev/null |'
);
unless ($ffmpeg_pid) {
print header(),
"Can't do ffmpeg: $!";
exit;
}
print header(-type => 'video/x-flv');
my $buffer;
while (read DATA, $buffer, 262144) {
print $buffer;
}
close DATA;
exit;
}

Pretty cool. I'm going to use this as a template for generating VLC VOD media on the fly using netcat. More to follow.

Anyway what happened is that I realized the flash video was working great but there was no audio. Playing with ffmpeg on the command line using the parameters above clued me in. ffmpeg didn't know how to decode AC3 and it also didn't know how to encode mp3.

I was surpised to find that the Medibuntu ffmpeg was missing some codecs that I know ffmpeg supports. I looked around the web for a way to compile ffmpeg with the missing codecs.

I found a bunch of apt-get source ffmpeg style how to's and figured out the DEBIAN_BUILD_OPTIONS="risky" part.

Had to install fakeroot to make the source by hand.

Another site had the details for getting AMR/3GP/3GPP/3GP2 AKA AMR narrow band and wide band. Found the how to for ffmpeg for Debian at this page. And the HOWTO with AMR at this page

I performed the following:

sudo apt-get build-dep ffmpeg
sudo aptitude install fakeroot liblame-dev liba52-0.7.4 liba52-dev libdts-dev libfaac0 libfaac0-dev libfaad0 libfaad-dev libxvidcore4 libxvidcore4-dev libx264-57 libx264-dev
apt-get source ffmpeg
wget http://www.3gpp.org/ftp/Specs/archive/26_series/26.104/26104-510.zip
wget http://www.3gpp.org/ftp/Specs/archive/26_series/26.204/26204-510.zip
unzip -q 26104-510.zip
unzip -q 26204-510.zip
cd ffmpeg-0.cvs20070307
mkdir libavcodec/amr_float
mkdir libavcodec/amrwb_float
cd libavcodec/amr_float
unzip -q ../../../26104-510_ANSI_C_source_code.zip
cd ../../libavcodev/amrwb_float
unzip -q ../../../26204-510_ANSI-C_source_code.zip
cd ../..

Ended up changing the following section in ffmpeg-0.cvs20070307/debian/rules in the source directory from this:

confflags += --enable-gpl --enable-pp --enable-swscaler --enable-pthreads
confflags += --enable-libvorbis --enable-libtheora --enable-libogg --enable-libgsm

to this:

confflags += --enable-gpl --enable-pp --enable-swscaler --enable-pthreads
confflags += --enable-libvorbis --enable-libtheora --enable-libogg --enable-libg
confflags += --enable-x264 --enable-liba52 --enable-libdts --enable-amr_nb
confflags += --enable-amr_wb

And finally I performed the following in the extracted source directory:

DEBIAN_BUILD_OPTIONS="risky" fakeroot debian/rules binary
cd ..
sudo dpkg -i *.deb

Blogged with the Flock Browser

Saturday, May 3, 2008

Ubuntu or Kubuntu on a IBM Thinkpad T21 or T22 Video Driver Fix

I recently upgraded to Ubuntu 8.04 Hardy Heron and experienced problems with Xorg and my savage video card.  Traced it back to the AGPSize and AGPMode options for the savage driver.  Now have working 3D acceleration aka DRI and AIGLX.  I've been missing that since the upgrade to version 7.xx.  Turns out AGPSize defaults to a 16MB window which causes a lockup or failure on my IBM Thinkpad T21 and T22 with acceleration turned on.  The Thinkpads have only 8MB video ram.

FYI I actually use Kubuntu but I upgraded all my Kubuntu systems just fine using the Ubuntu CD.

The working, 3D accelerated xorg.conf from my IBM Thinkpad T22:

# xorg.conf (X.Org X Window System server configuration file)
#
# This file was generated by dexconf, the Debian X Configuration tool, using
# values from the debconf database.
#
# Edit this file with caution, and see the xorg.conf manual page.
# (Type "man xorg.conf" at the shell prompt.)
#
# This file is automatically updated on xserver-xorg package upgrades *only*
# if it has not been modified since the last upgrade of the xserver-xorg
# package.
#
# If you have edited this file but would like it to be automatically updated
# again, run the following command:
#   sudo dpkg-reconfigure -phigh xserver-xorg

Section "InputDevice"
    Identifier    "Generic Keyboard"
    Driver        "kbd"
    Option        "XkbRules"    "xorg"
    Option        "XkbModel"    "pc101"
    Option        "XkbLayout"    "us"
    Option        "XkbOptions"    "lv3:ralt_switch"
EndSection

Section "InputDevice"
    Identifier    "Configured Mouse"
    Driver        "mouse"
    Option        "CorePointer"
    Option        "EmulateWheel"        "true"
    Option        "EmulateWheelButton"    "2"
    Option        "ZAxisMapping"        "4 5 8 9"
EndSection

Section "InputDevice"
    Identifier    "Synaptics Touchpad"
    Driver        "synaptics"
    Option        "SendCoreEvents"    "true"
    Option        "Device"        "/dev/psaux"
    Option        "Protocol"        "auto-dev"
    Option        "HorizEdgeScroll"    "0"
EndSection

Section "Device"
    Identifier    "Configured Video Device"
    Option        "AGPMode"        "2"
    Option        "AGPSize"        "8"
    #Option        "ShadowStatus"        "true"
    #Option        "ShadowFB"        "true"
    #Option        "NoAccel"        "true"
EndSection

Section "Monitor"
    Identifier    "Configured Monitor"
EndSection

Section "Screen"
    Identifier    "Default Screen"
    Monitor        "Configured Monitor"
    Device        "Configured Video Device"
    DefaultDepth    16
EndSection

Section "ServerLayout"
    Identifier    "Default Layout"
    Screen        "Default Screen"
    #InputDevice    "Synaptics Touchpad"
EndSection


Blogged with the Flock Browser

Sunday, March 9, 2008

sudoers on Mac OS X

In my last post I refer to a lot of commands that require superuser privileges without prefacing them with sudo. That is how I type the commands in.

So how do I do it? I don't login as a user with administrator privileges and I don't use su. I have a separate account with administrator privileges that I don't login to that I use as an unprivileged user in the username and password prompt to perform activities that require superuser privileges.

I used to su to the account with administrative privileges to run sudo commands. Until I educated myself a little bit by "man sudoers".

You should always use visudo to edit the /etc/sudoers file it checks the syntax of the file before committing your changes. Saving you from yourself by preventing you from thinking you have correctly made changes to the /etc/sudoers file when all you have done is locked out all your accounts with administrative privileges from performing any administrative tasks. visudo will notify you of an error and give you a chance to fix the problem or abandon your changes when you exit your editor before it commits your changes. If you decide to fix the problem visudo will relaunch your editor with the changes you made intact. It is up to you to decide whether to try repeatedly to get your syntax right or abandon your changes. I locked myself out of sudo on my Mac years ago by directly updating /etc/sudoers. I think I recovered from it by rebooting or fixing file permissions. Save yourself some time by always use visudo.

There are a couple of command line options to sudo that are very su like but stick to sudo's use your own password metaphor.

The "-s" command line option launches the user's own shell with superuser privileges. There are many options you can set that control what and whether any environment variables are carried over from the user's environment. The environment can also be cleared of certain environment variables and the forced setting of environment variables can be specified. In order to use the "-s" command line option to sudo you have to add the user's shell command to the list of executable programs. Which will usually be /bin/bash.

The "-i" command line option launches the root user's login environment. You can use an option to sudo to switch to a different user. You have to add the root user's shell to the allowed executable programs with the full path, /bin/sh.

By using either of these command line options to sudo you avoid having to repeatedly type sudo before every command. I also avoid having to add my everyday user to the administration group by adding just the commands I need for one liner's like port gem while still having a backdoor to any command through the shell command. There are a couple of installation programs that refuse to work with my setup. But they are easily fixed by logging in to my administration account to install them. The need to do that has become less over time.

Friday, March 7, 2008

Rails Requires RubyGems >= 0.9.4 Error on Mac OS X

Updated 2008-10-08:
Please note that I forgot to mention in this post the fact that Ruby on Rails is installed by default with OS X Leopard. This blog is most helpful to those who had previously installed rails using MacPorts and now want to move to the Apple supported version that comes with OS X Leopard.

I was following along with the Apple Developer Connection article that shows how to update and use the version of Ruby on Rails that comes with Mac OS X Leopard. I followed through the article without a hitch. Successfully creating the Events Rails Site and viewing it at localhost:3000. The only change I made outside of the article's instructions was to add the script/generate and script/console commands to the same place as the article recommends the script/server. I figured keeping the original data model definition commands with the rails project would help if I wanted to make any changes to the data model later. I moved the data model definition commands to a shell script located in the top directory of the project. Which will make a nice portable design convention. I have seen that method mentioned in a blog post before.

While I was browsing around the running instance of the Events rails site I decided to do a full update of all the gem packages on my system by way of "gem update" without the "--system". Big mistake or so I thought when I tried to start my own rails project a day later. I couldn't even get the generate commands to run. The response from every script/ command was the same "Rails requires RubyGems >= 0.9.4. I googled around for the solution which always seemed to say remove the Leopard version and install the MacPorts version. Which I didn't want to do since I was trying to get rid of the duplicate MacPorts packages in the first place. In fact I had thought I had removed all the ruby packages from MacPorts. But on reinspection I found 4 versions of Ruby installed and one active a la "port search ruby and installed". First I did a "port uninstall ruby and inactive" to get rid of the old versions. Then I found a ruby gem I had somehow missed when I had done a "port uninstall rb-* and installed". After removing the lone gem I got rid of the installed and active version of Ruby via "port uninstall ruby". A quick verification of /opt/local/bin verfied that there was no longer a ruby or gem or update_rubygems command left.

Of course I left out the part where I removed all of the manually installed gems under /opt/local/lib/ruby/1.8/rubygems by "rm -fr / opt/local/lib/ruby/1.8/rubygems". Don't quote me on that path as I am writing this post on my iPhone and recalling it from memory. I only know the path because I also reinstalled rubygems manually before I figured out what the real problem was. I would probably have missed the real problem if I hadn't installed rubygems with "ruby setup.rb" after downloading it. The install commands showed RubyGems beings installed to /opt/local/... I had to remove that copy of RubyGems manually. Which wasn't so hard. All the install commands with the full paths of the installed files were still in my scrollback buffer.

I also didn't mention that I had uninstalled all my gem packages from the Leopard side of things. Including the original packages under /System. Luckily one of the articles I'd googled had the complete list. It was easy enough to do a "gem install rails sqlite-ruby capistrano mongrel libxml-ruby ruby-openid ruby-yadis rubynode RedCloth sources ferret acts_as_ferret fcgi termios cgi_multipart_eof_fix daemons dnssd gem_plugin hpricot needle actionwebservice activesupport". The full list of preinstalled gem packages is available here. I abbreviated my gem install command to only include what is necessary as installing rails pulls in some of the other packages in the list like activerecord and rake.
Everything worked like a charm after that. Now on to writing that SFA app. After that I'll have to get back to my holy war to remove redundant MacPorts packages. I think I was up to removing those two verions of Python 2.4 and 2.5. Seeing as how Leopard ships with 2.5 and 2.4 was a buggy beast at best when compared to the speed and bug fixes of version 2.5.

Monday, February 18, 2008

Fix for iPhone not syncing some video

I use the iPhone. Before upgrading iTunes and the iPhone firmware I was able to load video on the iPhone using iTunes. Now I have problems getting the same video to sync to the iPhone. I also have problems getting some video podcasts to sync to the iPhone that used to sync just fine.

I did some research using ffmpeg -i on some of the files that synced and some of the ones that wouldn't sync. I found that the video that wouldn't sync all had something in common. The aspect ratio was showing up as 0:1 (for DAR and PAR).

I googled some iPhone video conversion parameters and found an ffmpeg parameter that gets any video to sync, -aspect 16:9. Now I just need to come up with a set of ffmpeg video conversion parameters that play for more than a few seconds before the audio drops out and the video stops playing.

Gonna try doing an ffmpeg copy and report back the results.

Update 2008-03-08

I found out a lot. The 0.49 version of ffmpeg that comes with ffmpegX works fine. It was either patched to deal with the QuickTime mpeg4 issues or uses default setting that work with QuickTime. Turns out QuickTime needs square pixels for mpeg4 and the harddup video filter to avoid a frame copy problem when it converts to the mpeg4 format as the duplicate frame flag gets lost in the translation. I haven't figured out how to do this with ffmpeg. But all the info on this issue is dealt with in a section of the mencorder documentation under the heading QuickTime. Mencoder uses ffmpeg under the "-lavc" flag.

I'll update with the details after I perform some tests.

Saturday, June 23, 2007

Common Gentoo Problem with Solution

In Gentoo GNU/Linux you usually compile all of your packages. It takes a little time for the little packages and a lot of time for the big ones. Much more time than a simple downloaded binary package on apt and rpm based systems.

This is a good thing. The benefits are customization of every package on the system, less cruft, and a system optimized from top to bottom.



It also results in the problem highlighted above. I recently upgraded my system to the latest profile and emerge -u world. Apparently some of the packages like digikam either haven't been upgraded recently or a package that was depended on by digikam was upgraded more recently.

Either way the way to fix the highlighted problem is to emerge digikam again. That way it will point to the newer libraries it depends on. And probably break anything that depends on digikam's libraries in the process. But that in a nut shell is Gentoo.



Hmm? What's this nfs USE flag that is disabled?



euse -i nfs shows the flag effects a bunch of packages. I'm going to enable it globally because I use nfs automount for file sharing between my Kubuntu, Gentoo, and OS X boxes. The more nfs support the better.



Now I can get back to uploading my albums. :)

FYI