Wednesday, January 22, 2014

I like ipython notebooks

I have been stretching my programming chops by working on the problems at http://projecteuler.net

I like mathematics. At a glance the exercises at http://projecteuler.net seemed very interesting. There are network walks and problems involving primes and other number types.

I used the typical development cycle at first. Write a script kill it if it runs too long. Optimize and repeat until I have a solution that runs in a reasonable amount of time. Then submit the solution. If it doesn't check out then write an alternate version by altering and trying to correct any perceived deficiencies. Check the solution against the generated solution and repeat.

In my idle time I worked these problems using this process mostly in a terminal on my personal server either in Emacs or in my alternate environment on my phone.

I had recently reacquainted myself with several programming languages I had encountered in the past: assembly, forth, aplus, j-console, lisp/scheme, prolog, java, ruby, JavaScript, coffeescript. And had also learned a new language or two that I wanted to get comfortable with: Haskell, f-sharp, Julia.

I figured I'd write a solution for each of the problems in each of the languages. I setup the environments for each and started with Haskell. What started out as a simple learning experience quickly proved to be more of a brain teaser than I expected. Learning Haskell while simultaneously stretching my programming chops was more work than I had expected.

After solving the first couple of problems in Haskell I switched to solving the first couple of problems in prolog. On the first prolog problem I ended up looking up a solution on Wikipedia that I reworked after reading and searching and understanding more and trying again. In prolog I am a neophyte at best and to call me a programmer would be generous.

When I got my prolog code to work and looked up the next problem it dawned on me that by solving these problems first in a language I am more comfortable in and am considered an expert in I might have a better chance of finishing a significant portion of the over 400 and growing list of problems presented at http://projecteuler.net and then I could attack the solutions in other languages from the point of knowing the solution in another language.

Python is one of my stronger languages. I know how to program in the functional style and in the OOP style. Which would be useful for thinking of solutions in some of the other languages. I started down that road with SL4A on my phone and python in a shell on my server with a pinch of git.

When I hit Wikipedia to lookup triangle numbers it was a real eye opener and led me to back to prime number theory to optimize a solution to a problem I had previously successfully solved involving primes. In that process and to my benefit I had created a virtualenv with a more recent version of ipython. Which includes notebook.

The command 'ipython notebook' in the the directory and virtualenv of your choosing is all it takes to get started. And I was hooked after I solved my my first http://projecteuler.net problem using a notebook. I teared through 5 or six more problems in less time than it took to solve one using traditional editing run cycles. And the benefit of a running shell environment that can be used for  documentation and has individual cells that only have to be evaluated if you want to add them to the solution makes for a very iterative development cycle.

I feel like I can finally keep a train of thought and I can feel my approach to coding at work improving despite the difference in environments.

I sidetracked myself last weekend and to get my head around the clusters tab of a notebook. The ease of launching clusters has me crashing running processes on my tiny micro AWS instance server with only 4 clusters. I resolved to drive farther off track and write some code in my project Euler notebook for spinning up and down temporary clusters using AWS big compute instances for brief period of time to divvy up some of the tougher problems. By writing solutions that are easy to process in parallel I could properly figure out solutions in short order.

Being able to start a solution and continue work on it in parts anywhere from any device is much more accessible than using ssh and a terminal. You have to try it to get it. Working on code in a browser seems like a toy at first. But then you throw some real problems at it and you begin to realize there is something to this style of programming. And then you shutdown your notebook kernel. And then you come back to it and realize you haven't lost anything. Everything you've typed is still there waiting to be evaluated or edited and reevaluated.

And should you get curious and ssh to your server and type 'ipython console --existing' you too will start to understand the power of ipython notebooks.

I know now that I will be using ipython notebooks for all the languages I plan on tackling. After I finish up to problem 15 in order I will start with ruby and Julia. Both languages are the natural next choice because they already have working iruby and ijulia solutions that have ipython notebooks built in.

I'll have to sidetrack again after getting up to problem 15 in each as I'll have to implement an ipython notebook extension to each of the other languages before I start to work on the problems in that language. The tight iterative train of thought development cycle of ipython notebooks is too useful to not use when I am a programmer and can plugin the solution. Which is the point of the entire exercise.

Tuesday, February 12, 2013

Crushing Python

I've begun thinking in Python a lot lately. ~/.mypyrc.py:
# standard library
from collections import defaultdict as DD,namedtuple as NT
import csv
from datetime import date as D,datetime as DT,time as T,timedelta as TD
import operator as op
import os
import re
from cStringIO import StringIO as SIO
import sys
import time
from urlllib2 import build_opener as BO,install_opener as IO,Request as R,urlopen as UO
from urlparse import urljoin as UJ,urlparse as UP
from zipfile import ZipFile as ZF
# third party libraries
from BeautifulSoup import BeautifulSoup as BS,BeautifulStoneSoup as BSS
import pytz
# initialization
IO(BO())
~/.profile or ~/.bash_profile:
PYTHONSTARTUP=~/.mypyrc.py
export PYTHONSTARTUP
Now starting python interactively loads the PYTHONSTARTUP file automagicaly:
python
>>> for f in ZF('a.zip').namelist:
...  print f
... 
>>> print BS(UO(R('http://google.com'))).find('title').text
>>> cols=zip(*csv.reader(open('a.csv')))
>>> rows=zip(*cols)

pypad - the python notepad web server

The plan is to take this simple python script and expand on it's note taking features with a GUI, console, and command line front end and additional storage options including a tiddlywiki style of appending the data store to the script.

The web server will eventually be an option rather than integral.

http://launchpad.net/pypad

I will fork this into a micro framework and also fork it into a python eval type remote or local shell. Maybe even a REPL embedded in a web page.