Climate Change and NYC Flooding

 Climate Change and NYC Flooding

True story:

The west side highway and FDR drive used to be elevated. The elevated west side highway was an active railway with elevated exits for cars and trains right into buildings several avenues over into the city. Those were the exits and there were ramps down to the street. It was profitable for the city to destroy the elevated highways and railways  and sell the steel for the elevated highway and extension exits. So they did. It's an island why wouldn't you elevate everything including the subway. Which used to be elevated train lines. Separate from the railways and highways. All worked just fine even when the city flooded. Climate change is a catch all excuse for the den of thieves we call our representatives to ask for more money to steal instead of spending our tax dollars on us.

There was also a massive set of independently operated trolley systems that were connected from one sided of the country to the other and they honored each other's transfers coast to coast.  Trolley lines that dropped you off down the street from your house all over all the cities of the US. There are a couple of documentaries about it that you can watch for free. How and Why Big Oil Conquered the World.   

Corbett has produced tons of documentaries on his site that explain how the world really works and why things are the way they are. We don't have to waste or lives scratching our heads like idiots wondering why when we know our actual history and all the facts.

As a kid I loved riding in a car through buildings in Manhattan and riding in electric old timey cabs with extra fold up seats because of all the room in the back when a car doesn't waste all its space on an engine and gas tank. The electric trolley on 14th Street was a special treat.

vscode you should know

search files
ctrl+p
type ? into search bar for help for other types of search

search editor commands
ctrl+shift+p or f1 or ctrl+p >

move line or selection up/down
alt+up/down

duplicate line or selection above/below
alt+shift+up/down

peek definition
alt+f12
you can scroll and edit in the pop-up overlay

goto definition
f12

add cursor at end of line or lines of selection
ctrl+shift+i

add cursor above below
ctrl+alt+shift+up/down

ctrl+`
open and toggle terminal window

ctrl+shit+`
open a new terminal

more here

pycharm you should know

search everywhere - file names, commands names, and symbol names across project
press shift key twice

search for a pycharm command by name or abbreviation or parts
ctrl+shift+a
use a space to separate parts of a search(f s) file save (fs) should work too

search for a file in the project space
ctrl+shift+n

open a recent file - searchable
ctrl+e

switch an open file or a open a panel
ctrl+tab start from the top of the open files list
ctrl+shift+tab start from the bottom of the panels list

return focus to editor window
esc

return focus to editor window and close last active panel
shif+esc

quick documentation popup for current cursor or selection
ctrl+q

quick popup of source snippet for current cursor or selection
ctrl+shit+i

return focus to last
navigate to previous or next edit location across files
ctrl+alt+left ctrl+alt+right

add a cursor AKA multi-cursor
alt+left mouse click
alt+shift+insert toggle multi-cursor mode for shifted selection

add multiple cursors and  rectangle selection
alt+shift+left mouse click and drag
use same selection method for rectangle pasting

add an import automatically to the top of the file by selecting from a popupalt+enter
use after typing the usage (datetime.datetime or just datetime)

duplicate the current line or selection
ctrl+d

move the current line or selection up or down
alt-shift-up or alt-shift-down or ctrl-shift-up or ctrl-shift-down

navigate to class method or function in current file by selecting from a popup
ctrl+f12

debug the current file
shift+f9

step into
f7

step over
f8

run the current file
shift+f10

run the current line or selection in the console
alt+shift+e

vim you should know

If you use vim you should try these:

Use the file browser for local file management.

:browse e .

Equivalent to command line:

vim .

Use the file browser for remote file management.

:browse e scp://hostname/

Equivalent to command line:

vim scp://hostname/

Pipe to 'view -' instead of 'less' when you want syntax highlighting.

Side by side diff with editing

:e file_v1

:vert diffsplit file_v2

Equivalent to command line:

vimdiff -v file_v1 file_v2

The Plant Paradox by Dr Gundry

Updated 11/2/2019: Added more details and more tips and more foods.

Dr Gundry the author of the Plant Paradox has added a YouTube channel.

https://www.youtube.com/channel/UCtxo0nTZjzlKJq5-vJq6s6g

Video from Dr. Gundry that is very approachable and immediately usable food advice https://cutyourcravings.com/180927B.php?n=tba

Here is a summary of my understanding of The Plant Paradox by Dr Gundry (Yes, I keep notes and I need to update these. It's very complicated.):

The Plant Paradox is the book I recommend reading

https://www.goodreads.com/book/show/33630505-the-plant-paradox

Podcast and Blog

http://drgundry.com/gundry-podcast/

Shopping List on Amazon

https://gundrymd.com/plant-paradox-shopping-list/

Shopping List PDF

https://gundrymd.com/wp-content/pdf/Plant-Paradox-Shopping-LIst.pdf

Yes and No List

https://tantor.com/the-plant-paradox-steven-r-gundry-md.html

It's a lot of information. Even if you know a lot.  Dr. Gundry takes many chapters to put into perspective the reasons why you want to avoid certain foods.

Dr. Gundry goes off on many worthwhile tangents. I'm down a size from 2XL to XL and 40 waist to 38 from last weekend and that was in 3 or 4 days and it's not even a calorie or portion restriction diet except for the meat and cheese AKA animal protein.

I don't feel hungry most of the time now. Especially not for the poisonous foods.

The dietary restriction on foods to avoid is a cure all by avoiding the bad and indulging the good. We are all suffering because of all the bad things we eat and popular diets and health recommendations lead us in unscientific tail chasing circles with only partially correct info and loads of bad recommendations.

It is not just sugar. In fact refined sugar AKA glucose is less bad for you than the fructose found in fruit. The availability of fruit all year round that is picked green before the lectins (plant created poison proteins that kill bugs) can be removed by the plant make it even worse. The fructose in fruits causes the brain to send out insulin from the taste but the insulin receptors don't recognize the fructose as sugar (glucose). This causes more insulin to go out and you to feel hungrier. This insulin cycle causes insulin resistance and is triggered by fructose (fruit sugar), syrups(corn, maple, agave, etc), and all sugar-free sweeteners (including stevia).

1. Sugar in everything you eat satisfies your body before the proteins arrive and your body immediately turns the protein and carbohydrates into sugar and then stores the sugar as fat.

2. Red meat proteins are similar enough for most of our body to use including lining our blood vessels. The immune system sees these proteins as foreign and attacks the body. Attacking our blood vessels and muscles, and tendons causing pain because it doesn't recognize the protein. The prevalence of rheumatoid arthritis in relatively young and middle aged adults is partially a symptom of too much red meat.

3. Most of the cow milk in the world has the A1 protein. A protein that our bodies find indigestible. A2 protein cow milk and  also any sheep or goat dairy have the A2 protein and are digestible.

4. All legumes are indigestible including cucumbers.

5. All fruit seeds have lectins and are poisonous as are the skins.

6. WGA, wheat germ a gluten is is a lectin that blocks insulin receptors and estrogen receptors permanently while turning on those receptors permanently.

7. Lectins get their poison from bacteria that makes its way in through your gut and creates a leaky gut that allows worse things in. The bacteria makes a home in your body and your immune system kicks in 24/7 causing symptoms that we call diseases. Lupus, autoimmune, Parkinson's, heart, allergies, dementia, Alzheimer's, rheumatoid arthritis, headaches, etc

8. NSAIDS are poison that make you addicted to the poison that they are (Advil, etc).

9. Night shades are poisonous. Eggplant, tomatoes. Peel the skins and throw away the seeds if you want to eat them.

10. Cashews are poisonous because they are full of lectin proteins even after they are sun-dried. Don't eat a five pound bag of cashews. You might end up poisoned in the hospital.

11. All grains are poisonous including soy and all the other beans (legumes) including peanuts.

12. If it has seeds it is a fruit (fructose sugar) don't eat it, even pickles. All fruits are picked green and don't have a chance to remove the lectins from the fruit when they ripen after being picked.  

13. Don't eat sugar ever if you can avoid it. If you do need a sweetener use lemon juice or lime juice) as the sugar content is negligible. Sugar is the lesser of the evils between fructose (fruit juice) and any artificial sweeteners including stevia. Better a handful of candy made from refined sugar not corn syrup which is a fructose than to eat a fake sweetener as the sugar will satisfy your hunger while the others like fruit juice will set you up for insulin problems (diabetes) and make you hungrier after eating them.

14. If the animal was fed grains consider it a grain and don't eat it - grass fed meat only and wild caught seafood only. They feed grain (lectins) and fish parts (mercury) to farm raised fish.

15. Liquid coconut oil (MCT/XCT) and avocados have the ketogenic hormones that lets your cells' mitochondria burn fat instead of sugar, using ketosis, without having to starve yourself of sugar to start ketosis in your body. This is an inefficient way to generate energy for your brain. This is a good thing. Ketosis allows your mitochondria to exercise a muscle they don't get to use very often.

16. Green bananas, green plantains, and green papayas are good but not when ripe.

17. Yams and sweet potatoes are good but not potatoes.

18. Avoid sweeteners. They make you feel hungry. It's not sugar but it activates the release of insulin making you hungrier. Cut out all sweeteners completely if you have diabetes. That means no fruit and no sugar too. Eat yams and avocados and pour lots of liquid coconut oil on top of everything. Also eats greens every day (not string beans or peas as they are legumes).

19. Cancer can't do ketosis (burn fat). It needs sugar and uses the foreign red meat protein to hide from your immune system. Cut out red meat and sugar and sweeteners completely if you have cancer. Eat lots of leafy greens, yams, cabbage, green plantains, green bananas, and lots of tree nuts and extra-virgin cold-pressed oils. Eat in total each week only a fist sized portion of anchovies or sardines and maybe a couple of pasture raised chicken eggs for animal protein.

20. Eat lots of green leafy vegetables. They are a resistant starch that is good for you. They also provide a platform for good gut bacteria to grow on. Even iceberg lettuce is really good for your good gut bacteria. 
Green bean, snow peas, string beans, peas, beans, and peanuts are all legumes and they are bad for you unless pressure cooked. Cooking legumes for a really long time does not break down the lectins. Only pressure cooking. Lentils only take 15 minutes in a pressure cooker but kidney beans take 45 minutes.

21. Don't eat raw fish sushi you are taking a bigger risk than you may realize. There is a bacteria that is only found in raw fish that can kill you and even if you survive it may take you years to recover even partially. Don't eat tuna, salmon, swordfish. They are full of mercury. Basically avoid all big fish including cod, halibut, and farm raised fish. Good fish are anchovies, and sardines packed in extra virgin cold pressed olive oil. Pour extra virgin cold pressed olive oil over all your food. It is very good for you. Think of your food as a way to get more olive oil in you. Don't get the non-virgin non-cold pressed olive oil. The good stuff has been removed.

22. Avocados, mushrooms, walnuts, macadamia nuts, Brazil nuts, pistachio nuts, coconuts, hemp seeds, and almonds are good for you. Avoid most other nuts and seeds. Coconut milk is sugar water and worse than that it is fructose sugar water so no coconut water.

23. Cassava, yams, and sweet potatoes are good. Potatoes are bad.

24. Only eat gluten-free grain free bread. A simple rule of thumb to making gluten-free grain bread or cakes or pancakes is half coconut flour plus half almond flour(without the skins) or green banana flour and a tablespoon of cassava flour or tapioca starch flour. It works for pancakes too. For gluten-free grain-free pancakes leave the batter thick and 

25. Eat 12 to 14 tablespoons of olive oil a day. It's good for you. So are a handful of nuts as a snack. Not peanuts as they are really bad for you even cooked.

26. Have a teaspoon of mercury free fish oil before each meal and added over meals.
27. Have a tablespoon of cold pressed hemp seed oil every morning with a drop of liquid vitamin D3 for calcium absorption.
28. Cold brewed coffee is really good for you. To avoid getting jittery I have fresh ground organic cacao beans (chocolate) in boiling water (hot cacao) instead of coffee in the morning and if I don't feel hungry I don't eat until lunch. Use a coffee grinder to make a fine powder out of 3 to 5 beans. If you grind it for more than 5 minutes it will turn into melted chocolate. You can set it in a mold in a freezer to make a pure cacao chocolate bar.
29. Fasting is really good for you. Fasting is not starving yourself. Drink lots of Italian sparkling water and eat a handful of nuts or greens or yam or some other small healthy snack if you need to while fasting. It's OK. The point of fasting is to get your body to turn on it's natural recycling behavior. It requires 16 hours of fasting before this happens. You may have to work your way up to 16 hours in stages.
30. Listen to your body and learn the difference between being hungry and wanting to eat. 3 meals a day may be keeping you from being healthy. Only eat when you are hungry. Don't eat when you are not hungry.
31. These are facts that can help you make better food choices. This is not a weight loss diet. It is a get rid of the sickness and pain diet. You may loose weight if you are overweight. You may gain weight if you are underweight. If the food choices in front of you don't give you a good choice then pick the lesser of the evils using your newly acquired knowledge or just eat what you want. You body will let you know just how bad that choice is.
32. Watch out for feeling better and then eating bad. If you have been eating really good and then start to eat bad foods again it may take a week or two for you health to decline. That is a good time to get back on track by eating healthy again and avoiding the bad poisonous foods.
32. Moringa leaves have everything a body needs if you eat the whole leaf. You can buy it as a powder or leaf form and put it in anything as a supplement or use it as a tea in powder or leaf form just remember to eat the tea leaves after you finish the tea. The dried leaves can be boiled and eaten as a green or added to a soup or salad (after boiling or if fresh). Moringa contains all of the 13 amino acids that a body needs but can't produce on it's own. The one natural supplement to rule them all. The dried leaves can be quickly turned into powder in a coffee grinder. As a rule of thumb you should always be drinking the water it is boiled in. Think tea or ramen or soup or as an ingredient in a casserole or stew or curry.
33. Seaweed in all it's forms is really good for you. 
34. A really good noodle substitute for ramen noodles and linguine and spaghetti is yam noodles AKA sweet potato noodles. Here is a link to an Amazon search sweet+potatoe+noodles. 
35. Don't be afraid of salt with iodine. It's good for you.

Dr, Gundry explains how you can eat some of the foods on the no list by preparing them (pressure cooking, skinning, deseeding) properly but recommends avoiding them entirely.

It's a lot of info. Read the book - it presents it all and tears down popular diets and explains if they work and what parts work and why. It also explains what to avoid and why and what to eat and why. The book is more avoid and the example meals are in a downloadable PDF. It's not really a diet in the traditional sense. It's more of a handbook. It makes it easy to make the right food choices by eating the lesser of the evils even if you can't get the good stuff.

Personal update 11/2/2019:

I have lost all joint pain and muscle aches and headaches and even stopped grinding my teeth so much. I have lost over 20 pounds since last year August 2018 without any portion control. Which isn't saying much because I lost 10 pounds in the first week while being super strict and the next 10 a month after that. I think running once around the block in the morning before hopping in the shower and using liquid coconut oil as the creamer in my hot cacao and over most of my meals has been a big help. I am definitely a big eater but filling myself with yams and green plantains covered in olive oil and lemon or lime juice when I get cravings has been a big help. I definitely have not been strict as this is not a strict diet. I've got to order some goat or sheep milk ice cream to avoid eating the A1 protein in craving mode.

Understanding Javascript Prototype Inheritence

    MyBaseObj = function(a) {
        this.a = a
    }
    MyObjBase.prototype.aSquared = function() {
        return a ^ 2
    }
    MyObj1 = function(a, b) {
        MyBaseObj.call(this, a)
        this.b = b
    }
    MyObj1.prototype = Object.create(MyBaseObj.prototype)
    MyObj1.prototype.constructor = MyObj1
    MyObj2 = function(a, c) {
        MyBaseObj.call(this, a)
        this.c = c
    }
    MyObj2.prototype = Object.create(MyBaseObj.prototype)
    MyObj2.prototype.constructor = MyObj2

Understanding J

The j language is a terse APL derivative programming language that uses language like concepts.

noun

verb

adjective

adverb

etc.

There are also several programming styles. Parameterized functions, tacit functions, state machines, and many many ways of overloading verbs to get them to do what you want without having to use the full vocabulary of the language.

Tacit functions being my favorite style that I latched on to like a tick on a deer after I'd written my first couple of tacit functions.

Tacit functions are all about changing the meaning of verbs based on a chaining order that determines whether a verb is monadic(one parameter passed on the left or right) or dyadic(left and right parameters) and whether one or both of those parameters are the result of the previous verb or verbs or the original or outer parameters or a combination of the two. Basically a repeating set of 2, a hook, or 3, a fork, verbs in a chain with one verb being the original behavior. More on that here http://www.jsoftware.com/help/learning/contents.htm

So, any who, my big breakthrough moment was when I looked at an example adverb definition and it struck me that my love of terse and tacit function definitions was keeping me from being as productive as I could be in j and thereby keeping me from learning more through experience than I can through pouring through the manuals for the perfect way to cut a character off of a tacit expression.

After my big breakthrough I dug in with gusto and quickly and easily rewrote some j code I'd written  a couple of years ago when I was first learning j. The lesson I have learned is that sometimes you have to use a few more characters to get things done in j. That is just the way it is most of the time.

That being said I still look forward to the next time I have an insight that let's me shave a few characters off of an expression but I will no longer try to force it. I am still a little bit leary of using explicit functions unless it is in order to learn how to write adverbs in j.

Python - exception avoidance

m1 = []

m2 = None

m1_total = m1 and max(m1) or None

m2_total = m2 and min(m2) or 0

J and Python Tips

J

$: self reference for recursion etc.

Have to experiment with it

Python

Use seq[:1] and seq[-1:] first and last instead of seq[0] and seq[-1] to avoid index out of range exception for empty sequences

Generally, for all index n use seq[n:n+1] instead of seq[n] to avoid exceptions without having to add extra logic checks

>>> [][3:4]

[]

j

What a difference a month of using an APL language variant, A Programming Language. I learned a bit more about ipython and the %%script cell command when I started playing with j in a ipython notebook. It seems like I can get by a problem at a time writing scripts in j and running  them and organizing them in an ipython notebook. I have been able to one up ipython by working in j-console and using Droidedit on my Android and Textpad on my iPad Mini.
Developing on my phone in J using the j-console app for Android has changed my mind about going much further down the path of an expanded language learning and improvement experiment. I've decided to stick to perfecting my j skills for the foreseeable future. J's super optimized algorithms and terse formulaic functional programming is exactly what I was looking for in a programing language.
I came to this realization after I reimplemented the first few problems from projecteuler.net in j.
For more check out the history of APL

Edit. oops, forgot to finish this post. I am now a half year into my j language learnings and have finished the first 17 problems in order at http://projecteuler.net

Go checkout j for yourself. The primer is the fastest way to get competent in j. http://jsoftware.com

Here is a search and replace I put together after talking about examining the efficiency and correctness/completenes/wordiness of the search and replace in the standard library for j. Next step profile this against the standard library verb and actually read through the code and add any missing features.

rep.ijs

NB. rep file contains srep verb

NB. original working concept

; 'xyz'&, each (#'abc')&}. each(('abc' E. 'abcdefabcghi') <;. 1 'abcdefabcghi')

NB. srep verb

srep =: 4 : '; (> {:x)&, each (#>{.x)&}. each ((>{.x) E. y) <;. 1 y'

smoutput 'example usage (''abc'';''xyz'') srep ''abc123abc456'''

smoutput ('abc';'xyz') srep 'abc123abc456'

I like ipython notebooks

I have been stretching my programming chops by working on the problems at http://projecteuler.net

I like mathematics. At a glance the exercises at http://projecteuler.net seemed very interesting. There are network walks and problems involving primes and other number types.

I used the typical development cycle at first. Write a script kill it if it runs too long. Optimize and repeat until I have a solution that runs in a reasonable amount of time. Then submit the solution. If it doesn't check out then write an alternate version by altering and trying to correct any perceived deficiencies. Check the solution against the generated solution and repeat.

In my idle time I worked these problems using this process mostly in a terminal on my personal server either in Emacs or in my alternate environment on my phone.

I had recently reacquainted myself with several programming languages I had encountered in the past: assembly, forth, aplus, j-console, lisp/scheme, prolog, java, ruby, JavaScript, coffeescript. And had also learned a new language or two that I wanted to get comfortable with: Haskell, f-sharp, Julia.

I figured I'd write a solution for each of the problems in each of the languages. I setup the environments for each and started with Haskell. What started out as a simple learning experience quickly proved to be more of a brain teaser than I expected. Learning Haskell while simultaneously stretching my programming chops was more work than I had expected.

After solving the first couple of problems in Haskell I switched to solving the first couple of problems in prolog. On the first prolog problem I ended up looking up a solution on Wikipedia that I reworked after reading and searching and understanding more and trying again. In prolog I am a neophyte at best and to call me a programmer would be generous.

When I got my prolog code to work and looked up the next problem it dawned on me that by solving these problems first in a language I am more comfortable in and am considered an expert in I might have a better chance of finishing a significant portion of the over 400 and growing list of problems presented at http://projecteuler.net and then I could attack the solutions in other languages from the point of knowing the solution in another language.

Python is one of my stronger languages. I know how to program in the functional style and in the OOP style. Which would be useful for thinking of solutions in some of the other languages. I started down that road with SL4A on my phone and python in a shell on my server with a pinch of git.

When I hit Wikipedia to lookup triangle numbers it was a real eye opener and led me to back to prime number theory to optimize a solution to a problem I had previously successfully solved involving primes. In that process and to my benefit I had created a virtualenv with a more recent version of ipython. Which includes notebook.

The command 'ipython notebook' in the the directory and virtualenv of your choosing is all it takes to get started. And I was hooked after I solved my my first http://projecteuler.net problem using a notebook. I teared through 5 or six more problems in less time than it took to solve one using traditional editing run cycles. And the benefit of a running shell environment that can be used for  documentation and has individual cells that only have to be evaluated if you want to add them to the solution makes for a very iterative development cycle.

I feel like I can finally keep a train of thought and I can feel my approach to coding at work improving despite the difference in environments.

I sidetracked myself last weekend and to get my head around the clusters tab of a notebook. The ease of launching clusters has me crashing running processes on my tiny micro AWS instance server with only 4 clusters. I resolved to drive farther off track and write some code in my project Euler notebook for spinning up and down temporary clusters using AWS big compute instances for brief period of time to divvy up some of the tougher problems. By writing solutions that are easy to process in parallel I could properly figure out solutions in short order.

Being able to start a solution and continue work on it in parts anywhere from any device is much more accessible than using ssh and a terminal. You have to try it to get it. Working on code in a browser seems like a toy at first. But then you throw some real problems at it and you begin to realize there is something to this style of programming. And then you shutdown your notebook kernel. And then you come back to it and realize you haven't lost anything. Everything you've typed is still there waiting to be evaluated or edited and reevaluated.

And should you get curious and ssh to your server and type 'ipython console --existing' you too will start to understand the power of ipython notebooks.

I know now that I will be using ipython notebooks for all the languages I plan on tackling. After I finish up to problem 15 in order I will start with ruby and Julia. Both languages are the natural next choice because they already have working iruby and ijulia solutions that have ipython notebooks built in.

I'll have to sidetrack again after getting up to problem 15 in each as I'll have to implement an ipython notebook extension to each of the other languages before I start to work on the problems in that language. The tight iterative train of thought development cycle of ipython notebooks is too useful to not use when I am a programmer and can plugin the solution. Which is the point of the entire exercise.

Crushing Python

I've begun thinking in Python a lot lately. ~/.mypyrc.py:

# standard library
from collections import defaultdict as DD,namedtuple as NT
import csv
from datetime import date as D,datetime as DT,time as T,timedelta as TD
import operator as op
import os
import re
from cStringIO import StringIO as SIO
import sys
import time
from urlllib2 import build_opener as BO,install_opener as IO,Request as R,urlopen as UO
from urlparse import urljoin as UJ,urlparse as UP
from zipfile import ZipFile as ZF
# third party libraries
from BeautifulSoup import BeautifulSoup as BS,BeautifulStoneSoup as BSS
import pytz
# initialization
IO(BO())

~/.profile or ~/.bash_profile:

PYTHONSTARTUP=~/.mypyrc.py
export PYTHONSTARTUP

Now starting python interactively loads the PYTHONSTARTUP file automagically:

python
>>> for f in ZF('a.zip').namelist:
...  print f
... 
>>> print BS(UO(R('http://google.com'))).find('title').text
>>> cols=zip(*csv.reader(open('a.csv')))
>>> rows=zip(*cols)

pypad - the python notepad web server

The plan is to take this simple python script and expand on it's note taking features with a GUI, console, and command line front end and additional storage options including a tiddlywiki style of appending the data store to the script.

The web server will eventually be an option rather than integral.

http://launchpad.net/pypad

I will fork this into a micro framework and also fork it into a python eval type remote or local shell. Maybe even a REPL embedded in a web page.

A Diatribe on Being Humble While Misunderstanding Incompetently

I do not subscribe to the I'm in charge so I feel powerful mentality. In fact, I feel the opposite about the administrator and programmer role.

I take the role names more literally in their humblest interpretation. An administrator is more like just a lowly administrator mired in the details. A programmer is more like just a computer in the calculator sense of the word. Both are tools that business minded entrepreneurs can make good use of.

That is not to say that I think little of what I do when I am in those roles. But more of a right sizing of my ego and attitudes toward my work and how I interact with others. By thinking humbly I can allow myself the freedom to expect more of others and to allow myself to look for the solutions to business problems outside of the programming solution straight jacket.

There are certain conclusions I have come to that have helped me to solve what are invariably business problems.

Firstly, there are no little people AKA users. There are only clients and customers who are invariably right ;)

Secondly, the people who actually use your programs on a daily basis are the one's who are going to decide whether your program is truly useful or not. Bouncing your ideas off of them and getting their feedback is only going to help you to deliver what is needed rather than what was in the specification.

Thirdly, it is easier for you to replicate a feature in a location convenient for the client(not user, user is the derogatory of client or customer) than to have that client click through to the location you have hidden the feature. Click of O(1) as opposed to Click of O(n), as n goes to infinity. n being the number of times the client has to click to find the feature that they want available one click away.

Fourthly, asking people outside of the development team to be responsible for their work is not a cardinal sin. If a graphic designer delivers images that are not in the right palette or format or pixel size, it is not your job to write a software solution to fix. You simply need to point out the problem and do a little hand holding to explain the problem and suggest a non-programmatic solution. This point bears repeating. If the client is unable to to access your website because they are using a crappy browser from the 90's don't change your code to handle that browser. Simply point out the problem and suggest a solution. I'm sure this could be loosely interpreted as telling the client how high to shove it. But I am only reiterating my point which is asking non-techie's to do their job correctly and responsibly is ok, acceptable, and an all around good thing.The only way people will act competent is if they are asked nicely to be competent and shown how to be competent.

Fifthly and lastly, there are tons of incompetent people out there. You, including me, are probably one of them. It's not the end of the world. We all sometimes think if someone doesn't know what we know then they are incompetent. Trust me this is flawed logic. As a simple proof to yourself, tell me what number I am thinking of right now. Look now I know something you don't know. Should I conclude you are incompetent with numbers and math? It's only one number should I extrapolate that to mean maybe you will have trouble with more than one number? Like addition, subtraction, multiplication, and division. Surely you suck at guessing games and logical thinking. Otherwise you would have guessed it already. This may seem like a trumped up example. Which it is. But it is also true that humans are bad predictors of almost anything. Especially when they are incompetent enough to think they know more than you or conversely put, when you think you know more than someone else. It's a paradox that you see around you every day. Why does management seem so incompetent? Is it because they are incompetent or are you so incompetent that you can't understand what you don't understand? Watch out for this one. Everybody does it including you. That is why it is known as a paradox. It is also why I use a number guessing game as an example. No one thinks a "What number am I thinking of?" guessing game to be fair except children.

The Expectation of Lack of Privacy in Python

One feature of Python that I thankfully rarely have to think of is the whole private, protected, public meme that seems to always popup in programming languages. Programmers coming from other programming languages always want to know how they can make an object's properties private.

Well, besides that fact that black boxes are only good for figuring out what went wrong after the crash. I'd have to say that the whole privacy meme in programming stems from the same sort of administrator == all powerful == programmer way of thinking. Responsibilities are not entitlements. Responsibilities are work.

Here is an example I wrote to clear up what I felt was a miscommunication about using the underscore to create private variables in Python:

Here is a short example I wrote that demonstrates my point about property getters and setters. I have tested cutting and pasting the following into an interactive python 2.5.2 session launched by typing python on the command line without any parameters. The # commented lines should match the output line above it. I hope this clarifies any miscommunication we may have had.

class One(object):
  def __init__(self, value):
    self._x = value
  def getx(self):
    return self._x ** 2
  def setx(self, value):
    self._x = value + 1
  x = property(getx, setx)

class Two(object):
  def __init__(self, value):
    self._my_x = value
  def getx(self):
    """ directly accessing a "private" member of another class """
    return self._my_x._x
  def setx(self, value):
    """ directly setting a "private" member of another class """
    self._my_x._x = value + 2
  # Not defining a setter for the polite_x property makes it read only
  @property
  def polite_x(self):
    """ using the exposed property of another class """
    return self._my_x.x
  x = property(getx, setx)

one = One(10)
two = Two(one)
one._x
#10

one.x
#100

one.x = 10
one._x
#11

one.x
#121

two._my_x._x
#11

two._my_x.x
#121

two.x
#11

two.polite_x
#121

two.x = 10
two.polite_x
#144

two.x
#12

one._x
#12

two.polite_x = 1
#Traceback (most recent call last):
# File "<stdin>", line 1, in <module>
#AttributeError: can't set attribute

I realized 30 seconds after I sent the above example. By looking up the meaning of the underscore and double underscore. That the communication problem was about the underscore as opposed to the double underscore. I was talking about the single underscore because I was asked about the underscore and was only told underscore as feedback when I tried to clarify by asking whether they were referring to the double or single underscore. Which should have been my first clue that I may have my answer judged by someone that didn't fully understand the question they were asking at the time. Which is a big no no. Unless you want to be seen as incompetent because you are being judged by someone who doesn't understand the questions they are asking. And may not be able to understand the answers to those questions. I am loath to admit it but I was just as culpable in this misunderstanding as my understanding of the double underscore was incomplete at that moment. As my mistaken derision of the double underscore came from the accessibility of members like __repr__ and __str__ and __dict__ and __len__. Class members with a double underscore on both sides of the name are system methods and are not meant or implied to being private at all. My lack of interest in making any of my code private was probably the real source of my having no real interest in reading past the disclaimer in the Python documentation which starts with:

9.6 Private Variables
“Private” instance variables that cannot be accessed except from inside an object, don’t exist in Python.

Thus my second email missive of the morning. Making my point all over again using the double underscore. Luckily :). Thank you benevolent dictator for thumbing your nose at private variables!

The private members rewritten with double underscores make those members seem enforceably private. This is because of the exception raised when trying to access those members outside of object and class methods. But this is a misnomer as leading double underscores are actually a name mangling scheme used to hide those members from name collisions with child classes. They can still be accessed directly.

Here is the same example written with "private" members that generate the same results and allows the same access. Despite the miscommunication of apparent privacy. This is because the double underscore's real purpose is for predictable name mangling not privacy.

class One(object):
  def __init__(self, value):
    self.__x = value
  def getx(self):
    return self.__x ** 2
  def setx(self, value):
    self.__x = value + 1
  x = property(getx, setx)

class Two(object):
  def __init__(self, value):
     self.__my_x = value
  def getx(self):
    """ directly accessing a "private" member of another class """
    return self.__my_x._One__x
  def setx(self, value):
    """ directly setting a "private" member of another class """
    self.__my_x._One__x = value + 2
  # Not defining a setter for the polite_x property makes it read only
  @property
  def polite_x(self):
    """ using the exposed property of another class """
    return self.__my_x.x
  x = property(getx, setx)

one = One(10)
two = Two(one)
one._One__x
#10

one.x
#100

one.x = 10
one._One__x
#11

one.x
#121

two._Two__my_x._One__x
#11

two._Two__my_x.x
#121

two.x
#11

two.polite_x
#121

two.x = 10
two.polite_x
#144

two.x
#12

one._One__x
#12

two.polite_x = 1
#Traceback (most recent call last):
# File "<stdin>", line 1, in <module>
#AttributeError: can't set attribute


Bit Flipping in Python

In Python you can access bits directly with the &, |, ^, ~, <<, >> operators on integer objects.

An example bit transformation might be one that moves all of the set bits in a 12 bit number to the left and all of the unset bits in the same 12 bit number to the right.

For example the 12 bit number 101101000001 would be transformed into the 12 bit number 111110000000 using that transformation.

This example bit manipulation can be concisely written in Python as:

n1 = int("101101000001", 2)
bc = sum(1 for i in range(12) if n1 & (1 << i))
n2 = int(("1" * bc) + ("0" * (12 - bc)))

In Python 2.6 and later this example can more concisely be written as:

n1 = 0b101101000001
bc = sum(1 for i in range(12) if n1 & (1 << i))
n2 = sum(1 << i for i in range(11, (11 - bc), -1))

The generic 12 bit manipulation can be more re-usably written as:

# The 12 bits
b12 = tuple(1 << i for i in range(11, -1, -1))

# There are only 13 solutions
result_numbers = tuple(sum(b12[i] for i in range(c)) for c in range(13))
result_strings = tuple(("1" * i) + ("0" * (12 - i)) for i in range(13))

original_number = int("101101000001", 2)
bit_count = sum(1 for i in range(12) if original_number & b12[i])
transformed_number = result_numbers[bit_count]
print result_strings[bit_count]

The Python 2.6 and later version is:

# The 12 bits
b12 = tuple(1 << i for i in range(11, -1, -1))

# There are only 13 solutions
result_numbers = tuple(sum(b12[i] for i in range(c)) for c in range(13))
result_strings = tuple(bin(i) for i in result_numbers)

original_number = 0b101101000001
bit_count = sum(1 for i in range(12) if original_number & b12[i])
# Updated 2011-01-08
# in python 2.7 and later the above can be further optimized
# using range(original_number.bit_length) in place of range(12)
transformed_number = result_numbers[bit_count]
print result_strings[bit_count]

There is no need to define a function call or set of function calls. Bit manipulation easily lends itself to mapping list indexes to result sets that can be prebuilt to aid in readability and execution speed.

python no overhead debugger on demand

One of the best ad-hock troubleshooting ways to figure out what caused an exception post mortem.

Run:

python -i your_script.py

After the exception you will be left at the python interpreter prompt staring at your exception.

Type:

import pdb
pdb.pm()

And you will be put into the context of the stack of the last exception. This means you can print/examine the local variables at the point of failure after the failure has occurred without having to change a line of code or import pdb in advance.

Another method that requires a little bit of preparation is to put the import pdb and pdb.set_trace() in a signal handler trap. *Moved the import of pdb out of the signal handler as imports in functions are deprecated. That way you can do a kill -SIGNAL PID or if the signal you trap is INT you can just Ctrl-C to get dropped into the debugger at any point. The signal handler technique is good for testing release candidates and released code because the signal handler doesn't create any runtime overhead.

Signal handler example. Debugger starts with Ctrl-C:

import pdb

import signal


def int_handler(signal, frame):
  pdb.set_trace()


if __name__ == "__main__":
  signal.signal(signal.SIGINT, int_handler)

Put that at the top of your script and you can start debugging your script at any point by typing Ctrl-C. Resume programe execution by typing exit at the Pdb prompt.

My Review of Roku HD Player

I wouldn't have let Roku post this to my blog except I do approve of this message.

Originally submitted at Roku

The best-selling HD Player (as known as Netflix Player by Roku) plays High Definition video and connects to surround sound audio.

Roku HD Player

Only needs Youtube and Hulu to be perfect

By Danyel from Bay Ridge on 12/22/2009

 

5out of 5

Pros: Great value, Built in Wi-Fi, Easy to use, Compact, Video selection, Easily movable, Easy to set up, High quality picture, Reliability

Best Uses: Bedroom, Living room, Traveling, Friends House

Describe Yourself: Power User, Technophile, Early adopter

Move it from living room to bedroom to finish watching in bed.

(legalese)

Python File Type Plugin for Vim - Part 1

* Made a correction below. Using the ~/.vim/ftplugin directory for a file type plugin causes the plugin to be reloaded repeatedly causing any autocommands to be appended for BufWrite actions. This causes multiple executions of every autocommand every time you save the file. A better location is the ~/.vim/plugin directory. Where the plugin is loaded once at startup. I have changed the location below. *

I wanted to add some IDE style conveniences to my preferred editor, console Vim. Rather than reinventing the wheel I decided to search the Internet and start with what was already out there. I found a bunch of very useful scripts and snippets. In part 1 I am documenting some of what I found here by publishing a snippet that I have only made minor changes to to make it useful for my needs.

This script needs to be put in a file named python.vim and saved in ~/.vim/plugin. If the directory does not exist it needs to be created. The scripts create the pyflakes, pylinks, and pychecker commands that can be used at the vim command prompt, ':'. The pylint, pyflakes, pychecker packages will have to be installed as prerequisites for using these commands.

All three commands act on the current buffer's file name. Running one of the commands at the vim command prompt will open a Vim quickfix window which will seem familiar to anyone who has used a graphical IDE for development.

You can press c to close the quickfix window when it has the focus. You can double click with a mouse under X or press enter to go to the error or warning associated with the current quickfix line. This will take you to the line referred to by that message. You will have to restart vim if you have already loaded a python file during the currently loaded Vim session.

Here is the code:

function! PythonGrep(tool)
  set lazyredraw
  " Close any existing cwindows.
  cclose
  let l:grepformat_save = &grepformat
  let l:grepprogram_save = &grepprg
  set grepformat&vim
  set grepformat&vim
  let &grepformat = '%f:%l:%m'
  if a:tool == "pylint"
    let &grepprg = 'pylint --output-format=parseable --reports=n --indent-string="        "'
  elseif a:tool == "pychecker"

    let &grepprg = 'pychecker --quiet -q'

  elseif a:tool == "pyflakes"

     let &grepprg = 'pyflakes'

  else

    echohl WarningMsg

    echo "PythonGrep Error: Unknown Tool"

    echohl none

  endif

  if &readonly == 0 | update | endif

  silent! grep! %

  let &grepformat = l:grepformat_save

  let &grepprg = l:grepprogram_save

  let l:mod_total = 0

  let l:win_count = 1

  " Determine correct window height

  windo let l:win_count = l:win_count + 1

  if l:win_count <= 2 | let l:win_count = 4 | endif

  windo let l:mod_total = l:mod_total + winheight(0)/l:win_count |

  \ execute 'resize +'.l:mod_total

  " Open cwindow

  execute 'belowright cw '.l:mod_total

  nnoremap   c :cclose

  set nolazyredraw

  redraw!
endfunction

command! Pyflakes call PythonGrep('pyflakes')
command! PyFlakes call PythonGrep('pyflakes')
command! Pychecker call PythonGrep('pychecker')
command! PyChecker call PythonGrep('pychecker')
command! Pylint call PythonGrep('pylint')
command! PyLint call PythonGrep('pylint')

" These three are successively more informative and aggressive in their
" warnings with pyflakes as the least noisy. Only uncomment one.
"autocmd BufWrite *.{py} :Pyflakes
autocmd BufWrite *.{py} :Pychecker
"autocmd BufWrite *.{py} :Pylint

In Vim a single double quote, ", is considered the beginning of a comment for the rest of the line.

The benefit of having the autocmd run one of the Python lint programs every time you save the file is that you can never get too far from sane working code as any errors and warnings pop up. This, in your face, style of development can keep you from pulling your hair out over something as simple as a semicolon or comma typo and can also warn you about more complex problems and proper coding style.

More to follow. I will try to combine the output of the three commands as their output doesn't seem to overlap for the most part. And the noisier programs (pychecker and pylint) don't appear to give the same advice when it comes to warnings. pylinker is by far the noisiest of them all but all it's warnings are configurable.

Warning: If you are using tabs for indenting you will have to modify the pylint parameter '--indent-string='. Set it to the empty string for tabs, or the actual number of spaces used for indenting. In my code above I have it set for 8 spaces which is project specific.

Step One: Insert Foot in Mouth

This is a follow up to my Your Fancy Language is Burdensome and Dangerous post.

Python and Ruby don't have the APL array/vector/matrix syntax built in to the core language. But a little nudge from someone made me do the little bit of research I should have done before I wrote that post. And I rediscovered that I did not originate the idea that these languages need array/matrix math syntax thought.

There is a gem for Ruby called NArray and an egg for Python called Numpy that not only add array syntax but also coherent coercion rules and the ability to apply functions across arrays without calling map. I'm glad I found it because they are easy as pie to use and do exactly what I wanted and a heck of a lot more.

Check out this Python example:

from numpy import *
a = array([1,2,3])
b = array([4,5,6])
print a+b #equals array([5,7,9])
print (a+b)/4
print (a+b)%6

Outputs
[5,7,9]
[1,1,2]
[5,1,3]

As you can see Numpy also replicates the integer type bug/feature which is very C like but is being dropped in Python 3000 in favor of coercing integers to float on divide. You can typecast the divisor to float to get floating point division and float results.