My Machine Learning Tool Box

I've been meaning, for sometime, to write about everything science and technology that I am passionate about. What better way to kick it all off, than to talk about my machine learning development toolbox.

Programming Language

I believe the programming languages you use is one of the most key tools in your toolbox.  Why? Because it affects the way you develop code, from the simplicity of code in expressing the solution, the readability of the written code, the availability of libraries for the wide range of problems you might encounter, the productivity with which you can develop software and doing it all without trading off too much in performance when it matters. I also believe in a small but powerful toolbox that you are an expert at than a very large one. For an average developer like myself, I find it easier to be an expert of one or two languages than claim to have an average understanding of 10 languages. Hence I choose my languages in my tool box carefully.

Hence my tool box consists of the following

1. Python
2. C/C++

Python is a scripting language, that is comparable to perl and ruby but superior in many respects. It has the same level of expressive power as Perl, yet is way more readable which is very important in software development. Code gets read many more times than it is written and it is important that it is very readable. Its an interpreted language but also has JIT implementation available for performance. And when further performance is needed, it is easy to convert the performance critical code  in to C/C++ directly or using a tool like Cython as an assist. With libraries like CFFI it is easy to wrap C libraries and make them available for use from python. And the range of libraries and tools available in python is legendary, from web development, mathematics, scientific computing.

C/C++ is a compiled lower level language that allows me to implement sections of the code that need to be performant. Generally speaking, 90% of the performance problems can be narrowed down to 10% of the code. So my development philosophy has always been to write what i need in python and drop down to C/C++ to improve that 10% of the code where performance matters. There tools like Cython and CFFI that make it brain dead simple to do.  For the sort of the code where scripting language is not an option C/C++ will fill in the space completely.

Go language from google as a future alternative to C/C++ has potential. I am keeping an eye on it. I would like to see how it works and interfaces with python. And if it keeps improving like it has been, it probably will replace C/C++ in my toolbox

Development Environment

iPython is a more advanced python shell/development environment that allows you to develop faster. It supports functionality like the ipython notebook that provides a browser placed environment to experiment with code, generate and view graphs, etc. interactively from within the browser. Additionally, the code and generated graphs can be interspersed with documentation that makes it a great tool to communicate ideas or generate interactive documentation.

virtualenv/pip is a set of tools in python that allows you to create virtual private instances of the python installation, there by allowing you to do development of python modules without messing with the central installation. These virtual instances are very light weight. Further, pip a more advanced package management tool allows for a "development mode" install of the library you are developing which is convenient for development. It creates symlinks to your development directory from the python installation directory, thereby letting you modify your python sources and running them without any additional installation requirement.

CFFI is python wrapper generator for C code. This allows you to take any C library and generate a python wrapper to the C code from the C header files with little or no change.

Mayavi is a 3D scientific visualization tool with a python interface that is very handy to view error surfaces of neurons or neural networks.  It allows me to take training errors, neuron signals, etc as they evolve during training and have them plotted as 3D surfaces that give better visual understand of how things are performing or where the problems are. Understanding machine learning needs an understanding of how error surfaces behave during training.

Libraries

Numpy/SciPy  - Machine learning involves a lot of floating point vector arithmetic. A library like Numpy allows you to do fast vector arithmetic from python and has a full library of statistical and scientific formulae implemented and ready for use.

Pandas - Think of Pandas as filling the holes in Numpy when specifically dealing with Time Series data.

Hardware Acceleration

Theano - GPGPU Acceleration - When working machine learning algorithms, you will quickly find that it is very vector math intensive, meaning you are going to be doing vector calculations a lot. This means that you are soon going to hit the limits of your main CPU, even if it has many cores. Though many of the vector math libraries can make use of multiple CPUs to accelerate your machine learning algorithms, being able to leverage one or more GPGPU units you may have on the system will be indispensable. GPGPU stands for General Purpose Graphic Processing Units. These are graphic accelerators from vendors like NVidia. But graphic processing requires a lot of vector arithmetic and hence they have been designed to be good array/vector processors which is exactly what is needed for machine learning.

A python library like Theano allows you to express your machine learning formulae and have them compiled to run on the GPU and hence achieving 10-100 fold increase in performance. If a GPU is not available, it is smart enough to use vector acceleration libraries that can make use of multiple CPU cores to still get better performance from python.

The Future

There  is still a scale issue if you want to do really massive models that might involve distributed computing. The GPGPU are evolving to be networking capable and I hope in the future theano evolves to be able to distribute its computation across a cluster of GPGPU nodes.

Machine Learning involves developing mathematical models for neurons and neural networks as well creating networks of networks. It would be nice to have an object orient library where I can implement neuron and network models driven by the power of theano behind it. It would allow for a more intuitive experimentation with machine learning models and networks.

In general the language you are comfortable, the tools you like to use are all quite personal. Over the years I have tried many of the language choices including perl, tcl, java, bash, etc. and have settled to the once above for the near term. Technology does change rapidly and these choices might change in the future. But this is my machine learning toolbox as it stands now.

Dry Rubbed, Honey Glazed, Slow Baked Chicken - A Thanksgiving Recipe

Most people cook a Turkey for Thanksgiving. But not everyone likes or eats turkey. So I made a dry rubbed, honey glazed, slow baked, chicken with crispy garlic bread and potato all in one dish.

Dry Rub Recipe - Make anytime and store
Ingredients
Salt - 1/3 cup
Jaggery - 4 - 1"x1/2"x1/2" pieces
Chilli powder - 7 tablespoons
Ground Pepper - 2 tablespoons
Allspice powder - 1 teaspoon
Garlic powder - 2 teaspoons
Ginger powder - 1 teaspoon
Cumin powder - 4 teaspoons
Onion powder - 2 tablespoons
Mustard powder - 2 teaspoons

Preparation
Mix and grind all dry rub ingredients into a coarse powder in a mortar or dry grinder. Store in a container until needed.


Marinading the Chicken - 12 hours
Keep the skin of the chicken intact and use a knife to separate the skin from the meat. Cut deep slits on the meat(not the skin). Sprinkle dry rub powder generously over the meat and massage into it the meat and slits. Rub the spice mix inside the chicken as well. Cover the meat by pulling the skin back over the meat. Sprinkle dry rub over the skin, as well. Place in a tray. Cover with plastic wrap and refrigerate for 12 hours.

Slow Baking the Chicken - 10 hours
Ingredients
Chicken, whole, with skin - 1
Bread - 8 slices
Potato, medium - 3 or 4
Fingerling potatoes(tiny) - 20
Garlic - 2 teaspoons
Onion powder - 1 tablespoons
Fresh Ground pepper - 3 teaspoons
Salt - 2 teaspoons
Chicken Stock - 1 cup
Lemon Juice - 1 tablespoon
Honey - 3 tablespoons
Vegetable Oil - 1 tablespoon

Preparation - 10 minutes
In a bowl mix garlic, onion powder, pepper, salt.
In a ceramic or glass baking tray, rub vegetable oil on the bottom. Layer with quarter inch slices of potatoes. Sprinkle the garlic/onion spice mix from the bowl over the sliced potatoes. Layer the sliced bread. Sprinkle some more of the garlic/onion spice mix from the bowl on top of the bread. Place the marinated chicken on top of the bread. Put the fingerling potatoes into the bowl, close and shake. Drizzle the vegetable oil into the bowl. Close and shake to coat evenly. Spread the fingerling potatoes around the chicken and put a couple inside it as well. Drizzle the liquid remaining in the bowl over the bread and potatoes.
Cover lightly with aluminium wrapper and a put a few knife slits into the wrapper.

Baking - 10 hours
Bake on the middle rack(not too close to the heat) at 200 degrees for 10 hours.  Increase temperature to 220 degrees for the last 2 hour and remove the foil. Mix the honey and lemon juice in a bowl and keep brushing the whole chicken every 1/2 hour. Add the chicken stock little by little into the sliced potatoes/bread to prevent them from drying too early.
When done at the end of 10 hours, the chicken skin, sliced potatoes, and bread should all be browned and crispy. The bread should not bet wet.

Serving
Let the bird rest for 10 minutes before carving. Serve a slice of the crusted potatoes and bread along with the fingerlings. Goes well with a bottle of medium bodied red wine.

Can be done with leg quarter of chicken as well for a smaller gathering.

Bongoo - My first rap song

He's a tease, my cheese,
My breath, my breeze,
The bee's knees, I love to squeeze,
My life's lease, across the 7 seas,
A hug, a kiss, and my spirit free's

He giggles, he squiggles,
When I hold him, he wiggles, he wriggles
Garbage cart, garbage cart, he niggles,
One gummy bear, no two, he tries and higgles,
What is this, what is that, everyday gets smarter, and riddles.

He's my workout, my mahout,
With whom I play out, till its lights out,
The reason, I don't washout, punch out, or feel like a copout,
The one I go to, when I burnout,
No doubt, the one I can't do without.

Every day I wandered, I sat on an pondered,
The meaning of life? I wept and wondered,
Searched far and wide and beyond the yonder,
Everyday and every night, I grow fonder,
I can tell you for sure, nothing in life grander.

Fly back, on a whim,  from Timbuktu,
Coz, I'll miss the way, he goes, Peekaboo,
Its black magic, its kung fu, no, jijutsu,
A smile, your defenseless, he subdues you,
My daily rendezvous with, the chatty, the naughty, the sidu bongoo.

In search of a meaning for the words "I Love You"

Why do you feel all warm and fuzzy when a 2 year old says "I Vuv You". Incase you did'nt already get it, thats baby talk for "I Love You"

People say "I Love You" all the time.  Do they really mean "I love the way I feel when I am around you"?

A mother says "I Love You" to her kids. A child says "I Love You"  to the father. Friends, cousins, soul mates and lovers say it to each other. Ironically, so do stalkers, desperados and the crazies who won't let go.

But what do they really mean.  What do you really mean when you say "I love you" to someone and why do you say it. Why does it evoke such craving for more. Why do some even go nuts in its pursuit.

For, such a meaning must convey such varied emotions, apply to all ages, a range of relationships and be applicable to states of mind as different as day and night.

The search for a universal meaning for those 3 words together eluded me for while.

The definition of the word 'love', according to the dictionary, an intense feeling of deep affection, a person or thing that one loves, to feel a deep romantic sexual attachment. But none of it explained all.

And then I pondered the why and the how I feel when I say that to someone or when someone else says that to me. Why do people crave it. Even get obsessed and addicted to it. It made more sense.

It seems to make sense for all the times and places when people say those words. People want to feel good about themselves, have a good self image. And if people feel that way when they are with someone, it only makes sense that they want more of that feeling.

So now I think, when someone says those three words "I Love You", what they are really saying is "I love how I feel when I am with you".

Quite a few people pointed out or argued that this was a very selfish perspective. And that love is selfless, giving, unquestioning and boundless in nature and more.

I just wanted to clarify that I don't disagree with these characterizations of what love is and what it is capable of.  I have no doubt they are all these and possibly more. When you wonder about something like love, the answer you get also depends on the question you ask.  I was merely trying to ask the why?

Like a child, I kept asking myself, why? why? why? at every answer or reason I came up with, and this process kept returning to the self, the how or why the self felt the way it did.  Returning to ones own heart.

This may sound clinical and unromantic at first, to look at it from this perspective. It doesn't change the fact that the person that is the object of one's affection is the one that makes you feel and want to be a better person, a better mother, a father or a friend. It doesn't take away from it the fact, there are some innate or hard earned qualities about that person that makes you feel that way. That the way you feel may even be the emotional price you pay for those qualities.

But its undeniable that the why, eventually kept me coming back to one conclusion. That when we say "I love you" one really means "I love who I am and how I feel when I am around you"

Tandoori Chicken Sandwich with Grilled Onions, Chipotle Chilli and Avacoda

Ingredients

• Sliced White Sandwich Bread
• Leftover Tandoori Chicken (Refer to Tandoori Chicken Recipe)
• 1 Chipotle Chilli
• 1 large red onion
• 1 avacado
• 1 Tomato
• A few Olives or Tepanade
• 1/2 teaspoon chilli flakes
• Dijon Mustard
• Mayonaise
• Pesto Sauce
• Chipotle Cheddar slices

Steps

1. Shred the left over Tandoori Chicken into small pieces
2. Slice the red onions and Chipotle Chilli into thin long slices and saute in olive oil for 4-5 minutes until the onions translucent and chilli slices are soft. Keep aside
3. Spread Dijon mustard whipped with chilli flakes on half the bread slices
4. Spread Pesto Sauce on the other bread slices.
5. Squirt a couple of strips of mayonaise on all the slices
6. Layer the shreded chicken, sauted onions/chilli, a couple of thin slices of tomato, a couple of sliced olives or tepanade, a couple of thin slices of avocado, and top it with just a few bits of the chipotle Cheddar slices
7. Close the sandwich the remaining bread slices and grill in a sandwich maker. If you don't have one you can grill it on a pan sprayed  with olive oil and placing a HOT and heavy cast iron pan on top of the sandwich  until both sides get toasted.
8. Slice and serve

Fried CatFish on bed of Cabbage, seasoned with Coconut and Poppy seeds

Ingredients

• 2 teaspoons of Poppy Seeds
• 2 teaspoons of Fennel
• 2 teaspoons of Jeera
• 1 1/2 tablespoon of Pepper corns
• 10 Curry leaves
• 1/3 can of coconut milk
• 1/3 can of water
• 1 medium sized Onion, chopped
• 1 teaspoon of Garlic Paste
• 2 medium sized tomatoes, chopped
• 6 pieces of Cat fish, each about 2 inch squares
• 1 quarter of medium sized Cabbage, chopped fine

Steps

1. Grind the Poppy seeds, Pepper Corns, Jeera and Fennel seeds in dry grinder
2. Heat the oil in a Tawa
3. Put the Curry leaves in the hot oil for 30 seconds
4. Add the ground powder from step 1 and fry for another minutes
5. Add garlic and fry
6. Add chopped onions an fry till the onions turn brown
7. Add the tomatos and fry till the tomatos well
8. Add coconut milk and equal amount of water and cook till the added water is gone
9. Remove some of the sauce into a bowl, enough to batter the pieces of fish with
10. Reduce the the gravy a bit further into a thick sauce
11. Add the finely chopped cabbage and fry for 1 minute
12. Take of the heat and transfer into a square serving plate
13. Batter the pieces of fish in the gravy saved for battering in step 9
14. Shallow fry in hot vegetable oil about 3 minutes each side
15. Lay the fried fish on on the bed of cooked Cabbage  

Poached Cat Fish in a Coconut Poppy seed gravy

Ingredients

• 6 pieces of Cat fish, each about 2 inch squares
• 2 teaspoons of Poppy Seeds
• 2 teaspoons of Fennel
• 2 teaspoons of Jeera
• 1 1/2 tablespoon of Pepper corns
• 10 Curry leaves
• 1/3 can of coconut milk
• 1/3 can of water
• 1 medium sized Onion, chopped
• 1 teaspoon of Garlic Paste
• 2 medium sized tomatoes, chopped

Steps

1. Grind the Poppy seeds, Pepper Corns, Jeera and Fennel seeds in dry grinder
2. Heat the oil in a Tawa
3. Put the Curry leaves in the hot oil for 30 seconds
4. Add the ground powder from step 1 and fry for another minutes
5. Add garlic and fry
6. Add chopped onions an fry till the onions turn brown
7. Add the tomatos and fry till the tomatos well
8. Add coconut milk and equal amount of water and cook till the added water is gone
9. Lay the pieces of Cat fish in the gravy, scoop an dour some of the gravy on top of the pieces of fish. After a couple of minutes and turn the fishes and cook for another couple of minutes
10. Get of the heat and transfer to a serving bowl

Persian Saffron Rice - A 24 hour process

Ingredients

• 1 cup Basmati rice
• 1/2 teaspoon sugar
• pleanty of salt
• 1 tablespoon butter
• 1 tablespoon of vegetable oil
• 3-4 pieces of saffron
• potato or uncooked chapathi or cooked naan, or yogurt  for "taadig"

Steps

1. Wash the Basmati Rice 5-6 times in water until the run off is clear
2. Soak the rice in salted water for a whole day, about 22 hours. The water must be very salty. Say like tablespoon on salt for every 2 cups of water that you use for soaking.
3. Drain the water out of the soaking rice
4. Bring to boil the drained salty water and add extra water to make sure there 3 cups of water boiling for every cup of rice
5. Add the rice to boiling water and cook for 10-15 minutes(depending on the rice) until the rice is cooked but not mushy. The rice should still have a bit of bite to it and not fully cooked.
6. When done dump the rice into a strainer to strain out the water.
7. Take 2 spoons of the boiling water and mix well with the sugar and saffron and keep aside
8. In a non-stick cooking pot, heat on high and drop the butter and the oil. 
9. When hot add 1/2 cup of water and stir well till the oil and water mix
10. Then poor the liquid out to a bowl, leaving a little bit of it in the bot
11. Layer the bottom of the pot with items for taadig. i.e. slices of potato, uncooked chapathi or a couple of spoons of yogurt mixed with couple of spoons of rice.
12. Then put the rice back into the pot on top of the taadig layer in a mound. 
13. Put some vertical holes into the mound for the rice to breath
14. Pour the water/oil mixture around the rice
15. Pour the saffron water on the rice.
16. Cover the pot with lid wrapped with towel
17. Turn the heat on Medium-High for 5 minutes
18. Then to Medium-Low and cook for 1 hour
19. Remove from heat and immediately immerse the pot in cold water to stop the cooking
20. Scoop out the rice in the bowl
21. Then cover the pot with a plate and invert pot to get the taadig out of the pot
22. "Taadig" is the crusted layer at the bottom and is considered a delicacy by itself