I asked AI aka ChatGPT "How to improve Social Media Algorithms to prevent extremism and promote balance in thinking within society"

 

I thought it would be an interesting question to pose to ChatGPT the most exciting AI engine at the current point in time, July 11, 2023. Basically ask AI how it thinks, AI algorithms in Social Media should be improved. 

Here is the fascinating answer I got.

I have to say I am very impressed with this response that feels Spot On!!

Improving social media algorithms to prevent extremism and promote balance in thinking within society is a complex challenge. While there is no definitive solution, here are a few suggestions that could potentially contribute to a more balanced and informed social media environment:

1. Transparency and User Control: Social media platforms should provide transparent explanations of how their algorithms work, including factors that influence content visibility and recommendations. Empowering users with control over their feed preferences and filtering options can help them customize their experience and reduce the echo chamber effect.

2. Diverse Content Recommendations: Algorithms should be designed to prioritize diversity and provide a range of perspectives on important topics. This could involve exposing users to content that challenges their existing beliefs, promoting reputable sources, and ensuring representation from various viewpoints.

3. Fact-Checking and Contextual Information: Integrating fact-checking mechanisms into the algorithm can help combat the spread of misinformation and disinformation. Platforms could work with reputable fact-checking organizations to provide accurate information and context alongside potentially misleading content.

4. Ethical Design and Responsible AI: Social media platforms should prioritize ethical design principles, taking into account the potential impacts of their algorithms on individuals and society. Responsible AI practices involve avoiding amplification of extremist or polarizing content and being mindful of the potential consequences of algorithmic decisions.

5. User Education and Media Literacy: Promoting media literacy and critical thinking skills among users is crucial. Social media platforms can collaborate with educators, organizations, and experts to provide resources, training, and initiatives that help users navigate information and engage in respectful and informed discussions.

6. Feedback Mechanisms and Continuous Improvement: Platforms should actively seek feedback from users and the wider community to understand the impact of their algorithms and make necessary adjustments. Regular evaluations, external audits, and ongoing research can contribute to refining algorithms and addressing potential biases or unintended consequences.

It's important to note that no single solution can completely eliminate extremism or ensure perfect balance. Combating extremism and promoting balanced thinking requires a multi-faceted approach involving collaboration among social media platforms, governments, users, and society at large.

So its not this problem cannot be solved, but if the purely capitalist structure motivated by corporate greed that is driving the social media giants will allow these changes to happen

Mushroom, Chicken Rice, with bacon, brussel sprouts and paneer, wrapped in Collard Greens

Ingredients

• 2 tablespoon olive oil
• 1/4 teaspoon of ground Sichuan peppers, ground to a powder
• 1/4 teaspoon all spice powder
• 1/4 teaspoon cloves, cardamom and cinnamon powder
• 1/2 teaspoon of jeera powder
• 1/2 teaspoon of fennel seed powder
• 2 spoons of chili powder
• 1/2 large onion finely chopped
• 1 can of chicken
• 6 pieces dried mushrooms
• 1 teaspoon of ketchup
• 2 teaspoon of sriracha sauce
• 4 slices of cooked bacon, finely chopped
• 1/2 cup cooked rice
• 1/2 cup paneer(cottage cheese) chopped in small pieces
• 5 teaspoons of grated asiago or parmesan cheese
• 3 large collard green leaves

1. Start with heating a pot. 
2. Add oil and heat. 
3. Add all seasoning powders and cook for a minute. 
4. Add chopped dried mushrooms and fry for a minute 
5. Add onions and fry for 2 minutes. 
6. Add 1 can of chicken after straining the water. 
7. After a minute add cooked bacon. 
8. Add the stem from the collard greens finely chopped.  
9. Add finely chopped paneer
10. Add left-over cooked rice and mix well
11. Switch off heat. Let it cool down for a few minutes
12. Add one egg and grate some asiago or parmesan cheese. Mix well and keep aside
13. Take the collard green leaves, without the stem.
14. Add the rice mixture into it and wrap it, holding with a tooth pick.
15. Sprinkle Salt on the collard green wrap.
16. Steam or Bake. Place in steamer and steam till the collard green is cooked well and soft but does not break apart. OR Place in a baking dish drizzle some oil over it and bake it at 400F for 30 minutes.  

Avakkai Apple Thokku

Ingredients

• 10 Raw Apples, preferably the very sour kind
• 1 cup Mustard Seeds
• 1.5 - 2 cups Spice Red Chili Powder
• 3/4 - 1 cup Salt
• 1/4 cup Turmeric Powder
• 1/4 tablespoon Asafoetida
• 3 tablespoon Fenugreek(Methi) Seeds
• 2 cups Sesame Oil
• 1/3 cup of Whole Garlic Cloves, Optional
• 1/2 cup Lemon Juice, needed if apple is not sour enough

Method

1. Adjust number of apples depending on size. Grate the apples, roughly, with the skin. Use a grater with bigger holes and not the fine ones.
2. Lightly toast the mustard seeds along with the fenugreek seeds for a few minutes in low heat and grind to a fine powder. This powder can be stored for later use if there some left after making the pickle.
3. Bring Oil to Medium Heat in a thick and deep pot.
4. Add the Garlic
5. A minute later add Asafoetida, add the grated apples, powder that was ground above, chili powder and 3/4 of a cup first of salt.
6. Let it fry for 5 minutes
7. Taste. 
• It should be spicy. 
• Check for salt, very mildly salty is good, but then salt is always a personal preference. 
• It will have a bitter bite from the mustard/fenugreek powder, but the bitterness will fade as the pickle ages in the fridge.
• Check for sourness
8. If salt is not sufficient, add as needed.
9. If sourness does not come through, add some lemon juice as needed.
10. Cook for an additional 15-20 minutes
11. Reduce heat to low-medium and slowly cook until all of the below happens.  
• The mixture reduces down in volume and thickness reaches Thokku consistence i.e a lightly thick paste
• The oil must separate out
• The water in the mixture is almost completely gone. Specially if you don't plan on refrigerating the pickle
12. Turn off heat, cover and let it cool down naturally.
13. Stores in glass jam jars, preferably refrigerated.  

The key to taste is aging it for a few days so the bitterness wears out.

The key to longer life when stored is 

• acidity from the sourness
• saltiness
• more oil, there should be a layer of the oil on top of the Pickle/Thokku when stored
• Making sure as much of the water has been cooked out. I tend do reduce it in the oven for a few hours, occasional stirring, for longer shelf life. 

This goes well with 

• just plain white rice
• curd rice
• as an accompaniment for dosa, idli
• as a spicy spread with mayo on sandwiches 

The Good Fart Day - Celebrating a Healthy Colon - May 14th

I hereby proclaim that May 14th, shall hence forth be known as

The Good Fart Day

A day that celebrates a healthy Colon and guilt free farting.

Here is a verse that says it all

The Theory of Farts

டர்ரும் புர்ரும் நிர்பயஹ,

(darrum burrum nir bayaha)

(loud farts are not to be feared)

குய்யும் முய்யும் மத்ய மஹ, 

(kuyyum muyyum madhya maha)

(medium sounding farts can be tolerated)

நிசப்தம் பிராண சந்கட்டி. 

(nisaptham praana sankatti)

(The quiet ones are life threatening)

This started of as my contribution to the ridiculous list of days celebrating all sort of non-sense.

Did you know there is World Cleavage Day. Really!?

How about World Naked Gardening Day.

But then I realized, If you have a breast cancer day, 

why not have a colon cancer awareness day.

But what a boring name.

So I decided to call it the 

"Good Fart Day"

So how about we celebrate our fine asses, by drinking to it, reminiscing on all the ass jokes that we can think of, may be giving your digestive system a break, giving it a good enema, and making sure we give it the respect that it deserves.

An lets always remember, the ass hole is always in charge

"I should be in charge," said the brain , "Because I run all the body's systems, so without me nothing would happen."

"I should be in charge," said the blood , "Because I circulate oxygen all over so without me you'd waste away."

"I should be in charge," said the stomach," Because I process food and give all of you energy."

"I should be in charge," said the legs, "because I carry the body wherever it needs to go."

"I should be in charge," said the eyes, "Because I allow the body to see where it goes."

"I should be in charge," said the rectum, "Because Im responsible for waste removal."

All the other body parts laughed at the rectum And insulted him, so in a huff, he shut down tight. Within a few days, the brain had a terrible headache, the stomach was bloated, the legs got wobbly, the eyes got watery, and the blood Was toxic. They all decided that the rectum should be the boss

The Moral of the story? Even though the others do all the work.... The ass hole is usually in charge

Lets keep him happy.

And so...... 

"Happy Good Fart Day"

வாழ்க்கை தாரகை (life is a star)

இது பொதிகை மலை, அதில் பனியின் புகை
         (idhu podhigai malai, adhil paniyin pugai)
         (Podhigai Mountains, the mists smoke)
இந்த  இரவின் பகை, என் கைகள் துணை
         (indha iravin pagai, en kaigal thunai)
         (The nights enmity, my arms shall guard)
பாரெலாம், தேடியும், என் மனம், உன்வசம்  
         (paarelaam, thediyum, en manam, un vasam)
         (World over, I have searched, but my heart's with you)
என் மனம், தாண்டவம், காரணம், உன்னிடம்
         (en manam, thaandavam, kaaranam, unnidam)
         (My heart, dances, the reason, is with you)

என் கண் அசைத்தாலே  உந்தன் ஞாபகம்
         (en kann asaindhaalae, undhan ngyabagam)
         (Even when my eyes move, its you that I think of)

நீ காமன் கலை, அதை ஓதும் சிலை
        (nee kaaman kalai, adhai odhum silai)
        (You are, the Art of love[kaman, is the god of love], the sculpture that speaks it)
உன் இதழ்கள் தனை, என் இதழில் இணை
        (un idhalgal thanai, en idhalil innai)
        (Let your lips, join with mine)
மோகமோ, என் வசம், நாணமோ, உன் வசம்
        (mogamo, en vasam, naanamo, un vasam)
        (passion is in me, shyness in you)
தேகமோ, சேரணும், தாகமோ, தீரணும்
        (thegamo, seranum, thhagamo, theeranum)
        (Our Bodies, should meet, thirsts to be quenched)

உன்  கை அணைத்தாலே, உள்ளம் போர்க்களம்
        (un kai anaithaalae, ullam porkalam)
        (If your arms hug, my heart comes a battle field)

உன் பார்வை வலை, அது எந்தன் சிறை
        (un paarvai valai, adhu endhan sirai)
        (Your look is a net,  that becomes my prison)
நீ பேசும் குரல், நான் தூங்கும் இசை
       (nee paesum kural, naan thoongum isai)
       (The voice when you speak, is the music I sleep to)
காலத்தின், போக்கெனும், சோர்விலா, ஓர் விஷம்,
       (kaalathin, pokaenum, sorvila, or visham)
       (Time, its passage, is a tireless, poison)
அதை போக்கவும், தாக்குமே, தேவதை, உன் ரசம்
       (adhai pokkavum, thaakumae, thevathai, un rasam)
       (To eliminate, it fights, goddess,  your essence.)

என் வாய் இசைத்தாலே உந்தன்  வாசகம்
       (en vaai isaithaalae, undhan vaasagam)
       (When my mouth sings, its your story)

உன் இமைகள் கலை, அசைக்கும் பரதக்கலை,
      (un imaigalkalai, asaikkum barathakalai)
      (Your eyelids, are moved by the art of barathanatyam[south indian classical dance])
என் திசை சாய்க்குமே,  உன் ஒரு புன்னகை
      (en thisai saaikumae, un oru punnagai)
      (It changes my direction, your one smile)
காந்தமோ, உன் இடை, சாந்தமோ, உன் நடை
      (kaandhamo, un idai, saanthamo, un nadai)
      (magnet, your hips, calm, your steps)
கடல் பாறையோ, கால்களை, தேய மோதுதே, உன் நகை
      (kadal paarayo, kaalkalai, theya modhuthey, un nagai)
      (rock at sea, my feet, to erode they crash, your giggles)

அகம்  அடைந்தாலே, வாழ்க்கை தாரகை
      (agam adainthalae, vaalkai thaarakai)
      (When I win your heart, my life is a star)

Copyright (c) Sarvi Shanmugham

Rent Vs Buy Comparson

To Rent or Buy. A question that never seems to have answers, and happens to be center of many debates when family and friends get together. And I have a had more than my share of that conversation.

A friend of mine, Subbu, started this spread sheet analyzing the Home Buying decision a while back and shared it with me. I had a few more questions and what ifs that I wanted to understand. So I decided to expand on this. I am sure a lot of you have questions on this topic. This is an attempt to settle this by the numbers.

Found and even better one from New York Times here use that

Well not really, but to provide a tool to debate with hard numbers, different assumptions and projections for the future.

Below, you will find that document published. I have protected most of the formulae so you wont be able to modify them. But the fields marked green in the first page are editable to allow you to test out different assumptions about the future.

The original interactive google spread sheet document can be found can be found here.

The one thing this document does not cover is the emotional aspects of owning your own home, the memories and the ups and downs that might come with it. Attempting that would be an exercise in futility.

The analysis, starts with the following assumptions

1. You have X amount of money for down payment, specified in Case 1, Down Payment Amount.
2. You have a monthly allocation of Y amount, specified in Case 1,  Maximum Allocation
3. Price home you want to buy
4. Rent you expect to pay for the house if you were to rent.

From here we look at different scenarios, change in price of the house, change in rent, change in interest rates, Expected appreciation for the house as well as the expected rate of return for money you might invest outside of the house.

You can always customize the different cases/scenarios from there, like 

1. delaying the buying decision for a few years if you think the house prices are too high
2. model an increase in interest rates X year from now, if you delay buying the house.
3. What if you decide to rent for the rest of your life.
4. Different rates of appreciation for the house
5. Different rates of return for your investment money.
6. Different inflation rates for rent and other expense.

The following is just a read only view of the document. If you want to try modifying the scenarios and the values, you want to use the above link to the original google sheets, make your own copy or download it to Excel and you can make modifications and try out different scenarios.

Use the Inputs tab below the graph to try out different scenarios.

Have fun trying it out. If you have any questions or corrections on the formulae I have used or  suggestions to improve, do let me know.

Copyright (c) Sarvi Shanmugham

My Quantopian Notes

While working on Quantopian.com, I have had to do a fair bit of learning in various areas

1. Learn Statistics
2. Operating on Data in Pandas
3. Trading Models

This is my attempt at documenting some of my learnings for my own and others.

Fundamentals Data Operations

Calculate Z-Score for specific fundamental fields in the Quantopian fundamentals DataFrame.

from scipy import stats

#Get rows you want

d=fund_df.loc[['pe_ratio','ev_to_ebitda']]

#Transpose it

d=d.T

#Drop NANs

d=d.dropna()

#Apply stats.zscore on that data

d=d.apply(stats.zscore)

#transpose it back

d=d.T

#rename the columns as needed

zscore=d.rename({'ev_to_ebitda':'ev_to_ebitda_zscore','pe_ratio':'pe_ratio_zscore'})

Calculate Z-Score for specific fundamental fields in the Quantopian fundamentals DataFrame, but do it group-wise by Morning Start Sector Code

from scipy import stats

#Transpose it

d=fund_df.T

#Drop NANs

d=d.dropna()

#Groupby Morning Star Code

d=d.groupby('morningstar_sector_code')

#Get rows you want

d=d[['pe_ratio','ev_to_ebitda']]

#Transform with stats.zscore on that grouped data

d=d.transform(stats.zscore)

#transpose it back

d=d.T

#rename the columns as needed

gzscore=d.rename({'ev_to_ebitda':'ev_to_ebitda_gzscore','pe_ratio':'pe_ratio_gzscore'})

Add these rows to back to fundamentals data

fund_df = pandas.concat([fund_df,zscore,gzscore]) 

Copyright (c) Sarvi Shanmugham

My EC2 Theano Keras Cluster Development Setup

Setting my EC2 environment to work on Machine Learning using GPU acceleration took a bit learning. Setting up EC2 was simple. I had to figure out how to do the following

1. Setup a EC2 cluster of nodes
2. Make sure there is a shared storage in EBS where the home directories are stored. Storage that will persist and be reused across multiple EC2 cluster starts and stops.
3. Setup the the networking between them that so they can talk to each other and have passwordless SSH between nodes in the cluster
4. Set them up to use their GPU, Theano and Keras
5. Set the master up for GUI Desktop so for developer convenience

So I thought I should document my steps for my own use in the future. But hopefully this will help others who come looking for a guide, just as I was a few days ago.

This is my development setup. I plan on building Machine Learning Models and run them on the GPU and eventually run it on a cluster of GPU. So I am planning ahead to make sure I have all the pieces I need to do that development.

My Local Machine setup

1. Make sure StarCluster is installed and is configured to use my EC2 account.
2. That it can be used to create clusters in my region
3. Create a volume where home directories will be stored and will persist across cluster starts/stops

My Node Setup

1. Make sure ubuntu image being used is upto date and secure.
2. EC2 GPU enabled StarCluster Ubuntu 14.04 image for cluster development
3. An EC2 VPC and Security Group to bring the nodes in the cluster together and allow them to be accessible.
4. Setup passwordless ssh access between nodes in the cluster
5. Numpy, Scipy and other libraries
6. Nvidia GPU tooling
7. Python VirtualEnv
8. Theano
9. Keras
10. EC2 Instance Setup
11. XFCE Desktop with X2GO

My Master Setup

1. All the steps from My Node Setup above
2. A XFCE desktop connected with X2GO for GUI access to the master node.

Install StarCluster on your local machine, MACOS in my case

The next step is to create an EBS storage volume using a standard StarCluster enabled image, so that it is created, formated and made available.

http://star.mit.edu/cluster/docs/latest/installation.html

Follow the quick start steps at

http://star.mit.edu/cluster/docs/latest/quickstart.html

to make sure you can start a basic default cluster using

starcluster start mycluster

starcluster sshmaster mycluster -u ubuntu

starcluster terminate mycluster

EC2 VPC andSecurity Group Setup

Create your own VPC in the VPN menu, and enable the following

1. VPC CIDR: Pick a range. Block sizes between /16 to /28. Example: 172.30.0.0/16
2. DNS Resolution: Yes
3. DNS Hostnames: Yes
4. Classic Link: Yes

Add a Security Group, and do the following

1. Give it a name
2. Add the VPC to the security group
3. Edit the Inbound Rules with
1. TCL, ALL TCP, ALL, 0.0.0.0/0
2. SSH(22), SSH, 22, 0.0.0.0/0
3. ALL ICMP, ICMP(1), ALL, 0.0.0.0/0

Ubuntu 14.04 updated to confirm the Shell Shock bug is fixed

Create EC2 instance of type g2.2xlarge to start with and use the latest standard Ubuntu AMI and using the above VPC

1. Confirm linux kernel information

uname -mrs

cat /etc/lsb-release  

2. Confirm that the Shell Shock bug does not exist in this image, the following command should not say vulnerable.

env x='() { :;}; echo vulnerable' bash -c "echo this is a test"

3. Upgrade packages

sudo apt-get update

sudo apt-get upgrade

sudo apt-get dist-upgrade

4. Upgrade kernel as follows. Got to http://kernel.ubuntu.com/~kernel-ppa/mainline/ and pick the latest version of the kernel within the the same major version number.

mkdir kernel

cd kernel/

wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19.8-vivid/linux-headers-3.19.8-031908_3.19.8-031908.201505110938_all.deb

wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19.8-vivid/linux-headers-3.19.8-031908-generic_3.19.8-031908.201505110938_amd64.deb

wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.19.8-vivid/linux-image-3.19.8-031908-generic_3.19.8-031908.201505110938_amd64.deb

sudo dpkg -i *.deb

cd ..

rm -rf kernel

sudo shutdown -r now

5. Verify the shell shock bug does not exist

env x='() { :;}; echo vulnerable' bash -c "echo this is a test"

6. Create and save the AMI for future use.

Create StarCluster enabled Ubuntu 14.04 AMI

The next step is to create a StarCluster enabled image based of the updated Ubuntu 14.04 AMI we created in the previous step.

1. Create new EC2 instance with the AMI created above or continue from the last section.

2. Update apt-get sources.list to uncomment the lines that add multiverse as a source and update

sudo vi /etc/apt/sources.list

sudo apt-get update

3. Install nfs-kernel-server and dependencies along with portmap. Ubuntu 14.04 uses RPC bind, but we can install portmap and make it work. 5. Download sg6.tar.gz from the following link7. Create and save Cluster AMI. You now have an Ubuntu 14.04 Image that you can use with StarCluster

sudo apt-get install nfs-kernel-server nfs-common portmap

sudo ln -s /etc/init.d/nfs-kernel-server /etc/init.d/nfs

sudo ln -s /lib/init/upstart-job /etc/init.d/portmap

sudo ln -s /lib/init/upstart-job /etc/init.d/portmap-wait

4. Use the customized version scimage_14_04.py script from my fork of StarCluster

git clone https://github.com/sarvi/StarCluster.git

sudo python StarCluster/utils/scimage_14_04.py  

5. Download sge6.tar.gz from the following URL into /home/ubuntu/

https://drive.google.com/folderview?id=0BwXqXe5m8cbWflY1UEpnVUpScVozbFVuMERaOE9sMktrX1dFQmhCU0tLbnItUEo0VkZxZFE&usp=sharing

6. Untar it into /opt

cd /opt  

sudo tar -zxvf /home/ubuntu/sge6.tar.gz

cd

rm sg6.tar.gz

rm -rf StarCluster

7. Create and save the AMI that can now be used in a StarCluster configuration

Setup Numpy, Scipy, CUDA and other libraries

The next step is to install Numpy, Scipy, CUDA compilers and tools, etc. It is recommended to have the python virtualenv tooling to allow you have different custom virtual python environments for developing software. The following commands should be get them installed.  

sudo apt-get update

sudo apt-get -y dist-upgrade

sudo apt-get install -y gcc g++ gfortran build-essential git wget linux-image-generic libopenblas-dev python-dev python-pip python-nose python-numpy python-scipy

sudo apt-get install -y python-virtualenv

sudo wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1404/x86_64/cuda-repo-ubuntu1404_7.0-28_amd64.deb

sudo dpkg -i cuda-repo-ubuntu1404_7.0-28_amd64.deb

sudo apt-get update

sudo apt-get install -y cuda

echo -e "\nexport PATH=/usr/local/cuda/bin:$PATH\n\nexport LD_LIBRARY_PATH=/usr/local/cuda/lib64" >> .bashrc

sudo shutdown -r now

Wait for the machine to reboot, relogin and continue installation and setup as follows

cuda-install-samples-7.0.sh ~/

cd NVIDIA\_CUDA-7.0\_Samples/1\_Utilities/deviceQuery

make

The following will make sure the CUDA was installed correctly and verify that the GPU is accessible and ready for use.

./deviceQuery

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 7.0, CUDA Runtime Version = 7.0, NumDevs = 1, Device0 = GRID K520

Result = PASS

cd ~/

rm -rf cuda-repo-ubuntu1404_7.0-28_amd64.deb

rm -rf NVIDIA\_CUDA-7.0\_Samples/1\_Utilities/deviceQuery

Create a virtual python environment that has to the global system packages installed, activate it by sourcing the activation script

virtualenv --system-site-packages theanoenv

source theanoenv/bin/activate

Install the Theano environment within the virtual environment. I do it this was so that I can work on theano itself to help fix bugs in the code. I can install it in an editable format and modify its code if needed. If you have no intention of modifying or updating theano, you can install it outside the virtualenv.

As a rule, I tend to install all tools that are bleeding edge, and stuff they depend on inside the virtual python environment.

pip install --upgrade --no-deps git+git://github.com/Theano/Theano.git

echo -e "\n[global]\nfloatX=float32\ndevice=gpu\n[mode]=FAST_RUN\n\n[nvcc]\nfastmath=True\n\n[cuda]\nroot=/usr/local/cuda" >> ~/.theanorc

Make sure the theano installation works and can use the GPU. This will acquire the GPU and start running theano tests on it. This will take a while. You can interrupt it once you know the GPU is being used and atleast some of the tests are running and passing

python -c "import theano; theano.test()"

Next pull Keras the modular machine learning library that builds on theano, so its sources are available to you. And pip install the code as editable, so that your changes to the keras sources can be run, debugged and test easily.

mkdir Workspace

cd Workspace

git clone https://github.com/fchollet/keras.git

pip install -e keras/

Next setup passwordless ssh between nodes in the cluster. For this you need to copy over the key file(*.pem) that you downloaded from amazon and that you use to ssh into your EC2 instance from your local machine. Copy this over to the instance you are working with. Then ssh-add this *.pem key

chmod 644 .ssh/authorized_keys

scp -i <your-public-encryption-key>.pem <your-public-encryption-key>.pem ubuntu@<public-ip-address-ec2-instance>:/home/ubuntu/

eval `ssh-agent`

ssh-add <your-public-encryption-key>.pem

Next verify that you can do a passwordless SSH, by trying an ssh into the same EC2 instance through its local IP address.

ssh ubuntu@<local-ip-address>

At this stage, you have everything installed and configure for working with GPU using theano and keras. This software configure can be used for masters and slaves in the cluster.

Create a slave AMI

But this would a good point to the go to the AWS menu and create and AMI, i.e. and Image based on the software and configuration of your current instance. You can launch future EC2 instances with the AMI that you create here. I call this a slave AMI since I would like to add a GUI desktop functionality into my master.

XFCE Desktop with X2GO for GUI access

I prefer to have a master machine running a GUI deskotp, with xterms to do my development on my master node. A setup that I can disconnect and connect back as needed. Where my development environment  is intact and allows me some continuity of development.

For this I setup and XFCE Desktop, that is known for its light foot print and X2GO for remote GUI access for its low bandwidth.

Add the X2Go Stable PPA

sudo add-apt-repository ppa:x2go/stable

sudo apt-get update

Install the XFCE packages and X2Go. Feel free to add other packages, but I purposely kept this selection small.

Installing "x2goserver-xsession" enables X2Go to launch any utilities specified under /etc/X11/Xsession.d/ , which is how a local X11 display or an XDMCP display would behave. This maximizes compatibility with applications.

sudo apt-get install xfce4 xfce4-goodies xfce4-artwork xubuntu-icon-theme firefox x2goserver x2goserver-xsession

Install X2Go Client and connect with it. In the X2Go Client "Session Preferences":

Specify "XFCE" as the "Session type."

If you have the SSH key in OpenSSH/PEM format, specify it in "Use RSA/DSA key for ssh connection".

If you have the ssh key in PuTTY .PPK format, convert it using PuTTYgen, and then specify it.

Or even better, just launch Pageant (part of the PuTTY suite,) load the .PPK key in Pageant, then in X2Go Client select "Try auto login (ssh-agent or default ssh key)".

Disable Screen Saver to minimize CPU usage

Create EBS storage to be mounted on all cluster nodes

The next step is to create an EBS storage volume using a standard StarCluster enabled image, so that it is created, formated and made available. And to move the home directory, /home/ubuntu on this storage. This will get mounted as /home in clusters and hence will act storage that will be persistent between clusters that created and destroyed.

Create EBS storage volume of desired size. Use an image-id from list show in the "starcluster listpublic" command. Specify a region where you want the volume to be created, in my case, us-west-2c where GPU nodes are available and cheap. 100 being the size in gigabytes

starcluster listpublic

starcluster createvolume --name=myhome --image-id=ami-04bedf34 100 us-west-2c

Note down the volume id that is created.

You will need to temporarily configure the star cluster config file to use the just created EBS mount as a shared storage at /myhome.

VOLUMES = myhome

..........................

[volume myhome]

VOLUME_ID = vol-595b0fbe

MOUNT_PATH = /myhome

This will mount the created EBS volume onto /myhome in a star cluster master and slave nodes

Now start a new cluster and log into the master node as user ubuntu

starcluster start mycluster

starcluster sshmaster mycluster -u ubuntu

Move the home directory to mounted storage
Then make sure the .gnupg directory is owned by user ubuntu, if not, change its ownership as follows. Then tar the ubuntu directory and save it into /myhome

sudo chown -R $(whoami) ubuntu/.gnupg

sudo tar -czvf /myhome/ubuntu.owner.tar.gz --same-owner ubuntu
cd /myhome

sudo tar -zxvf ubuntu.owner.tar.gz

Now change the starcluster configuration to now mount the shared EBS volume on /home instead of /myhome, terminate and restart the starcluster. You should now have an ubuntu home directory in EBS storage that is not is not lost when you start and restart your cluster.  

Create a master AMI

But this would a good point to the go to the AWS menu and create the master AMI, with GUI desktop functionality

Updates

I plan on keeping this page updated as my setup evolves and I refine the environment.

Copyright (c) Sarvi Shanmugham