Hello Montreal! PyCon 2014 underway!

So as many of you know I went to Montreal to attend PyCon 2014! As a up and coming “Pythonista,” this is very exciting. This is the first time I have travelled internationally, and also this is my first “major” conference! Looking to take in as much information as possible in these seven short days. Overall, Montreal is a great city. Everyone thus far is very nice, helpful, and understanding; despite my ability to speak the native tongue French. The weather is similar to that of Chicago, so not much to complain there 🙂

In terms of tutorials, I have attended the following thus far. I will give a title, a general overview, how I might apply it, and general reference links. I believe all the talks are recorded, I will share this as soon as I find out more information.

 

Topic: mrjob: Snakes on a Hadoop

Speaker: Jim Blomo

General Overview:

Essentially we walked through various examples, talking about MapReduce functions. We talked about how to orchestrate our logs in a manner that provides as much information in a single line vs multiple lines; Makes writing your jobs SO much easier. Essentially, Map and Reduce should be stateless; we are looking at an individual log line at a time, and the framework will merge values back together.

Essentially:

  • map(key1,value) -> list[key2,value2]
  • reduce(key2, list[value2]) -> list[value3]

We didn’t work with a direct Hadoop instance, but rather we worked in a simulated mode, where we had all the data locally. This was smart due to the possible instability of the wifi. (which if I might add has been kick ass! Great job PyCon team!) We worked through simple page visit MapReduce, to more complex multi step MapReduce jobs (get a list of users and their businesses reviewed, and convert it back to a match users together based on Jaccard Similarity). At the end, Jim talked about how to hook it up to a AWS instance of Hadoop and how to properly schedule your jobs. The nice advantage to AWS is that its easily scalable and takes a lot of the heavy lifting of getting Hadoop setup off your plate. Also if you need to spin up extra nodes, its just a nominal fee instead of having to possibly re architecture your own. Probably the biggest talk of the day for me because I am in a position to get more comfortable with Hadoop now. Before it just looked like a big scary elephant 🙂

Plan:

Now while I don’t have TB of data on my personal machine….there are tons of big data possibilities at work! Looking to see if I can take some of this back to the office, and write some MapReduce jobs on a lot of our data logs. With all this being said, the more efficient your logs are the better results you can get (or at least the easier it is to interpret). So I think this is going to become a bigger project soon. Glad I took this course. Very informative and broke the initial “scary” barrier of big data. BRING IT ON!

Links:

 

Topic: Build your own PiDoorbell!

Speakers: http://www.codechix.org/

General Overview:

In short, we took a Raspberry Pi (a small microcomputer) and wired it to a breadboard using the GPIO pins on the board. By doing this, we can have all sorts of fun with the circuits and program some cool things. We setup some sample led’s, but I finally got the Pi to snap this picture! 🙂

visitor-photo-2014:4:9-19:52

Plan:

So I plan on installing my PiDoorbell on my home’s door whenever someone gets closer than 24 inches away. This will allow me to snap and view a video stream of who is at the door before I get my lazy butt off my third floor apartment. (Remember, developers are supposed to be lazy ;)) The implementation in class only took a phone, and uploaded to Dropbox and set a SMS via Twilio. My plan is to tweak it to use the Google Drive API, and continue to get SMS or at least email alerts to my phone. This way, I can see when packages are delivered, or stolen O_O, even when I am not home. Long term, I want to add some way to interface with the person at the door; LED bar saying “I am on my way.” or “GO AWAY!” 🙂

Links:

 

Anyways that’s all I got for now! Looking forward to tomorrow’s tutorials on Flask and Intermediate Python! As always, learning all the time 🙂

Installing cx_Oracle on a Mac

UPDATE: I have updated this article to reflect the state of cx_Oracle and MacOS in 2017!

Spoiler Alert: It is way easier and this guide should be considered outdated!

Installing cx_Oracle on a Mac: 2017 Edition

So as previously mentioned, I got a Macbook Pro. Nearly two months later, I am loving it more and more. The only gripe I had was installing a specific package called cx_Oracle; a Python Oracle DB connection package. After a lot of attempts, I finally got a working copy installed on my local machine. I noticed there are many tutorials around the web, but they are a bit outdated, I am on Mavericks, so I will create a nice article not only for you guys, but also for myself.

Necessary Downloads:

  1. cx_Oracle — Source Code Only Option — (http://cx-oracle.sourceforge.net/)
  2. Oracle Instant Client Basic 64-bit (http://www.oracle.com/technetwork/topics/intel-macsoft-096467.html)
  3. Oracle Instant Client SDK 64-bit (http://www.oracle.com/technetwork/topics/intel-macsoft-096467.html)

Next Steps:
Assuming you downloaded all the above files, and they are found in your Downloads directory, run the following commands as sudo:

sudo su
mkdir /Users/<username_here>/oracle
mv /Users/<username_here>/Downloads/instantclient-* /Users/<username_here>/oracle
cd /Users/<username_here>/oracle
unzip instantclient-basic-macos.x64-11.2.0.3.0.zip
unzip instantclient-sdk-macos.x64-11.2.0.3.0.zip
cd instantclient_11_2/sdk
unzip ottclasses.zip
cd ..
cp -R ./sdk/* .
cp -R ./sdk/include/* .
ln -s libclntsh.dylib.11.1 libclntsh.dylib
ln -s libocci.dylib.11.1 libocci.dylib
vim ~/.bash_profile

 

Add the following lines (you may want to add these lines to your own accounts .bash_profile too):
export ORACLE_HOME=/Users/<username_here>/oracle/instantclient_11_2
export DYLD_LIBRARY_PATH=$ORACLE_HOME
export LD_LIBRARY_PATH=$ORACLE_HOME

 

Once you enter all the info, you should resource your bash_profile. After this, you setup instant client as well! I would validate all your settings are correct as well:

. ~/.bash_profile
echo $ORACLE_HOME # should be /Users/username_here/oracle/instantclient_11_2 if you followed this guide
echo $DYLD_LIBRARY_PATH # same as above
echo $LD_LIBRARY_PATH # same as above
which python # usually /usr/bin/python
which gcc # usually /usr/bin/gcc

If all of the above is set, then you are ready to proceed with the cx_Oracle Install!

. ~/.bash_profile
cd /Users/username_here/Downloads
tar -xzf cx_Oracle-5.1.2.tar.gz
cd cx_Oracle-5.1.2
sudo python setup.py build
sudo python setup.py install

If all went according to plan, it should have successfully installed! (if not post in the comments and I can help!) To test simply:

python
import cx_Oracle
# you may see some junk like:
/Library/Python/2.7/site-packages/cx_Oracle-5.1.2-py2.7-macosx-10.9-intel.egg/cx_Oracle.py:3: UserWarning: Module cx_Oracle was already imported from /Library/Python/2.7/site-packages/cx_Oracle-5.1.2-py2.7-macosx-10.9-intel.egg/cx_Oracle.pyc, but /Users/username_here/Downloads/cx_Oracle-5.1.2 is being added to sys.path

Hope this helps someone out there! 😀

UPDATE:
Seems like Oracle made a change to the basic zip since this was posted, the step when you are setting up symlinks. If you used my guide previously and you did:

ln -s libclntsh.dylib.11.2 libclntsh.dylib
ln -s libocci.dylib.11.2 libocci.dylib

you should unlink your symlinks and this should be…

ln -s libclntsh.dylib.11.1 libclntsh.dylib
ln -s libocci.dylib.11.1 libocci.dylib
I have updated the article since! 🙂
UPDATE 2:

William Smith, a commenter on the article, said he still ran into an error where Oracle software could not be found. He was able to get past the issue by using “sudo -E” with when installing:

$ sudo -E python setup.py install

The “-E” flag on sudo ensures that the local environment variables are preserved with root when the command is run. Thanks! Check out his full comment: http://joelvasallo.com/?p=276#comments

Updating Git on CentOS 6

Recently I had an issue with Pycharm, awesome software totally worth the investment by the way, where it was complaining about my version of Git. Took a bit of hacking around but I finally figured out my solution.

As of this writing, CentOS 6 has a default git version of 1.7.1.  This version is what you will have installed if you run the typical install command:

# yum install git

I needed to have at least version 1.7.1.11 for Pycharm to properly work. Found this quick fix online. You need to do this from a root account, but then the process is easy!

Add RPM Forge to Yum Repos

wget 'http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm'
rpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txt
rpm -i 'rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm'
yum clean all

Install New Version from RPM Forge

cd /etc/yum.repos.d
vim (or whatever) rpmforge.repo

change the enabled=0 flag to enabled=1 in the section labelled [rpmforge-extras].

yum  update
yum provides git

This will return a longer list of available git modules.

Install the newer git by copying the FULL REPO NAME.   For example:

yum install git-1.7.11.3-1.el6.rfx.x86_64
git --version

You should have the new release installed.

Now go back and edit rpmforge.repo and disable the rpmforge-extras repository.  Then ensure the yum directory is cleaned up.

yum update

When you are done, feel free to disable the rpmforge-extras back to 0. I personally would do this to ensure you have CentOS specific releases outside of RPMforge.

CentOS 6.4, Virtualbox, and You!

So this weekend, I went ahead and updated my VM at home. With that being said, I ran into problems getting 6.4 to boot, installing Virtualbox Guest Additions, and other things. Fear not, I documented my steps to help all of you out! I documented as I resolved the issue, hope this helps you not pull your hair out!

1. Stuck Progress Bar on boot after installing new release

Deleting the /etc/X11/xorg.conf and letting the guest OS create a new xorg.conf usually fixes the problem.

  1. Press any key during boot sequence.
  2. Press ‘e’ to edit boot commands
  3. Look for a line that begins with kernel and press ‘e’ on that line again.
  4. Add a 3 to the end of the line, save, and boot.

You should now be in command line when you boot. Login to your account, or root then:

[root@localhost~]# rm /etc/X11/xorg.conf

For those interested, here is a list of run level summaries.

  • Runlevel 0 and 6: halt and reboot the machine, respectively.
  • Runlevel 1: No services running, only root can login.
  • Runlevel 2: Users can login but no networking.
  • Runlevel 3: Networking and text-mode.
  • Runlevel 4: unused.
  • Runlevel 5: GUI.

2. VirtualBox Guest additions upgrade: install_x11_startup_app: no script given

Here’s what we get after trying to upgrade the Guest Additions:

Installing the Window System drivers Installing X.Org Server 1.11 modules ...done.
Setting up the Window System to use the Guest Additions ...done.
You may need to restart the hal service and the Window System (or just restart the guest system) to enable the Guest Additions.
Installing graphics libraries and desktop services components ...fail!
(See the log file /var/log/vboxadd-install-x11.log for more information.)
Press Return to close this window... install_x11_startup_app: no script given

Before installing the new guest additions version, you need to remove manually 2 symlinks:

[root@localhost~]# rm /usr/lib64/VBoxGuestAdditions
[root@localhost~]# rm /usr/share/VBoxGuestAdditions

3. VirtualBox Guest Additions Upgrade: Building the OpenGL support Module FAILED

Try the following:

[myusername@localhost~] su -
Password:
[root@localhost~]# cd /media/VBOXADDITIONS_4.1.12_77245/
[root@localhost VBOXADDITIONS_4.1.12_77245]# export MAKE='/usr/bin/gmake -i'
[root@localhost VBOXADDITIONS_4.1.12_77245]# ./VBoxLinuxAdditions.run