My Week at Hacker School

I first heard about Hacker School last summer. I had run into my friend Lindsey Kuper at a conference in Edinburgh. She told me that she had been a resident and that I should do it too.

"You’ll like it," she said. "I can put you in touch with the people.”

I like being told I'll like things--makes it so much easier to decide. I needed to finish my thesis and get a job, but I also like to say "yes" and deal with the consequences later. One "yes" led to another and I found myself booking travel and accommodations for a week in New York in November.

Since I was busy working on my thesis and getting a job, I did not try too hard to figure out what to expect. Plus I was told I would like it. I had opened a tab with the Hacker School website and skimmed Phil Guo’s blog post on his residency. When people asked me about it, I would say words like "immersive," "free," and "retreat for programmers" and then change the subject.

When the time came, I made my way to my Soho room and then to the address they had emailed me. The second-floor open office space of Hacker School was abuzz with all the chaos and excitement of the first day of summer camp. The beginning of my residency, they told me, coincided with the first day of a new "batch." Every six weeks, a group of new students begin their twelve-week Hacker School experience, joining a group of more seasoned Hacker Schools finishing their experience. Residents are given the unique opportunity to insert ourselves for one or two weeks at a time and observe the unfolding of this social experiment.

A hidden advantage of starting with a batch was that I was initially anonymous. After I acquired my key I took my place by the breakfast spread of pastries and fruit. My PhD has trained me well for standing next to free food: smile politely; introduce yourself; do not take too much food at once. The first person I met was student who had quit a math PhD program (which he had begun at an impressively young age, I later learned) and spent the last year programming and traveling the world. The second person I met was a student who started teaching herself Lisp while working in a bar. She had just quit her dog-walking business to do Hacker School.

After the fifth person it was time for orientation. Here I finally learned the key facts. While Hacker School has hours of being "in session," students and residents all have a key to the space at all hours. Outside of check-ins at 10am and weekly presentations on Thursdays, students are responsible for their Hacker School experiences. The students decide what they want to learn and how they want to go about it--the faculty are there only for guidance and check-ins. Hacker School differs from many of these other 12-week programs in that it is not focused on teaching a specific skill (unlike, for instance, the Ruby on Rails boot camps). In addition, there is a focus on learning skills (functional programming; cryptography) rather than completing specific projects. Some students have a project or direction in mind when they arrive, but many seem to just show up and see where the experience takes them. (You can read more about why they started Hacker School here.)

Orientation kicked off a fun and perspective-changing week. On the first day, I introduced myself, stated my interests (programming languages and functional programming among them), opened “office hours” and waited to see what would happen. I advised some people on learning Haskell and Scala--and also “paired” with them for periods of time. I talked to one student who had gotten into programming through activism and organizing (I later read online that he had been quite involved with Occupy Wall Street) and introduced him to a friend working with city data. After I gave talks about the history about programming and about my work, some people asked me about programming languages and also programming languages research (verification; static analysis). One morning by chance I sat next to a student, Pedro, and we struck up a collaboration to create the Markov Tweet generator that now powers @MarkovRMS. Pedro was eighteen, from Sao Paolo, and had the coolest development environment of anyone I'd ever met.

Pedro, Libby, Fernando and I had dim sum for lunch one day.

And Pedro was only one of many fascinating people I met that week. There were students who were teenagers and students in their fifties and sixties, with grown children. Some had not gone to college; other had PhDs--one woman had studied Computational Geometry and coauthored a paper with MIT professor Erik Demaine, one of the most well-known people in our field. Some people hoped to transition into a programming job: some from non-programming backgrounds and some from having taken breaks from programming to do startups focused on other things. (From the placement of alums, it seems that students are quite successful in getting the jobs they want.) Perhaps because Hacker School gives a second chance to people who have not been programming since birth, the program is far more diverse than the typical tech company.

At Hacker School I was impressed not just with the individual students, but with the community that Hacker School has created. The students are respectful and supportive of each other while working on problems together and also while doing social activities--of which there are many. One reason this is impressive is because tech and programming culture are infamous for being "macho," "aggressive," and unwelcome to women and other minorities. In addition, it is quite an achievement to create a safe environment when the students do not have the built-in trust that comes from having superficial similarities. While many companies leverage “like me” bias to build trust in their teams, the only thing Hacker Schoolers often have in common is a baseline level of programming ability and the fact that they have left their jobs, families, and friends to focus on programming.

Photo from alum Laura Bledait's blog.

While careful student selection accounts for part of the healthy community dynamics, Hacker School’s social rules play an important role in maintaining the safe environment:

• No feigning surprise. This is when someone will say something like "I've never had sushi before" and someone else responds with, “Really? I didn’t know there were people like that.” This exchange makes the second person feels superior and the first person self-conscious.
• No well-actuallys. A "well-actually" is when one person states a fact and someone else jumps in to correct the person on a minor technicality. Especially when people feel vulnerable and/or struggle with Impostor Syndrome, well-actuallys are much more harmful than they are productive.
• No backseat driving. Nobody likes backseat drivers.
• No subtle "isms." Hacker Schoolers are discouraged from engaging in sexism, racism, classicism, homophobia, etc. To avoid flame wars, people are also encouraged to take someone's word for it--without discussion--if they say an "ism" has occurred.

There is something to be said honest and open discussion and feedback, but these often require a level of trust and respect--both of which take time to build. Given that Hacker School brings together such a diverse group of people for such a short period of time, these rules are surprisingly effective in establishing an environment where people feel safe and supported.

The other aspect of Hacker School's culture that impressed me was how much the students were able to learn and produce despite the lack of curriculum or explicit goal structure. The weekly presentations--and job placement of the candidates--suggest that emphasizing learning and collaboration over specific projects and deadlines can be effective. I was amazed by the diversity and quality of the projects students showed during the weekly presentations. One guy had built an app that tells you where to get a cab based on previous cab pick-ups. Two students had built a simple web framework and then built some websites on top of it.  In addition to having fun work to show, it seemed like people had fun doing the projects in the first place.

The no-pressure atmosphere of Hacker School gave me the opportunity to finally try pair programming, a practice where one person does the typing and the other person reviews each line of code. To help with motivation and to prevent students from getting stuck, Hacker School encourages all students to "pair" on projects. Due to deadline pressure and perceived effectiveness of splitting up tasks my partner(s) and I had always agreed on interfaces and worked separately. With Pedro I discovered how productive it was to have two people look at code at the same time: for catching syntax errors, for finding deeper bugs, and for discussing how to construct the program. Working on code with someone else also allows you to finally have inside jokes about your code with someone besides yourself.

In short, Hacker School is a magical place. For those in a position to get involved, I highly recommend it. For everyone else, it's worth keeping up with what Hacker School is doing--how they are questioning our assumptions about how we learn and work. Hacker School showed me the value in being intentional about culture. It also showed me the value of taking time from my deadline-driven, task-driven life to learn--and play. Also, when I have my own research group, I am definitely going to encourage people to pair on programming and other projects.

Perhaps it was for the best that I did not try to understand what Hacker School was before I experienced it. There is nothing else quite like it. And it might be just what we need to build a more inclusive tech culture.

Experiment: Daily GitHub Checkins

I've been doing a lot of relatively mindless but decently labor-intensive code-related work (colloquially known, especially in the brogrammer community, as "coding bitch work"). I've been building up some web-based case studies in my Jeeves programming language. I've also had to take over some student code. Taking over this code was particularly painful because of all the managerial regret I felt: regret about not having made them document better, about not having made them do more work. The takeover process has involved a lot of commenting, test-writing, and the occasional small extension to test that I really know What's Going On.

Anyway, to try to mitigate the pain of these various tasks, or to spread it out and prolong it, I've decided to break from my usual model of nothing-nothing-nothing-OMGdeadline and do a small task every day that I work (which, note, does not include all days), big enough to warrant a GitHub checkin. (For those on the outside, being a computer science PhD student, at least if you're me, involves a lot of paper-reading, talk-preparing, writing, thinking, and "thinking" in addition to coding.) I hypothesized that this would be good for me to make incremental progress on some things that just aren't fun to do, as well as improve the general documentation state and cleanliness of my code and tools. I get pretty obsessed with arbitrary routine, so it's worked out decently well so far. (Check me out.) This policy has definitely made me write some documentation and tests I otherwise would not have written. (Although my pseudo-officemate Joe would argue that this is not "real work.") I'll report on things after we hit "OMGdeadline" and let you know how well it worked.

In the spirit of doing things in smaller increments, I'm also making it a goal to do smaller blog posts instead of the Blog Essays (also see my profile on Medium) I've gotten into a habit of doing. I've dramatically curbed my email habit (I wrote a thing here), so maybe these more frequent blog checkins will give my pent-up words somewhere to go.

Dual Booting Windows 8.1 and Ubuntu 14.04

It appears that Windows remains ahead in this operating systems arm race: dual booting with Linux has become even more difficult. Here are some updated instructions from the last time I dual booted, in an "idealized order" I have inferred through my various failures*.

1. Shrink the size of your Windows partition and create a new simple partition for your Linux installation to go in. (More.)
2. Get an Ubuntu image onto a DVD or a USB drive.
3. Turn off Fast Boot in Windows. (More.) If you don't do this, your system is going to boot straight into Windows every time.
4. Disable Secure Boot in your BIOS. (More.)
5. Enable UEFI and disable Legacy Boot in your BIOS. (More.) I'm not sure why this has to happen, but my Ubuntu Boot-Repair kept failing until I did this.
6. Boot from your image. (If you haven't turned off Fast Boot, you might discover that there are new ways of doing this in Windows 8.1. But you should have turned off Fast Boot.)
7. Follow the instructions and install Linux onto the partition you've set aside for it.
8. Run Boot-Repair to reinstall your GRUB.

After these many steps, you should be able to enjoy the pleasures of dual boot. Enjoy.

* I found this post to be quite a helpful resource during the process. Because I somehow still kept failing, I felt that my shorter summary may be helpful for people who, like me, thought they didn't need such a detailed step-by-step.

Run Your Research Demo Site on the Cloud

Last week, Travis Hance and I spent hours wading through the many blog posts of the internet to figure out how to set up a simple website on Amazon EC2 using our Jeeves language, which runs on Python and C++. Because we want to spare you this trouble, we put together this definitive* post for people who want to run the simplest possible research demo site on Amazon EC2. We cover the following:

1. How to set up an Amazon EC2 instance and SSH to it (to the install and configure whatever you like).
2. How to set up and configure an Apache web server on your Amazon EC2 instance.
3. How to set up your database and what to do if you want to host your own database on your Amazon EC2 instance.
4. How to configure virtual hosts on your Apache web server if you want to use the same server to host different projects on different subdomains.

This post assumes you have experience using Django and testing things on your local machine. We're using Django 1.6.2 with Apache 2.4.9. These instructions are tailored for an Ubuntu instance, but they probably generalize as well.

Is Amazon EC2 for me?

The first thing to do is to determine whether you need to run your own EC2 server. Amazon's Elastic Compute Cloud (EC2) gives you elastic compute in the cloud. The biggest win is you can easily change how much capacity you have with minimal friction. It's also just a nice way to host servers without managing your own physical machines.

If you just need vanilla Django hosting, then you should probably find some other hosting service that can manage things for you. In our case, we wanted to use the Z3 SMT solver, which runs on C++, so we needed to run our own server.

Fellow CSAILers may be interested to learn that I have also set up a mirror site on our department's OpenStack cloud. This is free for people in our department and is useful if you don't need permanent cloud data storage.

How to set up a cloud instance.

Once you decide you want to set things up on EC2, it's pretty easy to get started. As of the time we signed up, there is a free Linux tier that gives you 750 hours at no cost. Amazon recently announced further price cuts, so the situation may be even more exciting by now. To set up your own Amazon EC2 server, sign up here and follow the i nstructions for launching a new instance.

SSHing to your EC2 instance.

In order to SSH to your instance, you will need to set the permissions of your servers to allow this. You can do this by going to your EC2 management console and adding your IP address (or all IP address if you want to live on the edge) to the "Inbound" list of allowed SSH addresses.

You'll also have to use an RSA key, which you should have generated sometime during the setup. Go to the "Instances" tab under your console to get the public DNS name. Then you can SSH to your instance:

ssh -i [location of your RSA private key] [username]@[public DNS name]

For Ubuntu instances, the username is "ubuntu."

 

Installing software.

Congratulations! You now have root access on an EC2 instance. You have the freedom to install software the way you would on any other machine. You can check out a copy of your code, as well as everything you need to run it, this way.

How to run a web server.

We'll be describing how to use the Apache HTTP web server for serving websites off your machine. To run your server, first download Apache and the WSGI (Web Server Gateway Interface) module for interfacing with Python programs.

sudo apt-get install apache2 libapache2-mod-wsgi

Once you have done this, you should be able to access the Apache configuration file in /etc/apache2/apache2.conf. This file tells a webpage how to interact with Apache, by describing for instance how paths should be resolved.

To make sure your Apache server knows about your demo project, first you'll want to set your Python path and alias your / path to wherever your WSGI configuration file is.

WSGIScriptAlias / /home/ubuntu/srv/testproject/testproject/wsgi.py
WSGIPythonPath /home/ubuntu/srv/testproject

You'll also want to add "Alias" entries for the static/ and media/ directories:

Alias /static /home/ubuntu/code/jeeves/demo/conf/static
Alias /media /home/ubuntu/code/jeeves/demo/conf/media

Finally, you'll want to add a "Directory" entry to set the permissions for the directory where you'll be serving your Python files from.

<Directory /usr/share>
  AllowOverride None
  Require all granted
</Directory>

To put these changes into effect, restart your Apache server:

sudo /etc/init.d/apache2 restart

You'll also want to change the permissions of your static/ and media/ directories to make them owned by the www-data group.

sudo chown -R www-data:www-data path/to/static/
sudo chown -R www-data:www-data path/to/media/

Now everything should work! Go to your hostname in the browser and see for yourself. Okay, so it is likely that there were some configuration errors and you get a "Bad request" or other error. When this happens, it is helpful to check your Apache error log, which can be found in /var/log/apache2/error.log.

Oh, and for Apache configuration files: a gotcha is that order matters, so for redirects you should put the most specific first and the most general last. A consequence of this gotcha is that if you have aliased '/' and you already have a Directory entry for '/', you need to move this to be after the Directory entry for the directory aliased to '/'.

 

Setting up your database.

If your Django application uses a database, you'll want to hook that up as well. Django has pretty good documentation for how to edit your settings.py for the database of your choice. You may need to install Python-specific libraries for interfacing with these databases. For instance, for MySQL you will want to install the python-mysqldb Ubuntu package. Once you have configured your database settings, running "syncdb" will set up your tables:

python manage.py syncdb

We found that our site ran much faster if we hosted the database locally. We followed the standard instructions for installing and running a MySQL database. For those who have never done this before, here is what you should expect to do:

1. Install MySQL server.
2. Configure your server by, for instance, setting a password for the root user.
3. Start your MySQL server.
4. Create a new MySQL database for use by your web application.

EC2-related: if you want to be able to access your database through SSH from other hosts (for instance, to back up your database from elsewhere), you will need to add a SQL entry to your security settings permitting access from the allowed IP address(es).

 

Getting ready for production.

Now you are ready to go! For your website to look the most professional, you will want to set DEBUG = False in your settings.py file. Once you do this, you will need to make sure the ALLOWED_HOSTS list includes your domain. An easy way to do this is to add the host '*' to the list.

And make sure the secret key you use in production is secret! 

 

How to host multiple projects on one server.

You might want to serve multiple demos, each with their own Django projects. There are a couple of ways to do this. One is to do the appropriate aliasing in your Apache configuration file for different subdirectories (For instance, example.com/project1.). If you go this route, you will have to make sure your redirects, includes, etc. point to the right place.

Another option, the one we took, is to use virtual hosts to put each project on its own subdomain. Here is how to add each new virtual host:

1. Add a VirtualHost entry to your /etc/apache2/sites-available/[site name].conf file. For the main site the file is 000-default.conf.
2. Enable this site:
   

   a2ensite [site name]  

3. Reload your Apache configuration:
   

   sudo /etc/init.d/apache2 reload
   

Here is my VirtualHost configuration for jconf.jeeves.csail.mit.edu that lives in my /etc/apache2/sites-available/jconf.jeeves.csail.mit.edu.conf file. This post is getting long so I'm getting too lazy to explain all the parts, but you can see how I'm specifying paths, aliases, listening on port 80, and all that good stuff.

<VirtualHost *:80>
    ServerName jconf.jeeves.csail.mit.edu
    DocumentRoot /home/ubuntu/code/jeeves/demo/conf

    WSGIDaemonProcess jconf processes=5 threads=1
    WSGIScriptAlias / /home/ubuntu/code/jeeves/demo/conf/wsgi.py
    ErrorLog /var/log/apache2/jconf-error.log

    Alias /static /home/ubuntu/code/jeeves/demo/conf/static
    Alias /media /home/ubuntu/code/jeeves/demo/conf/media
    Alias /logs /home/ubuntu/code/jeeves/demo/conf/logs

    <Directory /home/ubuntu/code/jeeves/demo/conf>
      <Files wsgi.py>
        Order deny,allow
        Allow from all
      </Files>
    </Directory>
</VirtualHost>

Note that if you want things to run on subdomains, 1) you will need to use your own domain (rather than Amazon EC2's dynamically assigned DNS) and 2) you need to make sure you have DNS entries for the subdomains (you need to tell someone which IP addresses you would like for these subdomains to resolve to). There are instructions here about setting up your own domain name with EC2. Instructions for mapping subdomains will vary based on domain manager. (For CSAIL domains created with WebDNS, you can create subdomains by editing your hostname file and adding aliases for your subdomains.)

 

A final word.

There are a lot of details (version numbers; deprecation; death) involved with these web things, but it is so satisfying to get everything working. And if at first you don't succeed, try, try, try again.

* This claim is intended to be tongue-in-cheek. I had told Travis that there was so much misinformation on the internet that I wanted to write the definitive blog post. He laughed because this sentiment surely motivates every other post out there.

I'm Using Python Now

I have a confession. All these years I've been evangelizing strongly statically typed languages, I've been going home at night and using Python. Not all the time. It's more like the hungry vegetarian graduate student who comes to the free lunch and discovers only meat dishes. Sometimes you have to do what you have to do.

It started with small things here and there. Because of its libraries, Python has been my go-to language for web scraping jobs. Because of its relative concision to Java and its relative principles to PHP, Python has become my go-to language for web backend programming. And as the web is getting fancier and fancier, I've been doing more of this. As the web has been getting fancier, my research has also involved more of this. Web backend programming. And also Python.

It's true. I'm now doing research programming in Python*. For my PhD, I've been developing Jeeves, a new programming language (and soon-to-be web framework) for automatically enforcing privacy policies. Jeeves makes the programmer's life easier by making the language runtime responsible for keeping track of the privacy policies policies. As this approach happens completely at run-time rather than compile-time right now, it is a great fit for embedding into Python, which does most things dynamically (at run time). And so we switched from the Scala implementation we had been maintaining for a couple of years to Python.

The initial reason for switching to Python was that it is my favorite popular language for web backends. Popularity matters: a web framework is a big piece of software. Lots of people using it usually corresponds to lots of people developing and maintaining it, as well as more documentation for how to use it. I had found both of these to be issues with Scala web frameworks: the two popular Scala web frameworks, Lift and Scalatra, have learning curves that are quite steep. The people-power behind development, maintenance, and documentation also means that it is likely that lots of people will continue using it.

Another reason for the switch was that Python is what the (MIT undergraduate) kids are learning these days. I had thought that I could push Scala upon undergraduate research assistants using the force of my charisma, but this is harder than it might sound. It can take upwards of half a semester to teach even a motivated, bright undergraduate Scala. And given that undergraduates tend to be around for about a semester, that does not leave much productive work time.

After switching to Python, I learned that playing with language features in Python is much faster than in Scala. Much of the time spent coding in the Scala implementation involved an intricate dance with Scala's type system. While it's thrilling never knowing if your new tricks will allow you to save yourself from writing type-casing boilerplate, it takes a lot of time to convince the Scala type-checker that you're doing something reasonable in all cases. Sure, people might object that writing in Python is like swimming without a life jacket or something, but if you're an experienced swimmer trying out different moves in a small body of water, a life jacket is just going to hinder you. Same effect for experienced programmers prototyping a research language...

Something else that I've discovered is that people outside of programming languages seem much more excited about using my language when I tell them it is embedded in Python. These people include computational biologists, computer scientists working in areas other than programming languages (systems; the web), and undergraduates who are trying to work on our research project. A computational biochemist I spoke to who has been using logic programming was quite excited that we were embedding our language in Python. There are already many biological modelling toolkits written in Python, he said, so he could easily envision people picking up our tool.

I'm not saying everyone should use Python instead of Scala. I still stand by everything I say in my other blog post praising Scala: Scala is a less pretty version of the ML family of languages that is potentially way more usable because of its interoperability with Java. If you're trying to do quick-and-dirty language prototyping or trying to build a web framework, however, Scala may not be the tool for the job.

I know I'm not going to be able to check my Python programs before I run them. And I know I'm never going to be able to run my programs super fast. But I've looked into the tradeoff space and made my choice. Good thing Python is for girls.

* In the past few decades, the fashion has been to conduct programming languages research using statically typed languages with strong type systems. The idea is that since we have the compute power to check our programs before our run them, it is irresponsible not to. Thus using something like Python is considered somewhat scandalous.

Dual Booting Ubuntu and Windows 8

This blog post commemorates the better part of a work day I spent installing a dual boot of Ubuntu Linux 13.10 and Windows 8 on my Lenovo X230 Thinkpad. I did not expect to have such trouble, but I did, and once I came out in the open about it on Facebook and Twitter I heard from many others who have fought the good fight--and given up. So maybe this post will be helpful to some of you.

If you're looking for quick instructions for adding an Ubuntu partition to a machine with Windows 8 installed, here they are:

1. Manually partition your memory on your Windows 8 operating system. Go to the Control Panel, go to Disk Management, and create a new partition from your main partition. (More information here.)
2. Acquire an Ubuntu installation mechanism, either from Ubuntu or by downloading a disk image and burning it onto a flash drive or CD.
3. When starting up your computer, press "Enter" to interrupt the normal boot sequence. Press F12 to get into the boot menu. Select the option to boot from the flash/CD drive. Follow the directions to install Ubuntu.
4. The Ubuntu install will mess up your boot loader and prevent you from loading Windows properly. To fix this, install Boot Repair on your Linux system and select "Recommended Repair." This will reinstall your GRUB and do some other things.
5. The final piece of what you'll need to do is disable Secure Boot in your BIOS. Windows 8 uses it to make sure the pre-OS environment is secure. You can do it by pressing "Enter" at startup, getting into the BIOS options, and selecting "Disable" for the Secure Boot option. (More here on Secure Boot and here on disabling it.)

Read on for the full story.

The first question to address is why the dual boot. Linux is non-negotiable for coding. Besides feeling somewhat like it would be a waste to wipe out Windows, Windows is pretty useful for programs like Powerpoint, SolidWorks, and the new software that came with my drawing tablet (Autodesk, ArtRage, and Photoshop). Why don't I just get a Mac, you might ask. Maybe I'm waiting for free software to get good enough that I don't have to use Windows anymore. Or maybe I just haven't... yet...

The second question to address is why the dual boot and not virtual machines. At one point I was running a Windows virtual machine on Linux to use Powerpoint. And then my Windows decided to install updates... for ten minutes... during the beginning of my Research Qualifying Exam. After that, I decided dual boot was the better way to go. I hear having Windows as the host is better, but I mostly spend my time in Linux anyway. Windows is just for the special stuff every now and then.

Now for the story. I went on the Ubuntu website, downloaded a disk image, and burned it onto a disk as I normally do, expecting to be able to boot off the disk as usual. Ha. I tried turning on my computer a couple of times, thinking the system would detect the disk and boot off of it. That didn't work, I had thought because the boot order was not in my favor. I then wondered if I could take the easy route and use WUBI, the Ubuntu Windows installer, which was also part of the Ubuntu disk image. It looked like I was getting a little bit far in my Ubuntu installation: the system told me it succeeded and I even got a pretty "which OS do you want, Ubuntu or Win 8?" screen. But every time I tried to select the "Ubuntu" option I got a black screen of death saying my \ubuntu\winboot\wubuildr.mbr file was missing. A quick internet search revealed that WUBI is not to be used in conjunction with Windows 8 or UEFI hardware. (Okay, here I'll admit I tried reburning the CD at least once before doing this...) Apparently WUBI doesn't work with UEFI, the Universal Extensible Firmware Interface, because it uses grub4dos, which doesn't support GPT (GUID partition table) disks, which is a more flexible disk partitioning mechanism associated with UEFI.

I tried a little harder to boot off the disk and discovered that pressing "Enter" got me out of the normal boot sequence and F12 allowed me to boot off the disk. It looked like I successfully installed Linux again, until I shut down and tried to enter my Windows partition. There I got an equally scary screen saying my Windows couldn't be accessed anymore. I searched the error on Google and it said that I could probably address my boot issues using Boot Repair. I installed it, and during installation it reinstalled the GRUB (GNU GRand Unified Bootloader) and told me I needed to disable Secure Boot. Wondering if the second part was really true, I tried starting up Windows without disabling Secure Boot. No luck. It turns out that if Secure Boot is enabled, Windows 8 expects it to report back on certain properties that the dual boot breaks. (More here.) I went into the BIOS, found the Secure Boot [Enabled] option on the right-most screen, and set it to [Disabled]. (Apparently Linux systems can support Secure Boot now, but--unless I'm missing something--not for dual booting.)

And... then... it... worked! Now I have a working dual-boot of Windows 8 and Ubuntu 13.10. My usage of Windows 8 has been an endless source of amusement for my office friend Rishabh. ("Why do you have to do this just to get that to work?") Perhaps this can be the subject of a future blog post.

Upgrading to Scala 2.10.0

Scala 2.10.0 came out recently. It has many nice features I haven't used yet, for instance type tags, and production versions of many nice features that I was already using, for instance type Dynamic, implicit conversions, and reflective calls. I upgraded my main project to it and found it to be relatively painless.

My upgrade involved the following corresponding updates:

• Updating my SBT version. I had been using an older version of the SBT build tool (0.7.7), so I upgraded that to 0.12.1. (Any version higher than 0.11.0 should work.) This was the most painful of all since SBT has changed the way its build files work. The new SBT build system is simple, so it was mostly a matter of figuring out which variables to set and how.
• Updating library versions. The only dependency that affected me was that Scala 2.10.0 uses ScalaTest 1.9 instead of 1.8.
• Updating compiler options. Type Dynamic, implicit conversions, and reflective calls are no longer experimental and now correspond to the flags "-language:dynamics," "-language:implicitConversions", and "-language:reflectiveCalls." I also added the "-feature" flag for a reason I can't remember. A small other thing is that I had to define a selectDynamic function in addition to applyDynamic for types extending Dynamic.
• Cross-version packaging. I still compile a Scala 2.9.0-1 binary for my other project because it uses Scalatra 2.0.0--although 2.0.5-SNAPSHOT is supposedly compatible with Scala 2.10.0. SBT provides nice support cross-version compilation: define your versions, make sure you declare the right dependencies based on the Scala version, and then just use "+ package" to produce binaries for all Scala versions. Here are my build.sbt lines corresponding to that:
  

   crossScalaVersions := Seq("2.9.0-1", "2.10.0")  
   libraryDependencies <+= scalaVersion(v => v match {  
    case "2.9.0-1" => "org.scalatest" %% "scalatest" % "1.8" % "test"  
    case "2.10.0" => "org.scalatest" %% "scalatest" % "1.9" % "test"  
   })  
   scalacOptions <++= scalaVersion map (v => v match {  
    case "2.9.0-1" => Seq("-deprecation", "-unchecked", "-Xexperimental")  
    case "2.10.0" => Seq("-deprecation", "-unchecked", "-language:dynamics", "-language:implicitConversions", "-language:reflectiveCalls", "-feature")  
   })  
  

For your versioning needs, you may want to investigate this version investigator that Paul Phillips wrote.

And finally, aquestion for my Scala-using friends: I have been hesitant to upgrade to Scalatra 2.2.0. Anybody have a pointer to a summary of the concrete changes I need to make to port my code?

Treat Yo Self: Clean Up Your Code

A couple years ago, I discovered what I thought was the shortcut to building research systems.  Forget good software engineering practice!  Forget functional abstraction!  Copy-and-paste code all over the place; modify it to fit your needs.  Thinking before coding?  So college.  After all, premature optimization is the root of all evil.

In the beginning, this worked out well.  In my first year, even my advisor told me he was impressed with how quickly I got things working.  I saw in other research code the same patterns that I was learning to adopt: monstrous tangles of functionality with scant documentation.  I have found the secret to research productivity, I thought.

A couple deadlines later, I began to feel the consequences of my actions.  Pre-deadline, systems would begin to fall apart: a patch here revealed another hole there.  Post-deadline I had no desire to go back to disgusting soups of one-off functions, barely usable in the first place and certainly not reusable.  Much time was spent either avoiding my code or writing replacement code from scratch.

In other researchers, I observed that the few who had designed their systems well were able to make quick bug fixes and extensions.  For everyone else, either their code bit-rotted in obscurity or they were a slave to maintaining their systems for barely-satisfied users.  During paper deadlines, those who had built up good infrastructure could build comfortably on previous work while others ran around fighting fires and despairing.

During the course of grad school, my relationship with my code has become increasingly important.  Clean, modular, and well-documented code (with tests!) is not only less likely to have bugs but will be useful for longer.  Clean code provides a solid foundation for you and potentially other researchers.  Modular code makes it easier to reuse parts of your code. Also, knowing exactly what your code does just feels good.

On his blog, Harvard professor Michael Mitzenmacher advises graduate students to take a day every now and then to find better tools: for organizing papers read, for recording ideas and progress, etc.  Rewriting and refactoring code has become an important part of these activities for me.  Not only is refactoring useful, but it is also a relatively low-effort way to achieve a feeling of progress*.  For me, refactoring has become a treat for working hard.

Don't tell too many people now, but I think this is the real secret to research productivity.

* The importance of the feeling of progress is a topic worthy of its own blog post.

Should You Learn Scala?

It depends on what you want to accomplish*, but the answer is probably yes.

By way of background, Scala is a programming language originally developed at the Swiss university EPFL in an attempt to apply recent innovations in programming languages research to a language that can gain mainstream traction.  Researchers keep coming up with clever ways to it easier for programmers to write correct code and to write less code, but much of the time only academic programmers benefit because these features are implemented in boutique languages (for instance, OCaml and Haskell) that are not in widespread use in industry or open source projects.

On its own, without considering features such a libraries and interoperability, Scala is a nice language.  It is "multi-paradigm," supporting object-oriented programming and functional programming idioms.  Scala allows you to define types associated with both data attributes (with classes behaving like C++/Java classes) and behavior attributes (with traits corresponding to Java interfaces).  In the way of functional programming idioms, Scala functions are first-class (which means you can pass them around as values) and Scala also supports anonymous functions and currying (partial application of multi-argument functions).  This combination of features make it possible to write Scala programs that are quite concise and elegant**.

Scala's libraries and interoperability make it a force to be reckoned with.  Scala runs on the Java Virtual Machine (JVM) and can interoperate nicely with Java code.  This means that you can use Java libraries directly from Scala code.  It is also possible to call Scala code from Java (see here).  There is a nice post about how you can take advantage of Scala parser combinators, which use higher-order functions to help you write concise parsers, to write parsers for systems otherwise written in Java.  Not only can you do all of the things in Scala, you can easily write part of your program in Scala and the rest in Java.  (This is huge for mainstream adoption potential because so much of the world's code is written in Java and so many developers are taught in school!)

Learning Scala will make you more marketable.  Many companies are using Scala these days, including LinkedIn, FourSquare, Twitter, and Quora.  The founders of Quora spent a couple of months building a Scala web framework in order to use Scala.  Martin Odersky, the man behind Scala, is now behind the startup Typesafe for supporting Scala development.  There are some nice Scala web frameworks out there, including Lift and Play.  Scala is also gaining traction in the research community.  Many more people cared about our research on the Jeeves programming language for automatically enforcing privacy policies once we implemented it as an embedded domain-specific language in Scala.

Scala is also relatively easy to learn, especially compared to other strongly statically typed languages (OCaml; Haskell).  I have previously said that Python is the most marketable language for beginners, but I am beginning to change my mind.  Like Python, Scala has clean syntax, nice libraries, good online documentation, and lots of people in industry using it.  Unlike Python, Scala also has a static type system that can prevent you from doing bad things (whereas in Python the bad things just happen when you run the program).

If you are teaching a course using Java, you may want to consider switching to Scala.  I have recently had positive experiences teaching Scala to MIT undergraduates during recitation in 6.005, the software construction course taught primarily in Java.  During three recitations I introduced how to use functional idioms, how to build an evaluator for a simple language by using data types and pattern matching, and how to build an incredibly concise parser for the tiny language using parser combinators.  You may find my notes here, along with the accompanying code examples.

If you are considering the merits of Scala over other languages, you may also be interested in reading my Quora responses to the questions "What makes a good programming language?" and What are the advantages of Scala over C++ and Haskell?.

* If you are building scientific applications, a language with more math support (Matlab; Mathematica; R) may be superior.  If you just want to do a weekend project writing a Craigslist scraper, Python may be better because it is easier to learn and has great web scraping libraries.
** Okay, Scala programs are not as beautiful as OCaml or Haskell (I have another post on ML vs. Haskell) but it's so much more practical in many ways.

Getting started with OCaml

A friend with some experience with SML asked me how to get started with OCaml, so I thought I'd write this incomplete guide.

Apparently, the internet also has a "Getting started with OCaml" guide. My guide will remain competitive by being much more concise (and therefore, incomplete).

Steps to getting started:

1. Download an OCaml compiler from somewhere. Other useful things to have are ocamlfind (which finds your OCaml libraries), ocamllex (for lexing), and ocamlyacc (for parsing). Another useful thing is OCamlMakefile, which helps you manage builds with Make.
2. Read a tutorial or two. This is a good thing to read. Also, bookmark the Index of modules.
3. Write some programs while compiling frequently. (Compiling too late in the game may cause death of the Ocaml experience.)
4. If you want to lex and parse, look into ocamllex and ocamlyacc.

Some details:

• OCaml is very similar to SML. The only different parts are loops and things like that. I try to stay away from those because they are creepy.
• The file extension is .ml; the file extension for interfaces is .mli. You don't need an interface file; without one you export everything by default.
• The contents of file somefile.ml are in the Somefile namespace, which you can open with "open Somefile." By default all of these little guys are exposed. You can hide things by creating modules and writing module interfaces. Read more about that here.

Some important things to know:

• OCaml is the future because (arguably) a lot of people are using it. It is the new big thing for scientific computing because you can actually get good performance out of it!!
• OCaml is more practical than Haskell for some things and also yields better performance more easily.
• One thing OCaml is not great for is backwards compatibility. Each new release of OCaml supposedly breaks a lot of things. From my understanding a lot of libraries change/break, causing everyone to experience organ failure.

How I learned OCaml:

1. In fall of 2006, I spent a long time struggling with the MLton SMLNJ compiler. It gave me lots of obscure errors and I cried.
2. From fall of 2006 to spring of 2007, I came to not only be master* of the MLton compiler but also the Moscow ML compiler. These compilers are not only difficult to use but also often difficult to install, so you should be very impressed.
3. After learning SMLNJ was not really being maintained anymore, I said "screw this shit" and moved on to Haskell.
4. In January of 2009, I realized** that OCaml is the future. For my next project I decided to use OCaml, which I picked up in an ad hoc ways throgh the steps above, skipping step 2 (but compiling very frequently). I am still in this step.

* This is subjective.
** Through a combination of talking to people about what people used for different things, trying to do 3D arrays in Haskell, and meeting myself at Xavier Leroy.

Haskell vs. OCaml, or the ravings of a monomanic insomniac

Until fairly recently, I was a hardcore Haskell hacker who saw little reason to go elsewhere for needs of a higher-than-C level. Haskell is beautiful; Haskell is clever; Haskell has Parsec. A couple of weeks ago, I started using OCaml (after having used ML for a couple of semesters as an undergrad) and realized that you can get a lot farther in life if you are not limited to writing pretty code*.

Since then, I've gone back an forth between "OCaml is the future" and "But Haskell is so pretty." Here are my notes on OCaml vs. Haskell.

Pro-Haskell:

• Haskell does datatype constructors right. As in, Haskell datatype constructors are functions whose types are the type of the constructor. For instance, if we have a data type data Thing = C1 Int Int | C2 Bool, we get constructors of types (C1 :: Int -> Int -> Thing) and (C2 :: Bool -> Thing). In OCaml, constructors take explicit tuples. The analogous constructor C1 would need to take something of the form ([int], [int]), not just something of type (int * int).
• Haskell does type classes right. Not just type classes, but type class related utilities like deriving. Type classes are often nice for clever things but always nice for development. Being able to derive Show and Eq gets rid of a ton of boilerplate. Not believing in function overloading makes me not believe in ML! >:o
• Haskell is way prettier. Small things make Haskell code way easier to read and maintain. (For instance, the support for defining things below functions with where instead of binding everything for let. I should not have to trade off cluttering my toplevel namespace with cluttering my function definitions!
• Because Haskell is so clean and concise, it makes you produce prettier code in other languages.
• Higher-rank polymorphism and the ability to write down my forall types is very convenient.

Pro-ML:

• You can code without being constantly clever! In my senile senescence (which is also absolutely alliterative), I see this as a huge plus. ;)
• Not having to work in monads all the time to deal with state in any kind of okay way is a huge productivity plus of SML/OCaml. Don't believe me? I have the following points: 1) 3D IO Arrays, 2) gensym**, and 3) monad transformers (yuck!).
• OCaml has very nice library support (Batteries, arrays, etc.).
• OCaml seems to have a larger user base, so there are more people doing stuff in it, more people to ask when things go wrong, etc. etc.
• Though ML code is more verbose and generally less pretty than Haskell code, it disallows a lot of the clever use of syntax that obfuscates the meaning of Haskell code.

The conclusion is that while Haskell leaves a nicer taste in one's mouth, OCaml might be better for practical purposes, especially if you want to use 3D mutable arrays. It all depends on what you want to do.

* According to some fairly high standards of "pretty."
** Generate fresh variable names, for instance "__tempvar0." Note that being able to keep some state is nice so you can just increase your variable count each time.

25 most dangerous programming errors

I found this recent article on the top 25 programming errors linked from Lambda the Ultimate. From the article:

...experts from more than 30 US and international cyber security organizations jointly released the consensus list of the 25 most dangerous programming errors that lead to security bugs and that enable
cyber espionage and cyber crime. Shockingly, most of these errors are not well understood by programmers; their avoidance is not widely taught by computer science programs; and their presence is frequently not tested by organizations developing software for sale... Just two of them led to more than 1.5 million web site security breaches during 2008 - and those breaches cascaded onto the computers of people who visited those web sites, turning their computers into zombies.

The sensationalist tone of the article may be appropriate for discussing the very real and scary subject of how programming errors affect everyone in the world. People know how to fix these things, but consumers of software seem blissfully ignorant/tolerant of unnecessarily buggy software.

Listed errors include cross-site scripting, cleartext transmission of sensitive information, race conditions, and other classic errors. There are also things like "improper initialization" and "incorrect calculation." (The descriptions can be amusing: "Just as you should start your day with a healthy breakfast, proper initialization helps ensure...")

The JY Intro CS curriculum

I have long been of the the opinion that assembly (or even bits) is the best first programming language to teach, and I have long been appalled at the practice of teaching Python to poor naive starting programmers, but recent events* have caused me to change my opinion. The introductory curriculum I propose for people who have had none to light experience is:

• Introduction to Python via requiring students to write something making good use of Python's very nice libraries for doing nasty things, such as I/O, regular expressions, and interacting with the WWW.
• Introduction to why Python is not great by requiring students to write something in Python that is very slow for large sizes. Maybe implement a minimum spanning tree algorithm or something like that.
• Introduction to C! We will write the same thing in C. After a couple days of pointer chasing and pulling hair out, we will emerge as better, stronger, programmers. Now we can run our programs 1000 times for the time it took to run our last program once!
• Introduction to the necessary evil of object-oriented programming through revealing some difficulties of practicing good data abstraction in C. Maybe have people write something like a binary tree in C and have them try to make their library be more opaque about data members. (So OO programming is not actually necessary, but might be a necessary evil in present-day CS education for practical reasons.)
• Now we program with actually cool languages that do things right. Haskell, anyone? (This might be the bonus extra part of the class that might happen after the semester ends.)

When I am a professor, students will have evolved enough such that such a curriculum will not be considered ambitious at all. ;)

Python, despite the fact that it is incredibly unprincipled, is a decent first language because it gives you great mileage. You can go very far with Python in a week; you can go very far with Python with a few lines of code. (Adam was writing code that took logged onto password-protected sites and processed/parsed the HTML for keywords to download specific files within a couple of days.) Starting with Python makes people excited about their potential power and shows them that tools can be powerful. It is, however, bad to stay in Pythonland for too long because Python enforces no good programming practices. (You get too far being a bad programmer and so become in danger of never becoming a good programmer.)

Why would you want to become a good programmer? Besides the obvious reasons ("you will get much further when you build large projects", etc.), you can't live in Pythonland forever unless you plan to live forever (and have deadlines extend forever). Though Python has nice foreign function interfaces in C, there exist times when you will want to write your own C. One of my favorite conversations demonstrating my power from good choice of language (in discussing some small simulation to solve a problem in a randomized algorithms course):

• Naive friend: Does your thing take a really long time to run for n=10,000?
• Powerful Jean: Um yeah. It takes a whole minute, maybe?
• NF: Oh. Mine has been running since before dinner.
• PJ: Haha. Should have used C.

Why I previously did not believe in any Python at all. Programming abstractions are not opaque, so many things in programming make much more sense when one understands why they are the way they are. (For instance, the difference between a linked list and array--and the reason why linked lists exist at all--make more sense when you have a feel for memory and issues regarding contiguous memory.) It doesn't make a lot of sense to teach programming with broken tools 1) things will seem arbitrary to students unless they understand why things are broken, and 2) unless you are getting some mileage out of them. (I see bits as the least broken abstraction, and then assembly, etc.) One of my first languages was Java, one of the most broken languages of all time, and things made little sense to me until I learned C.

Note. Learning to program is not the same thing as learning to think. Learning to program is a process that involves getting to know one's tools. Learning to think is a process that involves developing one's mind so that one can make good use of tools. Thinking models are, unfortunately, not yet the same as the tools we currently have, so as of now these two sorts of things should be (initially) taught separately and in parallel. As of now I lump really high-level things like Scheme with the high-level thinking stuff because 1) Scheme is so simple and nice (has only one main rule and is unityped) that it is great for teaching reasoning about recursion, continuations, and other such things and 2) Scheme isn't the best for helping people develop great programming practices (modularity, incremental testing, good abstractions, etc.).


*I have been advising my boyfriend, Adam, an economics consultant, in learning to program. So far he has written a lot of useful Python to automate his work and he has written a binary tree in C. He is the coolest beginning C programmer around because I showed him gdb, valgrind, and gprof.

How enterprises use functional languages

A really nice paper by Wadler on the usefulness of functional programs, or "functional programs used in anger."

Pitfalls of teaching Java first

In catching up on my Lambda-the-Ultimate reading, I came upon this article which talks about the problems with the current state of computer science education.

I agree with some parts of it, such as what is wrong with teaching Java as a first programming language. Some of the other things they talk about seem more idiosyncratic.

haskell does as a language should

some haskell resources:

• gentle introduction to haskell 98
• glasgow haskell compiler
• monads for functional programming

history of C++

read what bjarne has to say .

hammer factory factory

this has been going around the internet for a while, but here is a link to joel spolsky's satire on software factories. as someone said, "there are too many nouns in java."

supporting the enforcement of namespaces

yesterday i was plagued by a terrible bug for a very long time. what was happening was this: i was using a class X* that was calling some third-party library that included a class Y*. i was calling a function in X that caused the destructor of Y to be called. whenever i called this function, i would get a seg fault occurring when the destructor of Y was called. some odd symptoms of this bug were that a string destructor was being called, but there were no strings anywhere, and i did not experience the bug when i took out certain linking dependencies.

when my host finally figured out what was going on, i had to laugh because it was so incredible. it turned out what had been going on was some dependency was bringing in another class Y had contained a string. when X called the destructor for its Y, the other destructor was being called, causing all sorts of problems. this should have been something caught by the linker, but for some reason it manifested itself as a runtime error.

lessons to be learned:

• the linker does not always know best. trust nothing!
• everything should be in a namespace. such problems could have been avoided if we had a n1::Y and a n2::Y.
• C++ is a beautiful language with very good enforcement of abstraction barriers.

one of the above statements is not like the other.

*names have been changed to preserve anonymity.