The best tech looks like no tech at all

Leap Motion and the Thalmic armband let you treat technology as an extension of your body.

Really, everything should be that way. Whether I’m architecting a piece of software or uploading photos of double backflip expert skier moves to Facebook seconds after doing a 360 off a halfpipe (watch the Thalmic video), the body should be the interface. No need for an intermediate keyboard, mouse, modal text editor, etc.

Countless tech companies market their user interfaces as “intuitive.” No interface that requires you to memorize the location of a bunch of buttons is intuitive. (Think about the amount of productivity lost in our daily efforts to translate thought into keystrokes. Phone apps, I’m looking at you too!)

The effectiveness of your interaction with technology should peak at the effectiveness of the actions you want the tech to take in the real world. It should not peak at the net of your intended actions and the effectiveness lost in the time, effort and rote memorization required to exploit an interface through an imperative intermediate device. (Imperative device here meaning one that accepts sets of instructions on how to accomplish the high-level task, like “turn left,” that you send to the machine).

The best interfaces are declarative. Tell the machine to move left, to fetch, to build. Don’t drive it, direct it or implement the details.

Stories: Eight months in New York (with pictures)

Pictures here.

New York City last October was a place where the weather was still compatible with t-shirts. The subway was neither baking with the heat of summer nor flooding with melting snow from passengers’ shoes. I much prefer places where cycling is a thing and hipsters are not a thing. But New York called.

Brownstone Brooklyn in the winter is so quaint. Twentysomething artists and hipsters stroll slowly down wide sidewalks, past chunks of ice, their scarves keeping cups of small-grower-sourced coffee warm until they arrive at brunch.

I’ve lived in Park Slope and Prospect Heights, Brooklyn, (and most recently, the East Village in Manhattan). I got to know Prospect Heights well enough that it felt like home. I had favorite coffee shops (like the Penny House Cafe. the best).

I got to know Prospect Park, too. I ran the main loop in my t-shirt during a snowstorm at least once, and did more miles there than I care to count. They were serene miles with memorable climbs, like up the stairs to Lookout Hill, which led to a silent overlook where I could see all of Brooklyn laid out below, often glittering in the snow.

While all this exploring and running and AirBnb-ing went on, I attended Hacker School at Etsy in DUMBO, Brooklyn. Memories include getting much better at programming, food carts, and super nice people.

Then I started working at Tutorspree, mostly in Python and JavaScript, but in some newfangled languages too. At work, I’ve seen a guy in a panda costume wandering towards Canal Street, a street preacher with a rock band-sized speaker strapped to a backpack condemn all of Tribeca to hell (did I use range() instead of xrange() again?), etc.

Of course there’s a guy in a panda costume walking down the street. It’s New York. Don’t question it.

At one point, I tried to rent an apartment in Park Slope. You see, I wanted the landlord to certify that the rent would remain constant throughout the year (he’d inserted a provision allowing it to go up during the lease term). He was unable to understand why a tenant would want to negotiate a contract. After a bit of polite negotiation, he dropped the pleasantries, got up and stormed out of the room, cursing all the way. I laughed it off; I’d simply never seen something so New York.

One random weekday night, on a late-evening Whole Foods run, I encountered an artist-dude-roommate from my first AirBnb in Park Slope while walking through the East Village. He got off his bike and tried to hire me on the spot to revamp his online art portfolio. Only here.

Months ago, I took the wrong bus from JFK airport, and ended up on a nonstop thirty minute ride to the bus depot in Far Rockaway. This was right after Hurricane Sandy had ravaged the area; looting had become somewhat of a problem. I was wearing an orange jacket, and was waving my iPhone around trying to find out which bus to take. To give you an idea of how rough the area is even without hurricanes: it’s in red on safety maps of Queens.

My fellow passengers and I were (at least once) threatened on the subway by people who are probably axe murderers. Whenever this happens, half the car usually gets out at the next stop and switches to the next car down. Hilarious.

And finally, I’ve developed a list of a few good things here in this crazy city. There are more, but these are my favorites.

  • Williamsburg. It’s like your college off-campus house, but more wild.
  • Prospect Park on a cold winter day. It’s much more natural than Central Park, and much more Brooklyn. It’s easier to feel like the city is slipping away as you take off down a woodsy trail.
  • Central Park’s northern part, up near Harlem. I once walked from the tip of Central Park to SoHo. The terrain changed from deep forest (not kidding; Central Park has forests and lakes up in its northern reaches) to grimy, slick, loud urban streets.
  • The Vanderbilt Avenue area of Prospect Heights. Nowhere else in Brooklyn does provides for such an interesting socioeconomic case study on gentrification. Gangstas roll past in souped-up Cadillacs as a younger, more well-to-do crowd heads out for brunch.
  • The Saturday farmers’ market on Grand Army Plaza. Just go there and you’ll see.
  • The Irish Hunger Memorial. Irish grass waves in the wind below the new World Trade Center, and a few knowledgable visitors whisper haunting reminders of the Irish potato famine that drove so many people to Ellis Island, right across the Hudson River from the memorial.

What’s your New York story?

ProblemMuncher: Maybe we should tackle open problems together

There was a passionate discussion on Hacker News recently. A math-loving software engineer wanted to play with equations and explore the reaches of the symbolic systems he really enjoys, but is instead working day-in, day-out on something that gives him a paycheck, a place to live, and a few minutes of free time for personal growth at the end of the day.

It’s the classic “I’m working all the time, but I could produce so much more happiness and possibly output if I had the freedom to do so” problem. It’s mainly a first world problem, but not only a first world problem, because a solution could really help people in other parts of the world who have the same problem but on a larger scale.

The questions I ask here are: Could online communities dedicated to solving open problems actually solve those problems? Also, could they help reduce economic friction in the labor market by producing a closer match between workers and skills?

Let me know what you think about this idea as a way to enable more people to work on open problems in math, computer science (including subfields like artificial intelligence) and other formalized fields and get paid.

This is a potential way to solve the friction between what workers are doing day-in, day-out and what they could be of doing, if freed from the need to make money in some field that isn’t their primary interest due to some form of economic inefficiency.

Those inefficiencies might include the research institutions’ tendency to focus on whatever is getting grant funding at the moment, and their failure to coordinate large-scale attempts to solve long-open problems.

ProblemMuncher: Munch on delicious open problems for fun and profit

I have this notion that there should be an online community that formalizes the search for solutions to open problems. Each open problem has directions and attempts. Proof verficiation software validates attempts, and senior community members discard fruitless directions.

Let’s say you want to tackle a hard unsolved math problem. Import all of the research directions to date, and log the attempts. Recursively log directions and attempts, and build up a tree of research into the problem.

Generous organizations and governments set up very large prizes for successful solutions, and when a solution is discovered and verified, contributors are rewarded according to merit. The person who provides the proof gets the largest prize, and the folks higher up in the tree for that problem who defined new directions get some money, too.

Research will accelerate, so it’s worth the money. It doesn’t solve the problem of tedious work in general - it only solves it for the specific case of workers who, due to friction and the incentive structure in the economy today, are excited and motivated to use their skills to do more good for the world than their 9-5 grind allows.

I think ProblemMuncher should probably exist anyways, just because it would be awesome to get brilliant people in industry involved in solving open problems.

Future directions

Add an automatic distributed isomorphism finder that suggests potentially fruitful new directions :-)

Generating software service clients from a taxonomy of intents and permissions

Or: An open-source ‘app store’ for software as a service, with an ecosystem full of thin or thick clients capable of materializing beautiful user interfaces from a standard way of representing application object permissions and the set of possible actions in that application.

In this article, intents are actions you can take in an app, like create, read, update, delete, merge, notify, etc.

Design is about what you can take away.

And now, having spoken of the men born of the pilot’s craft, I shall say something about the tool with which they work-the air-plane. Have you looked at a modern airplane? Have you followed from year to year the evolution of its lines? Have you ever thought, not only about the airplane but about whatever man builds, that all of man’s industrial efforts, all his computations and calculations, all the nights spent over working draughts and blueprints, invariably culminate in the production of a thing whose sole and guiding principle is the ultimate principle of simplicity?

It is as if there were a natural law which ordained that to achieve this end, to refine the curve of a piece of furniture, or a ship’s keel, or the fuselage of an airplane, until gradually it partakes of the elementary purity of the curve of ‘a human breast or shoulder, there must be the experimentation of several generations of craftsmen. In anything at all, perfection is finally attained not when there is no longer anything to add, but when there is no longer anything to take away, when a body has been stripped down to its nakedness.

- Antoine de Saint-Exupéry, Wind, Star and Stars (as translated by Lewis Galantière - more)

Background: Things as they are

The most important thing software-as-a-service service providers do is manipulate data on the backend. Value creation happens on the backend. The modern web app doesn’t trust clients running in the browser; instead, it acts as a black box that can intake data through an API. In the most modern Javascript+HTML5 clients, the internal API looks exactly like the external API client. This saavy API design means faster time-to-market for web products: Service providers can create just one API. Service providers’ on-staff developers create clients using the same API that customers use interact with the service from remote machines.

For example, providers from Google Analytics to Stripe to Gmail all provide services, and create most of their value on the backend. Stripe processes payments and deals with banks so you don’t have to. Google Analytics crunches numbers received from your website. And Gmail gives you secure, available email.

Past attempts at abstracting away the interface to 3rd-party services

The Web Intents project (http://webintents.org/) is aimed at consumer-facing apps that have a small number of possible actions. It includes no description language for object permissions.

There are more, and I may add them to this post a bit later on. As professor of my once wisely noted, “everything that is old in computing is new again.”

Abstracting away user interfaces

An app is really just two things. First, it’s a set of object permissions. That is, you can create a comment and update your own comments, but not others’ comments. You can create Groups, which hold Servers.

The old school of web application development encouraged an overemphasis on security through obscurity. This involved hiding references to server-side objects, lest someone discover an ID to an object they don’t own, and try to exploit that information to their advantage.

The most competent developers I know feel most confident relying primarily not on security through obscurity (which has its uses on the backend for sure) but in the difficulty of finding the prime factors of a large integer (see public-key cryptography) - in mathematics.

Firebase is the first step towards a future in which developers rely on clarity of security rules, and on the computational difficulty of breaking hash-based and public-key-based crypto, instead of on scattered protection mechanisms and obscurity only. Firebase makes this point in a brilliantly simple way in their docs:

Another important benefit of the Firebase security model is that it places all of your security logic in one place rather than sprinkling it around your code. Firebase then ensures that your logic is enforced consistently across all parts of your app, regardless of how data is accessed. This makes it very easy to reason about your app’s security or do formal security audits. The rules also allow for static analysis, meaning mistakes can be caught immediately when new rules are uploaded.

Where we should be going: The abstraction developers need most

Imagine a software ecosystem full of software-as-a-service providers who develop useful, value-creating backends that connect to any number of ‘server intent clients.’

Let me illustrate this future. I might want to install a monitoring service on my server. I use a (web or desktop-based) ‘server intent client’ to search for monitoring services. I type in ‘server monitoring’ and see a Python Package Index of Node Packaged Modules-type list of available services. It’s kind of like an app store, except anyone can build the app store because the interface used to connect to the app store is maintained by a community working group.

Each service intents client takes two things. First: A (JSON?) representation of object permissions (kind of like Firebase object permissions, but descriptive enough to allow for programmatically deducing what kind of interfaces would work well). Second: The set of possible actions an application may perform.

Here is an example. Let us say that I want to develop a server monitoring or security application or analytics service with all kinds of machine learning and cool, valuable stuff that clients will pay for. Well, I have no need to develop a frontend: I want to leave that to the frontend experts who are developing the service intents clients. Those clients do one thing - display object permissions and the set of possible intents for each app in a few simple, beautiful ways - and do it better than most web service or web app developers ever could.

If the grammar for the service intents protocol were English, I might type something like:

‘Servers have many groups. Servers and their children are scoped to owners. Owners can create, read, update and delete subusers; subusers are scoped to owners. Subusers can edit server names. Owners can take all analytics actions. Subusers can take all read-only analytics actions.’ Then, I’d be done with my intents protocol statement, and I’d build the most valuable part of my application: The backend algorithms that spit out the results my customers are paying for. The customer loads up my intents specification into a beautiful service intents client, and choose between drag-and-drop, autocomplete, command line or any number of other slick user interfaces for interacing with my service.

Why would a customer want to choose a user interface? A corporation might want to standardized its interface across many disparate applications that it pays for.

A friend points out that developers should be able to include a predefined interface. Perhaps developers can optionally specify the way service intents clients should handle 1-M relations, for example, by passing ‘drag_drop=True.’ But service intents clients would be responsible for interpreting the app beyond that.

That’s it’s; I’m done with the plumbing. Now I can move on to value creation by coding up the actual backend.

Primary Benefits to Society

There’s a big community of designers who may benefit from being able to focus on a number of competing open-source service intents clients that won’t be deprecated when a company is acquired or when a product is sold off.

Proprietary service intents clients might arise for very specialized needs like corporate apps, where employees need standardized training. Maybe the Microsofts of tomorrow make money selling service intents clients for hospitals, governments and banks.

Regular developers flourish by being able to create value extremely easing, and by sending their products out to service intents app repositories that act as distribution channels.

Why you should want service intents

In the future, software services take in data through APIs only, makes life easier for customers by manipulating that data, and return output through APIs only. Individuals and companies who want to make their lives easier download their favorite service intents client and get their favorite types of user interfaces rendered on the fly.

This is the ultimate abstraction over today’s software-as-a-service market: Service intents let developers focus on creating value by reducing web services and web apps to their essential functionality (or rather, to strings in a language representing that functionality), eliminating the need to create user interfaces. Also, because service intents make it easy to avoid overemphasis on security through obscurity and easy to rely more fully on security through cryptographic authentication and easy-to-specify authorization, it’ll be harder for developers to make common security mistakes or overlook security.

The way to get people to care about security is to make it implicit in the development process - to make it so obvious and easy to manage that you literally can’t create a functioning app without specifying correct object permissions. (It probably wouldn’t be long before the community created an object permissions validator as part of the software developers use in developing web services and web apps.)

Side Benefits

Service intents clients and web service or web app developers can be asked to support a few of the most popular federated authentication methods, like Mozilla Persona. This more or less solves the problem of having too many passwords and accounts to manage, because we’ll actually live in a world where we can make use of great software without running into web services or web apps that require users to create email+password accounts.

Let me know what you think about this. I typed it all up in the last hour or so, and I’m certainly not stuck on using JSON as the service intents definition language! I’m also very open to choosing a different or better name than ‘service intents’ for this project.

On radio and the American public conscience

The following is a note about how patterns of speech can make radio soothing and seemingly rational. It’s interesting to contrast the tones of expert or academic callers into NPR programs with the tones of the hosts and callers on political radio. The former is all about reflection, and the latter about both reflection and emotion.

Each scholar or expert on a Sunday morning talk show is an emissary from some sector of the public conscience. In between pauses, wedged between the “uh”s and “well”s and other indicators of spontaneous speech not normally included in the lexicon of intellectual public discourse, we hear about models of the world that are coherent and true in the speaker’s mind, and we know that some part - some people - in our collective conscience empathizes with this explanation of events.

Here, on radio, we can find verity not in the correspondance of these callers’ models of the world with reality, but in the intent of their spiels. Expressed in pausing, careful tones, we can find that this isn’t the way the law used to be, or isn’t the way people are meant to be treated, and that there is some better way, if only we too would listen and think in pauses and repeated instances of reflection, like those thinkers who dial into the American conscience each weekend to contribute their well-founded but dispassionately argued-for ideas.

Maybe the verity we seek is not to be defined in terms of the correspondance of the models of social science, many of which can be simultaneously mutually exclusive and entirely explanatory, to reality, but in terms of the way we talk to each other about issues. In reflective fits and starts, like experts calling into a Sunday morning talk show.

Building a BitTorrent client: Patterns, Modules and Algorithms

I just finished up my most recent Hacker School project, a BitTorrent client written in Python. Here’s a technical walkthrough of what’s involve in building a client for the popular peer-to-peer file-sharing protocol. You can check out the code here.

What is BitTorrent?

BitTorrent helps people download files more quickly by splitting them into pieces. Various computers called leechers and seeders join a BitTorrent swarm in order to download a data file. A computer, or peer, may be either a leecher only, a seeder only, or both. Every time your computer connects to a swarm, it becomes a peer. We’ll call the program peers use to access the swarm the BitTorrent client.

Pieces are, in turn, split into blocks. Peers request blocks from each other (leeching) and deliver those blocks (seeding). But peers only advertise that they have entire pieces. For example, a file might contain 10 pieces, each with 2 blocks, for a total of 20 blocks.

To be efficient, a BitTorrent client needs a downloading strategy to choose which parts of the file to download first. Because the information available to the strategy designer includes little about blocks, but does include which peers have complete pieces, piece downloading makes for some interesting algorithmic problems. Coming up with a good piece download strategy was my favorite part of the project.

Because of the dynamics of network traffic - I’ll post graphs later to explain this in more detail - downloading smaller pieces from a bunch of seeders is generally faster than downloading one large piece (or file) from a single computer. So people use BitTorrent to download large files.

What’s involved in building a client?

A BitTorrent client needs to read the metainfo file, which ends in .torrent. The metainfo file contains the information about how the data file (the file you will be downloading) is to be split up by the swarm. It also contains the tracker URI for fetching the list of peers that are currently part of the swarm in which you’d like participating in leeching and/or seeding. The metainfo file is encoded in a format called bencode.

The client must fetch the list of peers from the tracker, connect to some or all of them, and handshake with them. If the client has any pieces, it should let peers know by transmitting a bitfield immediately after the handshake. Peer should do the same by sending their own bitfield.

Interesting detour: Implementing bitfield parsing

A bitfield is, loosely, a string of bytes. Each bit in this string, starting with the leftmost bit at i=0, represents possession (1) or absence (0) of the ith piece in the list of pieces given by the metainfo file. In order to update the state of my client based upon the information received in incoming bitfields, I wrote some interesting bit manipulation logic that makes use of generators.

In Python, generators are functions that return generator objects. Generator objects return results at each call to yield. Basically, upon a call to yield, a generator function’s execution gets frozen until the caller decides to get the next value. It’s possible to iterate over the yields of generator objects. This is what the following expression does when join() is called.

image

(Note that in my code, _bits is also a generator function. This is irrelevant to the above line, and was just a really convenient way to output converted individual bits in a bitfield.)

The bitfield is at most floor(number of pieces in the data file / 8) + 1 bytes long, since there is a bit for every piece. But empty bits in a byte must be filled with either a 1 or a 0; the BitTorrent network protocol specifies that any partially-full bytes in a bitfield are zero-padded on the right. So there will be (number of pieces + 1) bytes in the bitfield if the number of pieces is not a multiple of 8.

Now we must turn each bit in the bitfield into a literal 1 or 0. Here’s a really simple way of doing this. First, we use ord() to get the value of each byte (this will be an integer between 0 and 255). Then, using the variable i as a counter, we can shift the byte right by i bits and immediately yield the bitwise AND of that result and 1. To see why this works, let’s look at the number 45, or 101101 in binary.

Shift it, pop it, then AND it

Shifting 45 right by 1 “pops” the 1 off of the right side of 101101, so we are left with 10110, or 22. ANDing 22 actually gives the value of the rightmost bit (since the second operand of AND is only 1 bit long, bitwise AND returns 1 if the last bit is 1, and 0 otherwise). Here, that’s 0. Just repeat this process 8 times and you’ll have shifted, popped and ANDed your way to 8 yields’ worth of literal 1s and 0s.

image

Once I have the bitfield converted to a string literal 1s and 0s, I can let my in-memory representation of a peer know whether (1) or not (0) this peer has a piece. I’ll get into what decrease_rarity does a bit later.

image

Project Components, Design Decisions and Cool Algorithms

There are three main parts to my project: the reactor, for doing message sending and receipt in asynchronous fashion; the message handler, for interpreting and constructing wire messages in the BitTorrent protocol; and the data coordinator, for reading and writing the stuff we’re downloading to the hard disk.

In this post, I’d like to walk you through each part. For the reactor part, I’ll talk about architecture and design decisions. For the message handler part, I’ll share a few useful Python modules I learned about in the course of this project. And for the data coordinator bit, I’ll share some algorithms I used to get good performance from my client.

1) The reactor. I wanted incoming and outgoing BitTorrent network traffic to any one peer to be written to or read from the network card only when that peer’s socket was ready for reading or writing.

This approach to I/O is asynchronous with respect to message production and sending. On the read side, it prevents an application from having to wait for incoming data on one socket when other sockets might actually have data.

The reactor pattern was a good fit here. For sending messages, I could queue up messages on my in-memory representation of each peer, and pop one or more messages off when that peer’s socket was ready for writing. I’d already used Twisted’s reactor pattern in an IRC bot (one of my earlier Hacker School projects). But I wanted to do this bit myself, so I built my own event loop using Python’s select module.

Python’s select module is a wrapper around the POSIX select API, and its select() method has basically the same behavior as its C equivalent.

Code here.

2) The message handler. I challenged myself to come up with a clean, generic way to parse BitTorrent protocol messages.

I was able to specify each message’s name in string format just once, and search for a method on my representation of peer objects (a Peer class) with that name. It’s actually a one-liner:

func(gettattr(self.parent, msg_name))

@classmethod: not always useless

Python’s @classmethod decorator turned out to be useful here. @classmethod lets you pass a method’s class as its first argument instead of the usual class instance (i.e., self). @classmethod allowed me to keep my DRY representation of the messages types tightly coupled to the only class which actually needed that information - WireMessage.

image

image

Dealing with binary data

The struct module was a big help in the message handler. It is useful for extracting various representations of data from strings whose contents are encoded in base 2, or binary. It turns out that BitTorrent encodes the messages peers and clients send to one another in binary. Or, to be more precise, BitTorrent generates data with no character encoding. The meaning in the data comes from its base 2 representation.

The code below uses destructuring to pull a four-tuple out of struct.unpack.

image

First, I use Python’s slice notation ([first:last:step_by]) to grab the handshake from the buffer recv() has written to. Let’s examine the argument to struct.unpack - BXs8x20s20s, where X is a variable-length string. If X is 10, unpack() will expect a string like this:

<1-byte unsigned char><10-byte char array><20-byte char array><20 byte char array>

As explained on the Format Characters section of the struct module docs, expected_length can be interpreted as an integer with a maximum value of 255 (sum(2**i for i in range(8))).

3) The data coordinator: A Torrent class. I decided to create a Torrent class to handle operations related to pieces, blocks and files. I’ll share the most interesting parts here.

In the BitTorrent protocol, a client attempts to download from a peer whenever two things are true. First, the client must be interested in a complete piece or part of a piece (a block) the peer has. Second, the peer must considered the client ‘unchoked.’

My client requests some pieces from a peer whenever the former receives an ‘unchoke’ message. But how do I make sure that my client doesn’t send unnecessary requests (for example, requests for blocks that have already been requested many times, even though other blocks may have only been requested a few times or not at all)?

The solution was a least-requested-blocks strategy. I simply suggest that my client ask a given peer for whichever X blocks have been requested the fewest numer of times. In the code below, you’ll notice that each time a block gets requested, its request counter goes up.

image

At some point, I will do a follow-up post about tweaking X to experimentally determine the impact of the number of consecutive block requests on the underlying TCP stream and on download performance.

Determining which pieces to request

Okay, so my client determines which blocks to request. But a block is part of a piece, and a file probably has many pieces. How do I determine which piece to request?

A peeer will only advertise that it has a piece when it has the entire piece - that is, when every block in the piece has been concatenated and the peer has confirmed that the sha1 hash of the resulting string or blob or file matches the one provided in the metainfo file. This means that if we were to use a least-requested-pieces strategy and randomly request the first few pieces, we would be discounting initial information that might result in more-productive requests. That information is how many peers have each piece.

Consider that in any swarm of peers, some peers may have a few pieces, other peers may have all, and some peers may have no pieces at all. Peers advertise this in their initial bitfield message, and in subsequent have messages.

Since we have information about which peers have which pieces, we can build a data structure that tells us which pieces are rarest. For simplicity and expediency, I have an array of 2-tuples, where the first element in every tuple is a Piece instance, and the second element is an array containing peer IDs. The more peer IDs the second element’s array contains, the less rare a piece is.

image

My code updates the array every time my client receives a bitfield or have with containing information about which peers have which pieces. I think this data structure can almost certainly be optimized by storing self.pieces as a tree and keeping it sorted at all times. I am going to create a more efficient solution during code review and post it here.

Conclusion

That’s all for now. I intend to come back and improve this post by adding more code with explanations after I do some code review. Now you know how a BitTorrent client works under the hood!

College, tell me why things are important. And three new courses to do just that.

Or: Why technical coursework should instill an understanding of Robert Pirsig’s Quality, and what to do about it.

During my first year at Chapel Hill, I found myself memorizing equations for calculus, function calls for computer science, and models for economics. Throughout my time at UNC, I asked smart friends why they were learning about the rational abstractions in their fields. The questions can be reverse-engineered to get a syllabus of three questions college should force students to ask.

I follow each question with a subheading (in [] brackets), but not the name, of a course I will suggest near the end of this essay that will empower students to answer each question with competence - and maybe even a newfound arete.

- What analytical rational tools that do not currently exist might be useful, and how would you go about developing them? (Got some good answers to this one. Most involved some specific form of stronger artificial intelligence. Most answers were things students learned about outside of school.)

[Romantically-inspired rational-tool production]

- How is the power of the rational tools you are using bounded? (For example, practical computational tools are usually bounded by the power of Turing machines. Mathematics generally cannot prove everything about itself without resorting to ideas outside of math - see Gödel’s incompleteness theorem. Economic policy involves tradeoffs that the models cannot moralize about.)

[The power of reason in your field]

- Can you place the rational abstractions you’re learning in a larger context? (For example, for what human purposes are we learning to code on practical implementations of Turing machines? Has nobody ever thought to start by making students figure out what might be best to study!?)

[The big picture]

Four years ago, a first grasping at my frustrations

My first year, I sat down with then-Chancellor Holden Thorp, who was awesome because he liked to stay in touch with campus life by speaking with students. I told Thorp that majors in science, math and other technical fields should begin their studies with an introduction to what people can do with these intellectual tools. He countered (very sensibly, given my unrefined argument and phrasing) that it’s best to actually teach the fundamentals of chemistry, biology, math, physics or computer science than to teach what their applications are.

But as time went on, I realized that’s not exactly what I meant.

I wanted to know why I was studying computation, how what I was currently learning related to the big picture, the big picture of the big picture, and so on, until I could place my learning in context of the ultimate big picture: human thought itself.

This sounds grandiose, but is actually not much to ask at all. If I believed higher education in science, math and technical fields as it is today were anything more than vocational training, then upon developing the views I am writing about here, my immediate reaction might have been be that we have philosophy departments for training in the theory of knowledge, and that I should go study philosophy. This reaction would have sent me spiraling down into modern philosophy, which can at times and without physicalism be, somewhat ironically, simultaneously internally consistent and vapid. It is also somewhat lacking in new ideas outside of mathematical logic.

(Note: Some old ideas do have much to teach us. But the point is that philosophy as a field evolves only so much, and its evolution does not really drive the evolution of our understanding of the world. The sciences and humanities - fields which actually apply rational inquiry - do that.)

I came to believe that teaching methods before theory is the essence of vocational education. Higher education is about context, theory, the study of abstractions for the ability to manipulate symbols in the appropriate way as to take advantage of those abstractions (methods), and, crucially, the study of abstractions for the sake of understanding the purpose of the abstractions.

The formulation of a problem is often more essential than its solution, which may be merely a matter of mathematical or experimental skill. To raise new questions, new possibilities, to regard old questions from a new angle, requires creative imagination and marks real advance in science.

- Albert Einstein

Smart students find it much more interesting and engaging when instructors explain the rational context for knowledge (i.e., symbolic manipulation for math, or abstract computational models for programming) before or at least while asking them to learn about methods (i.e. integrals or programming).

But, critically, it’s not just about motivating study of methods. It’s actually the case that this context makes for a more rigorous technical foundation for study. As I’ll show later, it turns out MIT and Stanford already do this.

Methods and theory should be one because, at the highest levels of intellectual activity, they already are

College is supposed to be a place for higher learning, but really, we undergraduates are just learning about the implementations of established techniques for problem solving. For example, if my math education had begun with an explanation that math is a symbolic manipulation system of limited power, I would have been much more interested in learning calculus and other specific methods because I would have understood that only by getting into the details of other peoples’ established techniques could I hope to arrive at an understanding of what it means to create good techniques in the first place.

This means, for example, that calculus students would come into college with a base level of knowledge in math, most of which was acquired through rote. But their introductory coursework would help them develop some rudimentary understanding of the fact that math is a system of symbolic manipulation of limited power whose methods we study not only because we hope to become proficient in those methods, but also because we hope to construct better, more mathematically beautiful methods.

Looking at methods’ details with an eye to esthetic is the zen bit of Zen and the Art of Motorcycle Maintenance and its approach to true intellectual mastery of a subject. Become a mechanic in order to see why abstractions are useful and generate better ones.

This isn’t essential to my argument, and is almost an aside, but one way we might tell which abstractions are good is something akin to Pirsig’s Quality.

Quality is the esthetic rational toolmakers compare their work against. Rational tools exist, and exist in their specific forms, for reasons motivated by an undefined but detectable rational or mathematical beauty.

The author doesn’t mean to suggest that students learn most from vocational curricula. He actually prescribes a path for self-study, I suppose because no vocational and few higher ed curricula care to enlighten students about rational toolmaking in their fields. For college, Pirsig ends up recommending Quality Studies. I think that class would be a moderately stimulating addition to a philosophy curriculum, but that it would be insufficient for higher technical education.

I’m not committed to Pirsig’s ideas. But I do believe that undergraduates can experience their own journey into the details (becoming motorcycle mechanics, so to speak), and emerge with a sense of what makes abstractions good instead of merely the ability to use abstractions, by making a study of rational toolmaking in their various technical fields.

As a friend put it, we need to ‘turn the scientific method upon reason itself.’ In doing so, we might find ways to focus its ends, sharpen its methods and make our tools more powerful and relevant.

And now [Phaedrus] began to see for the first time the unbelievable magnitude of what man, when he gained power to understand and rule the world in terms of dialectic truths, had lost. He had built empires of scientific capability to manipulate the phenomena of nature into enormous manifestations of his own dreams of power and wealth - but for this he had exchanged an empire of understanding of equal magnitude: an understanding of what it is to be a part of the world, and not an enemy of it.

- Zen and the Art of Motorcycle Maintenance

I think the system can be improved. People don’t learn what math *is* (symbolic manipulation of limited power, as Gödel showed with his incompleteness theorem, designed to solve certain problems humans face) or what computing *is* (implementations of abstract models closely tied to linguistics) or what physics *is* (abstractions, but probably actually just approximations, of the real world, wrapped in symbolic logic for easy manipulation by human brains) before learning about specific methods.

“The system” risks being a creativity-killer and deterring some bright students

By teaching them methods only, I think colleges are throwing away their brightest, most creative students. I think that generally speaking, colleges kill creativity in the same ways Sir Ken Robinson says primary education does. But nobody has really expressed the reason why very well. I think it is because college leaves Quality to philosophy and teaches methods to undergraduate students it treats as human equation-solvers who can only appreciate the place of methods after they study all of them.

The reason for this killing of creativity, I think, is that smart people are deterred when majors appear to be all about memorization of existing tools. COMP 455 opened my eyes to computability, abstract notions of computation, and other thought that helped answer my questions about why I’d bother to go sit in a room and learn about basic Java, data structures and algorithms, operating systems, networks et cetera.

I want to place rational tools in a larger context, analyze their power, and understand why they were created. I believe I am a good student because I want to know why I am studying a thing before I devote my life to it. For me, devotion to study before answering that “why” question meant acceptance without good reasons. It was entirely irrational.

I felt like I was mindlessly studying methods, with no direction beyond vocational application. I am capable of seeing the big picture, and was capable of that four years ago. I want more than vocational education; I want higher education.

An aside: Now that I have been enlightened, I am excited about MOOCs. I just have to teach myself where abstractions fit in the web of human understanding. Ironically, given our country’s focus on four-year college, I will be responsible for my own education as a motorcycle mechanic, so to speak.

How things should be: A general description

Higher education should include thinking about how our romantic notions actually lead to our rational abstractions. By starting out seeking something like Pirsig’s notion of Quality, students will see why the great thinkers of the past developed the abstractions they did. They will learn to appreciate what mathematical beauty is, and why it is more than mere esthetic.

How things should be: Two very specific solutions

Solution 1: Three classes that develop students’ abilities to critically analyze abstractions

I think undergraduates should take one theory class in their field (i.e. COMP 455/Models of Languages and Computation, or What is Modern Physics, What is Modern Math) before (potentially and probably, depending on the student’s prior exposure to the field) blindly and purposelessly memorizing techniques as if college is some kind of vocational program where you learn how to use existing tools.

I also think that everyone in technical fields should take an Introduction to Symbolic Logic, in order to appreciate what math is, the upper limits on its power, and to develop mathematical maturity easily. There is no reason students should be afraid of symbols. But every year, smart students drop out of math courses because they don’t understand what it is they are studying. True, some can’t do the work. But others are inquisitive and unwilling to blindly follow unlit paths. Light up the path a bit and you’ll get a flood of bright students from the humanities flowing into technical fields.

I think everyone in all fields should take Introdution to Linguistics and Computation. The ideas behind the two can be analyzed using similar toolkits. The parallels are fascinating to me, and even if they aren’t fascinating to everyone, everyone has to use both language and computation these days. The least higher education can do is enlighten us to their Quality.

There are three practical benefits to this three-course introduction to modern thought. First is the sudden, widespread development of mathematical maturity (or the state of being unafraid of symbols) among undergraduates. I believe that “my students don’t have mathematical maturity” is a cop-out. We should just teach students what symbolic manipulation is all about, since it is so fundamental to modern technical thought.

Second, I just think technical subjects would make a lot more sense, and fewer truly bright and extremely inquisitive students would give up, if they could place the methods they learn in college in context in a) their field of study and b) the whole of human knowledge. For example, students whose minds are open to the theory of computation while they study programming will develop a deeper understanding of why functional programming exhibits rational beauty. This seemingly lofty theoretical suggestion would actually have a low-level vocational impact by making them better job candidates.

Third, I think that treating theory (ex. abstract computation) and methods (ex. programming) as one is the only way to generate creativity on a broad scale in higher ed.

Solution 2: Develop an intuition for Quality in abstractions by viscerally experiencing their relative utility (i.e., by studying the lower-order abstractions first)

This solution is perhaps the most clever didactic approach to higher education I have seen because it relies on students’ introspection on the material to help them develop an understanding of the importance, meaning and romantic notions behind abstractions in the technical fields.

MIT teaches computer science from the ground up. They start with math and physics, building upon Maxwell’s equations and logic gates to get to computation.

I pose a question about the best students in society. Which of them, given this introspection-derived intuition for beauty in abstraction, for Pirsig’s Quality, could not be prepared to accept the crystallization of mathematically beautiful new thoughts, given a set of patterns to recognize from past inventors of mathematically beautiful abstractions? Wouldn’t these students be not just fully capable but actually driven, by an intuitive understanding of the romantic beauty of the abstractions, towards study that truly furthers human understanding (i.e., by the creation of more subjectively beautiful and therefore practically useful abstractions)?

And most importantly for the great masses of college students: Among average students, who would not be capable of at least critically analyzing abstractions and appreciating their value?

Conclusion

Only by placing reason in a larger context, analyzing the value of reason, and understanding why reason was created can we value life as more than a series of milestones, people as more than objects whose labor government or capital can exploit, and undergraduate education in technical fields as more than a four-year journey of rote (interspersed, if the student is lucky, with a touch of ethics unconnected to the rational tools that constitute the primary focus of study).

Riemann didn’t formalize the integral because he was an automaton with a reservoir of methodological knowledge. I feel that he must have had a full appreciation for the power of abstractions. Clearly, he needed methods to inform his thinking, and rigorous technical training to guide his work. He wasn’t guided by romantic notions of abstractions. But I beg you to tell me how many undergraduate students can even tell you what an equation is - what its power is, what its creation did for humanity, and suggest a couple directions for new abstractions.

And while not everyone can be a Riemann or Newton or whoever, at least give bright students the chance to ask “why” questions before they become completely disillusioned because of a missing-the-forest-for-the-trees curriculum.

Addendum: Related Observations and Predictions

Stanford, MIT and a few others already do what I am talking about. Most seem to do it using Solution 2. With the rise of MOOCs, colleges that take the vocational approach will fade away unless they go beyond mere vocational training via the approximately the way (instilling a sense of Quality) described above.

Addendum: Request for Comments

This essay could be completely wrong, and maybe college students should blindly study methods, because the world is so complicated that the ability to develop abstractions or at least appreciate others’ abstractions is beyond your average college student. I see things differently, which is why I wrote this.

This essay would be nothing without critical analysis, refinement, counter-examples and other critical discussion. Send your thoughts my way - please.

Addendum: What this essay is not

This essay is not a call for a new Great Books curriculum. It is true that the Great Books curriculum is dead. I think this is for good reason: The Great Books were written in another time, and I think their advice just isn’t applicable or great anymore.

This essay is also not a call for students to study philosophy only. One can argue that philosophy as a field is failing to make an impact on our world in the way it could. Philosophy as a study should be kept, but integrated into each technical subject. The equations ultimately come from a sense of what is valuable in the world, from a sense of what “it is to be part of the world, and not an enemy of it.”

Addendum: The three courses

What is X: The power of reason in your field, and the big picture

Introduction to Linguistics and Computation

Introduction to Symbolic Logic: Romantically-inspired rational-tool production*

* - This course must also teach Quality itself, and not just symbolic logic methods. That is, the ethos and open-minded didactic methods (yes, I think it is okay to use methods to teach non-methodical things) of the first two courses must be embedded in it.

Being extremely intentional about learning

I’m attending Hacker School, a “free-as-in-beer” “writers’ retreat for hackers,” here in New York City for the next three months. Hacker School is a chance to code up things that I wouldn’t have the chance to work on in a job, a startup or in school. Here, I can pick up a new language, technique or library simply because I want to learn.

The idea is to get much better at programming by focusing on writing lots of good code. In formal academic programs, students internalize formal descriptions of systems, learn how to use formal tools to clarify, analyze and solve computing problems. This scientific approach to computing doesn’t necessarily imbue one with engineering know-how. That’s why I’ve always found it instructive to work on web apps and businesses outside of school.

There are three things I’d like to accomplish while here.

But at Hacker School, nobody is pushing me to learn a particular framework, language, development technique or even idea. In fact, nobody asks students to set learning goals. So the items that follow are just skills I’ve realized are important for the kinds of things I want to create.

Here are the three things. First, I want to master test-driven development. Writing tests first and then coding against those tests means your code/program/product is working and robust by the time you finish the functionality. That can’t always be said for ex post facto testing. Second, I want to write good asynchronous code (the kind that makes nested callbacks manageable). Third, I want to contribute good code to major open-source projects.

Neither learning about computer science in school, applying that knowledge in a job, nor doing your own thing will afford you the opportunity to traverse the dense mesh of computing ideas guided only by your creativity and intuitive sense of what’s interesting. And few things are as exciting for me as interacting with smart people who can already do amazing things, but who might be twice as capable tomorrow after spending time on focused creative exploration.

That focused creative exploration is missing from education today. It’s even missing in the new online courses from Coursera, edX and Udacity. To succeed, students these days have to see everything they learn as an addition to their toolkit. Grow your toolkit, the new rules of the game say, and you’ll have more to draw from when you need to prove your value by exhibiting creative ability or problem-solving skills. If you want to be the best, you better have the biggest toolkit.

But you also better have the best context in which to place that toolkit. Learning by imitation fails to convey the fact that most human inventions, whether in the realm of ideas or in the physical world, arose via creativity and out of need. For example, Newton helped develop infinitesimal calculus because he needed it to describe physics, not because he wanted to hawk his ability to find minima and maxima on the job market. In today’s very complicated, detailed, high-tech world, being capable means having a big toolkit and a lot of context.

I think it’s time to create more learning communities based not around imitation and evaluation, but focused creative exploration. This would work for students who are extremely intentional about learning.

I think Hacker School is one such opportunity to become extremely intentional about one’s learning. More about that in a moment.

First, let me qualify the above statement about learning communities: It’s not as clear how students who are not so intentional about learning will fare in the 21st century. But think about this for a moment: Maybe the truly talented people in each field treat their work as craft more than as an application of some combination of academic principles and industry norms. It might not be too outrageous to hypothesize that being truly good at something comes from being extremely intentional about improving your knowledge, focused creative exploration, or both. The ‘focused’ bit can be with respect to time, or with respect to intensity of study.

As an example of being intentional about learning, I’m posting what I want to learn at Hacker School. I may edit what follows as I think of more things or come up with better ideas. Software clearly fits my model well, but I think focused creative exploration could work for economics, math and maybe even some parts of science (so, this post will have a sequel at some point).

I have picked out a number of languages, concepts and (code) libraries that I haven’t really taken the time to master. For each item on this list, I’m going to learn the item by creating a mini-project and posting the code online for peer review. Note how some items’ roadmaps start with simple learning exercises but progress to contributing to a major library or writing a library. (Part of intentional learning is thinking about levels of understanding; in this case, writing good, useful libraries can indicate a higher level of understanding relative to other coding accomplishments.)

0) Become really knowledgeable about the hardware-software interface by completing and extending all exercises in this book: http://www1.idc.ac.il/tecs/plan.html

  - If there’s a school that makes you implement a virtual machine in assembly language, go there. If they make you extend the exercise, you are getting your money’s worth.

1) Master Twisted, the Python framework for plugging whatever kind of computation you like, wherever you like, into an event loop. Do it by:

  - Writing an IRC bot that joins several channels and puts the messages into an event loop (done)

  - Completing this tutorial: http://krondo.com/?page_id=1327

  - Developing a solid “mental model” for thinking about asynchronous programming

  - Contribute the proactor pattern (I have a friend to thank for this idea)

2) Shell scripting: Be able to create, from memory and with minimal doc-diving on the scripting language reference, maintainable and flexible shell scripts. Mini-project:

  - From memory, create a script to do incremental backups of files and databases to some web service.

3) Keep asynchronous CoffeeScript readable. Not sure how I’ll do this yet.

  - Intro to the subject: http://gfxmonk.net/2010/07/04/defer-taming-asynchronous-javascript-with-coffeescript.html

4) Learn to make effective use of GNU Make

5) Become proficient at programming d3js animations (requires math and javascript skills)

6) Become proficient at using GDB to get a clear idea of how various programs use memory (specifically, develop an intuition for C and Python stack and heap allocation).

  - This will involve lots of fun over at Smash the Stack. Google it.

7) Develop a security framework for the Socketstream web framework. Block about half a dozen common types of attacks that naively-written web apps are vulnerable to. Use this project to learn how to create nice hook points for the attack blockers, so that other programmers can create pluggable attack detection modules for the framework.

8) Implement a few AI algorithms in Clojure. Use this opportunity to develop an intuition for what a functional programming language is good for (in particular, confirm or reject the notion suggested by one of my profs that it can be good for modeling cognitive processes).

9) Learn C++11x, and debug a few programs using gdb.

  - Starting with AI Tic-Tac-Toe but may move into some graphics stuff.

10) Understand zeromq. I like the idea and it helps run stock exchanges. So it seems cool.

Ideas to organize my learning:

- In-browser map of knowledge domains, where domains are defined by bookmarked articles.

This post ends here because it’s 2am. But you get the idea: focused creative exploration and being extremely intentional about learning are both good. Maybe more people should think up learning communities that foster these two things.

The app in the year 2015: What is it, and how is it made?

Let’s take a look at some of the things people could, until recently, only do well with native apps (that is, the kind you download from app stores):

  • Vector graphics. Vector images are more or less 1-dimensional matrices full of coordinates instead of grids of bits, meaning they scale to arbitrary sizes. In theory, that an image on a standard web page will look just as good when scaled up to fit on a web page the size of the moon.
  • Great user experience when the device is not connected to the Internet
  • Asynchronous background computation on the client side (i.e.., using your computer to do certain things on behalf of the app, instead of using some server to perform computation)
  • Real push notifications
  • Highly responsive browser-based user interfaces. No, not the ones that look like iOS but feel like they’re slowly stumbling home from a bar after every click.

Quite recently, all of these capabilities became available to web apps as browser vendors implemented the draft HTML5 specification. More significantly, the software development tools that developers needs in order to efficiently take advantage of the most promising features of HTML5 are now arriving.

These tools are immature, but when they reach beta status, the user experience for the average web application will be entirely different. After good tools are available, you will:

  • Have the weird experience of using web apps that let you play with data when you’re in the middle of the desert or high up on some remote mountain with no Internet connection. Even more strangely, these apps will automatically sync your changes with the server when you get a connection.
  • (Probably unknowingly) Allow your browser to perform computation in the background. Chrome could start factorizing prime numbers. These background computations will be dispatched by browser vendors’ implementations of Web Workers.
  • Receive messages from web apps without explicitly requesting said messages.

It used to be the case that clients (such as your browser) would dispatch requests to servers. Servers would answer these requests. The requests would be synchronous, meaning your browser would reload the page in order to fetch the server’s response, or asynchronous  meaning your browser would fetch the server’s response and update part of the web page when it came through.

If you’re learning about web development, stop for a moment. The most popular tools, like Django and Ruby on Rails, and even raw HTML, are rapidly being replaced by more efficient ones. Assumptions about the request/response model are baked into these tools, but that model is no longer the state-of-the-art way to develop interactive, speedy apps.

Implications for Developers

  • Go and learn about Web Sockets
  • The tools that are set to help developers take advantage of HTML5, Web Sockets and Web Workers work well with new time-saving languages (mostly by coincidence).
    Instead of writing HTML, you may now choose to write web pages in a nice template language with a strong developer community. In one such language, Jade, <div id=”home”></div> becomes #home. <p>Hello</p> becomes p Hello.

Implications for consumers and the app market

  • Businesses won’t need to build native apps in order to take advantage of most of the capabilities and performance features of native apps. What will happen to app stores’ revenue shares?
  • User experience will get really, really good.

A peak into the future

Let’s take a peek into the user experience these tools will enable.

It’s 2015, and you are headed to work. You visit the web page of an app your friend recommended. The entire app goes over the wire: the compiled, compressed output of an app built with tools descended from SocketStream, Meteor or Derby is downloaded to your device’s memory. You interact with the app - which is actually a web page - even though your phone lacks WiFi or mobile data because your subway train is passing through a long tunnel.

At the other end of the tunnel, you emerge in the sun, your task scheduling or purchase or appointment-making complete. After your phone re-establishes a connection, you get a buzz: The server has confirmed your doctor’s appointment, and your book will be delivered tomorrow.

You just ran markup and scripting code in a platform-independent virtual machine, and transacted business using a highly responsive, interactive interface.

But this isn’t the 1990s, and you didn’t download the Java Virtual Machine or Flash Player. And you didn’t send Apple 30% of your payment for the app. Anyone in developing parts of Africa, India or China with a phone half as good as yours can do what you did, because it’s 2015, and for now, the browser is how business gets done in the virtual economy.

Top