Unnaturally Long Attention Span

Here's a video where someone did an interesting thing. They graphed the locations of every Yellow Cab equipped with GPS in San Francisco over the course of a day. The intensity of the red represents how fast the cab is going.

http://clients.stamen.com/cabspotting/cabspotting_01.html

Doesn't this remind you of the videos of nerve firings in the brain?

As I mentioned before, I'm launching a new website called Diffbot on April 1st. Diffbot is a cool new kind of web-based RSS reader and bookmark manager. Have a handful of sites that you read daily? Diffbot lets you know when those sites have updated and only shows you the portion that changed. It's also just a convenient place to put your bookmarks so that they are accessible wherever you go. This is still very much a work in progress, so we'd really appreciate any feedback on how we could do better. You can now signup to get an invite when it's ready!

Many of my technical readers out there have job titles that match the regular expression "(Sr.|Jr.)? Software Engineer (I)*". For the non-geek readers, that means we call ourselves "Software Engineers" :-). But, what would you consider the difference is between someone who is a "Software Engineer" and someone who's title is a "Computer Programer"? They seem like similar lines of work, yet one seems to imply a higher level of education, perhaps at least a 4-year college instead of a technical trade school. Historically speaking, there has been a huge difference between an engineer and a programmer. During the second World War, "computers" where actually women that caculated artilliary projections using desk calculators. These women became the first computer programmers when they were assigned to program the ENIAC, a room-sized computer with 18,000 vacuum tubes. Engineering, on the other hand, implied not the people that operated the machines, but those that designed the system and solved the larger problems.

Most major universities separate the School of Engineering from the School of Sciences. Engineering includes departments like chemical, mechanical, nuclear, bio and electrical engineering. Science includes departments like chemistry, physics, biology and computer science. Although, it seems like there's a lot of duplication here with the sciences, supposedly, this is because engineering has some common skillset. Engineering cirricula require a certain type of maths. Engineers usually study topics like design, tolerances, robustness, production processes, and technical writing.

However, let's return to what is software engineering? One of the most popular degrees that Software Engineers graduate with is Computer Science, which is not an engineering degree. What I'd like to argue--and this point has been made before--is that software engineering, is not only not taught in formal education, but is consequently different from the other types of engineering. This is why large companies recruiters complain about fresh college grads knowing nothing about debugging, why there are so many software internships, why most software engineering jobs require a few to several years of prior experience--its because that's when you actually learn some "software engineering"!.

However, I say software engineering is fundamentally different from other types of engineering, because the field itself is still in its prenatal state. Maybe that is one of the reasons why it hasn't been developed yet in formal education; we don't really know yet what are the best practices and fundamental formulas in software engineering. The level of engineering that goes into building even the largest software projects is nothing like the level of engineering that goes into building something like a car. It's more like the amount required to build a snowman. Even when I was working at Microsoft on the Windows source code, the Hoover Dam of software engineering, there was very little engineering in place to manage the complexity and uncertainty. To get a sense of this, compare the reliability of your office buiding to Microsoft Office. If your plumbing system or electric system failed as much as my MS Outlook or Firefox crashes, you would be a very unhappy camper. Yet for information workers, both things are equally important to their day.

Software engineering is a newer discipline than mechanical engineering, or even electrical engineering. Obviously, the software world is undergoing a very rapid change right now. We haven't had time yet to sit back and understand the principles and fundamental formulae that govern software. There are lots of well-specified problems, where there is no agreement on what is the best algorithm. Sure, there are small groups of people every studying problems like software reliability, static source code analysis to identify software weaknesses, and theoretic guarantees for software correctness and performance. I think these efforts will become increasingly important.

I've explained in my mind what the distinction between software and other forms of engineering are, but why do I think this is an important issue? There's no problem with working in a field that is largely unstructured, complex, and ad-hoc. That's part of the excitement of being in a brand new field. Life is great as a software engineer. The problem is that software is increasingly replacing the function of physical objects and electrical components. That is, computers are used to replace other things. Your typewriter has been replaced by your word processor. The control center of your car has been replaced by a small computer running an embedded operating system. Your telephone has been replaced by a small computer which emulates the phones functions. The stock market itself has been infused with tons of small programs, trading trillions of your dollars. Take the typewriter as an example, the mechanical engineer that designed it knows that unless the few joints between the key and the hammer fail, your keystroke will translate into a mark on the paper. The materials in the product have been carefully chosen with respect to their well known structural flexibility, strength, and mass. I won't even begin to explain all the things that could go wrong between the time you hit a key on your computer and see a letter appear on screen. In this "design", the components involved were chosen because they seem to work. It's crazy talk to try to estimate how reliable this design is. Yet, this fundamental unreliability is what we entrust to keep our airplanes in the air, our cars on the highway, our bank accounts and financial markets secure. It's just a matter of time before a catastrophic software failure occurs (many major ones already have), or we decide to design responsible software. Software that works as reliably as a toaster. Software that just doesn't break, no matter what the user does.

Most Popular Posts:
Ranking Colleges Using Google and Open Source

Everyone in the sphere seems to have an opinion on what Web 2.0 means to them, and I will add my own here. Web 2.0 is the new software tradition which we have all but transitioned to. The features of this new tradition that differentiate it from the older tradition are that products are more focused on design rather than on capability/features. Take for example products like Flickr, Firefox, MacOS X, Office 12, and the slew of AJAX service-based microapps. A good example to illustrate this paradigm shift is MS Word. In earlier versions of Word(1-7), each release added more capability to the editor evolving it from something like Notepad to the current Word 2003, while essentially keeping the interface consistent and familiar. However, if you look at Word 12, the first thing you will notice is a new interface focused on making tasks more efficient and discoverable. Even the data is stored in open formats with the goal of making it easy to consume and access by third parties.

Why did this shift occur and does this mean that software companies will eventually evolve into pure design companies? It's conceivable that with technology becoming more and more accessible and the wider availability of powerful tools, the software company of the future may be staffed almost completely with artists, psychologists, anthropologists, and designers, with maybe a few technical school graduates to write the tools.

And also, why is it that features and capabilities are less emphasized now? Aren't those the cornerstone of the computer revolution--being empowered by technology?

The truth is, the software industry is stuck in a rut. You can see it across all specialties--office productivity software, on the web, in gaming--no new features have been added, no new types of websites, no new gameplay, just more efficiency, more polygons, more psuedo-chrome. Since we have no new features to add, we have been keeping busy by making things pretty and usable, to keep us employed. Why this rut? Some might say that it is because the industry has entered a stage of evolution rather than revolution. We've reached a critical mass where now the improvements will be in small increments. I agree and also disagree. I agree that is the state of the industry, but I think the cause for no new features is simply that we have no new technology.

Technology as a whole, even outside of IT, has actually slowed down. Where are the Bell Labs of today? PARC is a shell of its former self. What are the new Information theories and quantum theories, new internets. Technology innovation has flatlined after it was made unecessary after we came out of wartime. The internet itself is a wartime child.

Okay, I've gotten a little too caught up and started rambling, but I think the solution to this technological rut is clear. We need more fundamental research. We've reached the limit on how far we can milk the results of past research. Whether or not there is a wartime neccessity, we need to do this basic research in order to improve the capabilities of our systems, to claim that things are still getting better.

So, what kind of new capabilities should be developed? Computers today are used almost solely to input, output, store, or transmit human data. But, instead of just being repositories and pipes for the data, I believe computers can consume and reason with data, much like a human can. How this can be implemented in our current market, I'll talk about later.

Here's another interesting ranking: the World Press Freedom Index. The list is topped by Denmark. At the bottom of the list is North Korea. The US? 44th. Another interesting data point is the United States of America (in Iraq) [sic] listed with rank 137. Defenders of freedom?

The Human Rights Data Analysis Group , which used to be incubated under AAAS has just released a study they did on the analysis of the human rights violation datasets from Timor-Leste. I'm poring over their paper right now

The Profile of Human Rights Violations in Timor-Leste, 1974-1999
A Report by the Benetech Human Rights Data Analysis Group to the Commission on Reception, Truth and Reconciliation of Timor-LesteThe actual datasets are available for download as well. I encourage you to check them out(zipped CSV files):

the graveyard census database, called the Graveyard Census Database (GCD)
the fatal violations data from the Retrospective Mortality Survey (RMS)
the Multiple Systems Estimation (MSE) data file

Some of you may know that I've been working in my spare time on a new type of news reader. Until now, the service has been a closed test involving about 50 users. Last week, though, Leith and I have finally decided on a launch date for a public beta. It's going to be April Fools Day, 2006. That gives us less than two months to get things tidy (and boy are there a lot of things!). Stay tuned...

In the course of working with fellow geeks, I've noticed a few language usages that are unique to us geeks. One of these is the tendency to classify problems and issues as either "trivial" or "hard". Now, you have to understand, part of this standard geek colloquialism comes from the math tradition of formal proofs. Trivial is used to describe the obvious, non-interesting solution. "Hardness" could be a shortening of NP-hard, used to describe a class of problems that requires a lot of computation.

Next time you are tempted to use these labels, maybe consider some of the following more meaningful alternatives:

Instead of "This such-and-such problem is trivial"... consider replacing it with:

• This such-and-such problem is easy to solve!
• This such-and-such problem is small in scope.
  
• This such-and-such problem is easy to talk about, but would require a team of grad students 10 months to implement.

Instead of "So, we all know such-and-such is hard"... consider replacing it with:

• So, we all know such-and-such is computationally intensive.
• So, we all know such-and-such is impenetrable.
• So, we all know such-and-such is something I have no idea how to do.
  

"Researchers believe they have found a new compound that could finally kill the HIV/AIDS virus, not just slow it down as current treatments do. While most of the community is still hesitant to comment on this until it passes peer review, initial results show that their method attacks and kills ALL variations of the virus. A fast track through the FDA could have one of the world's leading problems licked in less than a decade."

Funniest thing heard on Slashdot

Theoretically, Chuck Norris' tears could cure AIDS, cancer, paraplegia, herpes, common cold, mouth ulcers, and hangovers. Too bad that it is impossible to make Chuck Norris cry...

Scientists have discovered a species of wasp, called Ampulex, that has evolved an ability to perform brain surgery on cockroaches. Not only that, but it has reverse-engineered the brain-physical map of the roach in order to control its movements, a feat which has only been performed by scientists in recent history. Seems like hacking is not only something that humans do.

The wasp slips her stinger through the roach's exoskeleton and directly into its brain. She apparently use ssensors along the sides of the stinger to guide it through the brain, a bit like a surgeon snaking his way to an appendix with a laparoscope. She continues to probe the roach's brain until she reaches one particular spot that appears to control the escape reflex. She injects a second venom that influences these neurons in such a way that the escape reflex disappears.

From the outside, the effect is surreal. The wasp does not paralyze the cockroach. In fact, the roach is able to lift up its front legs again and walk. But now it cannot move of its own accord. The wasp takes hold of one of the roach's antennae and leads it--in the words of Israeli scientists who study Ampulex--like a dog on a leash.

The zombie roach crawls where its master leads, which turns out to be the wasp's burrow. The roach creeps obediently into the burrow and sits there quietly, while the wasp plugs up the burrow with pebbles. Now the wasp turns to the roach once more and lays an egg on its underside. The roach does not resist. The egg hatches, and the larva chews a hole in the side of the roach. In it goes.

The larva grows inside the roach, devouring the organs of its host, for about eight days. It is then ready to weave itself a cocoon--which it makes within the roach as well. After four more weeks, the wasp grows to an adult. It breaks out of its cocoon, and out of the roach as well. Seeing a full-grown wasp crawl out of a roach suddenly makes those Alien movies look pretty derivative.

It appears now China has added http://google.cn to the "Great Firewall"

Tests on a Shanghai-based trace-route server, located at http://www.linkwan.com/vr2/, indicated that the site was being blocked at the government-operated backbone server. The analysis from the trace route said 'IP packets are being lost past network CHINANET backbone network at hop 4.'

Many people have been commenting on Google's recently implemented censorship of Chinese search results. If you search at google.cn from within China, certain sensitive queries such as "falun gong" or "democracy" will recieve censored results. Some have demanded that Google not comply with the Chinese government's request.

So, yesterday I was sitting in a small burrito shop on the corner of Rengstorff and Middlefield waiting for my order. I was scanning the local paper while observing the hyper-efficiency of the Mexican burrito assembly line. The reason I mention this is that a front page article featured a story of an American citizen's recent experience with the Chinese government. It really puts things in perspective.

Here's the full article. Some excerpts..

ET: Tell us about the trial.

Dr. Lee: The trial was conducted in such a way that I was denied every possible legal right. I have evidence which can prove my innocence and they did not allow me to present it. I tried to defend myself, but they did not allow me to do that, especially when I was talking about the reasons, why I was trying to reveal the truth of the persecution. They interrupted me many times; whenever I tried to speak, they would stop me. So I never got a chance to defend myself. And no evidence whatsoever was presented from me, even though I requested it many times.

...

ET: How were you treated?

Dr. Lee: When I was arrested in the very beginning, in Canton, and then was transferred to Guangzhou, they tortured me with handcuffs, which cut into my flesh to the bone; it was extremely painful. They did not allow me to sleep for 92 hours in total. They used the handcuffs as a tool to torture me. They pulled my arms upward, from the back, so it was extremely painful and I could not move at all. If I moved at all, the pain would get worse.

I conducted a hunger strike from the very beginning on January 22, 2003 until February 10—18 days in total.

I still insisted on practicing Falun Gong, so they handcuffed me on March 27, after the trial; the trial was March 21. They handcuffed me, and I started a hunger strike on the 30th, one more time. The reason I started was because I wanted to write an appeal letter, because the trial was unlawful, and I had to make three copies, plus one for myself, thus I had to write four copies of the appeal letter, and I was handcuffed.

On May 9 my appeal letter was denied, and I started a hunger strike again, because this was totally unlawful, so I wanted to protest again. After that, according to their law, that was the final decision, you had to be thrown into prison, so then I was transferred to Nanjing prison on the 12th of May. I was still doing a hunger strike to protest, and then on the 14th of May I received a phone call from the U.S. consulate and I said, "I have some materials over here, I want to give them to you. I will stop the hunger strike today, and if you do not get the materials in two weeks, I will start the hunger strike again." Because those materials are extremely important. They recorded all the details of the trial, what they (the authorities) said, what they did, why they did not allow me to do this and that, and also my appeal letters and how they tortured me. After two weeks, they did not send what I wanted, so I started a hunger strike again.

The forced feeding was extremely painful, and I resisted their treatment. So they just poked the big, thick tube into my nose and into my stomach. It was so irritating that I threw up several times. It was crazy—I was screaming as it was so painful. The cameraman, who was green—just from the police academy—he fainted, right at the scene. On June 2, the U.S. consulate received all the materials I wanted to give them.

They still had my Zhuan Falun [the central text for Falun Gong practitioners], and I said you need to give me this Zhuan Falun , or I won't stop the hunger strike. They started the force feeding again on June 3. Before they did, they yelled, "Taste the power of the people's democratic tyranny!" They were full of exultation before they were going to torture me, like it felt so good. They left the tubes in my nose, which I pulled out myself.

They used the forced feeding to torture me, actually.

They had these so-called group study sessions, anti-Falun Gong group study sessions. I was surrounded by 15 people, yelling at me. Falun Gong is this and that, so crazy, why do you believe in these things… calling me this every day. So I started a hunger strike again, because that was so bad. I was protesting against the torture.

I started a hunger strike again on July 14 to 18, and at that time they left the tube inside of my nose for 33 hours, and they had all of my body tied up on a bed. 33 hours I lied there without moving; it was extremely painful—agony.

Then came the brainwashing. From August, 2003 they started to force me to watch the videotapes. Every day, it was three hours of videotapes. Then they had condemnation meetings on Falun Gong, three-hour sessions; then the policemen would come and talk to me. This continued for three to four months, and they didn't see enough results, so they changed their strategy.

Starting at the end of 2003, they forced me to do slave labor, to make shoes. The shoes used a sort of industrial glue that contains benzene; it's very toxic and irritating, and I felt short of breath and had a headache, that sort of thing. The other thing is, I always tried to refuse to work, because I was innocent—I should not be doing this. They forced me to stand for 16 days, from morning to evening, stand in front of other prisoners; they verbally attacked me, insulted me, that kind of thing. If you didn't stand straight they would kick you and push you.

They tried to force me to confess to the crime, by using forced sitting. They say you have to sit there and think about your wrongdoings yourself, like repentance. You're forced to sit there, just sit there in a fixed position. The longest time was 48 days straight, with my heart problem surfacing as a result. After breakfast it was 7:30 a.m.; then you start sitting there, and at lunchtime you have lunch. Then you start sitting again after lunch, then have dinner, and then come back and sit there for some time and watch their CCTV (China Central Television). Every hour they gave me five minutes to walk around in the cell. The stool was this size [gestures to describes a stool less than a foot high, less than a foot wide, and around six inches deep], you sit there and then your bottom develops this hard callus—it's extremely painful, and your back is also very painful. I was exhausted for such a long time.

After maybe two to three weeks, my brain worked very slowly. When the consulate came, I found it very difficult to speak; somehow, my brain didn't work anymore!

ET: What was your day-to-day life like?

Dr. Lee: They have something for you all the time, like brainwashing sessions. It's always brainwashing, from the very beginning to the very end. They say you are a prisoner, you have committed a crime, that you have to be punished.

And this was an AMERICAN citizen.

Yeah, you don't want to mess with the Chinese government. I think what Google has done in order to bring their service to China is a positive step forward. They even display a message to the user when the results have been censored. Since many of the sensitive sites are filtered anyway by the Chinese government at the ISP level, showing the search results for those would have limited usefulness. Any change that we want to enact in China's humanitarian policies will have to happen at the governement level, not by companies.

An early text ad

GMail and Google Maps may be incremental improvements to web based email and mapping, but the reason Google is as talked about today as it is is because of its success on Wall Street. Despite a few recent lackluster product releases, we give Google the benefit of the doubt because, hey, if they were able to more than quadruple their stock price then they must be pretty smart guys. The majority of it's financial success, Google owes, to Adsense, Google's little text ads. (They have recently gone into graphic ads as well)

How sound is the concept of Adsense? Imagine what your elevator pictch for Adsense would be back in the day. Is it as useful as being able to buy books at bargain prices online or trading my used stuff with others? Well, I can only speak personally, but I have recieved little, if any value from Adsense. I've never found anything "relevant" to what I'm looking for in an Adsense ad. I've definitely tried, and I still don't understand it. When I search for something on Google the most relevant results are the natural search results. That is the justification of the search results--The ads inherently cannot be in the best interest of the user and I've trained up my mental adblocking circuits to just simply ignore them.

But, Google continues to milk money from Adsense, now up to $1.9 billion in this last quarter alone. Who's buying this stuff? I think the proper analogy is that Google is cashing in on the online Gold Rush it has created. It's selling picks and shovels to the miners when there is no gold in the mountains to be found. In the American Gold Rush, the real winners, at least financially, were the hotel owners and city developers that provided the support framework to the many westward bound migrants. Of the composition of Adsense traffic, I would conjecture that the majority are "mistake clicks" that do not lead to information the user actually wants to see. The design of the Adsense ad itself is optimized for these mistake clicks. For example if you hover over the Adsense ads on the right, you'll see that the entire surface of the ad is a clickable area, even the empty space between the ads. This goes counter to the web convention that only underlined, blue text represents a hyperlink. So the common behavior of underlining text as you read it will invoke the ad, causing the poor advertiser who thinks he has bid his money on a legitimate user to his site, to pay money to Google for the user's action.

In addition to these "mistake clicks" others have estimated that up to 20% of Adsense clicks are malicious clicks, or spam clicks. Google would never admit to this being a serious problem, but the advertisers themselves have noticed by examining their webserver logs that much of the traffic coming in from these Adsense clicks is suspicious.

So, there seems to be many misconceptions at multiple levels in the Adsense system, both between the user, Google, and the advertiser. How long can the parties involved remain in the dark? John Battelle, in a recent interview, said that this misconception persists because despite the false clicks, advertisers still believe that they are receiving some net benefit from Adsense, in terms of new users to their site. However, John thinks that there is some point at which these benefits will be outweighed by the costs of traffic acquisition.

Where will Google be when that point comes?

It's that time of year again when many people are deciding which college they should attend come fall. Whether they are a high school senior, aspiring professional, or seasoned veteran seeking an MBA, the ultimate decision of which college is based largely on reputation.

The perennial data source for college ratings that most students turn to is the US News America's Best Colleges Rankings. Despite the limited usefulness of reading rankings, there is only so much information you can get from first-hand experience. Taking the campus tour, talking with students and doing your own due diligence is about as far as you can go short of attending the school. The US News ranking methodology is based on many factors: peer assessment, faculty resources, selectivity, class sizes, and financial aid, among many others. What does it really mean to rank a list of schools? I can define a personal value function for determining which school I prefer to and by doing pairwise comparisons, I can sort a list of schools based on which would be the most beneficial to my personal goals. However, you can imagine though that an integration of these value functions over a large sample of students would pretty quickly create a ranking distribution that was rather flat. While the US News Rankings strive to give a complete overall view of the school, there are few weaknesses to its method. Some of these weaknesses have stirred criticism for the rankings by both students and colleges.

First, many of the metrics are subjective, such as peer assessment and selectively. This allows for "smoothing out" of the rankings to what the editors expect by "reinterpreting" the selectivity.

My major problem with the US News rankings, however, is that they are not free. In fact, only the top few schools rankings are viewable. Too see the whole list you have to pay $14.95.

So, to this end, I've decided to try my hand at generating my own rankings. Since I'm no expert in the field of evaluating colleges, I'm going to cheat and use statistical learning techniques. I'm going to do this with the help of just Google and some open source software. You won't even have to pay $14.95 to see the results!

First off, I found a list of American Universities from the Open Directory. I parsed out this page with a quick hand-written wrapper to get the names and URLs. Now, the fun part. What kind of "features" should I use for evaluating each school? This is where a bit of editorial control comes to bear. I wanted to capture the essence of what the US News methodology used, but I wanted to do this in a completely automated way using Google. So for each feature, I defined a Google Search query (shown in brackets) that would give me a rough approximation of that particular attribute:

1. Peer assessment [link:www.stanford.edu] - This is how some search engines approximate "peer assessment", by counting the number of other pages citing you
   
2. Size [site:www.stanford.edu] - a larger school would have a larger web, right? =)
   
3. Number of faculty [dr. "home page" site:www.stanford.edu] - hopefully those professors have websites that mention "dr." and "home page"
   
4. Scholarly Publications["Stanford University" in scholar.google.com]
5. News mentions ["Stanford University" in news.google.com]

So then I just iterate the list of the schools and perform each of those queries using the Google API. Let it run for a few hours and I have all my data. Now, you may be thinking there's no way that 5 numbers can tell you everything you need to make a decision on a school. Well, let's take a look at what the data looks like.

First, I load the data into WEKA, a free, open source data mining software package in Java. It implements many off-the-shelf classification and regression algorithms with an API and GUI interface. Let's take a look at few slices of the data:
This figure plots newsMentions on the x-axis against scholarlyPublications on the y-axis. The points that are plotted are those 50 schools that have a score in the US News rankings (schools beyond the 50th place don't have an overall score in the US News rankings). The color of the dots goes from low(blue) to high(orange). The color trend is blue in the lower-left to orange in the upper-right. As you can see, not only are the two Google queries correllated, they seem to also be jointly correlated to the US News score.
Plotting numberFaculty against scholarlyPublications shows also a positive correlation. So maybe these queries weren't totally bogus and have some informational content.

The next step is to fit a statistical model to the training data. The basic idea is to train the model on the 50 colleges with USNews scores and to test the model on all 1700+ American colleges. The natural first step is to try to fit a 5-dimensional line through the space. The best fit line is

USNewsRank = (-0.0003)peerAssessment+(0)sizeWeb+(0.0063)numFaculty+(0)scholarlyPubs +
0.0002 * newsMentions+68.7534.

This simple model has a root mean squared error(RMSE) of 10.4223. So, in the linear model the size of the web and number of scholarly Publications don't play a role. Fair enough, but can we do better?

The answer is yes. I next used a support vector machine model with a quadratic kernel function. This gave me a RMSE of 7.2724 on the training data. The quadratic kernel allows for more complex dependencies in the training data to be modelled, which is why the training error is lower, but would this result in better evaluation on the larger data set? There is no quick answer to this, but we can see from the ouput what the model predicted.

Name	USNews	SVM
University of Washington	57.4	98.929
Yale University	98	98.081
Harvard University	98	97.953
Massachusetts Institute of Technology	93	92.996
Stanford University	93	92.922
National University		92.523
Columbia University	86	92.255
Princeton University	100	90.609
New York University	65	85.271
University of Chicago	85	85.052
Indiana University		83.973
University of Pennsylvania	93	83.91
Duke University	93	79.487
University of Southern California	66	78.645
University of Pittsburgh		78.274
Cornell University	84	78.051
University of Florida		77.864
University of Colorado		76.877
The American College		76.597
University of California, Berkeley	78	76.192

You might have noticed that I have disabled my MusicSearch script. The page was up for a grand total of 3 days(1/13-1/15). During that time, it got over 3400 hits from around 11o0 unique visitors. That means the average person performed about 3 searches using the script. I'm not sure how it got so popular, but it seems that someone initially discovered my page through the blogspot network and posted it on French news site. From there, the promise of free music started spreading and my link started showing up at several other french, spanish, and greek sites as well as music forums. Some of those links are still there: (here, here, here). Since I don't read any of those languages, I don't really know what they are saying about my page. The Google Translation of the comments gives me this:

piranha - 15/01/2006 to 13:04:36 thank you for the bond! Nicolas - 16/01/2006 to 09:41:45 it was surely a good idea. When that functioned coconut - 17/01/2006 to 01:43:30 It was well yes, and there was also French song! Not like there bond

Although, my script later used caching, at some point the operator of the radio.blog network started to take notice of the traffic and sent me the following e-mail:

From: astro@mubility.com
To: mike@cs.stanford.edu
Hello,

I'm the owner of radio.blog.club.
What can I do that make you remove your script ?

What you do will bring us nowhere. When you make it easy to download MP3
from my website, you take both of us to an illegal level. One day or another, someone will ask to shut the site off because of this, and we will have no choice that take it down.

I'm not sure this is what you want. So PLEASE, remove your script.

Regards.

-astro

So, I have complied with astro's wishes, and removed my script, which was completely functioning at that time. Radio.blog's system encourages people to upload their music into publically accessible locations on the webserver. Their search engine at http://radioblogclub.com makes it easy to find anyone's radio.blog music.

Some people have been asking me how MusicSearch works and where the songs come from. Well, you know those embedded flash players that people put in their xanga/livejournal/myspaces that blast embarassing music everytime you try to visit your buddie's page in a library or coffeeshop? I took a look at the code for one of the popular ones, radio.blog. It turns out that the sound files are stored in an obvious, public directory. So, the MusicSearch that I wrote is just a simplified interface on the radio.blog search engine, which makes the public path to the audio file more explicit. The sound files are typically radio broadcast quality, 64kbps or lower, but give you a good sense of the song.

I've started building a page at http://ai.stanford.edu/~mike. It's a work in early progress. Also mirrored at http://miketung.com

Try out this thingy I just wrote : http://www.streetpricer.com/MusicSearch
Works only for Windows users with Windows Media Player(WMP). The songs are full tracks, not short clips, which you can download to your computer using "Save as..." in WMP. You'll notice that actually it generates an ASX playlist that plays in WMP. If you hit the "next track" button, it'll play other similar songs on that playlist.

UPDATE: If you can't get it working there are a few things you might need to do in WMP.
1. Turn off Shuffle in Play>
2. Turn on "Connect to the internet" in Tools > Options > Player
Hey, it only took me a few hours to write...

Count the number of men in the picture before and after it changes. Where does the extra man come from?

I've made quite a bit of money this past week trading stocks in my Roth IRA. Here's a few things that I learned, which I hope will be of use to you as well.

1. The reward function of playing the stock market game should NOT be the stock price itself. The reward function is actually SELLING stocks at a high margin over what you paid for them. So, if you are training a reinforcement learner (such as your brain or a Markov Decision Process) , no reward should be applied when the stock price increases, unless a sale is to be made.

Taking this into account, only two modes of short-term stock trading make sense to me:

1. Betting on a Bull Market
In this mode, you begin the trading near the beginning of a big event, such as the winter shopping season or the end of the fiscal year for a particular industry. You choose a basket of stocks that reflects how you predict the industry will be changed at the end of this quarter. Take all of the companies that you think will be affected positively, subtract out those which everyone else thinks will be positive (since those expectations are already reflected in the price) and buy those stocks. So for example, if you think AMD will beat out INTEL in the next batch of processor releases and it is counter to what the public believes, then buy it! Hold on to the stock until the event passes, and then sell your stock at the end of this 1-2 month period. Your gain/loss will reflect exactly how good your prediction was.

2. Interday/Week trading
The most important thing to remember is that each trade costs you $7 (with Scottrade, which is what I use), so you must try to make trades that at least cover you for the $7. This means that the trades necessarily have to be in larger volumes on stocks that have higher volitility. This translates into more risk. However, there are techniques to balance the risk by choosing a set of stocks that balance each other. Unlike the first mode, this type of trading can be done even when the market is flat, since you are just betting on the small changes that happen day-to-day. In addition to choosing stocks that move a lot from day-to-day, you want to choose stocks that have a sufficient information streams. What I mean, is that even though those penny stocks might have huge amounts of movement, if there is not much news related to that company, you don't really have enough information to make an informed trade. So, you really want to choose companies that have a mid-size stock, information-rich news stream, and those that are not likely to go out of business the next day. For this type of trading, you want to hold onto the stock for at most just a few days.