Monday, February 28, 2011

Linkedin.com keeps changing...

...and it's a bit frustrating for me testing my program.

The CSS it's what changes the most, and sometime my program relies on those. I have to make it flexible enough to include multiple parsing tokens.

I am glad at least I downloaded the searches previously done so that now I can run tests on those without connecting to the server, but some of those pages have changed now, so I have to constantly verify manually that what I rely on is also indeed online.

I am getting close to the full basic functionality of the program so hopefully I can put it online soon.

Friday, February 25, 2011

Encouraging Results

Of the 74 result pages that were giving me multiple relevant profiles, 55 are still giving me the same result after I inspect the education field in the results. Even if these ~55 were to be manually inspected it wouldn't be too bad compared to the 3220 total.

But of course there much more to improve and more data to mine out, so I'm confident I can bring this number even lower.

For example I could search for "Engineer" in the profile, I could go beyond the first 25 results when possible (it's applicable in limited cases) and I could see if the name matches perfectly the one stored in the records.


EDIT: I have currently cut it down to 32 just by introducing the "engineer" variant and by redistributing the weight of the words found in the profile. There is not much more unless I open the profiles and compare them.
I might as well do that since I have to do it for a later part of the program anyway.

30% of 63% ~ 20% of Relevant Pages at Least

YouTube - Paul Krugman on "The Return of Depression Economics?"
Interesting video to listen to while coding.

Anyway, about 30% of the result pages have something relevant so, that implies that about 20% of relevant pages come out of that chunk.

About 100 of these pages (about 1/6 of the results) have more than 1 possible relevant result. I now have to set up a function that establishes which one is the most relevant. It will probably take a while to make sure all are "sorted" correctly.

New Project Issues

Relationships Matter | LinkedIn

I encountered some new issues today as I went on with my project.
Turns out windows does not let you write more than 3220 files in one folder. And I had to write about 3490 in one, so I guess some of them got deleted. Weird behavior anyway. Why would they possibly do that?

So I lost about 250 webpages which were worth 7 seconds each. So about 30 min of computation. And what's worst is that I don't quite know which ones are gone and so it might just be better to run the entire program again. This time using DFS instead of BFS, this way fewer webpages will be saved in that folder, and the other ones will be processed right away in between an exponential distributed delay and the other.

I really wonder why Windows doesn't let you write more than 3220 files in a single folder. I mean, 3220 is not even a power of 2! it seems like a very arbitrary number. I wonder if it has to do with some security feature of Windows 7.

As soon as I got the first results in I did some statistical analysis to see where I was going, and of all the 3220 searches executed and saved,
- 17% have returned no result
- 63% have returned multiple results
- 6.50 % have returned public profile
- 12 % have returned a page saying that there is no public profile available (and that if I want to see the profile I would need to log in)

Encouraging I have to say. I thought that the 'no result' percentage would have been higher. Of course I still have to analyze the 63% of the result pages which might contain no profile at all.

I'm doing just that now. I just finished writing the function that will determine if any result has any relevancy, so I think I will run it on the entire sample and obtain some more data to analyze so that I may know the percentage of result pages actually have something relevant in them.

I'll be back posting it as soon as the program finishes running.

Wednesday, February 23, 2011

Other Weird Things

I was in the computer lab today and I was watching the Daily Show and the Colbert report while eating as I usually do when my friends came over.

Two of them started talking about how they prefer the Colbert Report rather than the Daily Show because of Colbert's humor. Then they went on to say that the actually agree with what Colbert literally says and not what he really means. One of them actually did not even understood Colbert's ironic humor until after 2 seasons. And this guy I thought was smart.


It saddens me on multiple levels. I really don't see how could you believe what Colbert is actually saying when he points out the flaws of these every single episode? and how can you not understand Colbert is using sarcasm to make the opposite point of what he is really saying?

This last point especially tells me that this person does not have a very good anti-bs system, also called critical thinking, and that he can be easily manipulated by authoritative figures how want to trick him.

It was very surprising to me to hear this from him, but I guess I shouldn't have since he is the one that said "greed is good" and that "business decisions should always overpower moral decisions".

And talking about that, Nobel Economist Joseph Stiglitz clarifies things at minute 20 of this video: http://www.youtube.com/watch?v=Tt_JlPCkQSs

and this is a very interesting talk that I suggest to watch in full, so here's the full video.




Thursday, February 3, 2011

Greed is Not Good

Just a quick reminder to everybody. Greed is not good.

I say this because lately me and another engineering friend of mine were discussing economic matter and, even though I know he has taken micro- and macro-economics, he told me that "Greed is good". That baffled me. That is not something you should expect of hearing out of a college student who has studied economics.

The general misconception about this topic is that greed makes markets efficient and deadweight loss disappear. This would also mean no government intervention in order for the 'greed' to work flawlessly.

The misconception is about the word 'greed' which is used as 'self-interest' while it really has a definition that is not so positive:

Greed - Wikipedia, the free encyclopedia: "Greed is an excessive desire to possess wealth or goods with the intention to keep it for one's self. Greed - like lust and gluttony - is a sin of excess. Greed is inappropriate expectation. However, greed is applied to a very excessive or rapacious desire and pursuit of wealth, status, and power.
As a secular psychological concept, greed is an inordinate desire to acquire or possess more than one needs or deserves, especially with respect to material wealth."

'Greed' is the blind accumulation of material goods, while 'self-interest' is the innate desire of pursuing what is good for you. To this I would add that greed is usually meant as "the ends justify the means" approach. In other words to say that 'greed is good' is to say that your own personal and material wealth is a good thing and if in the process of doing it you hurt somebody, that's fine anyway because greed is good.
'Self interest' on the other hand has a more relaxed meaning which is not limited to material good and not even personal benefit. It is more similar to 'the pursuit of happiness' rather than greed, and it certainly does not imply hurting somebody else.

The second issue with this mentality is that economists have proven to actually not work. The premise of Adam Smith 'invisible hand' is the context of a "perfectly competitive market". Such market would have a numerous number of small firms, all of which produce exact products, they all have free entry and exit costs, they are all 'price takers', they have and they all have the same information about the market and the product they are selling. In such market no government intervention is needed because their self interest will make each other work efficiently and the market wont generate inefficiencies.

The problem is that there is no such market in real life. Some get close enough that we can model it that way and thus argue for reducing government intervention, but the majority of the markets are not of this kind. Those that produce the most inefficiencies are not 'perfectly competitive markets'.

Summarizing:
  • It is known that greed is good in some cases, but these are not the most common by a long shot.
  • Economists do actually study these situations and when they describe the engine of markets they talk about 'self-interest' or 'maximizing satisfaction' and none of them use the word greed.
  • I take issue with the word greed being used to describe what it is not greed, especially because other people who might hear 'greed is good' distractingly don't associate the meaning that the speaker intended. I would say that unconsciously even the speaker would convince himself that greed is good and act greedy as a consequence.
  • Spreading this false sense that 'greed is good' in our society will change us culturally, making us more willing to accept greedy behavior from other people.