I encountered some new issues today as I went on with my project.
Turns out windows does not let you write more than 3220 files in one folder. And I had to write about 3490 in one, so I guess some of them got deleted. Weird behavior anyway. Why would they possibly do that?
So I lost about 250 webpages which were worth 7 seconds each. So about 30 min of computation. And what's worst is that I don't quite know which ones are gone and so it might just be better to run the entire program again. This time using DFS instead of BFS, this way fewer webpages will be saved in that folder, and the other ones will be processed right away in between an exponential distributed delay and the other.
I really wonder why Windows doesn't let you write more than 3220 files in a single folder. I mean, 3220 is not even a power of 2! it seems like a very arbitrary number. I wonder if it has to do with some security feature of Windows 7.
As soon as I got the first results in I did some statistical analysis to see where I was going, and of all the 3220 searches executed and saved,
- 17% have returned no result
- 63% have returned multiple results
- 6.50 % have returned public profile
- 12 % have returned a page saying that there is no public profile available (and that if I want to see the profile I would need to log in)
Encouraging I have to say. I thought that the 'no result' percentage would have been higher. Of course I still have to analyze the 63% of the result pages which might contain no profile at all.
I'm doing just that now. I just finished writing the function that will determine if any result has any relevancy, so I think I will run it on the entire sample and obtain some more data to analyze so that I may know the percentage of result pages actually have something relevant in them.
I'll be back posting it as soon as the program finishes running.
No comments:
Post a Comment