projects / tclrobot.git / commit
? search:
summary | shortlog | log | commit | commitdiff | tree
(parent: 975b229) | patch
File robots.txt now read the each domain.
authorAdam Dickmeiss <adam@indexdata.dk>
Fri, 8 Dec 2000 22:46:53 +0000 (22:46 +0000)
committerAdam Dickmeiss <adam@indexdata.dk>
Fri, 8 Dec 2000 22:46:53 +0000 (22:46 +0000)
commit04f32b20fae8795aab0c1f8b703394056f3efea3
tree24aec0935f37db3c6bb6447fa2f48ef463b94da8tree | snapshot
parent975b229918744816b12263fae523eaad4f8e2d87commit | diff
File robots.txt now read the each domain.
Pages are now fetched in a Round-robin fashion.
dcdot.tcl diff | blob | history
robot.tcl diff | blob | history
Simple Web harvester