summary |
shortlog | log |
commit |
commitdiff |
tree
first ⋅ prev ⋅ next
Adam Dickmeiss [Wed, 31 Oct 2001 08:51:49 +0000 (08:51 +0000)]
Robot saves metadata with unique names in directory "flat" (if it exists).
Adam Dickmeiss [Tue, 30 Oct 2001 08:29:54 +0000 (08:29 +0000)]
Pattern may be negated in rules (! as first character does that)
Adam Dickmeiss [Fri, 26 Oct 2001 13:26:11 +0000 (13:26 +0000)]
Implemented Allow/deny rules. Better Tcl autoconfig.
Adam Dickmeiss [Fri, 29 Jun 2001 22:25:55 +0000 (22:25 +0000)]
Yet another fix regarding relative links.
Adam Dickmeiss [Fri, 29 Jun 2001 21:47:31 +0000 (21:47 +0000)]
Added option to specify Accept-Language.
Adam Dickmeiss [Thu, 7 Jun 2001 08:17:00 +0000 (08:17 +0000)]
Fixes for robots.txt handling (bug introduced by previous commit).
Adam Dickmeiss [Thu, 7 Jun 2001 08:10:10 +0000 (08:10 +0000)]
Bug fix for relative links.
Adam Dickmeiss [Wed, 6 Jun 2001 09:37:18 +0000 (09:37 +0000)]
Added some character entities for mapping.
Adam Dickmeiss [Wed, 6 Jun 2001 07:10:31 +0000 (07:10 +0000)]
Added README. Ignore case in keywords in robots.txt.
Adam Dickmeiss [Tue, 5 Jun 2001 08:44:50 +0000 (08:44 +0000)]
maxDistance set to 50 default.
Adam Dickmeiss [Tue, 5 Jun 2001 07:46:00 +0000 (07:46 +0000)]
Remove characters after semicolon in header contents.
Adam Dickmeiss [Tue, 27 Feb 2001 10:45:44 +0000 (10:45 +0000)]
Minor changes.
Adam Dickmeiss [Mon, 26 Feb 2001 22:51:51 +0000 (22:51 +0000)]
Added config for zebra/zmbol.
Adam Dickmeiss [Tue, 23 Jan 2001 14:28:41 +0000 (14:28 +0000)]
Minor fix for anchor references.
Adam Dickmeiss [Tue, 23 Jan 2001 12:05:06 +0000 (12:05 +0000)]
Removed YAZ dependency.
Adam Dickmeiss [Tue, 23 Jan 2001 11:26:43 +0000 (11:26 +0000)]
Added options for the robot.
Adam Dickmeiss [Tue, 23 Jan 2001 09:20:32 +0000 (09:20 +0000)]
Multiple http connections. Bug fixes.
Adam Dickmeiss [Mon, 11 Dec 2000 17:11:03 +0000 (17:11 +0000)]
Fixed problem with links having .. for root directory of web server.
Thank you FrontPage.
Adam Dickmeiss [Sun, 10 Dec 2000 22:27:48 +0000 (22:27 +0000)]
Implemented robots.txt rules.
Adam Dickmeiss [Fri, 8 Dec 2000 22:46:53 +0000 (22:46 +0000)]
File robots.txt now read the each domain.
Pages are now fetched in a Round-robin fashion.
Adam Dickmeiss [Fri, 8 Dec 2000 08:55:35 +0000 (08:55 +0000)]
DCdot doesn't rely on htmlSwitch no more.
Adam Dickmeiss [Thu, 7 Dec 2000 20:16:11 +0000 (20:16 +0000)]
Added -nonest for htmlSwitch statement. Robot puts reference to
bad URLs in bad area.
Adam Dickmeiss [Mon, 27 Dec 1999 11:49:30 +0000 (11:49 +0000)]
Major speed improvement.
Adam Dickmeiss [Thu, 4 Feb 1999 21:32:00 +0000 (21:32 +0000)]
Updated configure script.
Per M. Hansen [Thu, 4 Feb 1999 20:37:25 +0000 (20:37 +0000)]
Changed tags for the output.
Adam Dickmeiss [Thu, 15 Oct 1998 13:27:19 +0000 (13:27 +0000)]
Minor changes.
Adam Dickmeiss [Thu, 15 Oct 1998 12:31:25 +0000 (12:31 +0000)]
Added configure script.
Adam Dickmeiss [Thu, 15 Oct 1998 12:30:59 +0000 (12:30 +0000)]
Buf fixes. Robot saves body of text without tags and java script sections.
Adam Dickmeiss [Tue, 6 Aug 1996 14:04:22 +0000 (14:04 +0000)]
Initial revision
projects / tclrobot.git / log