X-Git-Url: http://jsfdemo.indexdata.com/?a=blobdiff_plain;f=doc%2Fbook.xml;h=26fd0a5ba76102ef45101a54c69f50b86731e173;hb=9ac41f74e33f58fbbb507f0b3ae9ccdce306f525;hp=406e7bf8d0db6677b514cd20cc240c38413ccd03;hpb=747d1b739a4c770dd1af537d0280c75588e80f66;p=metaproxy-moved-to-github.git diff --git a/doc/book.xml b/doc/book.xml index 406e7bf..26fd0a5 100644 --- a/doc/book.xml +++ b/doc/book.xml @@ -17,19 +17,19 @@ --> ]> - + Metaproxy - User's Guide and Reference - MikeTaylor - - AdamDickmeiss MarcCromme + + MikeTaylor + 2006 Index Data ApS @@ -79,7 +79,7 @@ Metaproxy - is a standalone program that acts as a universal router, proxy and + is a stand alone program that acts as a universal router, proxy and encapsulated metasearcher for information retrieval protocols such as Z39.50, and in the future SRU and SRW. @@ -107,7 +107,7 @@ being more powerful, flexible, configurable and extensible. Among its many advantages over the older, more pedestrian work are support for multiplexing (encapsulated metasearching), routing by - database name, authentication and authorisation and serving local + database name, authentication and authorization and serving local files via HTTP. Equally significant, its modular architecture facilitites the creation of pluggable modules implementing further functionality. @@ -221,7 +221,7 @@ for more information. - We have succesfully built Metaproxy using the compilers + We have successfully built Metaproxy using the compilers GCC version 4.0 and Microsoft Visual Studio 2003/2005. @@ -378,7 +378,7 @@ \boost\lib, \boost\include. - For more informatation about installing Boost refer to the + For more information about installing Boost refer to the getting started pages. @@ -392,7 +392,7 @@ here. - Libxslt has other dependencies, but thes can all be downloaded + Libxslt has other dependencies, but these can all be downloaded from the same site. Get the following: iconv, zlib, libxml2, libxslt. @@ -424,7 +424,7 @@
Metaproxy - Metaproxy is shipped with NMAKE makfiles as well - similar + Metaproxy is shipped with NMAKE makefiles as well - similar to those found in the YAZ++/YAZ packages. Adjust this Makefile to point to the proper locations of Boost, Libxslt, Libxml2, zlib, iconv, yaz and yazpp. @@ -482,7 +482,7 @@ - After succesful compilation you'll find + After successful compilation you'll find metaproxy.exe in the bin directory. @@ -519,7 +519,7 @@ In general, packages are doctored as they pass through Metaproxy. For example, when the proxy performs authentication - and authorisation on a Z39.50 Init request, it removes the + and authorization on a Z39.50 Init request, it removes the authentication credentials from the package so that they are not passed onto the back-end server; and when search-response packages are obtained from multiple servers, they are merged @@ -615,7 +615,7 @@ as part of Metaproxy, and others may be provided by third parties and dynamically loaded. They all conform to the same simple API of essentially two methods: configure() is - called at startup time, and is passed a DOM tree representing that + called at startup time, and is passed an XML DOM tree representing that part of the configuration file that pertains to this filter instance: it is expected to walk that tree extracting relevant information; and process() is called every @@ -629,6 +629,7 @@ others are sinks: they consume packages and return a result (z3950_client, backend_test, + bounce, http_file); the others are true filters, that read, process and pass on the packages they are fed @@ -636,7 +637,9 @@ log, multi, query_rewrite, + record_transform, session_shared, + sru_z3950, template, virt_db). @@ -648,7 +651,7 @@ We now briefly consider each of the types of filter supported by the core Metaproxy binary. This overview is intended to give a - flavour of the available functionality; more detailed information + flavor of the available functionality; more detailed information about each type of filter is included below in the reference guide to Metaproxy filters. @@ -693,7 +696,7 @@ Figure out what additional information we need in: <literal>auth_simple</literal> (mp::filter::AuthSimple) - Simple authentication and authorisation. The configuration + Simple authentication and authorization. The configuration specifies the name of a file that is the user register, which lists username:password pairs, one per line, colon separated. When a session begins, it @@ -712,7 +715,7 @@ Figure out what additional information we need in: <literal>backend_test</literal> (mp::filter::Backend_test) - A sink that provides dummy responses in the manner of the + A partial sink that provides dummy responses in the manner of the yaz-ztest Z39.50 server. This is useful only for testing. Seriously, you don't need this. Pretend you didn't even read this section. @@ -720,13 +723,31 @@ Figure out what additional information we need in:
+ <literal>bounce</literal> + (mp::filter::Bounce) + + A sink that swallows all packages, + and returns them almost unprocessed. + It never sends any package of any type further down the row, but + sets Z39.50 packages to Z_Close, and HTTP_Request packages to + HTTP_Response err code 400 packages, and adds a suitable bounce + message. + The bounce filter is usually added at end of each filter chain + config.xml to prevent infinite hanging of for example HTTP + requests packages when only the Z39.50 client partial sink + filter is found in the + route. + +
+ +
<literal>frontend_net</literal> (mp::filter::FrontendNet) A source that accepts Z39.50 connections from a port specified in the configuration, reads protocol units, and feeds them into the next filter in the route. When the result is - revceived, it is returned to the original origin. + received, it is returned to the original origin.
@@ -734,8 +755,12 @@ Figure out what additional information we need in: <literal>http_file</literal> (mp::filter::HttpFile) - A sink that returns the contents of files from the local - filesystem in response to HTTP requests. (Yes, Virginia, this + A partial sink which swallows only HTTP_Request packages, and + returns the contents of files from the local + filesystem in response to HTTP requests. + It lets Z39.50 packages and all other forthcoming package types + pass untouched. + (Yes, Virginia, this does mean that Metaproxy is also a Web-server in its spare time. So far it does not contain either an email-reader or a Lisp interpreter, but that day is surely coming.) @@ -747,7 +772,8 @@ Figure out what additional information we need in: (mp::filter::Log) Writes logging information to standard output, and passes on - the package unchanged. + the package unchanged. A log file name can be specified, as well + as multiple different logging formats. @@ -775,6 +801,21 @@ Figure out what additional information we need in: + +
+ <literal>record_transform</literal> + (mp::filter::RecordTransform) + + This filter acts only on Z3950 present requests, and let all + other types of packages and requests pass untouched. It's use is + twofold: blocking Z3950 present requests, which the backend + server does not understand and can not honor, and transforming + the present syntax and elementset name according to the rules + specified, to fetch only existing record formats, and transform + them on the fly to requested record syntaxes. + +
+
<literal>session_shared</literal> (mp::filter::SessionShared) @@ -792,6 +833,16 @@ Figure out what additional information we need in:
+ +
+ <literal>sru_z3950</literal> + (mp::filter::SRUtoZ3950) + + This filter transforms valid + SRU/GET or SRU/SOAP requests to Z3950 requests, and wraps the + received hit counts and XML records into suitable SRU response messages. + +
<literal>template</literal> @@ -808,7 +859,7 @@ Figure out what additional information we need in: <section> <title><literal>virt_db</literal> - (mp::filter::Virt_db) + (mp::filter::VirtualDB) Performs virtual database selection: based on the name of the database in the search request, a server is selected, and its @@ -825,13 +876,16 @@ Figure out what additional information we need in: <literal>z3950_client</literal> (mp::filter::Z3950Client) - Performs Z39.50 searching and retrieval by proxying the + A partial sink which swallows only Z39.50 packages. + It performs Z39.50 searching and retrieval by proxying the packages that are passed to it. Init requests are sent to the address specified in the VAL_PROXY otherInfo attached to the request: this may have been specified by client, or generated by a virt_db filter earlier in the route. Subsequent requests are sent to the same address, which is remembered at Init time in a Session object. + HTTP_Request packages and all other forthcoming package types + are passed untouched.
@@ -856,34 +910,10 @@ Figure out what additional information we need in: - frontend_sru (source) - - - Receive SRU (and perhaps SRW) requests. - - - - - sru2z3950 (filter) - - - Translate SRU requests into Z39.50 requests. - - - - sru_client (sink) - SRU searching and retrieval. - - - - - srw_client (sink) - - - SRW searching and retrieval. + SRU/GET and SRU/SOAP searching and retrieval. @@ -921,37 +951,21 @@ Figure out what additional information we need in: implementation detail - they could just as well have been written in YAML or Lisp-like S-expressions, or in a custom syntax.) - - Since XML has been chosen, an XML schema, - config.xsd, is provided for validating - configuration files. This file is supplied in the - etc directory of the Metaproxy distribution. It - can be used by (among other tools) the xmllint - program supplied as part of the libxml2 - distribution: - - - xmllint --noout --schema etc/config.xsd my-config-file.xml - - - (A recent version of libxml2 is required, as - support for XML Schemas is a relatively recent addition.) -
- Overview of XML structure + Overview of the config file XML structure All elements and attributes are in the namespace - . + . This is most easily achieved by setting the default namespace on the top-level element, as here: - <yp2 xmlns="http://indexdata.dk/yp2/config/1"> + <metaproxy xmlns="http://indexdata.com/metaproxy" version="1.0"> - The top-level element is <yp2>. This contains a + The top-level element is <metaproxy>. This contains a <start> element, a <filters> element and a <routes> element, in that order. <filters> is optional; the other two are mandatory. All three are @@ -1000,14 +1014,14 @@ Figure out what additional information we need in: The following is a small, but complete, Metaproxy configuration file (included in the distribution as - metaproxy/etc/config0.xml). + metaproxy/etc/config1.xml). This file defines a very simple configuration that simply proxies to whatever back-end server the client requests, but logs each request and response. This can be useful for debugging complex client-server dialogues. - + @@ -1021,13 +1035,14 @@ Figure out what additional information we need in: + - + ]]> It works by defining a single route, called - start, which consists of a sequence of three + start, which consists of a sequence of four filters. The first and last of these are included by reference: their <filter> elements have refid attributes that refer to filters defined @@ -1035,18 +1050,51 @@ Figure out what additional information we need in: middle filter is included inline in the route. - The three filters in the route are as follows: first, a + The four filters in the route are as follows: first, a frontend_net filter accepts Z39.50 requests from any host on port 9000; then these requests are passed through a log filter that emits a message for each request; they are then fed into a z3950_client - filter, which forwards the requests to the client-specified - back-end Z39.509 server. When the response arrives, it is handed + filter, which forwards all Z39.50 requests to the client-specified + back-end Z39.509 server. Those Z39.50 packages are returned by the + z3950_client filter, with the response data + filled by the external Z39.50 server targeted. + All non-Z39.50 packages are passed through to the + bounce filter, which definitely bounces + everything, including fish, bananas, cold pyjamas, + mutton, beef and trout packages. + When the response arrives, it is handed back to the log filter, which emits another - message; and then to the front-end filter, which returns the - response to the client. + message; and then to the frontend_net filter, + which returns the response to the client.
+
+ Config file syntax checking + + The distribution contains RelaxNG Compact and XML syntax checking + files, as well as XML Schema files. These are found in the + distribution paths + + xml/schema/metaproxy.rnc + xml/schema/metaproxy.rng + xml/schema/metaproxy.xsd + + and can be used to verify or debug the XML structure of + configuration files. For example, using the utility + xmllint, syntax checking is done like this: + + xmllint --noout --schema xml/schema/metaproxy.xsd etc/config-local.xml + xmllint --noout --relaxng xml/schema/metaproxy.rng etc/config-local.xml + + (A recent version of libxml2 is required, as + support for XML Schemas is a relatively recent addition.) + + + You can of course use any other RelaxNG or XML Schema compliant tool + you wish. + +
@@ -1069,14 +1117,14 @@ Figure out what additional information we need in: The interaction between these two filters is necessarily complex: it reflects the real, irreducible complexity of multi-database searching in a protocol such - as Z39.50 that separates initialisation from searching, and in - which the database to be searched is not known at initialisation + as Z39.50 that separates initialization from searching, and in + which the database to be searched is not known at initialization time. It's possible to use these filters without understanding the details of their functioning and the interaction between them; the - next two sections of this chapter are ``HOWTO'' guides for doing + next two sections of this chapter are ``HOW-TO'' guides for doing just that. However, debugging complex configurations will require a deeper understanding, which the last two sections of this chapters attempt to provide. @@ -1114,7 +1162,7 @@ Figure out what additional information we need in: marc - indexdata.dk/marc + indexdata.com/marc ]]> @@ -1139,7 +1187,7 @@ Figure out what additional information we need in: Index Data's tiny testing database of MARC records: - + @@ -1154,21 +1202,22 @@ Figure out what additional information we need in: marc - indexdata.dk/marc + indexdata.com/marc all z3950.loc.gov:7090/voyager - indexdata.dk/marc + indexdata.com/marc 30 + -]]> +]]> (Using a virt_db @@ -1272,7 +1321,7 @@ Z> can be inconvenient in deployment, when users typically don't want to be bothered with problems of this kind and prefer just to get the records from the databases that are available. To obtain this - latter behaviour add an empty + latter behavior add an empty <hideunavailable> element inside the multi filter: @@ -1407,9 +1456,8 @@ Z> [Here there should be a diagram showing the progress of packages through the filters during a simple virtual-database search and a multi-database search, but is seems that your - toolchain has not been able to include the diagram in this - document. This is because of LaTeX suckage. Time to move to - OpenOffice. Yes, really.] + tool chain has not been able to include the diagram in this + document.]