Archives of the TeradataForum

Message Posted: Tue, 14 Dec 1999 @ 20:21:24 GMT

	<Prev	Next>		<<First	<Prev	Next>	Last>>

The cast/concatenate approach is good sometimes--BTEQ character handling uses field mode which has a fair amount of overhead plus the current 254 and 2GB limits. In a way, this is letting Teradata do more of the work at the cost of increased complexity in the SQL. Leveraging Access/Excel/MSQuery/SQL Server/MSDE/ADO etc. is good too when practical.

In the next release, TUF6.0 2Q2000, we plan to add 'Access Module' support to BTEQ. What's that? Sort-of like Inmods/Outmods but more powerful. We changed the utilities (Fastload, Multiload, TPump, FsatExport, and ARC) last year to encapsulate the details of I/O in Access Modules. The utilities now stream data through the Access Module API which supports abstract data stream functions like Open, Close, Read, Write, Checkpoint. We created Access Modules for Disk and Tape on MVS, VM, MP-RAS, NT, and Solaris. For NT, we offer an "OLE-DB Consumer" Access Module that enables the load utiliies to fire SQL at any OLE-DB Provider and flow the resulting rowset through the utilities. Also in TUF6.0 we will have a Nmed Pipes Access Module.

The API is documented, enabling PS, customers, 3rd parties to write custom Access Modules similarly to the historical use of Inmods/Outmods. For example, "Open Tape Backup" integration with Veritas was accomplished via Veritas writing an Access Module for ARC that effectively functions as a backup agent. We are now in concept on an XML Access Module to enable XML/relational mapping.

Once BTEQ has Access Module support, it can use any of these, as well as support custom access modules. Thus issues like the 2G limit, custom results 'rendering' (csv, xml) are addressed without changing BTEQ.

As Paral-X rolls out, the I/O modules become Read/Write Operators. A key difference is they are also passed metadata describing the rowset. We will provide inmod and Access Module adapters so existing inmods/Access Modules can be used without change. Again, the intent is to provide an open API and a set of productized modules--future ideas must be prioritized, but we're thinking about ODBC, XML, Message Queue (MSMQ, MQSeries), native Oracle/IFX/etc. Customers and PS can write custom Access Modules/Read Operators with a head start from templates that will be made available. In-stream transformation can be done in 'Filter Operators' that receive and output both data and metadata describing the input/output rowset. Documentation with more detail is targeted for early next year.

	<Prev	Next>		<<First	<Prev	Next>	Last>>