Archives of the TeradataForum
Message Posted: Wed, 19 Jun 2002 @ 17:28:50 GMT
| Subj: | | Re: Testing Methodology |
| |
| From: | | Jose Lora |
Hi Thomas.
Some of your reasons to move ETL away from the Teradata system are not really valid, for example :
| | - batch windows get smaller as the value of the warehouse grows | |
It always possible and recommendable to design your data warehouse without having Batch Windows as a requirement. In fact, NCR is
moving in that direction with the Active Data Warehouse concept and the improvements in the loading tools. Your design will also be
affected for user's Data availability requirements.
| | - at the same time data volumes and business logic complexity grows | |
This reason will affect both ETL strategies and I should say that an out-of-Teradata strategy will be more affected because more
business logic could mean more lockup transformations and bigger data volumes will involve expensive non parallel IO to move the data
required for the transformation outside Teradata
| | - I purchased the database for the user and not for the ETL process | |
This is a good point and the best solution is making the user a part of the ETL process and making him understand the benefits of
this approach. In my experience, user are very happy to have access to the original data in the database (temporarily) to validate
if the changing business rules.
| | - a separate ETL server is cheaper to expand than a Teradata system | |
That's true, however, an increase in CPU power for your ETL process will also increase the CPU power for the business process. In
the other hand, having an ETL server that only work during batch hours is not a very good way to use that box and do not improve user
response time.
| | - there are high performance tools that transform extremely fast, moving more data than I have typically seen | |
Sure there are, an after test some of them I couldn't find a match for the inherent parallelism of Teradata (for large data
volumes)
However, there is always a good reason to make transformation outside Teradata (number or date validation are not easy on
Teradata) and in this case, I would prefer using Inmod routines, Access Modules or tools like Genio (Hummingbird) that can stream
data directly to Fastload (doing all sort of simple row based transformations on the fly), to avoid the expensive non parallel
IO.
Regards.
Jose Lora
Systems Architect
Meredith Corporation
| 
