View as PDF

1. The Basics

2. The Product

3. The QD Environment

4. The Marketplace

5. General Technical Questions

6. Compression

7. Performance

8. Data Sources

9. Query Tools/Complementary Products

10. The QD Loader

11. Deployment

1. The Basics
   
   1. What is QD Technology?

      QD Technology is a premier provider of next generation database technology. By packaging enterprise data into a secure data mart that resides on your laptop PC, QD provides safe 24/7 read-only database access anytime and anywhere. The data mart QD creates on your laptop is optimized to deliver super fast query performance, and compressed to achieve high data volume in a small storage footprint.

      Companies who leverage QD will see significantly improved analyst productivity and expanded opportunity for analytic insights and reporting. Secondary benefits such as reduced storage costs, decreased contention for network resources, and support for an overall disaster mitigation program are also possible.

   2. Where are QD's offices?

      Our world headquarters is in Rutherford, NJ (outside New York City). We have a sales office in Los Angeles, CA, an engineering organization based in Austin, Texas and a team of over 50 developers and engineers in St. Petersburg, Russia.

   
   > Back to top

2. The Product
   1. What is QD and what does it do?

      QD is a Relational Database Management System (RDBMS) that performs SQL queries at speeds that exceed that of traditional RDBMSs.

      QD can take data from any ODBC-compliant RDBMS or properly delimited flat file and make it available to users in a read-only, compressed form. Users can then download the data in this form directly to their desktop or laptop.

      The read-only, compressed data stored locally in QD can be accessed with standard SQL tools and can be queried faster than would be possible from its native RDBMS.

   2. What benefits does QD deliver?

      The ultimate benefit of QD is 24/7 access and improved insight into the data stored in large corporate databases. Insight is improved because data can be combined from any number of disparate databases or flat files. Queries will also be faster than they have ever been before, and queries that were not even possible under traditional RDBMS's will be more than possible – they will be easy and fast with QD.

      Data needed by an analyst may often be distributed over several data sources that inhibit easy access. In other cases, database and system administrators have imposed strict policies that prohibit running intensive queries against the production database, or limit these queries to non-production hours. Queries that can't complete in the allotted time are not permitted to run. QD solves these data access issues. It enables analysts and other data users to run the queries they want against the data they need, and gain new insights into their corporate data and key business drivers.

   3. What are the product features that deliver these benefits?

      QD has three primary features:

      • Compression. QD takes one or more databases and compresses them down to about one-third to one-fifth of their original size. Greater compression results can be achieved depending on the nature of the data being compressed.

      • Performance. QD can access the compressed database at speeds that are faster than can be achieved by traditional RDBMS's. This is made possible through a combination of techniques including smart compression, more complete indexing of data, and a SQL engine that is optimized for querying.

      • Dedicated local copies. QD places a copy of the compressed database on a user's laptop or desktop. This allows the user to access it locally without impacting the production database, its servers, or the corporate network. The production server does not need to manage locks or logs or any other overhead.

   4. What are some of the secondary benefits to implementing QD?
      • Since personal computers running QD can access their databases without being connected to the network, the remote systems can operate even if the main database server goes down. This provides increased availability and even limited disaster tolerance for the database and its related applications.

      • Since the database is portable, you can bring the entire database to a customer meeting, even when no network access is available. This can give a resourceful salesperson or analyst the means to vastly improve data access. And since QD can compress and share the entire database, or even multiple databases, users have access to more data than ever before.

      • Once the simple initial set up is completed, minimal database administration is required to manage the read-only database copies on desktops and laptops.

      • Many large database implementations consist of a master database server, which feeds data to a data warehouse, which in turn makes data available to a series of smaller data marts. In some larger implementations, additional layers of hardware may be in use. QD simplifies the whole operation because it can place the entire database or data warehouse on a user's laptop or desktop. QD's infrastructure simplification can translate to significant financial savings in systems, networking, administration, support, licensing and software.

      • The master database is totally protected against any action a user might take on his laptop; a user cannot do anything to his read-only QD database that will affect the master database.

      • Data security is an integral part of the QD system architecture. QD server-side and client-side components deliver an integrated set of features to safeguard against outright data theft from the outside, as well as misuse and abuse of data on the inside.

      • By ensuring that an individual read-only capability would remain available even in the face of a larger enterprise level outage, QD can serve as one component in the support for an overall disaster mitigation program.

   5. What differentiates QD from a traditional RDBMS?
      • Network independence. Users can work on the QD without being connected to the main database server, or to any other external computer or network at all. Therefore, they can query their database while on an airplane or in a secure site where outside network access is prohibited.

      • A subscription service which provides users with a data feed (and email notification), whenever the QD is ready to provide a download.

      • A SQL engine that is optimized for query performance. Because QD is read-only, query execution is not encumbered by overhead or data structural inefficiencies owing to the need to be able to perform database inserts or updates.

      • Simple architecture. Because QD is read only, no complex infrastructure is required for transactional processing, locking, or updates. QD is a tight, compact solution.

      • Portability. The decompression algorithm travels with the table itself. If new decompression algorithms are brought into QD, all of the old tables can still be read.

      • Flexibility. Different rows of the database can be compressed with different compression algorithms. This allows higher levels of compression than could be provided by any single algorithm.

      • Future-proofing. Because we store information about the compression algorithm in the compressed database, a database that is compressed with today's release will still be readable and usable in all future releases of QD. We have not found another compression-based database where this is true.

   6. How are queries run against a QD database?

      QD works with your existing query tools and does not require that your data analysts learn any new tools or data analysis techniques.

   
   > Back to top

3. The QD Environment
   1. What environments offer the best fit for QD?
      • Where there are one or more large, busy databases where users and analysts cannot perform the kinds of queries, analysis, or reporting that need to be done

      • Where many remote employees could benefit from having continuous access to their database

      • Where the network or the database server are overburdened with database queries

      • Where database administrators cannot provide satisfactory service to all of their remote users

   2. In which environments might QD be a poor fit?
      • Where end-users require read-write access to the database.

      • Where end-users are required to use thin clients. A thin client will not have sufficient disk space or memory to store the compressed database or to execute complex queries.

      • Where real-time database access is required on laptops and desktops. Since updates must be prepared and downloaded to clients, it is inevitable for time to pass between data being added to the main database and its availability for download, and the actual download itself.

   
   > Back to top

4. The Marketplace
   1. Who are QD Technology's major competitors?

      QD is a unique product in the database marketplace today with no known competitors.

      While there are database accelerators, they are hardware-based solutions, or a mix of hardware and software and they only accelerate the database. They do not provide secure, compressed, local, read-only copies to the desktop or laptop.

      In many cases, accelerators are generally specific, niche-based tools that only work with one database, one kind of data, or one portion of a database's data. QD works with any ODBC-compliant database.

      Other products, such as traditional RDBMS's, business intelligence, or reporting tools are not competitors. QD takes the data that they manage or operate upon, and makes it available 24/7 while generally speeding up the work that these other applications already do.

      QD's advantage over all of these solutions comes from the balance between data accessibility, compression, and performance enhancement.

   
   > Back to top

5. General Technical Questions
   1. If the database is read-only, how can I update it? Does it require a complete refresh each time?

      The read-only nature of the QD means that users cannot change the copies of the database stored on their local machines. Incremental database updates can always be pulled to desktops and laptops from the main database server through QD's Incremental Update function.

      When an incremental update is prepared on the main database server, users are informed via email that it is ready, at which time they can download the update package to their desktops and laptops. The update is then automatically added to their existing QDB.

   2. From which databases on which platforms can QD accept data?

      QD can accept, process, and compress data from any ODBC-compliant RDBMS, regardless of its hardware platform, OS, or RDBMS application. QD can also process and compress flat file databases.

   3. What are the minimum recommended hardware and software configurations for QD?

      QD Server System

      • Supported OS's
        • Windows Server 2003 or 2000
        • Window XP SP1 or later
        • Windows Vista
      • Hardware/Software Requirements
        • Intel Pentium III or equivalent processor
        • 1 GB RAM (2 GB recommended)
        • 28 MB available hard disk space
        • MS Data Access Components 2.8 or later
        • Windows Installer 3.1 or later
        • ODBC drivers (2.8 compliant) for source db's
        • TCP/IP connection and Network bandwidth for loads and transmissions of QD Packages

      QD Client System

      • Supported OS's
        • same
      • Hardware/Software Requirements
        • Intel Pentium III or equivalent processor
        • 1 GB RAM
        • 16 MB available hard disk space
        • Windows Installer 3.1 or later
        • TCP/IP connection and Network bandwidth for downloads of QD Packages

   4. How does QD address security?

      The QD database can be encrypted using AES-256 encryption technology.

      If you choose not to encrypt the database, the data is still compressed using various compression algorithms and stored in libraries on the PC. Those algorithms are different depending on the kind of data that is being compressed. A license key is required to decompress the libraries. That compression serves to obfuscate the data, without truly encrypting it.

      In order to access the data through the standard QD client processes, a registered user-id and password must be provided upon the first query. Passwords are stored in the QD database using AES-256 encryption with a 16-bit key.

      The database can be set to expire at a predetermined time and date. The expiration time and date are set when the data is loaded onto the PC. When the expiration time arrives, the computer will either erase the data completely or simply lock out all attempts to access it, depending on configuration settings. Also, the data can be automatically erased or locked after a pre-determined number of failed login attempts.

      In addition, to deter against misuse and abuse of data by authorized users (or to support traceability of such suspected activity back to its source) QD provides an audit feature that captures and records critical events executed against a QDB. Audit information includes the actual SQL queries executed against a QDB along with the respective user id's of the individuals who executed them.

   5. Can I still use my production database?

      Yes. Your production database(s) will continue to run without change after QD is deployed. Since data analysts will no longer be performing queries against the production database, they will almost certainly see an improvement in its routine performance as well.

   6. Can users running QD on their laptop or desktop make changes to the primary database?

      The QDB on a laptop or desktop is designed to be read-only and it cannot accept changes from users running on that PC. The only changes that can be made to a QDB are updates from the primary database server.

      If a user wishes to change the master database, he must connect to it in the same fashion he used before QD was installed.

   7. Can QD take advantage of external storage?

      Yes. QD runs as an application on a desktop or laptop without regard for the type of storage that is used. If the application or the database itself is stored on a remote disk, via SAN, NAS, or other reliable network-based storage mechanism, there will be no impact on the functionality of QD.

   8. Can QD compress data from data cubes?

      QD is designed to accept data from traditional 2-dimension database tables, rather than from 3- or more-dimension data cubes.

      However, QD can be used as the source data for a utility such as SAS that generates cubes. The utility can simply be pointed at the QDB instead of the usual data mart or warehouse, to generate the cube. What's more, the cube will be generated far more quickly than it would without QD.

      Since QD dramatically compresses data, users may find that it's not necessary to restrict users to the contents of data cubes and may instead prefer users to have a much larger portion of the available data.

   
   > Back to top

6. Compression
   1. What specific compression techniques does QD employ?

      QD takes advantage of a variety of compression techniques:

      • Row packing. Since a QDB is read-only, QD does not need to reserve extra space for updates or deletions, or store predicate information for each block of row data. QD compresses each column based on the cardinality of the column's data (how many different entries appear in the column) and then it compresses each row using repetitive bit pattern techniques.

      • Floor values. In a business database the range of values associated with a particular row tends to be fairly narrow, although the numbers themselves may be large. QD also uses floors (essentially smaller numbers to represent larger numbers) for date and time columns; these represent actual numerical entries in a more compressed form.

      • Tokenization. When a database is blindly compressed with a single algorithm, the result can be a larger database. In most business applications, many columns will have a moderate number of values repeated many times. QD can create a token representing that repetition and then compress the column more efficiently. QD qualifies the column for tokenization and selectively tokenizes and can even share token translation lists between columns.

      QD also uses compressed data values in the column indexes. Further, it uses very large block-sizes (64K). If a conservative compression ratio of 5 and an original database block-size of 4K are assumed, that means that QD contains about 80 times the number of rows in each block of the QDB.

   2. Once in a compressed QDB, how is the data decompressed?

      When queries are made against the database, data is searched while remaining in its compressed form. The data gets decompressed only when it is returned as the result of a query. A shared library sits with the table that tells the query engine how to decompress it.

   3. Can compressed QD data be browsed?

      Yes, a user can browse the data in a compressed database. Browsing only decompresses the parts of the data that are actively being reviewed. Any tool that supports SQL can be used to browse the data.

   4. What are the variables that determine the compression ratio (QD vs. native RDBMS) other than the elimination of white space?

      The amount of compression that QD is able to achieve is based on the data content of each column in the database. Specifically, the actual data type, the range of values, and the number of unique values will determine the compression technique QD will use, especially for numeric and date fields. For non-numeric fields, the structure and cardinality of the column will be the primary factors that determine the amount of compression. For example, a telephone number is usually stored as a character field. However, it has a specific structure (nnn-nnn-nnnn) and is a mix of character and numeric sub-fields. The separation characters and the values of each sub-field can be compressed and stored separately.

      In addition, QD stores some columns by sharing bits within bytes. For example, Yes/No or True/False data can be stored in a single bit … by sharing the bit within a byte, up to eight columns of this kind of data can be stored in a single byte.

   
   > Back to top

7. Performance
   1. How does QD respond to ad hoc analytical read-only complex queries faster than a standard RDBMS on a PC?

      First, QD ensures that only those columns that are included in the query criteria will be retrieved from the disk in order for the query engine to compute the result set. The remaining row columns (i.e. those not included in the query criteria) are left alone. Retrieving only those columns that are necessary allows more data to be packed into fewer read operations, hence the greater query speed.

      Second, by compressing its data, the QDB can squeeze more data on a single page and then read it out faster. In any computer system, the disk is the performance bottleneck, as disk reads can be thousands of times slower than memory reads. A compressed database requires fewer read operations than an uncompressed one to retrieve its data.

      Because the data is compressed, transfers from disk to memory are faster and more data can be placed in cache memory. Data operations to memory are many hundreds of times faster than disk I/O operations.

      Since QD is read-only, no locking, logging, or other operations required in a read/write environment are required. As a result, QD can get data off the disk even faster.

   2. What gains in typical query performance result from high-levels of compression? If the compression is three to five times, is the query speed-up proportionate?

      A key value proposition is query performance. Speed depends on the data and the kinds of queries, however it should always be at least as fast as it is with standard databases and in many cases will be much faster.

      It is true that QD can compress source data by as much as 10 times, although three to five times is more common. In the 10x case, transferring the data off the disk would require about 90% less I/O transfer time. Despite that, the compression ratio is not necessarily directly related to query speed-up. Other factors can influence the speed-up, making it either faster or slower. Speed improvements are also achieved through the capture of the metadata.

      Additional performance enhancement is realized because the copy of the database that the user is accessing is local to his desktop or laptop. There is no contention for disks, CPU, memory, or other resources. The RDBMS does not incur any additional overhead from managing locks or simultaneous multiple user accesses.

   
   > Back to top

8. Data Sources
   1. What data sources does the QD support?

      Currently the data source is any ODBC database connection. QD can also compress and load flat files, either delimited or fixed, including e-mail logs.

      QD cannot currently manage images, as they are already compressed.

      QD cannot compress media files, though it can store pointers to the image, do the analytics and have links to the image.

   2. Can QD consolidate input data?

      Yes. QD can take data from multiple databases and consolidate it into a single compressed, high-performance, read-only database.

   3. How does QD work in enterprises that federate data sources using a services-based architecture?

      QD can take data from multiple data sources, including traditional RDBMSs, data warehouses, data marts and flat files and then combine it on the desktop, without requiring complex server-based federation tools.

   
   > Back to top

9. Query Tools/Complementary Products
   1. What SQL query building tools can be used with QD?

      Any SQL query tool that is compatible with SQL-92 will work with QD the same way it would work with any other RDBMS.

   2. How can QD be leveraged when an enterprise already has query applications in place?

      That's the best environment for QD. Enterprises with large, busy databases find that they cannot run the kinds of ad hoc queries that they want to. Often those queries can run for hours or days and seriously impact the performance of the production database.

      Since QD contains a complete database and runs on a local desktop or laptop, analysts are free to run queries against it without impacting production. They can take the time to develop complex queries, even making incremental improvements to the queries and rerunning them freely, without fear of affecting production.

      Unlike a standard transaction-based database, QD is optimized for these queries.

   3. If I have query applications in place today, how can I leverage a QD compressed database?

      Where pure SQL is employed in a query application, QD should work as a simple drop-in replacement database. However, where RDBMS-specific SQL customizations have been made, they will require modifications before they can be expected to work.

   4. Is everything in the ODBC standards fully supported?

      QD is fully compliant with the query portion of the ODBC Level 3 standard.

   5. Is JDBC supported?

      JDBC is supported through a JDBC/ODBC bridge.

   
   > Back to top

10. The QD Loader
    1. Can I still use my original database while the QD Loader is running against it?
       Yes. The QD Loader simply reads the database. It does not require any privileged or exclusive access to complete its task.

    2. How long does it take to load a database into the QD Loader?
       The current release of the QD Loader processes incoming data at approximately 7-10 GB per hour. Additional loader performance improvements are expected in future releases of QD.
    
    > Back to top

11. Deployment
    1. How long does it take to install QD?
       The installation of the QD software can be completed in less than an hour.

    2. Can I use my existing database with QD?
       In fact, you must use your existing database with QD. Since it does not permit database writes, it cannot replace Oracle, SQL Server, Sybase, Teradata, or any other RDBMS. QD makes those products work better and perform faster by providing users with local copies of the database. As a result, virtually all requests that come into the database will be write requests. Read requests will be handled on desktop/laptop-based copies of the database.

QD Technology, QD, QDT, and QD™ are trademarks of QD Technology LLC in the USA and other countries.

QD Technology may have patents or pending patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. The furnishing of this document does not give any company or individual any license to these patents, trademarks, copyrights, or other intellectual property rights except as expressly provided in any written license agreement from QD Technology.

Intel and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Microsoft, Windows, Windows NT, and SQL Server are registered trademarks or trademarks of Microsoft Corporation; Sun, Solaris, and Java are registered trademarks of Sun Microsystems, Inc.; Oracle is a registered trademark of Oracle Corporation; Sybase is a registered trademark of Sybase, Inc.; IBM, DB2, DB2 Universal Database, and OS/390 are registered trademarks of International Business Machines Corporation; Unix is a registered trademark in the U.S. and other countries, licensed exclusively through X/Open Company Limited.

Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. QD Technology LLC disclaims any proprietary interest in trademarks and trade names other than its own.