QD Technology

The Basics
The Product
The QD Environment
The Marketplace
General Technical Questions
Compression
Performance
Data Sources
Query Tools/Complementary Products
The QD Loader
Deployment

The Basics
1. What is QD Technology?
  QD Technology is a premier provider of next generation database technology. QD's goal is to deliver the world’s fastest, smallest, compressed database to your desktop or laptop.
  QD's database technology, known as The Quick Response Database™, takes data from your standard Relational Database Management System (RDBMS), compresses it into a QD database format that is optimized for query execution performance, and delivers the database to desktops and laptops. This empowers users to obtain answers to their SQL queries faster than they can today. This is accomplished using a unique technology for compressing and indexing databases.
  Companies who leverage QD will see significantly improved analyst productivity, add new value with closer-to-the-customer analytic insights and capabilities, decrease storage costs and improve disaster recovery resilience.
2. When was QD founded?
  August 2004.
3. When did you begin selling your products?
  Spring of 2006.
4. Where are QD's offices?
  Our world headquarters is located in Rutherford, NJ (outside New York City). We have a sales office in Los Angeles, CA, an engineering organization based in Austin, Texas and a team of over 50 developers and engineers in St. Petersburg, Russia.
> Back to top
The Product
1. What is QD and what does it do?
  QD is a Relational Database Management System (RDBMS) that performs SQL queries at speeds that exceed that of traditional RDBMSs.
  QD can take data from any ODBC-compliant RDBMS or properly delimited flat file and make it available to users in a read-only, compressed form. Users can then download the data in this form directly to their desktop or laptop.
  The read-only, compressed data stored locally in QD can be accessed with standard SQL tools and can be queried faster than would be possible from its native RDBMS.
2. Where did the idea for QD originate?
  The original idea for QD came from Jack Olson. Jack was CTO and VP of Development for Evoke Software and Peregrine Systems. Before that, he spent eight years at BMC Software as corporate architect for all DB2 products and 17 years at IBM as the chief architect a wide range of products including CICS, IMS, AIX and DB2 for Unix and OS/2.
  As Jack tells the story, "A few years back I was looking for an RDBMS that would give me extremely high performance to help me perform some very advanced data and assessment queries. Oracle and Sybase and the others were just burning up their servers which seemed unnecessary. So I did a web search to see if anybody had developed specialized databases with high performance for ad hoc queries and very large amounts of data. I found five companies in the space. None of them had any kind of serious traction in the market place.
  "Although each of the companies was still in business, they basically missed the mark with their products. None of them was willing to make the database read-only and so they couldn't deliver the required level of performance. Today only one of them is still in business and they are in a very specialized niche market.
  "So, I decided to bring the technology to market and do it right this time."
3. What benefits does QD deliver?
  The ultimate benefit of QD is 24/7 access and improved insight into the data stored in large corporate databases. Insight is improved because database queries will be faster than they have ever been before and because queries that were not possible under traditional RDBMS will be more than possible – they will be easy and fast with QD.
  With QD, data analysts can build more complex queries at their own speed and convenience. They will be able to test and build them iteratively; they can even make mistakes along the way, since running these queries will not impact the production database. Analysts can also run simple ad hoc queries against the database.
  In many organizations, database and system administrators have imposed strict policies that prohibit running intensive queries against the production database, or limit these queries to non-production hours. Queries that can't complete in the allotted time are not permitted to run. As a result, QD enables analysts and their organizations to run these queries and gain new insights into their corporate data and key business drivers.
4. What are the product features that deliver these benefits?
  QD has three primary features:
  - Compression. QD takes one or more databases and compresses them down to about one-third to one-fifth of their original size. Greater compression results can be achieved, depending on the nature of the data being compressed.
  - Performance. QD can access the compressed database at speeds that are faster than can be achieved by traditional RDBMS's. This is made possible through a combination of techniques including smart compression, more complete indexing of data, and a SQL engine that is optimized for querying.
  - Dedicated local copies. QD places a copy of the compressed database on a user's laptop or desktop. This allows the user to access it locally without impacting the production database, its servers, or the corporate network. The production server does not need to manage locks or logs or any other overhead.
5. What are some of the secondary benefits to implementing QD?
  - Since personal computers running QD can access their databases without being connected to the network, the remote systems can operate even if the main database server goes down. This provides increased availability and even limited disaster tolerance for the database and its related applications.
  - Since the database is portable, you can bring the entire database to a customer meeting, even when no network access is available. This can give a resourceful salesperson or analyst the means to vastly improve data access. And since QD can compress and share the entire database, or even multiple databases, users have access to more data than ever before.
  - Once the simple initial set up is completed, minimal database administration is required to manage the read-only database copies on desktops and laptops.
  - Many large database implementations consist of a master database server, which feeds data to a data warehouse, which in turn makes data available to a series of smaller data marts. In some larger implementations, additional layers of hardware may be in use. QD simplifies the whole operation because it can place the entire database or data warehouse on a user's laptop or desktop. QD's infrastructure simplification can translate to significant financial savings in systems, networking, administration, support, licensing and software.
  - The master database is totally protected against any action a user might take on his laptop; a user cannot do anything to his read-only QD database that will affect the master database.
  - Data security is an integral part of the QD system architecture. QD server-side and client-side components deliver an integrated set of features to safeguard against outright data theft from the outside, as well as misuse and abuse of data on the inside.
  - QD can federate tables from multiple databases into a single QDB (a physical Quick Response Database) which can then be queried as a single database.
6. What are the components of the QD system?
  The components of a QD system are partitioned across a server system and one or more client systems:
  - The QD Administrator is a GUI application that runs on the QD Server System, and which is used to define users, roles, and databases for authentication and authorization within QD.
  - The QD Loader automates the decision process for loading, compressing, and indexing the data. It reads the entire source database and examines each column in every table and schema to decide which indexing and compression methods to use for each part of the database. The Loader supports both full and incremental loading of the specified data sources.
  - The QD Server is a continuously running TCP/IP server application on the QD Server System whose purpose is to handle client requests for QDB downloads. The Server establishes the authority for the user to access databases on the QD Server System machine. Once it has authorized a user, the Server then enables replication of new databases, and updates to existing databases on the user's Client System.
  - The QD Client System Manager (CSM) is a GUI application that runs on the QD Client System, and which supports the definition and replication of the CSM user's local QDB. It connects via a TCP/IP secure SSL link to a constantly running QD Server application on the QD Server System.
  - The QD SQL Engine determines the most efficient execution plan for accessing a QDB's data in order to answer a specified SQL query. It then performs the optimized execution plan and provides the desired query result set.
  - The QD Notifier alerts the QD Client System user of available QDBs residing on the QD Server System.
7. What differentiates QD from a traditional RDBMS?
  - Network independence. Users can work on the QD without being connected to the main database server, or to any other external computer or network at all. Therefore, they can query their database while on an airplane or in a secure site where outside network access is prohibited.
  - A subscription service which provides users with a data feed (and email notification), whenever the QD is ready to provide a download.
  - A SQL engine that is optimized for query performance. Because QD is read-only, query execution is not encumbered by overhead or data structural inefficiencies owing to the need to be able to perform database inserts or updates.
  - Simple architecture. Because QD is read only, no complex infrastructure is required for transactional processing, locking, or updates. QD is a tight, compact solution.
  - Portability. The decompression algorithm travels with the table itself. If new decompression algorithms are brought into QD, all of the old tables can still be read.
  - Flexibility. Different rows of the database can be compressed with different compression algorithms. This allows higher levels of compression than could be provided by any single algorithm.
  - Future-proofing. Because we store information about the compression algorithm in the compressed database, a database that is compressed with today's release will still be readable and usable in all future releases of QD. We have not found another compression-based database where this is true.
8. How are queries run against a QD database?
  QD works with your existing query tools and does not require that your data analysts learn any new tools or data analysis techniques.
> Back to top
The QD Environment
1. What environments offer the best fit for QD?
  - Where there are one or more large, busy databases where users and analysts cannot perform the kinds of queries and data analysis that need to be done; those queries and analysis could consist of simple ad hoc queries, or more complex queries that an analyst develops over a long period of time.
  - Where many remote employees could benefit from having continuous access to their database.
  - Where the network or the database server are overburdened with database queries.
  - Where database administrators cannot provide satisfactory service to all of their remote users.
2. In which environments might QD be a poor fit?
  - Where end-users require read-write access to the database.
  - Where end-users are required to use thin clients. A thin client will not have sufficient disk space or memory to store the compressed database or to execute complex queries.
  - Where real-time database access is required on laptops and desktops. Since updates must be prepared and downloaded to clients, it is inevitable for time to pass between data being added to the main database and its availability for download, and the actual download itself.
> Back to top
The Marketplace
1. Who are QD Technology's major competitors?
  QD is a unique product in the database marketplace today with no known competitors.
  While there are database accelerators, they are hardware-based solutions, or a mix of hardware and software and they only accelerate the database. They do not provide secure, compressed, local, read-only copies to the desktop or laptop.
  In many cases, accelerators are generally specific, niche-based tools that only work with one database, one kind of data, or one portion of a database's data. QD works with any ODBC-compliant database.
  Other products, such as traditional RDBMS, business intelligence, or reporting tools are not competitors. QD takes the data that they manage or operate upon, and makes it available 24/7 while generally speeding up the work that these other applications already do.
  QD's advantage over all of these solutions comes from the balance between compression and performance enhancement, plus the advantages that come from placing the database on a local laptop or desktop.
> Back to top
General Technical Questions
1. If the database is read-only, how can I update it? Does it require a complete refresh each time?
  The read-only nature of the QD means that users cannot change the copies of the database stored on their local machines. Incremental database updates can always be pulled to desktops and laptops from the main database server through QD's Incremental Update function.
  When an incremental update is prepared on the main database server, users are informed via email that it is ready, at which time they can download the update package to their desktops and laptops. The update is then automatically added to their existing QDB.
2. From which databases on which platforms can QD accept data?
  QD can accept, process, and compress data from any ODBC-compliant RDBMS, regardless of its hardware platform, OS, or RDBMS application. QD can also process and compress flat file databases.
3. What are the minimum recommended hardware and software configurations for QD?
  QD Server System
  - Supported OS's
    - Windows Server 2003 or 2000
    - Window XP SP1 or later
    - Windows Vista
  - Hardware/Software Requirements
    - Intel Pentium III or equivalent processor
    - 1 GB RAM (2 GB recommended)
    - 28 MB available hard disk space
    - MS Data Access Components 2.8 or later
    - Windows Installer 3.1 or later
    - ODBC drivers (2.8 compliant) for source db's
    - TCP/IP connection and Network bandwidth for loads and transmissions of QD Packages
  QD Client System
  - Supported OS's
    - same
  - Hardware/Software Requirements
    - Intel Pentium III or equivalent processor
    - 1 GB RAM
    - 16 MB available hard disk space
    - Windows Installer 3.1 or later
    - TCP/IP connection and Network bandwidth for downloads of QD Packages
4. How does QD address security?
  The QD database can be encrypted using AES-256 encryption technology.
  If you choose not to encrypt the database, the data is still compressed using various compression algorithms and stored in libraries on the PC. Those algorithms are different depending on the kind of data that is being compressed. A license key is required to decompress the libraries. That compression serves to obfuscate the data, without truly encrypting it.
  In order to access the data through the standard QD client processes, a registered user-id and password must be provided upon the first query. Passwords are stored in the QD database using AES-256 encryption with a 16-bit key.
  The database can be set to expire at a predetermined time and date. The expiration time and date are set when the data is loaded onto the PC. When the expiration time arrives, the computer will either erase the data completely or simply lock out all attempts to access it, depending on configuration settings. Also, the data can be automatically erased or locked after a pre-determined number of failed login attempts.
  In addition, to deter against misuse and abuse of data by authorized users (or to support traceability of such suspected activity back to its source) QD provides an audit feature that captures and records critical events executed against a QDB. Audit information includes the actual SQL queries executed against a QDB along with the respective user id's of the individuals who executed them.
5. Can I still use my production database?
  Yes. Your production database(s) will continue to run without change after QD is deployed. Since data analysts will no longer be performing queries against the production database, they will almost certainly see an improvement in its routine performance as well.
6. Can users running QD on their laptop or desktop make changes to the primary database?
  The QDB on a laptop or desktop is designed to be read-only and it cannot accept changes from users running on that PC. The only changes that can be made to a QDB are updates from the primary database server.
  If a user wishes to change the master database, he must connect to it in the same fashion he used before QD was installed.
7. Can QD take advantage of external storage?
  Yes. QD runs as an application on a desktop or laptop without regard for the type of storage that is used. If the application or the database itself is stored on a remote disk, via SAN, NAS, or other reliable network-based storage mechanism, there will be no impact on the functionality of QD.
8. Can QD compress data from data cubes?
  QD is designed to accept data from traditional 2-dimension database tables, rather than from 3- or more-dimension data cubes.
  However, QD can be used as the source data for a utility such as SAS that generates cubes. The utility can simply be pointed at the QDB instead of the usual data mart or warehouse, to generate the cube. What's more, the cube will be generated far more quickly than it would without QD.
  Since QD dramatically compresses data, users may find that it's not necessary to restrict users to the contents of data cubes and may instead prefer users to have a much larger portion of the available data.
> Back to top
Compression
1. What specific compression techniques does QD employ?
  QD takes advantage of a variety of compression techniques:
  - Row packing. Since a QDB is read-only, QD does not need to reserve extra space for updates or deletions, or store predicate information for each block of row data. QD compresses each column based on the cardinality of the column's data (how many different entries appear in the column) and then it compresses each row using repetitive bit pattern techniques.
  - Floor values. In a business database the range of values associated with a particular row tends to be fairly narrow, although the numbers themselves may be large. QD also uses floors (essentially smaller numbers to represent larger numbers) for date and time columns; these represent actual numerical entries in a more compressed form.
  - Tokenization. When a database is blindly compressed with a single algorithm, the result can be a larger database. In most business applications, many columns will have a moderate number of values repeated many times. QD can create a token representing that repetition and then compress the column more efficiently. QD qualifies the column for tokenization and selectively tokenizes and can even share token translation lists between columns.
  QD also uses compressed data values in the column indexes. Further, it uses very large block-sizes (64K). If a conservative compression ratio of 5 and an original database block-size of 4K are assumed, that means that QD contains about 80 times the number of rows in each block of the QDB.
2. Once in a compressed QDB, how is the data decompressed?
  When queries are made against the database, data is searched while remaining in its compressed form. The data gets decompressed only when it is returned as the result of a query. A shared library sits with the table that tells the query engine how to decompress it.
3. Can compressed QD data be browsed?
  Yes, a user can browse the data in a compressed database. Browsing only decompresses the parts of the data that are actively being reviewed. Any tool that supports SQL can be used to browse the data.
4. What are the variables that determine the compression ratio (QD vs. native RDBMS) other than the elimination of white space?
  The amount of compression that QD is able to achieve is based on the data content of each column in the database. Specifically, the actual data type, the range of values, and the number of unique values will determine the compression technique QD will use, especially for numeric and date fields. For non-numeric fields, the structure and cardinality of the column will be the primary factors that determine the amount of compression. For example, a telephone number is usually stored as a character field. However, it has a specific structure (nnn-nnn-nnnn) and is a mix of character and numeric sub-fields. The separation characters and the values of each sub-field can be compressed and stored separately.
  In addition, QD stores some columns by sharing bits within bytes. For example, Yes/No or True/False data can be stored in a single bit ... by sharing the bit within a byte, up to eight columns of this kind of data can be stored in a single byte.
> Back to top
Performance
1. How does QD respond to ad hoc analytical read-only complex queries faster than a standard RDBMS on a PC?
  By compressing its data, the QDB can squeeze more data on a single page and then read it out faster. In any computer system, the disk is the performance bottleneck, as disk reads can be thousands of times slower than memory reads. A compressed database requires fewer read operations than an uncompressed one to retrieve its data.
  Because the data is compressed, transfers from disk to memory are faster and more data can be placed in cache memory. Data operations to memory are many hundreds of times faster than disk I/O operations.
  Since QD is read-only, no locking, logging, or other operations required in a read/write environment are required. As a result, QD can get data off the disk even faster.
2. What gains in typical query performance result from high-levels of compression? If the compression is three to five times, is the query speed-up proportionate?
  A key value proposition is query performance. Speed depends on the data and the kinds of queries, however it should always be at least as fast as it is with standard databases and in many cases will be much faster.
  It is true that QD can compress source data by as much as 10 times, although three to five times is more common. In the 10x case, transferring the data off the disk would require about 90% less I/O transfer time. Despite that, the compression ratio is not necessarily directly related to query speed-up. Other factors can influence the speed-up, making it either faster or slower. Speed improvements are also achieved through the capture of the metadata.
  Additional performance enhancement is realized because the copy of the database that the user is accessing is local to his desktop or laptop. There is no contention for disks, CPU, memory, or other resources. The RDBMS does not incur any additional overhead from managing locks or simultaneous multiple user accesses.
> Back to top
Data Sources
1. What data sources does the QD support?
  Currently the data source is any ODBC database connection. QD can also compress and load flat files, either delimited or fixed, including e-mail logs.
  QD cannot currently manage images, as they are already compressed.
  QD cannot compress media files, though it can store pointers to the image, do the analytics and have links to the image.
2. Can QD consolidate input data?
  Yes. QD can take data from multiple databases and consolidate it into a single compressed, high-performance, read-only database.
3. How does QD work in enterprises that federate data sources using a services-based architecture?
  QD can take data from multiple data sources, including traditional RDBMSs, data warehouses, data marts and flat files and then federate it on the desktop, without requiring complex server-based federation tools.
> Back to top
Query Tools/Complementary Products
1. What SQL query building tools can be used with QD?
  Any SQL query tool that is compatible with SQL-92 will work with QD the same way it would work with any other RDBMS.
2. How can QD be leveraged when an enterprise already has query applications in place?
  That's the best environment for QD. Enterprises with large, busy databases find that they cannot run the kinds of ad hoc queries that they want to. Often those queries can run for hours or days and seriously impact the performance of the production database.
  Since QD contains the complete database (or selected parts of it) and runs on a local desktop or laptop, analysts are free to run queries against it without impacting production. They can take the time to develop complex queries, even making incremental improvements to the queries and rerunning them freely, without fear of affecting production.
  Unlike a standard transaction-based database, QD is optimized for these queries.
3. If I have query applications in place today, how can I leverage a QD compressed database?
  Where pure SQL is employed in a query application, QD should work as a simple drop-in replacement database. However, where RDBMS-specific SQL customizations have been made, they will require modifications before they can be expected to work.
4. Is everything in the ODBC standards fully supported?
  QD is fully compliant with the query portion of the ODBC Level 3 standard.
5. Is JDBC supported?
  JDBC is supported through a JDBC/ODBC bridge.
> Back to top
The QD Loader
1. Can I still use my original database while the QD Loader is running against it?
  Yes. The QD Loader simply reads the database. It does not require any privileged or exclusive access to complete its task.
2. How long does it take to load a database into the QD Loader?
  The current release of the QD Loader processes incoming data at approximately 7 GB per hour. Additional loader performance improvements are expected in future releases of QD.
> Back to top
Deployment
1. How long does it take to install QD?
  The installation of the QD software can be completed in less than an hour.
2. Can I use my existing database with QD?
  In fact, you must use your existing database with QD. Since it does not permit database writes, it cannot replace Oracle, SQL Server, Sybase, Teradata, or any other RDBMS. QD makes those products work better and perform faster by providing users with local copies of the database. As a result, virtually all requests that come into the database will be write requests. Read requests will be handled on desktop/laptop-based copies of the database.

QD Technology, QD, QDT, and QD™ are trademarks of QD Technology LLC in the USA and other countries.

QD Technology may have patents or pending patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. The furnishing of this document does not give any company or individual any license to these patents, trademarks, copyrights, or other intellectual property rights except as expressly provided in any written license agreement from QD Technology.

Intel and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Microsoft, Windows, Windows NT, and SQL Server are registered trademarks or trademarks of Microsoft Corporation; Sun, Solaris, and Java are registered trademarks of Sun Microsystems, Inc.; Oracle is a registered trademark of Oracle Corporation; Sybase is a registered trademark of Sybase, Inc.; IBM, DB2, DB2 Universal Database, and OS/390 are registered trademarks of International Business Machines Corporation; Unix is a registered trademark in the U.S. and other countries, licensed exclusively through X/Open Company Limited.

Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. QD Technology LLC disclaims any proprietary interest in trademarks and trade names other than its own.