I tried simulating a network failure in mid-execution (I pulled out the network cable). The process on the database side terminated after completing the current transaction. The integrity of the data being archived was maintained. The launch statistics were updated correctly and also recorded error messages to the XML log file. All of these things worked as expected - great job!
–President
Consulting firm.
The Need for Database Archiving
Introduction
Database Archiving is about removing selected data records from transactional and operational databases especially which are rarely referenced, and storing them in an archive from where the records can be retrieved again, as necessary. Database Archiving Solutions are crucial for enterprises to improve system performance, meet compliance regulations, and indirectly save on time and money. The principles of database archiving are as under:
Assess: Determine which applications and versions are most in need of archiving, grouping them into categories based on your business requirements.
Classify: Document functional business rules and data retention policies to govern active, inactive and compliance-managed data.
Archive: Segregate historical business objects or transaction records from current activity. Safely move them to a secure archive.
Store: Store archived historical records securely and cost-effectively, according to the evolving business value.
Access: Apply service levels that provide decision makers with access to the historical records they need, when and how they need them.
Tune: Monitor operations to verify that archive operations continue to support desired service levels and access requirements.
Dispose: Prevent information assets from becoming information liabilities by deleting historical records after they are no longer required for compliance or business purposes
Challenges Faced
Large enterprises today are neck-deep with information flooded from all directions in their respective storage mediums. Unused data is continually eating up resources, and gobbling up the organization’s profits. Megabytes gave way to gigabytes, gigabytes have acceded to terabytes, and in the near future, terabytes is sure to escalate to petabytes. The data is growing relentlessly with organizations waking up to the reality of data explosion. The latest data retention clauses for pharmaceutical companies are at least 20 years while nuclear facilities will have to hoard data for 50 years. Database archiving solution helps in reducing costs, retaining data, and also aids in complying with regulation procedures.
The Sarbanes-Oxley Act, HIPAA and BASEL II are some of the laws and regulations related to data retention. It has been estimated that there are over 150 federal and state laws which elaborate extensively on the subject of data retention in the US. Today, the retention period is basically determined by the government itself. The data retention period which ranged from five to seven years previously, are now crossing the barriers of 20 to 70 years.
According to analysts at ByteandSwitch.com, management of backup and data recovery of the database in today's environment is a challenging proposition with a plethora of business requirements and needs to be taken care of. The volume of data is spurting doubly every year, forcing enterprises to shell out part of their financial resources for additional hardware and database software licenses.
Additionally, one of the most recent Gartner reports says that database archiving can significantly lower primary storage costs by transferring older data to relatively cheaper storage. Performance improvement and cost reduction is tremendous even for databases with size less than 200 GB. Analysts at both Gartner and Forrester also suggest the use of database archiving to lower primary storage costs significantly by transferring rarely accessed data to relatively cheap storage mediums and fast recovery formats like XML format.
Additional overheads of exploding data include the drastic changes in technology and increasing costs of storage hardware.
