Tiered Storage & How It Works

Tiered Storage is based on pools of storage whereby files are stored and accessed. These could be DAS (direct attached storage), NAS (Network Attached Storage) or SAN (Storage Area Network) running iSCSI or Fibre Channel. Typically a company would have a tier 1 storage platform and this would be the storage with the fastest access to the data, thereafter a company might have a large pool of SATA disk and tape library. The tier 1 storage would also be the most expensive to purchase and maintain, it is this storage that typically requires the biggest budget in order to store larger volumes of data.

Data Growth

Many organisations today are struggling with the sheer volumes of data being created and the issues of how to organise and store this data without the need to constantly back up, copy, move, and purchase new storage. A smarter approach involves implementing tiered storage, which categorises data based on its usage and importance.

The fundamental issue has been to keep all data—current and legacy—on spinning disks and then back this up. However, by adopting tiered storage, businesses can strategically determine what remains on tier 1 disk and what should move to lower-cost storage options, such as SATA RAID, cloud storage, tape, or optical.

A fact that has been known for many years is that up to 80% of your organisation’s data has not been accessed in the last 90 days, and at least 60% of it will not be required in the future. Tiered storage enables organisations to handle this disparity effectively, ensuring that less frequently accessed data is moved to more cost-efficient tiers.

By integrating tiered storage strategies, organisations can optimise performance, reduce costs, and avoid unnecessary spending on expanding high-performance storage unnecessarily.

HSM

The traditional way of archiving data was called HSM (hierarchical storage manager). This method evolved from the mainframe era, where all data in a given area was archived. During the 90s, HSM was the primary method for archiving information to a drive letter or device. However, today's businesses are moving towards tiered storage solutions, which address many of the challenges inherent in HSM.

The primary problems with HSM are as follows:

1. Data has to be pointed to a physical location.
2. No migration or policy rules can be applied to the data.
3. It is primarily a manual task.
4. Restores from archive are slow.
5. Users are not aware and cannot recover archived data. These operations must be carried out by system administrators.

Unlike HSM, tiered storage enables automated data migration based on policies, ensuring that data is stored in the most appropriate and cost-effective location. This approach solves many of the inefficiencies of traditional HSM while providing greater flexibility.

Due to increasingly complex data compliance regulations, the option to archive and store everything is not ideal. Tiered storage ensures compliance by allowing organisations to apply discovery tools, audit trails, and retrieval systems tailored to business needs.

Storing everything only increases the amount of storage space required, but tiered storage optimises storage use by moving less-accessed data to lower-cost tiers.

Users need to be able to find and recover files. Tiered storage enables self-service recovery without relying solely on system administrators.

Compliance officers require discovery tools and audit trails, which can be seamlessly integrated with tiered storage systems.

Keep it on spinning disk

The reluctance for companies to archive data is based upon a lack of knowledge and the fear of losing valuable information. The mindset of keeping it on RAID arrays and backing it up every night is still the example we encounter daily as a business. However, adopting tiered storage can significantly optimise data management and reduce costs.

As a consequence, the size of online data volumes is spiralling out of control, and storage management has become an ever-increasing challenge across many storage pools. Implementing tiered storage allows organisations to categorise data by its importance and usage, ensuring critical information stays on high-performance storage while older, less accessed data moves to cost-effective options.

• Server performance and data access are diminishing.
• Spiralling energy consumption and predicted power outages over the next 5-10 years.
• Business legislation and user demands are requiring companies to increase storage space to alleviate the problem.
• Data management puts high overheads on network and backup windows.
• DR policies for recovery take longer as all data needs to be restored rather than the most essential.
• The annual cost of managing this data over its lifetime is more than 5x the initial purchase price of the equipment.

Traditional ways of storing offline labelled tape or optical media on a shelf are problematic. Organisations today require information to be accessible at all times. This is where tiered storage can truly shine, providing a scalable solution that balances cost, performance, and accessibility. optical media on a shelf are problematic. Organisations today require information to be accessible at all times, making tiered storage a vital strategy for balancing accessibility, cost, and efficiency.

Active Data Archiving

Clearly for the reasons mentioned a better and more efficient way of managing data needs to be found.  Active Data Archiving through tiered storage is a method of managing data types by policy and controlling the movement of this information through different storage tiers and moving this information to a more suitable storage location. The ideal solution is to use a file system that natively stores the data where it belongs, according to the storage policies. Alternatively use file migration, this creates a symbolic link to the original files and moves the data to alternate tiered storage that could even be in the cloud!

An active tiered storage archive ensures the data is always available and accessible to users, albeit with a slight delay in restoring the information depending where and how it is stored.

Active Data Archiving overcomes many of these issues and enables companies to adopt an archiving strategy that can evolve and develop as business conditions dictate.

By deploying an Active Data Archiving strategy using tiered storage in any business will soon reap the rewards and wonder why they hadn’t done anything sooner.

Existing Investment in IT

Numerous organisations have a huge on-going investment in purchasing data storage systems and this investment is increasing year on year as the demands to store more information increases.  On average a businesses data is growing at 50%+ per annum due to video, audio, photos, AI, Machine Learning, social media and messaging.  After 3-5 years, this equipment is then replaced so the cycle continues.

In addition, a very cumbersome and time-consuming task in all organisations is to migrate data sets for various reasons: obsolete hardware, disk array full, etc.  A tiered storage solution easily migrates data from an existing storage to another over time.

With the implementation of a tiered storage solution we can actually extend the life of this investment by moving the data to a secure active archive, thereby freeing up valuable disk space on high performing storage solutions and slowing down the necessary and ongoing investment of more storage space giving a huge ROI benefit.  An additional benefit with active archiving is that you may be able to utilise your existing older storage systems to archive data.

Policies

As the need of corporate governance increases, companies need to know, retain and delete information by setting up policies. These policies contain rules which carry out instructions, such as:

• Copy
• Delete
• Migrate
• Move
• De-migrate
• Gather statistics
• Owner
• File size
• File type
• Date created
• Date Modified
• Last access date
• Attribute

Disaster Recovery

More organisations are adopting a DR strategy as part of their corporate governance and compliance regulations. A DR strategy can be a cluster of servers, remote data replication to a DR site or having backup tapes to restore.  An Active Tiered Storage Data Archive can significantly help in a DR environment by reducing the amount of information that is being backed up or replicated to a DR site. This will greatly assist DR restores as the only data on a file server needs to be restored.

In addition to the above, aged data can be automatically replicated to your DR site(s).

Storage Management

All data needs to be managed, how companies control and police the information lifecycle of their data is a continually growing problem. Many data compliance and corporate governance regulations require information to be stored and kept for a minimum number of years and thereafter deleted.

A tiered storage solution can alleviate all of these issues by creating policy rules based on the data type.  For example an insurer might issue a new policy to a customer and this information may need to be accessed over the following 60 days for accuracy and formal checks, thereafter it can be archived. The information will therefore remain on primary storage for the first 60 days, moving to SATA after this time and finally migrating to the cloud, tape or a Blu-ray optical disc after 120 days for long term archive.”

All this can be achieved with very little human intervention once the policy rules have been created and tiered storage setup and the whole information lifecycle process is fully automated.

Backup The Data

The backing up of data does not move the data through the tiered storage. All a backup does is ensure that the data can be restored in the event of a disaster. As data volumes rise the backup window shrinks and this again makes backup a more selective process rather than a backup everything policy.  A backup could be used to keep an archive copy of migrated data and this could be performed monthly or quarterly.

Corporate Governance and Data Compliance

The need for corporate governance and data compliance is becoming increasingly common in business. An Active Data Archive, combined with tiered storage, can become an essential part of a corporate governance and data compliance strategy. Tiered storage enables businesses to categorise and store data based on its value and usage, ensuring that critical data remains accessible while older or less relevant information is moved to cost-effective storage tiers.

By setting policies on data types, we can determine where it needs to be stored, how long it needs to be retained, and when it needs to be deleted or moved. With tiered storage, these policies can be implemented seamlessly, enabling the automated migration of data to appropriate storage tiers.

In addition to this, reporting and data discovery are necessary when trying to find or track file movements within the archive. This process, enhanced by tiered storage, can also identify file types that are not allowed on the corporate network and automatically delete the information or move it to another location for processing.

This combination of an Active Data Archive and tiered storage provides an essential part of any compliant archiving solution, balancing governance, cost, and efficiency.

Identifying The Data

The task many IT managers face is trying to identify current from historical data. We can provide this as a service and supply a detailed report outlining your IOP usage, who, what, where & when the files were last accessed. Once we identify our files we simply create simple policy rules to move the data based on age, file size, user/group, file extension, last accessed/modified date, project folder etc. These rules can then be scheduled to run out of hours and the data moved transparently through the storage tiers without the users knowledge. When the user wishes to access the file it is automatically pre-migrated to the primary tiered storage.

Reduced Costs

Once a tiered storage solution has been implemented the financial investment will be repaid in a relatively short time frame. As mentioned we can free up disk space on primary tier 1 storage, save energy by storing data on lower cost storage platforms and reduce our backup window.