Industry Info

Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3 Growth in the Entertainment & Media (EM) Market . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3 Archiving in the Entertainment and Media (EM) Defined . . . . . . . . . . . . . . . . . . . . . . . . . . .
5 Solutions and Products for the EM Archiving Market . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6 Issues and Opportunities in EM Archiving . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7 Format conversion and long term archive management . . . . . . . . . . . . . . . . . . . . . . . . . . .
7 Creating better metadata for archived content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9 Decreasing the overall costs of archiving . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9 Projections for Various Storage Media Used in EM Archiving . . . . . . . . . . . . . . . . . . . . . . .
10 Broader Asset Management Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11 About the Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Introduction

This report explores the current trends and drivers for digital archiving in the entertainment and professional media markets. Methodology and technologies utilized in digital archiving as well as future requirements will be explored. We also examine the requirements for long term content storage, look at various storage options and discuss how content administrators can create a reliable long-term archive.

Growth in the Entertainment & Media (EM) Market

To understand the demand for archiving in the Entertainment & Media market, it is important to begin with a discussion of the growth in content retained by all forms of EM companies from broadcast to post and animation to digital imaging and pre-press.

For companies participating in the entertainment and professional media markets their assets are primarily the rich content that they create and manage. A movie, a piece of music, a television program, a documentary, a video or audio file for web download and news or advertising pieces all generate revenue. They also contain large amounts of data and consume storage capacity. It is also important to note that the raw data collected to make these assets is much greater than what will be used in an actual product.

For instance, a professional animal videographer told me that he will shoot 10 hours of high resolution video to result in 5 minutes selected and edited for a television documentary. Those 10 hours of video were captured in 10 to 40 minute periods over a course of many months while waiting outside for hours at a time to capture these images. Some days he doesn’t get any images. Obviously, for content creation professionals there is a tremendous investment in the raw content.

These pieces of content are also unique and can never be repeated exactly the same—time moves on. As a result, raw and edited content are of inestimable value in preserving a record of natural history, historical events, hits and fads of a particular age and generally a cultural record of who we were and what mattered to us.

For the videographer to maintain this 10 hours of raw content he would need 648 GB of storage at HD-cam resolution. Retaining raw content can get expensive as it is continuously acquired and maintained. Of course this problem is not unique to videographers. From broadcast to post to imaging, all media and entertainment artists have to deal with storage of growing collections of raw content.

The effect of this is that professionals are creating and storing more content than ever before. Figure 1 shows projections out to 2012 for the growth of annual storage capacity requirements for the creation of professional moving image content . In 2007 over 700 petabytes of storage hardware are required while in 2012 this is estimated to climb to about 2.4 exabytes.

Figure 1. Annual Capacity Projections for Creation of Professional Moving Image Content

One of the drivers of this increase in storage capacity is the move to higher resolution content. As the resolution of video content increases the digital storage required increases. Table 1 shows the bandwidth and 1-hour storage capacity demands of typical professional media formats (including sound) and the primary applications that they are used in. The move from SD to HD results in 6X higher data rates and storage capacities while 2K to 4K results in 4X higher data rates and storage capacities. Bandwidth increases result in greater production of storage content and thus more demand for storage capacity.

Table 1. Example Raw Data Rates, Professional Media Standards

Resolution	Application	MB/sec	Storage (GB, 1 hour)
Uncompressed Audio (48 kHz/24-bit)	All	0.15	0.540
MiniDV/DVCAM/
DVCPRO (Digital Video)	Consumer, Corporate, Broadcast & News	3.6	12.96
IMX D10	Broadcast & News	6	21.6
HD Cam (compressed hi-def)	Broadcast, Post Production	18 (max)	64.8 (max)
SD Video (8-bit, uncompressed)	Broadcast, Post-Production	20	72.0
Uncompressed 601 (SDI) (10-bit standard def.)	Broadcast, Post-Production	34	122.4
High Res. HDTV (8-bit, 1080i)	Broadcast, Post-Production	120	432
Uncompressed 2K (10 bit-log)	Film Production, Television	300	1,080
Uncompressed 4K (10 bit-log)	Film Production, Some Television	1,200	4,320

It should be realized that Figure 1 gives an estimate for hardware storage assets required to make digital content in each year. The actual digital content generated using this hardware will tend to be even greater. Raw content stores alone can be considerable since a movie producer might edit hundreds of hours of content to come up with a 2 hour final product. Then there is the unused content saved for out-takes or even sequels and the digital intermediaries and effects created during production. Producers that make a practice of saving all their source raw material can amount in some cases with several petabytes of content.

When the digital content created and retained from prior years Figure 2. Cumulative Digital Storage Projection for is added in, the cumulative digital storage required for a Professional Video Digital Content Preservation. company to maintain their revenue generating content for the long term can become very great indeed. This is especially true as more ways to use and distribute captured content are identified (e.g. Internet and mobile phones) and the demand for richer and richer content increases.

Figure 2 shows estimates for storage capacity required to retain digital cumulative content created since 2004 . These estimates assume that only a part of the total raw content is preserved. If the percentage of raw content retained over the long term is greater than assumed here the total cumulative storage capacity could increase by 50% or greater than shown here.

To these estimates must also be added the digital storage required for the retention of digitally converted, formerly analog content such as old movies and television programs as well as music and digital still images. Also extra copies of content such as are needed for disaster recovery will add to the total storage requirement. As of 2007 the total unconverted analog video and movie content that could potentially be converted to digital form is on the order of 200 exabytes.

The growth of converted content depends upon the assumptions made about the conversion rate and resolution requirements as well as the continued rate of creation of analog content that may need to be converted. Altogether, including copies made of content archives for disaster recovery the total digital storage required for the cumulative preservation of professional digital content by 2012 is estimated to be many exabytes.

As can be imagined, storing all of this content on disk is not feasible. For this reason, the entertainment and media industry is at the forefront of using archiving to cost-effectively retain and manage this content.

Archiving in the Entertainment and Media (EM) Defined

An archive is a copy of data that is being retained for very long periods of time, usually in years and in some cases centuries. Archives are used throughout the entertainment and entertainment industry for storing content that is not being used in ongoing projects, but could be re-purposed or referenced in the future. An archive may be active, online, where it can be accessed relatively quickly or cold, offline, where it can be stored safely and economically, but may take a considerable amount of time to mount the digital storage medium and read the archived data. The average time to access archived data is the archive latency.

Editing and some other content industry segments also keep working archives of content on storage networks during the course of their work. These working archives are raw content and edited content that are protected during active work on a project. They are often kept in storage area networks (SAN) or network attached storage (NAS) systems used in the working studio. After a project is completed, the content of a working archive may be retained in a long term archive depending on the value and time to create intermediaries, certain effects, and other content used to develop the final cut.

Status of the Professional EM Archiving Market

Archiving within the EM market is currently driven by two factors: The need to cost-effectively retain content for re-use and the need to convert historical analog content to digital form to prevent degradation of content. Many sorts of facilities keep content for varying periods of time. Keeping completed content in long term archives is common practice by content owners, including movie and television production.

Raw content retention is not so complete and varies depending upon the policies and budgets of the facilities. Post, special effects and computer generation houses may keep some of their unique content for extended periods. As the cost of archiving declines more content will be retained by all of these facilities.

For preservation of new content, deciding what to archive can be difficult. The retention of digital entertainment and professional media involves costs in real estate, operation, management software and hardware. In practice data managers handling large and even small volumes of raw and edited content must make choices in what they will preserve for the long term because they often do not have the resources or budgets to save everything.

Preservation of source media containing the original content is almost always a hard requirement since it is difficult if not impossible to recreate, but this is often insufficient since raw source material does not capture any edits or metadata generated during the processing of the raw content to create a finished product. As a result, more content must be archived.

To this end, many movie studios and editing facilities will save a set of master movie films (with color separation preserved) as well as managed digital tapes of the content in a cold archive. In addition, many of these facilities will keep copies of content on-line in an active archive on a disk array or tape library for some period of time.

Many television networks, including major news networks, retain their library of analog video tape with a digitized index and metadata database of their analog tapes. They convert the analog content to a digital form and store it on a combination of disk arrays and tape libraries as they are accessed and used for historical content for current projects. Networks tend to keep cold as well as active archives so they can improve their odds of rapid content retrieval and successful long term content retention.

Smaller organizations, including smaller post-production shops and other facilities, may not have the resources to set up a complex and expensive archiving system. For these users it is advantageous to retain the source media and players to play them back later. Careful logging and management of these physical media combined with retention of the most valuable content in disk arrays or tape libraries may be a more affordable option.

While archiving will be driven long term by new content creation, there is also a sharp increase in digital storage capacity used for digital conversion and preservation of analog historical content. Costs for digital conversion are being reduced with the development of service providers that can do the conversion of bulk material at an attractive price and as the overall cost of storing digital archives is reduced. We project that several exabytes of digital storage will be used for digital archiving and content conversion and preservation by 2012 !

Long-term retention of digital files is not only a need of the entertainment and professional media industry. Regulatory and other pressures are driving many industries and companies to turn to long term data retention. The Storage Networking Industry Association (SNIA) started a 100 year archive project in 2004 and in August 2007 it released a survey of business professionals on their long term archiving needs. In that survey it was found that 80 percent of respondents have information they must keep over 50 years, and 68 percent of respondents said they must keep this data more than 100 years. Thus there is a strong incentive to develop hardware and management software to support long term retention requirements for several industries.

Solutions and Products for the EM Archiving Market

Solutions and products for archiving vary on the overall size and needs of an organization.

For smaller organizations, archiving may be a simple process in which final cuts from working storage is written out in digital format to media and placed in a cold, offline archive along with source tapes. Products used for data movement are typically asset managers but may also include backup applications. For digital content written out from working storage, archive media uses “commoditized” IT storage like digital tape, optical storage and even disk drives and disk drive cartridges. These commoditized products could include LTO and DLT tapes, DVD or blue laser DVD, write one or rewritable media as well as Iomega REV drives or disk drives in removable modules.

Factors in the choice of cold archive storage formats include not only cost, $/GB, but the expected life of the archive media. Media life depends upon the quality of that media as well as the environment that it is stored in. It is likely that inexpensive off-the-shelf optical disks, for instance, may not provide long term data preservation (many ten’s of years). Disk too, while an easy to use format, is not suitable for cold archives because of its life span. For this reason, most cold archives use tape (e.g. LTO) due to its portability, quality, and long shelf life. In any event, media should be stored in a climate controlled vault for high quality preservation.

Larger organizations will also use cold archives but in conjunction with some form of active archiving. In real world deployments, active archiving tends to include a hierarchical system consisting of multiple storage tiers. A disk storage array may be used on the front end to provide faster content transfers between artists during ongoing and recently completed projects. Less frequently used files will reside on a local tape or even optical disk library system for nearline access.

Quantum's StorNext - Intelligent Archiving Software

Quantum makes a software package for both active and cold archiving called StorNext. Composed of a shared file system and a storage manager for archiving, StorNext is used throughout entertainment and professional media for rapid data sharing, long term archiving, and content protection. The file system component of StorNext enables systems – UNIX, Windows, Linux, and Mac – to share a consolidated disk repository.

Archiving of content on the disk repository is managed by the Storage Manager module of StorNext. Storage Manager uses an embedded policy engine to transparently copy and move content from primary storage to active and cold archives. Content movement is handled based on multiple parameters that allow administrators to define both archiving and protection policies. For protection, copies of files are made to a secondary tier, typically tape, within a set number of minutes or hours after the file is closed.

For archiving, StorNext moves content to active archive tiers of disk, disk with de-duplication, and tape. This movement frees up primary disk capacity for use by new projects and minimizes the amount of primary storage needed. Archived files or protection copies of files can be sent to a cold archive for long term storage as needed. Additionally, StorNext offers alerting services, data migration services and integrity checks to validate the quality of archived data when it is re-used.

Disk arrays used for active archiving typically use SATA hard drives since these provide higher storage capacity for a given price and provide good reliability under less frequently accessed conditions expected in an archive application. There are even SATA storage arrays called MAIDs (massive array of inactive disks) that keep most of the SATA drives in the array powered down most of the time, providing significant power and heat load savings for a data center.

Data movement between tiers in an active archive can be accomplished via administrative action (manually moving files to different storage locations) but this is a cumbersome process and can result in data loss. To alleviate this issue, most organizations that use active archiving rely on products that automate data movement between tiers. These products often also include the ability to protect content by cloning of files and allowing them to be placed in a cold archive. Active archiving products are built to include or work side by side with content asset managers.

Some of the companies making software used for asset management and digital archive management in the entertainment and professional media markets (either by themselves or in combination) include: Dalet, Etere, Masstech Group, Quantum, SGI, SGL and XenData. Cold archiving is done by companies such as Iron Mountain that specialize in retaining records and assets in controlled environments.

Issues and Opportunities in EM Archiving

In this section we shall examine some important issues in the archiving of digital entertainment and professional media content to get a better understanding of them and to understand how they can be managed. This shall include what is involved in dealing with format conversions, protection of archived data from disasters at the primary data center, longevity of various possible storage media and total operating costs in maintaining a digital media archive.

Format conversion and long term archive management

Over the course of time storage media formats and interfaces become obsolete. This poses special problems for a long term digital archive. In the case of retained source media extra copies of the reader devices for the media as well as systems that these readers can attach to are required to access this media in the future. If the retention time is especially long this could even include a string of devices that can be used sequentially to convert older media contents into then current formats and with interfaces that then current devices support.

Thus to be secure in the future use of source media an archivist needs to also keep all the hardware and other software needed to read and convert that content into a form that will be readable to future hardware and software. Over the course of time this can become quite a collection of devices and can become a serious management issue itself.

Regardless of whether the original source media are retained, if the content data is transferred into a media archive system using hard disk drives, optical disks or magnetic tape format obsolescence still forces the administrator to convert older storage formats into newer ones as time goes by. Storage formats that include backwards compatibility mitigate this issue, but do not solve it.

This issue becomes more pronounced as the size of the digital archive increases. The time to transfer or convert digital content from the old media to the new can be overwhelming unless there are sufficient old and new resources on hand.

De-Duplication Takes Center Stage In Creating An Active Archive

Entertainment archives are growing in size as the amount of content being acquired and the content resolution increases. Combined with the additional data being generated during the production process a very large amount of digital storage capacity can be generated from the acquisition through production and finally distribution. Some of this data is kept for a very long time in a long term archive—preferably a very long time. Some of this content such as in the production process is generally only kept during an active project and for some period afterwards before being discarded. Whether the content files are kept forever or for a short time digital storage of large files can become expensive.

Cold archives are less expensive since the content (often on tapes) is kept off line in some low cost storage facility. While this is great for protection of the content it doesn’t make it readily available to be accessed and used. An active archive is much better for using content. However many facilities do not use active archives to as great a degree as they could because of the costs and complexities of on-line storage.

Storage management software can play a key role in making a more effective use of active archives. One of the ways to improve the utilization of disk based active archives is to use de-duplication technology. De-duplication, such as performed by the StorNext software, involves analysis of chunks of the data by storage software, looking to see if the content in that chunk is unique. If it is the chunk (or blocklet) is stored to disk and storage capacity is used. If the blocklet is not unique and it has been stored before it is discarded. This is not a problem in recreating the file since a record is stored by the software that explains how the desired file can be reconstructed by the blocklets.

continued page 8 sidebar >

In a sense archiving (even cold archiving) should not be a static process. When the archive load becomes too large choices will have to be made about which content to transfer and preserve on the new format. Format choices should always be made with a consideration of backwards compatibility. Otherwise archive transfers could become a constant process.

The rate of obsolescence for storage device formats varies with the devices. For hard disk drives the stability of the interface and the drive specification that controls that interface sets the backward compatibility. For SCSI drives and for fibre channel disk drives that use the SCSI specification there is a very long format stability (backward compatibility) on drive commands.

On the other hand the SCSI hard drive connection interface evolves to a faster version, roughly every 5 to 7 years. Recently both SCSI and ATA, found in the past in personal computers, made a conversion from parallel to serial connections. The resulting SAS and SATA interfaces are completely different from the interfaces used in older format drives. Thus disk drive interfaces change relatively frequently with time from an archivist point of view.

An archival system built around the use of disk drives must take this into account and either have a means to migrate to newer drives using adapters as the old drives wear out or eventually to move data from the older drive arrays to a new set. Thus the useful life of an active disk array is probably somewhere in the range of 5 to 7 years (the actual functional life with appropriate spares could be as high as 10 years). This time frame will also be influenced by service contract costs. After year 3 to 5 service costs for arrays tend to go up dramatically resulting in the desire to swap an array, even if it is functioning adequately.

For a MAID system with less active disks this time could be longer than 10 years. Often disk storage systems (as well as tape and optical) are replaced not because they no longer work but because there is technology with much more storage capacity, better performance and lower operating costs available.

The format obsolescence rate for tape (a common digital archival media) is variable depending on the format and technology. Different tape standards take different approaches.

For example, the LTO Consortium's roadmap communicates the member’s intention to provide read/write capability one generation back and read capability for two generations back. The intent is to provide that capability at each specific generation's original density and performance. Thus an LTO-4 drive which is capable recording 800GB per cartridge at 120 MB/sec must be able to read and write up to 400GB on LTO-3 cartridges at the manufacturer's originally specified Gen3 performance and must also be able to read Gen2 cartridges, again at the originally specified performance.

The LTO Consortium has a track record of introducing new generations every 20 30 months (note that DLT tape technology roadmaps have similar prior generation read requirements). Thus archival tape systems have a total format life (for reading) of about 6 years. An existing tape system could last much longer and the media is rated to last up to 20 years. Tape storage systems (especially libraries) probably have a useful life of 10-15 years.

Optical storage media may be more stable in media development than tape but generally there is a major format change every 10 years and interfaces develop like those of other computer peripherals. Optical storage media of the right quality can probably last about 10 years under the right storage conditions and optical archive libraries probably have about a 10 year useful life.

With such a process the savings in storage capacity grows with time and with the accumulation of files. With StorNext’s variable block range redundancy checking the reduction in redundancy can be as much as 10X depending upon the content. Also since the software adjusts itself to the users content it becomes a more custom solution rather than a generic system.

Such a de-duplication technology can even make active archives attractive for short term content protection such as in a production studio and certainly to create an active archive for a content owner or content distributor. Technologies such as de-duplication can be a boon to busy entertainment storage system administrators working on restrained budgets. More compact storage assets are easier to manage and migrate as newer technology becomes available and by enabling an active archive for your facilities you can make sure that your content assets are available when they are needed.

Thus an 8-10 year life of an archival storage system is likely for all these storage devices. Sometime before an older system is to be retired the content it contains must be moved to replacement storage system.

To handle this format conversion manual processes could be used, but they are error prone and time consuming, taking staff away from other projects. Storage management software should include a way to automatically control the format refresh. Management software can read archive data from an older generation tape environment, verify the content is still intact, and then automatically rewrite to the latest generation of tape.

By having this process follow rules defined by the client, their specific data retention and protection needs can easily be factored into the process. This also allows for maximum reuse of tape media and ultimately minimizes long term data retention cost. Over a 100 year archive life this process is liable to happen at least 10 times. This operation benefits from active software management to make sure such media progressions occur according to a pre-set schedule and that transfers are successful before the older media and storage systems are retired.

Creating better metadata for archived content

Creating better metadata to represent the data in the archive and creating the means to manage this metadata are keys to better search and discovery of historical content. While today much metadata is entered manually technologies are being developed to enable searching data based upon matches with audio, still or moving video content. As these technologies develop the resulting metadata could be incorporated into professional video metadata formats such as MXF to create powerful ways of accessing older content archives.

Storage devices and storage management software should be designed to make use of these automated metadata generation capabilities as they are developed. Archive management software should search through active archive content, creating new metadata based upon rich media attributes. In the absence of this, or in environments that use customized metadata formats, an archive management software should be as tightly aligned with the asset manager as possible so that artists can track and manage content as they need.

Decreasing the overall costs of archiving

The cost of archiving is a function of several factors. Some of the more important of these factors are storage media utilization and the costs of maintaining the data center where the storage assets are kept. Storage management software is a key element of reducing the overall costs of storage.

Storage management that can find unneeded duplication of data and not back up multiple copies of the same files is one method to reduce overall system digital storage capacity as well as reducing the bandwidth requirements for moving this data around.

Although generally higher in archive environments, storage capacity utilization is generally rather low. Disk storage utilization of less than 50% is not uncommon. It is possible to improve storage utilization by creating virtualization of storage assets so that storage can be provided to the user as needed rather than relying over pre-provisioning storage. By reducing the overall hardware and facilities costs (which tend to be much greater over the long run) good storage management software can more than pay for itself.

Tape Media Designed for EM Professionals

Quantum makes an interesting storage solution that includes a file system on the tape drive. The products are the Quantum A-Series Professional Video Solutions which are available on LTO and DLT tape drive platforms. This product provides video professionals with an archive solution offering up to 6.4 TB (native) of removable networked storage capacity in a tape-based file system that is MXF(1)-aware.

Such systems make the tape drive a network attached storage system (NAS) which can directly access content. A storage library based on such tape drives and tape format could also be part of an automated metadata generation and recording system such as described above. The independent file systems in the drives could generate and record enhanced metadata much faster. Also if the enhanced metadata from the individual tapes is retained in the overall storage system, finding and accessing desired content will be much faster.

The cost of maintaining storage facilities greatly exceeds the initial hardware costs. Heating and air conditioning are essential to successful operation of an active storage facility. Reducing the heat load from the storage systems will do a lot to reduce overall facility costs. The use of smaller disk drives in drive arrays (particularly if these are lower RPM SATA drives) will reduce the heat generated by these drives and since the drives can be packed tighter in an array than larger form factor disk drives the overall storage array footprint can be smaller for an equivalent storage capacity. This reduces the size of the data center space required for the active archive.

For larger archives tape is an important format because it does not require a large amount of power. Media is offline except when in use, in a tape drive. The frame itself requires no power except for the robotics and drives, which when not in use are idle. Tape libraries can scale to multiple petabytes of storage without requiring significant power consumption in comparison to a disk array of the same capacity.

Projections for Various Storage Media Used in EM Archiving

Table 1 gives projections for annual demand of new storage media (tape, optical or hard disk drive) for entertainment and professional media archiving of newly generated content. Digital conversion of analog content and its preservation can increase these numbers considerably but prediction of this trend is hard to do. Suffice it to say that the overall demand could easily be twice that shown here.

As can be seen in this table tape remains the most used archival storage media. The anticipated growth in optical media depends upon the development of advanced high capacity optical technology such as holographic media.

Table 2. Projections for the Use of Various Archival Storage Media in the Media and Entertainment Industry

A constant problem with a cold digital archive is staying ahead of the obsolescence of the storage media used. As time goes on digital storage technology improves and the storage media and drives change. If the data storage on a cold archived media is not migrated to new storage technology when it becomes available there is a danger that the data won’t be readable as time goes by. Thus successful preservation of archived digital content must also include data migration management to deal ahead of time with the risks of format obsolescence.

Broader Asset Management Systems

A final topic of discussion is how archiving fits into a larger asset management system. An asset management system is a set of processes designed to control digital assets used in active and inactive projects. Processes should cover every aspect of the content lifecycle from tracking metadata, physical location from active usage to archive, and handling data protection.

While there is not one catch-all product that covers asset management some broad strokes can be made about the design of a system. A digital media archive management system should do the following:

· Protect the digital assets for many years through active management and checking of the hardware and storage environment.
· Perform systematic format updates and transfers of the digital content as needed to avoid potential access loss due to format obsolescence.
· Alert the administrator of problems with the content, environment or storage hardware early enough for the administrator to take action and if necessary shut down and protect assets in the event of a crisis.
· Provide organized access to the digital archives by retaining an appropriate database of content metadata and perhaps indexing of the actual content on those assets.
· If an active archive, control the access and delivery of digital content when needed and in a timely manner.
· If a cold archive, initiate delivery and mounting of the media from the cold archive and delivery of the content where it is needed.
· Manage disaster recovery requirements such as mirroring or backup of managed data to remote data centers to make sure content can be recovered if the original copy is somehow damaged.

New developments in long term archive management include:

· Active indexing of still and moving digital images during the archiving process—more advanced metadata creation and management allowing easier search and use of this content.
· More active and continuous checking of archived content to make sure the data integrity is OK and to detect growing problems.

File-based access to digital content from the storage media to enhance content access.

Conclusions

This report has explored the important trends and drivers for the archiving of entertainment and professional media content. Management software as well as hardware play an important role in digital media archives. We have looked at the requirements as well as future expected developments in storage management. We have also looked at some of the providers of digital media management software.

The decreasing cost of digital storage and the capture and editing of content enabled by digital technology has increased the number of facilities and producers of content.

Digital technology and increased available communication bandwidth have also allowed the development of new ways to distribute content such as the internet and mobile phone networks. This has also increased the supply of new professional (as well as non-professional) content. Higher resolution is being required in professional content and the amount of digital footage being acquired for the final produced product has increased considerably. In addition, older analog content is being digitized in greater amounts with time. All of these factors have caused digital storage capacity requirements to swell.

As total content capacity increases so does the amount of content that is being archived in either cold or active archives (or both). Long term preservation of large digital archives will lead the industry to solve new issues associated with format conversion, metadata creation and management as well as methods to reduce the total cost of operating a digital media archive. Preservation of digital data from the possible destruction of the primary storage system can be dealt with by having off-site copies of the content or by maintaining remote mirrors or backup of the content.

The material presented in this report was largely extracted from the 2007 Entertainment Content Creation and Digital Storage Report or from the data set created in putting together that report. For more information or to order a copy of this report please go to http://www.tomcoughlin.com/techpapers.htm.

Low-cost systems let almost anyone become a TV producer

By ANDREW D. SMITH / The Dallas Morning News
asmith@dallasnews.com

Video companies say they're already redefining live entertainment with systems that let anyone with a few thousand bucks produce live television shows and stream them over the Internet.

High schools can broadcast road games to fans back home. Rock bands broadcast concerts around the globe. Churches broadcast sermons for vacationing members.

But only time will tell how many new types of entertainment Americans will make and view, thanks to products from both tech giants and lesser-known names like Plano's ViewCast Corp. and San Antonio's NewTek Inc.

"Live television broadcasts have always required million-dollar equipment and armies of expert operators," said Philip Nelson, NewTek's senior vice president for strategic development.

"With new technology cutting those costs more than a hundredfold, we'll see an incredible flood of new programming. The lower cost of failure will free people to gamble with new concepts. I don't think we can even imagine some of the stuff we'll be watching 10 years from now."

Mr. Nelson's grand predictions belie the humble appearance of modern production equipment.

Many products, like Adobe's Visual Communicator 3, are nothing but software for regular computers. ViewCast's best-selling conversion cards look like electronic scraps. Even NewTek's TriCaster, which the company bills as an entire production studio in a box, looks more like the back of a small television than the leading edge of a revolution.

The anonymous sockets that line a TriCaster accept audio and video from cameras, microphones and computers. Users hook the TriCaster to a keyboard and monitor, which shows a virtual production studio.

A big box on the monitor displays the outgoing video feed. Several smaller boxes display the various incoming video feeds, along with any audio or video clips that have been stored.

Producers control the device by clicking on whatever feed they wish to air. They can even put graphics over the images – such as names and titles for speakers – or insert people into virtual sets.

Assembling a show on the fly feels much like playing a video game, which is exactly what NewTek wants.

"Much like Windows made computer operation visually intuitive, the software in these systems makes video production visually intuitive, so you don't have to be an expert to do it," said Tim Siglin, a contributing editor for Streaming Media.

"These systems don't just allow one person to produce a show," Mr. Siglin said. "They force one person to do the job because there's only one workstation. It's very intense to be one person scrambling to do the work that once occupied several folks."

Still, experience shows that the technology really does enable lone operators to make broadcast-quality video on the fly.

MTV used TriCasters to produce segments of The Hills. Fox Sports used TriCasters to produce its online pre-game and post-game reports from this year's World Series. The Academy of Television Arts and Sciences used them to produce its broadcasts of the Daytime Emmys.

Comedian Tom Green, who rose to fame with a series on MTV, looked like a has-been after a bout with cancer and the failure of his movie, Freddie Got Fingered.

No one would support a comeback by risking the money needed for a traditional television show, so Mr. Green began broadcasting a talk show from his house and streaming it over the Internet.

Such low costs free Mr. Green to follow his passions rather than formulas designed to maximize viewers and ad revenue. If he's having fun talking to a guest, he keeps talking – sometimes for hours.

Anyone with a camera, a computer and an Internet connection can post videos to YouTube or Google Video without spending a dime. Anyone with a webcam and a few bucks to spend on Web hosting can send live programming over the air.

But YouTube videos lack the excitement of live television, and single-camera webcasts bear little resemblance to professional television.

To create something like broadcast-quality television, users need a good video camera, microphones, a computer and some editing software. The total cost of such a setup falls well short of $1,000, assuming that people use their existing computers.

Folks who want to go a step further and do live broadcasts probably need to add $5,000 to $10,000 for production equipment and another few bucks for Web hosting, which comes cheap from companies like Akamai Technologies.

Such low costs have led to an explosion of sports programming online. Small colleges that never appear on ESPN or the networks have begun broadcasting their football and basketball games on Web sites. Schools of all sizes have begun broadcasting minor sports over their Web sites.

Archiving in the Entertainment and Professional Media MarketThomas M. Coughlin Coughlin Associates, Inc. August 2007CONTENTS