Calm before the Storm: Helping a Federal Agency Plan for a Coming Wave of Big Data Needs

For the first time ever, we know the size of the problem we are facing and how fast the train is coming. Thank you for opening our eyes!

Situation

A federal regulatory agency was anticipating an enormous growth in the volume of data to be ingested over the next three to five years, driven by growth of the regulated industry, changes in technology, and expansion of the agency’s regulatory mandate. However, there was virtually no enterprise-wide visibility into current data ingestion and historical growth upon which to base projections and preparations. The agency was flying blind, and the IT organization lacked the information necessary to effectively support the needs of its business users, who were already beginning to experience the pain of rapidly increasing data volumes (e.g., slow processing, downtime). Ultimately, this was affecting the agency’s ability to effectively deliver on their mission.

Approach

Censeo conducted a 10-week diagnostic to size the agency’s current data footprint, estimate anticipated data growth, pinpoint capacity constraints, and identify management implications and strategic options. To assemble this baseline, the Censeo team developed collaborative relationships with over 30 business and IT stakeholders across the agency, and overcame an overwhelming culture of organizational silos that had hindered past attempts at enterprise-wide coordination and visibility.

Through these collaborative relationships, Censeo identified and sized more than 200 data sources being ingested or generated by the agency, amounting to over 3PB of data. Censeo then developed analytically rigorous projections of data volume growth, which revealed that the agency should expect growth of 30-40 percent annually over the next three to five years. In addition, Censeo found that the agency had been significantly underestimating the growth of their data consumption needs.

The Censeo team also worked with the IT organization to understand the agency’s capacity to handle the anticipated growth in data. The team identified multiple hard constraints (e.g., storage capacity, end-of-support for mission-critical enterprise storage environments) that would be triggered within two years if management did not act—and act quickly.