For years, the University of Arizona relied on Extract, Transform, and Load (ETL) processes and a suite of legacy data tools to meet its growing data needs as well as the ever-increasing data storage needs, it was clear the University of Arizona was ready for a shift. With a vision for modernization, the University Analytics and Institutional Research (UAIR) team, encouraged by Chief Information Officer Barry Brummund, evolved their approach to move from a traditional data warehouse to a modern data lakehouse environment. The university is uniquely positioned to lead in this endeavor, ready to push the boundaries of data ingestion and integration.

Under the leadership of Ravneet Chadha, Associate Vice President and Chief Data Officer, UAIR began exploring scalable, cutting-edge techniques to transform data from a basic utility into a powerful driver of institutional growth.

A DECADE OF EVOLUTION

The university’s data modernization journey began in 2010 with the implementation of PeopleSoft HR and Student systems, as well as Kuali Financial and Research systems. In 2018, the university took a major step forward, migrating to Amazon Web Services (AWS) under a “Bring Your Own License” model. By 2022, the Data Warehouse Modernization Initiative was launched to explore advanced ETL and ELT tools through proof-of-concept projects, shifting from long-term licensed technologies to flexible solutions like Software as a Service (SaaS), Reporting as a Service (RaaS), and application programming APIs).

In just the past year, the university has successfully migrated third-party systems such as Academic Insights and Trellis Salesforce to AWS, reaffirming its commitment to modernizing data management and addressing the future challenges of higher education.

Image

ETL (Extract, Transform, Load) is a traditional data integration method that transfers data from a source to a data warehouse, involving separate extraction and transformation processes. In contrast, ELT (Extract, Load, Transform) loads raw data directly into the data warehouse, enabling transformations within the warehouse and offering greater flexibility for analyzing both structured and unstructured data. Choosing between ETL and ELT significantly impacts data storage, analysis, and processing strategies.
 

UAIR’S DATA ENGINEERING TEAM: LEADING THE CHARGE

Under the leadership of Shiva Chidara, who brings over 18 years of ETL experience, UAIR’s Data Engineering Team has effectively transferred data from UAccess Systems and other sources into the Enterprise Data Warehouse (EDW) since the initiation of the Data Warehouse Modernization project in 2022. Through nightly batch loads, the Data Engineering Team guarantees that the data is fresh and easily accessible each morning via UAccess Analytics, highlighting a strong commitment to data availability, efficiency, and quality.

In recent months, the Data Engineering Team has led the Data Warehouse Modernization initiative through both internal and external collaborations. They prioritized identifying an infrastructure that satisfied the university’s data storage and performance needs, ultimately selecting AWS for its support of both ETL and ELT processes. With this robust tool and ongoing support from AWS professionals, the team has committed the past 12 months to improving data warehousing processes, paving the way for a more efficient, insightful, and data-informed future for the campus community.

“The primary reason for transitioning from ETL to ELT is to leverage more efficient tools that accelerate data processing, shorten load times and enable quicker insights, all of which make analytics more accessible to campus users”

— Shiva Chidara, Data Engineering Team Lead

During the data warehouse modernization implementation, the Data Engineering Team transformed the organization’s approach to data management by conducting comprehensive training sessions on AWS. Through hands-on learning and collaboration with AWS experts and UAIR’s Systems team to provide guidance on best practices, the team acquired the skills necessary to successfully integrate diverse data sources into the new system. With a focus on innovation, the team designed numerous AWS Glue jobs, enabling the ingestion of new data while consolidating previously overlooked reports. This effort was not merely technical; it aimed to revitalize the university’s entire data landscape. By leveraging best practices for ELT workflows, the team developed a streamlined process that facilitated efficient operations.

Developing critical process resources has ensured the UAIR team adheres to best practices in database view development, data validation, and migration throughout this modernization, fostering a unified approach to making data more accessible and actionable at the U of A.

SYSTEMS SUPPORT & OAS UPGRADE

The Data Engineering Team did not navigate this complex project alone; they received crucial support from UAIR’s Systems and Security Team, led by Business Intelligence Architect Doug Hester. Hester and his team of middleware analysts played a vital role in ensuring seamless systems integration, enhanced workflows, and improved customer support. Their expertise was instrumental in executing tool upgrades and system enhancements, transforming potential challenges into opportunities for growth. The Systems Team not only supported the Data Engineering Team but also empowered the university to embrace a future where data management is not only efficient but strategic.

THE INCEPTION OF VISUALIZER

In early 2024, as the data warehouse modernization project gained momentum, UAccess Analytics underwent a significant upgrade managed by Hester and UAIR’s Systems Team. This upgrade was particularly noteworthy because UAccess Analytics is powered by Oracle Analytics, which continuously aims to improve user experience. Their latest upgrade, OAS 2023, introduced an innovative tool called Visualizer, designed to transform how users engage with institutional data.

“Visualizer enables customers to not only work with the datasets and data models we’ve provided and pre-created, but they can also incorporate their own datasets.”

– Doug Hester, BI Architect & Systems Team Lead

Visualizer allows provisioned users to effortlessly integrate their own datasets with the institutional data available in UAccess Analytics, creating new opportunities for analysis and insight. As users explore this powerful data tool, they will discover two key features: the ability to upload and incorporate personal data and an expanded range of visualization options. With Visualizer, users can generate compelling visuals using graphs, charts, and maps, layering custom datasets with the robust institutional data from UAccess Analytics. This enhances data exploration, making it more intuitive and effective for producing meaningful insights.

COLLABORATION & SYSTEMS SUPPORT

This transformation was supported by UAIR’s Systems and Security Team, led by Business Intelligence Architect Doug Hester. Hester’s team ensured smooth systems integration, tool upgrades, and workflow enhancements. Their work not only addressed technical challenges but also paved the way for strategic data management, positioning the university for future success.

THE ERA OF SELF SERVICE

The data warehouse modernization initiative has significantly enhanced self-service capabilities, positioning the institution as a leader in empowering users to create their own reports—an uncommon advantage in higher education. By transitioning away from IBM DataStage’s multi-year licensing contracts, the university not only improved data integration and analytics but also significantly minimized storage expenses while gaining a more scalable and flexible infrastructure.

Through this transformation, the University of Arizona is now equipped with a modern data management framework that enhances processing capabilities, supports strategic decision-making, and fosters a culture of innovation. With leaders like Brummund and Chadha at the helm, the university is prepared to tackle the challenges of tomorrow with agility and insight.