Background
Our client is largest independent processor of flat-rolled steel and manufacturer of industrial cylinders and pressure vessels in the United States.
Heavily customized BI stack (Informatica mappings, Data Warehouse, RPD Models, and Dashboards) hosted on Oracle Cloud. Cloudera Data lake on AWS to capture detailed operational data from machines, sensors, and shop floor applications to help improve production efficiency and quality.
However, both the Oracle DW and Cloudera Data lake were unstable and unable to integrate with BI tools, hence there was a strong need to modernize the cloud data and analytics platform.
Current Challenges
The client faced several challenges with their existing data infrastructure:
- Stability of the Cloudera data lake – many outages and incidents
- Data redundancy across data warehouse, data lake, and tableau extracts
- Tableau connectivity issue to CDP Impala
- Tableau extracts required to hit query performance targets
- Unable to reliably hit data latency SLA’s
- Complexity of environment requires additional resources
- Lacking sufficient visibility over data pipeline health and data quality
Solution
To address these challenges, we proposed the following solution:
- Recommend and prove out the target state data and analytics architecture
- Evaluate and shortlist technology and tools (Matillion, Talend, DBT, Streamsets, Airflow, Astronomer)
- Pilot two uses on the recommended architecture on Data Warehouse (ETL process) and Data Lake (streaming data) to meet success criteria including acceptable performance
- Help setup Infrastructure of (AWS, Snowflake, Streamsets, environment
- Extend the pilot implementation to migrate DW tables (54 Dim and 60 Fact) and Datalake artifacts (274 base tables and 73 views) for about 800 users
- Setup and automate CI / CD pipeline with Github actions and branches for Streamsets + Snowflake with scheduling and monitoring tools
The Results
- 30% performance improvement by converting Oracle Cloud and Cloudera to AWS-Snowflake stack.
- 20% SLA improvement by modernizing Oracle and Cloudera workloads on AWS-Snowflake.
- 50% cost and time savings when transforming Informatica workflows and Oracle EDW to AWS.
- Reduction in operational expenses, over $350k annual savings from consolidating Data Warehouse and Data Lakes.
Migrating data from on-premises servers to the AWS cloud significantly improved reporting capabilities. The unified and scalable platform facilitated self-service BI, reducing manual reporting efforts, enhancing data availability, and enabling data-driven decision-making.
After looking into a handful of the articles on your web site, I seriously like your technique of blogging.
I saved it to my bookmark site list and will be checking back in the near future.
Please check out my website as well and let me know how you feel.
Thank you for your feedback and for bookmarking our site! We’re glad you enjoy our blogs. We’ll definitely check out your website. If you need any support with AI, Data, or Cloud services, feel free to reach out.
Keep exploring our insights (Blogs, Success Stories, Use Cases, and News) here: https://narwal.ai/insights/
contact us: https://narwal.ai/contact-us/
What’s Happening i’m new to this, I stumbled upon this I’ve discovered It absolutely helpful and it
has aided me out loads. I hope to contribute &
aid different customers like its helped me. Good job.
Thank you! We’re glad you found our content helpful. At Narwal, we specialize in AI, Data, Cloud, and Quality Engineering services. If you’re looking to contribute and help others, we’d love to connect with you. Explore our services and see how we can work together to drive innovation and success.
Check our services here: https://narwal.ai/services/
contact us: https://narwal.ai/contact-us/
Greetings from Ohio! I’m bored at work so I decided to browse your
site on my iphone during lunch break. I enjoy the knowledge you provide here and can’t wait to take a look
when I get home. I’m shocked at how quick your blog loaded on my mobile ..
I’m not even using WIFI, just 3G .. Anyways, superb site!
You actually make it seem so easy with your presentation but I find this matter to be actually something that I think I would never understand. It seems too complex and extremely broad for me. I’m looking forward for your next post, I’ll try to get the hang of it!
Hmm it seems like your blog ate my first comment (it was super long) so I guess I’ll just sum it up what I submitted and say, I’m thoroughly enjoying your blog. I too am an aspiring blog blogger but I’m still new to the whole thing. Do you have any tips for newbie blog writers? I’d certainly appreciate it.
My brother suggested I might like this blog. He was entirely right. This publish actually made my day. You cann’t consider just how a lot time I had spent for this info! Thank you!