View Details

We are excited to announce the acquisition of Octopai, a leading data lineage and catalog platform that provides data discovery and governance for enterprises to enhance their data-driven decision making. Cloudera’s mission since its inception has been to empower organizations to transform all their data to deliver trusted, valuable, and predictive insights. With AI and generative AI powering the next wave of business applications, the real competitive edge lies in collecting vast amounts of data and deeply understanding and leveraging it for business value. Cloudera, together with Octopai, will make it easier for organizations to better understand, access, and leverage all their data in their entire data estate – including data outside of Cloudera – to power the most robust data, analytics and AI applications. This acquisition delivers access to trusted data so organizations can build reliable AI models and applications by combining data from anywhere in their environment.

Propel AI and analytic success with better data discovery and data catalogingAs organizations collect a vast and diverse array of data sources, they face significant challenges in achieving a comprehensive understanding of their data. This includes having full visibility into the origin of the data, the transformations it underwent, its relationships, and the context that was added or stripped away from that data as it moved throughout the enterprise. In today’s heterogeneous data ecosystems, integrating and analyzing data from multiple sources presents several obstacles: data often exists in various formats, with inconsistencies in definitions, structures, and quality standards. Additionally, multiple copies of the same data locked in proprietary systems contribute to version control issues, redundancies, staleness, and management headaches. This dampens confidence in the data and hampers access, in turn impacting the speed to launch new AI and analytic projects.

Founded in 2016, Octopai offers automated solutions for data lineage, data discovery, data catalog, mapping, and impact analysis across complex data environments. Combining Octopai capabilities with Cloudera’s AI powered hybrid data platform provides deeper data understanding, enhanced security, and robust data governance – essential for driving AI and analytics success. The combined platform will integrate data – from wherever it originates and wherever it is stored (cloud or on prem) – to deliver real-time insights required for faster decision making and predictive generative AI applications for personalized customer experiences. By adding the Octopai platform, Cloudera customers will benefit from:

Enhanced Data Discovery: Octopai’s automated data discovery enables instantaneous search and location of desired data across multiple systems. It leverages knowledge graphs to keep track of all the data sources and data flows, using AI to fill the gaps so you have the most comprehensive metadata management solution.

Data Trust and Quality: Octopai’s multi-layered data lineage solution provides the most complete, in-depth, and trustworthy automated lineage so data users can always trust the data and the insights generated from it. The end-to-end lineage also automates tasks such as predicting the impact of a process change, analyzing the impact of a broken process, discovering parallel processes performing the same tasks, and performing root cause analysis to uncover the source of reporting errors. This guarantees data quality and automates the laborious, manual processes required to maintain data reliability.

Robust Data Catalog: Organizations can create company-wide consistency with a self-creating, self-updating data catalog. This automated data catalog always provides up-to-date inventory of assets that never get stale. Octopai’s 50+ connectors make it easy to capture the metadata from different data sources and maintain the catalog automatically so users always know what data is available, where it can be found, what it represents, and who is responsible for it.

AI Co-pilot: The co-pilot empowers data teams with a real-time, unified workspace that automates, optimizes, and interprets scripts while providing immediate insights into data lineage. It allows users to mitigate risks, increase efficiency, and make data strategy more actionable than ever before.

The path forward with data governance and metadata managementWith this acquisition, Cloudera bolsters its rich metadata management with Octopai’s market leading data discovery and data lineage capabilities, enabling customers to understand and trust their data across not just the Cloudera platform but the entire enterprise ecosystem. Together, Cloudera and Octopai will help reinvent how customers manage their metadata and track lineage across all their data sources. This will also accelerate deployment of new data products for AI, gen AI, and analytics applications. It will increase the discovery of the data products and ensure the usability and consistent delivery of these data products, providing essential elements of a data mesh architecture for self-service decentralized access to data.

The post Octopai Acquisition Enhances Metadata Management to Trust Data Across Entire Data Estate appeared first on Cloudera Blog.

View Details

Large Language Models (LLMs) will be at the core of many groundbreaking AI solutions for enterprise organizations. Here are just a few examples of the benefits of using LLMs in the enterprise for both internal and external use cases:

Optimize Costs. LLMs deployed as customer-facing chatbots can respond to frequently asked questions and simple queries. These enable customer service representatives to focus their time and attention on more high-value interactions, leading to a more cost-efficient service model.

Save Time. LLMs deployed as internal enterprise-specific agents can help employees find internal documentation, data, and other company information to help organizations easily extract and summarize important internal content.

Increase Productivity. LLMs deployed as code assistants accelerate developer efficiency within an organization, ensuring that code meets standards and coding best practices.

Several LLMs are publicly available through APIs from OpenAI, Anthropic, AWS, and others, which give developers instant access to industry-leading models that are capable of performing most generalized tasks. However, these LLM endpoints often can’t be used by enterprises for several reasons:

• Private Data Sources: Enterprises often need an LLM that knows where and how to access internal company data, and users often can’t share this data with an open LLM.
• Company-specific Formatting: LLMs are sometimes required to provide a very nuanced formatted response specific to an enterprise’s needs, or meet an organization’s coding standards.
• Hosting Costs: Even if an organization wants to host one of these large generic models in their own data centers, they are often limited to the compute resources available for hosting these models.

The Need for Fine TuningFine tuning solves these issues. Fine tuning involves another round of training for a specific model to help guide the output of LLMs to meet specific standards of an organization. Given some example data, LLMs can quickly learn new content that wasn’t available during the initial training of the base model. The benefits of using fine-tuned models in an organization are numerous:

• Meet Coding Formats and Standards: Fine tuning an LLM ensures the model generates specific coding formats and standards, or provides specific actions that can be taken from customer input to an agent chatbot.
• Reduce Training Time: AI practitioners can train “adapters” for base models, which only train a specific subset of parameters within the LLM. These adapters can be swapped freely between one another on the same model, so a single model can perform different roles based on the adapters.
• Achieve Cost Benefits: Smaller models that are fine-tuned for a specific task or use case perform just as well as or better than a “generalized” larger LLM that is an order of magnitude more expensive to operate.

Although the benefits of fine tuning are substantial, the process of preparing, training, evaluating, and deploying fine-tuned LLMs is a lengthy LLMOps workflow that organizations handle differently. This leads to compatibility issues with no consistency in data and model organization.

Introducing Cloudera’s Fine Tuning Studio
To help remedy these issues, Cloudera introduces Fine Tuning Studio, a one-stop-shop studio application that covers the entire workflow and lifecycle of fine tuning, evaluating, and deploying fine-tuned LLMs in Cloudera’s AI Workbench. Now, developers, data scientists, solution engineers, and all AI practitioners working within Cloudera’s AI ecosystem can easily organize data, models, training jobs, and evaluations related to fine tuning LLMs.

Fine Tuning Studio Key Capabilities Once the Fine Tuning Studio is deployed to any enterprise’s Cloudera’s AI Workbench, users gain instant access to powerful tools within Fine Tuning Studio to help organize data, test prompts, train adapters for LLMs, and evaluate the performance of these fine-tuning jobs:

• Track all your resources for fine tuning and evaluating LLMs. Fine Tuning Studio enables users to track the location of all datasets, models, and model adapters for training and evaluation. Datasets that are imported from both Hugging Face and from a Cloudera AI project directly (such as a custom CSV), as well as models imported from multiple sources such as Hugging Face and Cloudera’s Model Registry, are all synergistically organized and can be used throughout the tool – completely agnostic of their type or location.
• Build and test training and inference prompts. Fine Tuning Studio ships with powerful prompt templating features, so users can build and test the performance of different prompts to feed into different models and model adapters during training. Users can compare the performance of different prompts on different models.
• Train new adapters for an LLM. Fine Tuning Studio makes training new adapters for an LLM a breeze. Users can configure training jobs right within the UI, either leave training jobs with their sensible defaults or fully configure a training job down to custom parameters that can be sent to the training job itself. The training jobs use Cloudera’s Workbench compute resources, and users can track the performance of a training job within the UI. Furthermore, Fine Tuning Studio comes with deep MLFlow experiments integration, so every metric related to a fine tuning job can be viewed in Cloudera AI’s Experiments view.
• Evaluate the performance of trained LLMs. Fine Tuning Studio ships with several ways to test the performance of a trained model and compare the performance of models between one another, all within the UI. Fine Tuning Studio provides ways to quickly test the performance of a trained adapter with simple spot-checking, and also provides full MLFlow-based evaluations comparing the performance of different models to one another using industry-standard metrics. The evaluation tools built into the Fine Tuning Studio allow AI professionals to ensure the safety and performance of a model before it ever reaches production.
• Deploy trained LLMs to production environments. Fine Tuning Studio ships natively with deep integrations with Cloudera’s AI suite of tools to deploy, host, and monitor LLMs. Users can immediately export a fine-tuned model as a Cloudera Machine Learning Model endpoint, which can then be used in production-ready workflows. Users can also export fine tuned models into Cloudera’s new Model Registry, which can later be used to deploy to Cloudera AI’s new AI Inferencing service running within a Workspace.
• No-code, low-code, and all-code solutions. Fine Tuning Studio ships with a convenient Python client that makes calls to the Fine Tuning Studio’s core server. This means that data scientists can build and develop their own training scripts while still using Fine Tuning Studio’s compute and organizational capabilities. Anyone with any skill level can leverage the power of Fine Tuning Studio with or without code.

An End-to-End Example: Ticketing Support AgentTo show how easy it is for GenAI builders to build and deploy a production-ready application, let’s take a look at an end-to-end example: fine tuning an event ticketing customer support agent. The goal is to fine tune a small, cost-effective model that , based on customer input, can extract an appropriate “action” (think API call) that the downstream system should take for the customer. Given the cost constraints of hosting and infrastructure, the goal is to fine tune a model that is small enough to host on a consumer GPU and can provide the same accuracy as a larger model.
Data Preparation. For this example, we will use the bitext/Bitext-events-ticketing-llm-chatbot-training-dataset dataset available on HuggingFace, which contains pairs of customer input and desired intent/action output for a variety of customer inputs. We can import this dataset on the Import Datasets page.

Model Selection. To keep our inference footprint small, we will use the bigscience/bloom-1b1 model as our base model, which is also available on HuggingFace. We can import this model directly from the Import Base Models page. The goal is to train an adapter for this base model that gives it better predictive capabilities for our specific dataset.

Creating a Training Prompt. Next, we’ll create a prompt for both training and inference. We can utilize this prompt to give the model more context on possible selections. Let’s name our prompt better-ticketing and use our bitext dataset as the base dataset for the prompt. The Create Prompts page enables us to create a prompt “template” based on the features available in the dataset. We can then test the prompt against the dataset to make sure everything is working properly. Once everything looks good, we hit Create Prompt, which activates our prompt usage throughout the tool. Here’s our prompt template, which uses the instruction and intent fields from our dataset:

Train a New Adapter. With a dataset, model, and prompt selected, let’s train a new adapter for our bloom-1b1 model, which can more accurately handle customer requests. On the Train a New Adapter page, we can fill out all relevant fields, including the name of our new adapter, dataset to train on, and training prompt to use. For this example, we had two L40S GPUs available for training, so we chose the Multi Node training type. We trained on 2 epochs of the dataset and trained on 90% of the dataset, leaving 10% available for evaluation and testing.

Monitor the Training Job. On the Monitor Training Jobs page we can track the status of our training job, and also follow the deep link to the Cloudera Machine Learning Job directly to view log outputs. Two L40S GPUs and 2 epochs of our bitext dataset completed training in only 10 minutes.

Check Adapter Performance. Once the training job completes, it’s helpful to “spot check” the performance of the adapter to make sure that it was trained successfully. Fine Tuning Studio offers a Local Adapter Comparison page to quickly compare the performance of a prompt between a base model and a trained adapter. Let’s try a simple customer input, pulled directly from the bitext dataset: “i have to get a refund i need assistance”, where the corresponding desired output action is get_refund. Looking at the output of the base model compared to the trained adapter, it’s clear that training had a positive impact on our adapter!

Evaluate the Adapter. Now that we’ve performed a spot check to make sure training completed successfully, let’s take a deeper look into the performance of the adapter. We can evaluate the performance against the “test” portion of the dataset from the Run MLFlow Evaluation page. This provides a more in-depth evaluation of any selected models and adapters. For this example, we will compare the performance of 1) just the bigscience/bloom-1b1 base model, 2) the same base model with our newly trained better-ticketing adapter activated, and finally 3) a larger mistral-7b-instruct model.

As we can see, our rougueL metric (similar to an exact match but more complex) of the 1B model adapter is significantly higher than the same metric for an untrained 7B model. As simple as that, we trained an adapter for a small, cost-effective model that outperforms a significantly larger model. Even though the larger 7B model may perform better on generalized tasks, the non-fine-tuned 7B model has not been trained on the available “actions” that the model can take given a specific customer input, and therefore would not perform as well as our fine-tuned 1B model in a production environment.

Accelerating Fine Tuned LLMs to ProductionAs we saw, Fine Tuning Studio enables anyone of any skill level to train a model for any enterprise-specific use case. Now, customers can incorporate cost-effective, high-performance, fine-tuned LLMs into their production-ready AI workflows more easily than ever, and expose models to customers while ensuring safety and compliance. After training a model, users can use the Export Model feature to export trained adapters as a Cloudera Machine Learning model endpoint, which is a production-ready model hosting service available to Cloudera AI (formerly known as Cloudera Machine Learning) customers. Fine Tuning Studio ships with a powerful example application showing how easy it is to incorporate a model that was trained within Fine Tuning Studio into a full-fledged production AI application.

How can I Get Started with Fine Tuning Studio?Cloudera’s Fine Tuning Studio is available to Cloudera AI customers as an Accelerator for Machine Learning Projects (AMP), right from Cloudera’s AMP catalog. Install and try Fine Tuning Studio following the instructions for deploying this AMP right from the workspace.

Want to see what’s under the hood? For advanced users, contributors, or other users who want to view or modify Fine Tuning Studio, the project is hosted on Cloudera’s github here: https://github.com/cloudera/CML_AMP_LLM_Fine_Tuning_Studio.

Get Started Today!Cloudera is excited to be working on the forefront of training, evaluating, and deploying LLMs to customers in production-ready environments. Fine Tuning Studio is under continuous development and the team is eager to continue providing customers with a streamlined approach to fine tune any model, on any data, for any enterprise application. Get started today on your fine tuning needs, and Cloudera AI’s team is ready to assist in fulfilling your enterprise’s vision for AI-ready applications to become a reality.

The post Introducing Cloudera Fine Tuning Studio for Training, Evaluating, and Deploying LLMs with Cloudera AI appeared first on Cloudera Blog.

View Details

Cloudera’s partner ecosystem delivers best-of-breed technology solutions to joint customers from the biggest names in the industry and is a core pillar of the company’s growth strategy.

Cloudera is committed to fostering collaboration with partners, growing relationships, and innovating for the future. To elevate Cloudera’s partner ecosystem, the company recently announced the promotion of Michelle Hoover to Senior Vice President of Global Alliances & Channels.

Michelle brings over 20 years of experience to her role, having managed software alliances and partners at many key industry organizations, including two years at Cloudera. Her extensive background includes management and executive positions at Confluent, Red Hat, and Oracle. Most recently, she was VP of Cloud & AI Ecosystem Partners at Cloudera. Michelle’s deep partnership expertise and strong relationships within the data and AI ecosystem make her a great leader of the Cloudera alliances and partner channels strategies.

Let’s get to know Cloudera’s newest SVP, Michelle Hoover.

Michelle, congratulations on your new position. What excites you most about your new role?

I’m truly excited to step into this position and work toward improving and growing our partner ecosystem because of the position Cloudera and our partners are in today and the opportunities that lie ahead.

Cloudera is the only true hybrid data platform that enables enterprises to analyze, control, and modernize data, analytics, and AI. Organizations recognize the value of hybrid data platforms that run anywhere and create true value for our customers.

But we don’t do this alone. We work closely with our partners to ensure that the treasure trove of data our customers hold can be easily managed into actionable insights no matter where the data resides. This positions the partner ecosystem well for the future by creating an incredible opportunity to generate services and grow with us.

What is your vision for the Cloudera partner ecosystem as you take on this role?

One of my goals is to build a robust system integrator partner business. Regional and global system integrators are critical to making decisions as we work with large enterprise customers who rely on them.

We must double down on our focus on that community of partners, along with independent software vendors, who are more important than ever to our customers with the advent of AI and new applications. It’s my objective to make sure those two communities of partners are successful with Cloudera. We can only be as successful as our partner ecosystem, so it is fundamental to our growth strategy to help partners excel with customers.

Our diverse partner set delivers the best portfolio of solutions to make it easy for customers to run their businesses in the cloud – whether that’s with AWS, Microsoft, or Google – or in a hybrid environment with Dell, IBM, and others. We want to help our customers leverage our partner ecosystem to make it easier to solve their problems and address their specific needs.

The key to partnerships is making sure it’s a win-win situation. How can our partners be successful and build a robust business around our customers? It’s important to explore new channels and routes to do that.

What makes Cloudera such a unique place to work?

The people are what really sets Cloudera apart. Everyone at Cloudera is passionate and engaged in collaboration across lines of business to deliver the most innovative world-class solutions for our customers in any environment.

That passion is something I have seen extend across the culture at Cloudera as well. Take, for example, our Cloudera Cares program. Employees in this program volunteer to help our surrounding communities through acts of service and give back to those in need. These initiatives embody the collective spirit of Cloudera employees worldwide. It’s a pillar of our company and builds a strong and rewarding culture that I’m proud to be a part of.

What advice would you give to partners when it comes to rapidly growing technologies such as AI?

My advice is to align with partners you can trust and know that it’s more important than ever to be able to trust your data.

Cloudera is uniquely positioned in that way. It offers technologies to help organizations trust their data, and it will continue to lead in strategic partnerships. We offer products and solutions to help bring AI models to your data versus bringing the data to the AI model.

When considering a partner vendor, remember that your customers innovate and take advantage of newer technologies, but you also need to know that the solutions you build and the vendors you partner with will be around for a long time.

Trust your data and trust the vendors you partner with to manage that data.

What advice would you give your younger self or young professionals today?

Early in my career, I was told that everyone you meet can help you in some way. There’s value in everybody.

It’s something that I believe is particularly relevant in my work in the partner space today. Just as every individual you meet or work with can help you, in the partner ecosystem, every partner has the potential to help you support your customers. Cloudera partners are important – both current partners and new partners that may emerge.

How do you like to spend your free time?

I love the outdoors, nature, competitive sports, like tennis, and spending time with my husband and two young adult kids. I’ve also been trying to take more opportunities to explore new areas of interest such as gardening and travel. My travel bucket list includes Australia and New Zealand, so I would love to visit soon.

Learn more about the Cloudera partner ecosystem.

The post Meet Michelle Hoover, Cloudera’s new SVP of Global Alliances and Channels appeared first on Cloudera Blog.

View Details

Today we are excited to announce the release of a new Cloudera Accelerator for Machine Learning (ML) Projects (AMP) for PDF document analysis, “Document Analysis with Command R and FAISS”, leveraging Cohere’s Command R Large Language Model (LLM), the Cohere Toolkit for retrieval augmented generation (RAG) applications, and Facebook’s AI Similarity Search (FAISS).

Document analysis is crucial for efficiently extracting insights from large volumes of text. It has wide-ranging applications including legal research, market analysis, and scientific research. For example, cancer researchers can use document analysis to quickly understand the key findings of thousands of research papers on a certain type of cancer, helping them identify trends and knowledge gaps needed to set new research priorities.

Before the widespread use of LLMs, document analysis was primarily conducted through manual methods and rule-based systems. These methods were often time-consuming, labor-intensive, and limited in their ability to handle complex language nuances and unstructured data.

The development of advanced LLMs, such as Cohere’s Command R, and AI Platforms, such as Cloudera Artificial Intelligence (CAI), made it easier than ever for enterprises to deploy high-impact document analysis applications. We created our “Document Analysis with Command R and FAISS” AMP to make that process even easier.

Cohere’s Command R Family of Models are advanced LLMs that leverage state-of-the-art transformer architectures to handle complex text generation and understanding tasks with high accuracy and speed, making them suitable for enterprise-level applications and real-time processing needs. They were made to be easily integrated into various applications, offering scalability and flexibility for both small-scale and large-scale implementations. The Cohere Toolkit is a collection of pre-built components enabling developers to quickly build and deploy retrieval augmented generation (RAG) applications.

CAI is a robust platform for data scientists and Artificial Intelligence (AI) practitioners to build, train, deploy, and manage models and applications at scale. AMPs are one-click deployments of commonly used AI/ML-based prototypes that reduce time to value by providing high-quality reference examples leveraging Cloudera’s research and expertise to showcase cutting-edge AI applications.

This AMP is a single project launched from CAI that automatically deploys an application, loads vectors into a FAISS vector store, and enables interfacing with Cohere’s Command R LLM to perform document analysis. The image below illustrates the Retrieval-Augmented Generation (RAG) architecture used by the AMP, and how the components of Cohere, FAISS, the user’s knowledge base, and Streamlit work together to create a ready-to-use Generative AI use case.

This project brings together several exciting new themes to Cloudera’s AMP library, especially in terms of RAG. Facebook’s open source FAISS is a library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. By leveraging it in this AMP, Cloudera demonstrates its flexibility in vector search applications and adds this capability on top of its adoption of Milvus, Chroma, Pinecone, and others in its existing AMP catalog.

Additionally, the AMP leverages LangChain’s AI toolkit that takes advantage of custom connectors to Cohere and FAISS to enable advanced semantic search and summarization capabilities in a clean and easy to understand code base. It also utilizes Cohere’s embed-english-v3.0 model which is tailor made for generating high-quality text embeddings from English language inputs and excels in capturing semantic nuances. By using Streamlit for the UI, users have a simple starting template, which can be the basis for a full-scale production deployment.

More on how the “Document Analysis with Command R and FAISS” AMP works and how to deploy it can be found in this Github Repository.

Be on the lookout for more news from Cohere and Cloudera as we work together to make it easier than ever to deploy high-performance AI applications.

The post Unlocking Faster Insights: How Cloudera and Cohere can deliver Smarter Document Analysis appeared first on Cloudera Blog.

View Details

Over the last few months, Cloudera has been traversing the globe hosting our EVOLVE24 event series. It has been a time full of excitement, innovative ideas, and connection with our partners and customers. It also provided a moment for us to launch an important initiative for Cloudera: our Women Leaders in Technology (WLIT) initiative.

WLIT is a global initiative developed to create a forum wherein women and allies in tech leadership roles can connect with and demonstrate to women and girls that it is possible to enter, grow, and thrive in the tech industry. It aims to shine a light on the gender imbalance in the industry, provide insight into policies and programs that help foster a stronger more diverse workforce, and create networking opportunities for women. This program goes beyond the critical work of our Womens+ ERG which aims to cultivate among Cloudera employees an inclusive environment that supports and encourages women to advance their skills and leadership potential through connection, mentorship, collaboration, recruiting, retention, and discussion. WLIT is industry-wide and seeks to connect, inspire, and elevate Clouderans as well as cross-sector leaders.

During EVOLVE New York, the WLIT group came together for a luncheon panel designed to kick off a conversation among the women—and allies—in the room and in the tech space more broadly about the challenges faced by women in tech and how to overcome them. The panel discussion included: Manasi Vartak, of Cloudera, Nichola Hammerton of Deutsche Bank, and Melissa Dougherty of AWS. Moderating the conversation was Forbes Reporter, Zoya Hasan. Zoya edits the Forbes 30 Under 30 lists, including U30 U.S., Europe, and Local, co-authors a weekly newsletter, and writes features on young founders.

Let’s dive into the panel discussion and a few of the biggest takeaways from our participants.

Building Inclusive Data-Driven Organizations: Leadership Strategies for the Modern WorkplaceAs it stands, women currently account for approximately 25% of the technology workforce. And that number only gets smaller the further up you advance in your career, with women holding just 11% of executive roles in the technology space. But, it’s about much more than a number. As Zoya pointed out in her opening remarks, women in technology are not just a statistic and we should be doing everything we can to flip prevailing assumptions to demonstrate that it’s not women in technology, instead, it is just people in technology who happen to be women.

As we started the discussion, our panelists covered several pressing issues surrounding how women leaders find success in building inclusive and data-driven organizations. The speakers covered everything from different leadership approaches to overcoming systemic barriers and driving organizational transformation and inclusion. Here are a few key takeaways:

It’s never too early (or late) to enter into a STEAM field. This was one of the key points raised during the discussion. There is tremendous value in encouraging women to get involved early, whether that’s in technology, mathematics, or other STEAM-related subjects. Likewise, while getting started early is important, it’s also not the only way to get into these disciplines. When it comes to carving out a career in technology, it’s never too late to take the first step.

Find a mentor who can help you grow in your career. For women looking to succeed professionally in technology, having a mentor can be incredibly impactful. Mentors, both women and men, bring plenty of experience and insight from their own lived experiences that can help you better understand how to handle various situations, deepen your networks, and provide trusted guidance in a competitive field. Cultivating a company culture that prioritizes and facilitates mentorship and sponsorship is imperative.

Don’t put limits on what you’re capable of. We’re all susceptible to second-guessing and self-doubt. But it’s important to recognize that feeling and work on overcoming it. Whether it’s thinking a project or task is too challenging for your skills or that you’re underqualified for a job you want (or maybe even already have), imposter syndrome is a feeling all the panelists could relate to but agreed it is rarely, if ever, justified.

“As women leaders in this space, it’s so important to share our experiences and learnings with other women to help encourage them in their own careers. Our first WLIT event has been incredibly rewarding and having the opportunity to connect with so many people throughout the luncheon goes to show just how important this community is.” – Manasi Vartak, Chief AI Architect, Cloudera

Our first WLIT event was an incredible experience, and we were so thrilled to see how engaged attendees were throughout the luncheon and how active the Q&A portion was. With the launch of our WLIT group, we hope to grow this community and support women throughout their technology careers—at Cloudera, our partner organizations, our customers, and beyond.

Find out more about Cloudera’s Women Leaders in Technology initiative and join our LinkedIn group to get involved.

The post Looking Back on Our First Women Leaders in Technology Event appeared first on Cloudera Blog.

View Details

In this Employee Spotlight, we sat down with Julia Ostrowski to learn about her time at Cloudera, what she loves about her job, her experience on both sides of Cloudera’s mentorship program, and her impressive volunteer work.

Meet Julia Ostrowski Julia is the Director of Enterprise Entitlement at Cloudera and has been with the company since 2019, joining via Hortonworks. Outside of her typical responsibilities, Julia is deeply involved in various philanthropic initiatives within Cloudera as well as in her own free time.

“Whenever there is an opportunity to do something good in the Santa Clara office for a couple of hours, I always sign up for it, no matter what it is,” Julia said. “There are so many rewarding initiatives that Cloudera puts together, from mentorships to taking care of animals or helping people in need. It’s a great part of Cloudera’s culture.”

Julia’s Cloudera Career Journey When Julia first arrived at Cloudera, she worked in Support as the manager of the Support Product Management team, helping ensure Cloudera’s COEs and Support Managers were able to provide world-class technical support to our customers. In 2022 she was offered the opportunity to join the growing and dynamic IT department under Olivia Keenaghan as a part of Business Applications. Now, she’s responsible for managing a small team that reviews the entire entitlement business process, taking ownership of any parts of the lifecycle that were distributed and/or unowned.

This was quite the career shift, but she described switching to the IT department as “fantastic,” crediting an amazing manager, a great team, and rewarding responsibilities.

“Making the change was such a welcoming experience,” she said. “I still get to my laptop each day with a smile on my face. I like the challenge of never knowing exactly what each day will bring, and I get excited to solve complex problems to help Cloudera grow. I have always said that at Cloudera your career is in your hands. I’ve seen people go from one team to a completely different team with such success and grace, totally supported by their former and new managers. You just don’t see that at every company, unfortunately”

Julia’s team consisting of Ken McCarthy and Susy Mena-McCarthy tracks and manages all of Cloudera’s public cloud metering for billing purposes and consistently optimizes processes to support business needs. Their current big project is Project Lionheart: the replacement of a licensing server to allow for more flexibility and much more control over the insight gained into customer usage over their on-premises product.

While Julia has a real passion for solving problems and driving value during her day-to-day job, she is always eager to leverage Cloudera’s extensive giving and volunteering programs to volunteer her time to help others.

Mentoring and Volunteering at Cloudera Cloudera offers a mentorship program to help employees improve and navigate their careers. They gain advice, set goals, and learn how to handle difficult situations at work, such as a project not going according to plan. Julia, who shared that she’s benefited from being a mentee herself both at Cloudera currently, and at previous organizations, volunteers to mentor up to four colleagues at a time and views this act as paying it forward.

“I really enjoy it,” she said. “I’ve gotten so much out of having a mentor that I feel obligated to be a mentor myself, but fortunately, I do love it. I appreciate being able to meet new people at Cloudera and do my best to help guide them in their careers.”

Julia says that a lot of the time, the key is to keep a good attitude, but the biggest piece of advice she offers is that uncertainty is not always a bad thing as long as you have a path forward.

“It’s okay not to know exactly where you will be in ten years, five years, or even two years,” she said. “Expecting to have your whole career path mapped out down to the last rock or pebble is a very old-school way of thinking, but you should always have your next goal in mind. It doesn’t have to be a new title or a particular salary, but it could be a new skill set that you want to add to your repertoire, for example. I always ask mentees, ‘What is your next goal?’ and work with them on finding a path there.”

Cloudera Cares Volunteer Program Julia also lends her time to philanthropic initiatives like Cloudera’s Teen Accelerator Program, an initiative organized by Cloudera’s volunteer group, Cloudera Cares.

The Teen Accelerator Program is a partnership with the Boys & Girls Club of America in both Tennessee and the San Francisco Bay Area. This program offers students a six-week paid internship program at Cloudera and 1:1 employee mentorship, helping facilitate opportunities in corporate America for high school students in under-resourced communities.

At first, Julia explained, she was a little nervous to start working with the program because she had not had much interaction with teens. However, despite that initial nervousness, she dove right in and was hooked instantly as the program tapped into her passion for helping others.

“It was such a rewarding experience,” Julia said. “I asked myself, ‘What did I want to know about the world of business when I was 16 years old?’ The first time I was in an office in my early years, I didn’t know how to dress or what to do. And there are so many simple questions. I remember not knowing whether you have to raise your hand to go to the bathroom or what ‘CEO’ means. I put myself in those shoes, and I’m glad I did. It was honestly fantastic.”

During the teen mentorship, Julia partnered with Cy Jervis (Senior Manager, Support Knowledge Programs) to virtually guide the teen mentee through the ins and outs of Cloudera, introducing him to key departments, and allowing him to ask questions and engage with colleagues. By the end of the program, he left with a solid understanding of how modern tech companies operate.

“It was so interesting to hear all our colleagues go in-depth sharing what they do and what their department works on that I was even learning right along with him,” Julia said. “I wasn’t expecting to have so much fun, but he was so engaged, and it was just great.”

Julia also takes advantage of another Cloudera volunteer initiative by spending some of her time with Second Harvest, a food distribution group in her local area of San Jose, California. Through this volunteer program, a group of Cloudera employees (championed by Executive Briefing Program Manager Amanda Allen) assist the organization in distributing food to people in need.

“Cloudera has quite a few volunteer opportunities, and this one is one of my absolute favorites,” Julia said. “I take advantage of every opportunity to take part in it. Whether it’s sunny or raining out, it doesn’t matter. I get to go and spend time with other fantastic Clouderans who I wouldn’t normally see, and I truly enjoy it. This is a great organization that puts food in the hands of people who really need it, and I am so appreciative of Amanda’s strong advocacy in organizing these opportunities for Cloudera.”

Julia is truly passionate about her philanthropic efforts and helping those in need both through Cloudera and outside of work. She continues to foster dogs and cats for local animal rescues (and occasionally they find a permanent home with her!). She loves being able to make a difference in her role in Enterprise Entitlement and appreciates the flexibility Cloudera granted her to explore a new position in IT that she truly enjoys.

Read our last employee spotlight here.

The post #ClouderaLife Employee Spotlight: Julia Ostrowski appeared first on Cloudera Blog.

View Details

In August, we wrote about how in a future where distributed data architectures are inevitable, unifying and managing operational and business metadata is critical to successfully maximizing the value of data, analytics, and AI. One of the most important innovations in data management is open table formats, specifically Apache Iceberg, which fundamentally transforms the way data teams manage operational metadata in the data lake. By maintaining operational metadata within the table itself, Iceberg tables enable interoperability with many different systems and engines.

The Iceberg REST catalog specification is a key component for making Iceberg tables available and discoverable by many different tools and execution engines. It enables easy integration and interaction with Iceberg table metadata via an API and also decouples metadata management from the underlying storage. It is a critical feature for delivering unified access to data in distributed, multi-engine architectures.

That’s why Cloudera added support for the REST catalog: to make open metadata a priority for our customers and to ensure that data teams can truly leverage the best tool for each workload– whether it’s ingestion, reporting, data engineering, or building, training, and deploying AI models.

Snowflake and Cloudera: Better TogetherIn the spirit of open data and engine freedom, Cloudera is excited to partner with Snowflake to bring the most comprehensive open data lakehouse, and the freedom it provides, to all of our customers.

Snowflake is one of the most popular platforms for data sharing, business intelligence (BI), reporting, and dashboarding due to its ease of use, self-service capabilities, and the performance of its execution engine. Snowflake is a prominent contributor to the Iceberg project, understanding the value it brings to its customers in terms of interoperability, data management, and data governance.

By leveraging Cloudera to build and manage Iceberg tables, Snowflake customers can make a single, consistent, and accurate view of their data available for their BI users without moving or copying data to other systems. They can take advantage of Cloudera’s true hybrid architecture and even provide easy access to on-premises data sources by leveraging Apache Ozone.

They can also leverage a single view of their data for any other Cloudera or third-party engine for other analytic workloads, including streaming, advanced analytics, and AI/ML.

With Snowflake’s engine, Cloudera customers get easy self-service access to their data for BI and interactive dashboards anywhere their data lives, including multiple public clouds and on-premises.

The Cloudera + Snowflake AdvantageThe partnership between Cloudera and Snowflake gives several advantages to joint customers:

• Lower Total Cost of Ownership: Reducing data copies and data movement while guaranteeing engine and infrastructure freedom enables customers to reduce storage, compute, and operational costs of maintaining their analytics stack.
• Choose the best tool for the job: By keeping data in open formats, customers can choose the environment and tools that provide the most ideal balance of cost and performance on a workload-by-workload basis. Customers have access to multiple public and private clouds and on-premises data stores, and they can use any engine that can read or write to Iceberg tables.
• True hybrid: Customers have full access to data stores on-premises and in every cloud without undertaking an expensive and complex migration project. They are free to choose the infrastructure best suited for each workload. Cloudera Shared Data Experience (SDX) enables customers to enforce consistent security and governance policies across all of their environments –even if data moves across clouds.

Try Cloudera and Snowflake TodayTogether, Cloudera and Snowflake deliver the most comprehensive hybrid open data lakehouse. It enables customers to confidently address virtually any analytic use case, from self-service BI that delivers actionable intelligence to business users to AI that transforms business processes and powers differentiated customer experiences.

Both platforms are free to try today. Try Cloudera’s open data lakehouse on AWS for 5 days for free here, or try Snowflake for free for 30 days here.

The post Cloudera and Snowflake Partner to Deliver the Most Comprehensive Open Data Lakehouse appeared first on Cloudera Blog.

View Details

In recent years, machine learning operations (MLOps) have become the standard practice for developing, deploying, and managing machine learning models. MLOps standardizes processes and workflows for faster, scalable, and risk-free model deployment, centralizing model management, automating CI/CD for deployment, providing continuous monitoring, and ensuring governance and release best practices.

However, the rapid rise of large language models (LLMs) has introduced new challenges around computing cost, infrastructure needs, prompt engineering, and other optimization techniques, governance, and more. This requires an evolution of MLOps into what we now call “large language model operations” (LLMOps).

Let’s explore some key differentiating areas where LLMOps introduce novel processes and workflows compared to traditional MLOps.

• Expanding the Builder Persona: Traditional ML applications largely involve data scientists building models, with ML engineers focusing on pipelines and operations. With LLMs, this paradigm has shifted. Data scientists are no longer the only ones involved—business teams, product managers, and engineers play a more active role, particularly because LLMs lower the barrier to entry for AI-driven applications. The rise of both open-source models (e.g.; Llama, Mistral) and proprietary services (e.g., OpenAI) have removed much of the heavy lifting around model building and training. This democratization is a double-edged sword. While LLMs can be easily integrated into products, new challenges like compute cost, infrastructure needs, governance, and quality must be addressed.
• Low-Code/No-Code as a Core Feature: In MLOps, tools were primarily designed for data scientists, focusing on APIs and integrations with Python or R. With LLMOps, low-code/no-code tooling has become essential to cater to a broader set of users and make LLMs accessible across various teams. A key trend is how LLMOps platforms now emphasize user-friendly interfaces, enabling non-technical stakeholders to build, experiment, and deploy LLMs with minimal coding knowledge.
• More Focus on Model Optimization: When using LLMs, teams often work with general-purpose models, fine-tuning them for specific business needs using proprietary data. Therefore, model optimization techniques are becoming central to LLMOps. These techniques, such as quantization, pruning, and prompt engineering, are critical to refining LLMs to suit targeted use cases. Optimization not only improves performance but is essential for managing the cost and scalability of LLM applications.
• Prompt Engineering: A completely new concept introduced by LLMOps is prompt engineering—the practice of crafting precise instructions to guide the model’s behavior. This is both an art and science, serving as a key method for improving the quality, relevance, and efficiency of LLM responses. Tools for prompt management include prompt chaining, playgrounds for testing, and advanced concepts like meta-prompting techniques where users leverage one prompt to improve another prompt, which should be part of an LLMOps stack. Techniques like Chain of Thoughts and Assumed Expertise are becoming standard strategies in this new domain.
• The Emergence of Retrieval-Augmented Generation (RAG): Unlike traditional ML models, many enterprise-level GenAI use cases involving LLMs rely on retrieving relevant data from external sources, rather than solely generating responses from pre-trained knowledge. This has led to the rise of Retrieval-Augmented Generation (RAG) architectures, which integrate retrieval models to pull information from enterprise knowledge bases, and then rank and summarize that information using LLMs. RAG significantly reduces hallucinations and offers a cost-effective way to leverage enterprise data, making it a new cornerstone of LLMOps. Building and managing RAG pipelines is a completely new challenge that wasn’t part of the MLOps landscape. In the LLMOps life cycle, building and managing a RAG pipeline has replaced traditional model training as a key focus. While fine-tuning LLMs is still critical (and similar to ML model training), it brings new challenges around infrastructure and cost. Additionally, the use of enterprise data in RAG pipelines creates new data management challenges. Capabilities like vector storage, semantic search, and embeddings have become essential parts of the LLMOps workflow—areas that were less prevalent in MLOps.
• Evaluation and Monitoring is Less Predictable: Evaluating and monitoring LLMs is more complex than with traditional ML models. LLM applications are often context-specific, requiring significant input from subject matter experts (SMEs) during evaluation. Auto-evaluation frameworks, where one LLM is used to assess another, are beginning to emerge. However, challenges like the unpredictability of generative models and issues like hallucination remain difficult to address. To navigate these challenges, many companies first deploy internal LLM use cases, such as agent assistants, to build confidence before launching customer-facing applications.
• Risk Management and Governance: Model risk management has always been a critical focus for MLOps, but LLMOps introduces new concerns. Transparency into what data LLMs are trained on is often murky, raising concerns about privacy, copyrights, and bias. Additionally, making LLMs auditable and explainable remains an unsolved problem. Enterprises are beginning to adopt AI risk frameworks, but best practices are still evolving. For now, focusing on thorough evaluation, continuous monitoring, creating a catalog of approved models, and establishing governance policies are essential first steps. AI governance will be a central pillar of LLMOps tooling going forward.

As enterprises adopt LLMs, the shift from MLOps to LLMOps is essential for addressing their unique challenges. LLMOps emphasizes prompt engineering, model optimization, and RAG. It also introduces new complexities in governance, risk management, and evaluation, making LLMOps crucial for successfully scaling and managing these advanced models in production.

For enterprises interested in learning more about leveraging LLMs, click here.

The post The Evolution of LLMOps: Adapting MLOps for GenAI appeared first on Cloudera Blog.

View Details

The open data lakehouse is quickly becoming the standard architecture for unified multifunction analytics on large volumes of data. It combines the flexibility and scalability of data lake storage with the data analytics, data governance, and data management functionality of the data warehouse. Open table formats are a key component of this architecture, as they provide many of the capabilities of traditional data warehousing directly on data lake storage, and Apache Iceberg is quickly becoming the standard format for vendors and customers alike.

Iceberg has many features that drastically reduce the work required to deliver a high-performance view of the data, but many of these features create overhead and require manual job execution to optimize for performance and costs. To make the data lakehouse even easier to manage, Cloudera is introducing Cloudera Lakehouse Optimizer, which intelligently automates Iceberg table maintenance so many of these jobs automatically run in the background. Let’s take a look at some of the features in Cloudera Lakehouse Optimizer, the benefits they provide, and the road ahead for this service.

Cloudera Lakehouse Optimizer FeaturesCloudera Lakehouse Optimizer runs automatic, policy-based Iceberg table optimization tasks based on user configurations and Iceberg table statistics. Automatic optimization jobs include:

Compaction: Companies often ingest many small files, such as with micro batching or streaming ingestion, and reading multiple small files can negatively impact query performance. Compaction is a process that rewrites small files into larger ones to improve performance. Cloudera Lakehouse Optimizer autonomously determines the best time to automatically compact data files so users always have the best performance from their tables. It also prioritizes the tables that must be optimized based on the usage patterns so we are only optimizing when there is real ROI.

Table Cleanup: As tables grow, they often accumulate unused data files, manifest files, and snapshots that aren’t needed anymore. Users may want to perform table maintenance functions, like expiring snapshots, removing old metadata files, and deleting orphan files, to optimize storage utilization and improve performance. Cloudera Lakehouse Optimizer will autonomously determine the best time to perform these maintenance tasks and ensure tables always utilize optimal storage.

In addition to optimization and policy-based controls, Cloudera Lakehouse Optimizer features observability for optimization jobs, so data teams can see and understand how their policies are impacting the health and performance of their tables and storage.

The BenefitsCloudera Lakehouse Optimizer provides several benefits for companies managing Iceberg tables:

• They experience lower Total Cost of Ownership (TCO) as a result of optimizing their storage footprint and reducing query runtimes.
• They can deliver a high-performance of their data by reducing the number of files that need to be read in a query.
• They reduce data management effort and overhead by automating some of the most tedious lakehouse maintenance tasks.

Fig 1. Cloudera internal benchmarks demonstrate significant cost savings using Cloudera Lakehouse Optimizer to maintain Iceberg tables. Actual results will vary depending on actual usage.

The Road AheadThe features we are launching in Cloudera Lakehouse Optimizer solve two very important challenges for companies who want to move to an open data lakehouse architecture. This is just the first step in advancing Cloudera’s vision of making it easier than ever to deliver a high-performance view of your data. Down the road, we plan to add support for more optimization features, including reorganizing partitions to solve data distribution problems that can impact query performance, and query optimization.

The goal for all of these features is to ensure that Cloudera is the best platform for managing and delivering access to Iceberg tables, and that the path to adopting an open data lakehouse is easier than ever.

Our Open Data Lakehouse is Free to TryYou can try Cloudera’s open data lakehouse on AWS for free today. Go sign up for our 5-day trial here to see for yourself.

The post Cloudera Lakehouse Optimizer Makes it Easier Than Ever to Deliver High-Performance Iceberg Tables appeared first on Cloudera Blog.

View Details

We are thrilled to announce the general availability of the Cloudera AI Inference service, powered by NVIDIA NIM microservices, part of the NVIDIA AI Enterprise platform, to accelerate generative AI deployments for enterprises. This service supports a range of optimized AI models, enabling seamless and scalable AI inference.

BackgroundThe generative AI landscape is evolving at a rapid pace, marked by explosive growth and widespread adoption across industries. In 2022, the release of ChatGPT attracted over 100 million users within just two months, demonstrating the technology’s accessibility and its impact across various user skill levels.

By 2023, the focus shifted towards experimentation. Enterprise developers began exploring proof of concepts (POCs) for generative AI applications, leveraging API services and open models such as Llama 2 and Mistral. These innovations pushed the boundaries of what generative AI could achieve.

Now, in 2024, generative AI is moving into the production phase for many companies. Businesses are now allocating dedicated budgets and building infrastructure to support AI applications in real-world environments. However, this transition presents significant challenges. Enterprises are increasingly concerned with safeguarding intellectual property (IP), maintaining brand integrity, and protecting client confidentiality while adhering to regulatory requirements.

A major risk is data exposure — AI systems must be designed to align with company ethics and meet strict regulatory standards without compromising functionality. Ensuring that AI systems prevent breaches of client confidentiality, personally identifiable information (PII), and data security is crucial for mitigating these risks.

Enterprises also face the challenge of maintaining control over AI development and deployment across disparate environments. They require solutions that offer robust security, ownership, and governance throughout the entire AI lifecycle, from POC to full production. Additionally, there is a need for enterprise-grade software that streamlines this transition while meeting stringent security requirements.

To safely leverage the full potential of generative AI, companies must address these challenges head-on. Typically, organizations approach generative AI POCs in one of two ways: by using third-party services, which are easy to implement but require sharing private data externally, or by developing self-hosted solutions using a mix of open-source and commercial tools.

At Cloudera, we focus on simplifying the development and deployment of generative AI models for production applications. Our approach provides accelerated, scalable, and efficient infrastructure along with enterprise-grade security and governance. This combination helps organizations confidently adopt generative AI while protecting their IP, brand reputation, and compliance with regulatory standards.

Cloudera AI Inference ServiceThe new Cloudera AI Inference service provides accelerated model serving, enabling enterprises to deploy and scale AI applications with enhanced speed and efficiency. By leveraging the NVIDIA NeMo platform and optimized versions of open-source models like Llama 3 and Mistral, businesses can harness the latest advancements in natural language processing, computer vision, and other AI domains.

Cloudera AI Inference: Scalable and Secure Model Serving The Cloudera AI Inference service offers a powerful combination of performance, security, and scalability designed for modern AI applications. Powered by NVIDIA NIM, it delivers market-leading performance with substantial time and cost savings. Hardware and software optimizations enable up to 36 times faster inference with NVIDIA accelerated computing and nearly four times the throughput on CPUs, accelerating decision-making.

Integration with NVIDIA Triton Inference Server further enhances the service. It provides standardized, efficient deployment with support for open protocols, reducing deployment time and complexity.

In terms of security, the Cloudera AI Inference service delivers robust protection and control. Customers can deploy AI models within their virtual private cloud (VPC) while maintaining strict privacy and control over sensitive data in the cloud. All communications between the applications and model endpoints remain within the customer’s secured environment.

Comprehensive safeguards, including authentication and authorization, ensure that only users with configured access can interact with the model endpoint. The service also meets enterprise-grade security and compliance standards, recording all model interactions for governance and audit.

The Cloudera AI Inference service also offers exceptional scalability and flexibility. It supports hybrid environments, allowing seamless transitions between on-premises and cloud deployments for increased operational flexibility.

Seamless integration with CI/CD pipelines enhances MLOps workflows, while dynamic scaling and distributed serving optimize resource usage. These features reduce costs without compromising performance. High availability and disaster recovery capabilities help enable continuous operation and minimal downtime.

Feature Highlights: Hybrid and Multi-Cloud Support: Enables deployment across on-premises, public cloud, and hybrid environments, offering flexibility to meet diverse enterprise infrastructure needs. * Model Registry Integration: Seamlessly integrates with Cloudera AI Registry, a centralized repository for storing, versioning, and managing models, enabling consistency and easy access to different model versions. * Detailed Data and Model Lineage Tracking: Ensures comprehensive tracking and documentation of data transformations and model lifecycle events, enhancing reproducibility and auditability. * Enterprise-Grade Security: Implements robust security measures, including authentication, authorization*, and data encryption, helping ensure that data and models are protected both in transit and at rest. * Real-time Inference Capabilities: Provides real-time predictions with low latency and batch processing for large datasets, offering flexibility in serving AI models based on different needs. * High Availability and Dynamic Scaling: Features high availability configurations and dynamic scaling capabilities to efficiently handle varying loads while delivering continuous service. * Advanced Language Model: Support with pre-generated optimized engines for a diverse range of cutting-edge LLM architectures. * Flexible Integration: Easily integrate with existing workflows and applications. Developers are provided open inference protocol APIs for traditional ML models and with an OpenAI compatible API for LLMs. * Multiple AI Framework Support: Integrates seamlessly with popular machine learning frameworks such as TensorFlow, PyTorch, Scikit-learn, and Hugging Face Transformers, making it easy to deploy a wide variety of model types. * Advanced Deployment Patterns: Supports sophisticated deployment strategies like canary and blue-green deployments, as well as A/B testing, enabling safe and gradual rollouts of new model versions. * Open APIs: Provides standards-compliant, open APIs for deploying, managing, and monitoring online models and applications*, as well as for facilitating integration with CI/CD pipelines and other MLOps tools. * Performance Monitoring and Logging: Provides comprehensive monitoring and logging capabilities, tracking performance metrics such as latency, throughput, resource utilization, and model health, supporting troubleshooting and optimization. * Business Monitoring: Supports continuous monitoring of key generative AI modeI metrics like sentiment, user feedback, and drift that are crucial for maintaining model quality and performance.

The Cloudera AI Inference service, powered by NVIDIA NIM microservices, delivers seamless, high-performance AI model inferencing across on-premises and cloud environments. Supporting open-source community models, NVIDIA AI Foundation models, and custom AI models, it offers the flexibility to meet diverse business needs. The service enables rapid deployment of generative AI applications at scale, with a strong focus on privacy and security, to help enterprises that want to unlock the full potential of their data with AI models in production environments.

• feature coming soon – please reach out to us if you have questions or would like to learn more.

The post Deploy and Scale AI Applications With Cloudera AI Inference Service appeared first on Cloudera Blog.

View Details

Cloudera customers understand the potential impact of data, analytics, and AI on their respective businesses — reducing costs, managing risk, improving customer satisfaction, and generating new business opportunities that help to increase market share.

But, what is the ultimate impact of all this effort and investment on each of us in our daily lives? At EVOLVE in Singapore, the Manila Electric Company, Meralco, won the Cloudera 2024 Data Impact Award in the Leadership and Transformation category for its customer-centric and data-driven transformation. In addition to literally powering countries, there were other impressive stories from award finalists and winners that focused on the human element. Specifically, the People and Society award category highlighted data’s positive influence on the world at large.

That got us thinking: How could we share our customer stories in a way that clearly articulates the impact of Cloudera on our daily lives? So we built an interactive tour showcasing that impact throughout a typical day. This tour features examples of data, analytics, and AI delivering memorable and tangible experiences for every-day people.

For example, SAIC-Volkswagen delivers insights and services to drivers through an application powered by Cloudera. In fact, 8 out of the top 10 global automakers use Cloudera to deliver a connected vehicle experience.

BT Group is among 7 of the top 10 global telecommunications companies that use Cloudera to improve the customer experience and operational efficiency. They process 5 times the data in a third of the time, enabling them to deliver better network performance to their customers.

And pharmaceutical companies like IQVIA use Cloudera to provide visibility across the R&D pipeline, accelerating the development of life-saving drugs.

These are just a couple of the stories we highlight in our interactive tour. You can see the rest for yourself here.

And if you want to learn why the world’s most recognized enterprises trust Cloudera for business-critical data, analytics, and AI initiatives, you can read their stories here.

The post The Global Impact of Cloudera in Our Daily Lives appeared first on Cloudera Blog.

View Details

The journey from a great idea for a Generative AI use case to deploying it in a production environment often resembles navigating a maze. Every turn presents new challenges—whether it’s technical hurdles, security concerns, or shifting priorities—that can stall progress or even force you to start over.

Cloudera recognizes the struggles that many enterprises face when setting out on this path, and that’s why we started building Accelerators for ML Projects (AMPs). AMPs are fully built out ML prototypes that can be deployed with a single click directly from Cloudera Machine Learning . AMPs enable data scientists to go from an idea to a fully working ML use case in a fraction of the time. By providing pre-built workflows, best practices, and integration with enterprise-grade tools, AMPs eliminate much of the complexity involved in building and deploying machine learning models.

In line with our ongoing commitment to supporting ML practitioners, Cloudera is thrilled to announce the release of five new Accelerators! These cutting-edge tools focus on trending topics in generative AI, empowering enterprises to unlock innovation and accelerate the development of impactful solutions.

Fine Tuning StudioFine tuning has become an important methodology for creating specialized large language models (LLM). Since LLMs are trained on essentially the entire internet, they are generalists capable of doing many different things very well. However, in order for them to truly excel at specific tasks, like code generation or language translation for rare dialects, they need to be tuned for the task with a more focused and specialized dataset. This process allows the model to refine its understanding and adapt its outputs to better suit the nuances of the specific task, making it more accurate and efficient in that domain.

The Fine Tuning Studio is a Cloudera-developed AMP that provides users with an all-encompassing application and “ecosystem” for managing, fine tuning, and evaluating LLMs. This application is a launcher that helps users organize and dispatch other Cloudera Machine Learning workloads (primarily via the Jobs feature) that are configured specifically for LLM training and evaluation type tasks.

RAG with Knowledge GraphRetrieval Augmented Generation (RAG) has become one of the default methodologies for adding additional context to responses from a LLM. This application architecture makes use of prompt engineering and vector stores to provide an LLM with new information at the time of inference. However, the performance of RAG applications is far from perfect, prompting innovations like integrating knowledge graphs, which structure data into interconnected entities and relationships. This addition improves retrieval accuracy, contextual relevance, reasoning capabilities, and domain-specific understanding, elevating the overall effectiveness of RAG systems.

RAG with Knowledge Graph demonstrates how integrating knowledge graphs can enhance RAG performance, using a solution designed for academic research paper retrieval. The solution ingests significant AI/ML papers from arXiv into Neo4j’s knowledge graph and vector store. For the LLM, we used Meta-Llama-3.1-8B-Instruct which can be leveraged both remotely or locally. To highlight the improvements that knowledge graphs deliver to RAG, the UI compares the results with and without a knowledge graph.

PromptBrew by Vertav80% of Generative AI success depends on prompting and yet most AI developers can’t write good prompts. This gap in prompt engineering skills often leads to suboptimal results, as the effectiveness of generative AI models largely hinges on how well they are guided through instructions. Crafting precise, clear, and contextually appropriate prompts is crucial for maximizing the model’s capabilities. Without well-designed prompts, even the most advanced models can produce irrelevant, ambiguous, or low-quality outputs.

PromptBrew provides AI-powered assistance to help developers craft high-performing, reliable prompts with ease. Whether you’re starting with a specific project goal or a draft prompt, PromptBrew guides you through a streamlined process, offering suggestions and optimizations to refine your prompts. By generating multiple candidate prompts and recommending enhancements, it ensures that your inputs are tailored for the best possible outcomes. These optimized prompts can then be seamlessly integrated into your project workflow, improving performance and accuracy in generative AI applications.

Chat with your Documents This AMP showcases how to build a chatbot using an open-source, pre-trained, instruction-following Large Language Model (LLM). The chatbot’s responses are improved by providing it with context from an internal knowledge base, created from documents uploaded by users. This context is retrieved through semantic search, powered by an open-source vector database.

In comparison to the original LLM Chatbot Augmented with Enterprise Data AMP, this version includes new features such as user document ingestion, automatic question generation, and result streaming. It also leverages Llama Index to implement the RAG pipeline.

To learn more, click here.

The post Streamlining Generative AI Deployment with New Accelerators appeared first on Cloudera Blog.

View Details

We’re more than a week into Hispanic Heritage Month, which started on September 15 and continues through October 15. This month is an annual celebration in the United States that honors the contributions, culture, and achievements of Hispanic and Latinx Americans. Over the next few weeks, we’ll be gathering with fellow Clouderans to reflect on and celebrate, the achievements of the Hispanic and Latinx communities here in the U.S. and across the globe.

Fundamentally, this month is about sharing what makes each of us unique and learning about the cultural differences across these communities to better understand the experiences that unite us all.

It’s primed to be a busy month, so let’s dive into what we have planned for an exciting month of learning, growing, and giving back.

A Month Full of Action and LearningCloudera’s calendar is full of engaging activities to help not just bring awareness to the Hispanic and Latinx experience and history but also to give back and make an impact on the next generation.

At our most recent event, Cloudera volunteers helped Hispanic and Latinx students at under-resourced schools enhance their LinkedIn profiles. We spent time offering constructive recommendations to build on what they already had and align them with professional standards. Our goal was to enable these students to find more secure employment opportunities, build a strong professional brand, and learn from what other Latinx and Hispanic Americans are doing, both in their careers and via networking settings like LinkedIn.

Looking ahead, the rest of the month is jam-packed with fun and informative activities too. On October 8, we’re hosting a Hispanic Heritage Month Workshop and a Hispanic and Latin American History and Culture Trivia Event. Both offer a great opportunity for attendees to celebrate the diverse histories, languages, and heritage of Hispanic and Latinx cultures across the U.S. and Latin America. During trivia, we’ll test our knowledge on everything from famous Hispanic and Latinx figures to historical events to pop culture knowledge.

Whether it’s through volunteering or trivia, there are numerous opportunities for Clouderans to get involved and learn about their colleagues’ experiences. Each activity this month will help attendees gain an appreciation for the long, rich, and complex history that got us here, and better understand the unique perspectives that shape the Latinx and Hispanic experience around the world.

Reflecting on A Month of Learning and GrowthThis month is so important in helping bring a real, meaningful level of engagement and connection to Clouderans. As we participate in each of these events, it’s important to step back and appreciate the many contributions Hispanic and Latinx Americans have made to the world while also acknowledging that the concept of Hispanic or Latinx is not a monolith. So many amazing cultures and communities are encompassed by those labels and it’s important to highlight and celebrate the unique aspects of each one.

Hispanic Heritage Month has been around in the U.S. since 1988. Since then, it’s evolved from a weeklong event into its own month-long celebration. This evolution is a testament to the profound impact of Hispanic and Latinx culture and history in the U.S. As we near the final weeks of celebration, it is rewarding to see just how invested everyone has been in building a warm, welcoming environment for team members to connect and showcase the things that make them unique.

Learn more about how Cloudera is building a more inclusive and diverse workplace.

The post Celebrating Hispanic Heritage Month with Cloudera appeared first on Cloudera Blog.

View Details

According to recent survey data from Cloudera, 88% of companies are already utilizing AI for the tasks of enhancing efficiency in IT processes, improving customer support with chatbots, and leveraging analytics for better decision-making. More and more enterprises are leveraging pre-trained models for various applications, from natural language processing to computer vision. For that reason, Cloudera is evaluating a new line of business: Cloudera Integrated Data and AI Exchange (InDaiX).

InDaiX provides data consumers with unparalleled flexibility and scalability, streamlining how businesses, researchers, and developers access and integrate diverse data sources and AI foundational models, expediting the process of Generative AI (GenAI) adoption.

As part of this evaluation process with InDaiX, Cloudera is conducting workshops with end users to better understand the practical use cases that enterprises are hoping to use AI for. InDaiX is being evaluated as an extension of Cloudera to include:

• Datasets Exchange:
  • Industry Datasets: Comprehensive datasets across various domains, including healthcare, finance, and retail.
  • Alternative Datasets: Unique datasets, such as location intelligence and social media data, providing novel insights for various applications.
  • Synthetic Datasets: High-quality synthetic data generated using state-of-the-art techniques, ensuring privacy and compliance.
• AI Foundational Models Exchange: Access to pre-trained AI models, including natural language processing models, computer vision models, and reinforcement learning models, catering to various industry needs.
• Unique Data Integration and Experimentation Capabilities: Enable users to bridge the gap between choosing from and experimenting with several data sources and testing multiple AI foundational models, enabling quicker iterations and more effective testing.
• Scalability and Flexibility: Cloudera’s scalable architecture supports the growing volume and variety of data from InDaiX, allowing the platform to expand and adapt to changing data needs without compromising performance.

Offering a vast variety of data choices and AI foundational models, InDaiX enables businesses to create new data assets at scale, enabling them to run workloads in private or public clouds, choosing the most suitable environment for their needs. For data providers, InDaiX enhances distribution by reaching Cloudera’s established customer base and provides valuable feedback on data usage and integration with AI models. This dual value proposition makes InDaiX an essential platform for driving innovation and operational efficiency in the data and AI markets.

By offering high-quality, privacy-compliant, and scalable data solutions and AI models, Cloudera is cementing itself as the go-to platform for businesses, researchers, and developers seeking to build bespoke, innovative data-driven insights and applications.

For enterprises looking to work and participate in the evaluation process, learn more about how to get involved here.

The post Cloudera Evaluates Integrated Data and AI Exchange Business Line to Optimize Data-Driven Generative AI Use Cases appeared first on Cloudera Blog.

View Details

Imagine a world where your sensitive data moves effortlessly between clouds – secure, private, and far from the prying eyes of the public internet. Today, we’re making that world a reality with the launch of Cloudera Private Link Network.

Organizations are continuously seeking ways to enhance their data security. One of the challenges is ensuring that data remains protected as it traverses different cloud environments. Cloud provider solutions like AWS PrivateLink and Azure Private Link are a step in the right direction, but they often fall short of providing a comprehensive solution across multiple clouds.

Cloudera Private Link Network is designed to provide seamless, private connectivity between your cloud workloads and the Cloudera Control Plane. Cloudera Private Link Network ensures that data never leaves your secure, private network, even across multiple cloud environments, offering peace of mind in an increasingly privacy-conscious climate.

Why Cloudera Private Link Network?As industries like financial services, healthcare, and pharmaceuticals continue to navigate strict data privacy policies, the need for secure, private connectivity has never been more critical. Cloudera Private Link Network addresses these concerns head-on by providing a unified, cross-cloud private connectivity service that goes well beyond what’s currently available.

1. Integrated Solution: Whether you’re connecting workloads in AWS or Azure, Cloudera Private Link Network provides a seamless, integrated solution that eliminates the need for multiple vendor-specific solutions.
2. Better Security Posture: Cloudera Private Link Network ensures that your data remains within a secure, private network, significantly reducing the attack surface. This is particularly crucial for highly regulated industries where even the slightest data exposure can have serious consequences.
3. Reduced Network/CloudOps Load: By utilizing the Cloudera Private Link Network, you free your network and CloudOps teams from having to design, test, deploy, manage, and monitor multi-cloud connectivity. These teams can spend their time on more strategic work.
4. Lower TCO: Unlike point solutions that require you to pay for individual, atomic links on a per link basis, with Private Link Network, you pay to access the network on a per-VPC or per-account basis. The pricing is based on consumption so you only pay for what you use. Ease of management also results in lower operational costs.
5. Network Performance: With Cloudera Private Link Network, security doesn’t come at the cost of network performance. Your operations remain smooth and efficient, even as your data remains fully protected.

Cloudera Private Link Network is a game changer for industries where data privacy is paramount. Here’s a sample of how it’s making a difference:

• Financial Services: For financial institutions managing large volumes of sensitive customer and financial data, Cloudera Private Link Network enables secure connectivity while ensuring compliance with regulations and policies by keeping data off of the public internet.
• Pharmaceuticals: Pharmaceutical companies need to integrate sensitive data, including Personally Identifiable Information (PII), across the R&D pipeline to accelerate the development of life-saving medications. Cloudera Private Link Network provides a secure environment where teams can integrate and analyze sensitive information without fear of exposing that data outside the organization.
• Healthcare: Healthcare providers are under pressure to leverage data to deliver patient centricity and a continuum of care . Cloudera Private Link Network enables providers to build a proactive, data-driven healthcare experience while maintaining strict privacy standards, reducing the risk of data loss and ensuring data privacy.

At Cloudera, we understand that ease of use is just as important as security. That’s why Cloudera Private Link Network is designed to be user friendly and flexible, with two deployment options to suit your organization’s needs. Customers can turn on these options using the Cloudera Command Line Interface (CLI), enabling this functionality on demand.

• VPC Option: This option is ideal for organizations that prefer to use Cloudera’s CLI for a complete setup, offering end-to-end control and management.
• Authorization Option: This option is perfect for customers who want to integrate Cloudera Private Link Network with their existing cloud automation tools, such as Terraform or AWS CloudFormation, without the need for cross-account permissions.

PricingSecuring your cloud infrastructure shouldn’t come with hidden costs. That’s why Cloudera Private Link Network uses a consumption-based pricing model, ensuring that you only pay for what you use. This approach provides transparency and predictability, enabling organizations to scale their use of Private Link Network according to their needs, whether it’s for single cloud virtual networks or entire Cloudera accounts that span multiple cloud accounts.

Click here for more information on Cloudera Private Link Network.

The post Cloudera Launches Private Link Network for Secure, Internet-Free Cloud Connectivity appeared first on Cloudera Blog.

View Details

Without a doubt, Artificial Intelligence (AI) is revolutionizing businesses, with Australia’s AI spending expected to hit $6.4 billion by 2026. However, according to The State of Enterprise AI and Modern Data Architecture report, while 88% of enterprises adopt AI, many still lack the data infrastructure and team skilling to fully reap its benefits. In fact, over 25% of respondents stated they don’t have the data infrastructure required to effectively power AI. We also found that over 39% of respondents said that almost none of their employees are currently using AI.

Interestingly, Gartner has predicted that at least 30% of GenAI projects will be abandoned after proof of concept by the end of 2025. With that in mind, the question then becomes: How will you embrace technologies and projects when you can’t see the time to value that AI will bring to the organization?

Translating AI’s Potential into Measurable Business ImpactIt can’t be denied that a mature enterprise data strategy generates better business outcomes in the form of revenue growth and cost savings. Organizations also see improvements in customer experience, operational efficiency, and supply chain optimization.

However, to fully realize the benefits of AI and its perceived value, organizations must measure their AI objectives against key business metrics used internally. This alignment is crucial for the progression of these projects. It also becomes the basis for communicating to internal stakeholders to secure sustained funding and financial investment. Adopting common business metrics also enhances the likelihood of successful implementation and value realization from these investments.

OCBC Bank’s adoption of AI has effectively impacted revenue generation and better risk management. In addition, it has improved developers’ efficiency by 20%.

Ensuring AI’s Trust with IntentAI projects cannot begin without trust. Trusting AI equates to trusting the data it uses, meaning it must be accurate, consistent, and unbiased. Ethical AI depends on trustworthy data, guaranteeing equitable outcomes that reflect the company’s principles.

This means access to data completeness is critical. Yet, it’s a challenge for 55% of organizations that suggest accessing all of their company’s data is more daunting than a root canal.

Ensuring AI trust involves understanding your data and scrutinizing data sources, quality, access, and storage within your organization. Consider the intent, potential biases, and implications of AI decisions. Empathize with customers’ perspectives on data usage to guide ethical practices. If you wouldn’t approve of how the data would be used, it’s a sign to reassess your approach.

Kick-starting Your AI JourneySo, how do you transition an AI project from concept to full production and reap its benefits? Here are some tips for organizations starting on their ethical AI journey:

• Formulate a data strategy. This starts and ends with business value. Look at the organization’s mission, vision, and key objectives, and develop a holistic approach that involves people, processes, and technology to leverage your data assets and develop capabilities and use cases to support business objectives.
• Know Your Data, Know Your Intent. Ask yourself: is the data integrated into your systems reliable, and can you trust your organization’s intentions for using that data? A deliberate and thoughtful design of AI systems is crucial to ensure the outcomes are fair and unbiased, reflecting the organization’s ethos and principles. Organizations must have a clear vision of what they aim to achieve with AI to avoid missing out on its benefits or, worse, damaging their reputation and customer trust.
• Utilize a modern data platform that unifies the data lifecycle. Your data platform should facilitate the implementation of modern data architectures – data mesh, fabric, or open data lakehouse – with security and governance as the foundation. This platform should enable your organization to handle the complex data challenges that arise daily across different functions, enabling seamless deployment of workloads between on-premise and cloud (or multi-cloud) without workload refactoring. Most importantly, it should maintain data traceability and uphold stringent security policies and access controls from one environment to another.

AI Assistants – Democratize AI For UsersWhat’s in trend today may not be tomorrow, and it’s possible that public LLMs will soon become a thing of the past before the next disruptive technology comes along. Perhaps you find accessing your data challenging or you lack the technical skills in-house to build and deploy GenAI capabilities.

Fortunately, modern data platforms with AI Assistants can facilitate AI adoption across the organization, giving Data Analysts access to ‘conversational AI’ capabilities and all everyday users faster access to their data-driven insights.

Learn more about how Cloudera can help accelerate your enterprise AI adoption.

The post Moving Your AI Pilot Projects to Production appeared first on Cloudera Blog.

View Details

Register for EVOLVE24 in Dubai (September 12, 2024) to hear from industry leaders on why hybrid solutions are essential for navigating an increasingly complex regulatory environment.

A prominent global bank was thrust into the spotlight for all the wrong reasons. The institution was hit with a staggering fine – multiple billions – for failing to comply with new data protection regulations that ultimately led to a customer data breach. The breach, which exposed sensitive information, not only resulted in financial penalties but also caused significant reputational damage. Customers lost trust, investors questioned the bank’s governance, and competitors seized the opportunity to highlight the incident swaying customers away from the bank with messaging about data privacy and incentives.

Another scenario: A major lender rolls out a new AI-driven credit scoring system to streamline loan approvals. The system was expected to reduce processing times and improve customer satisfaction. However, six months into its implementation, regulators discovered that the AI model had been trained on biased historical data and was inadvertently discriminating against certain demographic groups, leading to unfair lending practices.

Regulators determined the bank was not compliant with anti-discrimination laws and data protection regulations, as the AI system lacked transparency and failed to meet the required standards for fairness. The bank was fined $100 million and ordered to audit and overhaul its AI practices. The incident not only resulted in financial penalties but also sparked public outrage, damaging the bank’s reputation and leading to a significant loss of customer trust.

While these scenarios are hypotheticals, the risk is real.

For good reason, the financial services industry is facing an increasingly complex regulatory landscape, particularly when it comes to data privacy and the use of artificial intelligence. However, as regulations become more stringent and data governance demands grow, financial institutions are under immense pressure to manage their data with greater precision, making effective data management within a hybrid cloud environment essential.

How a Hybrid Cloud Architecture Empowers Regulatory ComplianceA hybrid cloud architecture has emerged as a crucial strategy for financial institutions to navigate these regulations while maintaining innovation and operational efficiency. By combining the best of on-premises and cloud environments, hybrid architectures offer a flexible, secure, and scalable data management solution that empowers financial institutions to maintain compliance, enhance security, and adapt to regulatory changes—all while optimizing costs and ensuring business continuity.

Let’s review some of the more critical regulations and the impact of a hybrid cloud architecture.

Privacy RegulationsPrivacy regulations like GDPR (EU), CCPA (California, US), LGPD (Brazil), APPI (Japan), and PIPL (China) have profoundly influenced how financial institutions manage personal data. Implementing a hybrid cloud architecture offers several key advantages in complying with these stringent requirements:

Data Sovereignty and LocalizationMany privacy laws require certain types of data to be stored within specific geographic boundaries. Hybrid cloud allows financial institutions to maintain sensitive data on-premises or in private clouds within the required jurisdictions while leveraging public cloud resources for non-sensitive workloads.

Granular Data ControlHybrid cloud enables financial institutions to implement fine-grained access controls and data classification systems. This allows for better management of personal data, making it easier to comply with data subject rights (e.g., right to access, right to be forgotten) mandated by regulations like GDPR and CCPA.

Enhanced Security MeasuresHybrid cloud allows for the implementation of robust security measures, including encryption, tokenization, and data masking. These techniques are crucial for protecting personal data and meeting the security requirements of privacy regulations.

Compliance Monitoring and ReportingHybrid cloud often includes tools that facilitate continuous compliance monitoring and automated reporting. This capability is essential for financial institutions to maintain transparency and accountability in line with regulatory requirements.

Disaster Recovery and Business ContinuityA hybrid cloud’s ability to distribute workloads across different environments provides a strong foundation for disaster recovery and business continuity. This ensures that personal data remains protected and accessible even in the event of a system failure or cyberattack.

AI-Specific RegulationsAs AI becomes increasingly integral to financial services, regulations like the EU AI Act (EU), AIDA (Canada), the Digital India Act (India) and most recently the California S.B. 1047 AI bill, are emerging to ensure its ethical and responsible use. Navigating these regulations requires a robust infrastructure, and hybrid cloud architectures are proving to be essential in meeting these new challenges in the following ways:

Transparency and ExplainabilityAI regulations often require organizations to provide transparency in their AI decision-making processes. Hybrid cloud environments can facilitate the storage and processing of AI models and their associated data, allowing for easier auditing and explanation of AI outcomes.

Model GovernanceHybrid cloud enables financial institutions to implement comprehensive model governance frameworks. This includes version control, model testing, and validation processes, which are crucial for complying with AI regulations that demand rigorous oversight of AI systems.

Data Quality and Bias MitigationMany AI regulations focus on ensuring fairness and preventing bias in AI systems (Ethical AI). Hybrid cloud architectures allow for better data management and quality control, helping financial institutions maintain high-quality, diverse datasets for training AI models and mitigating potential biases.

Financial Services-Specific RegulationsFinancial institutions face additional industry-specific regulations that impact their IT infrastructure choices. Hybrid cloud architectures are well-suited to address these requirements:

Basel III and IV: These regulations focus on capital adequacy, stress testing, and market liquidity risk. Hybrid cloud architectures provide the computational power needed for complex risk calculations and stress tests while allowing sensitive data to remain on-premises or in private clouds.

MiFID II: This regulation requires extensive record-keeping and reporting. Hybrid cloud architectures offer the scalability to handle large volumes of transaction data while maintaining the security needed for sensitive financial information.

DORA (Digital Operational Resilience Act): DORA focuses on the digital operational resilience of financial institutions. Hybrid cloud architectures enhance operational resilience by providing redundancy, disaster recovery capabilities, and the ability to quickly scale resources in response to operational challenges.

ESG Regulations: As ESG (Environmental, Social, and Governance) reporting becomes mandatory, financial institutions need robust data management and analytics capabilities. Hybrid cloud architectures provide the flexibility to collect, store, and analyze vast amounts of ESG-related data while ensuring compliance with data privacy regulations.

How can Cloudera’s Hybrid Data Platform help address regulatory compliance?Cloudera’s hybrid data platform is a comprehensive solution for financial institutions navigating today’s complex regulatory environment while striving for innovation and operational efficiency while reducing risk. By integrating on-premises, private, and public cloud resources into a unified architecture, Cloudera helps organizations address data sovereignty requirements mandated by international privacy regulations such as GDPR, CCPA, and PIPL. The platform’s advanced security and governance features, powered by Cloudera’s Shared Data Experience (SDX), ensure compliance with AI-specific regulations like the EU AI Act and AIDA, delivering transparency, explainability, and robust model governance

For regulations, including Basel III/IV, MiFID II, and DORA, Cloudera’s scalable analytics capabilities support intricate risk calculations, comprehensive record-keeping, and enhanced operational resilience. The platform’s flexibility enables institutions to adapt swiftly to changing regulatory demands while harnessing advanced analytics and AI for critical functions such as fraud detection, risk modeling, and ESG reporting. By providing a cohesive environment for managing and analyzing data across hybrid and multi-cloud deployments, Cloudera empowers financial institutions to stay compliant, improve customer experiences, and drive innovation in a dynamic digital landscape.

To learn more about Cloudera’s work with financial institutions, click here.

The post The critical role of a hybrid cloud architecture in ensuring regulatory compliance in financial services appeared first on Cloudera Blog.

View Details

Achieving equality and fairness requires ongoing effort, and for a business to be truly successful, it’s critical to raise awareness and create leadership and growth opportunities for underrepresented communities.

We are proud to share that Cloudera was recently named a Finalist for the Ragan 2024 Corporate Social Responsibility (CSR) & Diversity Awards under the Mentoring Program category for our Sponsorship Program’s commitment to Diversity, Equity, and Inclusion (DE&I). At Cloudera, we believe that true success exists when we foster an inclusive environment where everyone can thrive.

Cloudera’s DE&I strategy centers on a multifaceted approach at every level of the organization. Central to this is establishing clear goals and objectives, regular reviews of diversity analytics, and enhanced transparency and accountability. The company actively engages leaders and team members across all business units to ensure widespread participation and ownership of DE&I initiatives.  

The Ragan Awards celebrate the most successful campaigns, initiatives, people, and teams in the communication, public relations, marketing, and employee wellbeing industries. Its Mentoring Program award goes to organizations that champion a mentorship program that is instrumental in advancing the careers of employees from underrepresented communities. All finalists are recognized at a special event in New York City on September 27, before category winners are announced.

Cloudera’s Sponsorship Program addresses the disparity between mentorship and sponsorship experienced by underrepresented groups (URGs). Historically, these communities have not received the same opportunities as others and lack active advocates and supporters of their career advancement. The primary goal of Cloudera’s Sponsorship Program is to amplify the high-potential talent within these groups, increase their visibility within the Company, and actively advocate for them, which results, oftentimes, in talented individuals advancing into leadership positions.

“This program was life-changing for me,” said a Sponsorship Program participant. “There were many opportunities to reflect on my areas of improvement and also on the responsibility that I have to lead my own career. Besides that, to have the opportunity to learn with the Cloudera executive was remarkable.”

How the Sponsorship Program Works Cloudera’s Sponsorship Program operates on a cohort-based model, spanning six months. It strategically pairs senior leaders, or “sponsors,” with “proteges”, Clouderans with diverse backgrounds across all company functions. This highly intentional pairing ensures that proteges receive personalized support and advocacy from experienced leaders, committed to advancing diversity and inclusion.

Through Cloudera’s partnership with Sounding Board, a renowned leadership coaching firm, proteges also receive additional professional coaching and support throughout the duration of the program. This enhances participants’ personal and professional development, empowering them to navigate challenges and seize opportunities for growth. 

The Work Behind the Scenes Cloudera’s Sponsorship Program involves meticulous planning and coordination to ensure the success of each cohort. Program administrators work closely with senior leadership to identify suitable sponsors and proteges, considering each participant’s unique strengths and aspirations.

Once pairs are established, sponsors are encouraged to advocate for their proteges, leveraging their influence and networks to create visibility and exposure to other senior leaders and professional development opportunities. Simultaneously, proteges engage in one-on-one coaching sessions with Sound Board’s certified professional development experts, focusing on career advancement, skill development, and leadership competencies. Throughout, sponsors, proteges, and program administrators collaborate closely to monitor progress, address challenges, and celebrate achievements. 

Keys to Success Cloudera’s Sponsorship Program is evaluated on its ability to elevate high-potential talent, increase their visibility within the company, and foster diverse leadership pipelines. It conducts engagement and diversity surveys to draw a comparative analysis of program participants’ sentiment relative to the rest of the employee population. These statistically backed reviews measure the number of proteges transitioning into leadership roles, participant survey feedback regarding coaching and sponsorship support, and the overall impact diversity initiatives have on organizational culture.

According to participant feedback, 84% of Cloudera’s Sponsorship Program graduates believe that people from all backgrounds have equal opportunities to succeed at Cloudera, while 83% feel like a valued member of the organization. Regular assessments and feedback mechanisms ensure that the program remains responsive to participants evolving needs while also aligning with Cloudera’s commitment to promoting DE&I initiatives.

Cloudera is honored to be recognized as a Finalist in the Mentoring Program category for the Ragan 2024 CSR & Diversity Awards. The Company remains committed to advancing the Sponsorship Program and to providing mentorship opportunities for employees from diverse backgrounds.

To learn more about Cloudera’s DE&I initiatives, click here.

The post DEI-focused Cloudera Sponsorship Program Named Finalist for Ragan Awards 2024 CSR & Diversity Awards appeared first on Cloudera Blog.

View Details

It’s nearing the end of the summer in North America, and one report has been a staple on my reading list for more than a decade: the Flexera State of the Cloud Report. The annual survey of hundreds of global IT decision makers assesses cloud strategies, migration trends, and important considerations for companies moving to the cloud or managing cloud environments. As a long-running report, it’s also a valuable resource for understanding the evolution of cloud strategies and priorities over time, with trend data that shows how strategies and priorities have also evolved over time. I’ve referenced the latest iteration of the report dozens of times since its inception.

The 2024 edition of the Flexera State of the Cloud report was released in March and, as usual, it serves as a fantastic resource for data, analytics, and AI leaders as they consider the infrastructure and platform options for their architecture. Here are a few key takeaways from the report:

Managing Cloud Spend Remains a top Challenge – Even Overtaking SecurityOne of the biggest surprises of the 2023 report was that managing cloud spend overtook security as the top challenge for organizations for the first time in 11 years. Cloud spend remained on top for the second year in a row, with public cloud spend exceeding budgets by an average of 15%. The economic uncertainty that many companies have faced in the past two years has exacerbated cost overruns, and most data teams should expect greater scrutiny over their public cloud consumption.

There are two ways to combat the high costs of public clouds. The first is to architect for hybrid deployments. The cloud is ideal for workloads with intermittent or burst capacity requirements, like training AI models. But companies can save money by running other workloads with predictable resource requirements on-premises. The other way is to be strategic about where and when to leverage SaaS platforms, which take control over workload tuning and resource allocation out of the user’s hands in favor of ease of use. That trade-off is not always necessary or ideal.

Increased Adoption of Multi-Cloud StrategiesMulti-cloud strategies continue to dominate. 89% of respondents report using multiple clouds, up from 87% in 2023. The dominance of multi-cloud is the result of the increased parity in functionality and ecosystems between the hyperscalers, as well as the desire to avoid lock-in with any individual cloud provider.

Data teams operating in multi-cloud environments must make some critical architectural decisions. The first is to leverage open formats, including Apache Parquet at the file level and Apache Iceberg at the table level, to ensure that data is both transferable between clouds and interoperable with a wide range of tools for different use cases. The second and perhaps the most critical component of multi-cloud architectures is unified security and governance across the entire data estate, so sensitive data is always protected and data consumers have access to a consistent and accurate view of the data wherever it lives.

A Lot of Data Will Remain On-PremisesMany organizations still prefer to keep sensitive data on-premises, including consumer data, corporate financial data intellectual property, research data, and more, while the majority of non-sensitive data is destined for the public cloud. This result is intuitive for anyone who has spent time talking to customers in highly regulated industries like banking and healthcare, but it can serve as a healthy sanity check for customers who are feeling pressure to migrate. The reality is that, despite a lot of cloud hype, less than 20% of all companies who participated in the survey plan to move their sensitive data to the cloud.

Cloudera Customers Have an AdvantageThe good news for Cloudera customers, as they consider their cloud strategy, is that it doesn’t really matter whether they plan to migrate sensitive data to a public cloud or leave it on-premises. As the only true hybrid platform for data, analytics, and AI, Cloudera enables customers to freely choose any infrastructure for their data analytics workloads, and that data remains in open formats and available for a wide range of workloads, from data engineering to Business Intelligence to AI and ML. It’s portable, meaning that if infrastructure requirements change, it’s easy to move. It’s interoperable, so data teams and data consumers can choose the best tool or execution engine on a workload-by-workload basis. And it’s protected by a unified governance and security solution, so customers can rest assured that whether their data is in the cloud or on-premises, it’s safe and accessible only by the right users.

Try Cloudera TodayCloudera is available for AWS customers to try today. Deploy one of three use case patterns and get your hands on the platform that can dramatically simplify and accelerate your cloud journey.

The post Add Flexera’s State of the Cloud Report to Your Summer Reading List appeared first on Cloudera Blog.

View Details

In the heart of India’s tech hub, Bangalore, you’ll find our Center of Excellence (CoE), an innovation hub focused on technological advancement. Established in 2014, this center has become a cornerstone of Cloudera’s global strategy, playing a pivotal role in driving the company’s three growth pillars: accelerating enterprise AI, delivering a truly hybrid platform, and enabling modern data architectures. It is an engine of cutting-edge solutions that keep us close to our open source roots, and drive our mission of making data and analytics accessible and easy for everyone.

The Indian Talent ProwessOver nearly a decade, the Bangalore CoE has grown into a robust hub, housing over 600 employees, primarily in engineering roles, which form the backbone of product innovation and customer support together with the team in Chennai. The other teams responsible for delivering these products include performance engineering, data engineering, storage, control plane, test infrastructure, security, performance, partner certification, release engineering, and site reliability. This diverse range of expertise ensures that our solutions are comprehensive and of the highest quality to support the data journeys of top enterprises globally.

We operate three major engineering centers worldwide in the US, India, and Hungary. This strategic distribution allows Cloudera to drive continuous innovation and provide timely support to its global customer base. Within India, the engineering expertise in Chennai and Bangalore play a strategic role to global growth & progress due to its rich talent pool and historical contributions to open-source projects like Iceberg. The center’s prominence in the open-source community has made it a magnet for top talent.

Our team in Bangalore is instrumental in providing comprehensive cloud support for end-to-end delivery of projects, keeping major releases up to date with the latest features, as well as innovations including features in Cloudera Machine Learning (CML) that empower customers to develop, test, train, and deploy models within their data environments. The team plays a crucial role in delivering features and enhancements through designing and building solutions that accelerate the journey from exploration to production, and scale their Machine Learning workloads.These work toward supporting Generative AI applications for enterprises in a hybrid cloud environment, ensuring that all their data is AI-ready.

Forging ahead with innovations in machine learning and cloud computing, the team has refined flows for Generative AI workloads and developed an alert system for Cloud, which enables customers to achieve significant savings in cloud costs. With the team propelling these advancements, Cloudera retains its foothold as a leader providing cutting-edge machine learning and data science solutions worldwide.

Local Innovation with Global ImpactThe impact of our Bangalore CoE extends beyond the borders of India. The team works on global projects that drive significant growth across industries, such as finance, telcos and manufacturing. Solutions engineered here are designed with a global perspective in mind, ensuring they meet the diverse needs of our global and Indian customers.

Cloudera is assisting leading banks, stock exchanges, and top organizations across industries in India by providing the infrastructure needed to analyze the data. This support helps them in cost reduction, increased revenue, and enhanced operational efficiency. These sectors frequently handle large volumes of sensitive data demanding robust security and there is a growing need to manage large and complex data sets with better strategies and infrastructure.

We believe that data can make what is impossible today, possible tomorrow. Our Bangalore and Chennai CoE play a key role in empowering organizations to unlock the full potential of their data. By developing robust and secure data platforms, our team ensures businesses can derive meaningful insights that drive a competitive advantage. From enhancing data governance and security to optimizing data workflows and enabling real-time analytics, our team’s contributions are instrumental in shaping the future of data-driven enterprises.

The center’s ability to handle all facets of software development from ideation to delivery ensures that innovation is continuous. 600+ engineers with a passion for open source technologies, have helped accelerate the enterprise AI and machine learning story for hundreds of customers — across a series of generative AI use cases such as agentic applications on our lakehouse, co-pilots for increased productivity, and text summarization at scale. The unique collocation has also helped customers move to the cloud faster and deploy secure hybrid solutions that empower them to transform data of all types on any public or private cloud into valuable and trusted insights.

Looking ahead, Cloudera plans to continue its investments in the Bangalore CoE, with plans to hire an additional 50 engineers this year alone. The focus will be on developing talent to fuel enterprise AI and hybrid platform capabilities. Despite challenges in finding niche skill sets, Cloudera’s commitment to upskilling and providing growth opportunities remains steadfast.

As Cloudera continues to evolve, the Bangalore CoE will undoubtedly remain at the forefront, pushing the boundaries of what’s possible in the world of big data and AI.

The post Cloudera’s Bangalore Center of Excellence – Local Innovation Driving Global Impact appeared first on Cloudera Blog.

View Details

The CRN Tech Innovator Awards spotlight innovative products and services across 36 categories, with winners chosen by CRN staff from over 320 product applications. This year, we’re excited to share that Cloudera’s Open Data Lakehouse 7.1.9 release was named a finalist under the category of Business Intelligence and Data Analytics.

These awards, held annually, are intended to help solution providers identify IT products and services that are truly innovative and deliver customer value. The Awards showcase IT vendor offerings that provide significant technology advances – and partner growth opportunities – across technology categories including AI and AI infrastructure, cloud management tools, IT infrastructure and monitoring, networking, data storage, and cybersecurity.

Embracing the Open Data LakehouseThe selection of Cloudera’s Open Data Lakehouse signals just how important this platform has become with the rise of artificial intelligence (AI) and generative AI (GenAI) alike.

AI is at the forefront of nearly every business’ list of priorities. Its potential is huge when it comes to carving out a competitive edge or just boosting operational efficiency. But even as adoption continues to accelerate, many organizations find themselves struggling with how to fully tap into the power of AI.

The root of the problem comes down to trusted data. Pockets and siloes of disparate data can accumulate across an enterprise or legacy data warehouses may not be equipped to properly manage a sea of structured and unstructured data at scale. Successful AI implementations require businesses to adequately access and collect, often disparate and siloed, data across hybrid environments.

The latest release of the Open Data Lakehouse on private cloud brings a number of features, expanded support, and capabilities that can make managing that data easier. The latest platform release includes Apache Iceberg support to unlock opportunities for enterprises to apply mission-critical data to AI and address the most error-prone processes, generate new use cases, improve overall performance, and reduce costs.

The platform is truly unique, also offering critical capabilities like zero downtime upgrades and security enhancements to limit disruptions and improve business continuity. Additionally, this release of Open Data Lakehouse includes a mix of Apache Ozone capabilities, like quotas, snapshots, and disaster recovery enhancements. Open Data Lakehouse also offers expanded support for Python 3.10 and RHEL 9.1, all of which add another layer of compatibility and flexibility.

Driving AI Innovation on the Open Data LakehouseWe’re incredibly honored and excited to be named a finalist among hundreds of innovative products and solutions that were considered for CRN’s Tech Innovator Awards. Many businesses understand the need for effective AI and analytics within their own operations but struggle with getting their data architectures to support it.

Cloudera’s platform offers portable, cloud-native, analytics deployable across infrastructures, all while maintaining consistent data governance and security. Leveraging a modern data architecture like the Open Data Lakehouse for private cloud helps cut through that complexity to deliver meaningful insights and truly effective AI, at scale.

Learn more about how Cloudera’s Open Data Lakehouse for private cloud can help fuel your AI journey.

The post Cloudera Open Data Lakehouse Named a Finalist in the CRN Tech Innovator Awards appeared first on Cloudera Blog.

View Details

By now, every organization, regardless of industry, has at least explored the use of AI, if not already embraced it. In today’s market, the AI imperative is firmly here, and failing to act quickly could mean getting left behind. But even as adoption soars, struggles remain, and scalability continues to be a major issue. Organizations are quick to adopt AI, but getting it established across the organization brings a unique set of challenges that come into play.

Whether it’s rapidly rising costs, an inefficient and outdated data infrastructure, or serious gaps in data governance, there are myriad reasons why organizations are struggling to move past adoption and achieve AI at scale in their enterprises.

But with the right technology partner, businesses can accelerate their adoption and maximize the value of both their own data and the AI outputs it can generate. Let’s take a closer look at what they face and how Cloudera is uniquely positioned to help them find success.

A New Set of ChallengesGetting up and running with AI is not always straightforward. Anytime a technology is integrated into a business, there’s potential for a new set of challenges to take hold. At the core of many of those is the issue of trust, specifically trusted data. Trusted data is what makes the outputs of AI not just accurate, but impactful in decision making. Ensuring data is trustworthy comes with its own complications.

Cloudera’s State of Enterprise AI and Modern Data Architecture survey identified several challenges when it comes to data. Among those challenges, survey respondents identified merely having too much data (35%) and good governance (36%) as serious obstacles. Trust represents a unique challenge for business leaders—the insights gathered from data are only useful if those leaders know they can trust it. Essentially, if the business data that’s fed into AI models is bad, the resulting insights that come from it will be flawed as well. As data volume grows, data silos proliferate, making it harder for leadership to manage their collective data estates. That lack of knowledge also directly feeds into the problem of governance, as those gaps leave room for data to be misused or mishandled.

Businesses require a modern data architecture that is ready to support the needs of a contemporary business. With this architecture, businesses can build in greater flexibility and scalability in their existing infrastructure to support the rise of AI. These architectures enable enterprises to future-proof their AI models, drive innovation, and stand out in competitive markets.

Cloudera is Your Trusted AI PartnerGetting past these challenges and successfully tapping into the power of AI requires businesses to work with a technology partner proven to deliver crucial data and analytics solutions for hybrid cloud, industry-leading AI expertise, and a strong foundation to support and future-proof AI investments.

With Cloudera, organizations can build a data infrastructure that is flexible and scalable enough to ensure that as the use of AI grows, the data that fuels it will keep pace. Cloudera ensures data governance is robust to protect data as it’s used in AI models and keep that use in line with internal standards and external regulations.

Likewise, Cloudera’s open data lakehouse presents a standout option for organizations to leverage as a foundational part of their data infrastructure. This platform brings together the flexibility of data lakes with the power of a data warehouse all in one place. The open data lakehouse is critical for organizations looking to harness AI, helping run analytics on data—structured and unstructured—at scale. It’s tailor-made for the data challenges that often hinder AI adoption. Particularly as organizations are inundated with more and more data, an open data lakehouse serves as a strong foundation to help them keep pace. With Cloudera and the only true hybrid platform for data, analytics, and AI, organizations can eliminate data silos and empower data teams to collaborate on the same data with the tools of their choice on any public cloud and private cloud.

Ultimately, every AI journey will encounter a bump in the road. But there’s no need to face those challenges alone. Cloudera brings a wide set of technology, solutions, and know-how to demystify the process and ensure that AI is implemented securely, effectively, and simply enough to rapidly scale and generate maximum business value.

Read the full survey report and learn how Cloudera can help accelerate your AI journey.

The post AI Challenges and How Cloudera Can Help appeared first on Cloudera Blog.

View Details

Data practitioners are consistently asked to deliver more with less, and although most executives recognize the value of innovating with data, the reality is that most data teams spend the majority of their time responding to support tickets for data access, performance and troubleshooting, and other mundane activities. At the heart of this backlog of requests is this: data is hard to work with, and it’s made even harder when users need to work to get or find what they need.

As a long-time partner to some of the largest enterprises in the world, we recognize the critical role Cloudera plays in making data teams and data consumers successful in their day-to-day work. That’s why we are rolling out a significant update to the Cloudera platform homepage, including a new set of features we designed to provide a more intuitive and efficient experience for data practitioners.

The decision to revamp the Cloudera UI was driven by our commitment to enhancing user experience and addressing the evolving needs of our customers. We have always listened closely to our users and tailored our solutions to meet their specific requirements. Over the years, our platform has grown in capability, offering a diverse range of services and tools. However, while the traditional tile-based homepage was functional, it did not fully support the intuitive navigation and quick access to information that our users require. This UI improvement ensures that our platform remains at the forefront of user-friendly design, making it easier for users to take advantage of the full potential of our services.

By implementing these changes, our goal is to create a more cohesive, intuitive, and efficient user interface that simplifies navigation, enhances the discoverability of features within the platform, and improves overall user productivity. With much quicker access to frequently used tools and services, integrated analytics for quick insights, comprehensive guides for exploring new solutions, and a powerful search function, users can now navigate the platform with greater ease and efficiency.

Key Enhancements: Streamlined Navigation: + Navigating through multiple pages to access frequently-used services and workspaces has historically been a time-consuming task for our users. The new Favorites* feature addresses this problem by enabling users to bookmark their most frequently used Data Hubs, services, and workspaces, making them available wherever you open Cloudera. It ensures that users can quickly reach the tools and services they rely on the most, making their daily operations smoother and more efficient. + Additionally, several changes have been made to streamline navigation across the platform. These enhancements reduce the number of clicks required, ensuring users can quickly reach the tools and services they are looking for.

• Analytics Summary:

  • It is essential for data teams to quickly access and understand the health and performance of their data services. The Analytics Summary section addresses this need by integrating key Observability Dashboard metrics directly into the homepage. Users can now view a summary of analytics for individual clusters and virtual warehouses without navigating away from the main page. This section enables users to select and display operational insights for specific services, such as Data Hub, Data Engineering, and Data Warehouse, providing immediate insights into their operations. Bringing these metrics to the homepage helps users monitor performance and make data-driven decisions more effectively.

• Solution Explorer:

  • The Solution Explorer section is designed to help users discover and explore new innovations within the Cloudera platform tailored to specific roles and needs. It offers a comprehensive guide to various solutions, including Enterprise AI, Open Data Lakehouse, Scalable Data Mesh, Unified Data Fabric, and Hybrid Data Platform, with more to come! This section provides detailed descriptions and relevant documentation for users whose roles benefit from these capabilities, such as data scientists, analytics professionals, and database administrators. By centralizing information about Cloudera’s latest innovations, the Solution Explorer makes it easier for users to stay informed and leverage the full potential of the platform’s capabilities.

• Quick Start:

  • The Quick Start section is designed to help users perform essential data tasks with ease and efficiency. This section offers a step-by-step guide for common activities, such as connecting to or importing data, querying and transforming data, and visualizing data. Each guide includes links that take users directly to the relevant sections in Cloudera’s documentation, providing a supportive experience for both new and experienced users.

• Documentation Search:

  • The new Documentation Search feature adds a convenient search bar at the top of the homepage, enabling users to quickly find the information they need within Cloudera’s extensive documentation. Whether you need guidance on a specific feature, troubleshooting tips, or detailed technical documentation, the Documentation Search makes it easy to access the comprehensive resources available.

We’re rolling out this new UI gradually, and we’d love for you to try it out. You can enable the new homepage by clicking the “Enable New UI” toggle button in the top-right navigation bar. We know many of our customers are comfortable using the current UI, and that’s fine! You can opt in or opt out with the click of a button.

This homepage update is just the beginning. The new UI is designed to expand beyond the homepage, and we will gradually integrate it into individual services across the platform. Our vision is to make it easier than ever for our customers to deliver actionable insights to the business by providing the most intuitive and user-friendly experience for working with data.

We encourage you to explore the new homepage and share your thoughts. Your feedback is crucial in helping us refine and improve the Cloudera experience for all users.

The post Navigating the Future with Cloudera’s Updated Interface appeared first on Cloudera Blog.

View Details

Cloudera is excited to announce a partnership with Allitix, a leading IT consultancy specializing in connected planning and predictive modeling. This collaboration is set to enhance Allitix’s offerings by leveraging Cloudera’s secure, open data lakehouse, empowering enterprises to scale advanced predictive models and data-driven solutions across their environments.

How Cloudera and Allitix Fit TogetherAllitix will now leverage Cloudera’s open data lakehouse to help its enterprise clients eliminate data silos and integrate plans across functions through connected planning. This facilitates improved collaboration across departments via data virtualization, which allows users to view and analyze data without needing to move or replicate it. Cloudera’s data lakehouse provides enterprise users with access to structured, semi-structured, and unstructured data, enabling them to analyze, refine, and store various data types, including text, images, audio, video, system logs, and more. Allitix enterprise clients will also benefit from the enhanced data security, data governance, and data management capabilities offered with Cloudera’s open data lakehouse.

Through this strategic partnership, Allitix applications will enable business users to more easily work with data in the lakehouse, collaborate across functions with this data, and use it to build advanced predictive models, giving its end customers a competitive edge.

“Allitix constantly seeks to do more for our customers, and our extensive search showed that Cloudera is best in class to service our clients’ end-to-end data needs,” said Jon Pause, Practice Director for Data and Advanced Tools at Allitix. “We love Cloudera’s hybrid model, coding portability, and open-source AI approach. And through this partnership, we can offer clients cost-effective AI models and well-governed datasets as this industry charges into the future.”

Allitix will leverage Cloudera’s open data lakehouse to support its connected planning solutions for enterprise clients and partners across various markets, including regulated industries such as finance, healthcare, pharmaceuticals, and consumer packaged goods. This will enable these clients and partners to make more informed strategic decisions regarding marketing, operations, customer success, overall business strategy, and more.

“This partnership is a significant win for enterprise customers,” said Andy Moller, SVP of Global Alliances and Ecosystem at Cloudera. “With Cloudera and Allitix, they can develop complex predictive data models to make crucial business decisions. These large, regulated organizations depend heavily on data management and security. This strategic partnership strengthens our connection with business users through Allitix solutions and extends our technology into new markets.”

Data-backed Decisions Through Predictive ModelsPredictive models use historical data and analytics to forecast future outcomes through mathematical processes. They help organizations allocate resources appropriately, anticipate potential challenges, and identify market trends. This capability is crucial for enterprises to make more informed financial and resource allocation decisions. Cloudera’s open data lakehouse will enable Allitix customers to build more comprehensive predictive models, leading to faster, data-driven decision making.

“Cloudera’s open data lakehouse is the core functionality that enables self-service analytics, governance, and cost-cutting architecture,” Pause said. “Through this partnership, our customers will benefit from more democratized data reducing risk to all downstream projects while significantly cutting their variable IT spend.”

Cloudera’s open data lakehouse unlocks the power of enterprise data across private and public cloud environments. It streamlines data ingestion, analytics, and the development of AI and ML models, allowing enterprises to feed their data into a large language model (LLM) to build advanced AI applications in a secure governed environment. The Cloudera platform allows businesses to build AI applications hosted by any open-source LLM they choose, allowing for scaling across an enterprise for a variety of users and data sets.

“Cloudera partners with the world’s most innovative companies – across industries – to bring our leading trusted data management platform to organizations leveraging the technologies of tomorrow,” Moller said. “This strategic partnership with Allitix will empower enterprises to harness our world-class data management platform, driving innovation, operational excellence, and competitive advantages.”

Learn more about how you can partner with Cloudera.

The post Cloudera Partners with Allitix to Fuel Enterprise Connected Planning Solutions appeared first on Cloudera Blog.

View Details

Over the past several years, data leaders asked many questions about where they should keep their data and what architecture they should implement to serve an incredible breadth of analytic use cases. Vendors with proprietary formats and query engines made their pitches, and over the years the market listened, and data leaders made their decisions.

The most interesting thing about their choices is that, despite the millions of marketing dollars vendors spent trying to convince customers that they built the next greatest data platform, there has been no clear winner.

Many companies adopted the public cloud, but very few organizations will ever move everything to the cloud, or to a single cloud. The future for most data teams will be multi-cloud and hybrid. And although there is clear momentum behind the data lakehouse as the ideal architecture for multi-function analytics, the demand for open table formats including Apache Iceberg is a clear signal that data leaders value interoperability and engine freedom. It no longer matters where the data is. What matters is how we understand it and make it available to share, and use.

The direction is clear. Proprietary formats and vendor lock-in are a thing of the past. Open data is the future. And for that future to be a reality, data teams must shift their attention to metadata, the new turf war for data.

The need for unified metadataWhile open and distributed architectures offer many benefits, they come with their own set of challenges. As companies seek to deliver a unified view of their entire data estate for analytics and AI, data teams are under pressure to:

• Make data easily consumable, discoverable, and useful to a wide range of technical and non-technical data consumers
• Improve the accuracy, consistency, and quality of data
• Ensure the efficient querying of data, including high availability, high performance, and interoperability with multiple execution engines
• Apply consistent security and governance policies across their architecture
• Achieve high performance while managing costs

The answer to unifying the data has traditionally been to move or copy data from one source or system to another. The problem with that approach is that data copies and data movement actually undermine all five of the points above, increasing costs while making it more difficult to manage and trust the data as well as the insights derived from it.

This leads us to a new frontier of data management, which is especially critical for teams managing distributed architectures. Unifying the data isn’t enough. Data teams actually need to unify the metadata.

There are two types of metadata, and they both serve critical functions within the data lifecycle:

Operational metadata supports the data team’s goals of securing, governing, processing, and exposing the data to the right data consumers while also keeping queries against that data performant. Data teams manage this metadata with a metastore.

Business metadata is metadata that supports data consumers who want to discover and leverage that data for a broad range of analytics. It provides context so users can easily find, access, and analyze the data they’re looking for. Business metadata is managed with a data catalog.

Many solutions manage at least one of these types of metadata well. A few solutions manage both. However, there are very few platforms that can unify and manage business and operational metadata from on-premises and cloud environments as well as metadata from multiple disparate tools and systems. Additionally, almost none of the available tools do all of that and also provide the automation required to scale these solutions for enterprise environments.

Cloudera is built on open metadataCloudera’s open data lakehouse is built on Apache Iceberg, which makes it easy to manage operational metadata. Iceberg maintains the metadata within the table itself, eliminating the need for metadata lookups during query planning and simplifying formerly complex data management tasks like partition and schema evolution. With Cloudera’s open data lakehouse, data teams store and manage a single physical copy of their data, eliminating additional data movement and data copies and ensuring a consistent and accurate view of their data for every data consumer and analytic use case.

Cloudera also supports the REST catalog specification for Iceberg, ensuring that table metadata is always open and easily accessible by third-party execution engines and tools. While a lot of vendors are focused on locking in metadata, Cloudera remains cloud- and tool-agnostic to ensure customers continue to have the freedom to choose.

Cloudera is also working on accessing and tracking metadata outside of the Cloudera ecosystem, so data teams will have visibility across their entire data estate, including data stored in a variety of other platforms and solutions.

Automating business metadata is the key to achieving scaleWhile operational metadata is often generated by a system and maintained within Iceberg tables, business metadata is often generated by domain experts or data teams. In an enterprise environment, which often features hundreds or even thousands of data sources, files, and tables, scaling the human effort required to ensure these datasets are easily discoverable is impossible.

Cloudera’s vision is to augment the data catalog experience and remove the manual effort of generating business metadata. Customers will be able to leverage Generative AI to ensure that every dataset is properly tagged and classified, and is easily discoverable. With an automated business metadata solution, data consumers and data teams can easily find the data they’re looking for, even with huge catalogs, and no dataset will fall through the cracks.

Unified security and governanceData teams strive to balance the need for broad access to data for every data consumer with centralized security and governance. That task becomes much more complicated in distributed environments, and in situations where the data moves from its source to another destination.

Cloudera Shared Data Experience (SDX) is an integrated set of security and governance technologies for tracking metadata across distributed environments. It ensures that access control and security policies that are set once still apply wherever and however that data is accessed, so data teams know that only the right data consumers have access to the right datasets, and the most sensitive data is protected. Unlike decentralized and siloed data systems, having a centralized and trusted security management layer makes it easier to democratize data with the confidence that nobody will have unauthorized access to data. From a governance perspective, data teams have control over and visibility into the health of their data pipelines, the quality of their data products, and the performance of their execution engines.

The metadata turf wars have just begunAs data teams adopt hybrid, distributed data architectures, managing metadata is critical to providing a unified self-service view of the data, to delivering analytic insights that data consumers trust, and to ensuring security and governance across the entire data estate.

Chief Data Analytics Officers can take some important lessons from the data wars onto this new battlefield:

1. Choose open metadata: Don’t lock your metadata into a single solution or platform. Iceberg is a great tool for ensuring openness and interoperability with a large commercial and open source software ecosystem.
2. Unify metadata management: Invest in a metadata management solution that unifies operational and business metadata across all environments and systems, even third-party tools and platforms.
3. Automation and Scalability: Leverage automation to handle the scale and complexity of creating and managing metadata in large, distributed environments.
4. Centralized Security and Governance: Ensure that security and governance policies are consistently applied and enforced across the entire data landscape to protect sensitive data and ensure the health and performance of your data estate.

These are the guiding principles of Cloudera’s metadata management solutions, and why Cloudera is uniquely positioned to support an open metadata strategy across distributed enterprise environments.

Learn more about Cloudera’s metadata management solutions here.

The post The Data Turf Wars are Over, But the Metadata Turf Wars Have Just Begun appeared first on Cloudera Blog.

View Details

In this Employee Spotlight, we sat down with Stephanie Han to learn about her tenure at Cloudera, her journey from accounting to leading diversity, equality & inclusion (DEI) programs, and her impressive volunteer work.

Meet Stephanie Han Stephanie is a Senior Program Manager in the HR team at Cloudera. She’s been with the company since 2019 and plays a key role in a variety of employee-centric initiatives including Cloudera’s employee volunteering program, talent management program, and both its DEI and environmental, social, and governance (ESG) efforts. She exudes enthusiasm about all the work she’s involved in, is thankful for the variety of initiatives she gets to work on and the diverse perspectives she works alongside.

“Building a diverse team is so important because it allows us to learn from one another,” Stephanie said. “That is very true with my team. We all come from different backgrounds – both professional and lived experiences. It’s been so amazing to be able to grow with my team and learn from our different ways of thinking, ways of doing, and ways of being.”

Making the Switch from Accounting to HR – A Unique Opportunity at Cloudera When Stephanie first joined Cloudera, she found herself in a role doing what she had always done – finance and accounting. During her time as senior operations revenue manager, Stephanie worked on revenue close processes, financial reporting, and more. However, after she started managing employees, she found her passion for professional development and the opportunity to help teammates grow and advance their careers. She wanted to do more than just help “close the books,” and this inspired her to seek a career change to HR.

She knew switching from a 12-year accounting career to HR would be difficult, but she remained persistent, conducting as many informal interviews with HR professionals as she could. Fortunately, those calls led her back to Cloudera.

“The right opportunity at the right time came up,” Stephanie said. “All the skill sets I gained in my previous role in accounting and revenue operations helped me to seamlessly hit the ground running in HR and program management. I’m a big believer that building skill sets is more important than actual job titles, and I was thankful that the leaders at Cloudera recognized that too.”

Stephanie took a consultancy position on Cloudera’s DEI team in October 2020 before earning a full-time role in April 2021. The team later became part of the HR department and Stephanie welcomed the opportunity to further integrate into areas of interest.

Growing Cloudera’s DEI Programs After joining Cloudera’s DEI team, Stephanie helped to build the company’s Teen Accelerator Program, which has a partnership with the Boys and Girls Club of America in both Tennessee and the San Francisco Bay Area. This program facilitates opportunities in corporate America for high school students in under-resourced communities. The Teen Accelerator program offers students a six-week paid internship program at Cloudera and 1:1 employee mentorship. Managing this group of employee mentors inspired Stephanie to begin volunteering alongside colleagues as well.

In 2022, Stephanie joined Cloudera’s Ambassador Network, the company’s global network of philanthropic champions, dedicated to planning and executing giving and volunteering initiatives. After serving in this role for two years, Stephanie expressed interest and subsequently expanded her professional responsibilities to include the management of this critical culture initiative. Under Stephanie’s management, the Ambassador Network is wrapping up its inaugural Summer of Service campaign, which supplements Cloudera’s existing annual Week of Giving campaign. Both initiatives consist of philanthropic events and global volunteer opportunities.

“It was somuch fun working with my team to plan our first Summer of Service and seeing Clouderans so excited to plan events giving back to their local communities and beyond,” Stephanie said.

Her interest in expanding her expertise and impact did not stop there. Now, Stephanie’s scope also includes Cloudera’s ESG and talent management programs as well.

“I feel very privileged to be a part of this team,” Stephanie said. “Volunteering as an ambassador for several years was really rewarding. It’s really unique to now have the opportunity to expand my scope and work on this initiative – as well as others of interest, like ESG and Talent Management – as part of my formal responsibilities too. Having the support of our HR Leadership to continue to build my experience and expertise is invaluable.”

One of the other areas of interest Stephanie has taken on is talent management. She serves as a member of the talent management team, which is made up of employees from various HR groups who are incredibly passionate about helping people grow their careers. The team works on succession planning, development plans, and maps out ways to help employees grow within Cloudera.

“Talent management is an area I’m grateful for during my time at Cloudera,” Stephanie said. “It’s an area I took great interest in before this role, and it’s one of the reasons I pursued a career change. I wanted to work with people and help them improve and succeed professionally, and now I get to do that through various unique and impactful initiatives at Cloudera.”

Empowering Professional Development Around the Globe After the 2015 earthquake devastated much of Nepal, Stephanie, who lives in the Bay Area, visited the country to see the landscape, talk to locals, and try to address widespread needs. Now, Stephanie devotes a great amount of her free time to helping people in Nepal through group volunteer work. Recently, she helped start the Institute of Higher Learning (IHL), which aims to equip the Nepali people with skills to help them pursue a better life.

“We realized there’s very little that wecan do living halfway across the world, but one way we could really make an impact is by setting up a learning institute,” Stephanie said. “There’s a wealth of resources here in the Bay Area and many people are trained in various transferable fields including IT and Engineering. Nepal is quickly developing, so there will be jobs needed in the IT space, for example. We have people in my group who come from various professional backgrounds, and it is our mission to teach and mentor students to help open the door to greater economic opportunities.”

The institute is starting its first English class and hopes to launch technology classes and coding camps in the future.

Following Passions of Helping People Grow Career changes are not always easy, but after a 12-year career in accounting, Stephanie managed to find a role she was truly passionate about – empowering others to grow and thrive at work while bringing people together through DEI and HR initiatives. In parallel, she continues to focus her free time on her philanthropic work to help students in Nepal and appreciates the ability to weave that passion for impact into her work here at Cloudera.

Read our last employee spotlight here.

The post #ClouderaLife Employee Spotlight: Stephanie Han appeared first on Cloudera Blog.

View Details

Enterprise IT leaders across industries are tasked with preparing their organizations for the technologies of the future – which is no simple task. With the use of AI exploding, Cloudera, in partnership with Researchscape, surveyed 600 IT leaders who work at companies with over 1,000 employees in the U.S., EMEA and APAC regions. The survey, ‘The State of Enterprise AI and Modern Data Architecture’ uncovered the challenges and barriers that exist with AI adoption, current enterprise AI deployment plans, and the state of data infrastructures and data management.

The State of Enterprise AIIt will likely come as little surprise that businesses across the world are swiftly incorporating AI into their operations, with 88% of surveyed companies already utilizing this transformative technology. AI is starting to revolutionize industries by changing how a business operates and the teams within. The departments leading this adoption are IT (92%), Customer Service (52%), and Marketing (45%). Across these business areas, AI is enhancing efficiency in IT processes, improving customer support with chatbots, and leveraging analytics for better decision-making.

Among various AI implementations, Generative AI (GenAI) stands out as the most popular, with 67% of respondents utilizing generative models in some capacity. Companies are deploying GenAI using several architectures: exposing data to open-source models without training on it (60%), training open-source models on their data (57%), using open-source models trained on-premises or in private clouds (50%), and developing proprietary Large Language Models (LLMs) or Small Language Models (26%).

In addition to GenAI, respondents noted they are deploying predictive (50%), deep learning (45%), classification (36%) and supervised learning (35%) applications.

Challenges in Implementing AIImplementing AI does not come without challenges for many organizations, primarily due to outdated or inadequate data infrastructures. While every business has adopted some form of data architecture, the types they use vary widely. The majority of organizations store their data in private clouds (81%), but other architectures are also prevalent, including public clouds (58%), on-premises mainframes (42%), on-premises distributed systems (31%), other physical environments (29%), and data lakehouses (19%).

Navigating the complexity of modern data landscapes brings its own set of challenges. Key issues include data security and reliability (66%), escalating data management costs (48%), compliance and governance challenges (38%), overly complex processes (37%), siloed and difficult-to-access data (36%), mistrust in connecting private data and inaccuracies in AI models (32%), and the need for standardized data formats (29%).

Adding to these complexities is the rapidly evolving nature of data technologies and the growing volume of data businesses must manage. Ensuring that AI implementations are effective and secure requires continuous adaptation and investment in robust, scalable data infrastructures. This is essential for businesses aiming to leverage AI for competitive advantage and operational efficiency.

Leveraging Modern Data ArchitecturesIn today’s landscape, the only way to ensure data reliability is through the adoption of modern data architectures. These advanced architectures provide critical flexibility and visibility, acting as a blueprint for accelerating the extraction of insights and value from data. They simplify data access across organizations, breaking down silos and making data easier to understand and act upon.

When asked about the most valuable advantages of hybrid data architectures, respondents highlighted data security (71%) as the primary benefit. Other significant advantages include improved data analytics (59%), enhanced data management (58%), scalability (53%), cost efficiency (52%), flexibility (51%), and compliance (37%).

Modern data architectures support the integration of diverse data sources and formats, providing a cohesive and efficient framework for data operations. This integration is essential for businesses aiming to leverage data-driven strategies, ensuring that their data infrastructure can meet the demands of evolving technologies and increasing data volumes. By adopting these architectures, organizations can position themselves to unlock new opportunities and drive innovation through reliable and accessible data.

The enhanced security, transparency, accessibility, and insights provided by modern data architectures directly contribute to a business’s agility, adaptability, and informed decision-making. These factors are crucial for future-proofing data infrastructure, ensuring it remains robust over time, and achieving tangible ROI from AI implementations.

To gain more insights from Cloudera’s latest survey report, click here.

The post An Overview of Cloudera’s AI Survey: The State of Enterprise AI and Modern Data Architecture appeared first on Cloudera Blog.

View Details

Cloudera’s data-in-motion architecture is a comprehensive set of scalable, modular, re-composable capabilities that help organizations deliver smart automation and real-time data products with maximum efficiency while remaining agile to meet changing business needs. In this blog, we will examine the “why” behind streaming data and review some high-level guidelines for how organizations should build their data-in-motion architecture of the future.

Businesses everywhere seek to be more data-driven not just when it comes to big strategic decisions, but also when it comes to the many low-level operational decisions that must be made every day, every hour, every minute, and, in many cases, every second. The transformative power of incremental improvement at the operational level has been proven many times over. Executing better on the processes that add value to your value chain is bound to reap benefits. Take a hypothetical manufacturer for example. On the shop floor, myriad low-level decisions add up to manufacturing excellence, including:

• Inventory management
• Equipment health and performance monitoring
• Production monitoring
• Quality control
• Supply chain management

It’s no wonder that businesses are working harder than ever to embed data deeper into operations. In 2022, McKinsey imagined the Data-Driven Enterprise of 2025 where winner-takes-all market dynamics incentivizes organizations to pull out all the stops and adopt the virtuous cycle of iterative improvement. It was very telling that, of the seven characteristics highlighted in that piece, the first two are:

• Data should be embedded in every decision, interaction, and process
• Data should be processed and delivered in real time

Notice that McKinsey isn’t talking about how fast data is created. They are talking about data being processed and delivered in real time. It is not the speed at which data is created that determines an organization’s response time to a critical event, it’s how quickly they can execute an end-to-end workflow and deliver processed data that determines their response. A sensor on a machine recording a vibration, on its own, has very little value. What matters is how fast that data can be captured, processed to put that vibration reading within the context of the machine’s health, used to identify an anomaly, and delivered to a person or system that can take action.

Businesses are challenged, however, with transforming legacy architectures to deliver real-time data that is ready for business use. For many organizations, the analytics stack was built to consolidate transactional data in batches, often over multiple steps, to report on Key Performance Indicators (KPIs). They were never built for real-time data, yet they are still the primary means of moving and processing data for most data teams. To achieve this, real-time data must first come to rest and wait to make its way through the stack. By the time it is ready for analysis, it is a historical view of what happened, and the opportunity to act on events in real time has passed, reducing the value of the insights.

The growing number of disparate sources that business analysts and data scientists need access to further complicates efforts. Unfortunately, a lot of enterprise data is underutilized. Underutilized data often leads to lost opportunities as data loses its value, or decays, over time. For example, 50% of organizations admit that their data loses value within hours, and only 26% said their streaming data is analyzed in real time. If an organization is struggling to utilize data before it decays, it fails to fully leverage the high-speed data in which it has invested.

Before we go any further, let’s clarify what data in motion is. Data in motion, simply put, is data that is not at rest, such as data in permanent storage. It includes data that is streaming – a continuous series of discrete events that happen at a point in time, such as sensor readings. It also includes data that is currently moving through an organization’s systems. For example, a record of login attempts being sent from an authentication server to a Security Information and Event Management tool is also data in motion. By contrast, data at rest isn’t doing much besides waiting to be queried. Data in motion is active data that is flowing.

Data-in-motion architecture is about building the scalable data infrastructure required to remove friction that might impede active data from flowing freely across the enterprise. It’s about building strategic capabilities to make real-time data a first-class citizen. Data in motion is much more than just streaming.

Delivering real-time insights at scale with the efficiency and agility needed to compete in today’s business environment requires more than just building streaming pipelines to move high-velocity data into an old analytics stack. The three key elements of a data-in-motion architecture are:

• Scalable data movement is the ability to pre-process data efficiently from any system or device into a real-time stream incrementally as soon as that data is produced. Classic Extract, Transform, & Load (ETL) tools have this functionality, but they typically rely on batching or micro-batching as opposed to moving the data incrementally. Thus, they are not built for true real-time.
• Enterprise stream management is the ability to manage an intermediary that can broker real-time data between any number of “publishing” sources and “subscribing” destinations. This capability is the backbone of building real-time use cases, and it eliminates the need to build sprawling point-to-point connections across the enterprise. Management involves utilizing tools to easily connect publishing and subscribing applications, ensure data quality, route data, and monitor health and performance as streams scale.
• Democratized stream processing is the ability of non-coder domain experts to apply transformations, rules, or business logic to streaming data to identify complex events in real time and trigger automated workflows and/or deliver decision-ready data to users. This capability converts large volumes of raw data into contextualized data that is ready for use in a business process. Domain experts need to have access to inject their knowledge into data before it is distributed across the organization. A traditional analytics stack typically has this functionality spread out over multiple inefficient steps.

To transform business operations with data embedded in every process and decision, a data-in-motion architecture must be able to capture data from any source system, process that data within the context of the processes and decisions that need to be made, and distribute it to any number of destinations in real time. As organizations scale, the benefits of data in motion grow exponentially. The hallmark of an effective data-in-motion architecture is maximal data utilization with minimal latency across the organization. Examples of this include:

• An order flowing across an e-commerce organization to provide real-time updates to marketing, fulfillment, supply chain, finance, and customer service, enabling efficient operations and delighting customers.
• A user session on a telco network flowing across the organization and being utilized by various processes, including fraud detection, network optimization, billing, marketing, and customer service.

With data in motion enabling true real-time, analysts can get fresh, up-to-the-second, processed data ready for analysis, improving the quality of insights and accelerating their time to value.

A data-in-motion architecture delivers these capabilities in a way that makes them independently modifiable. That way, organizations can adopt technology that meets their current needs and continue to build their streaming maturity as they go. It should be easy to do things like onboard a new sensor stream when a manufacturing production line has been retrofitted with sensors by using data movement capabilities to bring data into an existing stream without modifying the entire architecture. We should be able to add new rules to how we manage streaming data without rebuilding connectivity to the source system. Similarly, it should be easy to add new logic into real-time monitoring for cybersecurity threats when we identify a new tactic. As demand for real-time data continues to grow and new data sources and applications come online, it should be effortless to scale up the necessary components independently without compromising the efficient use of resources. The speed with which an enterprise can make changes to the way they capture, process, and distribute data is essential for organizational agility.

Capturing, processing, and distributing real-time data at scale is critical to unlocking new opportunities to drive operational efficiency. The ability to do so at scale is the key to reaping greater economic value. The ability to remain agile is critical to sustaining innovation speed. Additionally, the value of architectural simplicity can not be understated. In a recent paper, Harvard Business School professor and technology researcher Marco Iansiti collaborated with Economist Ruiging Cao to model “Data architecture coherence” and the cascading benefit of sustained innovation speed across an enterprise. A coherent data architecture in Professor Iansiti’s definition is simple to understand and modify, and one that is well aligned with business processes and broader digital transformation goals. Professor Iansiti theorizes that the real driving force behind the innovation speed of many digital natives is not culture as much as it is a coherent data architecture that lends itself well to a rapid iteration approach to business process optimization. Reduction in redundant tools and process steps can be quantified in terms of licensing, resource utilization, personnel impacts, and administrative overhead. However, these benefits are dwarfed by the sustained innovation speed required to execute constant incremental improvements at the operational level that coherent data architectures deliver.

Cloudera’s holistic approach to real-time data is designed to help organizations build a data-in-motion architecture that simplifies legacy processes for data movement as it scales.

Ready to take action? Find out how a data-in-motion architecture can help you improve critical processes and get the most out of your data.

The post What Makes Data-in-Motion Architectures a Must-Have for the Modern Enterprise appeared first on Cloudera Blog.

View Details

Cloudera recently unveiled the latest version of Cloudera Private Cloud Base with the Zero Downtime Upgrade (ZDU) feature to enhance your user experience. The goal of ZDU is to make upgrades simpler for you and your stakeholders by increasing the availability of Cloudera’s services.

How Do You Keep IT Infrastructure (and Buses) Running and Avoid Downtime?Before I dive into the depths of ZDU, let me provide an analogy inspired by a customer. Citizens of large cities heavily depend on their local metro systems to plan their day-to-day lives. People need to get to work, go to the doctor, and get groceries, and it’s up to their local transportation department to ensure they make it to their destinations reliably. Managing IT infrastructure starts to look like a city’s transportation infrastructure when you realize that end users also depend on the reliability of IT systems to complete work and get home on time. IT organizations have the thankless job of ensuring infrastructure is up to date and patched against the latest vulnerabilities while downtime is kept to a minimum. Similarly, transportation agencies reduce downtime through innovations like automatic inflating tire systems; so it’s about time we innovate too. That begs the question – if clusters are like buses, how do I inflate the tires while the bus is en route? Meaning, how do I keep my infrastructure running and avoid downtime?

Enter Zero Downtime UpgradesZDU is an answer to the increased demands on IT infrastructure brought on by internal stakeholders and external customers becoming global. The days when IT infrastructure could be brought down at night or on weekends to apply updates are disappearing. Similar to how a bus needs tune-ups, IT infrastructure needs maintenance to perform major upgrades, apply performance enhancements to scale workloads, or patch vulnerabilities to keep your environments safe. Cloudera helps you with this maintenance by delivering improvements and vulnerability patches in Service Packs and Cumulative Hotfixes (CHFs) . Although applying Service Packs and CHFs is a straightforward process, you do need to restart services. Therefore, Cloudera Private Cloud Base needs to adopt a fundamental change to the upgrade and patch process to reduce and eventually eliminate workload downtime.

ZDU isn’t Cloudera’s first experience with providing the ability to upgrade services with no downtime. Rolling upgrades and restarts have been available in services like HDFS and YARN. This feature, which is still available in Cloudera Private Cloud Base, allows users to restart some of Cloudera’s fundamental services with reduced capacity and no downtime

Diving into Zero Downtime UpgradesWith that context let’s dive into how ZDU in Cloudera Private Cloud Base keeps your end users “on the bus” while performing critical maintenance. ZDU allows platform administrators to perform major upgrades, apply service packs, and cumulative hotfixes with minimum to no downtime. The first innovation of this experience was improving Cloudera Manager’s upgrade process. The service upgrade sequence is optimized to account for service dependencies and to limit the time a service experiences reduced capacity. These optimizations improve upgrade time whether performing a regular full downtime upgrade or the ZDU experience. Next, services are improved to either add the ability to upgrade without downtime or reduce the amount of downtime one may experience.

Let’s talk about what you should expect during a zero downtime upgrade. When initiating an upgrade with Cloudera Manager, you will first be presented with a checklist page to ensure your cluster is ready for an upgrade. After completing the checklist, you can perform a regular or zero downtime upgrade. Once the ZDU begins, the Cloudera Manager will begin to upgrade the services in two stages. First, services that will experience some downtime will be upgraded. This ensures that any service downtime is predictable and is only experienced at the beginning of your upgrade window. Next, Cloudera Manager will perform upgrades on the rest of the services that will experience reduced capacity, but zero downtime. When Cloudera Manager completes the sequence of commands, validation of the cluster by administrators can occur much like a regular upgrade before being finalized. If any issues occur during the process, Cloudera Private Cloud Base now supports downgrades to allow a cluster to adopt the previous version without losing any metadata.

The Cloudera team is passionate about helping you confidently tackle their toughest data and AI challenges. This first step into Zero Downtime Upgrades is a big achievement in providing a revolutionary experience for cluster administration teams. Ultimately our goal is to provide you with the tools to keep the buses rolling and passengers moving so we can all make it home on time.

To learn more, visit our product page.

The post Zero Downtime Upgrades – Redefining Your Platform Upgrade Experience appeared first on Cloudera Blog.

View Details

Late last week, the tech world witnessed a significant disruption caused by a faulty update from CrowdStrike, a cybersecurity software company that focuses on protecting endpoints, cloud workloads, identity, and data. This update led to global IT outages, severely affecting various sectors such as banking, airlines, and healthcare. Many organizations found their systems rendered inoperative, highlighting the critical importance of system resilience and reliability.

However, amidst this disruption, one Cloudera customer reported that although many of their systems were impacted, Cloudera’s data-in-motion stack specifically demonstrated remarkable resilience, experiencing no downtime. Here, we’ll briefly discuss the incident, and how Cloudera protected its customers’ most critical analytic workloads from potential downtime.

The Incident: A Brief OverviewThe CrowdStrike incident, which stemmed from a problematic update to their Falcon platform, caused widespread compatibility issues with Microsoft systems. This resulted in numerous systems experiencing the infamous Windows “blue screen of death” among other operational failures. While this incident did not involve a cyberattack, the technical glitch led to significant disruptions to global operations.

Cloudera’s Resilience – Data in Motion and the Entire Cloudera Data PlatformThe Cloudera customer reported that despite many of their systems going down, Cloudera services running on Linux instances in Amazon Web Services (AWS) remained up and functional. These services included their data-in-motion stack, but it’s important to note that Cloudera’s entire platform and all hybrid cloud data services are equally resilient largely due to Cloudera’s focus on high availability, disaster tolerance, and long history serving mission-critical workloads to our large enterprise customers.

Cloudera offers the only open true hybrid platform for data, analytics and AI, and with that comes unique opportunities for supporting high availability and disaster tolerance. With portable data services that can run on any cloud, and on-premises, you can configure a variety of available sites that mix between different clouds and include on-premises resources, reducing the dependency on a single platform, vendor, or service to operate. For more information on how Cloudera is designed for resilience, read the Cloudera blog on Disaster Recovery, and follow the Cloudera Reference Architecture for Disaster Recovery for guidance and best practices to further your own resilience and availability goals with Cloudera.

Data in motion is a set of technologies, including Apache NiFi, Apache Flink, and Apache Kafka, that enable customers to capture, process, and distribute any data anywhere, enabling real-time analytics, AI, and machine learning. These technologies are key components for many mission-critical workloads and applications – from network monitoring and service assurance in telecommunications to fraud detection and prevention in financial services. Real-time workloads, when they are mission critical, carry the additional weight of timeliness, and, as such, a potential outage could have a significantly greater business impact compared to less time-critical workloads.

Fortunately for this and many other Cloudera customers, data in motion has been designed with Cloudera’s most exacting standards for high availability and disaster tolerance, including support for hybrid cloud, ensuring even if some components were to have a dependency on a CrowdStrike affected system or service, it would not have presented itself as a single point of failure for the platform. The continuity of service that they experienced underscores the reliability and resilience of Cloudera, even in the face of significant external disruptions, as well as Cloudera’s potential for reducing the business impact of cloud provider outages.

Architect for Resilience, Especially for Real-Time Applications

The CrowdStrike incident is not the first major service disruption that businesses have experienced, and it very likely will not be the last. The cloud provides many benefits from a cost, flexibility, and scalability perspective, especially for analytic workloads. However, it also comes with some operational risk. Many workloads and applications that rely on the real-time capturing, processing, and analysis of data have zero tolerance for downtime.

Cloudera’s platform, and the data-in-motion stack, are built with resilience in mind. Cloudera’s unique approach to hybrid cloud and investment in proven architectures for high availability and disaster tolerance can mitigate the challenges many companies have experienced in the past few days, protecting their mission-critical workloads and ensuring business continuity.

Learn more about Cloudera and data in motion here.

The post Resilience in Action: How Cloudera’s Platform, and Data in Motion Solutions, Stayed Strong Amid the CrowdStrike Outage appeared first on Cloudera Blog.

View Details

There’s nothing worse than wasting money on unnecessary costs. In on-premises data estates, these costs appear as wasted person-hours waiting for inefficient analytics to complete, or troubleshooting jobs that have failed to execute as expected, or at all. They manifest as idle hardware waiting for urgent workloads to come in, ensuring sufficient spare capacity to run them amidst noisy neighbors and resource-hungry, lower-priority workloads. In the public cloud, these cost management issues are compounded by consumption rates, where compute is often overused due to a lack of visibility into optimization opportunities.

With observability, you gain more than just the information about what’s happening in your infrastructure, workloads, and related services. You can tap into insights such as where to optimize for the biggest gains, what you can do to fix workloads that don’t run, and how you can save money in the cloud.

Observability for your most secure dataFor your most sensitive, protected data, we understand even the metadata and telemetry about your workloads must be kept under close watch, and it must stay within your secured environment. You may be behind heavy firewalls, or even in a completely air-gapped environment, and sending telemetry to any third-party service is just not an option. For scenarios like this, we have now created Cloudera Observability Premium on-premises service. Simply install Cloudera Observability Premium services in your data center, set it up to receive Cloudera telemetry, and enjoy all the premium benefits without any data or metadata ever leaving your secured environment. If you have multiple Cloudera environments, as long as they can all connect to the same on-premises observability server you’ve installed, you can use this service across your organization and benefit from federated telemetry and centralized visibility.

Observability for your public cloud data estateWe all know how fast things move in the public cloud, and nothing moves faster, it seems, than the bill! One way we can help you regain control and reduce overspending in the cloud is through real-time monitoring and real-time automatic actions. Can you imagine being able to stop runaway jobs before you have to pay for them? Imagine no more. With Cloudera Observability’s latest innovation in real-time monitoring, customers running Cloudera DataHub on a public cloud can take full advantage of this feature, along with many other high-value capabilities, and start saving on cloud costs today.

New Data Observability capabilitiesYou’ve seen how Cloudera Observability Premium can tell you what you’re doing with your data – how many resources you’re using to process it, query it, and more. But what about the data itself? Wouldn’t it be great if you could also have some observability into what tables are hot and cold? Cloudera Observability Premium now includes features to measure your data’s temperature, and identify which tables are used the most, and what their health condition and other measures are like – all so you can improve data quality, performance, and health.

The data temperature feature lets us see whether hot or cold data sets are deployed optimally, including the underlying file sizes and partitioning styles. This allows you to quickly determine if your most important data is managed efficiently. You can easily check if these data sets are correctly secured, properly stored to minimize bottlenecks during analysis, and effectively partitioned to remain performant as they grow. With this added telemetry, you can take control of your data and ensure optimal use of one of your company’s most precious assets, driving even more business value from it.

Cloudera Observability does it againCloudera Observability Premium users have seen the advantages of immediately addressing issues and concerns related to the health and optimization of their data centers. Now, on-premises users with the most secure data centers can enjoy these same benefits—all without any metadata or telemetry leaving their protected environments. With Cloudera Observability Premium for DataHub on public cloud, we extend these benefits to public cloud workloads, where our customers run their mission-critical and complex applications. With these new observability features, you’ll maximize your investment and eliminate unnecessary spending.

To learn more, click here. Get Observability for your data center today as a SaaS application or reach out to your local Cloudera sales representative and let us know where you’d like to start.

The post Introducing Cloudera Observability Premium appeared first on Cloudera Blog.

View Details

In today’s data-driven world, businesses rely heavily on their dashboards to make informed decisions. However, traditional dashboards often lack the intuitive interface needed to truly harness the power of data. But what if you could simply talk to your data and get instant insights?

In the latest version of Cloudera Data Visualization, we’re introducing a new AI visual that helps users leverage the power of Large Language Models (LLMs) to “talk” to their data. Cloudera Data Visualization now leverages the latest advancements in natural language processing to transform your business dashboards into intelligent platforms.

Gone are the days of tedious filtering schemes and dropdown menus. With Cloudera Data Visualization, users can now have interactive conversations with their data, thanks to its seamless integration with LLMs of their choosing.. This means users can ask questions in plain language and receive accurate, contextually relevant responses. Say goodbye to static dashboards and hello to a whole new level of engagement.

One of the most remarkable features of the AI visual is its ability to understand context. For example, if a user asks their office supply data sets about “binders,” Cloudera Data Visualization automatically recognizes that the query might be referring to both durable and economy types that are sold by your organization. This level of intelligence streamlines the analysis process and saves valuable time. In the example below, the assistant is answering a question about sales performance of a particular product in a particular region.

AI Visual in Cloudera Data Visualization

And don’t worry about losing track of the data behind the insights. Cloudera Data Visualization allows users to easily delve deeper into the underlying data, providing transparency and fostering trust in the results. This means users can build powerful visual dashboards and reports, and also have an additional layer of contextual intelligence through the AI visual for a comprehensive business intelligence workflow. So, whether you’re a data scientist, business analyst, or executive, Cloudera Data Visualization revolutionizes the way you interact with data. It empowers users to make faster, more informed decisions by putting the power of natural language processing at their fingertips.

Dashboards, Visuals and Apps in Cloudera Data Visualization

Ready to experience the future of business intelligence? Cloudera customers can now access the Technical Preview of this new AI visual within any Data Visualization dashboard or application, and see firsthand how LLMs can transform dashboards to make it easier to surface insights. For more information on these features and our AI capabilities, visit our Enterprise AI page. When you’re ready, you can request a demo at the bottom of the page to see how these capabilities can work in the context of your business. The future of data insight from visualization is here – and it’s smarter than ever before.

The post Revolutionize Your Business Dashboards with Large Language Models appeared first on Cloudera Blog.

View Details

In the last couple of years, AI has launched itself to the forefront of technology initiatives across industries. In fact, Gartner predicts the AI software market will grow from $124 billion in 2022 to $297 billion in 2027. As a data platform company, Cloudera has two very clear priorities. First, we need to help customers get AI models based on trusted data into production faster than ever. And second, we need to build AI capabilities into Cloudera to give more people access to data-driven insights for their everyday roles.

At our recent Cloudera Now virtual event, we announced three new capabilities that support both of our AI priorities: An AI-driven SQL assistant, a Business Intelligence (BI) chatbot that converses with your data, and an ML copilot that accelerates machine learning development. Let’s take a deeper dive into how these capabilities accelerate your AI initiatives and support data democratization.

SQL AI Assistant: Your New Best FriendWriting complex SQL queries can be a real challenge. From finding the right tables and columns to dealing with joins, unions, and subselects, then optimizing for readability and performance, and doing all of that while taking into account the unique SQL dialect of the engine, it’s enough to make even the most seasoned SQL developer’s head spin. And at the end of the day, not everyone who needs data to be successful in their day-to-day work is an SQL expert.

Imagine, instead, having a domain expert and a SQL guru always by your side. That’s exactly what Cloudera’s SQL AI assistant is. Users simply describe what they need in plain language, and the assistant will find the relevant data, write the query, optimize it, and even explain it back in easy-to-understand terms.

Under the hood, the assistant uses advanced techniques like prompt engineering and retrieval augmented generation (RAG) to truly understand your database. It works with many large language models (LLMs), whether they are public or private, and it effortlessly scales to handle thousands of tables and users simultaneously. So whether you’re under pressure to answer critical business questions or just tired of wrestling with SQL syntax, the AI assistant has your back, enabling you to focus on what really matters – getting insights from your data.

AI Chatbot in Cloudera Data Visualization: Your Data’s New Best FriendBI dashboards are undeniably useful, but they often only tell part of the story. To gain meaningful and actionable insights, data consumers need to engage in a conversation with their data, and ask questions beyond simply the “what” that a dashboard typically shows. That’s where the AI Chatbot in Cloudera Data Visualization comes into play.

The chatbot resides directly within your dashboard, ready to answer any question you pose. And when we say “any question,” we mean it. Why are sales down in the Northeast? Will this trend continue? What actions should we take? The chatbot leverages the context of the data behind the dashboard to deliver deeper, more actionable insights to the user.

A written answer is a great way to start understanding your data, but let’s not forget the power of the visuals in our dashboards and reports. The chatbot eliminates the burden of clicking through dropdowns and filters to find answers. Simply ask what you want to know, in plain language, and the chatbot will intelligently match it to the relevant data and visuals. It’s like having a dedicated subject matter expert right there with you, ready to dive deep into the insights that matter most to your business.

Cloudera Copilot for Cloudera Machine Learning: Your Model’s New Best FriendBuilding machine learning models is no easy feat. From data wrangling to coding, model tuning to deployment, it’s a complex and time-consuming process. In fact, many models never make it into production at all. But what if you had a copilot to help navigate all of the challenges related to deploying models in production?

Cloudera’s ML copilots, powered by pre-trained LLMs, are like having machine learning experts on call 24/7. They can write and debug Python code, suggest improvements, and even generate entire applications from scratch. With seamless integration to over 130 Hugging Face models and datasets, you have a wealth of resources at your disposal.

Whether you’re a data scientist looking to streamline your workflow or a business user eager to get an AI application up and running quickly, the ML copilots support the end-to-end development process and get models into production fast.

Elevate Your Data with AI AssistantsBy embedding AI assistants for SQL, BI, and ML directly into the platform, Cloudera is simplifying and enhancing the data experience for every single user. SQL developers will be more efficient and productive than ever. Business analysts will be empowered to have meaningful, actionable conversations with data, uncovering the “why” behind the “what.” Additionally, data scientists will be empowered to bring new AI applications to production faster and with greater confidence.

For more information on these features and our AI capabilities, visit our Enterprise AI page. When you’re ready, you can request a demo at the bottom of the page to see how these capabilities can work in the context of your business.

The post Introducing Cloudera’s AI Assistants appeared first on Cloudera Blog.

View Details

In the fast-evolving landscape of data science and machine learning, efficiency is not just desirable—it’s essential. Imagine a world where every data practitioner, from seasoned data scientists to budding developers, has an intelligent assistant at their fingertips. This assistant doesn’t just automate mundane tasks but understands the intricacies of your workflows, anticipates your needs, and dramatically enhances your productivity at every turn. Welcome to the era of Cloudera Copilot for Cloudera Machine Learning.

The Evolution of AI-Powered AssistanceAt Cloudera, we understand the challenges faced by data practitioners. The complexities of modern data workflows often translate into countless hours spent coding, debugging, and optimizing models. Recognizing this pain point, we set out to redefine the data science experience with AI-driven innovation.

Cloudera Copilot for Cloudera Machine Learning integrates cutting-edge large language models directly into the machine learning service. This integration empowers developers and data scientists alike with advanced capabilities for code completion, generation, and troubleshooting. Whether you’re tackling data transformation challenges or refining intricate machine learning models, our Copilot is designed to be your reliable partner in innovation.

Accelerating Productivity with AIThe Cloudera Copilot for Cloudera Machine Learning redefines data practitioners’ workflows across critical areas: code generation and autocompletion, debugging and troubleshooting, and code understanding and exploration.

In code generation and autocompletion, the Copilot dramatically speeds up the development lifecycle. It begins by generating initial notebooks to kickstart projects and continues to assist in writing functions, test cases, or documentation. This practical support speeds up project initiation and maintains consistent coding practices. By automating these essential tasks, the Copilot frees up developers’ time to concentrate on innovation and problem-solving rather than spending valuable time on repetitive tasks. This integrated assistant fosters a more efficient development process and boosts overall productivity in data science and machine learning endeavors.

When it comes to debugging and troubleshooting, the Copilot reduces time to solution by clarifying error messages, identifying bugs within code, and proposing practical solutions such as: analyzing static code, recognizing common issues, and providing proactive recommendations to address them. This proactive approach not only improves code quality but also empowers developers to resolve issues more efficiently, fostering a smoother development process and minimizing disruptions to project timelines.

In the realm of code understanding and exploration, the Cloudera Copilot for Cloudera Machine Learning speeds up developer onboarding and assists in navigating complex projects. By capturing project-specific knowledge, the Copilot can assist new team members in grasping project structures, helping with high-level questions, and explaining implementation details. This capability accelerates learning curves and fosters better collaboration within teams of varying expertise, ultimately leading to more efficient development cycles and enhanced project outcomes.

The Future of Data Science, TodayAs we continue to innovate within Cloudera Machine Learning, the introduction of Cloudera Copilot marks a significant leap forward in data science practitioner productivity capabilities. Our commitment to empowering data practitioners extends beyond mere automation—it’s about equipping them with the tools to rapidly drive innovation and achieve meaningful outcomes.

Get Started TodayAre you ready to unlock the full potential of your data workflows? Explore the capabilities of Cloudera Copilot for Cloudera Machine Learning and experience firsthand how AI-powered assistance can transform your productivity.

The post Unparalleled Productivity: The Power of Cloudera Copilot for Cloudera Machine Learning appeared first on Cloudera Blog.

View Details

Juneteenth holds profound significance in the history of freedom and equality for Black Americans. Also known as Freedom Day or Emancipation Day, Juneteenth commemorates the anniversary of June 19, 1865, when news of the Emancipation Proclamation reached Galveston, Texas, finally declaring freedom for enslaved Americans held in the Confederacy–more than two years after the proclamation was issued in on January 1, 1863.

Black Americans have celebrated Juneteenth across the US–predominantly in the south and southwestern regions –for over 150 years. However, awareness and recognition of its importance has only come into the mainstream since it was recognized as a federal holiday in 2021.

Juneteenth is a momentous occasion that marked the culmination of a long and arduous struggle for freedom and justice. It calls for celebration, but also provides an important opportunity to reflect on the experiences of Black Americans throughout history and the challenges they still face today. As we celebrated Juneteenth in the U.S., we honored the resilience, courage, and perseverance of those who fought for freedom and equality then, and of those who continue to fight for true equality and equity today.

Cloudera is committed to fostering a culture of inclusivity, diversity, and belonging that empowers all employees to bring their whole, authentic selves to work every day. One of the ways we did this was by providing opportunities for our employees to educate themselves on the experiences of their colleagues and community members.

This year, we encouraged our employees to come together to celebrate their shared humanity and reaffirm their commitment to building a more just and equitable world for all. We urged them to take the time to engage in celebrations and activities that expanded their awareness of the holiday, including by joining our Juneteenth History & Culture Jeo-Party, which took place on June 20th. The event featured a fast-paced and competitive game that celebrated and shared education on Black history and culture.

“A vital part of our culture at Cloudera is ensuring all voices are heard, and that we take the time to connect with others and understand the unique backgrounds that shape our lives. Our Juneteenth History & Culture Jeo-Party was an incredible success, filled with engaging dialogue and great insight into both Black history and the day-to-day experiences of our fellow colleagues.” – Dipto Chakravarty, Chief Product Officer

Interested in resources to learn about Juneteenth? We encourage you to watch this video from a previous Understanding Juneteenth event and review our Understanding Juneteenth Resource Guide.

For more on Cloudera’s commitment to Diversity, Equality and Inclusion, click here.

The post The Importance of Recognizing Juneteenth appeared first on Cloudera Blog.

View Details

The world of Generative AI (GenAI) is rapidly evolving, with a wide array of models available for businesses to leverage. These models can be broadly categorized into two types: closed-source (proprietary) and open-source models.

Closed-source models, such as OpenAI’s GPT-4o, Anthropic’s Claude 3, or Google’s Gemini 1.5 Pro, are developed and maintained by private and public companies. These models are known for their state-of-the-art performance and extensive training on vast amounts of data. However, they often come with limitations in terms of customization, control, and cost.

On the other hand, open-source models, such as Llama 3 or Mistral, are freely available for businesses to use, modify, and deploy. These models offer greater flexibility, transparency, and cost-effectiveness compared to their closed-source counterparts.

Advantages and Challenges of Closed-source ModelsClosed-source models have gained popularity due to their impressive capabilities and ease of use. Platforms like OpenAI’s API or Google Cloud AI provide businesses with access to powerful GenAI models without the need for extensive in-house expertise. These models excel at a wide range of tasks, from content generation to language translation.

However, the use of closed-source models also presents challenges. Businesses have limited control over the model’s architecture, training data, and output. This lack of transparency can raise concerns about data privacy, security, and bias. Additionally, the cost of using closed-source models can quickly escalate as usage increases, making it difficult for businesses to scale their GenAI applications.

The Rise of Open-source Models: Customization, Control, and Cost-effectivenessOpen-source models have emerged as a compelling alternative to closed-source models, and usage has been on the rise. According to GitHub, there was a 148% year-over-year increase in individual contributors and a 248% rise in the total number of open-source GenAI projects on GitHub from 2022 to 2023. With open-source models, businesses can customize and fine-tune models to their specific needs. By training open-source models on enterprise-specific data, businesses can create highly tailored GenAI applications that outperform generic closed-source models.

Moreover, open-source models provide businesses with complete control over the model’s deployment and usage. According to data gathered by Andreessen Horowitz (a16z), 60% of AI leaders cited control as the primary reason to leverage open source. This control enables businesses to ensure data privacy, security, and compliance with industry regulations. Open-source models also offer significant cost savings compared to closed-source models, as businesses can run and scale these models on their own infrastructure without incurring excessive usage fees.

Selecting the right GenAI model depends on various factors, including the specific use case, available data, performance requirements, and budget. In some cases, closed-source models may be the best fit due to their ease of use and state-of-the-art performance. However, for businesses that require greater customization, control, and cost-effectiveness, open-source models are often the preferred choice.

Cloudera’s Approach to Model Flexibility and DeploymentAt Cloudera, we understand the importance of flexibility in GenAI model selection and deployment. Our platform supports a wide range of open-source and closed-source models, allowing businesses to choose the best model for their specific needs.

Fig 1. Cloudera Enterprise GenAI Stack
Openness and interoperability are key to leverage the full GenAI ecosystem.

With Cloudera, businesses can easily train, fine-tune, and deploy open-source models on their own infrastructure. The platform provides a secure and governed environment for model development, enabling data scientists and engineers to collaborate effectively. Our platform also integrates with popular open-source libraries and frameworks, such as TensorFlow and PyTorch, ensuring compatibility with the latest advancements in GenAI.

For businesses that prefer to use closed-source models, Cloudera’s platform offers seamless integration with leading public cloud AI services, such as Amazon Bedrock. This integration allows businesses to leverage the power of closed-source models while still maintaining control over their data and infrastructure.

Find out how Cloudera can help fuel your enterprise AI journey.

The post Empowering Enterprise Generative AI with Flexibility: Navigating the Model Landscape appeared first on Cloudera Blog.

View Details

Pride Month is underway and we at Cloudera are looking forward to joining the global celebration of diversity, equity and the ongoing effort for LGBTQ+ (Lesbian, Gay, Bisexual, Transgender, Queer/Questioning) rights and recognition.

Pride Month serves as a reminder that the fight for equality and equity for members of the LGBTQ+ community is not over. It is a call to action to continue to support and advocate for the rights of our LGBTQ+ colleagues and friends within our organization and our communities.

We are proud that Cloudera’s culture is built on a strong foundation of fostering inclusivity and creating an environment where everyone feels valued, respected, and empowered to bring their authentic selves to work. In past years, Cloudera employees have hosted events, both in-person and virtually, to celebrate. Last year, our team in Cork, Ireland joined those efforts and hosted its first on-site Pride event. The event brought together over 80 employees to celebrate the 30th anniversary of the decriminalization of same-gender loving relationships in Ireland. The event proved to be a resounding success, but don’t just take our word for it. Here’s just a sampling of what our Cork team had to say.

“Our Cork Pride events have gotten bigger and bigger with each passing year and we want to surpass that again this year. We are very fortunate in Cloudera to have the backing of our Leadership sponsors across all areas of the Organisation and without them, these events would not be possible.” Charlotte Keating, Senior Operations Analyst

“Being a member of the LGBTQ+ community, I was delighted to have the opportunity to give Clouderans in Ireland the chance to experience a colorful pride celebration, even people outside of the community should experience at least once in their life!” – Sean Murphy Phelan, Associate Business System Analyst

“The growing participation among the Clouderans both within and outside the community to take part in organising and attending the Pride events amplifies the culture of acceptance, welcome, and celebration that has been created in Cloudera over the previous years and will continue to grow through their support, participation, and help.” – Ódhlan Duffy, Cloudera Cork Intern, FSS.

Building on that success, the Cork Team is excited to announce that this year’s celebration will be held on Wednesday, June 26. Once again, they invite Cloudera’s global workforce to join the festivities which they promise will be even bigger and bolder this year. The event will include party games, a special performance by Cork’s own drag performer Krystal Queer, and inspiring conversations around the triumphs and struggles of the LGBTQ+ community. Attendees will also hear from Cloudera leadership and members of the EMEA+ ERG Committee about our ongoing efforts in promoting Diversity, Equality, and Inclusion within our workforce. Additionally, the Cork Pride Committee will join the team to share insights into their mission and the significance of Pride in the local community.

The celebration will continue in August, when Cloudera participates in Cork’s Pride Parade for the second year. This all-out colorful event draws people from all walks of life to honor the LGBTQ+ community. This year’s theme, ‘Unity in Community,’ is a powerful call to action focusing on inclusivity and acceptance, as well as the importance of strong support systems. We encourage all employees and our broader community to attend and celebrate the LGBTQ+ community, commemorate their LGBTQ+ peers, family, and friends, and to expand their knowledge of the history of Pride and the powerful contributions of the volunteers at Cork Pride in our community.

For more on Cloudera’s commitment to Diversity, Equality and Inclusion, click here.

The post Cloudera Unveils Plans for Annual Pride Celebration in Cork appeared first on Cloudera Blog.

View Details

At a time when artificial intelligence (AI) and tools like generative AI (GenAI) and large language models (LLMs) have exploded in popularity, getting the most out of organizational data is critical to driving business value and carving out a competitive market advantage. To reach that goal, more businesses are turning toward hybrid cloud infrastructure – with data on-premises, in the cloud, or both – as a means to tap into valuable data.

But for all the excitement and movement happening within hybrid cloud infrastructure and its potential with AI, there are still risks and challenges that need to be appropriately managed—specifically when it comes to the issue of data governance. Inherently, a hybrid cloud infrastructure allows data to move between environments, which can make that data vulnerable to not only security risks but also lapses in compliance with internal standards or external regulations like the General Data Protection Regulation (GDPR) or California Consumer Privacy Act (CCPR) and even the Health Insurance Portability and Accountability Act (HIPAA).

The need for effective data governance itself is not a new phenomenon. It’s something that’s always been an important task alongside everyone’s day-to-day workflows. It’s also something that, unlike other projects, is always happening. With hybrid cloud infrastructure firmly cemented as the preferred approach to data infrastructure, governance needs to be at the top of every to-do list.

Barriers to Good Data Governance RemainA Cloudera survey found that 72% of enterprise leaders agreed that data governance was an enabler of business value. But whether it’s a lack of buy-in from leadership, a disparate set of tools and users, or increasingly siloed data, a broad range of complications can prevent data governance measures from being comprehensive and impactful. As hybrid cloud architecture leads the way, data silos—a challenge that most IT leaders are all too familiar with—have increasingly been a pain point when managing governance. Any time operations are run through a hybrid architecture with multiple environments, it becomes difficult to maintain a holistic view of what’s happening across the organization as a whole, with different workloads running in isolated pockets, if things like data access and visibility are not managed effectively.

This, understandably, presents a major challenge when it comes to ensuring data governance practices are effectively implemented—even in the most data-driven organizations. The Cloudera survey found that over one quarter (26%) of enterprise practitioners reported they had anywhere from 51 to 100 data silos spread across their organizations. To break those silos and achieve a meaningful, comprehensive level of data governance, both large enterprises and data-driven organizations must prioritize and fully integrate solutions that help boost data visibility in a hybrid setting while also ensuring consistent compliance with both business-level data guidelines and external regulation.

Establishing Comprehensive and Impactful Data GovernanceSo, how should organizations go about data governance in a hybrid cloud? Some IT and business leaders attempt a more fragmented approach to governance, but while somewhat effective, that can create ‘islands’ of perfection—a place where governance appears to be happening effectively, leading IT leaders to assume it’s not as much of a concern. While these islands may seem sound up close, at an enterprise level, they become fragmented, isolated, and prone to collapse should governance change in other areas. Ultimately, the key to good governance must leverage solutions and architectures that provide an enterprise-level, unified, viewpoint of governance and data visibility.

In a hybrid cloud setting, data gravity sees smaller bodies of data pulled in toward larger ones where the center of gravity exists. Creating, managing, and maintaining each of those connections adds up over time and hinders data governance efforts. That’s where adopting the right hybrid data platform can help transform those operations and achieve a true hybrid cloud experience. With the right solution, businesses gain the ability to leverage architectures like a data fabric—a type of data architecture designed to give a unified view of data across an organization, regardless of where data is stored or how it is structured.

It also infuses automation into data management, handling not just the unlocking of data but also the sorting and investigating of information, determining what goes where and who needs access, bolstering internal guidelines and access rules that feed back to data governance. In the context of comprehensive data governance, implementing a data fabric has become a vitally important first step.

Learn more about how Cloudera can help guide you on your journey toward comprehensive data governance.

The post Where Does Data Governance Fit Into Hybrid Cloud? appeared first on Cloudera Blog.

View Details

Hadoop. The first time that I really became familiar with this term was at Hadoop World in New York City some ten or so years ago. There were thousands of attendees at the event – lining up for book signings and meetings with recruiters to fill the endless job openings for developers experienced with MapReduce and managing Big Data.

This was the gold rush of the 21st century, except the gold was data. Two companies were at the center of it all, handing out the proverbial pickaxes: Cloudera and Hortonworks.

After countless open-source innovations ushered in the Big Data era, including the first commercial distribution of HDFS (Apache Hadoop Distributed File System), commonly referred to as Hadoop, the two companies joined forces, giving birth to an entire ecosystem of technology and tech companies.

You could argue that the Big Data and analytics movement would not have happened without Cloudera. And all of those massive volumes of data are now today’s data lakes – more than 25 exabytes managed by Cloudera.

But, let’s make one thing clear – we are no longer that Hadoop company.

Welcome to Today’s ClouderaSince the Big Data era, Cloudera has made massive investments with the north star of delivering customer-focused innovation wherever our customers run their business-critical data and analytics. This includes running analytics at the edge, supporting multi-cloud environments, treating Apache Iceberg as a first-class citizen, and introducing many more innovations like data observability.

That investment and support have resulted in the first true hybrid platform for data, analytics, and AI, backed by a seasoned and proven leadership team, with a go-to-market strategy focused on ensuring our customers’ success in the future of Enterprise AI.

The Future of Enterprise AI, Delivered TodayIf the Big Data era was this century’s gold rush, then AI is the next moon shot. Hyperbole doesn’t really apply when you speak to the potential impact of AI for every business and person on earth (and beyond). But, what is essential to putting AI into practice to improve productivity? Again, it’s all about the data, but more specifically, trusted data so that you can trust in Enterprise AI.

Only Cloudera has the ability to help organizations overcome the three barriers to trust in Enterprise AI:

• Readiness – Can you trust the safety of your proprietary data in public AI models? Cloudera’s true hybrid approach ensures you can leverage any deployment, from virtual private cloud to on-premises data centers, to maximize the use of AI.
• Reliability – Can you trust that your data quality will yield useful AI results? With Cloudera’s modern data architectures, you can ensure your data is of high quality, well-governed, and managed as a single data estate.
• Responsibility – Can you trust your AI models will give meaningful insight? Cloudera’s support for both open and closed models for enterprise AI available to all form factors ensures you have the choice, flexibility, and ability to cross-compare and ensure useful outcomes that you can trust.

With last week’s acquisition of Verta’s operational AI platform, we are deepening our technology and talent to accelerate AI innovation and, more specifically, simplify the process of bolstering customers’ private datasets to build retrieval-augmented generation (RAG) and fine-tuning applications. As a result, developers – regardless of their expertise in machine learning – will be able to develop and optimize business-ready large language models (LLMs). These bold acquisitions, a continual release of innovations, and key partnerships from the ecosystem, including NVIDIA, will enable all companies to prosper in the Enterprise AI era.

The Enterprise Runs on ClouderaInnovative technology aside, the best evidence to show how a vendor has evolved to meet the business-critical use cases of its customers are through success stories.

Cloudera plays a central role not only at work but in all of our daily lives – from the money you save and spend, to the energy and connectivity in your home, to the car you are driving (and your insurance rates), to the phone and network that you are using, to the life-saving drugs and healthcare that keep you and your loved ones healthy.

A recent customer story – OCBC Bank has accelerated its data strategy with Cloudera – illustrates the power of Cloudera for machine learning use cases, particularly in the area of GenAI with business impact:

• OCBC’s Next Best Conversation, a centralized platform that uses machine learning to analyze real-time contextual data from customer conversations related to sales, service and more. The bank increased their revenue by more than $100M annually by using the data to identify the most relevant information for each customer and curate personal experiences across communication channels.
• OCBC also developed a credit card fraud detection solution that reduced the volume of transactions reviewed by anti-money laundering compliance analysts and increased the accuracy rate of identifying suspicious transactions. They developed smarter processes on the platform by introducing chatbots to take over 10% of customer interactions on their website.

But, What Happened to Hadoop?Many of our customers store and manage their data – much of it unstructured – in HDFS, particularly in on-prem environments. And, with the growing popularity of object storage, we support a variety of S3 object stores from our partners for customers who want cloud-native architectures delivered on public and private clouds.

That open approach is key to enabling our customers to analyze data wherever it resides. Instead of moving the data each time to the compute that you want to use, you just keep all the data in its current place and bring the compute to the data. That is the key to our open data lakehouse architecture.

Also, we have seen significant adoption in Apache Ozone, a scalable, redundant, and distributed object store optimized for big data workloads running on-premises. In fact, Cloudera customers have just exceeded 1 exabyte of data stored in Ozone. In addition, customers can use the Ozone file system with key Apache technologies, including Apache Hive, Apache Spark, and Apache Iceberg, as well as any S3-compatible workload.

Those are just a few examples of how Cloudera constantly evolves with customer-led innovation to prepare everyone for a truly open future of data, analytics, and AI. That’s today’s Cloudera.

To learn more about groundbreaking innovations and customer stories, join us at Cloudera EVOLVE, the industry’s premier data and AI conference. We hope to see you there.

The post Addressing the Elephant in the Room – Welcome to Today’s Cloudera appeared first on Cloudera Blog.

View Details

Of all of the emerging tech of the last two decades, artificial intelligence (AI) is tipping the hype scale, causing organizations from all industries to rethink their digital transformation initiatives asking where it fits in. In Financial Services, the projected numbers are staggering. According to a recent McKinsey & Co. article, “The McKinsey Global Institute (MGI) estimates that across the global banking sector, [Generative AI] could add between $200 billion and $340 billion in value annually, or 2.8 to 4.7 percent of total industry revenues.”

While these numbers reflect the potential impact of broad implementation, I’m often asked by our Financial Services customers for suggestions as to which use cases to prioritize as they plan Generative AI (GenAI) projects, and AI more broadly.

In truth, the question is usually framed more like, “How are my competitors using AI and GenAI?” and “What business use cases are they focused on?”

What Should Institutions Invest In?The truth is, the industry is rapidly adopting AI and GenAI technologies to drive innovation across various domains. Traditional machine learning (ML) models enhance risk management, credit scoring, anti-money laundering efforts and process automation. Meanwhile, GenAI unlocks new opportunities like personalized customer experiences through virtual assistants, automated content creation, advanced risk and compliance analysis, and data-driven trading strategies.

Some of the biggest and well-known financial institutions are already realizing value from AI and GenAI:

• JPMorgan Chase uses AI for personalized virtual assistants and ML models for risk management.
• Capital One leverages GenAI to create synthetic data for model training while protecting privacy.
• BlackRock utilizes GenAI to automatically generate research reports and investment summaries.
• Deloitte employs AI for risk, compliance, and analysis while also using ML models for fraud detection.
• HSBC harnesses ML for anti-money laundering efforts based on transaction patterns.
• Bridgewater Associates leverages GenAI to process data for trading signals and portfolio optimization.

The key is identifying high-value, high-volume tasks that can benefit from automation, personalization and rapid analysis enabled by ML, AI, and GenAI models. Prioritizing use cases that directly improve customer experiences, operational efficiency and risk management can also drive significant value for the industry.

AI and ML for Risk ManagementML models can analyze large volumes of data to identify patterns and anomalies indicating potential risks such as fraud, money laundering or credit default, enabling proactive mitigation. In credit scoring and loan underwriting, AI algorithms evaluate loan applications, credit histories and financial data to assess creditworthiness and generate more accurate approval recommendations than traditional methods. ML models enhance anti-money laundering (AML) compliance by detecting suspicious transaction patterns and customer behaviors. Additionally, AI and robotic process automation (RPA) improve operational efficiency by automating repetitive tasks like data entry, document processing, and report generation.

Quick Wins with GenAI OpportunitiesFinancial institutions can achieve quick wins by leveraging GenAI to enhance or improve a range of use cases including customer service, operations, and decision-making processes.

Customer experiences

One significant application is in creating personalized customer experiences. AI-powered virtual assistants and chatbots can understand natural language queries, enabling them to provide tailored financial advice, product recommendations, and support. This personalized approach will improve customer satisfaction and engagement.

Content creation

Another area where AI will make a substantial impact is in automated content creation. GenAI models can automatically generate a wide range of materials, including marketing content, research reports, investment summaries and more. By analyzing data, news, and market trends, these models produce high-quality content quickly and efficiently, freeing up human resources for more strategic tasks.

Risk and compliance analysis

Risk and compliance analysis is another critical application of AI in finance. AI can rapidly analyze complex legal documents, regulations, financial statements and transaction data to identify potential risks or regulatory and compliance issues. This capability allows financial institutions to generate detailed assessment reports swiftly, ensuring they remain compliant with evolving regulations and mitigate risks effectively.

Trading and portfolio optimization

GenAI can play a pivotal role in trading and portfolio optimization by processing vast amounts of data to generate actionable insights and trading signals. These insights enable the implementation of automated investment strategies, additional variables in decision-making and optimized portfolio management allowing financial institutions to deliver superior investment performance to their clients.

The Opportunities are Compelling, but Significant Challenges Must be AddressedData privacy and security in the financial sector demand rigorous protection measures for sensitive information. This includes robust encryption, stringent access controls and advanced anonymization techniques to ensure financial data remains secure. Moreover, ensuring AI decision-making processes are transparent and explainable is crucial for meeting regulatory compliance standards. This transparency helps in understanding and verifying AI-driven decisions, thereby fostering trust.

Addressing biases and errors in training data is essential to prevent the propagation of incorrect insights. Bias mitigation ensures that AI systems provide fair and accurate outcomes, which is critical for maintaining the integrity of financial services. Additionally, safeguarding AI systems against data manipulation attacks and exploitation for fraudulent activities is vital to address cybersecurity vulnerabilities. This involves implementing strong defensive measures and continuously monitoring for potential threats.

Adhering to industry regulations and guidelines is necessary to ensure fairness and accountability in AI decision-making processes. Compliance with these standards helps in maintaining governance and regulatory oversight, which are essential for building a trustworthy AI ecosystem.

Monitoring for new sources or transmission channels of systemic risks introduced by AI adoption is crucial for managing systemic financial risks. These might include unforeseen vulnerabilities in AI models, reliance on flawed or biased data, or new types of cyber threats targeting AI systems. Understanding how these risks can spread within the financial system is critical to safe and effective AI. For instance, an error in an AI model used by one financial institution could propagate through interconnected systems and markets, affecting other institutions and leading to broader financial instability. Not addressing these risks can impact the entire financial system, not just individual entities, and have the potential to cause widespread disruption and significant economic consequences.

Additionally, proactive governance frameworks, security protocols and regulatory guidance will be crucial as financial institutions continue exploring the potential of AI.

How Cloudera helps Financial Institutions on their AI and Gen AI journeyCloudera helps financial institutions harness the power of AI and GenAI while navigating the associated risks. Cloudera provides a secure, scalable and governed environment for managing and analyzing vast volumes of structured and unstructured data, essential for training accurate and unbiased AI models. Integrated ML and AI tools allow financial institutions to develop, deploy and monitor AI models efficiently, streamlining the implementation of the aforementioned use cases.

Cloudera’s advanced data management capabilities ensure the highest levels of data privacy and security while data lineage and governance features help institutions maintain transparency and compliance with regulatory requirements.

With Cloudera, financial institutions can unlock the full potential of AI and GenAI while mitigating risks, ensuring responsible adoption, and driving innovation in the industry.

The post Making an AI Investment: How Finance Institutions are Harnessing the Power of AI and Generative AI appeared first on Cloudera Blog.

View Details

More than two-thirds of companies are currently using Generative AI (GenAI) models, such as large language models (LLMs), which can understand and generate human-like text, images, video, music, and even code. However, the true power of these models lies in their ability to adapt to an enterprise’s unique context. By leveraging an organization’s proprietary data, GenAI models can produce highly relevant and customized outputs that align with the business’s specific needs and objectives.

Structured and Unstructured Data: A Treasure Trove of InsightsEnterprise data encompasses a wide array of types, falling mainly into two categories: structured and unstructured. Structured data is highly organized and formatted in a way that makes it easily searchable in databases and data warehouses. This data often includes fields that are predefined, such as dates, credit card numbers, or customer names, which can be readily processed and queried by traditional database tools and algorithms.

On the other hand, unstructured data lacks a predefined format or structure, making it more complex to manage and utilize. This type of data includes a variety of content such as documents, emails, images and videos. Thankfully, GenAI models can harness the insights hidden within both structured and unstructured data. As a result, these models enable organizations to unlock new opportunities and gain a 360 degree view of their entire business.

For example, a financial institution can use GenAI to analyze customer interactions across various channels, including emails, chat logs, and call transcripts, to identify patterns and sentiments. By feeding this unstructured data into an LLM, the institution can generate personalized financial advice, improve customer service, and detect potentially fraudulent activities.

The Role of an Open Data Lakehouse in Seamless Data AccessTo fully capitalize on the potential of GenAI, enterprises need seamless access to their data. This is proving to be a challenge for businesses – only four percent of business and technology leaders described their data as fully accessible. This is where an open data lakehouse comes into play. It is the building block of a strong data foundation necessary to adopt GenAI. An open data lakehouse breaks down data silos and enables the integration of data from various sources, making it readily available for GenAI models.

Cloudera’s open data lakehouse provides a secure and governed environment for storing, processing, and analyzing massive amounts of structured and unstructured data. With built-in security and governance features, businesses can ensure that their data is protected and compliant with industry regulations while still being accessible for GenAI applications.

By feeding enterprise data into GenAI models, businesses can create highly contextual and relevant outputs. For instance, a manufacturing company can use GenAI to analyze sensor data, maintenance logs, production records and reference operational documentation to predict potential equipment failures and optimize maintenance schedules. By incorporating enterprise-specific data, the GenAI model can provide accurate and actionable insights tailored to the company’s unique operating environment – helping drive ROI for the business.

Real-world Examples of Data-driven Generative AI SuccessOCBC Bank, a leading financial institution in Singapore, has leveraged GenAI to enhance its customer service and internal operations. By feeding customer interaction data and financial transaction records into LLMs, OCBC Bank has developed AI-powered chatbots that provide personalized financial advice and support. The bank’s teams built Next Best Conversation, a centralized platform that uses machine learning to analyze real-time contextual data from customer conversations related to sales, service, and other variables to deliver unique insights and opportunities to improve operations. The bank has also used GenAI to automate document processing, reducing manual effort and improving efficiency.

A global pharmaceutical company has utilized GenAI to accelerate drug discovery and development. By integrating structured and unstructured data from clinical trials, research papers, and patient records, the company has trained GenAI models to identify potential drug candidates and predict their efficacy and safety. This data-driven approach has significantly reduced the time and cost associated with bringing new drugs to market.

These real-world examples demonstrate the transformative power of combining enterprise data with GenAI. By leveraging their unique data assets, businesses across industries can unlock new opportunities, drive innovation, and gain a competitive edge.

Learn more about how Cloudera can help accelerate your enterprise AI journey.

The post Fueling Enterprise Generative AI with Data: The Cornerstone of Differentiation appeared first on Cloudera Blog.

View Details

A constant flow of breaking news from the data lakehouse space is making notable tech headlines this week.

On Tuesday, Databricks announced that it will acquire Tabular, a data management company founded by the creators of Apache Iceberg, Ryan Blue, Daniel Weeks, and Jason Reidfor. The deal was for an unconfirmed sum, but some reports suggest that amount to be between $1B and $2B (and allegedly outbidding Snowflake). The move aims to unify the two most popular open-souce lakehouse formats – Apache Iceberg and Linux Foundation Delta Lake – to enhance data compatibility across different formats.

The prior day, Snowflake – still dealing with the aftermath of last week’s data breach – announced Polaris Catalog, a vendor-neutral, open catalog for Apache Iceberg. The company also announced at its annual user conference that Polaris Catalog will be open sourced in the next 90 days.

So, how do you make sense of all these announcements and what does this mean to you?

Iceberg is the Champion in the Table Format WarDatabricks putting this much value in Iceberg is proof that Delta Lake has lost the table format war, and Iceberg is the clear winner. Iceberg will further become, and will remain, the de facto standard for large-scale data and analytics deployments for the long run.

Cloudera was a first mover in adopting Iceberg as central and native to our data, analytics, and AI platform – reinforcing our credibility as the best vendor to work with when you want managed Iceberg data estates, at scale, across all clouds and on-premises.

How Open is Your Open Source?Despite its claims as the open data lakehouse company, Databricks is NOT well known for being true to open source. Unlike Tabular, Databricks has made commercial versions as proprietary implementations of open source technology in a bid to retain customer lock-in, and it will remain to be seen if this move changes that approach.

Cloudera is a neutral party that manages Iceberg without vendor lock-in and at scale – in all clouds and on-premises. Cloudera also counts as customers many of the other large organizations that directly contribute to the project. That’s truly open source.

Tabular Does Not Own IcebergTabular was founded by the originators of the Iceberg project. The company has about 20% of the Iceberg contributors and committers on staff (companies like AWS, Google, Dremio, Starburst, Adobe, Apple, Netflix, and more), which make up the bulk of the contributions. It has a healthy community, unlike Delta Lake, and a lot of big tech companies who are invested in keeping it open source and vendor independent.

This is a risky and costly acquisition by Databricks, particularly if the 80% of the committers decide that other committer affiliations weaken the mission to remain open source for all.

Welcome to the PartyCloudera has been ahead of this game for years. Our 2022 open lakehouse position blog post was essentially the blueprint for the Databricks acquisition announcement.

Iceberg has, and continues to be, central to Cloudera’s open data lakehouse architecture across hybrid clouds – not just something to be used on the side. Databricks failed to gain adoption for Delta Lake from communities and third-party vendors, and now must make this BIG and costly bet. At the same time, Snowflake’s Polaris catalog timing shows that they’ve been forced into this space as the market and customers have moved Iceberg as the central table format for their data two years after Cloudera.

They are both not only late to join the party, but will miss the fun–and opportunity–as they play catch up to those of us who have been here from the start.

The post Databricks Follows Cloudera by Adopting Iceberg, While Snowflake Mulls Open Source Approach appeared first on Cloudera Blog.

View Details

Recently, Cloudera, alongside OCBC, were named winners in the“Best Big Data and Analytics Infrastructure Implementation” category at The Asian Banker’s Financial Technology Innovation Awards 2024. This recognition underscores the importance of trusted data when building AI and generative AI (GenAI) models and serves as a testament to the impact that reliable data can have in real world use cases.

As we celebrate this win, let’s explore the work that Cloudera and OCBC did together and why a trusted AI is so critical to effective AI.

The Role of AI in Banking2024 continues to witness the rapid development of AI and its applications, with GenAI leading the charge. The McKinsey Global Institute estimates that across various industries worldwide, generative AI could contribute an annual value ranging from US$2.6 trillion to US$4.4 trillion. Banking, in particular, stands to gain significantly, with an estimated increase in revenue of US$200 billion to US$340 billion, equivalent to between 9% and 15% of their operating profits.

To keep pace as banking becomes increasingly digitized in Southeast Asia, OCBC was looking to utilize AI/ML to make more data-driven decisions to improve customer experience and mitigate risks.

Through Cloudera, OCBC built a data lake and an Enterprise Data Science platform in a private cloud environment to introduce a more resilient infrastructure and platform capable of managing projects with increasing volume, variety, and velocity of data, while also enabling real-time analytics. With Cloudera, OCBC’s Next Best Conversation platform was able to analyze real-time contextual data from customer conversations, resulting in a revenue increase.

While these are great proof points to demonstrate how business value can be driven by AI/ML, this was only made possible with trusted data.

Trusted Data is the Foundation of AIAccording to a Cloudera survey, Data Architecture and Strategy in the AI Era, 57% of APAC organizations are at least early-stage adopters of AI. Gartner’s predictions also estimate 80% of enterprises will adopt GenAI APIs and models or deploy GenAI-enabled applications in production environments by 2028.

With adoption growing, organizations need to understand that without good quality and trusted data, there is no way to implement AI. Unreliable data leads to unreliable AI models, like building a house on a foundation of sand that might shift and collapse. On the other hand, trusted data is like building a house on solid concrete by ensuring AI models are trained on traceable and accurate information, enabling them to establish patterns and deliver dependable results. This translates into sharper insights for better decision-making capabilities and improved business outcomes.

Data biases are another element banks need to take into account, as data can unknowingly harbor biases which may be amplified and skew insights – like rejecting loans submitted by eligible applicants. Trusted data practices reduce this risk by ensuring data sets are well-balanced and representative, fostering fairer and more ethical AI implementations.

Lastly, data security is paramount, especially in the finance industry. Highest standards of data governance prioritize robust security features – such as storage encryption, data access authorisation, and data stewardship – as well as access controls to protect sensitive customer information, mitigating the risk of breaches from either external threats, or carelessness and negligence from internal stakeholders.

Smarter and Safer Banking for the FutureWith the continuous evolution and adoption of AI and Generative AI (GenAI), the future of banking is rapidly transforming. Trusted data is paramount for banks to unlock the true potential of AI and ML, which will in turn provide personalized customer experiences, better fraud detection for enhanced security, and ultimately pave the way for a smarter and safer banking future for all.

Learn more about how Cloudera helped OCBC unlock business value with trusted data.

The post The Award Winning Formula: How Cloudera Empowered OCBC With Trusted Data To Unlock Business Value from AI appeared first on Cloudera Blog.

View Details

It’s not a surprise that in today’s challenging economic landscape, rising costs pose a significant threat to the telecommunications industry. Consider that in 2022, Bain Capital was predicting that Telcos would grapple with increased personnel and escalating operating costs due to inflation. And here we are.

To combat these challenges, telcos must proactively seek opportunities to streamline operations and optimize revenue streams. From embracing automation to digital transformation initiatives, data plays a significant role in powering cost control strategies. According to consulting firm BCG’s 2024 Telco Value Creators Report, leading telcos are “radically optimizing costs through next-generation network architecture and core-to-cloud transformation” as well as “exploring ways to deploy GenAI (generative AI) that will transform each step in the industry’s value chain.”

Adapting to these economic shifts is crucial for telcos not only to survive but also to thrive in an increasingly competitive market. With the strategic use of open-source solutions and generative AI, the industry can not only implement cost-effective approaches but also pave the way for enhanced efficiency and scalability.

“Because gen AI democratizes access to powerful capabilities, any telco—a small operator or large incumbent—can reshape customer expectations and its organizational efficiency. In doing so, they can potentially narrow previously unassailable competitive advantages and overturn long-standing barriers to growth.” McKinsey

How Cloudera powers cost-control strategiesWith Cloudera’s experience delivering data and AI into the telecommunications industry, we conducted an internal assessment as to the areas where we have jointly built further value. Here are ten areas where Cloudera has enabled telcos to leverage open source and generative AI to implement cost control measures.

Ten Strategies for Success1. Reduce cloud costs by running large regular data workloads on-premThe public cloud is flexible, agile, accessible, and a great option for your business, looking to innovate fast and develop new ideas. The costs can increase and decrease, but as new workloads mature, that variability dies away. As the Rakuten CMO Geoff Hollingworth has said, “[a]s soon as a workload gains known predictable load, move it to private cloud for simple economic reasons.” Most telecom companies experience “sticker shock” when public cloud operations go into full production, and seek FinOps and other solutions to help – careful workload management can accelerate cost reduction by offering more efficient alternatives.

2. Deploy Flink for high volume data streamingOne major carrier is ingesting 3PB every single day into its Cloudera environment, mostly made up of network data, logs and telemetry from its extensive network across the USA. That’s 3,000 terabytes, every single day. To move that volume of data, old-school batch file processing simply doesn’t work, and real-time streaming is imperative. Flink, supported by Cloudera for many years with SQL Stream Builder, is one of the latest innovations in the Apache stable and is designed to respond to higher volume and velocity demands, while significantly lowering the cost of high-volume data operations. Combined with Kafka and Nifi, open-source data streaming operates at carrier grade to seamlessly engage at carrier-grade network scale.

3. Network anomaly detection as the grounding for network automationAs telcos seek to reduce the cost of operations, the network persists as its highest-cost domain. Automating network operations remains the nirvana for the industry, and to a large extent that begins in service assurance with network anomaly detection. While traditional tools like fault and performance management have their uses, automating anomaly detection in network streams can trigger demands for configuration changes or capacity reallocation to offset potential issues, and reduce the incidence or impact of network degradations. TIM Sparkle in Italy, one of the largest providers of global network in the world, has been using Cloudera CDP to support network anomaly detection for many years.

4. Generative AI for network anomaly description and resolutionNetworks are complicated. They are multi-vendor, they integrate Telecom and IT standards, with wireless, wireline and legacy models, and your customers expect a single view of their operations. Trying to find and address network defects can be time-consuming and difficult to understand, and often institutional knowledge can reside in a few key individuals, creating risks to the business should one of them leave. Generative AI can describe network issues in natural language as they arise, and potentially even potential causes and resolutions, based on historical performance, vendor communities and other documentation. One of the largest operators in the Middle East is now using Cloudera to explore how Generative AI can improve its operational performance by combining public services such as Whisper.ai and Hugging Face, both partners of Cloudera, with on-prem local data for hyper-contextualised applications.

5. Rationalise data mediation platforms with open source NiFiHistorically, data mediation platforms were used to collect data from the network and ‘translate’ that data into consumable transactions for billing, service assurance and security operations. They were often vertically focused, only serving telecom operations, as those businesses had ‘unique needs’ in terms of scale and complexity. Open source tools like Nifi can replace those systems at a fraction of the cost, and scale is no longer a challenge that can only be met by vertical appliance solutions – as LG uPlus found out. With Cloudera as the network data mediation layer for its entire wireline and 3G/4G/5G wireless service assurance functions, they are ingesting over 400TB of network telemetry per day.

6. Hybrid cloud data architecture for public cloud workload offloadWhile public cloud offers significant advantages over on-prem for data and AI, such solutions come with constraints. Regulatory, security and jurisdiction preferences and laws each may require that certain workloads or particular data sets cannot reside on public cloud infrastructure. Similarly, as mentioned above, certain workloads may vary in resource requirements when being designed, so it’s cheaper to avoid sizing to the maximum resource requirement in that early phase. However, these workloads can become much more predictable and consistent in terms of scale when they mature, making an on-prem solution more cost-effective. Over time, as workloads grow or shrink, and as regulations, privacy postures and company policies evolve, new options may become available. Having the capacity to move workloads around, from on-prem to public cloud, between public clouds, and back to the data center, can offer tools to make sure that the best option is always available for the workload. One major European telco had been working on a plan to migrate on-prem workloads to the public cloud, but when the Schrems II decision on GDPR and related matters caused their privacy posture to change. Cloudera allowed them to persist an architecture on-prem, but also to retain an option to migrate workloads to the public cloud as and when that posture changed again.

7. Re-platform appliance-based processes for lower TCOAppliances combine hardware and software into a dedicated solution for large-volume data workloads, and in the past, they provided performant solutions for specific workloads. They were expensive, but the bang for the buck was great. Today, however, with open source Data LakeHouse solutions from Cloudera, based on open source technologies like Apache Iceberg, those appliances now cost far more than is necessary, though telcos are often stuck with them as legacy, with too many dependent applications. The Cloudera platform can offload data from these appliances – which are usually charged for on a per-terabyte basis – while retaining the front-end interface and avoiding any disruption to the dependent applications. This can dramatically reduce the cost of operations as hundreds of Cloudera clients have discovered, including Saudi Telecom.

8. Redeploy internal open-source data and AI support teamsWhen building open source solutions for your internal operations, your IT support will have to develop capabilities to support those internal customers on both the applications that you have developed and on the open source components inside. As those applications scale, and require enhanced availability, Cloudera can provide enterprise-grade support for core application components and relieve pressure on internal commitments. Nokia faced a similar dilemma for its AVA suite of products, and rather than recruit specialists in Iceberg, HBase, Ozone, and other Apache open source components, partnered with Cloudera to deliver its telecom analytics solutions. Nokia focuses on the telecom vertical solutions, while Cloudera focuses on the data infrastructure powering it along.

9. Use high-volume network streaming data and generative AI to construct real-time customer profiles for hyper-personalised agent responseJust as automation is required in the network, so too is automation required in customer support. Traditionally, customer profiles derive from BSS data, focusing mainly on payment, interaction and billing information. Network data can build layers of significant passive experience data, building nuance into each customer interaction that will move the needle on customer experience. More and more telecom operators are using Generative AI running on Cloudera to dynamically translate and interpret voice interactions with real-time recommendations, sentiment analysis and CSR guidance. MTN has been combining mobile money and telecom network data to drive digital campaigns to even greater levels of targeting across Africa, and in one recent case improved campaign uptake by 186%.

10. Use Cloudera Data Flow to centralize data distribution and minimize data redundancyWhile there is no shortage of data in telecom operations, neither is there any shortage of demand for that data. Network operations and customer management are both significant consumers, but within those groups, there are multiple data consumers, and on top of that there are B2B, finance, IT, fleet, channels and countless other groups all looking for data sets, and often the same data. This can lead to redundancy, unnecessary spending, and a loss of control over how data is being used – a real security risk. Some of the largest telcos in the world are using Cloudera Data Flow and Cloudera Observability – all based on open source technologies – to optimize their data and resource consumption and get their arms around its vast reservoirs of enterprise and network data, and cloud and data center resource allocations.

For almost twenty years, Cloudera has led the evolution of open source data and AI, and has been its leading innovator. With eighty of the top one hundred telecommunications service providers running Cloudera, and over 25 Exabytes under management – that’s as much as any of the hyperscalers – Cloudera is the industry powerhouse when it comes to high-end data and AI engineering.

Cloudera will be sponsoring this year’s TM Forum Digital Transformation World at Copenhagen, Denmark, June 18-20. Come and talk to us about your data workloads, we’d love to explore those with you! Click here to request a meeting, and we’ll be in touch!

The post Delivering Effective AI for Telecom Companies: Trusted, Open, Hybrid appeared first on Cloudera Blog.

View Details

We are excited to announce a tech preview of Cloudera AI Inference service powered by the full-stack NVIDIA accelerated computing platform, which includes NVIDIA NIM inference microservices, part of the NVIDIA AI Enterprise software platform for generative AI. Cloudera’s AI Inference service uniquely streamlines the deployment and management of large-scale AI models, delivering high performance and efficiency while maintaining strict privacy and security standards.

It integrates seamlessly with our recently launched AI Registry, a central hub for storing, organizing, and tracking machine learning models throughout their lifecycle.

Cloudera AI Registry: Centralized Model Management

By combining the AI Registry with advanced inference capabilities, Cloudera provides a comprehensive solution for modern MLOps, enabling enterprises to efficiently manage, govern, and deploy models of any size across public and private clouds.

The new AI Inference service offers accelerated model serving powered by NVIDIA Tensor Core GPUs, enabling enterprises to deploy and scale AI applications with unprecedented speed and efficiency. Additionally, by leveraging the NVIDIA NeMo platform and optimized versions of open-source LLMs like LLama3 and Mistral models, enterprises can take advantage of the latest advancements in natural language processing, computer vision, and other AI domains.

Cloudera AI Inference: Scalable and Secure Model Serving

Key Features of Cloudera AI Inference service:* Hybrid cloud support: Run workloads on premises or in the cloud, depending on specific needs and requirements, making it suitable for enterprises with complex data architectures or regulatory constraints. * Platform-as-a-Service (PaaS) Privacy: Enterprises have the flexibility to deploy models directly within their own Virtual Private Cloud (VPC), providing an additional layer of protection and control. * Real-time monitoring: Gain insights into the performance of models, enabling quick identification and resolution of issues. * Performance optimizations: Up to 3.7x throughput increase for CPU-based inferences and up to 36x faster performance for NVIDIA GPU-based inferences. * Scalability and high availability: Scale-to-zero autoscaling and HA support for hundreds of production models, ensuring efficient resource management and optimal performance under heavy load. * Advanced deployment patterns: A/B testing and canary rollout/rollback allow gradual deployment of new model versions and controlled measurement of their impact, minimizing risk and ensuring smooth transitions. * Enterprise-grade security: Service Accounts, Access Control, Lineage, and Audit features maintain tight control over model and data access, ensuring the security of sensitive information.

The tech preview of the Cloudera AI Inference service provides early access to these powerful enterprise AI model serving and MLOps capabilities. By combining Cloudera’s data management expertise with cutting-edge NVIDIA technologies, this service enables organizations to unlock the potential of their data and drive meaningful outcomes through generative AI. With its comprehensive feature set, robust performance, and commitment to privacy and security, the AI Inference service is critical for enterprises that want to reap the benefits of AI models of any size in production environments.

To learn more about how Cloudera and NVIDIA are partnering to expand GenAI capabilities with NVIDIA microservices, read our recent press release.

The post Cloudera Introduces AI Inference Service With NVIDIA NIM appeared first on Cloudera Blog.

View Details

In an era where artificial intelligence (AI) is reshaping enterprises across the globe—be it in healthcare, finance, or manufacturing—it’s hard to overstate the transformation that AI has had on businesses, regardless of industry or size. At Cloudera, we recognize the urgent need for bold steps to harness this potential and dramatically accelerate the time to value for AI applications. It’s with this vision for the future and a commitment to continually enhancing our customers’ experiences that we announce the latest milestone in our journey: the acquisition of Verta’s Operational AI platform.

Why Verta? Cloudera is a leader in data management and enterprise AI. With 25% of the world’s data under management, it’s a mandate for us to shape the future for our customers and partners by innovating at the speed of business. This acquisition isn’t just about growth; it’s about deepening our IP and adding even more talent to better serve our customers with unmatched expertise and innovative solutions. The Verta platform will expand Cloudera’s enterprise AI portfolio and meet the growing demand across our global user base so that we can accelerate innovation for our customers. Leveraging Verta’s IP, we will simplify the process of bolstering customers’ private datasets to build custom retrieval-augmented generation (RAG) and fine-tuning ​applications​. As a result, ​developers — regardless of their expertise in machine learning — will be able to develop and optimize business-ready large language models (LLMs).

Furthermore, Verta’s technology, such as model catalog, model development capabilities, model monitoring and AI governance tools, combined with Cloudera’s hybrid platform for data, analytics and AI powered by our open data lakehouse and data services for MLOps and more to modernize data architectures, can unlock new possibilities for our diverse global customer base. This powerful combination will ensure our customers’ data estates and data architecture is well prepared and future-proofed to handle increased demands for data ingestion, data curation, data conflation, data quality and governance for actionable insight with our enterprise AI apps on our open data lakehouse.

Why now? It’s been a banner year for AI. The rise of Gen AI, of course, drove the dramatic surge in visibility and adoption. However, our customers asking what will keep AI relevant and accelerate its adoption for measurable business impact. Cloudera is setting a new pace in the race for AI adoption by strengthening our team to deliver results and accelerating our AI model lifecycle capabilities so that our customers can operationalize AI faster and drive forward with strategic initiatives that ultimately automate operations to reduce costs and focus on revenue generation.

Through this acquisition, Cloudera is not just preparing for the future – we are actively shaping it. Our commitment to integrating cutting-edge technologies with our solutions ensures that we remain customer obsessed, and our users are equipped to ride the next wave of enterprise AI and outpace the competition.

Read more about the announcement in our press release.

The post Acquisition of Verta’s Operational AI Platform Will Transform Cloudera’s AI Vision to Reality appeared first on Cloudera Blog.

View Details

The financial services industry is undergoing a significant transformation, driven by the need for data-driven insights, digital transformation, and compliance with evolving regulations. In this context, Cloudera and TAI Solutions have partnered to help financial services customers accelerate their data-driven transformation, improve customer centricity, ensure compliance with regulations, enhance risk management, and drive innovation.

Cloudera and TAI Solutions in Financial ServicesCloudera has a strong presence in the financial services sector, with 82% of the largest global banks, four of the top five stock exchanges, eight out of the top ten wealth management firms, and all four of the top credit card networks among its customers. Additionally, Cloudera has over 25 central banks and a dozen financial regulators as customers, providing it with valuable insights into the innovations in the sector.

TAI Solutions provides IT services and solutions to major players in the financial services industry, particularly in the banking and insurance sectors. The organization designs, develops, and manages IT processes, infrastructures, and applications tailored for banks and insurance companies to build and enhance their digital capabilities. Its expertise spans cloud enablement, modern application development, BigData and ML, Cloud Journey, DevOps, Microservices, platform, and architecture changes for financial services firms.

TAI Solutions’ Partnership with ClouderaTAI Solutions has a strategic partnership with Cloudera, leveraging Cloudera’s enterprise data management solutions to provide data-driven insights and digital transformation services to clients, particularly in the financial services industry. They offer strategic advisory, implementation services, and support around Cloudera products, utilizing Cloudera’s Enterprise Data Hub to create a unified platform for clients to store, access, and analyze unlimited data from multiple frameworks securely and efficiently.

Common Business Use CasesFinancial service customers use Cloudera to improve customer experience, manage risk, prevent financial crime, drive operational efficiency, and streamline compliance with regulations. Key use cases include customer journey/customer 360, regulatory compliance, financial crime prevention, risk management, market risk, credit risk, liquidity risk, operational risk, systemic risk, climate risk, intraday risk management, finance, integrated risk and finance view, treasury management, advanced analytics, and emerging technology.

A Q&A With Cloudera and TAI Solutions:What are some real-life success stories of financial services customers that TAI Solutions worked on, what Cloudera services were implemented, and what were the business outcomes achieved?

• Christian Simonelli, of TAI Solutions: One of Italy’s major banks, working with TAI, reorganized their existing data lifecycle management processes to explore advanced machine learning scenarios, streaming analytics, and data lineage capabilities. They deployed a proof-of-concept version of CDP Private Cloud and CDP Public Cloud, facilitating the client’s exploration of Cloudera’s hybrid cloud functionalities and a new data model. The client opted to adopt Kafka and Flink with Iceberg on Cloudera Private Cloud for streaming analytics scenarios and Cloudera Machine Learning and Data Warehouse on CDP Public Cloud for machine learning model development and data visualization applications.

What are some of the reasons that TAI Solutions’ customers choose Cloudera?

• Christian: TAI Solutions’ customers choose Cloudera due to its adaptability to specific reference cases, its open-source and market standard solutions, comprehensive suite of data services, and successful adoption in the cloud, offering continuity with on-premises solutions in terms of governance and interface. The decision to implement the SDX layer, capable of abstracting technology present on-premises and in the cloud, has been successful in realizing hybrid on-premises/cloud solutions without the need to modify organizational processes.

Regulation and risk are a big focus for financial institutions. Can you elaborate a bit more on that?

• Joe Rodriguez, Sr. Managing Director Financial Services for Cloudera: Regulations like Basel IV, the EU AI Act, DORA, GDPR, and ESG regulations require more transparency, better controls, and exponentially more data and compute, presenting a complex challenge for banks, particularly around data management. Implementing a modern data architecture is vital to compliance, and those banks that can underpin compliance with such an architecture will be well prepared.

What are some of the business use cases financial services customers are focused on to use AI?

• Joe: There are promising use cases for traditional ML/AI in financial services, including risk management, credit scoring and loan underwriting, anti-money laundering, and process automation. Generative AI also provides a range of new opportunities, such as personalized customer experiences, automated content creation, risk and compliance analysis, and trading and portfolio optimization.

Helping Financial Services Customers Take on Data Analytics and AIThe partnership between Cloudera and TAI Solutions is well-positioned to help financial services customers navigate the complexities of data analytics, ML/AI, and GenAI, while ensuring compliance with evolving regulations. As the financial services market continues to grow and evolve, the importance of data-driven insights and digital transformation will only increase, making partnerships like this crucial for the sector’s future success.

Learn more about how Cloudera’s partner ecosystem can support your most challenging use cases.

The post Bringing Financial Services Business Use Cases to Life: Leveraging Data Analytics, ML/AI, and Gen AI appeared first on Cloudera Blog.

View Details

Cloudera customers run some of the biggest data lakes on earth. These lakes power mission-critical, large-scale data analytics and AI use cases—including enterprise data warehouses. Nearly two years ago, Cloudera announced the general availability of Apache Iceberg in the Cloudera platform, which helps users avoid vendor lock-in and implement an open lakehouse. With an open data lakehouse powered by Apache Iceberg, businesses can better tap into the power of analytics and AI.

One of the primary benefits of deploying AI and analytics within an open data lakehouse is the ability to centralize data from disparate sources into a single, cohesive repository. By leveraging the flexibility of a data lake and the structured querying capabilities of a data warehouse, an open data lakehouse accommodates raw and processed data of various types, formats, and velocities. This unified data environment eliminates the need for maintaining separate data silos and facilitates seamless access to data for AI and analytics applications.

Here’s what implementing an open data lakehouse with Cloudera delivers:

• Integration of Data Lake and Data Warehouse: An open data lakehouse brings together the best of both worlds by integrating the storage flexibility of a data lake with the query performance and structured querying capabilities of a data warehouse.
• Openness: The term “open” in open data lakehouse signifies interoperability and compatibility with various data processing frameworks, analytics tools, and programming languages. This openness promotes collaboration and innovation by empowering data scientists, analysts, and developers to leverage their preferred tools and methodologies for exploring, analyzing, and deriving insights from data. Whether it’s traditional SQL-based querying, advanced machine learning algorithms, or complex data processing workflows, an open data lakehouse provides a flexible and extensible platform for accommodating diverse analytics workloads.
• Scalability and Flexibility: Like traditional data lakes, an open data lakehouse is designed to scale horizontally, accommodating large volumes of data from diverse sources. It provides flexibility in storing both raw and processed data, allowing organizations to adapt to changing data requirements and analytical needs. As data volumes grow and analytical needs evolve, organizations can seamlessly scale their infrastructure horizontally to accommodate increased data ingestion, processing, and storage demands. This scalability ensures the data lakehouse remains responsive and performant, even as data complexity and usage patterns change over time.
• Unified Data Platform: An open data lakehouse serves as a unified platform for data storage, processing, and analytics, eliminating the need for maintaining separate data silos and ETL (Extract, Transform, Load) processes. Deploying AI and analytics within an open data lakehouse promotes data democratization and self-service analytics, empowering users across the organization to access, analyze, and derive insights from data autonomously. By providing a unified and accessible data platform, organizations can break down data silos, democratize access to data and analytics tools, and foster a culture of data-driven decision-making at all levels. This democratization of data and analytics enhances organizational agility and competitiveness and promotes a more collaborative and data-literate workforce.
• Support for Modern Analytics Workloads: With support for both SQL-based querying and advanced analytics frameworks (e.g., machine learning, graph processing), an open data lakehouse caters to a wide range of analytics workloads, from ad-hoc querying to complex data processing and predictive modeling.

Open data lakehouse architecture represents a modern approach to data management and analytics, enabling organizations to harness the full potential of their data assets while embracing openness, scalability, and interoperability.

Learn more about the Cloudera Open Data Lakehouse here.

The post Unify your data: AI and Analytics in an Open Lakehouse appeared first on Cloudera Blog.

View Details

Behind every business decision, there’s underlying data that informs business leaders’ actions. As the market landscape across verticals from financial services to healthcare and manufacturing grows increasingly competitive, those decisions need to happen ever faster and to make them, businesses need to rely on data to reveal insights quickly, as near-real-time as possible. Delivering the most business value possible is directly linked to those decisions and the data and insights that inform them.

It’s not enough for businesses to implement and maintain a data architecture. The unpredictability of market shifts and the evolving use of new technologies means businesses need more data they can trust than ever to stay agile and make the right decisions. To tackle that goal, data and analytics leaders need to adopt modern data architectures that deliver greater flexibility and visibility and serve as a blueprint for accelerating the process of gathering insights and value from data.

Modern Data Architectures are Ready for the FutureThere is an important distinction between data architecture and modern data architecture. Data architecture is what defines the structures and systems within an organization responsible for collecting, storing, and accessing data, along with the policies and processes that dictate how data is governed. When we talk about modern data architecture, there are several unique benefits to this kind of approach. A modern data architecture simplifies data access across organizations and unlocks data from their silos, making it easier to know and understand that data. This ensures that the right, trusted data is able to be used to feed AI and analytics effectively.

Modern data architectures deliver key functionality in terms of flexibility and scalability of data management. This form of architecture can handle data in all forms—structured, semi-structured, unstructured—blending capabilities from data warehouses and data lakes into data lakehouses. They also incorporate functionalities like data processing, data integration, and data governance as part of the overall structure for data systems.

Making the Leap from Data Architecture to Modern Data ArchitectureThere are a multitude of paths businesses can take to successfully implement a modern data architecture. Among those paths to implementation, here are three key approaches to strengthening and modernizing data architecture:

Data Lakehouses: This offers businesses a combined approach, bringing together key pieces of data lakes and data warehouses to deliver the flexibility of a data lake with the performance of a data warehouse. A data lakehouse eliminates the need for data duplication and synchronization, helping keep costs down while building trust with a single source of truth. In particular, open data lakehouses are well-suited for a modern data architecture, delivering self-service access to analytics that help organizations generate insights quickly and manage data on public or private cloud.

Data Fabric: This is a type of data architecture intended to help automate the unlocking of data sources at scale while also providing self-service access to the data that still adheres to both governance and data access rules. In a hybrid cloud environment, data fabric helps organizations bring all of that disparate data together into one single source and gives businesses a single point of access for data stored in multiple systems without exposing data to undue compliance risk.Data Mesh: A data mesh involves decentralization of data and analytics where each domain is fed by its own data fabric and has its own data lakehouse. The idea is to create a “mesh” of data products connected to various parts of an organization versus having a central repository of data and analytics controlled by an individual IT team that is responsible for the needs of multiple domains. The visibility, accessibility, and insights that derive from a modern data architecture feed directly into a business’s ability to adapt quickly, handle change, and make well-informed decisions based on real-time data—all of which play an important part in future-proofing data infrastructure for the long term.

Learn more about how Cloudera can help you achieve a modern data architecture.

The post Laying the Foundation for Modern Data Architecture appeared first on Cloudera Blog.

View Details

In modern enterprises, the exponential growth of data means organizational knowledge is distributed across multiple formats, ranging from structured data stores such as data warehouses to multi-format data stores like data lakes. Information is often redundant and analyzing data requires combining across multiple formats, including written documents, streamed data feeds, audio and video. This makes gathering information for decision making a challenge. Employees are unable to quickly and efficiently search for the information they need, or collate results across formats. A “Knowledge Management System” (KMS) allows businesses to collate this information in one place, but not necessarily to search through it accurately.

Meanwhile, ChatGPT has led to a surge in interest in leveraging Generative AI (GenAI) to address this problem. Customizing Large Language Models (LLMs) is a great way for businesses to implement “AI”; they are invaluable to both businesses and their employees to help contextualize organizational knowledge.

However, training models require huge hardware resources, significant budgets and specialist teams. A number of technology vendors offer API-based services, but there are doubts around security and transparency, with considerations across ethics, user experience and data privacy.

Open LLMs i.e. models whose code and datasets have been shared with the community, have been a game changer in enabling enterprises to adapt LLMs, however pre-trained LLMs tend to perform poorly on enterprise-specific information searches. Additionally, organizations want to evaluate the performance of these LLMs in order to improve them over time. These two factors have led to development of an ecosystem of tooling software for managing LLM interactions (e.g. Langchain) and LLM evaluations (e.g. Trulens), but this can be much more complex at an enterprise-level to manage.

The SolutionThe Cloudera platform provides enterprise-grade machine learning, and in combination with Ollama, an open source LLM localization service, provides an easy path to building a customized KMS with the familiar ChatGPT style of querying. The interface allows for accurate, business-wide, querying that is quick and easy to scale with access to data sets provided through Cloudera’s platform.

The enterprise context for this KMS can be provided through Retrieval-Augmented Generation (RAG) of LLMs, to help contextualize LLMs to a specific domain. This allows the responses from a KMS to be specific and avoids generating vague responses, called hallucinations.

The image above demonstrates a KMS built using the llama3 model from Meta. This application is contextualized to finance in India. In the image, the KMS explains that the summary is based on Indian Taxation laws, even though the user has not explicitly asked for an answer related to India. This contextualization is possible thanks to RAG.

Ollama provides optimization and extensibility to easily set up private and self-hosted LLMs, thereby addressing enterprise security and privacy needs. Developers can write just a few lines of code, and then integrate other frameworks in the GenAI ecosystem such as Langchain, Llama Index for prompt framing, vector databases such as ChromaDB or Pinecone, evaluation frameworks such as Trulens. GenAI specific frameworks such as Chainlit also allow such applications to be “smart” through memory retention between questions.

In the picture above, the application is able to first summarize and then understand the follow-up question “can you tell me more”, by remembering what was answered earlier.

However, the question remains: how do we evaluate the performance of our GenAI application and control hallucinating responses?

Traditionally, models are measured by comparing predictions with reality, also called “ground truth.” For example if my weather prediction model predicted that it would rain today and it did rain, then a human can evaluate and say the prediction matched the ground truth. For GenAI models operating in private environments and at-scale, such human evaluations would be impossible.

Open source evaluation frameworks, such as Trulens, provide different metrics to evaluate LLMs. Based on the asked question, the GenAI application is scored on relevance, context and groundedness. Trulens therefore provides a solution to apply metrics in order to evaluate and improve a KMS.

Image Credit: Trulens.org

The picture above demonstrates saving the earlier metrics in the Cloudera platform for LLM performance evaluation

With the Cloudera platform, businesses can build AI applications hosted by open-source LLMs of their choice. The Cloudera platform also provides scalability, allowing progress from proof of concept to deployment for a large variety of users and data sets. Democratized AI is provided through cross-functional user access, meaning robust machine learning on hybrid platforms can be accessed securely by many people throughout the business.

Ultimately, Ollama and Cloudera provide enterprise-grade access to localized LLM models, to scale GenAI applications and build robust Knowledge Management systems.

Find out more about Cloudera and Ollama on Github, or sign up to Cloudera’s limited-time, “Fast Start” package here.

The post Building and Evaluating GenAI Knowledge Management Systems using Ollama, Trulens and Cloudera appeared first on Cloudera Blog.

View Details

Hybrid cloud plays a central role in many of today’s emerging innovations—most notably artificial intelligence (AI) and other emerging technologies that create new business value and improve operational efficiencies. But getting there requires data, and a lot of it. More than that, though, harnessing the potential of these technologies requires quality data—without it, the output from an AI implementation can end up inefficient or wholly inaccurate. Something that Cloudera and Foundry research found 36% of IT leaders said ranked as a top challenge.

To attain that level of data quality, a majority of business and IT leaders have opted to take a hybrid approach to data management, moving data between cloud, on-premises -or a combination of the two – to where they can best use it for analytics or feeding AI models. The challenge is not solved, though, by simply adopting a hybrid cloud infrastructure. Meaningful results, and a scalable, flexible data architecture demand a ‘true’ hybrid cloud approach to data management.

What do we mean by ‘true’ hybrid? Let’s dive deeper.

Reshaping Data Management in the CloudThere are a handful of factors that elevate hybrid cloud strategy to the level of ‘true’ hybrid, but for our purposes, we’ll zero in on some of the most important facets of this approach. One of the main pieces that separates ‘true’ hybrid is the ability to operate as a single platform across both data center and cloud, as well as at the edge. Although a step in the right direction, supporting multiple clouds alone does not provide a complete solution for organizations with an existing data center investment or those needing some data and analytics to remain on-premises. In a true approach, data and workloads can move freely and multi-directionally between environments. This form of hybrid also goes a level deeper than one may find in a standard hybrid cloud, accounting for the entirety of the data lifecycle, whether that’s the point of ingestion, warehousing, or machine learning—even when that end-to-end data lifecycle is split between entirely different environments.

Data comes in many forms. Understanding and handling structured, semi-structured, or unstructured data is a core part of what makes a hybrid approach valuable. ‘True’ hybrid incorporates data stores that are capable of maintaining and harnessing data, no matter the format. One element that is essential to achieving ‘true’ hybrid is open data lakehouses, capable of managing those vast swaths of unstructured or semi-structured data and making it available in the right environments for analysis or AI models. These lakehouses greatly reduce complexity that can stack up in traditional environments over time, resulting in a much simpler experience and workflow for employees.

Adopting the right hybrid cloud approach opens up visibility and boosts data access which will, in turn, generate business value. Take, for example, our work supporting LGIM, empowering the company to develop the flexibility to respond to market shifts, tap into new opportunities, and bolster ESG efforts. Working together, Cloudera helped the company build a strong foundation to generate even more value from its data for the future.

Building a Unified Approach to Data GovernanceBut all that data can only be leveraged properly if the people in charge can see it, understand it, and know quickly where and how it’s being deployed. The right approach includes implementing a solution that delivers a single, common control plane no matter where data is, what state it’s in, or how it’s being deployed in analytics. That also means ‘true’ hybrid allows organizations to avoid sprawling sets of tools that only work in specific environments. With a single, consistent view of all data, this hybrid approach provides a clearer picture of performance, resource allocation, and expenses end-to-end for data and analytics. That cross-functionality inherent in ‘true’ hybrid cloud also brings a crucial boost to data governance. With data constantly in motion in a hybrid environment, this approach helps increase visibility, making it easier to apply rules to data and maintain compliance as it moves from where it is born to where it is needed.

Embracing a ‘true’ hybrid cloud approach isn’t just about solving immediate challenges or even solely about improving data governance. Change is a constant when it comes to the state of data architecture, and the pace of innovation is only getting faster as time goes on. With that, we’re seeing the importance of ‘true’ hybrid cloud as organizations begin to shift, favoring data architecture that’s highly flexible, scalable, and adaptable.

Learn more about how Cloudera can support your hybrid data management needs.

The post What Separates Hybrid Cloud and ‘True’ Hybrid Cloud? appeared first on Cloudera Blog.

View Details

Businesses today face unique challenges, whether it’s with hybrid cloud, AI, data analytics, or all of the above. Delivering solutions that can address those challenges effectively requires a robust ecosystem of partnerships. At the center of this critical ecosystem is the partner marketing team at Cloudera, who work tirelessly in pursuit of excellence for customers—and as a result, we’re proud to share that six of our very own Clouderans have been recognized by CRN as part of this year’s Women of the Channel. Each of these women play an integral role in fostering Cloudera’s partner ecosystem and delivering on the most pressing customer needs.

Every year, CRN publishes its Women of the Channel project to honor female executives for their accomplishments during the previous year, as well as the far-reaching impact these women have on the broader technology industry. Winners of this award are women who have played a prominent role in the IT channel ecosystem with either a channel organization, solutions provider, or distributor.

Let’s celebrate Cloudera’s own Women of the Channel and learn a bit more about the winners.

Natascha Lee, Head of Global Partner & Alliance Marketing – Natascha—a six-time winner of the Women of the Channel awards—boasts over 20 years of channel marketing experience across multiple B2B technology firms, including her time here at Cloudera. In her position, Natascha has led the transformation of Cloudera’s already high-performing Global Partner & Alliance Marketing team, weaving together the art and science of marketing to drive demand. She and her team’s work has been instrumental in aggressively pursuing revenue goals and growing market share, as well as has directly contributed to major growth within the partner marketing program. During her time with Cloudera, Natascha has helped facilitate major partner integrations with organizations like NVIDIA, among others, as the company continues to strengthen its offerings.

Valaretha Brown, Senior Partner Marketing Manager – Valaretha has delivered impactful results and growth opportunities through her work executing creative strategic marketing programs at Cloudera. A three-time winner of the Women of the Channel awards, Valaretha regularly aligns with trusted advisor partner contacts to deliver innovative demand-generation programs. Valaretha works collaboratively with complex, highly matrixed partner organizations to uncover strategic initiatives that make an immediate impact. As a part of that role, Valaretha has been at the heart of incorporating partners in marketing-wide programs, driving significant new leads, and delivering cobranded, impactful content for joint campaigns.

Lan Chu, Senior Partner Marketing Manager – As Senior Marketing Manager in Partner Marketing at Cloudera, Lan leverages her more than 20 years of expertise in marketing strategy and demand generation to work closely with cross-functional teams, partners, and vendors to deliver channel-focused marketing programs that amplify sales and drive business success for Cloudera’s entire partner ecosystem. In fact, you can learn more about Lan and her work with the partner marketing team here. Being named as a Women of the Channel recipient is a reflection of Lan’s dedication to elevating partner marketing programs at Cloudera and her deep understanding of what the needs of customers truly are.

Naomi Gravelding,Partner Marketing Manager – As a veteran of software startups, Naomi has a deep understanding of the space and the skillset to rapidly react to changes in the market. Her work has helped to expand partner marketing communications initiatives and fold in new and improved strategies and processes to boost awareness of Cloudera’s partner ecosystem and extend the company’s reach with current and prospective partners. Naomi built Cloudera’s first LinkedIn Partner Showcase page, which allows partners to engage directly with the broader partner ecosystem. Through her efforts to grow the partner marketing program, Naomi has helped increase reach among partners and prospects and establish clear guidelines for effective digital communications.

Janet O’Sullivan, Senior Partner Marketing Manager – Janet’s dynamic and far-reaching role managing partners across four continents has had a significant impact in fostering growth, developing a strong partner ecosystem, and generating significant ROI, all in service of delivering on customers’ most important needs. Her efforts have been highly successful, generating a 50% growth of the partner pipeline in her region collaborating with partners and internal stakeholders on more than 80 activities. Those campaigns ranged from some of the biggest industry events to smaller, much more focused, account-based marketing (ABM) activities. She has also played an essential role in driving the partner marketing team’s marketing development funds within the partner ecosystem.

Caitriona Snell, Senior Manager, AI/ISV Partner Marketing and Marketing Operations – Caitriona is another integral member of Cloudera’s partner marketing team, driving operational excellence and managing long-term projects that increase global revenue opportunities in the channel. Caitriona has established herself as a leader and even acts as a subject matter expert on marketing initiatives for other teams. As a winner of Women of the Channel, she is being recognized for her role in piloting a lead distribution program across multiple geographies that would ultimately lead to significant deal registrations and a vast number of new opportunities. She has also played an important role in connecting Cloudera technology and solutions to broader environmental, social, and governance (ESG) reporting needs—an aspect that has become increasingly important for customers and, likewise, important for channel partners to help address.

Learn more how Cloudera’s partner ecosystem can support your hybrid cloud journey.

The post Six Clouderans Earn CRN Women of the Channel Distinction appeared first on Cloudera Blog.

View Details

Apache Iceberg is vital to the work we do and the experience that the Cloudera platform delivers to our customers. Iceberg, a high-performance open-source format for huge analytic tables, delivers the reliability and simplicity of SQL tables to big data while allowing for multiple engines like Spark, Flink, Trino, Presto, Hive, and Impala to work with the same tables, all at the same time. The features within the Iceberg table format play an important role in making data architecture more effective.

Cloudera and Iceberg have become increasingly interconnected since it was integrated into the Cloudera platform in 2022. Leveraging Cloudera’s platform, powered by Iceberg, organizations can transform their data and analytics capabilities with open data lakehouses, making the most of data from across the entire enterprise with unique tools and no unnecessary data movement or transformations along the way. Building an open data lakehouse with Iceberg delivers significant benefits increasing self-service access, ease of use, flexibility, and delivering unified security and governance for all data. It accelerates the entire data lifecycle from streaming and ingestion to processing, analytics, and AI.

One of the reasons Cloudera integrated Iceberg was its openness, with engine agnostic development and very broad community support. This enables unbiased, accelerated innovation and, for the first time, provides a common standard for all data in the organization, irrespective of the processing engine. It’s also one of the reasons why we see such a wide adoption of Iceberg in the market. The Iceberg community is deeply important to us at Cloudera.

As it continues to grow, we continue to invest heavily in providing opportunities for learning, networking, and understanding exactly what this technology can do to benefit organizations and their data and analytics needs. With that in mind, we’re excited to share that Cloudera is a sponsor of this year’s Iceberg Summit 2024. The event, taking place virtually from May 14-15, features a variety of speaking sessions from experts, community members, and practitioners who will share insights and best practices for leveraging the full power of Iceberg.

This virtual event brings together a wide range of attendees for two days filled with technical talks, breakout sessions, and panels that cover the real-world experiences of data practitioners and developers working with Apache Iceberg as their table format. From data pipelines into Iceberg to data governance, the event will hit on a broad range of topics surrounding Iceberg.

Register now and join us at the Iceberg Summit or follow the link to learn more about Cloudera’s Iceberg integration.

The post Join us at the Iceberg Summit 2024 appeared first on Cloudera Blog.

View Details

In the last couple of years, AI has launched itself to the forefront of technology initiatives across industries. In fact, Gartner predicts the AI software market will grow from $124 billion in 2022 to $297 billion in 2027. As a data platform company, Cloudera has two very clear priorities. First, we need to help customers get AI models based on trusted data into production faster than ever. And second, we need to build AI capabilities into Cloudera to give more people access to data-driven insights for their everyday roles.

At our recent Cloudera Now virtual event, we announced three new capabilities that support both of our AI priorities: An AI-driven SQL assistant, a Business Intelligence (BI) chatbot that converses with your data, and an ML copilot that accelerates machine learning development. Let’s take a deeper dive into how these capabilities accelerate your AI initiatives and support data democratization.

SQL AI Assistant: Your New Best Friend

Writing complex SQL queries can be a real challenge. From finding the right tables and columns to dealing with joins, unions, and subselects, then optimizing for readability and performance, and doing all of that while taking into account the unique SQL dialect of the engine, it’s enough to make even the most seasoned SQL developer’s head spin. And at the end of the day, not everyone who needs data to be successful in their day-to-day work is an SQL expert.

Imagine, instead, having a domain expert and a SQL guru always by your side. That’s exactly what Cloudera’s SQL AI assistant is. Users simply describe what they need in plain language, and the assistant will find the relevant data, write the query, optimize it, and even explain it back in easy-to-understand terms.

Under the hood, the assistant uses advanced techniques like prompt engineering and retrieval augmented generation (RAG) to truly understand your database. It works with many large language models (LLMs), whether they are public or private, and it effortlessly scales to handle thousands of tables and users simultaneously. So whether you’re under pressure to answer critical business questions or just tired of wrestling with SQL syntax, the AI assistant has your back, enabling you to focus on what really matters – getting insights from your data.

BI Chatbot: Your Data’s New Best Friend

BI dashboards are undeniably useful, but they often only tell part of the story. To gain meaningful and actionable insights, data consumers need to engage in a conversation with their data, and ask questions beyond simply the “what” that a dashboard typically shows. That’s where Cloudera’s BI chatbot comes into play.

The chatbot resides directly within your dashboard, ready to answer any question you pose. And when we say “any question,” we mean it. Why are sales down in the Northeast? Will this trend continue? What actions should we take? The chatbot leverages the context of the data behind the dashboard to deliver deeper, more actionable insights to the user.

A written answer is a great way to start understanding your data, but let’s not forget the power of the visuals in our dashboards and reports. The chatbot eliminates the burden of clicking through dropdowns and filters to find answers. Simply ask what you want to know, in plain language, and the chatbot will intelligently match it to the relevant data and visuals. It’s like having a dedicated subject matter expert right there with you, ready to dive deep into the insights that matter most to your business.

ML Copilot: Your Model’s New Best Friend

Building machine learning models is no easy feat. From data wrangling to coding, model tuning to deployment, it’s a complex and time-consuming process. In fact, many models never make it into production at all. But what if you had a copilot to help navigate all of the challenges related to deploying models in production?

Cloudera’s ML copilots, powered by pre-trained LLMs, are like having machine learning experts on call 24/7. They can write and debug Python code, suggest improvements, and even generate entire applications from scratch. With seamless integration to over 130 Hugging Face models and datasets, you have a wealth of resources at your disposal.

Whether you’re a data scientist looking to streamline your workflow or a business user eager to get an AI application up and running quickly, the ML copilots support the end-to-end development process and get models into production fast.

Elevate Your Data with AI Assistants

By embedding AI assistants for SQL, BI, and ML directly into the platform, Cloudera is simplifying and enhancing the data experience for every single user. SQL developers will be more efficient and productive than ever. Business analysts will be empowered to have meaningful, actionable conversations with data, uncovering the “why” behind the “what.” Additionally, data scientists will be empowered to bring new AI applications to production faster and with greater confidence.

For more information on these features and our AI capabilities, visit our Enterprise AI page. When you’re ready, you can request a demo at the bottom of the page to see how these capabilities can work in the context of your business.

The post Introducing Cloudera’s AI Assistants appeared first on Cloudera Blog.

View Details

The Gartner Data and Analytics Summit in London is quickly approaching on May 13th to 15th, and the Cloudera team is ready to hit the show floor! The theme of this year’s summit, “Generating Value Together: Creating Synergies between Data, Analytics & AI,” could not have come at a better time as we push forward on our AI and analytics journey together. With the rapid pace of AI innovations, today’s leaders must connect, share, and learn from each other on how to best leverage AI technology to effectively analyze data to our advantage.

At the summit, attendees will hear from chief data analytics officers (CDAOs), as well as data, IT, and analytics leaders spearheading their AI-driven strategies to deliver significant value to their enterprises. We’re at a crucial point in time where trusted data is fundamental for driving new AI use cases, enabling real-time operations, and allowing enterprises to easily scale.

With that, let’s take a closer look at what the Cloudera team will be doing over the course of the event.

Taking AI from Pilot to Production at Enterprise ScaleHow well businesses implement emerging enterprise AI and ML use cases—predictive, generative, and beyond—will define the next wave of market leaders. While most enterprises are experimenting with AI, moving from pilots to full-scale AI solutions in production brings new challenges. Each initiative requires careful planning, domain knowledge, and a battle-ready enterprise data platform.

On the second day of the summit, May 14 at 11:00 am BST, Cloudera’s own Senior Director of Product Marketing, Wim Stoop, will be spearheading a conversation on capitalizing AI opportunities with our customers, data leaders from BT Group, OTP Bank, and eMAG. In his conversation titled AI from Pilot to Production at Enterprise Scale, learn from Wim and his expert panel on how they are capitalizing on this AI opportunity and identifying AI use cases for transformation.

Hybrid Horses for Courses: The Right Cloud for AI from Pilot to Production at ScaleLater, on May 14 at 12:40 pm BST, hear from Mark Samson, one of Cloudera’s solutions engineering directors, on whether a data center or cloud deployment is best for your organization’s data platform and architecture.

As far as data storage and processing resources go, there’s cloud, and then there’s your data center. The data for your AI projects can come from either, yet different stages of the development lifecycle require different resource profiles and scalability. In his talk titled Hybrid Horses for Courses: The Right Cloud for AI from Pilot to Production at Scale, Mark will discuss the pros and cons of the two infrastructure choices for each of the lifecycle stages, highlighting the key capabilities organizations need in a platform to be able to move between them seamlessly.

This year’s summit will be an exciting deep dive into the best way to achieve organizational goals with AI and human intelligence led by some of the world’s top leaders.

In addition to our speaking sessions, the Cloudera team will also be on the show floor at booth #211, engaging with fellow attendees and sharing insights into how we can work together to achieve enterprise AI at scale.

Learn more about what’s happening at this year’s summit and check out the full session lineup.

The post We’ll See You at the Gartner Data and Analytics Summit appeared first on Cloudera Blog.

View Details

Generative AI (GenAI) has taken the world by storm, promising to revolutionize industries and transform the way businesses operate. From generating creative content to automating complex tasks, the potential applications of GenAI are vast and exciting. However, implementing GenAI in an enterprise setting comes with its own set of challenges.

At Cloudera, we understand the complexities of enterprise GenAI adoption. Our extensive experience in data management and AI has given us unique insights into the factors that determine the success of GenAI initiatives.

Let’s take a closer look at these challenges, and dive into Cloudera’s three key pillars for successful implementation.

The Challenges of Enterprise Generative AI

Implementing GenAI in an enterprise environment is no small feat. Businesses face several significant challenges, including:

1. Data Integration: GenAI models require access to vast amounts of high-quality, diverse data. Enterprises often struggle to integrate their data silos and ensure secure, seamless access to necessary data.
2. Model Infrastructure Flexibility: The rapidly evolving landscape of GenAI models demands a flexible and modular infrastructure. Rigid infrastructures can hinder innovation and slow down the development and deployment of AI applications.
3. Security, Governance, and Transparency: GenAI introduces new security and governance challenges that extend beyond the data lake. Ensuring the security of models, managing AI-generated content, and maintaining transparency in AI decision-making processes are critical concerns.

Cloudera’s Three Pillars for Enterprise Generative AI Success

To help enterprises overcome these challenges and unlock the full potential of GenAI, there are three key pillars that define success:

1. Data as the Differentiator: An enterprise’s data is its most valuable asset and the key differentiator in GenAI. Leveraging their unique data, businesses can create highly customized and contextually relevant AI models that set them apart from competitors.
2. Flexibility in Model Selection and Deployment: Cloudera’s platform supports the flexibility to choose and easily switch between open-source and closed-source models, allowing businesses to adapt to their evolving needs. This flexibility ensures that enterprises can always leverage the best models for their specific use cases.
3. Comprehensive Security, Governance, and Lineage: Our platform provides a robust framework for security, governance, and lineage that extends beyond the data lake to encompass the entire GenAI ecosystem. This comprehensive approach ensures that businesses can trust their data and AI models, maintaining the highest standards of security and transparency.

Following these three pillars will be crucial to helping businesses overcome the emerging challenges that surround GenAI. Learn more about how Cloudera can support your GenAI implementation.

The post Navigating the Enterprise Generative AI Journey: Cloudera’s Three Pillars for Success appeared first on Cloudera Blog.

View Details

Organizations are challenged today to become both more data driven and more nimble to adapt quickly to changing conditions. These challenges are the driving forces behind much of their digital transformation or “modernization” efforts. Digital Transformation is defined as the process of integrating digital technology into all areas of a business to create and capture value in new ways, effectively “datifying” all processes while remaining agile enough to make continuous incremental improvements.

To support these transformation efforts, Cloudera is thrilled to announce that all Data-in-Motion capabilities will be available 2H 2024 as independently deployable Kubernetes operators. Now, organizations that strive to be more data-driven can take advantage of modern containerized services to accelerate innovation and scale efficiently.

Regardless of industry or use case, there are two key themes that always arise when executing on digital transformation strategies.

1. Data needs to be shared in real time so it can be embedded deeper into everyday operational processes across the organization that are working from the same ground-truth.
2. Applications, data, and supporting infrastructure must be broken down from the monolithic, tightly coupled architectures of the past into smaller building blocks that are independently modifiable, reconfigurable, and scalable.

So, the strategic capabilities that are needed to execute on innovation projects in support of digital transformation goals or scale them to the enterprise as they evolve are:

• The ability to capture, process, and distribute any and all data in real time to any user, application or system
• The ability to rapidly deploy, provision, manage, and scale efficiently across hybrid and multi-cloud environments.

This new release will deliver both of those strategic capabilities.

These offerings are also significant industry “firsts,” helping bring the power of open-source technology to the enterprise.

Cloudera Flow Management – Kubernetes Operator is the only commercially supported Kubernetes deployment for Apache NiFi while Cloudera Streaming – Kubernetes Operator is the only offering for both Apache Kafka and Apache Flink in Kubernetes form factor that can be run anywhere. These services provide a unique and powerful combination of streaming data movement, real-time processing, and streaming analytics with low code development experiences that make delivery of real-time pipelines efficient at massive scale.

Staying true to our commitment to deliver complete capabilities for streaming data across hybrid environments, this release represents a significant milestone for Cloudera and our enterprise customers. Customers who wish to build streaming pipelines can quickly deploy and manage scalable clusters that support their real-time data needs in any environment in the cloud or on premise. They can do this independent of the Cloudera Data Platform instances, giving greater speed and flexibility than ever before.

Customers choose Cloudera’s Data-in-Motion tools for numerous reasons including:

• Our breadth of integrated capability
• Open-source innovation with enterprise-grade delivery
• Low-code development experiences
• Efficient scalability

The value of our holistic approach to Data-in-Motion is an overall simplification and efficiency of pipeline architecture able to deliver actionable data across the enterprise for all real-time use cases and evolve with changing data or business requirements.

At the same time, many organizations choose Kubernetes as an enterprise standard deployment form factor for good reason. There is tremendous value just in having a standard – it is much easier to make an administrative team efficient and to manage resources while maintaining security when there is a single enterprise standard for them to work with. As far as container orchestration tools go, the industry has settled on Kubernetes as the de facto standard, meaning the market is full of IT professionals who are already proficient with Kubernetes. Kubernetes has earned that position due to a number of factors

• Commitment to open source
• Automation tools
• Scalability
• Cloud-agnostic flexibility.

Organizations that orchestrate Kubernetes containers take advantage of efficiencies in deployment and management.

Together, a holistic set of streaming services containerized as Kubernetes operators delivers the key strategic capabilities all organizations need to execute on their digital transformation goals and continually evolve over the long term. Cloudera Flow Management – Kubernetes Operator, and Cloudera Streaming – Kubernetes Operator bring a new level of hybrid portability to Data-in-Motion, which is critical for any organization operating in multiple clouds and data centers and has requirements around where data must live, such as data sovereignty laws in the EU. Deploying via these new operators as independent services improves innovation speed- development clusters can be set up in just minutes by any IT admin. From there, they can use their container orchestration tooling of choice to monitor, manage, and scale deployments.

There are economic reasons to deploy on-premises too. Many organizations are facing cost and latency challenges related to streaming data in the cloud. From a cost perspective, the sheer volume of streaming data, as well as ingress and egress fees and variable compute requirements all add up to high prices for streaming workloads in particular. On the performance side, network latency is an unavoidable reality of cloud computing, and in the case of streaming workloads, it can result in missed SLAs for performance. This new offering is the only supported Kubernetes operator for both Kafka and Flink, allowing our customers to reduce costs and latency by bringing the stream processing right to the data as opposed to pushing everything to the cloud!

An example of Data-in-Motion at scale could be a Cloudera customer delivering cyber security pipelines that continuously evolve. Our customer is able to collect, process, and filter log data from hundreds of thousands of distributed devices, streaming that data for high-speed ingestion into a cyber data lake for analysis much more efficiently than a SIEM-only approach. After first delivering this data in real time to threat analysts and dramatically reducing mean time to detection, the team was easily able to reconfigure existing pipelines to deliver similar data to systems reliability teams who needed it. As their needs evolved, they added stateful real-time processing to their pipelines to monitor for patterns that represented threats that their analysts can update with simple SQL queries against streaming data.

Other use cases that require the ability to orchestrate real-time data and evolve quickly are fraud prevention, supply chain optimization, cybersecurity, personalized offers, and generative AI, to name a few.

This is an important milestone for the open source community and an example of Cloudera’s continued commitment to delivering the industry’s most comprehensive set of streaming capabilities across hybrid environments to truly capture, process, and distribute any data anywhere.

For more information or to see a demo, we invite you to join our webinar on June 11th here

The post Accelerating Deployments of Streaming Pipelines – Announcing Data in Motion on Kubernetes appeared first on Cloudera Blog.

View Details

One of the worst-kept secrets among data scientists and AI engineers is that no one starts a new project from scratch. In the age of information there are thousands of examples available when starting a new project. As a result, data scientists will often begin a project by developing an understanding of the data and the problem space and will then go out and find an example that is closest to what they are trying to accomplish. This is a standard practice, but it has some key drawbacks that don’t always get discussed. This includes:

• There is no guarantee that the code you find is using best practices
• The credentials of a given author are often vague
• The environment may not be compatible
• Security and legal risks

With these issues in mind, Cloudera is thrilled to announce the release of Accelerators for ML Projects (AMPs). AMPs are fully built, end-to-end solutions that provide data scientists with a ready-to-go MVP for various AI use cases, significantly reducing development time. With a single click, AMPs build, deploy, and set up continuous monitoring of enterprise-ready machine learning (ML) applications.

Each AMP is a prototype that encapsulates industry-leading practices for tackling complex ML challenges. The workflow—from data ingestion and model training to model deployment—is meticulously defined within a YAML configuration file. This allows for seamless transitions, whether you’re running examples locally or deploying processes automatically in Cloudera Machine Learning.

Best of all, every AMP is fully open source. Even though they are easiest to deploy in Cloudera Machine Learning, each project provides a README with instructions on how to deploy in any environment—another reminder that Cloudera will always be committed to the open source community.

Cloudera’s AMP catalog provides three different types of AMPs for you to choose from. (1) AMPs built with Cloudera engineering, (2) AMPs from HuggingFace Spaces, and (3) AMPs built by community contributors.

Now, let’s dive into these 3 unique types of AMPs and how they can be used.

Cloudera Engineering AMPsAMPs built by Cloudera engineering provide the largest number of examples to choose from. These AMPs are built and supported by research teams that focus on the latest and greatest in AI and ML. They go through a rigorous testing and review process to guarantee that they provide the highest quality reference projects for our enterprise customers to choose from. These AMPs are also continuously reviewed and updated to maintain compatibility with new versions of Python and the various libraries they leverage.

One of our most popular AMPs in this catalog is the LLM Chatbot Augmented with Enterprise Data. This project demonstrates how to use the popular retrieval augmented generation (RAG) architecture to add enterprise context to the responses of a locally hosted large language model (LLM) using a hosted Milvus instance as a vector store. This is a great starting point for enterprises looking to leverage their proprietary data for chatbot applications without the risk of exposing that data.

HuggingFace Spaces AMPsHuggingFace Spaces are very similar to AMPs, and as HuggingFace is one of the key members of Cloduera’s AI partnership ecosystem, it only made sense to integrate them directly into the AMP catalog. Like AMPs, Spaces are ML demo applications that are self-contained and instantly ready to deliver value upon deployment. HuggingFace has built an unmatched community of the best and brightest data scientists, and Spaces are where this community shares its best projects. With a staggering 180,000+ projects to draw from, this integration gives Cloudera customers streamlined access to an unparalleled array of projects to choose from.

Community AMPsThe strength of Cloudera doesn’t end with its engineering staff. Our strength is our community, from solutions engineers to professional services employees embedded in the world’s leading technical organizations to the practitioners who use Cloudera to solve real-world problems every day. Our community AMP catalog is where anyone can contribute best-in-class solutions to an open-source repository of meaningful projects.

This catalog is where we add standout submissions from Cloudera’s global hackathon events. Most recently, we hosted a Climate and Sustainability Hackathon in partnership with AMD. With over 2,000 participants from across the world, the hackathon invited the brightest minds to contribute solutions that would help combat the effects of climate change.

Get Started with Accelerators for ML Projects TodayDon’t just take our word for it, try it yourself. We’re offering a free five-day trial for Cloudera on public cloud. In this trial environment, users have the ability to launch AMPs from our entire catalog.

Find out how AMPs can accelerate your AI use cases, delivering your AI MVP with a single click!

The post SoftBank Selects Cloudera Data Platform to Leverage Customer Intelligence While Ensuring Data Security appeared first on Cloudera Blog.

View Details

This month is Allyship April—a time dedicated to deepening our understanding of allyship and its profound impact on fostering inclusive cultures.

Allyship isn’t merely a buzzword; it’s a fundamental commitment to actively support and advocate for marginalized individuals and communities within our organization. This month, we’ve engaged in meaningful conversations, challenged our assumptions, and committed to tangible actions that drive positive change. Together, we’re creating a workplace where everyone feels valued, respected, and empowered to thrive.

As part of this goal, we caught up with Antoine Burrell, a senior engineering manager here at Cloudera and leader of our Cloudera Black Employee Network (CBEN+) Employee Resource Group, and explored what it means for him to be an ally. From understanding privilege to practicing empathy and amplifying marginalized voices, there’s much to explore and learn together.

Let’s hear what Antoine had to say about allyship.

What does allyship mean to you?To me, allyship means taking the time to listen to and understand your peers. As someone who isn’t the youngest in the room anymore, I’ve realized the importance of empathizing with different generations and their experiences. My children, who are 19, often remind me of this by introducing me to their friends and sharing their struggles. It’s eye-opening to see how different the world is for them compared to my upbringing.

Reflecting on accepting certain norms today, such as LGBTQ+ identities, I recognize the evolution in societal attitudes. Growing up, these identities weren’t as openly accepted, and we lacked the language to discuss them openly. However, I’ve consciously tried to understand and accept these differences through allyship. Allyship, for me, involves challenging the biases ingrained in my upbringing and actively seeking to understand and support those whose experiences differ from mine. It’s about listening more and judging less, creating space for diverse voices to be heard and respected.

How does Cloudera support allyship across the company?Unlike any other company I’ve been with, Cloudera takes concrete steps to support allyship, fostering an inclusive environment where every voice is valued and celebrated. One of the standout initiatives is the active engagement of our Employee Resource Groups (ERGs). These ERGs are not just placeholders on paper; they’re vibrant communities that actively champion diversity and inclusion.

Our leadership’s hands-on involvement in allyship initiatives sets us apart, from our CEO to our Executive Vice Presidents. Beyond symbolic gestures, Cloudera empowers employees to take tangible action through volunteer opportunities and charity events, reflecting our commitment to allyship as a lived reality ingrained in our corporate culture.

Why do you think allyship is vital in the workplace?Allyship is crucial in the workplace because it fosters diversity of thought and breaks down barriers that hinder collaboration. It’s about recognizing and respecting individual differences while understanding that they do not impede our ability to achieve common goals. By creating an environment where everyone feels valued and supported, we can unleash the full potential of our team members.

As a leader, I focus on nurturing people, believing that investing in their well-being drives success for both individuals and the organization. Ultimately, allyship isn’t just about being inclusive—it’s about leveraging our differences to fuel innovation and excellence.

What pillar of allyship is most meaningful to you? How do you use elements of these pillars every day?Allyship has four pillars: education and awareness; listening and empathy; amplification and advocacy; and action and accountability. Although each of these pillars works together to provide a framework for individuals to support marginalized or underrepresented people, the two that stand out to me are listening and empathy. Without active listening and empathy, every other pillar and action loses effectiveness. Listening is the foundation for education, awareness, amplification, advocacy, action, and accountability.

I believe in the power of actively engaging with others, hearing their stories, and empathizing with their experiences. I gain valuable insights that inform my actions and advocacy efforts by prioritizing listening. As the saying goes, “You have one mouth and two ears, so you should listen twice as much as you talk.” By practicing empathy and cultivating a culture of listening, we can create workplaces where everyone feels seen, heard, and valued.

What are some ways you try to be an ally at work?One of the primary ways I strive to be an ally at work is through my role as the lead of the Cloudera Black Employee Network (CBEN+). In this capacity, I focus on understanding the needs of my immediate team and the broader community of Black employees within the organization. Whether it’s providing support and resources or simply lending an empathetic ear, I prioritize actively listening and responding to the needs of my colleagues.

I also make a conscious effort to put the needs of others before my own agenda. Whether it’s a colleague reaching out for assistance or guidance, I strive to be attentive and responsive, offering help wherever possible. For instance, recently, when a colleague sought assistance outside of my department’s scope, I didn’t hesitate to leverage my network to connect them with the appropriate resources. It wasn’t about personal gain but rather about fostering a culture of support and collaboration.

Learn more about our Diversity, Equality, and Inclusion efforts at Cloudera.

The post #ClouderaLife Allyship April Q&A with Antoine Burrell appeared first on Cloudera Blog.

View Details

Back in October, we announced the first-ever Cloudera Climate and Sustainability Hackathon, powered by AMD. The Hackathon was intended to provide data science experts with access to Cloudera machine learning to develop their own Accelerated Machine Learning Project (AMP) focused on solving one of the many environmental challenges facing the world today.

More than 2,300 data scientists competed in the Climate and Sustainability Hackathon—a record number of Cloudera Hackathon participants for an incredibly important cause. And that record number meant hard work for our judging panel, who were tasked with evaluating the nearly 200 innovative AMPs that our participants created to determine the top three winners.

The judging process took place over two phases from October 2023 to March 2024. In the first phase, the judges awarded points based on uniqueness and innovation of the idea and the problem it would solve; the approach to how the project would employ machine learning; the articulated benefits to the environment; and how differentiated the solution was from other applications. In phase two, the judges further narrowed the submission pool by evaluating the code quality, web application quality, and potential impact of the AMP.

The result? Three incredibly inspiring projects capable of targeting some of our most pressing climate challenges. But you’ll have to stay tuned to find out just who those winners are when we announce this week.

In the meantime, get to know some of the judges who made this Hackathon such a success.

Jake BengtsonJake is the Technical Evangelism Director at Cloudera, where he specializes in demonstrating how Cloudera’s Lakehouse supports Generative AI and the entire data lifecycle. His journey in Cloudera began as a Senior Product Marketing Manager over machine learning lifecycle products. Jake previously honed his skills at ExxonMobil, starting as a Data Scientist and eventually progressing to a Data Science and Analytics Solution Architect role. In addition, he held the role of Senior Data Scientist and Data Science Manager at FarmersEdge, adding a startup and geospatial facet to his experience.

Mike HowerMike Hower is a sustainability communication leader with experience on both sides of the podium — as a journalist and communication strategist for leading global companies. As the founder and fractional sustainability communication executive at Hower Impact, Mike works closely with companies to translate their sustainability strategy into stories that stick while navigating the risks and opportunities of an evolving communications landscape. Mike has served as a communication strategist at both large and small consultancies, including Edelman, and has led in-house communication efforts at mission-driven startups. Mike shares insights and ideas as a LinkedIn influencer, writing his ENGAGE newsletter and blog, and hosting “The Sustainability Communicator” podcast.

Debbie KrugerDebbie is the Director of Internal Communications and Social Impact at Cloudera, where she’s led innovative strategies in social impact and employee engagement for the past two years. With a background spanning from corporate giants like Marsh & McLennan Companies and Verizon to fashion retailer ANN INC., Debbie brings cross-industry experience developing impactful CSR initiatives that leverage corporate resources to make sustainable and long-lasting impacts on companies, climate, and communities.

Vijay SudanVijay is a seasoned corporate responsibility and sustainability executive with experience leading global environmental sustainability strategy for three different Fortune 100 companies, most recently at The Walt Disney Company. Previously, he managed global Social Impact initiatives for 21st Century Fox and led News Corporation’s Global Energy Initiative, helping them to become the first carbon-neutral global media company in 2010.

Yi-Hsun TsaiYi-Hsun Tsai is a Senior Engineering Manager of the Enterprise AI Platform team at Cloudera, where he has led the team building the Enterprise AI & ML platform for supporting end-to-end machine learning & Generative AI operations. Prior to Cloudera, Yi-Hsun was a founding team member of Sense, which later became the Cloudera Data Science Workbench and Cloudera Machine Learning, as well as a founding team member of the Post TV video team at Washington Post and built the Post TV platform for serving live streaming or video on demand from the Washington Post newsroom. Yi-Hsun holds a Master of Science in Engineering in Computer and Information Science from the University of Pennsylvania. Originally from Taiwan and now based in San Francisco, Yi-Hsun enjoys traveling around the globe with his wife and his six-year-old son when he is not building the next generation of Enterprise AI platform with his team.

Check in next week for the winner announcement and learn more about Cloudera’s AMD community here.

The post Climate and Sustainability Hackathon—Meet the Judges! appeared first on Cloudera Blog.

View Details

Regulations often get a bad rap. You may have heard the old idiom “cut the red tape” which means to circumvent obstacles like regulations or bureaucracy. But in many – if not most )– cases the underlying need for regulations outweighs the burden of compliance. In the financial sector, regulations are essential for financial institutions to maintain stability by preventing excessive risk-taking, ensuring adequate capitalization and reducing the likelihood of failures or financial crises. Regulations require the implementation of robust risk management practices, prevention of financial crimes and promotion of competition. Moreover, they help maintain confidence in the financial system, encouraging consumers, enterprises and investors to trust institutions with their money.

With that said, consider the impact digital technology has made on the industry with the adoption of hybrid and multi-cloud strategies. While these enablers have streamlined operations, inspired innovation and accelerated cost optimization, governing bodies would be negligent if they didn’t address the cyber-risk associated with digital, internet-based, and third-party technology solution providers that present a broadened threat landscape.

In Europe, the EU is taking key steps to bring uniformity and an increased focus on risk mitigation within the financial sector. The introduction of the Digital Operational Resilience Act (DORA) will affect both the institutions (financial entities) and technology service providers, like Cloudera, that serve the financial sector across member states.

What is DORA?DORA is a regulation by the European Commission, made effective in January of 2023, with compliance required by January 2025. As the financial sector is increasingly dependent on information and communication technology (ICT) and ICT service providers (ICTSPs) – as defined by the act – to deliver financial services, DORA is intended to enhance the operational resilience of the EU’s financial sector against cyber threats and incidents. DORA focuses on ensuring the continuous functioning of digital services provided by financial entities (FEs), such as banks, investment firms, and market infrastructures.

Here are some of the key objectives and requirements of DORA:

• Addresses ICT risk management comprehensively in the financial sector and harmonizes rules across the EU
• Requires FEs to identify, assess and manage ICT risks, establish policies to safeguard systems and data, and develop business continuity plans
• Mandates incident reporting, resilience testing, and third-party risk management for FEs
• Establishes an oversight framework for critical ICTSPs like cloud platforms and data analytics services
• Allows FEs to exchange cyber threat information with arrangements that comply with GDPR and other data laws

The consequences of non-compliance can be severe as FEs may face administrative fines up to 10 million euros or 5% of their total annual turnover, whichever is higher, for serious infringements.

The consequences reach critical ICTSPs as well. “Critical” ICTSPs are those whose disruption or failure could have a significant impact on society, the economy, or national security. These ICTSPs may face fines of up to 1% of average daily worldwide turnover.

The Impact on Data Platform ICTSPsData platform ICTSPs, such as Cloudera, may fall under DORA’s scope and if so, will need to adhere to strict data security standards, implement robust encryption and access controls, and demonstrate operational resilience in the face of cyber threats.

Here are the key ways DORA may affect data platforms:

• Critical ICTSPs will be subject to a new oversight framework and directly supervised by EU authorities such as EBA, ESMA, and EIOPA
• There are requirements for sound monitoring of ICT third-party risks and the inclusion of necessary details in contracts with FEs
• Non-EU companies that qualify as FEs or ICTSPs to FEs may be impacted by extraterritorial enforcement
• Contracts between FEs and ICTSPs must include specific details on monitoring and compliance with DORA rules
• ICTSPs will need to provide evidence to FE clients on their ICT risk management practices and resilience
• ICTSPs must have mechanisms to report major ICT-related incidents to their FE clients.
• There is an allowance for threat information sharing between FEs and ICTSPs, if done in compliance with GDPR
• ICTSPs may need to enhance incident response and share cyber threat intelligence with FE clients
• Resilience testing of ICT systems and tools is required
• ICTSPs could be subject to audits and on-site inspections by EU supervisory authorities
• Non-EU companies providing critical ICT services to FEs in the EU may fall under DORA’s scope
• Data platforms headquartered outside the EU but serving EU FEs will need to comply with DORA

How Cloudera Helps FEs Comply with DORA RequirementsCloudera helps FEs comply with the EU’s Digital Operational Resilience Act (DORA) in several key ways.

Security and GovernanceCloudera provides a Shared Data Experience (SDX) that delivers consistent data security, governance, and control across the entire data lifecycle and across all environments – public cloud, private cloud and on-premises. With SDX, FEs can set data access controls and policies once, and they are automatically enforced across data and analytics in hybrid and multi-cloud deployments, even as data and workloads move between them. This helps FEs meet DORA’s requirements around sound ICT risk management practices and safeguarding of systems and data

PortabilityCloudera’s container architecture enables flexibility to move data and applications between different environments – public cloud, private cloud and on-premises. This portability helps address DORA’s concerns around cloud vendor lock-in and enables operational resilience for FEs. FEs can also move workloads as needed while maintaining consistent security and compliance

Comprehensive Data Lifecycle ManagementCloudera enables FEs to manage the end-to-end data lifecycle by integrating streaming, analytics, and machine learning on a single platform. This helps develop critical applications to address current and future needs, supporting DORA’s ICT risk management objectives.

Open Source and InteroperabilityCloudera’s platform is based on open source which accelerates innovation and eases concerns about vendor lock-in, a key DORA concern. It enables interoperability with a broad range of analytic and business applications that FEs rely on.

Hybrid and Multi-Cloud Deployment OptionsCloudera can be deployed on any public cloud, private cloud or on-premises, providing FEs the flexibility and control to manage data in adherence with DORA rules. The hybrid, multi-cloud capabilities enable FEs to maintain strict enterprise data security and governance across all their ICT environments.

As FE’s move toward DORA compliance, Cloudera provides a unified, secure and portable hybrid data platform that can help FEs meet several key requirements of the EU’s DORA regulation around ICT risk management, data security, governance, resilience and multi-cloud flexibility. Cloudera’s core capabilities align well with DORA’s objectives to enhance the digital operational resilience of the financial sector.

For more on how Cloudera helps FEs, click here.

The post Navigating the Digital Operational Resilience Act appeared first on Cloudera Blog.

View Details

Artificial intelligence (AI) is something that, by its very nature, can be surrounded by a sea of skepticism but also excitement and optimism when it comes to harnessing its power. With the arrival of the latest AI-powered technologies like large language models (LLMs) and generative AI (GenAI), there’s a vast amount of opportunities for innovation, growth, and improved business outcomes right around the corner. All of that technology, though, depends on data to be successful.

With the backdrop of optimism and interest in these technologies, the Cloudera team headed to Orlando last month for the Gartner Data and Analytics Summit, which brought together industry leaders including Chief Data and Analytics officers (CDAOs) and data and analytics (D&A) leaders to share insights and learn how data, analytics, and AI can transform their operations.

Here are a couple of the biggest takeaways we had from our time at the event.

More Businesses Are Taking a Holistic Approach to Data StrategyOne of the more common trends we saw coming up through conversations during the summit was the need for a reframing of how we approach data strategy—taking a much more holistic viewpoint to it than organizations otherwise would have in past years. In those discussions, it was clear that everyone understood the need to treat data estates more cohesively as a whole—that means bringing more attention to security, data governance, and metadata management, the latter of which has become increasingly popular.

In the context of emerging technologies, like GenAI, there is always a healthy dose of skepticism that comes along with its use. What we’ve seen is that more businesses are looking at the potential of those tools through the lens of trust—specifically, do I trust these open models with my data? Do I trust my data to be ready for use in AI? Do I trust the models to give me useful insights? Answering these questions is an important piece of being adequately prepared to leverage GenAI or LLMs. And that’s where we’ve seen this more holistic approach come into play, as more businesses are doing due diligence and making sure all these things—security, governance, and metadata—are clean and ready to support new use cases.

Modernization Is Foundational to Generating Business ValueFor all the talk of GenAI, one thing remains clear—many organizations still need, and are looking, to modernize and move workloads that are still running on legacy capacity, into something modern. It’s not just a matter of deciding to move to the cloud wholesale, there’s a desire, and increased willingness, to complement the stack provided by a given cloud service provider (CSP) and supplement it with tools and solutions that are tailored to a business’s specific needs, often across clouds and on-premises. With that, we’ve seen heightened excitement when it comes to third-party solutions—particularly integrating those into existing infrastructure in a way that helps drive the maximum possible business value.

The emphasis on modernization is also driven, in part, by a push among businesses to get data moving and accessible in real time. More and more organizations are realizing that the immediacy of their data and the ability to be proactive as close to a point in time when a change happens is invaluable. That immediate access and real-time capability is what keeps businesses agile and enables them to leverage GenAI and other tools successfully.

Preparing For an AI-powered FutureThere’s plenty of optimism and interest surrounding GenAI and AI more broadly. But even in all the excitement, the steps required to make those technologies impactful were at the top of mind for attendees. Among other shifting trends, we saw just how much the approach to data management is shifting, with data strategies moving to account for the data that feeds AI use cases and ultimately makes them trustworthy, and successful.

Learn more about how Cloudera is accelerating enterprise AI.

The post A Look Back at the Gartner Data and Analytics Summit appeared first on Cloudera Blog.

View Details

Businesses around the world rely on an extensive network of partnerships to deliver quality customer experiences—and it’s no different here at Cloudera. Cloudera is building a robust partner ecosystem to meet the unique needs of its customers, working to provide exceptional and fulfilling experiences that help make Cloudera a leader in the multi-cloud data platform space. With nearly 20 years of marketing experience, Lan Chu is no stranger to collaborating across teams to build strong, robust marketing programs. Having worked across marketing teams from companies in food delivery to software development, Lan understands the importance of partner success to support positive customer outcomes. Lan exemplifies the importance of creative marketing, innovative solutions, and building trust across Cloudera’s extensive and essential partner ecosystem.

To delve deeper into the importance of partnerships, we spoke with Lan about her experience as a senior partner marketing leader at Cloudera.

What makes strong partnerships so vital? How do they impact the work happening at Cloudera? A partner ecosystem isn’t just a nice addition; it’s a fundamental requirement in today’s market. I’m in lockstep with our partners daily, ensuring we all have the plans to succeed. Collaborating closely with strategic partners allows Cloudera to offer comprehensive solutions that meet diverse customer needs and drive growth. By aligning efforts and objectives with partners, Cloudera ensures coordinated sales, marketing, and product development efforts, resulting in more successful customer outcomes and enabling us to deliver outcomes and objectives that allow us and our partners to grow.

Partnerships are not just about mutual support; they are the catalyst of innovation at Cloudera. They open doors to new ideas, technologies, and market trends, propelling the development of cutting-edge solutions. This collaborative approach empowers us to outpace the competition, deliver solutions that truly meet our customers’ needs, and maintain our leadership position in the industry. The potential of partnerships to fuel innovation is limitless, and it’s a journey we’re excited to embark on with our partners.

What excites you about working with Cloudera’s partner ecosystem? I thrive on challenges and I’m empowered in partner marketing to develop new strategies to deliver optimal results. It’s all about aligning diverse stakeholders to move in the same direction, gaining buy-in, and delivering results that benefit everyone involved. Working in partner marketing aligns with my passion for technology and startups, offering opportunities for autonomy, innovation, and rapid learning and growth.

While it can be daunting, especially as I focus on multi-partner motions throughout the year, it’s ultimately about serving the customer. They want to see the full solution offering, not just individual parts. By bringing together various partners, we provide comprehensive solutions that address specific use cases and modernize data architecture. Collaborating with partners to create value for our customers and drive success for all parties involved is exciting.

What is the best way to build a strategic partner marketing program? Building a strategic partner marketing program begins with identifying those key partners and exploring similarities in the market. By the nature of my role, I find myself collaborating with teams across Cloudera to support partner and channel initiatives.

Despite the challenge of balancing partnerships with major cloud providers, it’s essential for us to acknowledge the reality of customers operating in a hybrid cloud environment. Discussions in our partner work often revolve around understanding customer needs and creating interconnected solutions. For instance, in a recent project focused on generative AI (GenAI), I engaged with cloud providers and hardware vendors to develop a fully integrated program which included a 3 part webinar series addressing the pivotal points of GenAI. This series emphasized the tools and resources needed for success in the GenAI space and showcased Cloudera’s expertise through customer success stories.

While the journey may be challenging, my background, particularly within the high-tech sector, has equipped me with the tools to navigate complex partnerships and drive impactful marketing initiatives.

A lot of time and effort goes into these partnerships. How does our partner marketing strategy help Cloudera succeed as a business? In my experience, partner marketing programs are indispensable for a business’s success because they are inherently customer-centric. Customers expect a rich ecosystem of solutions, and partnering with other companies allows businesses to meet those demands effectively.

When hyperscalers, hardware vendors, or other technology companies partner with us, it’s because they see value for themselves and their customers. Every partnership is mutually beneficial, with both parties focused on delivering value to their respective customer bases. This customer-centric approach aligns perfectly with our growth themes, which revolve around AI, hybrid solutions, and modern data architecture. Partners are crucial in supporting these themes, enabling us to effectively deliver comprehensive solutions that address customer needs. Without our partners, achieving true hybridity, enabling AI capabilities, or modernizing data architecture would be challenging. Therefore, viewing everything through the lens of partnership underscores the nature of our business, with partners playing a vital role in helping us deliver value to our customers.

To help us fuel the best business outcomes at Cloudera, we are committed to building a rich and robust partner ecosystem, and we’re committed to helping our partners unlock the potential of Cloudera’s enterprise data cloud. The Cloudera Partner Network provides a direct path to benefits, including enablement, marketing, adaptable selling models, technology certifications, and solution competencies that customers trust.

Learn more about partnering with Cloudera here.

The post Learn About Cloudera’s Partner Network appeared first on Cloudera Blog.

View Details

It’s been said that the Federal Government is one of, if not the largest, producer of data in the United States, and this data is at the heart of mission delivery for agencies across the civilian to DoD spectrum. Data is critical to driving the innovation and decision-making that improves services, streamlines operations and strengthens national security. Moreover, as emerging technologies like generative AI proliferate across Federal use cases, the need for trusted data that is secure, governed and ready for AI has never been more acute.

I am pleased to announce that Cloudera has achieved FedRAMP “In Process”, a significant milestone that underscores our commitment to providing the public sector with secure and reliable data management solutions across on-prem, hybrid and multi-cloud environments. This designation reaffirms Cloudera’s dedication to delivering enterprise-grade, government-scale data management capabilities while maintaining a hyper-focus on data security.

Cloudera has long been recognized as a leader in the field of data management, offering a comprehensive suite of products and services designed to meet the unique needs of government agencies. While Cloudera has a firm history supporting public sector use cases like smart military bases, Department of Defense flight safety, and other public sector use cases, the FedRAMP “In Process” designation will make hybrid-cloud data management available to an even greater number of federal agencies.

Purpose Built for SecurityHere at Cloudera, we focus on ensuring that our platform meets the highest standards of security and regulatory compliance. We understand the importance of protecting sensitive government data from unauthorized access, and we have implemented robust security measures to safeguard this information.

FedRAMP requires that we meet strict security standards to protect government data. Cloudera is perfectly suited for FedRAMP and securing government data due to our integrated approach to security and governance. With independent operation from compute and storage layers, Cloudera ensures that security and governance are foundational elements of our platform. This integrated approach helps simplify and streamline the implementation of security controls required by FedRAMP. Our use of metadata also ensures persistent context across all analytics and cloud environments, making it easier to maintain consistent security policies and access controls.

The Road AheadAs we push forward to FedRAMP Moderate Authority to Operate and, subsequently, FedRAMP High authorization, Cloudera remains steadfast in its commitment to delivering both a true hybrid data platform and cybersecurity excellence that will empower federal agencies and their partners to get the most out of their data management.

Learn more about how partnering with Cloudera can help you strengthen your federal agency’s hybrid data infrastructure.

The post FedRAMP In Process Designation, A Milestone in Cybersecurity Commitment appeared first on Cloudera Blog.

View Details

We’re excited to share that Gartner has recognized Cloudera as a Visionary among all vendors evaluated in the 2023 Gartner® Magic Quadrant for Cloud Database Management Systems. This recognition underscores Cloudera’s commitment to continuous customer innovation and validates our ability to foresee future data and AI trends, and our strategy in shaping the future of data management.

Cloudera, a leader in big data analytics, provides a unified Data Platform for data management, AI, and analytics. Our customers run some of the world’s most innovative, largest, and most demanding data science, data engineering, analytics, and AI use cases, including PB-size generative AI workloads. Cloudera has over 25 Exabytes of high-value business data under management, making us one of the larger data hyperscalers in this MQ. This recognition is a testament to our vision and ability as a strategic partner to deliver an open and interoperable Cloud data platform, with the flexibility to use the best fit data services and low code, no code Generative AI infused practitioner tools.

Download the complimentary 2023 Gartner Magic Quadrant for Cloud Database Management Systems report.

The Gartner Magic Quadrant report is complemented with the Gartner Critical Capabilities for Analytics report which scores all the vendors on their technical strengths. We scored the highest in hybrid, intercloud, and multi-cloud capabilities because we are the only vendor in the market with a true hybrid data platform that can run on any cloud including private cloud to deliver a seamless, unified experience for all data, wherever it lies. Our cutting-edge Shared data experience (SDX) service provides a unified control plane for common security, governance and metadata management on all structured and unstructured data. This is critical for our customers to gain visibility into all their data, residing anywhere, and deliver trusted, secure data for all analytics and AI workloads. Moreover, the portability and interoperability of our data services bring simplicity, flexibility, and agility to all data workloads.

Download the complimentary 2023 Gartner Critical Capabilities for Analytics Report to view the technical scores of each vendor on the three key analytic use cases.

The following are key attributes of our platform that set Cloudera apart:

Unlock the Value of Data While Accelerating Analytics and AIThe data lakehouse revolutionizes the ability to unlock the power of data. It unifies structured data with unstructured data giving a holistic view of the business and enhancing the efficacy of AI by providing it with more structured and accessible business data as context. By bringing data warehouse-like capabilities to the lake it eliminates the need for multiple data stores, simplifies data engineering pipelines, increases data reliability, and improves the efficiency of both data scientists and data analysts. Cloudera’s open data lakehouse accelerates BI query performance by over 40% while also making it easy for data scientists to explore the latest datasets for their models. The integrated data services from stream analytics, data flow, and data engineering to data warehousing and machine learning all operate on a single copy of data residing in Iceberg tables for a true multi-function analytics experience.

This improves collaboration across teams, empowers users with self-service data access and reduces compute and storage requirements while minimizing integration tax. It gives the flexibility to teams to use their choice of tools/engines without worrying about ETL. Cloudera’s real-time streaming service is unique in the industry because it not only delivers microbatch ingestion like other intelligent data platforms but also provides stream ingestion and stream data processing so you can process the stream in real time before ingesting the events into the lakehouse.

Guarantee Data Ecosystem InteroperabilityMany organizations struggle to unlock the full value of data due to disparate tools and data sources, and poor data quality. Organizations manage an increasing variety of single purpose databases, resulting in increased cost, complexity, management overhead, and risk.

Cloudera customers deploy a very diverse data infrastructure that consists of hundreds of data sources, multiple clouds, and multiple processing engines. Our commitment to open source, investments in integrating open standards, and building open APIs ensures that our platform and services interoperate in any data ecosystem. With the integration of Apache Iceberg table format, we allow any compute engine to access data from Cloudera’s data lakehouse including engines outside of our platform like Snowflake, Google Big Query, AWS Redshift, and more. This has helped our customers reduce their Snowflake and Databricks cloud costs while still using the tool of their choice.

Democratize Data, Analytics, and AI As per a study conducted by Harvard Business School, The Wharton School, The Warwick Business School, and MIT Sloan, Generative AI can improve a highly skilled worker’s performance by as much as 40% compared with workers who don’t use it.

At Cloudera, we are embedding AI in all our products to improve productivity of data professionals. These intelligent services automate tasks, optimize workloads, simplify UI, and make it easier for more users to interact with the data. \

We have embedded LLMs in our Hue interface so you can write SQL queries in English or any other language. LLMs in our data visualization tool allow business users to talk to their data using a chatbot and share dashboards and reports with ease. LLMs in our Cloudera Machine Learning (CML) service intelligently assist you in data science and AI projects. AI in our Observability solution optimizes your workloads and makes recommendations based on your specific usage. All this intelligence built into the platform democratizes data, analytics, and AI.

We are also continuously building tools and services to help organizations build AI applications with Cloudera. Cloudera’s quick starter AI projects called AMPs package the code, models, datasets, and configurations required to deploy a new ML project, accelerating the speed of deploying new LLM use cases by 50%. Unlike software, ML models need continuous tuning. They need to be monitored and refreshed continuously to prevent model drift. Cloudera’s MLFlow service monitors models and makes it easy to unplug and plug new models as and when needed. Cloudera’s real-time streaming service ensures you are fine-tuning your models with the latest data to prevent model drift and accelerate the rate of learning. The private cloud data services and write once read anywhere portability guarantee that you can use LLMs without ever leaking your company secrets to the public while reducing the inference costs.

Deliver Trusted Data for Trusted AITrusted AI needs trusted data. This is data that is reliable, complete, fully understood and has the right business context. Data that has lineage so you can trace back the source and all the processing it underwent. Increased confidence in data results in trusted AI. Cloudera’s data catalog combined with the Shared Data Experience (SDX) service gives full visibility into the data so you can find your datasets easily, understand your data with lineage and metadata information, and access the right data with unified security and governance.

Adopt Data Mesh to Power the New Wave of AIData is evolving from a valuable asset to being treated as a product. Adopting the data mesh philosophy is not merely a trend but a strategic imperative to unlock the true power of data. This approach is vital to gaining a deeper understanding of data, using it to drive actionable insights, and building a repository of trusted data that is maintainable, portable, and easily shareable across diverse teams. The essence of data mesh lies in transferring the ownership of data into the hands of the users and creators and amplifying the utility of data across various use cases.

Implementing a data mesh requires changing the organization’s data culture and elevating data literacy across all teams. The tools and platform play a pivotal role in increasing the visibility of data, easing its manageability, facilitating data product sharing, and delivering federated governance across all data assets. Cloudera’s lakehouse with integrated portable data services, SDX, and data catalog capabilities along with data flow service for integrating data from literally any source, has helped companies implement a data mesh fostering global collaboration and information sharing across teams.

Empower Customers to Take Back Control of Their Data:Only Cloudera lets you take back control of your data, workloads, deployment, and spend. With the Cloudera data lakehouse, you have full control of your data, with the flexibility to store it in the format you like, run your workloads on whichever cloud you want, and process it with whatever engine you like. All this without any vendor lock-ins or additional ingress or egress costs and while having full control over your cloud spend. Cloudera’s industry-leading financial governance capabilities and observability solution help you minimize cloud spend and storage costs giving you full visibility into your workloads. We also see customers reducing their cloud spend by repatriating non-fluctuating workloads to private cloud where they get all the benefits of cloud for instance—scale, flexibility, and agility—while maintaining full control over their cloud spend

Generative AI has accelerated the adoption of AI. But before you jump into Generative AI and LLMs, companies need strong data fundamentals. Data sitting in swamps is of no use until it can be made actionable with the right analytics and AI. LLM’s ability to mine unstructured data is remarkable which has unleashed new business use cases and enhanced the ability to respond to customer requests, present personalized offers, and accelerate R&D. Cloudera customers like OCBC have already deployed LLMs using their enterprise data as context to improve customer experience and operational efficiency.

Cloudera’s hybrid, multi-cloud data platform delivers the strong data fundamentals you need to accelerate your Generative AI journey. Sign up for a trial to see for yourself.

The post Cloudera Named a Visionary in the Gartner MQ for Cloud DBMS appeared first on Cloudera Blog.

View Details

At a time when AI is exploding in popularity and finding its way into nearly every facet of business operations, data has arguably never been more valuable. More recently, that value has been made clear by the emergence of AI-powered technologies like generative AI (GenAI) and the use of Large Language Models (LLMs). But, even with the backdrop of an AI-dominated future, many organizations still find themselves struggling with everything from managing data volumes and complexity to security concerns to rapidly proliferating data silos and governance challenges.

As organizations continue to navigate this AI-driven world, we set out to understand the strategies and emerging data architectures that are defining the future. To do this, Cloudera commissioned a study with Foundry, Data Architecture and Strategy in the AI Era, surveying over 600 IT decision-makers in North America, northern Europe region of EMEA, and APAC.

Let’s explore some of the most important findings that the survey uncovered.

Tapping into AI’s Full PotentialIt’s not just hype and talk when it comes to AI—a majority of surveyed respondents (three out of five) said their organizations were at least in the early stages of adopting AI in their operations while only eight percent said they had yet to make any plans for AI adoption. And of those organizations working on some stage of AI adoption, a few of the top benefits included increased productivity (35%), enhanced operational efficiency (33%), improved customer experience (33%), and optimized supply chain and logistics (33%).

The benefits are clear, and there’s plenty of potential that comes with AI adoption. But that doesn’t mean it’s all smooth sailing for organizations putting AI into practice. Among the most common challenges to achieving AI adoption at scale were data quality and availability (36%), scalability and deployment (36%), integration with existing systems and processes (35%), and change management and organizational culture (34%). Ultimately, when it comes to reaching the full potential of AI, the organizations that are able to overcome these complexities and find, classify, and expose data to the right people will be able to find sustained success at scale.

Mapping Out the Keys to SuccessThe path to successfully implementing AI at enterprise scale is built on three critical elements: modern data architecture, unified data management, and versatile, secure data platforms. Of surveyed respondents, companies that are leading the way toward AI adoption are focusing on these three areas.

• Modern data architecture: A flexible approach is critical for building a modern architecture, with IT leaders recognizing the importance of data lakes or lakehouses for managing the large volumes of unstructured and semistructured data required for AI model training. In fact, two thirds of respondents agreed that data lakehouses were crucial to reducing pipeline complexity.
• Unified data management: Survey respondents overwhelmingly (90%) understood the importance of unifying their data lifecycle on a single platform as an essential part of analytics and AI. And nearly half (46%) of surveyed IT leaders said their organization interacts with every stage of the data lifecycle process. Gaining complete control and visibility into every aspect of data gives IT leaders the capabilities needed to drive AI-fueled innovation.
• Flexible, secure data platforms: From a long-term perspective, a hybrid data management approach, including both on-prem and public cloud infrastructure and data strategy, is the preferred path forward. While only one third of respondents currently deploy multicloud or hybrid data architectures, 93% of those respondents agreed that “multicloud and hybrid capabilities for data and analytics are key for an organization to adapt to change.”

The potential of AI is massive and is quickly moving from the theoretical into actual implementation across a vast number of businesses. And as it does, having a modern data architecture is proving to be a critical, foundational, part of successfully scaling the technology and reaching its full potential—something that the survey results reveal and that IT leaders are acutely aware of within their own organizations. Ultimately the organizations that successfully implement AI will be those that are able to demonstrate high levels of confidence in training data, model integrity, and respect for security and privacy.

Check out the full survey report for additional insights into the future of AI and data architecture.

The post Data Architecture and Strategy in the AI Era appeared first on Cloudera Blog.

View Details

The ongoing progress in Artificial Intelligence is constantly expanding the realms of possibility, revolutionizing industries and societies on a global scale. The release of LLMs surged by 136% in 2023 compared to 2022, and this upward trend is projected to continue in 2024. Today, 44% of organizations are experimenting with generative AI, with 10% having already implemented it in operational settings. Companies must act now in order to stay in the AI Race.

But for many enterprises, the biggest hurdle is getting started. How do you adapt a foundational model to your specific needs? What about security, privacy, and trust concerns? How much is all this really going to cost? These questions are legitimate, and this is precisely where Cloudera plays a role.

Cloudera: Your Trusted Partner in AIWith over 25 Exabytes of Data Under Management and hundreds of customers leveraging our platform for Machine Learning, Cloudera has a long and successful history as an industry leader. Cloudera has had an in-house expertise in Machine Learning and has been helping customers deliver machine learning applications, build ML models, and deliver ML ops for the past decade. Now that the industry has advanced into Large Language Models and more intelligent forms of modeling, LLM Ops, fine tuning, and serving capabilities, Cloudera has seamlessly evolved alongside this expanding ecosystem.

We’ve built a comprehensive AI stack designed to empower businesses to develop AI applications faster and more efficiently. Our Lakehouse architecture seamlessly integrates with your existing data infrastructure, and we offer a complete solution with cutting edge innovations such as Accelerators for ML Projects, advanced AI inference capabilities powered by NVIDIA, and open tooling for every step of the AI lifecycle.

“The world of AI is moving at unprecedented speed and companies that do not modernize their stack now risk being left behind. It is more important than ever for companies to support the deployment of customer AI use cases, which is why I am excited about Cloudera’s Enterprise AI Fast Start. This release will remove the complexity from getting started with AI, delivered as a cost-effective bundle to deliver immediate business value.” said Sanjeev Mohan, Principal, SanjMo & Former Gartner Research VP, Data & Analytics.

Introducing the Enterprise AI Fast Start: Accelerating AI Applications for EnterprisesThe Enterprise AI Fast Start removes the complexity from getting started with AI and allows customers to have a much faster time to value for their AI use cases. This accelerator will provide access to the entire AI lifecycle, giving customers a robust, hybrid-cloud solution for production LLM ModelOps & real-time inferences.

Available for both public and private clouds, the Enterprise AI Fast Start is a time-limited offer delivered as a cost-effective bundle that fits within your budget and delivers immediate business value which includes:

Cloudera professional services to build a Proof-of-Concept AI application. Choose from high value use cases like Chat Q&A, Copilot, or Text Summarization.

• Cloudera Machine Learning Public or Private Cloud. Our leading AI platform includes:
• 1. 1. Notebooks: Develop your applications and models using your preferred language and framework. 2. Model Registry and Endpoints: Effortlessly manage your models through their lifecycle, including hosting and web app integration. 3. Pre-built Demos and Reference Architectures (Accelerators for Machine Learning Projects – AMPs): Get a head start with ready-made solutions designed for specific AI use cases. 4. Containerized Compute Sessions: Run your development and testing tasks with ease.
• On Demand Cloudera Machine Learning training to build internal expertise

The Benefits of Embracing Enterprise AI Fast StartWith the Enterprise AI Fast Start, your organization can:

• Develop hundreds of AI applications: Improve efficiency and gain valuable insights across your business.
• Achieve Faster Time to Value: Deploy applications quickly and see results sooner.
• Use Fewer Resources: Get started with AI without a massive investment.

Many of our customers are already reaping the benefits of Enterprise AI deployments.

“Utilizing Cloudera within a modern architecture has allowed us to accelerate our ability to develop insightful models and Gen AI use cases that ultimately create more value for our customers. Our pursuit of innovative AI and data science technology will continue to help us develop even more impactful projects in the near future.” said Dr. David Hardoon, Group Chief Data & AI Officer, Union Bank of the Philippines.

Don’t wait any longer! Get started with the Enterprise AI Fast Start today and join the leaders who are shaping the future with Artificial Intelligence. Please find more information about this offering on our website.

The post Don’t Get Left Behind in the AI Race: Your Easy Starting Point is Here appeared first on Cloudera Blog.

View Details

Meet Cloudera’s new Senior Vice President of Global Communications, Jess Hohn-Cabana. In this Employee Spotlight, we’ll get to know more about Jess, her new role, and her recent award win at the 2024 Ragan Top Women in Communications Awards.

Get to Know Jess: A Seasoned Leader in Tech Communications and BrandingComing to Cloudera with nearly three decades of experience in tech communications and branding, Jess is a leader and a visionary on all things storytelling. Jess recently joined the Cloudera team after several years at Rocket Software and ASG Technologies, where she led internal and external communication strategies with a focus on PR initiatives, thought leadership, M&A communications, and executive communications. Jess also has extensive experience in brand strategy, analyst relations, and influencer relations, successfully building awareness for global bands and creating meaningful relationships with key audiences.

To delve deeper into Jess’s thoughts, experiences, and motivations, let’s turn the spotlight onto her with a few questions and answers.

What is one goal you hope to accomplish in your new role?At Cloudera, my goal is to transform a virtually unknown company into a globally recognized brand. Enhancing Cloudera’s brand reputation and visibility requires creating narratives that focus less on product and feature functionality to messaging that resonates with our key audiences. This messaging should highlight how we help large organizations solve some of the most complex data challenges and strengthen their business operations.

We must also position Cloudera as a thought leader in our space. By focusing on thought leadership and strategic and consistent messaging, we can not only elevate Cloudera’s reputation but also influence industry conversations, differentiate ourselves from competitors, and ultimately contribute to our long-term success.

Why have you been drawn toward tech companies throughout your career?I began my career at an Advertising and PR agency, working with clients ranging from retail to manufacturing to tech companies. However, my journey truly took off when I transitioned into a communications role at a software company in Silicon Valley. This is where I found myself captivated by the fast-paced energy and the ever-evolving landscape of technology. I mean, so much has changed even in the last year with AI, cloud computing, and the metaverse. In order to be successful in tech communications, you need to be able to evolve alongside these changes and innovations – something that is not just necessary but exciting. I feel particularly excited about joining Cloudera because of the continuous innovation happening within the business to help our customers solve their toughest AI and data challenges.

What is an underrated quality or skill in the communications field to have? How has that skill helped you?There are several I could list, but I think the most important is the ability to actively listen. Active listening involves more than just hearing someone; it entails genuinely striving to comprehend the significance and intent behind their words. Communication professionals often spend time doing the speaking, but I’ve learned in my own career that having the ability to truly listen is critical for effective communication and forming meaningful connections. By actively listening, I can tailor communication strategies that address specific needs and requirements, resulting in more fruitful collaborations and successful outcomes.

You were recently named a Leader in Ragan Communications’ list of Top Women in Communications for 2024. What does it feel like to be honored alongside other female industry experts and visionaries who redefine the way communicators connect, inform, and innovate?I know it may sound cliché, but it really is an honor. I was recognized alongside some truly talented and incredible leaders, industry trailblazers, and inspiring women. All of the honorees have accomplished some amazing feats throughout their careers, whether just starting out in the world of communications or industry veterans like me. While my name is the one listed, I see it as a team win – representative of all the intelligent and dynamic individuals I’ve had the pleasure of working with – both past and present.

To read more employee spotlights, click here.

The post #ClouderaLife Employee Spotlight: Jess Hohn-Cabana appeared first on Cloudera Blog.

View Details

We’re excited to announce that Cloudera has been named the Best Medium Workplace in Ireland, one of the Best Workplaces in Costa Rica, and one of Ireland’s Best Workplaces for Women for 2024. These recognitions underscore Cloudera’s ongoing efforts to prioritize employee well-being, professional development, and collaborative work environments. It’s an honor to be listed amongst some of the world’s leading workplaces as a leader in the hybrid cloud space.

Cloudera is a vital hub for data management and enterprise AI solutions across the world, and our Ireland Center of Excellence plays a critical role. The center has been core to our employee growth and development, having witnessed significant expansion in the last year to offer support across various functions—including HR, Finance, Operations, Sales, IT, Marketing, and Engineering. This growth is just part of what contributed to our placement as the Best Medium Workplace in Ireland in 2024. This marks Cloudera’s third consecutive year on the list, moving from second place in 2023 to the top spot this year. The recognition stems from feedback from 200 Cloudera employees in Ireland, contributing to an extensive survey covering 17 critical aspects of workplace culture. Cloudera’s exceptional culture emphasizes customer support, global thinking, camaraderie, and enjoyment, making it a standout workplace, and we couldn’t be more honored for this recognition.

In addition to being named the Best Medium Workplace in Ireland, Cloudera also earned recognition as one of Ireland’s Best Workplaces for Women 2024, reflecting our company’s commitment to creating supportive environments for female employees. With a strong representation of women in management roles, positive sentiments on career development opportunities, and efforts to foster external relationships to encourage female participation in the industry, Cloudera sets a high standard for gender diversity and inclusion across the globe.

Our Ireland office is not the only one celebrating. Cloudera is also now one of the Best Workplaces in Costa Rica, underscoring our commitment to fostering a positive and inclusive workplace culture. This recognition reflects Cloudera’s dedication to creating an environment where employees thrive professionally and personally. Through direct feedback from our workforce in Costa Rica, Cloudera has been recognized as a culture leader. With strong efforts to create a diverse and inclusive atmosphere, comprehensive benefits for employees, a far-reaching social impact, and a collaborative mindset, Cloudera is a vibrant and welcoming workplace that allows its employees to bring their most authentic selves to work.

With a commitment to building a high-trust work environment, Great Place to Work® recognizes organizations like Cloudera that truly put an emphasis on employee well-being and professional growth. Cloudera’s dedication to fostering an inclusive workplace culture, supporting women’s advancement, and providing ample opportunities for learning and development solidify our position as a top employer in the world. If a career at Cloudera sounds like a great fit for you, check out our open opportunities here.

The post Cloudera Recognized as a Great Place to Work in Ireland and Costa Rica appeared first on Cloudera Blog.

View Details

As enterprise AI technologies rapidly reshape our digital environment, the foundation of your cloud infrastructure is more critical than ever. That’s why Cloudera and Red Hat, renowned for their open-source solutions, have teamed up to bring Red Hat Enterprise Linux (RHEL) to Cloudera on public cloud as the operating system for all of our public cloud platform images. Let’s dive into what this means and why it’s a game-changer for our customers.

Previously, Cloudera customers on public cloud were provided with Virtual Machine images built on CentOS Linux. As CentOS Linux approaches its end-of-life in June 2024, and considering RHEL’s widespread adoption among our customers, our transition to a RHEL-based infrastructure reflects both our customers’ preferences and the market’s direction. As a trusted enterprise Linux OS, RHEL boasts certification across hundreds of clouds and thousands of hardware and software vendors. This integration will accelerate InfoSec approvals for Cloudera customers, enhance their security posture, and deliver a long-term, vendor-backed and supported operating system.

Consider it like fortifying the foundation of a house: moving to RHEL is about laying a stronger, more supportive groundwork for your data platform. Much like reinforcing a home to support future expansions and tech upgrades, this shift to RHEL helps to confirm that your digital infrastructure not only supports but accelerates this transformation. On top of that, this solid base is crucial for effectively harnessing the power of enterprise AI and generative AI, enabling your organization to build and scale advanced AI apps with confidence.

The switch to RHEL as the standard operating system for Cloudera on public cloud isn’t just a change – it’s an upgrade for both our customers and ourselves. While this shift naturally aligns with customer preferences for quicker InfoSec approvals and long-term support, it also brings operational improvements for Cloudera itself. The adoption of a RHEL-based system allows for expedited resolutions of Critical Vulnerabilities and Exposures (CVE) and grants access to RHEL’s Identity Management system, offering superior performance and scalability over open-source FreeIPA. RHEL’s robust automation and management capabilities free up valuable DevOps resources, allowing teams to focus on innovation with Cloudera rather than getting bogged down by the constant maintenance other operating systems demand. This upgrade is particularly beneficial for those looking to leverage next-gen hardware architectures and explore cutting-edge Gen AI use cases, which RHEL is finely tuned to support.

“Cloudera’s move to RHEL for our public cloud offering is a decisive step towards delivering unparalleled security and support for our customers. This evolution underscores our commitment to not just meet, but exceed the expectations of the enterprises and organizations we serve. By harnessing RHEL, we’re enhancing our platform to empower customers with a more secure, scalable, and supported cloud environment, ready to take on the demands of tomorrow’s data and AI challenges.” – Karthik Krishnamoorthy, Vice President of Product Management, Cloudera.

For those transitioning from CentOS Linux (or other distributions), Cloudera will offer an easy migration path. This process will be managed by our teams, ensuring that our customers continue operating smoothly without the need to rebuild their infrastructure.

Cloudera’s commitment to customer success remains a cornerstone of our service model. As part of our integration with RHEL, we are introducing a RHEL mirror, specifically curated to provide the components required for Cloudera use cases. Should customers require packages not included in this specialized mirror, they can easily open a support ticket with us, providing justification for the intended use of the additional components. It’s important to note that this limited RHEL mirror is exclusively accessible for Cloudera deployments on public cloud.

The integration of Cloudera with RHEL marks a significant advancement, offering our customers a seamless experience that combines Cloudera’s ability to tackle data and AI challenges with the reliability Red Hat is known for. This move not only streamlines compliance and security measures, but also delivers a more sustainable and scalable operating system foundation.

“We are pleased to extend our collaboration with Cloudera further as they make Red Hat Enterprise Linux the standard for their public cloud runtime. Together, we are dedicated to providing supported and hybrid cloud-ready solutions that can improve security and position our joint customers at the forefront of technological innovation, including generative AI,” said Gunnar Hellekson, vice president and general manager, Red Hat Enterprise Linux Business Unit, Red Hat.

We invite you to harness the strengths of Cloudera and Red Hat, driving your organization forward with efficiency and innovation. Get started today with our comprehensive resources to guide you through the journey:

• Start our new 5-day trial
• Why Cloudera
• Cloudera Data Platform

The post Cloudera’s RHEL-volution: Powering the Cloud with Red Hat appeared first on Cloudera Blog.

View Details

This year I had the pleasure of joining world leaders, business titans, and changemakers at the 54th World Economic Forum in Davos to grapple with complex challenges that demand collective action. It was validating to see that AI wasn’t just a fringe topic – it was the protagonist.

We clearly saw how much potential AI holds to help create and foster new business opportunities, forge stronger partnerships, find new avenues for dialogue, and achieve more efficient use of emerging technologies. From climate action to economic development, discussions revolved around how AI can be harnessed to solve pressing global problems, but also included a healthy dose of caution for its potential pitfalls.

These are my main takeaways on the future of AI from an action-packed week:

1. Generative AI is a productivity multiplier for workforces (not a replacement) that every company and function needs to adopt to stay competitiveI made this point during a SAP panel discussion on Generative AI – “In the last 40 years there hasn’t been a technology that is going to impact all personas in all industries like Gen AI”.

Analysis shared during the “Thinking through Augmentation” panel reinforced my view. The study found that Generative AI can automate or augment a significant portion of tasks across ALL industries and functional areas. The percentage of tasks impacted ranges from 72% in the financial services sector to 21% in the agricultural sector and from 73% in the IT function to 43% in the supply chain function.

No company can risk missing out on these productivity gains. During the same SAP panel, Niko Mohr of Mckinsey further solidified the benefits of AI: “We found that companies using AI intensively had a 2.5x higher Total Shareholder Return growth rate vs. those that did not”

Generative AI won’t destroy jobs but will make them better as less than half of impacted tasks are expected to be automated: for example only 35% (of 72%) in the financial services sector and 32% (of 73%) in the IT function.

Sam Altman also speaking in Davos put it best: “[Generative AI] will change the world much less than we all think and it will change jobs much less than we all think. We will all operate at a little bit higher level of abstraction. We will all have access to a lot more capability.”

Image: Jobs of Tomorrow: Large Language Models and Jobs

2. The Generative AI revolution is just beginning and significant reskilling is needed to accelerate itA lot of time was spent discussing the exciting future of Generative AI but there was a general consensus that we have a long road ahead of us in fully realizing its promise. One AI expert I talked to said we likely won’t be able to automate 50% of current work activities until 2045 and that’s even a full decade sooner than predictions made before the release of ChatGPT.

Another researcher noted 70% of companies are in exploration mode in terms of Generative AI adoption while only 19% are in pilot or production. My observations echo these findings and Microsoft CEO Sataya Nadella’s comments during his keynote interview: Last year was about the art of possible. This year is about how we get AI to scale.

To this end, several CEOs stressed the need for a widespread reskilling of their workforces to drive usage and see productivity gains. One referenced survey found that executives expect to retrain 40% of their employees over the next three years as a result of implementing AI tools. A lack of internal expertise still remains the number one blocker to deploying AI solutions.

Accenture CEO Julie Sweet did a good job exposing the main driver of the AI skill gap: “[Organizations] have to be willing to upskill as the current workforce has not received this training in their education. We have to partner with governments to change basic education. It’s not going to help now but we need to think 10, 20, 30 years ahead. Education in every country has to change.”

Cloudera has made solving this issue core to our product strategy. Our mission is to democratize AI by making it accessible to enterprises up and down the adoption curve. Cloudera has built a strong enterprise AI partnership ecosystem with leading technology providers to help our customers maximize the value they get from AI. We’ve also created a robust catalog of Applied Machine Learning Prototypes (AMPs) that make it easy for developers to quickly customize, test, and deploy a wide range of AI applications built on foundational models from customer support call transcription to internal Q&A chatbots.

3. Trust is everything, especially with AIMy final takeaway is aligned with the 54th World Economic Forum’s overarching theme of “Rebuilding Trust”. The topic extends beyond fighting corruption in public institutions. Trust is critical to AI. We’ll never transform our economies with AI if we can’t trust its outputs.

The foundation of this trust starts with high-fidelity input data. This is something we understand all too well at Cloudera. The most impactful AI outcomes are built on trusted, secure, data.

The world’s largest companies trust us with their most private and sensitive datasets. Public exposure of this sensitive data from uncontrolled AI use is a key risk our customers are looking to mitigate. They also fear running amok of an international patchwork of data privacy and AI regulations.

All companies need to center security and compliance when developing AI applications to combat these concerns. Adoption will accelerate when leaders can trust that AI applications are secure and compliant.

Salesforce CEO and Davos Veteran Marc Benioff made a similar point during a star-studded panel: “We’re not quite at this threshold moment [with AI] because we don’t quite trust it yet. We have to cross the bridge on trust”

This year’s World Economic Forum was incredibly rewarding to be a part of. The theme of trust, coupled with an emphasis on AI, is deeply aligned with the goals of our solutions at Cloudera. As we march toward the future, it’s exciting to be part of AI capabilities beginning to take hold at such a large scale.

Learn more about how Cloudera can help you build Enterprise AI you can trust.

The post Our Time at Davos: Key Takeaways from the World Economic Forum appeared first on Cloudera Blog.

View Details

As we enter into a new month, the Cloudera team is getting ready to head off to the Gartner Data & Analytics Summit in Orlando, Florida for one of the most important events of the year for Chief Data Analytics Officers (CDAOs) and the field of data and analytics. We’re at a crucial point in time where the excitement and potential surrounding AI has elevated the importance of improving access to the mission-critical data that helps organizations implement it at scale.

This year, members of the Cloudera team will be hosting speaking sessions during the event where we’ll be sharing insights around generative AI and the tools needed to leverage emerging technologies at enterprise scale. Here’s a snapshot of what we’ll be up to during the event:

Understanding and Overcoming the Limits of Large Language ModelsThe potential for AI, and particularly tools like Large Language Models (LLMs), is vast, but that doesn’t come without drawbacks and complications. Even with all the advancements surrounding these technologies, there’s still a critical role for human involvement in fueling these models and generating the best possible business outcomes.

On day two of the summit, March 11 at 5:55 pm EDT, Cloudera Principal Solutions Engineer, Ian Brooks, will be diving into LLMs and exploring how important human involvement is to improving the outputs from those LLMs as part of his presentation, How To Improve AI Systems? Add a Human To The Loop: An Introduction to RLHF & DPO.

Delivering a Competitive Edge with AI at ScaleThe hype cycle with AI is intense and growing by the day. There’s a vast sea of opportunity when it comes to scaling the technology at an enterprise level, creating new use cases and driving greater value. With a deep level of expertise and understanding when it comes to tapping into AI, the conference presents a unique chance to highlight learnings from those who have reached those goals successfully.

On day three of the summit, March 12 at 11:00 am EDT, Cloudera’s Technical Evangelism Director, Jake Bengston, will be a part of the session, Game-Changing Gen AI: Advancing from Pilot to Production at Enterprise Scale. Together with Heeje Cho, senior data engineer at Illumina, and Gregory Floyd, IT Enterprise Architect, Specialist at Southern Company, the session will cover a mix of stories and lessons learned from some of the most seasoned AI leaders at global enterprises. Jake, Heeje, and Gregory will hit on everything from generative AI best practices to follow, avoidable pitfalls, and other learnings aimed to help listeners achieve production-ready generative AI and gain a competitive edge.

This year’s summit is set to be an exciting deep dive into the rapidly evolving world of AI and data and analytics bringing together some of the most innovative companies in the space. In addition to our speaking sessions, attendees can also stop by booth #1534 on-site to connect with Cloudera experts or just have a little fun playing some life-size Jenga or on our Nintendo Switch.

Learn more about this year’s summit and the full lineup of activities and speaking sessions.

The post A Look Ahead at the Gartner Data & Analytics Summit appeared first on Cloudera Blog.

View Details

In this article, we will walk you through the process of implementing fine grained access control for the data governance framework within the Cloudera platform. This will allow a data office to implement access policies over metadata management assets like tags or classifications, business glossaries, and data catalog entities, laying the foundation for comprehensive data access control.

In a good data governance strategy, it is important to define roles that allow the business to limit the level of access that users can have to their strategic data assets. Traditionally we see three main roles in a data governance office:

• Data steward: Defines the business rules for data use according to corporate guidance and data governance requirements.
• Data curator: Assigns and enforces data classification according to the rules defined by the data stewards so that data assets are searchable by the data consumer.
• Data consumer: Derives insights and value from data assets and is keen to understand the quality and consistency of tags and terms applied to the data.

Within the Cloudera platform, whether deployed on premises or using any of the leading public cloud providers, the Cloudera Shared Data Experience (SDX) ensures consistency of all things data security and governance. SDX is a fundamental part of any deployment and relies on two key open source projects to provide its data management functionality: Apache Atlas provides a scalable and extensible set of core governance services, while Apache Ranger enables, monitors, and manages comprehensive security for both data and metadata.

In this article we will explain how to implement a fine grained access control strategy using Apache Ranger by creating security policies over the metadata management assets stored in Apache Atlas.

Case IntroductionIn this article we will take the example of a data governance office that wants to control access to metadata objects in the company’s central data repository. This allows the organization to comply with government regulations and internal security policies. For this task, the data governance team started by looking at the finance business unit, defining roles and responsibilities for different types of users in the organization.

In this example, there are three different users that will allow us to show the different levels of permissions that can be assigned to Apache Atlas objects through Apache Ranger policies to implement a data governance strategy with the Cloudera platform:

• admin is our data steward from the data governance office
• etl_user is our data curator from the finance team
• joe_analyst is our data consumer from the finance team

Note that it would be just as easy to create additional roles and levels of access, if required. As you will see as we work through the example, the framework provided by Apache Atlas and Apache Ranger is extremely flexible and customizable.

First, a set of initial metadata objects are created by the data steward. These will allow the finance team to search for relevant assets as part of their day-to-day activities:

• Classifications (or “tags”) like “PII”, “SENSITIVE”, “EXPIRES_ON”, “DATA QUALITY” etc.
• Glossaries and terms created for the three main business units: “Finance,” “Insurance,” and “Automotive.”
• A business metadata collection called “Project.”

NOTE: The creation of the business metadata attributes is not included in the blog but the steps can be followed here.

Then, in order to control the access to the data assets related to the finance business unit, a set of policies need to be implemented with the following conditions:

The finance data curator <etl_user> should only be allowed to:

• Create/read classifications that start with the word “finance.”
• Read/update entities that are classified with any tag that starts with the word “finance,” and also any entities related to the “worldwidebank” project. The user should also be able to add labels and business metadata to those entities.
• Add/update/remove classifications of the entities with the previous specifications.
• Create/read/update the glossaries and glossary terms related to “finance.”

The finance data consumer <joe_analyst> should only be allowed to:

• View and access cClassifications related to “finance” to search assets.
• View and access entities that are classified with tags related to “finance.”
• View and access the “finance” glossary.

In the following section, the process for implementing these policies will be explained in detail.

Implementation of fine-grained access controls (step by step)In order to meet the business needs outlined above, we will demonstrate how access policies in Apache Ranger can be configured to secure and control metadata assets in Apache Atlas. For this purpose we used a public AMI image to set up a Cloudera Data Platform environment with all SDX components. The process of setting up the environment is explained in this article.

1. Authorization for Classification TypesClassifications are part of the core of Apache Atlas. They are one of the mechanisms provided to help organizations find, organize, and share their understanding of the data assets that drive business processes. Crucially, classifications can “propagate” between entities according to lineage relationships between data assets. See this page for more details on propagation.

1.1 Data Steward – admin userTo control access to classifications, our admin user, in the role of data steward, must perform the following steps:

1. Access the Ranger console.
2. Access Atlas repository to create and manage policies.
3. Create the appropriate policies for the data curator and the data consumer of the finance business unit.

First, access the Atlas Ranger policies repository from the Ranger admin UI

Image 1 – Ranger main page

In the Atlas policy repository:

Image 2 – Atlas policies

The first thing you will see are the default Atlas policies (note 1). Apache Ranger allows specification of access policies as both “allow” rules and “deny” rules. However, it is a recommended good practice in all security contexts to apply the “principle of least privilege”: i.e., deny access by default, and only allow access on a selective basis. This is a much more secure approach than allowing access to everyone, and only denying or excluding access selectively. Therefore, as a first step, you should verify that the default policies don’t grant blanket access to the users we are seeking to restrict in this example scenario.

Then, you can create the new policies (eg. remove the public access of the default policies by creating a deny policy; note 2) and finally you will see that the newly created policies will appear at the bottom of the section (note 3).

After clicking the “Add New Policy” button:

Image 3 – Create policy over finance classification

1. First, define a policy name and, if desired, some policy labels (note 1). These do not have a “functional” effect on the policy, but are an important part of keeping your security policies manageable as your environment grows over time. It is normal to adopt a naming convention for your policies, which may include short-hand descriptions of the user groups and/or assets to which the policy applies, and an indication of its intent. In this case we have chosen the policy name “FINANCE Consumer – Classifications,” and used the labels “Finance.” “Data Governance,” and “Data Curator.”
2. Next, define the type of object on which you want to apply the policy. In this case we will select “type-category” and fill with “Classifications” (note 2).
3. Now, you need to define the criteria used to filter the Apache Atlas objects to be affected by the policy. You can use wildcard notations like “”. To limit the data consumer to only search for classifications starting with the work finance, use FINANCE (note 3).
4. Finally, you need to define the permissions that you want to grant on the policy and the groups and users that are going to be controlled by the policy. In this case, apply the Read Type permission to group: finance and user: joe_analyst and Create Type & Read Type permission to user: etl_user. (note 4)

Now, because they have the Create Type permission for classifications matching FINANCE*, the data curator etl_user can create a new classification tag called “FINANCE_WW” and apply this tag to other entities. This would be useful if a tag-based access policy has been defined elsewhere to provide access to certain data assets.

1.2 Data Curator – etl_user userWe can now demonstrate how the classification policy is being enforced over etl_user. This user is only allowed to see classifications that start with the word finance, but he can also create some additional ones for the different teams under that division.

etl_user can create a new classification tag called FINANCE_WW under a parent classification tag FINANCE_BU.

To create a classification in Atlas:

Image 4 – Atlas classifications tab

1. First, click on the classification panel button (note 1) to be able to see the existing tags that the user has access to. You will be able to see the assets that are tagged with the selected classification. (note 3)
2. Then, click on the “+” button to create a new classification. (note 2)

A new window open, requiring various details to create the new classification.

Image 5 – Atlas classifications creation tab

1. First, provide the name of the classification, in this case FINANCE_WW, and provide a description, so that colleagues will understand how it should be used.
2. Classifications can have hierarchies and those inherit attributes from the parent classification. To create a hierarchy, type the name of the parent tag, in this case FINANCE_BU.
3. Additional custom attributes can also be added to later be used on attribute-based access control (ABAC) policies. This falls outside of the scope of this blog post but a tutorial on the subject can be found here.
4. (Optional) For this example, you can create an attribute called “country,” which will simply help to organize assets. For convenience you can make this attribute a “string” (a free text) type, although in a live system you would probably want to define an enumeration so that users’ inputs are restricted to a valid set of values.

After clicking the button “create” the newly created classification is shown in the panel:

Image 6 – Atlas classifications tree

Now you can click on the toggle button to see the tags in tree mode and you will be able to see the parent/child relationship between both tags.
Click on the classification to view all its details: parent tags, attributes, and assets currently tagged with the classification.

1.3 Data Consumer – joe_analyst userThe last step on the Classification authorization process is to validate from the data consumer role that the controls are in place and the policies are applied correctly.

After successfully logging in with user joe_analyst:

Image 7 – Atlas classifications for finance data consumer

To validate that the policy is applied and that only classifications starting with the word FINANCE can be accessed based on the level of permissions defined in the policy, click on the Classifications tab (note 2) and check the list available. (note 3)

Now, to be able to access the content of the entities (note 4), it is required to give access to the Atlas Entity Type category and to the specific entities with the corresponding level of permissions based on our business requirements. The next section will cover just that.

1. Authorization for Entity Types, Labels and Business Metadata In this section, we will explain how to protect additional types of objects that exist in Atlas, which are important within a data governance strategy; namely, entities, labels, and business metadata.

Entities in Apache Atlas are a specific instance of a “type” of thing: they are the core metadata object that represent data assets in your platform. For example, imagine you have a data table in your lakehouse, stored in the Iceberg table format, called “sales_q3.” This would be reflected in Apache Atlas by an entity type called “ceberg table,” and an entity named “sales_q3,” a particular instance of that entity type. There are many entity types configured by default in the Cloudera platform, and you can define new ones as well. Access to entity types, and specific entities, can be controlled through Ranger policies.

Labels are words or phrases (strings of characters) that you can associate with an entity and reuse for other entities. They are a light-weight way to add information to an entity so you can find it easily and share your knowledge about the entity with others.

Business metadata are sets of related key-value pairs, defined in advance by admin users (for example, data stewards). They are so named because they are often used to capture business details that can help organize, search, and manage metadata entities. For example, a steward from the marketing department can define a set of attributes for a campaign, and add these attributes to relevant metadata objects. In contrast, technical details about data assets are usually captured more directly as attributes on entity instances. These are created and updated by processes that monitor data sets in the data lakehouse or warehouse, and are not typically customized in a given Cloudera environment.

With that context explained, we will move on to setting policies to control who can add, update, or remove various metadata on entities. We can set fine-grained policies separately for both labels and business metadata, as well as classifications. These policies are defined by the data steward, in order to control activities undertaken by data curators and consumers.

2.1 Data Steward – admin userFirst, it’s important to make sure that the users have access to the entity types in the system. This will allow them to filter their search when looking for specific entities.In order to do so, we need to create a policy:

Image 8 – Atlas entity type policies

In the create policy page, define the name and labels as described before. Then, select the type-category “entity”(note 1). Use the wildcard notation () (note 2) to denote all entity types, and grant all available permissions to etl_user and joe_analyst.(note 3*)
This will enable these users to see all the entity types in the system.

The next step is to allow data consumer joe_analyst to only have read access on the entities that have the finance classification tags. This will limit the objects that he will be able to see on the platform.

To do this, we need to follow the same process to create policies as shown in the previous section, but with some modifications on the policy details:

Image 9 – Example Atlas finance entity policies

1. As always, name (and label) the policy to enable easy management later.
2. The first important change is that the policy is applied on an “entity-type” and not in a “type-category.” Select “entity-type” in the drop-down menu (note 2) and type the wildcard to apply it to all the entity types.
3. Some additional fields will appear in the form. In the entity classification field you can specify tags that exist on the entities you want to control. In our case, we want to only allow objects that are tagged with words that start with “finance.” Use the expression FINANCE. (note 3*)
4. Next, filter the entities to be controlled through the entity ID field. In this exercise, we will use the wildcard () (note 4) and for the additional fields we will select “none.” This button will update the list of permissions that can be enforced in the conditions panel. (note 4)*
5. As a data consumer, we want the joe_analyst user to be able to see the entities. To implement this, select the Read Entity permission. (note 5)
6. Add a new condition for the data curator etl_user but this time include permissions to modify the tags appropriately, by adding the Add Classification, Update Classification & Remove Classification permissions to the specific user.

In this way, access to specific entities can be controlled using additional metadata objects like classification tags. Atlas provides some other metadata objects that can be used not only to enrich the entities registered in the platform, but also to implement a governance strategy over those objects, controlling who can access and modify them. This is the case for the labels and the business metadata.

If you want to enforce some control over who can add or remove labels:

Image 10 – Example Atlas finance label policy

1. The only difference between setting a policy for labels versus the previous examples is setting the additional fields filter to “entity-label” as shown in the image and fill with the values of labels that want to be controlled. In this case, we use the wildcard () to enable operations on any label on entities tagged with FINANCE classifications.
2. When the entity-label is selected from the drop-down, the permissions list will be updated. Select Add Label & Remove Label permission to grant the data curator the option to add and remove labels from entities.

The same principle can be applied to control the permissions over business metadata:

Image 11 – Example Atlas finance business metadata policy

1. In this case, one must set the additional fields filter to “entity-business-metadata” as shown in the image and fill with the values of business metadata attributes that want to be protected. In this example, we use the wildcard () to enable operations on all business metadata attributes on entities tagged with FINANCE classifications.
2. When you enable the entity-business-metadata drop-down, the permissions list will be updated. Select Update Business Metadata permission to grant the data curator the option to modify the business metadata attributes of financial entities.

As part of the fine grained access control provided by Apache Ranger over Apache Atlas objects, one can create policies that use an entity ID to specify the exact objects to be controlled. In the examples above we have often used the wildcard (*) to refer to “all entities;” below, we will show a more targeted use-case.

In this scenario, we want to create a policy pertaining to data tables which are part of a specific project, named “World Wide Bank.” As a standard, the project owners required that all the tables are stored in a database called “worldwidebank.”

To meet this requirement, we can use one of the entity types pre-configured in Cloudera’s distributions of Apache Atlas, namely “hive_table”. For this entity type, identifiers always begin with the name of the database to which the table belongs. We can leverage that, using Ranger expressions to filter all the entities that belong to the “World Wide Bank” project.

To create a policy to protect the worldwidebank entities:

Image 12 – Example Atlas Worldwide Bank entity policy

1. Create a new policy, but this time don’t specify any entity classification, use the wildcard “*” expression.
2. In the entity ID field use the expression: worldwidebank
3. In the Conditions, select the permissions Read Entity, Update Entity, Add Classification, Update Classification & Remove Classification to the data curator etl_user to be able to see the details of these entities and enrich/modify and tag them as needed.

2.2 Data Curator – etl_user userIn order to allow finance data consumer joe_analyst to use and access the worldwidebank project entities, the data curator etl_user must tag the entities with the approved classifications and add the required labels and business metadata attributes.

Login to Atlas and follow the process to tag the appropriate entities:

Image 13 – Data curator entity search

1. First, search for the worldwidebank assets using the search bar. You can also use the “search by type” filter on the left panel to limit the search to the “hive_db” entity type.
2. As data curator, you should be able to see the entity and be allowed to access the details of the worldwidebank database entity. It should have a clickable link to the entity object
3. Click on the entity object to see its details.

After clicking the entity name, the entity details page is shown:

Image 14 – Worldwide Bank database entity detail

In the top of the screen, you can see the classifications assigned to the entity. In this case there are no tags assigned. We will assign one by clicking on the “+” sign.

In the “Add Classification” screen:

Image 15 – Worldwide Bank database tag process

1. Search for the FINANCE_WW tag and select it.
2. Then fill the appropriate attributes if the classification tag has any. (Optional in Image 5, in the 1.2 Data Curator – etl_user user section above.)
3. Click on “add.”

That will tag an entity with the selected classification.

Now, enrich the worldwidebank hive_db entity with a new label and a new business metadata attribute called “Project.”

Image 16 – Worldwide Bank database tag process

Now, enrich the worldwidebank hive_db entity with a new label and a new business metadata attribute called “Project.”

1. To add a label, click “Add” on the labels menu.
2. Type the label in the space and click “save.”
3. To add a business metadata attribute, click “Add” on the business metadata menu.
4. Click on “Add New Attribute” if it’s not assigned or “edit” if it already exists.
5. Select the attribute you want to add and fill the details and hit “save.”

NOTE: The creation of the business metadata attributes is not included in the blog but the steps can be followed here.

With the “worldwidebank” Hive object tagged with the “FINANCE_WW” classification, the data consumer should be able to have access to it and see the details. Also, it is important to validate that the data consumer also has access to all the other entities tagged with any classification that starts with “finance.”

2.3 Data Consumer – joe_analyst userTo validate that the policies are applied correctly, login into Atlas:

Image 17 – Finance data assets

Click on the classifications tab and validate:

• The list of tags that are visible based on the policies created in the previous steps. All the policies must start with the word “finance.”
• Click on the FINANCE_WW tag and validate the access to the “worldwidebank” hive_db object.

After clicking on the “worldwidebank” object:

Image 18 – WorldWideBank database asset details

You can see all the details of the asset that where enriched by the finance data curator in previous steps:

• You should see all the technical properties of the asset.
• You should be able to see the tags applied to the asset
• You should see the labels applied to the asset.

• You should see the business metadata attributes assigned to the asset.

• Authorization for Glossary and Glossary TermsIn this section, we will explain how a data steward can create policies to allow fine-grained access controls over glossaries and glossary terms. This allows data stewards to control who can access, enrich or modify glossary terms to protect the content from unauthorized access or mistakes.

A glossary provides appropriate vocabularies for business users and it allows the terms (words) to be related to each other and categorized so that they can be understood in different contexts. These terms can be then applied to entities like databases, tables, and columns. This helps abstract the technical jargon associated with the repositories and allows the user to discover and work with data in the vocabulary that is more familiar to them.

Glossaries and terms can also be tagged with classifications. The benefit of this is that, when glossary terms are applied to entities, any classifications on the terms are passed on to the entities as well. From a data governance process perspective, this means that business users can enrich entities using their own terminology, as captured in glossary terms, and that can automatically apply classifications as well, which are a more “technical” mechanism, used in defining access controls, as we have seen.

First, we will show how as a data steward you can create a policy that grants read access to glossary objects with specific words in the name and validate that the data consumer is allowed to access the specific content.

3.1 Data Steward – admin userTo create a policy to control access to glossaries and terms, you can:

Image 19 – Glossary control policy

1. Create a new policy, but this time use the “entity-type” AtlasGlossary and AtlasGlossaryTerm. (note 1)
2. In the entity classifications field, use the wildcard expression: *
3. The entity ID is where you can define which glossaries and terms you want to protect. In Atlas, all the terms of a glossary include a reference to it with an “@” at the end of its name (ex. term@glossary). To protect the “Finance” glossary itself, use Finance; and to protect is terms, use @Finance (note 2).
4. In the Conditions, select the permissions Read Entity to the data consumer joe_analyst to be able to see the glossary and its terms. (note 3)

3.2 Data Consumer – joe_analyst userTo validate that only “Finance” glossary objects can be accessed:

Image 20 – Finance Atlas glossary

1. Click on the glossary tab in the Atlas panel.
2. Check the glossaries available in the Atlas UI and the access to the details of the terms of the glossary.

ConclusionThis article has shown how an organization can implement a fine grained access control strategy over the data governance components of the Cloudera platform, leveraging both Apache Atlas and Apache Ranger, the fundamental and integral components of SDX. Although most organizations have a mature approach to data access, control of metadata is typically less well defined, if considered at all. The insights and mechanisms shared in this article can help implement a more complete approach to data as well as metadata governance. The approach is critical in the context of a compliance strategy where data governance components play a critical role.

You can learn more about SDX here; or, we would love to hear from you to discuss your specific data governance needs.

The post Metadata Management & Data Governance with Cloudera SDX appeared first on Cloudera Blog.

View Details

Artificial Intelligence (AI) is primed to reshape the way just about every business operates. Cloudera research projected that more than one third (36%) of organizations in the U.S. are in the early stages of exploring the potential for AI implementation. But even with its rise, AI is still a struggle for some enterprises. AI, and any analytics for that matter, are only as good as the data upon which they are based. And that’s where the rub is. Struggling to access and collect, oftentimes disparate and siloed, data across environments that are required to power AI, many organizations are unable to achieve the business insight and value they had hoped for. Faced with unique challenges around distributed data infrastructures, governance, and an evolving security landscape, enterprises need the right support to fully tap into AI quickly.

To power our customers’ data, AI, and analytics needs, we are unveiling the next phase of our open data lakehouse, featuring several enhancements built to quickly scale enterprise AI and deliver unprecedented business value. Cloudera is now the only provider to offer an open data lakehouse with Apache Iceberg for cloud and on-premises. This marks a significant milestone for the platform: according to IDC, today about half of the world’s enterprise production data under management is on-prem. The latest release of the Cloudera platform delivers a one-of-a-kind set of capabilities to bring the same open data lakehouse functionality from the cloud into those data centers. The platform is ready to address the complexities of managing highly sensitive, yet critical, company data while still extracting the most value from its use.

Let’s dive deeper into three of the most impactful features included in this update.

Apache IcebergThe addition of Apache Iceberg support for the Cloudera platform unlocks opportunities for enterprises to apply mission-critical data to AI and address some of the most error-prone processes, enabling them to generate new use cases, improve overall performance, and reduce costs. Iceberg delivers the open table format so that enterprises can put AI to work on their data all in an on-premises setting. This approach brings new compute engines into the fold, adding Spark, Flink, Impala, and NiFi, enabling concurrent access and processing of datasets within Iceberg.

With built-in features like time travel, schema evolution, and streamlined data discovery, Iceberg empowers data teams to enhance data lake management while upholding data integrity. Things like in-place schema evolution and ACID transactions on the data lakehouse are critical pieces for organizations as they push to achieve regulatory compliance and adhere to policies like the General Data Protection Regulation (GDPR). The powerful platform data security and governance layer, Shared Data Experience (SDX), is a fundamental part of the open data lakehouse, in the data center just as it is in the cloud.

Apache OzoneAs AI and other advanced analytics continue to grow in scale, performance and scalable data storage will need to expand right along with them. Specifically for the data center, Apache Ozone delivers greater scalability, at a lower cost, helping organizations drive greater business value. With the Cloudera platform’s latest update, new features give customers the tools they need to incorporate greater security and strengthen enterprise readiness. The latest generation of our platform includes Ozone features like improved replication, improved quotas for volumes, buckets to facilitate cloud-native architectures, and snapshots, which are also now able to support data storage at the bucket and volume levels.

Zero Downtime UpgradesBeyond improvements to Iceberg and Ozone, the platform now boasts Zero Downtime Upgrades (ZDU). ZDU gives organizations a more convenient means of upgrading. Rolling upgrades are now supported for HDFS, Hive, HBase, Kudu, Kafka, Ranger, YARN, and Ranger KMS. ZDU ensures customers experience minimal workflow disruptions and ultimately reduce or even eliminate lengthy and costly downtimes.

By adding ZDU, customers get a powerful boost to productivity with capabilities like one-stage upgrades and auto upgrades of large clusters. And for the platform components that are still expected to experience downtime, this update ensures they are optimized through Cloudera Manager and able to quickly restart. This marks a key improvement to previous iterations where some of the services, like Queue Manager, were often the first pieces to go down and some of the last ones to restart. Those services are now able to get back up and running in a matter of minutes, right at the start of the ZDU.

AI is quickly cementing itself as a key part of generating maximum business value out of enterprise data. Getting to that value though, means utilizing data and analytics in the environment that they are most well-suited to run—that’s what makes a hybrid approach so crucial. And that’s also what makes Cloudera so unique. The Cloudera platform offers portable, cloud-native, analytics that can be deployed across infrastructures, all while maintaining consistent data governance and security. Available for cloud and now also for the data center.

Learn more about the next generation of Cloudera Data Platform for Private Cloud.

The post A Closer Look at The Next Phase of Cloudera’s Hybrid Data Lakehouse appeared first on Cloudera Blog.

View Details

Businesses often need to aggregate topics because it is essential for organizing, simplifying, and optimizing the processing of streaming data. It enables efficient analysis, facilitates modular development, and enhances the overall effectiveness of streaming applications. For example, if there are separate clusters, and there are topics with the same purpose in the different clusters, then it is useful to aggregate the content into one topic.

This blog post walks you through how you can use prefixless replication with Streams Replication Manager (SRM) to aggregate Kafka topics from multiple sources. To be specific, we will be diving deep into a prefixless replication scenario that involves the aggregation of two topics from two separate Kafka clusters into a third cluster.

This tutorial demonstrates how to set up the SRM service for prefixless replication, how to create and replicate topics with Kafka and SRM command line (CLI) tools, and how to verify your setup using Streams Messaging Manger (SMM). Security setup and other advanced configurations are not discussed.

Before you beginThe following tutorial assumes that you are familiar with SRM concepts like replications and replication flows, replication policies, the basic service architecture of SRM, as well as prefixless replication. If not, you can check out this related blog post. Alternatively, you can read about these concepts in our SRM Overview.

Scenario overviewIn this scenario you have three clusters. All clusters contain Kafka. Additionally, the target cluster (srm-target) has SRM and SMM deployed on it.

The SRM service on srm-target is used to pull Kafka data from the other two clusters. That is, this replication setup will be operating in pull mode, which is the Cloudera-recommended architecture for SRM deployments.

In pull mode, the SRM service (specifically the SRM driver role instances) replicates data by pulling from their sources. So rather than having SRM on source clusters pushing the data to target clusters, you use SRM located on the target cluster to pull the data into its co-located Kafka cluster.Pull mode is recommended as it is the deployment type that was found to provide the highest amount of resilience against various timeout and network instability issues. You can find a more in-depth explanation of pull mode in the official docs.

The records from both source topics will be aggregated into a single topic on the target cluster. All the while, you will be able to use SMM’s powerful UI features to monitor and verify what’s happening.

Set up SRMFirst, you need to set up the SRM service located on the target cluster.

SRM needs to know which Kafka clusters (or Kafka services) are targets and which ones are sources, where they are located, how it can connect and communicate with them, and how it should replicate the data. This is configured in Cloudera Manager and is a two-part process. First, you define Kafka credentials, then you configure the SRM service.

Define Kafka credentialsYou define your source (external) clusters using Kafka Credentials. A Kafka Credential is an item that contains the properties required by SRM to establish a connection with a cluster. You can think of a Kafka credential as the definition of a single cluster. It contains the name (alias), address (bootstrap servers), and credentials that SRM can use to access a specific cluster.

1. In Cloudera manager, go to the Administration > External Accounts > Kafka Credentials page.
2. Click “Add Kafka Credentials.”
3. Configure the credential.

The setup in this tutorial is minimal and unsecure, so you only need to configure Name, Bootstrap Servers, and Security Protocol lines. The security protocol in this case is PLAINTEXT.

1. Click “Add” once you’re done, and repeat the previous step for the other cluster (srm2).

Configure the SRM serviceAfter the credentials are set up, you’ll need to configure various SRM service properties. These properties specify the target (co-located) cluster, tell SRM what replications should be enabled, and that replication should happen in prefixless mode. All of this is done on the configuration page of the SRM service.
1. From the Cloudera Manager home page, select the “Streams Replication Manager” service.
2. Go to “Configuration.”
3. Specify the co-located cluster alias with “Streams Replication Manager Co-located Kafka Cluster Alias.”
The co-located cluster alias is the alias (short name) of the Kafka cluster that SRM is deployed together with. All clusters in an SRM deployment have aliases. You use the aliases to refer to clusters when configuring properties and when running the srm-control tool. Set this to:

Notice that you only need to specify the alias of the co-located Kafka cluster, entering connection information like you did for the external clusters is not ended. This is because Cloudera Manager passes this information automatically to SRM.

1. Specify External Kafka Accounts.This property must contain the names of the Kafka credentials that you created in a previous step. This tells SRM which Kafka credentials it should import to its configuration. Set this to:

2. Specify all cluster aliases with “Streams Replication Manager Cluster” alias.
   The property contains a comma-delimited list of all cluster aliases. That is, all aliases you previously added to the Streams Replication Manager Co-located Kafka Cluster Alias and External Kafka Accounts properties. Set this to:

3. Specify the driver role target with Streams Replication Manager Driver Target Cluster.
   The property contains a comma-delimited list of all cluster aliases. That is, all aliases you previously added to the Streams Replication Manager Co-located Kafka Cluster Alias and External Kafka Accounts properties. Set this to:

4. Specify service role targets with Streams Replication Manager Service Target Cluster.
   This property specifies the cluster that the SRM service role will gather replication metrics from (i.e. monitor). In pull mode, the service roles must always target their co-located cluster. Set this to:

5. Specify replications with Streams Replication Manager’s Replication Configs.
   This property is a jack-of-all-trades and is used to set many SRM properties that are not directly available in Cloudera Manager. But most importantly, it is used to specify your replications. Remove the default value and add the following:

6. Select “Enable Prefixless Replication”
   This property enables prefixless replication and tells SRM to use the IdentityReplicationPolicy, which is the ReplicationPolicy that replicates without prefixes.

7. Review your configuration, it should look like this:

8. Click “Save Changes” and restart SRM.

Create a topic, produce some recordsNow that SRM setup is complete, you need to create one of your source topics and produce some data. This can be done using the kafka-producer-perf-test CLI tool.

This tool creates the topic and produces the data in one go. The tool is available by default on all CDP clusters, and can be called directly by typing its name. No need to specify full paths.

1. Using SSH, log in to one of your source cluster hosts.
2. Create a topic and produce some data.

Notice that the tool will produce 2000 records. This will be important later on when we verify replication on the SMM UI.

Replicate the topicSo, you have SRM set up, and your topic is ready. Let’s replicate.

Although your replications are set up, SRM and the source clusters are connected, data is not flowing, the replication is inactive. To activate replication, you need to use the srm-control CLI tool to specify what topics should be replicated.

Using the tool you can manipulate the replication to allow and deny lists (or topic filters), which control what topics are replicated. By default, no topic is replicated, but you can change this with a few simple commands.

1. Using SSH, log in to the target cluster (srm-target).
2. Run the following commands to start replication.

Notice that even though the topic on srm2 doesn’t exist yet, we added the topic to the replication allow list as well. The topic will be created later. In this case, we are activating its replication ahead of time.

Insights with SMMNow that replication is activated, the deployment is in the following state:

In the next few steps, we will shift the focus to SMM to demonstrate how you can leverage its UI to gain insights into what is actually going on in your target cluster.

Notice the following:

1. The name of the replication is included in the name of the producer that created the topic. The -> notation means replication. Therefore, the topic was created with replication.
2. The topic name is the same as on the source cluster. Therefore, it was replicated with prefixless replication. It does not have the source cluster alias as a prefix.
3. The producer wrote 2,000 records. This is the same amount of records that you produced in the source topic with kafka-producer-perf-test.
4. “MESSAGES IN” shows 2,000 records. Again, the same amount that was originally produced.

On to aggregation After successfully replicating data in a prefixless fashion, its time move forward and aggregate the data from the other source cluster. First you’ll need to set up the test topic in the second source cluster (srm2), as it doesn’t exist yet. This topic must have the exact same name and configurations as the one on the first source cluster (srm1).

To do this, you need to run kafka-producer-perf-test again, but this time on a host of the srm2 cluster. Additionally, for bootstrap you’ll need to specify srm2 hosts.

Notice how only the bootstraps are different from the first command. This is crucial, the topics on the two clusters must be identical in name and configuration. Otherwise, the topic on the target cluster will constantly switch between two configuration states. Additionally, if the names do not match, aggregation will not happen.

After the producer is finished with creating the topic and producing the 2000 records, the topic is immediately replicated. This is because we preactivated replication of the test topic in a previous step. Additionally, the topic records are automatically aggregated into the test topic on srm-target.

You can verify that aggregation has happened by having a look at the topic in the SMM UI.

The following indicates that aggregation has happened:

1. There are now two producers instead of one. Both contain the name of the replication. Therefore, the topic is getting records from two replication sources.
2. The topic name is still the same. Therefore, perfixless replication is still working.
3. Both producers wrote 2,000 records each.
4. “MESSAGES IN” shows 4,000 records.

SummaryIn this blog post we looked at how you can use SRM’s prefixless replication feature to aggregate Kafka topics from multiple clusters into a single target cluster.

Although aggregation was in focus, note that prefixless replication can be used for non-aggregation type replication scenarios as well. For example, it is the perfect tool to migrate that old Kafka deployment running on CDH, HDP, or HDF to CDP.

If you want to learn more about SRM and Kafka in CDP Private Cloud Base, hop over to Cloudera’s doc portal and see Streams Messaging Concepts, Streams Messaging How Tos, and/or the Streams Messaging Migration Guide.

To get hands on with SRM, download Cloudera Stream Processing Community edition here.

Interested in joining Cloudera?At Cloudera, we are working on fine-tuning big data related software bundles (based on Apache open-source projects) to provide our customers a seamless experience while they are running their analytics or machine learning projects on petabyte-scale datasets. Check our website for a test drive!

If you are interested in big data, would like to know more about Cloudera, or are just open to a discussion with techies, visit our fancy Budapest office at our upcoming meetups.

Or, just visit our careers page, and become a Clouderan!

The post Using Streams Replication Manager Prefixless Replication for Kafka Topic Aggregation appeared first on Cloudera Blog.

View Details

It’s hard to believe it’s been 15 years since the global financial crisis of 2007/2008. While this might be a blast from the past we’d rather leave in the proverbial rear-view mirror, in March of 2023 we were back to the future with the collapse of Silicon Valley Bank (SVB), the largest US bank to fail since 2008.

While there are clear reasons SVB collapsed, which can be reviewed here, my purpose in this post isn’t to rehash the past but to present some of the regulatory and compliance challenges financial (and to some degree insurance) institutions face and how data plays a role in mitigating and managing risk. There will inevitably be another global financial crisis, but robust data capabilities allow institutions globally to better adapt to regulations, implement compliance strategies, and predict risk. It’s a future state worth investing in. I’m reminded of a quote by Dr. Emmett Brown in Back To The Future, Part III: “Marty, the future isn’t written. It can be changed…you know that. Anyone can make their future whatever they want it to be.”

Well, sort of.

The opportunities are great, but so are the challengesTo make that future a reality, regulatory challenges confronting banks and insurance companies demand a constant reassessment of compliance strategies and operational frameworks. From stringent data protection measures to complex risk management protocols, institutions must not only adapt to regulatory shifts but also proactively anticipate emerging requirements, as well as predict negative outcomes.

To get there, institutions need to navigate some serious challenges, including:

• Technological innovation: New technologies, such as artificial intelligence, cloud computing, and data processing, can present difficulties for compliance departments and industry regulators, especially the risk and compliance requirements associated with data, infrastructure, and security.
• Regulatory compliance: Compliance has become a significant challenge due to the increasing number of regulations that banks and insurance companies must adhere to. Global bodies such as the Financial Stability Board (FSB), the International Organization of Securities Commissions (IOSCO), the International Association of Insurance Supervisors (IAIS), and the Basel Committee for Banking Supervision (BCBS) all impact the work of national regulators, and ultimately compliance within the institution. On top of these, each nation has its own list of regulations.
• Cybersecurity and data privacy: As news of data breaches become increasingly common, resulting in elevated privacy concerns, regulatory and compliance standards are becoming more stringent. Financial institutions must navigate a complex landscape of privacy laws, such as GDPR, the UK Data Protection Act 2018, and the California Consumer Privacy Act (CCPA). They are also subject to the Digital Operational Resilience Act (DORA), which is the European Union’s regulation aiming to strengthen the IT security of financial entities such as banks, insurance companies, and investment firms, and ensure that the financial sector in Europe is resilient against cyber threats and operational disruptions.
• Fintech and regulatory arbitrage: The rise of fintech firms poses challenges for financial regulation authorities, as these entities may be subject to little or no oversight. This can lead to regulatory arbitrage, where firms, including banks, engage in activities that exploit loopholes or the differences in regulatory standards between jurisdictions.
• Risk management and governance: Regulators are increasingly focusing on risk governance, risk sustainability, and the detection, mitigation, tracking, and remediation of threat actors. Financial institutions must demonstrate robust risk accountability and governance, as well as maintain consumer protections.
• Cultural shift and technology adoption: Traditional banks and insurance companies must adapt to the emergence of fintech firms and changing business models. This requires a shift towards a technology-first attitude and the adoption of digital solutions to address industry challenges.
• TBTM (too big to manage): A financial institution is TBTM when growth and complexity hinder effective oversight and the ability to manage complex operations. Institutions deemed TBTM may face added regulatory scrutiny as they are viewed to inevitably have persistent weaknesses and commit repeat offenses.
• AI Regulation: The most notable, the European Union’s AI Act, focuses on the safe, responsible, fair, and transparent use of AI technologies. The EU’s initial objectives aim to “ensure that AI systems placed on the Union market and used are safe and respect existing law on fundamental rights and Union values; ensure legal certainty to facilitate investment and innovation in AI; enhance governance and effective enforcement of existing law on fundamental rights and safety requirements applicable to AI systems and facilitate the development of a single market for lawful, safe and trustworthy AI applications and prevent market fragmentation.”

Seeing the future in a modern data architectureThe key to successfully navigating these challenges lies in the adoption of a modern data architecture. By leveraging cutting-edge technology and an efficient framework for managing, analyzing, and securing data, financial institutions can streamline operations and enhance their ability to meet compliance requirements efficiently, while maintaining a strong focus on risk management.

Some of the key benefits of a modern data architecture for regulatory compliance include:

• Enhanced data governance and compliance: Modern data architecture incorporates data governance practices and security controls to ensure data privacy, regulatory compliance, and protection against unauthorized access or breaches.
• Improved data accessibility: By providing self-service data access and analytics, modern data architecture empowers business users and data analysts to analyze and visualize data, enabling faster decision-making and response to regulatory requirements.
• Data integration and ingestion: With robust data integration capabilities, a modern data architecture makes real-time data ingestion from various sources—including structured, unstructured, and streaming data, as well as external data feeds—a reality.
• Scalability and future-proofing: Modern data architecture offers robust data integration capabilities, allowing efficient and real-time data ingestion from various sources, including structured databases, unstructured data, streaming data, and external data feeds.
• Collaboration and communication: Modern data architecture fosters cooperation among data engineers, data scientists, and operations teams, leading to more efficient deployment, monitoring, and maintenance of data pipelines. This results in enhanced efficiency in compliance processes.
• Risk management and compliance: By leveraging advanced analytics techniques and providing real-time insights, modern data architecture helps financial institutions better manage risk and maintain compliance.
• Regulatory reporting: A modern data architecture can support banks, financial services firms, and insurers in meeting changing regulatory reporting requirements by providing a comprehensive and efficient framework for data governance, management, and analysis.

The promise of a modern data architecture might seem like a distant reality, but we at Cloudera believe data can make what is impossible today, possible tomorrow. By implementing a modern data architecture with Cloudera Data Platform (CDP), financial and insurance institutions can effectively address the challenges posed by the rapidly evolving regulatory landscape and ensure compliance with global and national regulations.

The post Back to the Financial Regulatory Future appeared first on Cloudera Blog.

View Details

IntroductionApache Iceberg has recently grown in popularity because it adds data warehouse-like capabilities to your data lake making it easier to analyze all your data—structured and unstructured. It offers several benefits such as schema evolution, hidden partitioning, time travel, and more that improve the productivity of data engineers and data analysts. However, you need to regularly maintain Iceberg tables to keep them in a healthy state so that read queries can perform faster. This blog discusses a few problems that you might encounter with Iceberg tables and offers strategies on how to optimize them in each of those scenarios. You can take advantage of a combination of the strategies provided and adapt them to your particular use cases.

Problem with too many snapshotsEverytime a write operation occurs on an Iceberg table, a new snapshot is created. Over a period of time this can cause the table’s metadata.json file to get bloated and the number of old and potentially unnecessary data/delete files present in the data store to grow, increasing storage costs. A bloated metadata.json file could increase both read/write times because a large metadata file needs to be read/written every time. Regularly expiring snapshots is recommended to delete data files that are no longer needed, and to keep the size of table metadata small. Expiring snapshots is a relatively cheap operation and uses metadata to determine newly unreachable files.

Solution: expire snapshotsWe can expire old snapshots using expire_snapshots

Problem with suboptimal manifestsOver time the snapshots might reference many manifest files. This could cause a slowdown in query planning and increase the runtime of metadata queries. Furthermore, when first created the manifests may not lend themselves well to partition pruning, which increases the overall runtime of the query. On the other hand, if the manifests are well organized into discrete bounds of partitions, then partition pruning can prune away entire subtrees of data files.

Solution: rewrite manifestsWe can solve the too many manifest files problem with rewrite_manifests and potentially get a well-balanced hierarchical tree of data files.

Problem with delete filesBackgroundmerge-on-read vs copy-on-writeSince Iceberg V2, whenever existing data needs to be updated (via delete, update, or merge statements), there are two options available: copy-on-write and merge-on-read. With the copy-on-write option, the corresponding data files of a delete, update, or merge operation will be read and entirely new data files will be written with the necessary write modifications. Iceberg doesn’t delete the old data files. So if you want to query the table before the modifications were applied you can use the time travel feature of Iceberg. In a later blog, we will go into details about how to take advantage of the time travel feature. If you decided that the old data files are not needed any more then you can get rid of them by expiring the older snapshot as discussed above.

With the merge-on-read option, instead of rewriting the entire data files during the write time, simply a delete file is written. This can be an equality delete file or a positional delete file. As of this writing, Spark doesn’t write equality deletes, but it is capable of reading them. The advantage of using this option is that your writes can be much quicker as you are not rewriting an entire data file. Suppose you want to delete a specific user’s data in a table because of GDPR requirements, Iceberg will simply write a delete file specifying the locations of the user data in the corresponding data files where the user’s data exist. So whenever you are reading the tables, Iceberg will dynamically apply those deletes and present a logical table where the user’s data is deleted even though the corresponding records are still present in the physical data files.

We enable the merge-on-read option for our customers by default. You can enable or disable them by setting the following properties based on your requirements. See Write properties.

Serializable vs snapshot isolationThe default isolation guarantee provided for the delete, update, and merge operations is serializable isolation. You could also change the isolation level to snapshot isolation. Both serializable and snapshot isolation guarantees provide a read-consistent view of your data. Serializable Isolation is a stronger guarantee. For instance, you have an employee table that maintains employee salaries. Now, you want to delete all records corresponding to employees with salary greater than $100,000. Let’s say this salary table has five data files and three of those have records of employees with salary greater than $100,000. When you initiate the delete operation, the three files containing employee salaries greater than $100,000 are selected, then if your “delete_mode” is merge-on-read a delete file is written that points to the positions to delete in those three data files. If your “delete_mode” is copy-on-write, then all three data files are simply rewritten.

Irrespective of the delete_mode, while the delete operation is happening, assume a new data file is written by another user with a salary greater than $100,000. If the isolation guarantee you chose is snapshot, then the delete operation will succeed and only the salary records corresponding to the original three data files are removed from your table. The records in the newly written data file while your delete operation was in progress, will remain intact. On the other hand, if your isolation guarantee was serializable, then your delete operation will fail and you will have to retry the delete from scratch. Depending on your use case you might want to reduce your isolation level to “snapshot.”

The problemThe presence of too many delete files will eventually reduce the read performance, because in Iceberg V2 spec, everytime a data file is read, all the corresponding delete files also need to be read (the Iceberg community is currently considering introducing a concept called “delete vector” in the future and that might work differently from the current spec). This could be very costly. The position delete files might contain dangling deletes, as in it might have references to data that are no longer present in any of the current snapshots.

Solution: rewrite position deletesFor position delete files, compacting the position delete files mitigates the problem a little bit by reducing the number of delete files that need to be read and offering faster performance by better compressing the delete data. In addition the procedure also deletes the dangling deletes.

Rewrite position delete files

Iceberg provides a rewrite position delete files procedure in Spark SQL.

But the presence of delete files still pose a performance problem. Also, regulatory requirements might force you to eventually physically delete the data rather than do a logical deletion. This can be addressed by doing a major compaction and removing the delete files entirely, which is addressed later in the blog.

Problem with small filesWe typically want to minimize the number of files we are touching during a read. Opening files is costly. File formats like Parquet work better if the underlying file size is large. Reading more of the same file is cheaper than opening a new file. In Parquet, typically you want your files to be around 512 MB and row-group sizes to be around 128 MB. During the write phase these are controlled by “write.target-file-size-bytes” and “write.parquet.row-group-size-bytes” respectively. You might want to leave the Iceberg defaults alone unless you know what you are doing.

In Spark for example, the size of a Spark task in memory will need to be much higher to reach those defaults, because when data is written to disk, it will be compressed in Parquet/ORC. So getting your files to be of the desirable size is not easy unless your Spark task size is big enough.

Another problem arises with partitions. Unless aligned properly, a Spark task might touch multiple partitions. Let’s say you have 100 Spark tasks and each of them needs to write to 100 partitions, together they will write 10,000 small files. Let’s call this problem partition amplification.

Solution: use distribution-mode in writeThe amplification problem could be addressed at write time by setting the appropriate write distribution mode in write properties. Insert distribution is controlled by “write.distribution-mode” and is defaulted to none by default. Delete distribution is controlled by “write.delete.distribution-mode” and is defaulted to hash, Update distribution is controlled by “write.update.distribution-mode” and is defaulted to hash and merge distribution is controlled by “write.merge.distribution-mode” and is defaulted to none.

The three write distribution modes that are available in Iceberg as of this writing are none, hash, and range. When your mode is none, no data shuffle occurs. You should use this mode only when you don’t care about the partition amplification problem or when you know that each task in your job only writes to a specific partition.

When your mode is set to hash, your data is shuffled by using the partition key to generate the hashcode so that each resultant task will only write to a specific partition. When your distribution mode is range, your data is distributed such that your data is ordered by the partition key or sort key if the table has a SortOrder.

Using the hash or range can get tricky as you are now repartitioning the data based on the number of partitions your table might have. This can cause your Spark tasks after the shuffle to be either too small or too large. This problem can be mitigated by enabling adaptive query execution in spark by setting “spark.sql.adaptive.enabled=true” (this is enabled by default from Spark 3.2). Several configs are made available in Spark to adjust the behavior of adaptive query execution. Leaving the defaults as is unless you know exactly what you are doing is probably the best option.

Even though the partition amplification problem could be mitigated by setting correct write distribution mode appropriate for your job, the resultant files could still be small just because the Spark tasks writing them could be small. Your job cannot write more data than it has.

Solution: rewrite data filesTo address the small files problem and delete files problem, Iceberg provides a feature to rewrite data files. This feature is currently available only with Spark. The rest of the blog will go into this in more detail. This feature can be used to compact or even expand your data files, incorporate deletes from delete files corresponding to the data files that are being rewritten, provide better data ordering so that more data could be filtered directly at read time, and more. It is one of the most powerful tools in your toolbox that Iceberg provides.

RewriteDataFiles

Iceberg provides a rewrite data files procedure in Spark SQL.

See RewriteDatafiles JavaDoc to see all the supported options.

Now let’s discuss what the strategy option means because it is important to understand to get more out of the rewrite data files procedure. There are three strategy options available. They are Bin Pack, Sort, and Z Order. Note that when using the Spark procedure the Z Order strategy is invoked by simply setting the sort_order to “zorder(columns…).”

Strategy option Bin Pack + It is the cheapest and fastest. + It combines files that are too small and combines them using the bin packing approach to reduce the number of output files. + No data ordering is changed. + No data is shuffled. * Sort + Much more expensive than Bin Pack. + Provides total hierarchical ordering. + Read queries only benefit if the columns used in the query are ordered. + Requires data to be shuffled using range partitioning before writing. * Z Order + Most expensive of the three options. + The columns that are being used should have some kind of intrinsic clusterability and still need to have a sufficient amount of data in each partition because it only helps in eliminating files from a read scan, not from eliminating row groups. If they do, then queries can prune a lot of data during read time. + It only makes sense if more than one column is used in the Z order*. If only one column is needed then regular sort is the better option. + See https://blog.cloudera.com/speeding-up-queries-with-z-order/ to learn more about Z ordering.

Commit conflictsIceberg uses optimistic concurrency control when committing new snapshots. So, when we use rewrite data files to update our data a new snapshot is created. But before that snapshot is committed, a check is done to see if there are any conflicts. If a conflict occurs all the work done could potentially be discarded. It is important to plan maintenance operations to minimize potential conflicts. Let us discuss some of the sources of conflicts.

1. If only inserts occurred between the start of rewrite and the commit attempt, then there are no conflicts. This is because inserts result in new data files and the new data files can be added to the snapshot for the rewrite and the commit reattempted.
2. Every delete file is associated with one or more data files. If a new delete file corresponding to a data file that is being rewritten is added in future snapshot (B), then a conflict occurs because the delete file is referencing a data file that is already being rewritten.

Conflict mitigation1. If you can, try pausing jobs that can write to your tables during the maintenance operations. Or at least deletes should not be written to files that are being rewritten. 2. Partition your table in such a way that all new writes and deletes are written to a new partition. For instance, if your incoming data is partitioned by date, all your new data can go into a partition by date. You can run rewrite operations on partitions with older dates. 3. Take advantage of the filter option in the rewrite data files spark action to best select the files to be rewritten based on your use case so that no delete conflicts occur. 4. Enabling partial progress will help save your work by committing groups of files prior to the entire rewrite completing. Even if one of the file groups fails, other file groups could succeed.

ConclusionIceberg provides several features that a modern data lake needs. With a little care, planning and understanding a bit of Iceberg’s architecture one can take maximum advantage of all the awesome features it provides.

To try some of these Iceberg features yourself you can sign up for one of our next live hands-on labs.

You can also watch the webinar to learn more about Apache Iceberg and see the demo to learn the latest capabilities.

The post Optimization Strategies for Iceberg Tables appeared first on Cloudera Blog.

View Details

IntroductionIn the previous blog post we covered the high availability feature of Cloudera Operational Database (COD) in Amazon AWS. Cloudera recently released a new version of COD, which adds HA support to Microsoft Azure-based databases in the Cloud. In this post, we’ll perform a similar test to validate that the feature works as expected in Azure, too. We will not repeat ourselves, so it’s assumed that technologies and concepts like HA, Multi-AZ, and operational databases are already known to the reader through the previous blog post.

Preparation“Availability zones” in Azure are slightly different from AWS. Unlike in AWS, one cannot just utilize the subnets to assign resources to the availability zone. Virtual networks and subnets are zone redundant in Azure so the availability zone needs to be specified for virtual machines and public IPs to distribute the VMs across availability zones. See Azure availability zones. See Azure zone service and regional support to understand the regions and services that support availability zones.

To use the Multi-AZ feature for every component in the platform, the following prerequisites need to be met:

1. Azure PostgreSQL Flexible Server: The Azure region that you select should support Azure PostgreSQL Flexible Server and the instance types to be used. See Flexible Server Azure Regions.
2. Zone-Redundant Storage (ZRS): The ADLS gen two storage account should be created as zone-redundant storage (ZRS). To specify ZRS via Azure CLI during storage account creation, the –sku option should be set to Standard_ZRS. Below is the Azure CLI command:

Cloudera allows FreeIPA servers, enterprise data lake, and data hub to be configured as Multi-AZ deployment. To set up a Multi-AZ deployment, availability zones need to be configured at the environment level. We can optionally specify an explicit list of availability zones as part of CDP environment creation. If not given, all availability zones, i.e. 1, 2, and 3, will be used.

Below is the CDP CLI command for the same:

For existing environments, we can use CLI to configure a list of AZs. Below is the CLI command:

The list of configured availability zones can be verified on the summary page for the environment on CDP UI:

We can also update the list of availability zones via CDP UI. While updating the list of availability zones for an environment, it can only be extended, which means we cannot remove the availability zones.

To configure FreeIPA as Multi-AZ, it needs to be specified as part of environment creation via CLI or GUI. Below is the CLI command:

To configure the data lake as Multi-AZ, it needs to be specified as part of data lake creation via CLI or GUI. Below is the CLI command:

Note: Only enterprise data lake can be configured as Multi-AZ.

For the Multi-AZ data lake, nodes for each instance group will be distributed across configured availability zones. This can be verified by looking at nodes on CDP UI as shown below for the core host group:

Multi-AZ data lake will also use Postgres Flexible Server since it supports HA.

In addition to the Multi-AZ option, we can also specify the list of AZs for specific instance groups if needed. The list of availability zones for specific instances needs to be a subset of AZs configured at the environment level. If not specified, AZs configured for the environment will be used. For the Multi-AZ data hub, nodes for each instance group will be distributed across configured availability zones for the instance group. This can be verified by looking at nodes on CDP UI.

To create a Multi-AZ COD cluster, use the following CLI command:

COD automates the data hub creation completely: assuming we already have the required entitlements in COD, we can just create a new database that will be automatically allocated to all available AZs. Our test cluster has been created with the light duty option, meaning it has nine nodes (two masters, one leader, one gateway, and five workers) accommodated in three AZs. Pop the hood and see what it looks like in Azure portal:

Names of virtual machines are a bit cryptic. The allocation looks like this:

In the simulation we’re going to stop virtual machines in AZ number 2, which will also bring down the HBase active master (master 0), so the backup master (master 1) has to take over the role. The way we do the simulation is different from the AWS test case because we cannot define a similar network rule to block the traffic. Instead, we just gracefully stop and restart the nodes on Azure portal, but it is still suitable to verify HBase failover behavior.

Test clientWe use the same command line to start the standard HBase load test tool as a test client which will send write requests to the cluster while we’re simulating a failure:

hbase ltt -write 10:1024:10 -num_keys 10000000

DemoCOD is showing a green state, so we can start.

First, we stop the virtual machines on the Azure portal screen and see what happens. The client starts to experience the failure at 13:46 with exceptions: timeout, unable to access region, and no route to host errors.

The backup master takes over the master role and finishes the boot process at 13:50. It’s showing we only have three live region servers.

Once RITs (region in transition) processes are finished, the client recovers and starts making progress at 13:52.

The COD console shows we have node failure and the cluster is running on degraded performance.

We restart the nodes now. The client doesn’t experience any change and keeps progressing. Performance is not impacted in this test scenario, because this single client does not put enough load on the five or three workers.

All five region servers have joined the cluster and have started receiving write requests.

The COD console shows that we’re back in business and had a six-minute outage in write requests.

SummaryIn this blog post we simulate an availability zone failure in the Microsoft Azure cloud environment with Cloudera Operation Database service. We’ve confirmed that HBase can detect the failure and recover the service by booting the backup master to take over the master role in a few minutes and transition unavailable regions to live region servers. The client also noticed the failure and experienced a seven to eight minute outage, but after HBase recovered it was able to continue processing without manual intervention.

However, there are a few things to note regarding the test. First, it’s impossible to simulate a real-world AZ outage in any cloud environment. Cloud providers simply don’t support that, unfortunately, so we can only try to approach it as closely as possible. A real-world outage would be different in some regard. For instance, for our simulation we did a graceful stop command on VMs. In a real-world scenario, it could take more time for HBase to detect the failure and recover.

Second, performance is a critical aspect of an operational database and it’s severely impacted by an entire availability zone failure. This must be closely monitored and manually addressed by reducing the load or bringing up new worker nodes in the available regions. COD has the auto-scaling feature that comes to the rescue in a situation like this.

The post High Availability (Multi-AZ) for Cloudera Operational Database appeared first on Cloudera Blog.

View Details

In Cloudera deployments on public cloud, one of the key configuration elements is the DNS. Get it wrong and your deployment may become wholly unusable with users unable to access and use the Cloudera data services. If the DNS is set up less ideal than it could be, connectivity and performance issues may arise. In this blog, we’ll take you through our tried and tested best practices for setting up your DNS for use with Cloudera on Azure.

To get started and give you a feel for the dependencies for the DNS, in an Azure deployment for Cloudera, these are the Azure managed services being used:

• AKS cluster: data warehouse, data engineering, machine learning, and Data flow
• MySQL database: data engineering
• Storage account: all services
• Azure database for PostgreSQL DB: data lake and data hub clusters
• Key vault: all services

Typical customer governance restrictions and the impactMost Azure users use private networks with a firewall as egress control. Most users have restrictions on firewalls for wildcard rules. Cloudera resources are created on the fly, which means wildcard rules may be declined by the security team.

Most Azure users use hub-spoke network topology. DNS servers are usually deployed in the hub virtual network or an on-prem data center instead of in the Cloudera VNET. That means if DNS is not configured correctly, the deployment will fail.

Most Cloudera customers deploying on Azure allow the use of service endpoints; there is a smaller set of organizations that do not allow the use of service endpoints. Service endpoint is a simpler implementation to allow resources on a private network to access managed services on Azure Cloud. If service endpoints are not allowed, firewall and private endpoints will be the other two options. Most cloud users do not like opening firewall rules because that will introduce the risk of exposing private data on the internet. That leaves private endpoints the only option, which will also introduce additional DNS configuration for the private endpoints.

Connectivity from private network to Azure managed servicesFirewall to InternetRoute from firewall to Azure managed service endpoint on the internet directly.

Service endpointAzure provides service endpoints for resources on private networks to access the managed services on the internet without going through the firewall. That can be configured at a subnet level. Since Cloudera resources are deployed in different subnets, this configuration must be enabled on all subnets.

The DNS records of the managed services using service endpoints will be on the internet and managed by Microsoft. The IP address of this service will be a public IP, and routable from the subnet. Please refer to the Microsoft documentation for detail.

Not all managed services support services endpoint. In a Cloudera deployment scenario, only storage accounts, PostgreSQL DB, and Key Vault support service endpoints.

Fortunately, most users allow service endpoints. If a customer doesn’t allow service endpoints, they have to go with a private endpoint, which is similar to what needs to be configured in the following content.

Private EndpointThere is a network interface with a private IP address created with a private endpoint, and there is a private link service associated with a specific network interface, so that other resources in the private network can access this service through the private network IP address.

The key here is for the private resources to find a DNS resolve for that private IP address. There are two options to store the DNS record:

• Azure managed public DNS zones will always be there, but they store different types of IP addresses for the private endpoint. For example:
  • Storage account private endpoint—the public DNS zone stores the public IP address of that service.
  • AKS API server private endpoint—the public DNS zone stores the private IP of that service.
• Azure Private DNS zone: The DNS records will be synchronized to the Azure Default DNS of LINKED VNET.

Private endpoint is eligible to all Azure managed services that are used in Cloudera deployments.

As a consequence, for storage accounts, users either use service endpoints or private endpoints. Because the public DNS zone will always return a public IP, the private DNS zone becomes a mandatory configuration.

For AKS, these two DNS alternatives are both suitable. The challenges of private DNS zones will be discussed next.

Challenges of private DNS zone on Azure private networkImportant AssumptionsAs mentioned above for the typical scenario, most Azure users are using a hub-and-spoke network architecture, and deploy custom private DNS on hub VNET.

The DNS records will be synchronized to Azure default DNS of linked VNET.

Simple Architecture Use CasesOne VNET scenario with private DNS zone:When a private endpoint is created, Cloudera on Azure will register the private endpoint to the private DNS zone. The DNS record will be synchronized to Azure Default DNS of linked VNET.

If users use custom private DNS, they can configure conditional forward to Azure Default DNS for the domain suffix of the FQDN.

Hub-and-spoke VNET with Azure default DNS:With hub-spoke VNET and Azure default DNS, that is still acceptable. The only problem is that the resources on the un-linked VNET will not be able to access the AKS. But since AKS is used by Cloudera, that does not pose any major issues.

The Challenge PartThe most popular network architecture among Azure consumers is hub-spoke network with custom private DNS servers deployed either on hub-VNET or on-premises network.

Since DNS records are not synchronized to the Azure Default DNS of the hub VNET, the custom private DNS server cannot find the DNS record for the private endpoint. And because the Cloudera VNET is using the custom private DNS server on hub VNET, the Cloudera resources on Cloudera VNET will go to a custom private DNS server for DNS resolution of the FQDN of the private endpoint. The provisioning will fail.

With the DNS server deployed in the on-prem network, there isn’t Azure default DNS associated with the on-prem network, so the DNS server couldn’t find the DNS record of the FQDN of the private endpoint.

Configuration best practicesAgainst the background

Option 1: Disable Private DNS ZoneUse Azure managed public DNS zone instead of a private DNS zone.

• For data warehouse: create data warehouses through the Cloudera command line interface with the parameter “privateDNSZoneAKS”: set to”None.”

• For Liftie-based data services: the entitlement “LIFTIE_AKS_DISABLE_PRIVATE_DNS_ZONE” must be set. Customers can request this entitlement to be set either through a JIRA ticket or have their Cloudera solution engineer to make the request on their behalf.

The sole drawback of this option is that it does not apply to data engineering, since that data service will create and use a MySQL private DNS zone on the fly. There is at present no option to disable private DNS zones for data engineering.

Option 2: Pre-create Private DNS ZonesPre-create private DNS zones and link both Cloudera and hub VNETs to them.

The advantage of this approach is that both data warehouse and Liftie-based data services support pre-created private DNS zones. There are however also a few drawbacks:

• For Liftie, the private DNS zone needs to be configured when registering the environment. Once past the environment registration stage, it cannot be configured.
• DE will need a private DNS zone for MySQL and it doesn’t support pre-configured private DNS zones.
• On-premises networks can’t be linked to a private DNS zone. If the DNS server is on an on-prem network, there are no workable solutions.

Option 3: Create DNS Server as a Forwarder.Create a couple of DNS servers (for HA consideration) with load balancer in Cloudera VNET, and configure conditional forward to Azure Default DNS of the Cloudera VNET. Configure conditional forward from the company custom private DNS server to the DNS server in the Cloudera subnet.

The drawback of this option is that additional DNS servers are required, which leads to additional administration overhead for the DNS team.

Option 4: Azure-Managed DNS ResolveCreate a dedicated /28 subnet in Cloudera VNET for Azure private DNS resolver inbound endpoint. Configure conditional forward from custom private DNS to the Azure private DNS resolver inbound endpoint.

SummaryBringing all things together, consider these best practices for setting up your DNS with Cloudera on Azure:

• For the storage account, key vault, postgres DB
  • Use service endpoints as the first choice.
  • If service endpoint is not allowed, pre-create private DNS zones and link to the VNET where the DNS server is deployed. Configure conditional forwards from custom private DNS to Azure default DNS.
  • If the custom private DNS is deployed in the on-premises network, use Azure DNS resolver or another DNS server as DNS forwarder on the Cloudera VNET. Conditional forward the DNS lookup from the private DNS to the resolver endpoint.
• For the data warehouse, DataFlow, or machine learning data services
  • Disable the private DNS zone and use the public DNS zone instead.
• For the data engineering data service
  • Configure the Azure DNS resolver or another DNS server as a DNS forwarder on the Cloudera VNET. Conditional forward the DNS lookup from the private DNS to the resolver endpoint. Please refer to Microsoft documentation for the details of setting up an Azure DNS Private Resolver.

For more background reading on network and DNS specifics for Azure, have a look at our documentation for the various data services: DataFlow, Data Engineering, Data Warehouse, and Machine Learning. We’re also happy to discuss your specific needs; in that case please reach out to your Cloudera account manager or get in touch.

The post DNS Zone Setup Best Practices on Azure appeared first on Cloudera Blog.

View Details

OverviewThis blog post describes support for materialized views for the Iceberg table format in Cloudera Data Warehouse.

Apache Iceberg is a high-performance open table format for petabyte-scale analytic datasets. It has been designed and developed as an open community standard to ensure compatibility across languages and implementations. It brings the reliability and simplicity of SQL tables to big data while enabling engines like Hive, Impala, Spark, Trino, Flink, and Presto to work with the same tables at the same time. Apache Iceberg forms the core foundation for Cloudera’s Open Data Lakehouse with the Cloudera Data Platform (CDP).

Materialized views are valuable for accelerating common classes of business intelligence (BI) queries that consist of joins, group-bys and aggregate functions. Cloudera Data Warehouse (CDW) running Hive has previously supported creating materialized views against Hive ACID source tables. Starting from the CDW Public Cloud DWX-1.6.1 release and the matching CDW Private Cloud Data Services release, Hive also supports creating, using, and rebuilding materialized views for Iceberg table format.

The key characteristics of this functionality are:

• Source tables of the materialized view are Iceberg tables (the underlying file format could be Parquet, ORC).
• The materialized view itself is an Iceberg table.
• Materialized views can be partitioned on one or more columns.
• Queries containing joins, filters, projections, group-by, or aggregations without group-by can be transparently rewritten by the Hive optimizer to use one or more eligible materialized views. This can potentially lead to orders of magnitude improvement in performance.
• Both full and incremental rebuild of the materialized view are supported. Incremental rebuild can be done under qualifying conditions.

Create Iceberg materialized viewFor the examples in this blog, we will use three tables from the TPC-DS dataset as our base tables: store_sales, customer and date_dim.

These tables are created as Iceberg tables. For instance:

create table store\_sales ( `ss\_sold\_time\_sk` int, … … `ss\_net\_profit` decimal(7,2)) PARTITIONED BY ( `ss\_sold\_date\_sk` int) stored by iceberg stored as orc; It is the same for the other two tables. We populated the tables using INSERT-SELECT statements by reading from text format source tables but they can be populated through any ETL process.

Let’s create a materialized view that joins the three tables, has filter conditions, and does grouped aggregation. Such a query pattern is quite common in BI queries. Note that the materialized view definition contains the ‘stored by iceberg’ clause. Furthermore, it is partitioned on the d_year column.

drop materialized view year\_total\_mv1;create materialized view year\_total\_mv1 PARTITIONED ON (dyear) stored by iceberg stored as orc tblproperties ('format-version'='2')ASselect c\_birth\_country customer\_birth\_country ,d\_year dyear ,sum(ss\_ext\_sales\_price) year\_total\_sales ,count(ss\_ext\_sales\_price) total\_count from customer ,store\_sales ,date\_dim where c\_customer\_sk = ss\_customer\_sk and ss\_sold\_date\_sk = d\_date\_sk and d\_year between 1999 and 2023 group by c\_birth\_country ,d\_year; Show materialized view metadataSimilar to a regular table, you can describe the materialized view to show metadata.

DESCRIBE FORMATTED year\_total\_mv1; A few key characteristics are listed below (extracted from the DESCRIBE output):

As shown above, this materialized view is enabled for rewrites and is not outdated. The snapshotId of the source tables involved in the materialized view are also maintained in the metadata. Subsequently, these snapshot IDs are used to determine the delta changes that should be applied to the materialized view rows.

SHOW MATERIALIZED VIEWS;

The last column indicates that the materialized view can be incrementally maintained in the presence of insert operations only. If the base table data is modified through an UPDATE/DELETE/MERGE operation, then the materialized view must go through a full rebuild. In a future version, we intend to support incremental rebuild for such cases.

A materialized view can also be explicitly disabled for rewrites. This is similar to disabling indexes in databases for certain reasons.

ALTER MATERIALIZED VIEW year_total_mv1 DISABLE REWRITE;

Conversely, it can be enabled as follows:

ALTER MATERIALIZED VIEW year_total_mv1 ENABLE REWRITE;

Query planning using materialized view Let’s first consider a simple case where the grouping columns and aggregate expression exactly match one of the materialized views.

explain cboselect c\_birth\_country customer\_birth\_country ,d\_year dyear ,sum(ss\_ext\_sales\_price) year\_total\_sales from customer ,store\_sales ,date\_dim where c\_customer\_sk = ss\_customer\_sk and ss\_sold\_date\_sk = d\_date\_sk and d\_year between 2000 and 2003 group by c\_birth\_country ,d\_year; CBO PLAN:

HiveProject(customer\_birth\_country=[$0], dyear=[$3], year\_total\_sales=[$1]) HiveFilter(condition=[BETWEEN(false, $3, 2000, 2003)]) HiveTableScan(table=[[tpcds\_iceberg, **year\_total\_mv1**]], table:alias=[tpcds\_iceberg.**year\_total\_mv1**]) The above CBO (cost based optimizer) plan shows that only the year_total_mv1 materialized view is scanned and a filter condition applied since the range filter in the query is a subset of the range in the materialized view. Thus, the scans and joins of the three tables in the original query are not needed and this can improve performance significantly due to both I/O cost saving and the CPU cost saving of computing the joins and aggregations.

Now consider a more advanced usage where the group-by and aggregate expressions in the query don’t exactly match the materialized view but can potentially be derived.

explain cboselect c\_birth\_country customer\_birth\_country ,avg(ss\_ext\_sales\_price) year\_average\_sales from customer ,store\_sales ,date\_dim where c\_customer\_sk = ss\_customer\_sk and ss\_sold\_date\_sk = d\_date\_sk and d\_year between 2000 and 2003 group by c\_birth\_country; CBO PLAN:

HiveProject(customer\_birth\_country=[$0], year\_average\_sales=[CAST(/($1, COALESCE($2, 0:BIGINT))):DECIMAL(11, 6)]) HiveAggregate(group=[{0}], agg#0=[sum($1)], agg#1=[sum($2)]) HiveFilter(condition=[BETWEEN(false, $3, 2000, 2003)]) HiveTableScan(table=[[tpcds\_iceberg, year\_total\_mv1]], table:alias=[tpcds\_iceberg.**year\_total\_mv1**]) Here, the materialized view year_total_mv1 contains the SUM and COUNT aggregate expressions which are used to derive the AVG(ss_ext_sales_price) expression for the query. Further, since the query contains GROUP BY c_birth_country only, a second-level grouping is done on c_birth_country to produce the final output.

Incremental and full rebuild of materialized viewWe will insert rows into the base table and examine how the materialized view can be updated to reflect the new data.

Due to the table modification, Iceberg creates new snapshots and the metadata table “snapshots” can be examined to view the new snapshot version:

SELECT * FROM tpcds\_iceberg.store\_sales.snapshots; Note that the materialized view is now marked outdated for rewriting because their contents are now stale:

DESCRIBE FORMATTED year\_total\_mv1; Outdated for Rewriting: Yes

Running the original query now will not leverage the materialized view and instead do the full scan of the source tables followed by the joins and group-by.

Let us now rebuild the materialized view:

ALTER MATERIALIZED VIEW year\_total\_mv1 REBUILD; This does an incremental rebuild of the materialized view by reading only the delta changes from the store_sales table. Hive does this by asking the Iceberg library to return only the rows inserted since that table’s last snapshot when the materialized view was last rebuilt/created. It then computes the aggregate values for these delta rows after joining them with the other tables. Finally, this set of rows is outer joined with the materialized view using the grouping columns as the join key and the appropriate aggregate values are consolidated—for example, the old sum and the new sum are added together and the old min/max aggregate values may be replaced with the new one depending on whether the new value is lower/higher than the old one.

The rebuild of the materialized view is triggered manually here but it can also be done on a periodic interval using the scheduled query approach.

At this point, the materialized view should be available for query rewrites:

DESCRIBE FORMATTED year\_total\_mv1;Outdated for Rewriting: **No** Re-running the original query will again use the materialized view.

Qualifying conditions for incremental rebuildAn incremental rebuild is not possible under the following situations:

• If the base table was modified through a DELETE/MERGE/UPDATE operation.
• If the aggregate function is anything other than SUM, MIN, MAX, COUNT, AVG. Other aggregates such as STDDEV, VARIANCE, and similar require a full scan of the base data.
• If any of the source tables were compacted since the last rebuild. Compaction creates a new snapshot consisting of merged files and it is not possible to determine the delta changes since the last rebuild operation.

In such situations, Hive falls back to the full rebuild. This fall-back is done transparently as part of the same REBUILD command.

A Note on Iceberg materialized view specificationCurrently, the metadata needed for materialized views is maintained in Hive Metastore and it builds upon the materialized views metadata previously supported for Hive ACID tables. Over the past year, the Iceberg community has proposed a materialized view specification. We intend to adopt this specification in the future for Hive Iceberg materialized view support.

Performance with materialized viewsIn order to evaluate the performance of queries in the presence of materialized views in Iceberg table format, we used a TPC-DS data set at 1 TB scale factor. The table format was Iceberg and the underlying file format was ORC (similar tests can be performed with Parquet but we chose ORC as most Hive customers use ORC). We ran the ANALYZE command to gather both table and column statistics on all the base tables.

We started with twenty three TPC-DS queries and created variants of them such that we had a total of fifty queries in the workload. Each query had between one to three variants. A variant was created by one of the following modifications: (a) adding extra columns in the GROUP-BY clause (b) adding extra aggregation function in the SELECT list, and (c) adding or modifying single table WHERE predicates. We obtained the EXPLAIN CBO (cost based optimization) plan in JSON format for all the fifty queries and supplied the plans to a materialized view recommender that is supported by Cloudera Data Warehouse. Based on the ranked recommendations, we picked the top seven materialized views and created them in the Iceberg table format. We ran the fifty query workload on a CDW Hive virtual warehouse on AWS using a large t-shirt size (see Virtual Warehouse sizes) . Each query was run three times and the minimum total execution time was captured. The query performance results are shown below with and without the materialized view rewrite enabled. The following configuration option is toggled for this:

SET hive.materializedview.rewriting = false;

Out of the fifty queries, there are sixteen queries which the optimizer planned using materialized views. A few of the longer running queries benefited the most by the materialized views – for example the query65 a, b, c variants showed a reduction of nearly 85% in the elapsed time. Overall, across all queries, the average reduction in total elapsed time was 40%. We also looked at only the query compilation time overhead for queries that did not hit the materialized views. A slight increase of 4% in the average query compilation time, approximately 60 milliseconds, was observed due to the optimizer attempting to evaluate the feasibility of using materialized views.

This performance evaluation focused on the query rewrite performance using materialized views. In a future blog, we will evaluate the incremental versus full rebuild performance.

ConclusionThis blog post describes the materialized view support in Hive for the Iceberg table format. This functionality is available in Cloudera Data Warehouse (CDW) Public Cloud deployments on AWS and Azure as well as in CDW Private Cloud Data Services deployments. Users can create materialized views on Iceberg source tables, and Hive will leverage these to accelerate query performance. When the source table data is modified, incremental rebuild of the materialized view is supported under qualifying conditions (stated above); otherwise, a full rebuild is done.

The support for Apache Iceberg as the table format in Cloudera Data Platform and the ability to create and use materialized views on top of such tables provides a powerful combination to build fast analytic applications on open data lake architectures. Sign up for one of our next hands-on labs to try Apache Iceberg on Cloudera’s lakehouse and see the benefits and ease of using materialized views. You can also sign up for the webinar to learn more about the benefits of Apache Iceberg and watch the demo to see the latest capabilities.

AcknowledgementThe authors would like to acknowledge the assistance of Soumyakanti Das in gathering the performance results.

The post Accelerating Queries on Iceberg Tables with Materialized Views appeared first on Cloudera Blog.

View Details

How enterprise-grade data management creates better and more efficient care.

In the last few years, the acceptance of telehealth has become more widespread as patients and providers found they could maintain continuity through phone and video collaboration, instead of in-person visits. In many cases, a level of care that once required a drive to the clinic or hospital could be delivered over a mobile phone or laptop, with no travel and no waiting room. This evolution of healthcare has set the stage for a new social acceptance of technology as a medium for healthcare delivery and remotely collected data as a key component to diagnosis. I think this is just the tip of the iceberg for the future of data-driven healthcare.

Thinking outside of the boxRecently I read an excellent article in de Volkskrant about a gentleman in the Netherlands whose aging father suffers from mild dementia in London. Relocation wasn’t possible, so he installed cameras, shared access computer capabilities, phone screening services, and microphones in his father’s home in London. All of this with the purpose of monitoring, supporting, and communicating with his father remotely, over 500 kilometers away.

Besides monitoring for health and behavioral risks, he was able to monitor his father’s financial risk as seniors tend to be a regular target for financial fraud through phishing and other tactics.

Although his father had to get used to the resulting loss of privacy, he accepted the oversight since it mitigated some of his son’s worries. This has allowed them to stay in near constant contact and communication, and the son has access to critical data, even though they live in different countries.

Staying home longerAlthough not always the perfect solution, technology is enabling seniors to live in their own homes longer than before. Like in the story above, it can help loved ones keep track of at-risk family members, and it reduces some of the increasing demands put on professional caregivers.

The healthcare tech industry is taking strides to further enable seniors and people with health issues to maintain independent lives where they want, for longer, through connected devices that provide real-time or near real-time data for alerts or diagnosis.

Some of the emerging technologies include:

– Medical devices for remote monitoring of biological conditions

– Slip and fall/location and movement sensors

– Biomarker detection sensors for cancer

– Smart devices such as plugs, lights, locks, stoves, and refrigerators

With the help of these and other emerging technologies, people can stay in their homes longer and be monitored to get the right care when they need it.

But what about privacy?All of this technology does require some serious thinking and discussion about privacy; the peace of mind of constant monitoring comes at a price. If possible, this should be discussed openly and honestly with family members, caregivers, and care receivers. A simple Google search will show that there are many suggestions from elderly homes, care centers, medical practices, law firms, and associations for seniors that address how to have these conversations.

Ideally, all the data gathered from these devices is not only received by a primary caregiver but also by relevant medical and other urgent care providers (alarm services, etc.). The patient is constantly being assessed for health issues, and immediate action can be taken when needed.

This information benefits the whole health ecosystem: care receivers get personalized help, caregivers are more informed and involved, and family members worry less. Medical and care professionals get faster and more complete information on the patient’s needs while medical systems and health insurers can analyze summarized data to predict care needs and prioritize services and spending around prevention and care.

Enterprise data management for enhanced careTo accomplish this will require enterprise-grade data management with the ability to take in different types of data from different sources, combine these in a coherent picture of the “patient,” analyze them in real time for urgent situations, and send the outcomes to a workflow so action gets taken. All while guaranteeing privacy and compliance with the many laws and regulations covering these types of data.

We at Cloudera offer these capabilities in our CDP data platform. It is gratifying to see Cloudera customers embark on this journey. Data truly is the lifeline of society. Let’s nurture it. We would love to discuss with you how we can support you in your data driven healthcare initiatives. Please reach out to me at mhesseling@cloudera.com.

The post Health Care Outside of the Box appeared first on Cloudera Blog.

View Details

Forrester Research recently released The Forrester Wave: Streaming Platforms, Q4 2023. We are happy to share that Cloudera ranked as a strong performer, with a top 3 score in the current offering category.

We believe the report represents a major shift in the market. Traditionally, Forrester and other analysts have looked at data movement, streaming and analytics tools separately. This year they broadened their criteria to streaming platforms in recognition of market needs for complete capabilities, development tools and scalability — all capabilities Cloudera has invested in over several years. . The recommendations for organizations evaluating streaming platforms are well informed- these are necessary to scaling streaming to unlock real-time across the enterprise.

Cloudera offers the most complete set of end-to-end capabilities to capture, process and distribute any data anywhere across environments. We do this by integrating the data capture, movement, message brokering, stream processing and analytics capabilities of Nifi, Kafka and Flink into a single platform with developer tooling to maintain development velocity as organizations scale, take on new data sources and evolve to meet changing business requirements. This completeness of capabilities, democratization of access via tooling and integration of the full lifecycle of streaming data was scored highly in the wave. In fact, the Forrester report had this to say…

“Cloudera is a good fit for customers who wish to have the breadth and depth of data management services of the big cloud service providers without the infrastructure lock-in”

Cloudera’s vision is to place data in motion at the center of the modern data architecture by enabling the efficient capture, processing and distribution of all enterprise data in real time with low code tools that make it easier to build streaming applications. The holistic approach brings a simplicity to the architecture that helps organizations achieve massive scale. By integrating powerful open source technologies with thriving development communities on behalf of our customers we are giving them the best of both worlds- the continuous innovation of the open source community with the stability and ease of deployment of an enterprise technology partner.
Our near term roadmap is all about further simplifying deployments and improving autoscaling for maximum efficiency.

We believe our hybrid strategy will pay off as well. Most enterprises operate in a hybrid multi-cloud environment where the capture, processing and distribution across environments is essential. Real-time stream processing use cases in particular (powered by Apache Flink) will often require on-prem to process enormous amounts of data fast enough to power next generation automation use cases.

The takeaway is that any organization evaluating streaming technology should be talking to Cloudera if they haven’t already!

Sign up for a trial of DataFlow for data distribution use cases
Download Cloudera Stream Processing Community edition for stream processing use cases.

The post Cloudera Named Strong Performer in New Forrester Wave for Streaming Platforms appeared first on Cloudera Blog.

View Details

Delivering transformational innovation and accurate business decisions requires harnessing the full potential of your organization’s entire data ecosystem. Ultimately, this boils down to how reliable and trustworthy the underlying data that feeds your insights and applications is. This applies to modern generative AI solutions that are particularly reliant on trusted, accurate, and context-specific data.

Implementing the right platform is half the battle won—so congratulations on your choosing Cloudera’s industry-leading hybrid data platform for building your data solutions on a foundation of trusted data. The other half of the equation requires your team’s emphasis to shift to sustained excellence in managing and optimizing your data ecosystem—better known as Day 2 operations. In this blog, we’ll cover the highlights of our recently published Day 2 Operations Guide and why it matters to enterprises.

In the fast-paced world of cloud-native products, mastering Day 2 operations is crucial for sustaining the performance and stability of Kubernetes-based platforms, such as CDP Private Cloud Data Services. Day 2 operations are akin to the housekeeping of a software system—vital for maintaining its health and stability. At Cloudera, our commitment to excellence extends beyond your deployment on Day 0 and Day 1, and into the critical phase of system maintenance and optimization.

Before delving into Day 2 operations for Cloudera on private cloud, let’s quickly demystify the jargon and define what these “days” mean.

• Day 0 —Design and Preparation: Focuses on designing and preparing for your installation, including gathering requirements, planning architecture, allocating resources, setting up network and security, and documentation creation.
• Day 1 —Deployment and Migration: Involves the actual deployment and initial configuration of the platform, including installation, configuration, testing, troubleshooting, and setting up monitoring tools, as well as migrating your data and workloads onto the platform.
• Day 2 —Operations and Optimization: Focuses on ongoing platform operations, including regular maintenance, user support, performance tuning, scaling, security monitoring, and updating documentation.

We’ve included a more detailed example of what Days 0, 1, and 2 involve in the appendix if you’re interested. You were right if you guessed that these key steps won’t necessarily all happen in a day!

To sum up, Day 2 operations involve meticulous attention to regular maintenance, proactive user support, and ongoing performance tuning. This is the stage where scalability becomes a reality, adapting to growing data and user demands while continuously fortifying security measures. Moreover, it is a period of dynamic adaptation, where documentation and operational protocols will adapt as your data and technology landscape change.

How does Cloudera support Day 2 operations?For a cloud-native data platform that supports data warehousing, data engineering, and machine learning workloads launched by potentially thousands of concurrent users, aspects such as upgrades, scaling, troubleshooting, backup/restore, and security are crucial. Cloudera on private cloud is designed to manage these and more automatically. The rest of the blog covers precisely how the platform handles monitoring and troubleshooting of the platform when breakages happen.

Cloudera offers a multi-faceted approach to health checks, monitoring, and troubleshooting, including:

1. Environment health checks, host-level health checks, data backup, and proactive monitoring and alerting. While this blog summarizes our Day 2 operations, we have published a detailed guide to help you through every step of the way here. Cloudera makes running these health and environment checks easy through the control plane UI as an action command.
2. Status indicators at the component level that illustrate the state of the platform: healthy, warning, and critical. The threshold level for these alerts can be configured on the control pane to tailor the warning/critical alerts for specific health checks to a specific customer environment.

Monitoring and alertingProactive monitoring is key to maintaining a healthy and efficient Kubernetes environment. Cloudera’s data services on private cloud allow administrators to define custom alert rules based on PromQL expressions. These rules are designed to automatically trigger alerts when specific events occur, ensuring that any potential issues are promptly identified and addressed. These alerts can be viewed on the management console dashboard, and configured alert receivers can send notifications to specified endpoints, keeping the team informed and responsive.

The demonstration below illustrates configuring a custom alert for a Cloudera Data Services install using PromQL expressions.

Navigate to the management console using the below instructions:

To add a custom alert rule, click “add alert rule” button above and ensure the following fields bolded are populated; the others are optional:

• Name
• Severity
• Enable Alert
• Message
  • Summary
  • For Cause
• Source
  • Workload Type
• PromQL Expression

For backup/restore of the platform data protection, Cloudera offers a data recovery system (DRS) out of the box that enables administrators to facilitate backup and restore of the Kubernetes platform. Cloudera recommends taking backups before any maintenance activity or upgrade to mitigate risks and restore the environment as needed. Additionally, these backup operations can be run while the cluster is up without impacting the running workloads. This functionality allows our customers to run periodic backups or as needed during business hours and maintenance windows.

ConclusionCloudera ensures that our customers are supported throughout their operational life cycle by focusing on continuous improvement, optimization, and adaptation. This ongoing support is crucial in a landscape where data requirements and interactions are constantly growing and evolving. Day 2 operations are pivotal in maintaining the platform’s stability and elevating the customer experience for users within the cluster. These operations ensure a seamless, efficient, and reliable service, impacting tenant satisfaction and trust in the platform.

Check out the Day 2 Operations Guide as you plan your upgrades to Cloudera’s Data Services on private cloud and bookmark it for future reference as you operate your state-of-the-art data platform. Stay tuned for upcoming blogs on managing Day 0 and Day 1 operations to optimize your upgrade.

Appendix

| Day 0 (Design & Preparation) | Day 1 (Deployment) | Day 2 (Operations & Optimization) | | * On “Day 0,” an administrator would focus on designing and preparing to install Cloudera CDP Private Cloud Data Services on ECS. * Tasks might include: + Gathering requirements: understand the specific needs of your organization and the goals of deploying Cloudera CDP Private Cloud. + Planning the architecture: design the system architecture, considering factors like scalability, security, and performance. + Resource allocation: determine the hardware and cloud resources required for the installation. + Network setup: configure the network infrastructure to ensure connectivity and data flow. + Security considerations: define security policies and implement necessary measures to protect data and resources. + Documentation: create documentation detailing the installation process and system configurations. | * “Day 1” involves the actual deployment and initial configuration of Cloudera CDP Private Cloud Data Services on ECS. * Key activities for this day include: + Installation: deploy Cloudera CDP components and services on the ECS infrastructure following Cloudera’s public documentation. + Configuration: set up initial configurations, including cluster settings, user access, and data storage configurations. + Testing: validate the installation and ensure that the services are operational. + Troubleshooting: address any issues or errors encountered during deployment. + Monitoring: set up monitoring tools to monitor system performance and resource utilization. | * “Day 2” focuses on ongoing operations, maintenance, and optimization. * Tasks might include: + Regular maintenance: perform routine tasks such as backups, software updates, and security patches. + User support: assist users with any issues or questions about Cloudera CDP services. + Performance tuning: continuously optimize the system for better performance and resource utilization. + Scaling: if needed, scale your infrastructure to accommodate growing data and user demands. + Security monitoring: continually monitor and enhance security measures to protect your data. + Documentation updates: keep documentation up-to-date with any changes or improvements made to the system. |

The post Mastering Day 2 Operations with Cloudera appeared first on Cloudera Blog.

View Details

Replication is a crucial capability in distributed systems to address challenges related to fault tolerance, high availability, load balancing, scalability, data locality, network efficiency, and data durability. It forms a foundational element for building robust and reliable distributed architectures. It is also important to have multiple options (like normal and prefixless replication) to do the replication process, since every solution has its own advantages.

Streams Replication Manager (SRM) is an enterprise-grade replication solution that enables fault tolerant, scalable, and robust cross-cluster Kafka topic replication. SRM replicates data at high performance and keeps topic properties in sync across clusters. Replication can be dynamically enabled for topics and consumer groups. SRM also delivers custom extensions that facilitate installation, management, and monitoring, making SRM a complete replication solution that is built for mission-critical workloads.

IntroductionKafka as an event streaming component can be applied to a wide variety of use cases. SRM provides cross-cluster Kafka topic replication to make it more fault tolerant and robust. SRM is based on the Mirror Maker 2 (MM2) component of Kafka, which is the improved version of Mirror Maker (MM1). MM1 has been used for years in large-scale production environments, but not without several limitations—that is why MM2 was introduced.

These are some of the MM1 limitations that MM2 addresses:

• Topics are created with default configuration, often needed to be repartitioned manually.
• ACL and configuration changes are not synced across mirrored clusters. This makes it difficult to manage multiple clusters.
• Records are repartitioned with DefaultPartitioner. Semantic partitioning may be lost.
• Any configuration change means the cluster must be bounced. This includes adding new topics to the whitelist, which may be a frequent operation.
• No mechanism to migrate producers or consumers between mirrored clusters.
• No support for exactly once delivery. Records may be duplicated during replication.
• Rebalancing causes latency spikes, which may trigger further rebalances.

When SRM replicates a topic, it renames the topic in the target cluster by prefixing the name of the topic with the alias (name) of the source cluster. This differs from the way replication worked in MM1, where the target topics had the same name as the source (thus “prefixless”). The MM1 behavior is crucial for some use-cases. For example, cluster migration scenarios cannot be correctly carried out with the default replication behavior of SRM, the MM1 behavior is a must. Up until now, this type of replication was not available or fully supported. Moreover, MM1 was deprecated in one of the more recent releases of Kafka (Kafka 3.0.0) and its use is no longer recommended.

To address this, Cloudera introduced a new MM1-compatible mode in SRM. Starting with Cloudera Data Platform (CDP) Private Cloud Base 7.1.9, prefixless replication is generally available with replication monitoring support in SRM. This makes it possible to migrate cluster replication workloads from the deprecated MM1 to SRM without change in the replicated topic names.

Replicated topic namesThe naming of the replicated topics is defined by the replication policy that SRM is configured to use. By default, SRM uses the DefaultReplicationPolicy, which adds the source cluster alias as a prefix to the name of replicated topics. In the past, this was the only policy available natively in SRM and the design of the replication monitoring features in the service was based on the assumption that every replicated topic would always have a prefix. Therefore, SRM service role instances were only able to monitor replication flows that used a replication policy that uses prefixes, such as the DefaultReplicationPolicy.

Once the IdentityReplicationPolicy was introduced, users were able to replicate topics without having prefixes added to the replicated topic names. Due to the design of the SRM service though, these replications could not be monitored until the release of CDP Private Cloud Base 7.1.9.

Note: SRM supports custom topic naming policies through a plugin called replication policy. There are two different Replication policy types shipped with SRM by default:

• DefaultReplicationPolicy – default policy. Prefixes topic names with “.”
• IdentityReplicationPolicy – policy which does not change topic names during replication. (with this policy, replication monitoring does not work until CDP 7.1.9 release)

Remote topic discoverySRM needs to be able to know which topics are replicas and what are their respective source topics. It relies on the replication policy and the topic naming conventions to discover replica topics by default. The process lists all of the topic names of a cluster, then detects the source cluster name. When using the DefaultReplicationPolicy, SRM knows that a topic is a replica when it has a prefix that is a valid cluster alias (.). The replica topic name contains the alias of the source cluster and name of the source topic. For instance, the topic name can be source-cluster.topic-name. In this case source-cluster will be the alias of the source cluster, while topic-name will be the name of the topic in the source cluster.

This discovery procedure has some limitations, since it relies on topic naming conventions to provide source cluster information. When the IdentityReplicationPolicy is used, the source cluster cannot be identified by this method. Furthermore, the current state of the replication (stopped, active, etc.) has no connection with the replica topic detection—if a topic has been removed from the SRM replication configuration, the logic will still detect the prefixed topic as a replica topic.

The above shortcomings were addressed in the CDP Private Cloud Base 7.1.9. In this release, SRM is shipped with a new property Use Internal Topic For Remote Topics Discovery, which is enabled for new installations. For upgraded clusters, this feature will be disabled by default to ensure that existing SRM deployments will continue to work without changes in behavior.

When Use Internal Topic For Remote Topics Discovery is enabled, SRM drivers will write the list of source topic—target topic pairs that have to be replicated to an internal, compacted topic (srm-meta.internal), stored on the target cluster. SRM drivers will periodically check which topics need to be replicated and will write updates to the internal topic as needed.

Clients trying to discover replica topics are able to scan the “srm-meta.internal” topic, and consume the latest message—which lists the currently replicated topics. This data also contains the source-target topic name mappings. It makes the feature independent of the ReplicationPolicy that is in use.

Prefixless replicationFrom CDP 7.1.9, SRM supports data replication, checkpointing, and monitoring with the IdentityReplicationPolicy. Identity replication, or prefixless replication, means that replica topics’ names will be the same as on the source cluster (MM1-compatible mode, but with the advantages of MM2). The IdentityReplicationPolicy can also be used for topic aggregation use cases, where the same topic on multiple clusters are replicated to the same identically-named “aggregated topic” on a different cluster. Of course, aggregation can be avoided if DefaultReplicationPolicy is in use or if the separate source clusters have different topic names.

To enable prefixless replication for SRM, you only need to select the “Enable Prefixless Replication” property in the SRM service configuration.

When “Enable Prefixless Replication” is selected, SRM must also enable the “Use Internal Topic For Remote Topics Discovery” feature due to the limitations of replica discovery mentioned previously in this blog. Fortunately, Cloudera Manager handles this automatically, so if a user enables the “Enable Prefixless Replication” option, Cloudera Manager will override the configuration of “Use Internal Topic For Remote Topics Discovery” to enable it.

Prefixless replication is not free of limitations or caveats. Be aware of the following:

• Replication loop detection is not supported

As a result, you must ensure that topics are not replicated in a loop between your source and target clusters. You can ensure this by setting up your topic allow and deny lists (also known as topic filters) in a way that’s appropriate for your use case.

For example, assume you have two replications that replicate topics between two clusters, but in different directions. If both replications include topic_1, they must never be enabled at the same time.

• All SRM services must use the same replication policy

For example, if you want to use prefixless replication then all of the SRM services should use IdentityReplicationPolicy. In case of prefixed replication DefaultReplicationPolicy should be used everywhere. Clusters connected by replication flows, regardless of the number of SRM services, should only use one ReplicationPolicy. Otherwise, replications will be mixed up and unwanted side effects can happen.

• Group offset sync should be disabled

SRM makes a mapping about Kafka message offsets of the source and target clusters. Offset checkpoints are stored in the source clusters and they will be interpreted only if the message is coming from the current source cluster. If more source clusters have the same group offsets, then they can interfere with each other, so group offset sync should be disabled.

• Not all REST API endpoints and SMM UI features are supported
  • The /v2/topic-metrics/{target}/{downstreamTopic}/{metric} endpoint of the SRM Service v2 API does not work properly with prefixless replication. Use the /v2/topic-metrics/{source}/{target}/{upstreamTopic}/{metric} endpoint instead.
  • The replication metric graphs shown on the Topic Details page of the SMM UI do not work with prefixless replication. The graph is not displayed.

SummaryPrefixless replication enables you to use MM1-like replication behavior in CDP while having access to the many enterprise ready features that SRM provides. While aggregation is the main use case for prefixless replication, it can also be used to build traditional replication pipelines that provide a safety net for your Kafka data if things go amiss. Better yet, prefixless replication is also a perfect tool to migrate that old Kafka deployment running on CDH, HDP, or HDF to CDP.

In addition, the changes and improvements to remote topic discovery that were introduced alongside prefixless replication make SRM more robust than ever as some core features within SRM, like replication monitoring, no longer need to rely on topic prefixes to function.

If you want to learn more about SRM and Kafka in CDP Private Cloud Base, hop over to Cloudera’s doc portal and see Streams Messaging Concepts, Streams Messaging How Tos, and/or the Streams Messaging Migration Guide. This is the first of a two-blog series, to continue your journey on Streams Replication, click here.

To get hands on with SRM, download Cloudera Stream Processing Community edition here.

Interested in joining Cloudera?At Cloudera, we are working on fine-tuning big data related software bundles (based on Apache open-source projects) to provide our customers a seamless experience while they are running their analytics or machine learning projects on petabyte-scale datasets. Check our website for a test drive!

If you are interested in big data, would like to know more about Cloudera, or are just open to a discussion with techies, visit our fancy Budapest office at our upcoming meetups. Or, just visit our careers page, and become a Clouderan!

The post Streams Replication Manager Prefixless Replication appeared first on Cloudera Blog.

View Details

In the dynamic landscape of modern manufacturing, AI has emerged as a transformative differentiator, reshaping the industry for those seeking the competitive advantages of gained efficiency and innovation. As we navigate the fourth and fifth industrial revolution, AI technologies are catalyzing a paradigm shift in how products are designed, produced, and optimized.

With the ability of manufacturers to store a huge volume of historical data, AI can be applied in general business areas of any industry, like developing recommendations for marketing, supply chain optimization, and new product development. But with this data—along with some context about the business and process—manufacturers can leverage AI as a key building block to develop and enhance operations.

There are many functional areas within manufacturing where manufacturers will see AI’s massive benefits. Here are some of the key use cases:

1. Predictive maintenance: With time series data (sensor data) coming from the equipment, historical maintenance logs, and other contextual data, you can predict how the equipment will behave and when the equipment or a component will fail. With AI, it can even prescribe the appropriate action that needs to be taken and when.
2. Quality: Use cases like visual inspection, yield optimization, fault detection, and classification are enhanced with AI technologies. While outcomes within industry segments will vary, the potential is huge. For example, improving yield in the semiconductor industry even by a small fraction of a percentage point could save millions of dollars.
3. Demand forecasting: AI can be used to forecast demand for products based on historical data, trends, and external factors such as weather, holidays, seasonality, and market conditions.

While AI stands to drive smart intelligent factories, optimize production processes, enable predictive maintenance and pattern analysis, personalization, sentiment analysis, knowledge management, as well as detect abnormalities, and many other use cases, without a robust data management strategy, the road to effective AI is an uphill battle.

The universal industrial data challengeData—as the foundation of trusted AI—can lead the way to transform business processes and help manufacturers innovate, define new business models, and establish new revenue streams. Yet many manufacturing executives say they are challenged in adopting new technologies, including AI for new use cases. According to Gartner, 80 percent of manufacturing CEOs are increasing investments in digital technologies—led by artificial intelligence (AI), Internet of Things (IoT), data, and analytics. Yet Gartner reports that only eight percent of industrial organizations say their digital transformation initiatives are successful. That is a very low number.

The lack of universal industrial data has been one of the major obstacles slowing the adoption of AI among mainstream manufacturers. Advanced technologies are only part of the digital transformation story. Manufacturers who want to get ahead must understand data’s role and value. With the very low cost of sensors: new equipment is being standardized with sensors and old manufacturing equipment is being retrofitted with sensors. Manufacturers now have unprecedented capacity to collect, utilize, and manage massive amounts of data.

In this age of industrial IoT, it’s possible to rapidly introduce tools to produce actionable results with huge data sets. But without the highest level of trust in these data, AI/ML solutions render questionable analysis and below-optimal results. It is not uncommon for organizations to construct solutions with faulty assumptions about data—the data contains every scenario of interest and the algorithm will figure it out. Without a thorough grounding with trusted data and a robust data platform, AI/ML approaches will be biased and untrusted, and more likely to fail. Simply put, many organizations fail to realize the value of AI because they rely on AI tools and data science that is being applied to data which is faulty to begin with.

Trusted AI begins with trusted dataWhat resolves the data challenge and fuels data-driven AI in manufacturing? Develop a data strategy built on a robust data platform.

Manufacturing operations and IT have to work hand-in-hand to develop a data-centric culture, with IT responsible for end-to-end data life cycle management focused on reliability and security.

There are several best practices specifically when it comes to the data:

• You don’t need to boil the ocean. Start with a pilot problem on the manufacturing floor that needs to be solved.
• Identify the use cases that help manufacturing operations add value. Let that dictate the data you want to collect.
• Build out capabilities to collect and ingest data with IT/OT convergence, and collect and ingest the shop floor and equipment data onto a centralized platform on the cloud.
• Add appropriate contextual data (IT/business data), which is critical in AI analysis of manufacturing data.
• Eliminate data silos. Data from multiple sources must be centralized and stored on a common data lake so that you will have one source of truth across the value chain.
• Apply AI tools and data science to the data that you trust and provide insights to the appropriate people or the system to make the best, most informed decisions.

The value of a hybrid data platformAI can help manufacturers improve operations and achieve the next level of operations excellence. But the key is to focus on data first, not complex AI systems. Manufacturing organizations still use legacy infrastructure and data sources on varied types of platforms (on-prem, provide cloud, public cloud etc.). To resolve these challenges, it’s essential to leverage a hybrid data platform where data can be collected and ingested from any system and in turn delivered to any system or platform.

Cloudera provides end-to-end data life cycle management on a hybrid data platform, which includes all the building blocks needed to build a data strategy for trusted data in manufacturing. The key capabilities include ingesting data, preparing data, storing data, and publishing data, along with common security and governance capabilities across the data life cycle. Cloudera enables data transfer from anywhere to anywhere (private cloud, public cloud, on-prem, and platform agnostic), giving manufacturing the ability to use next-gen AI tools and applications on “trusted” data. Find out more about Cloudera Data Platform (CDP), the only hybrid data platform for modern data architectures supporting AI in manufacturing with data anywhere at Manufacturing at Cloudera.

The post Achieving Trusted AI in Manufacturing appeared first on Cloudera Blog.

View Details

In this article, we will walk you through the process of implementing fine grained access control for the data governance framework within the Cloudera platform. This will allow a data office to implement access policies over metadata management assets like tags or classifications, business glossaries, and data catalog entities, laying the foundation for comprehensive data access control.

In a good data governance strategy, it is important to define roles that allow the business to limit the level of access that users can have to their strategic data assets. Traditionally we see three main roles in a data governance office:

• Data steward: Defines the business rules for data use according to corporate guidance and data governance requirements.
• Data curator: Assigns and enforces data classification according to the rules defined by the data stewards so that data assets are searchable by the data consumer.
• Data consumer: Derives insights and value from data assets and is keen to understand the quality and consistency of tags and terms applied to the data.

Within the Cloudera platform, whether deployed on premises or using any of the leading public cloud providers, the Cloudera Shared Data Experience (SDX) ensures consistency of all things data security and governance. SDX is a fundamental part of any deployment and relies on two key open source projects to provide its data management functionality: Apache Atlas provides a scalable and extensible set of core governance services, while Apache Ranger enables, monitors, and manages comprehensive security for both data and metadata.

In this article we will explain how to implement a fine grained access control strategy using Apache Ranger by creating security policies over the metadata management assets stored in Apache Atlas.

Case IntroductionIn this article we will take the example of a data governance office that wants to control access to metadata objects in the company’s central data repository. This allows the organization to comply with government regulations and internal security policies. For this task, the data governance team started by looking at the finance business unit, defining roles and responsibilities for different types of users in the organization.

In this example, there are three different users that will allow us to show the different levels of permissions that can be assigned to Apache Atlas objects through Apache Ranger policies to implement a data governance strategy with the Cloudera platform:

• admin is our data steward from the data governance office
• etl_user is our data curator from the finance team
• joe_analyst is our data consumer from the finance team

Note that it would be just as easy to create additional roles and levels of access, if required. As you will see as we work through the example, the framework provided by Apache Atlas and Apache Ranger is extremely flexible and customizable.

First, a set of initial metadata objects are created by the data steward. These will allow the finance team to search for relevant assets as part of their day-to-day activities:

• Classifications (or “tags”) like “PII”, “SENSITIVE”, “EXPIRES_ON”, “DATA QUALITY” etc.
• Glossaries and terms created for the three main business units: “Finance,” “Insurance,” and “Automotive.”
• A business metadata collection called “Project.”

NOTE: The creation of the business metadata attributes is not included in the blog but the steps can be followed here.

Then, in order to control the access to the data assets related to the finance business unit, a set of policies need to be implemented with the following conditions:

The finance data curator should only be allowed to:

• Create/read classifications that start with the word “finance.”
• Read/update entities that are classified with any tag that starts with the word “finance,” and also any entities related to the “worldwidebank” project. The user should also be able to add labels and business metadata to those entities.
• Add/update/remove classifications of the entities with the previous specifications.
• Create/read/update the glossaries and glossary terms related to “finance.”

The finance data consumer should only be allowed to:

• View and access cClassifications related to “finance” to search assets.
• View and access entities that are classified with tags related to “finance.”
• View and access the “finance” glossary.

In the following section, the process for implementing these policies will be explained in detail.

Implementation of fine-grained access controls (step by step)In order to meet the business needs outlined above, we will demonstrate how access policies in Apache Ranger can be configured to secure and control metadata assets in Apache Atlas. For this purpose we used a public AMI image to set up a Cloudera Data Platform environment with all SDX components. The process of setting up the environment is explained in this article.

1. Authorization for Classification TypesClassifications are part of the core of Apache Atlas. They are one of the mechanisms provided to help organizations find, organize, and share their understanding of the data assets that drive business processes. Crucially, classifications can “propagate” between entities according to lineage relationships between data assets. See this page for more details on propagation.

1.1 Data Steward – admin userTo control access to classifications, our admin user, in the role of data steward, must perform the following steps:

1. Access the Ranger console.
2. Acces Atlas repository to create and manage policies.
3. Create the appropriate policies for the data curator and the data consumer of the finance business unit.

First, access the Atlas Ranger policies repository from the Ranger admin UIImage 1 – Ranger main page

In the Atlas policy repository:Image 2 – Atlas policies

The first thing you will see are the default Atlas policies (note 1). Apache Ranger allows specification of access policies as both “allow” rules and “deny” rules. However, it is a recommended good practice in all security contexts to apply the “principle of least privilege”: i.e., deny access by default, and only allow access on a selective basis. This is a much more secure approach than allowing access to everyone, and only denying or excluding access selectively. Therefore, as a first step, you should verify that the default policies don’t grant blanket access to the users we are seeking to restrict in this example scenario.Then, you can create the new policies (eg. remove the public access of the default policies by creating a deny policy; note 2) and finally you will see that the newly created policies will appear at the bottom of the section (note 3).

After clicking the “Add New Policy” button: Image 3 – Create policy over finance classification

1. First, define a policy name and, if desired, some policy labels (note 1). These do not have a “functional” effect on the policy, but are an important part of keeping your security policies manageable as your environment grows over time. It is normal to adopt a naming convention for your policies, which may include short-hand descriptions of the user groups and/or assets to which the policy applies, and an indication of its intent. In this case we have chosen the policy name “FINANCE Consumer – Classifications,” and used the labels “Finance.” “Data Governance,” and “Data Curator.”
2. Next, define the type of object on which you want to apply the policy. In this case we will select “type-category” and fill with “Classifications” (note 2).
3. Now, you need to define the criteria used to filter the Apache Atlas objects to be affected by the policy. You can use wildcard notations like “”. To limit the data consumer to only search for classifications starting with the work finance, use FINANCE** (note 3).

Finally, you need to define the permissions that you want to grant on the policy and the groups and users that are going to be controlled by the policy. In this case, apply the Read Type permission to group: finance and user: joe_analyst and Create Type & Read Type permission to user: etl_user. (note 4)

Now, because they have the Create Type permission for classifications matching FINANCE***, the data curator etl_user can create a new classification tag called “FINANCE_WW” and apply this tag to other entities. This would be useful if a tag-based access policy has been defined elsewhere to provide access to certain data assets.

1.2 Data Curator – etl_user userWe can now demonstrate how the classification policy is being enforced over etl_user. This user is only allowed to see classifications that start with the word finance, but he can also create some additional ones for the different teams under that division.

etl_user can create a new classification tag called FINANCE_WW under a parent classification tag FINANCE_BU.

To create a classification in Atlas:Image 4 – Atlas classifications tab

1. First, click on the classification panel button (note 1) to be able to see the existing tags that the user has access to. You will be able to see the assets that are tagged with the selected classification. (note 3)

Then, click on the “+” button to create a new classification. (note 2)

A new window open, requiring various details to create the new classificationImage 5 – Atlas classifications creation tab

1. First, provide the name of the classification, in this case FINANCE_WW, and provide a description, so that colleagues will understand how it should be used..
2. Classifications can have hierarchies and those inherit attributes from the parent classification. To create a hierarchy, type the name of the parent tag, in this case FINANCE_BU.
3. Additional custom attributes can also be added to later be used on attribute-based access control (ABAC) policies. This falls outside of the scope of this blog post but a tutorial on the subject can be found here.

(Optional) For this example, you can create an attribute called “country,” which will simply help to organize assets. For convenience you can make this attribute a “string” (a free text) type, although in a live system you would probably want to define an enumeration so that users’ inputs are restricted to a valid set of values.

After clicking the button “create” the newly created classification is shown in the panel:Image 6 – Atlas classifications tree

Now you can click on the toggle button to see the tags in tree mode and you will be able to see the parent/child relationship between both tags.

Click on the classification to view all its details: parent tags, attributes, and assets currently tagged with the classification.

1.3 Data Consumer – joe_analyst userThe last step on the Classification authorization process is to validate from the data consumer role that the controls are in place and the policies are applied correctly.

After successfully logging in with user joe_analyst:Image 7 – Atlas classifications for finance data consumer

To validate that the policy is applied and that only classifications starting with the word FINANCE can be accessed based on the level of permissions defined in the policy, click on the Classifications tab (note 2) and check the list available. (note 3)

Now, to be able to access the content of the entities (note 4), it is required to give access to the Atlas Entity Type category and to the specific entities with the corresponding level of permissions based on our business requirements. The next section will cover just that.

1. Authorization for Entity Types, Labels and Business Metadata In this section, we will explain how to protect additional types of objects that exist in Atlas, which are important within a data governance strategy; namely, entities, labels, and business metadata.

Entities in Apache Atlas are a specific instance of a “type” of thing: they are the core metadata object that represent data assets in your platform. For example, imagine you have a data table in your lakehouse, stored in the Iceberg table format, called “sales_q3.” This would be reflected in Apache Atlas by an entity type called “ceberg table,” and an entity named “sales_q3,” a particular instance of that entity type. There are many entity types configured by default in the Cloudera platform, and you can define new ones as well. Access to entity types, and specific entities, can be controlled through Ranger policies.

Labels are words or phrases (strings of characters) that you can associate with an entity and reuse for other entities. They are a light-weight way to add information to an entity so you can find it easily and share your knowledge about the entity with others.

Business metadata are sets of related key-value pairs, defined in advance by admin users (for example, data stewards). They are so named because they are often used to capture business details that can help organize, search, and manage metadata entities. For example, a steward from the marketing department can define a set of attributes for a campaign, and add these attributes to relevant metadata objects. In contrast, technical details about data assets are usually captured more directly as attributes on entity instances. These are created and updated by processes that monitor data sets in the data lakehouse or warehouse, and are not typically customized in a given Cloudera environment.

With that context explained, we will move on to setting policies to control who can add, update, or remove various metadata on entities. We can set fine-grained policies separately for both labels and business metadata, as well as classifications. These policies are defined by the data steward, in order to control activities undertaken by data curators and consumers.

2.1 Data Steward – admin userFirst, it’s important to make sure that the users have access to the entity types in the system. This will allow them to filter their search when looking for specific entities.

In order to do so, we need to create a policy: Image 8 – Atlas entity type policies

In the create policy page, define the name and labels as described before. Then, select the type-category “entity”(note 1). Use the wildcard notation (*) (note 2) to denote all entity types, and grant all available permissions to etl_user and joe_analyst.(note 3)

This will enable these users to see all the entity types in the system.

The next step is to allow data consumer joe_analyst to only have read access on the entities that have the finance classification tags. This will limit the objects that he will be able to see on the platform.

To do this, we need to follow the same process to create policies as shown in the previous section, but with some modifications on the policy details:Image 9 – Example Atlas finance entity policies

1. As always, name (and label) the policy to enable easy management later.
2. The first important change is that the policy is applied on an “entity-type” and not in a “type-category.” Select “entity-type” in the drop-down menu (note 2) and type the wildcard to apply it to all the entity types.
3. Some additional fields will appear in the form. In the entity classification field you can specify tags that exist on the entities you want to control. In our case, we want to only allow objects that are tagged with words that start with “finance.” Use the expression FINANCE*. (note 3)
4. Next, filter the entities to be controlled through the entity ID field. In this exercise, we will use the wildcard (*) (note 4) and for the additional fields we will select “none.” This button will update the list of permissions that can be enforced in the conditions panel. (note 4)
5. As a data consumer, we want the joe_analyst user to be able to see the entities. To implement this, select the Read Entity permission. (note 5)
6. Add a new condition for the data curator etl_user but this time include permissions to modify the tags appropriately, by adding the Add Classification, Update Classification & Remove Classification permissions to the specific user.

In this way, access to specific entities can be controlled using additional metadata objects like classification tags. Atlas provides some other metadata objects that can be used not only to enrich the entities registered in the platform, but also to implement a governance strategy over those objects, controlling who can access and modify them. This is the case for the labels and the business metadata.

If you want to enforce some control over who can add or remove labels:Image 10 – Example Atlas finance label policy

1. The only difference between setting a policy for labels versus the previous examples is setting the additional fields filter to “entity-label” as shown in the image and fill with the values of labels that want to be controlled. In this case, we use the wildcard () to enable operations on any label on entities tagged with FINANCE classifications.
2. When the entity-label is selected from the drop-down, the permissions list will be updated. Select Add Label & Remove Label permission to grant the data curator the option to add and remove labels from entities.

The same principle can be applied to control the permissions over business metadata:Image 11 – Example Atlas finance business metadata policy

1. In this case, one must set the additional fields filter to “entity-business-metadata” as shown in the image and fill with the values of business metadata attributes that want to be protected. In this example, we use the wildcard () to enable operations on all business metadata attributes on entities tagged with FINANCE classifications.
2. When you enable the entity-business-metadata drop-down, the permissions list will be updated. Select Update Business Metadata permission to grant the data curator the option to modify the business metadata attributes of financial entities.

As part of the fine grained access control provided by Apache Ranger over Apache Atlas objects, one can create policies that use an entity ID to specify the exact objects to be controlled. In the examples above we have often used the wildcard (*) to refer to “all entities;” below, we will show a more targeted use-case.

In this scenario, we want to create a policy pertaining to data tables which are part of a specific project, named “World Wide Bank.” As a standard, the project owners required that all the tables are stored in a database called “worldwidebank.”

To meet this requirement, we can use one of the entity types pre-configured in Cloudera’s distributions of Apache Atlas, namely “hive_table”. For this entity type, identifiers always begin with the name of the database to which the table belongs. We can leverage that, using Ranger expressions to filter all the entities that belong to the “World Wide Bank” project.

To create a policy to protect the worldwidebank entities:Image 12 – Example Atlas Worldwide Bank entity policy

1. Create a new policy, but this time don’t specify any entity classification, use the wildcard “*” expression.
2. In the entity ID field use the expression: worldwidebank
3. In the Conditions, select the permissions Read Entity, Update Entity, Add Classification, Update Classification & Remove Classification to the data curator etl_user to be able to see the details of these entities and enrich/modify and tag them as needed.

2.2 Data Curator – etl_user userIn order to allow finance data consumer joe_analyst to use and access the worldwidebank project entities, the data curator etl_user must tag the entities with the approved classifications and add the required labels and business metadata attributes.

Login to Atlas and follow the process to tag the appropriate entities:Image 13 – Data curator entity search

1. First, search for the worldwidebank assets using the search bar. You can also use the “search by type” filter on the left panel to limit the search to the “hive_db” entity type.
2. As data curator, you should be able to see the entity and be allowed to access the details of the worldwidebank database entity. It should have a clickable link to the entity object
3. Click on the entity object to see its details.

After clicking the entity name, the entity details page is shown:Image 14 – Worldwide Bank database entity detail

In the top of the screen, you can see the classifications assigned to the entity. In this case there are no tags assigned. We will assign one by clicking on the “+” sign.

In the “Add Classification” screen:Image 15 – Worldwide Bank database tag process

1. Search for the FINANCE_WW tag and select it.
2. Then fill the appropriate attributes if the classification tag has any. (Optional in Image 5, in the 1.2 Data Curator – etl_user user section above.)
3. Click on “add.”

That will tag an entity with the selected classification.

Now, enrich the worldwidebank hive_db entity with a new label and a new business metadata attribute called “Project.” Image 16 – Worldwide Bank database tag process

To add a label, click “Add” on the labels menu.

• Type the label in the space and click “save.”

To add a business metadata attribute, click “Add” on the business metadata menu.

• Click on “Add New Attribute” if it’s not assigned or “edit” if it already exists.
• Select the attribute you want to add and fill the details and hit “save.”

NOTE: The creation of the business metadata attributes is not included in the blog but the steps can be followed here.

With the “worldwidebank” Hive object tagged with the “FINANCE_WW” classification, the data consumer should be able to have access to it and see the details. Also, it is important to validate that the data consumer also has access to all the other entities tagged with any classification that starts with “finance.”

2.3 Data Consumer – joe_analyst userTo validate that the policies are applied correctly, login into Atlas:Image 17 – Finance data assets

Click on the classifications tab and validate:

• The list of tags that are visible based on the policies created in the previous steps. All the policies must start with the word “finance.”

Click on the FINANCE_WW tag and validate the access to the “worldwidebank” hive_db object.

After clicking on the “worldwidebank” object:Image 18 – WorldWideBank database asset details

You can see all the details of the asset that where enriched by the finance data curator in previous steps:

• You should see all the technical properties of the asset.
• You should be able to see the tags applied to the asset
• You should see the labels applied to the asset.

• You should see the business metadata attributes assigned to the asset.

• Authorization for Glossary and Glossary TermsIn this section, we will explain how a data steward can create policies to allow fine-grained access controls over glossaries and glossary terms. This allows data stewards to control who can access, enrich or modify glossary terms to protect the content from unauthorized access or mistakes.

A glossary provides appropriate vocabularies for business users and it allows the terms (words) to be related to each other and categorized so that they can be understood in different contexts. These terms can be then applied to entities like databases, tables, and columns. This helps abstract the technical jargon associated with the repositories and allows the user to discover and work with data in the vocabulary that is more familiar to them.

Glossaries and terms can also be tagged with classifications. The benefit of this is that, when glossary terms are applied to entities, any classifications on the terms are passed on to the entities as well. From a data governance process perspective, this means that business users can enrich entities using their own terminology, as captured in glossary terms, and that can automatically apply classifications as well, which are a more “technical” mechanism, used in defining access controls, as we have seen.

First, we will show how as a data steward you can create a policy that grants read access to glossary objects with specific words in the name and validate that the data consumer is allowed to access the specific content.

3.1 Data Steward – admin userTo create a policy to control access to glossaries and terms, you can:

Image 19 – Glossary control policy

1. Create a new policy, but this time use the “entity-type” AtlasGlossary and AtlasGlossaryTerm. (note 1)
2. In the entity classifications field, use the wildcard expression: *
3. The entity ID is where you can define which glossaries and terms you want to protect. In Atlas, all the terms of a glossary include a reference to it with an “@” at the end of its name (ex. term@glossary). To protect the “Finance” glossary itself, use Finance; and to protect is terms, use @Finance (note 2).
4. In the Conditions, select the permissions Read Entity to the data consumer joe_analyst to be able to see the glossary and its terms. (note 3)

3.2 Data Consumer – joe_analyst userTo validate that only “Finance” glossary objects can be accessed:Image 20 – Finance Atlas glossary

1. Click on the glossary tab in the Atlas panel.
2. Check the glossaries available in the Atlas UI and the access to the details of the terms of the glossary.

ConclusionThis article has shown how an organization can implement a fine grained access control strategy over the data governance components of the Cloudera platform, leveraging both Apache Atlas and Apache Ranger, the fundamental and integral components of SDX. Although most organizations have a mature approach to data access, control of metadata is typically less well defined, if considered at all. The insights and mechanisms shared in this article can help implement a more complete approach to data as well as metadata governance. The approach is critical in the context of a compliance strategy where data governance components play a critical role.

You can learn more about SDX here; or, we would love to hear from you to discuss your specific data governance needs.

The post Metadata Management and Data Governance with Cloudera SDX appeared first on Cloudera Blog.

View Details

As described in our recent blog post, an SQL AI Assistant has been integrated into Hue with the capability to leverage the power of large language models (LLMs) for a number of SQL tasks. It can help you to create, edit, optimize, fix, and succinctly summarize queries using natural language. This is a real game-changer for data analysts on all levels and will make SQL development faster, easier, and less error-prone.

This blog post aims to help you understand what you can do to get started with generative AI assisted SQL using Hue image version ​​2023.0.16.0 or higher on the public cloud. Both Hive and Impala dialects are supported. Please refer to the product documentation for more information about specific releases.

Getting started with the SQL AI AssistantLater in this blog we will walk you through the steps of how to configure your Cloudera environment to use the SQL AI Assistant with your supported LLM of choice. But first, let’s explore what the SQL AI Assistant does, and how people would use it within the SQL editor.

Using the SQL AI AssistantTo launch the SQL AI Assistant, start the SQL editor in Hue and click the blue dot as shown in the following image. This will expand the SQL AI toolbar with buttons to generate, edit, explain, optimize and fix SQL statements. The assistant will use the same database as the editor, which in the image below is set to a DB named tpcds_10_text.

The toolbar is context aware and different actions will be enabled depending on what you are doing in the editor. When the editor is empty, the only option available is to generate new SQL from natural language.

Click “generate” and type your query in natural language. In the edit field, press the down arrow to see a history of query prompts. Click “enter” to generate the SQL query.

The generated SQL is presented in a modal together with the assumptions made by the LLM. This can include assumptions about the intent of the natural language used, like the definition of “top selling products,” values of needed literals, and how joins can be created. Now, you can insert the SQL directly into the editor or copy it to the clipboard.

When there is an active SQL statement in the editor the SQL AI Assistant will enable the “edit,” “explain,” and “optimize” buttons. The “fix” button will only be enabled when the editor finds an error, such as a SQL syntax error or a misspelled name.

Click “edit” to modify the active SQL statement. If the statement is preceded by a NQL-comment then that prompt can be reused by pressing tab. You can also just start typing a new instruction.

After using edit, optimize, or fix, a preview shows the original query and the modified query differences. If the original query has a different formatting or keyword upper/lower case than the generated query, you can enable “Autoformat SQL” at the top of the modal for a better result.

Click “insert” to replace the original query with the modified one in the editor.

The optimize and the fix functionality do not need user input. To use them simply select a SQL statement in the editor, and click “optimize” or “fix” to generate an improved version displayed as a diff of the original query, as shown above. “Optimize” will try to improve the structure and performance without impacting the returned result of running the query. “Fix” will try to automatically fix syntactic errors and misspelling.

If you need help making sense of complex SQL then simply select the statement, and click “explain.” A summary and explanation of the SQL in natural language will appear. You can choose to insert the text as a comment above the SQL statement in the editor as shown below.

Supported AI models and servicesThe SQL AI Assistant is not bundled with a specific LLM; instead it supports various LLMs and hosting services. The model can run locally, be hosted on CML infra or in the infrastructure of a trusted service provider. Cloudera has been testing with GPT running in both Azure and OpenAI, but the following service-model combinations are also supported:

Note: Cloudera recommends using the Hue AI assistant with the Azure OpenAI service.

The supported AI models are pre-trained on natural language and SQL but they have no knowledge of your organization’s data. To overcome this the SQL AI Assistant uses a Retrieval Augmented Generation (RAG)-based architecture where the appropriate information is retrieved for each individual SQL task (prompt) and used to augment the request to the LLM. During the retrieval process it uses the Python SentenceTransformers framework for semantic search, which by default utilizes the all-MiniLM-L6-v2 model. The SQL AI Assistant can be configured with many pre-trained models for better multi-lingual support. Below are the models tested by Cloudera:

Data shared with the LLM modelsIt is important to understand that by using the SQL AI Assistant you are sending your own prompts and also significant additional information as input to the LLM. The SQL AI Assistant will only share data that the currently logged-in user is allowed to access, but it is of utmost importance that you use a service that you can trust with your data. The RAG-based architecture reduces the number of tables sent per request to a short list of the most likely needed, but there is currently no way to explicitly exclude certain tables; consequently, info about all tables that the logged-in user can access in the database could be shared. The list below details exactly what is shared:

• Everything that a user inputs in the SQL AI Assistant
• The selected SQL statement (if any) in the Hue editor
• SQL dialect in use (Hive, Impala for example)
• Table details such as table name, column names, column data types and related keys, partitions and constraints
• Three sample rows from the tables (following the best practices specified in Rajkumar et al, 2022)

Prerequisites for enabling the AI assistantThe administrator must obtain clearance from your organization’s infosec team to make sure it is safe to use the SQL AI Assistant because some of the table metadata and data, as mentioned in the previous section, is shared with the LLM.

Configuring the SQL AI AssistantGetting started with the SQL AI Assistant is a straightforward process. First arrange access to one of the supported services and then add the service details in Hue’s configuration.

Using Microsoft Azure OpenAI serviceMicrosoft Azure provides the option to have dedicated deployments of OpenAI GPT models. Azure’s OpenAI service is much more secure than the publicly hosted OpenAI APIs because the data can be processed in your virtual private cloud (VPC). Considering the added security, Azure’s OpenAI is the recommended service to use for GPT models in the SQL AI Assistant. For more information, see the Azure OpenAI quick start guide.

Step 1. Azure subscriptionFirst, get Azure access. Contact your IT department to get an Azure subscription. Subscriptions could be different based on your team and purpose. For more information, see subscription considerations.

1. Azure Open AI accessCurrently, access to this service is granted only by application. You can apply for access to Azure OpenAI by completing the form at https://aka.ms/oai/access. Once approved, you should receive a welcome email.

2. Create resourceIn the Azure portal, create your Azure OpenAI resource: https://portal.azure.com/#home.

In the resource details page, under “Develop”, you can get your resource URL and keys. You just need any one of the two provided keys.

1. Deploy GPTGo to Azure OpenAI Studio at https://oai.azure.com/portal and create your deployment under management > Deployments. Select gpt-35-turbo-16k or higher.

2. Configure SQL AI Assistant in HueNow that the service is up and running with your model, the last step is to enable and configure the SQL AI assistant in Hue.

3. Log in to the Cloudera Data Warehouse service as DWAdmin.

4. Go to the virtual warehouse tab, locate the Virtual Warehouse on which you want to enable this feature, and click “edit.”
5. Go to “configurations” > Hue and select “hue-safety-valve” from the configuration files drop-down menu.

Edit the text under the desktop section by adding a subsection called ai_interface. Populate it as shown below by replacing the angle bracket values with those from your own service:

Using OpenAI service1. Open AI platform sign upRequest access to the Open AI platform from your IT department or go to https://platform.openai.com/ and create an account if allowed by your company’s policies.

1. Get the API keyIn the left menu bar, navigate to AI keys. You should be able to view existing keys or create new ones. The API key is the only thing you need to integrate with the SQL AI Assistant.

2. Configure SQL AI Assistant in HueFinally, enable and configure the SQL AI assistant in Hue.

3. Log in to the data warehouse service as DWAdmin.

4. Go to the virtual warehouse tab, locate the Virtual Warehouse on which you want to enable this feature, and click “edit.”
5. Go to “configurations” > Hue and select “hue-safety-valve” from the configuration files drop-down menu.
6. Edit the text under the desktop section by adding a subsection called ai_interface. Only two key value pairs are needed as shown below. Replace the value with the API key from Open AI.

Amazon Bedrock ServiceAmazon Bedrock is a fully managed service that makes foundation models from leading AI startups and Amazon available via an API. You must have an AWS account with Bedrock access before following these steps.

1. Get your access key and secret

Get the access key ID and the secret access key for using Bedrock-hosted models in Hue Assistant:

1. Go to IAM console: https://console.aws.amazon.com/iam
2. Click “users” in the left menu
3. Find the user who needs access
4. Click “security credentials”

5. Go to the “access keys” section and find your keys there.

6. Get Anthropic Claude accessClaude from Anthropic is one of the best models available in Bedrock for SQL-related tasks. More details are available at https://aws.amazon.com/bedrock/claude/. Once you have access, you will be able to try Claude in the text playground under the Amazon Bedrock service.

7. Configure SQL AI Assistant in HueFinally, enable and configure the SQL AI assistant in Hue.

8. Log in to the data warehouse service as DWAdmin.

9. Go to the virtual warehouse tab, locate the virtual warehouse on which you want to enable this feature, and click “edit.”
10. Go to “configurations: > Hue and select “hue-safety-valve” from the configuration files drop-down menu.
11. Edit the text to make sure the following sections, subsections and key value pairs are set. Replace the and the with the values from your AWS account.

SQL AI Assistant configurationsService- and model-related configurations are under ai_interface, and semantic search related configurations used for RAG are under the semantic_search section.

Limitations and known issuesThe configurable LLMs are very good at generating and modifying SQL. The RAG architecture provides the proper context. But there is no guarantee suggestions from LLMs, or from human experts, are always accurate. Please be aware of the following:

• Non-deterministic: LLMs are non-deterministic. You cannot guarantee the exact same output for the same input every time, and different responses for very similar queries can occur.
• Ambiguity: LLMs may struggle to handle ambiguous queries or contexts. SQL queries often rely on specific and unambiguous language, but LLMs can misinterpret or generate ambiguous SQL queries, leading to incorrect results.
• Hallucination: In the context of LLMs, hallucination refers to a phenomenon where these models generate responses that are incorrect, nonsensical, or fabricated. Occasionally you might see incorrect identifiers or literals, or even table and column names, if the provided context is incomplete or user input simply doesn’t match any data.
• Partial context: The RAG architecture provides context to each request but it has limitations and there is no guarantee the context sent to the LLM will always be complete.

The SQL AI Assistant is now available in tech preview on Cloudera Data Warehouse on Public Cloud. We encourage you to try it out and experience the benefits it can provide when it comes to working with SQL. Additionally, check out the overview blog on SQL AI Assistant to learn how it can help data and business analysts in your organization speed up data analytics. Check out the SQL AI Assistant documentation Reach out to your Cloudera team for more details.

The post Setting up and Getting Started with Cloudera’s New SQL AI Assistant appeared first on Cloudera Blog.

View Details

Cloudera DataFlow for the Public Cloud (CDF-PC) is a complete self-service streaming data capture and movement platform based on Apache NiFi. It allows developers to interactively design data flows in a drag and drop designer, which can be deployed as continuously running, auto-scaling flow deployments or event-driven serverless functions. CDF-PC comes with a monitoring dashboard out of the box for data flow health and performance monitoring. Key performance indicators (KPIs) and associated alerts help customers monitor what matters for their use cases.

Many organizations have invested in central monitoring and observability tools such as Prometheus and Grafana and are looking for ways to integrate key data flow metrics into their existing architecture.

In this blog we will dive into how CDF-PC’s support for NiFi reporting tasks can be used to monitor key metrics in Prometheus and Grafana.

Target architecture: connecting the piecesThe key to insightful Grafana dashboards is having access to relevant application metrics. In our case, these are NiFi metrics of our flow deployment. We therefore need to be able to expose NiFi metrics for Prometheus so it can scrape them before we can build dashboards in Grafana. CDF-PC’s support for Prometheus reporting tasks and inbound connections allows Prometheus to scrape metrics in real time. Once the metrics are in Prometheus, querying it and building dashboards on top of it in Grafana is straightforward. So let’s take a closer look at how we get from having metrics in our flow deployment to a fully featured Grafana dashboard by implementing the target architecture shown in Figure 1 below.

Configuring a CDF deployment to be scraped by PrometheusStarting with CDF-PC 2.6.1, you can now programmatically create NiFi reporting tasks to make relevant metrics available to various third party monitoring systems. The Prometheus reporting task that we’ll use for this example creates an HTTP(S) metrics endpoint that can be scraped by Prometheus agents or servers. To use this reporting task in a CDF-PC deployment, we have to complete the following steps:

1. Ensure that the HTTP(s) metrics endpoint is reachable from Prometheus by configuring an inbound connections endpoint when creating the deployment.
2. Create and configure the Prometheus reporting task using the CDP CLI after successful deployment creation.

Creating a deployment with an inbound connections endpointWhen creating a deployment, CDF-PC gives you the option to allow NiFi to receive data by configuring an inbound connections endpoint. When the option is checked, CDF-PC will suggest an endpoint hostname that you can customize as needed.

The inbound connections endpoint gives external applications the ability to send data to a deployment, or in our case, connect to a deployment to scrape its metrics. In addition to the endpoint hostname we also have to provide at least one port that we want to expose. In our case we’re using Port 9090 and are exposing it with the TCP protocol.

After you have created your deployment with an inbound connection endpoint, navigate to the NiFi configuration tab in the deployment manager where you will see all relevant information to connect external applications to the deployment. Now that the deployment has been created, we can move on to the next step—creating the reporting task.

Creating and configuring the NiFi Prometheus reporting taskWe can now use the CDP CLI to create and configure the Prometheus reporting task. Download and configure the CDP CLI. Make sure that you are running at least version 0.9.101 by running cdp –version.

The command we’re going to use is the cdp dfworkload create-reporting-task command. It requires the deployment CRN, environment CRN and a JSON definition of the reporting task that we want to create. Copy the deployment CRN from the deployment manager, get the environment CRN for the associated CDP environment, and start constructing the command.

cdp dfworkload create-reporting-task --deployment-crn crn:cdp:df:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:deployment:eb2717f3-1bdf-4150-bd33-5b15d715bc7d/5cdc4d43-2991-4d4c-99fc-c400cd15853d --environment-crn crn:cdp:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:bf58748f-7ef4-477a-9c63-448b51e5c98f The missing piece is providing the information about which reporting task we want to create. All supported reporting tasks can be passed in using their configuration JSON file. Here’s the JSON configuration for our Prometheus reporting task. It consists of a Prometheus-specific properties section followed by generic reporting task configuration properties such as whether the reporting task should be started, how frequently it should run, and how it should be scheduled.

{ "name": "PrometheusReportingTask", "type": "org.apache.nifi.reporting.prometheus.PrometheusReportingTask", "properties": { "prometheus-reporting-task-metrics-endpoint-port": "9090", "prometheus-reporting-task-metrics-strategy": "All Components", "prometheus-reporting-task-instance-id": "${hostname(true)}", "prometheus-reporting-task-client-auth": "No Authentication", "prometheus-reporting-task-metrics-send-jvm": "false" }, "propertyDescriptors": {}, "scheduledState": "RUNNING", "schedulingPeriod": "60 sec", "schedulingStrategy": "TIMER\_DRIVEN", "componentType": "REPORTING\_TASK" }

| Configuration Property | Description | | prometheus-reporting-task-metrics-endpoint-port | The port that this reporting task will use to expose metrics. This port must match the port you specified earlier when configuring the inbound connection endpoint. | | prometheus-reporting-task-metrics-strategy | Defines granularity on which to report metrics. Supported values are “All Components,” “Root Process Group,” and “All Process Groups.” Use this to limit metrics as needed. | | prometheus-reporting-task-instance-id | The ID that will be sent alongside the metrics. You can use this property to identify your deployments in Prometheus. | | prometheus-reporting-task-client-auth | Does the endpoint require authentication? Supported values are “No Authentication,” “Want Authentication,” or “Need Authentication”. | | prometheus-reporting-task-metrics-send-jvm | Defines whether JVM metrics are also exposed. Supported values are “true” and “false.” |

Table 1: Prometheus configuration properties of the NiFi Prometheus reporting task.

You can either pass the JSON file as a parameter to the CLI command or reference a file. Let’s assume we’re saving the above JSON content in a file called prometheus_reporting_task.json.

Now we can construct our final CLI command that will create the desired reporting task:

cdp dfworkload create-reporting-task --deployment-crn crn:cdp:df:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:deployment:eb2717f3-1bdf-4150-bd33-5b15d715bc7d/5cdc4d43-2991-4d4c-99fc-c400cd15853d --environment-crn crn:cdp:environments:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:environment:bf58748f-7ef4-477a-9c63-448b51e5c98f --file-path prometheus\_reporting\_task.json

After executing the command, you should get a response back that contains the reporting task CRN:{ "crn": "crn:cdp:df:us-west-1:9d74eee4-1cad-45d7-b645-7ccf9edbb73d:reportingTask:eb2717f3-1bdf-4150-bd33-5b15d715bc7d/66a746af-018c-1000-0000-00005212b3ea"} To confirm that the reporting task was created, navigate to the NiFi configuration tab in the deployment manager and verify that the reporting task section reflects the reporting tasks you created using the CLI.

Now that our flow deployment and reporting task have been created, we can move on to the next step and configure the Prometheus server to scrape this deployment.

Configuring Prometheus to monitor a CDF deploymentTo define a new scraping target for Prometheus, we need to edit the Prometheus configuration file. Open the prometheus.yaml file to add the CDF deployment as a target.

Create a new job, e.g. with CDF deployment as its name. Next, copy the endpoint hostname of your CDF deployment from the deployment manager and add it as a new target.

scrape_configs:

# The job name is added as a label `job=<job\_name>` to any timeseries scraped from this config. - job\_name: "CDF Deployment" # metrics\_path defaults to '/metrics' # scheme defaults to 'http'. static\_configs: - targets: ["wikipediaprometheus.inbound.dfx.q5crnxxe.xcu2-8y8x.dev.cldr.work:9090"] Apply the configuration changes and navigate to the Prometheus web console to confirm that our CDF deployment is being scraped. Go to the Status→Targets and verify that your CDF Deployment is “Up.”

Once Prometheus has started scraping, you can explore all NiFi metrics in the metrics explorer and start building your Prometheus queries.

Sample Grafana dashboardGrafana is a popular choice for visualizing Prometheus metrics, and it makes it easy to monitor key NiFi metrics of our deployment.

Create a Prometheus connection to make all metrics and queries available in Grafana.

Now that Grafana is connected to Prometheus, you can create a new dashboard and add visualizations.

Let’s say we want to create a graph that represents the data that this deployment has received from external sources. Select “add visualization” on your dashboard and make sure your Prometheus connection is selected as the data source.

Select the nifi_amount_bytes_received metric. Use the label filters to narrow down the component in the flow. By using component_name and “Hello World Prometheus,” we’re monitoring the bytes received aggregated by the entire process group and therefore the flow. Alternatively you can monitor all components if no filter is defined or monitor individual processors too.

With all NiFi metrics being available in Grafana, we can now build a full dashboard tracking all relevant metrics. In the example below we’re monitoring total bytes received/sent, the number of flow files queued in all components, the average lineage duration and the current NiFi JVM heap usage which help us understand how our flows are doing.

ConclusionThe NiFi Prometheus reporting task, together with CDF inbound connections makes it easy to monitor key metrics in Prometheus and create Grafana dashboards. With the recently added create-reporting-task CDF CLI command, customers can now automate setting up Prometheus monitoring for every new deployment as part of their standard CI/CD pipeline.

Try out CDF-PC using the public five day trial and check out the Prometheus monitoring demo video below for a step-by-step tutorial.

The post Monitoring Cloudera DataFlow Deployments With Prometheus and Grafana appeared first on Cloudera Blog.

View Details

his November, Cloudera hosted its annual Week of Giving – a dedicated time each year for employees to come together as a community – to engage with one another and to make a positive impact in our society.

From Sao Paolo to Seattle, London to Shanghai, nearly 500 Clouderans participated, supporting causes meaningful to them. Together, we supported 345 nonprofit organizations – donating over 1,300 volunteer hours and $93,000.

“Participating in Week of Giving was one of the best moments of this year, being able to connect with other colleagues while doing something meaningful outside of our day to day job.”

Take a look at some of the highlights from throughout the week:

Giving back is an integral part of Cloudera’s culture. In addition to Week of Giving, we are proud to match employee donations, incentivize volunteering and cultivate a network of global volunteer ambassadors to ensure we carry the momentum of Week of Giving throughout the year.

The post Cloudera Week of Giving Recap appeared first on Cloudera Blog.

View Details

Imagine you’ve just started a new job working as a business analyst. You’ve been given a new burning business question that needs an immediate answer. How long would it take you to find the data you need to even begin to come up with a data-driven response? Imagine how many iterations of query writing you’d have to go through.

In this scenario, you also have reports that need updating as well. Those contain some of the biggest hair-ball queries you’ve ever seen. What do they mean? Imagine how long it takes to unravel those queries just to understand them, let alone make modifications to fit new business requirements.

Also, those crazy queries don’t always run the most efficient way possible. Some are returning errors that are difficult to find—and if you’re missing KPIs you have to fix, optimize, and measure every bit of code, which can take a considerable amount of time and trial and error.

What a nightmare! Now imagine you had a personal assistant who knew everything about your data sets and was an expert in SQL, sitting alongside you every step of the way to help you quickly problem solve, write optimized code, explain queries, and much more. That would be amazing wouldn’t it? Well imagine it no longer, as Cloudera’s SQL AI Assistant is exactly that!

Creating a query when you’re new to a data modelWhether you’re new to a role, or just new to a given data source, finding data is 90 percent of the query creation problem. However, with the new SQL AI Assistant, this is no longer a chore. All you have to do is launch the SQL AI Assistant, and ask it to generate a query based on a natural language prompt.

In this example, we’re going to look for a list of stores ordered by their performance in terms of total sales. To do that, we’ll launch the SQL AI Assistant, select “generate” from the menu and enter “get store name, store id, manager, zip code, total sales of each store, and sort by total sales in ascending order“ as our prompt.

In the “assumptions” field, we see how the SQL AI Assistant looked over our data model; compared to what we’re looking for, it was able to find the right tables, columns, and joins needed to provide a query that will give us the list we’re looking for. No more searching for tables and columns and digging into cryptic metadata with time consuming trial and error just to find the right data sets. And as a bonus, we even get the query written for us, saving us even more time!

Editing an existing query to refine the resultsFollowing along from the generation example above, let’s say we have a query and we want it to be a little more precise. We still need to examine the data to determine the right tables, columns, joins, and more to refine the query, and when we’re new to the data set this takes time. Even if the data are clear, if this isn’t a query we wrote in the first place; it can be hard to decide where to add additional joins and where clauses, etc., and not mess up the entire result. Have no fear, the SQL AI Assistant is here, and can help.

Let’s say that the list of stores by sales just isn’t helping us understand our performance measures quite right. Larger stores with more sales people will surely have larger sales. Maybe what we really want is a breakdown by sales representative by store, so we can see who has the best average sales per teammate, to get a better picture of what’s happening? So, to do that, with our original query in the query editor field, we can use the “edit” menu item from the SQL AI Assistant and write a prompt for just what we want to add—and not restate the entire problem we’re solving. In this case, we’re just going to ask the SQL AI Assistant to “add sales per employee and sort by sales per employee where sales per employee is total sales divided by the number of employees.”

Here, we see the difference between the original query (on the left) and the new query (on the right) so we can see exactly what the SQL AI Assistant is proposing as the change to the query itself. We also see an “assumptions” field that explains what it found for the additional data needed to refine the results. If we like these changes, we can “insert” them into the editor as our new query. Note, that we may also optionally include both the original prompt and the additional detail prompt in the comments of the new query so we keep track of the history of how we made this query as well.

Making sense of a complicated queryQuite often we come across queries we didn’t write, and the last known author cannot be found. Or, if you’re like me, it’s a query you wrote, but so long ago you cannot remember what it does. When it’s a simple query, that’s no big deal. But what if it is a complicated query with cryptic table and column names, and even when you run it and see the result set, you’ve got no idea how it works? And you’ve got to make a change to it to include more details or refine the result. Well the SQL AI Assistant still has you covered. Like an expert on both your data model and SQL, it will read the query and explain in natural language exactly what it does.

To do this, simply paste the query into the SQL editor field, and select “explain” from the SQL AI Assistant to get your explanation. In this example, we had this query to understand:

After running the explain process, you’ll see a natural language description of the query.

The SQL AI Assistant recognizes data-centric elements as well; where possible it will recognize things like comparing to the value 1.2 is the same as 20 percent above average. The explanation can be inserted into the SQL editor as a comment so we can keep, and modify, this explanation together with the query wherever we’re saving and documenting it.

Optimizing any querySometimes we are looking at a query that just seems overly complex. However, simplifying it for better readability or even faster performance can be a daunting, iterative task full of trial and error. Not anymore: with the SQL AI Assistant, you can easily ask for help to take any query and see if we can make it better. In this example, we have a query that contains many sub-selects and is hard to read and understand. If we paste this query into the SQL editor field and select “optimize” from the SQL AI Assistant menu, we will be given an optimized form of the query, if one is possible to create.

The result is a side-by-side comparison of the original query and an optimized form of it, together with the explanation of what we did to make it better: we made easier to read, easier to maintain, and possibly faster to execute. In this case we see the multiple sub-selects were converted into simple joins.

Fixing a query that won’t runSometimes we are struggling with a query that has a syntax error, but we can’t find it no matter how hard we stare at the code. The SQL AI Assistant can also help us in these cases as well. From anything as simple as a syntax error to anything as complex as a logical fault (such as a circular dependency), if you have the query in the SQL Editor you can simply select FIX from the menu, and see the recommendations the SQL AI Assistant finds for us.

In the example above, we see a side-by-side comparison of the query that wouldn’t run, and the fixed version. We see we forgot to close a bracket in the column list, we missed a space in the “group by” phrase, and we misspelled “limit” as “limits.”.

We also see one more correction that is interesting—in the “from” clause, we misspelled the table name as “stor_sales” instead of “store_sales.” That isn’t a syntax error, but certainly will be caught by the engine trying to run this query. The SQL AI Assistant also caught this mistake and offered us a correction for it, too.

After all the mistakes are caught, we can insert the corrected query into the editor, and will find it will now run.

Using the SQL AI Assistant, we can dramatically improve our work by having an intelligent SQL expert by our side, one that also knows our data schema very well. We can save time finding the right data, building the right syntax, and getting any new query started, with the generate feature. We can easily refine queries with the edit feature, make queries run better with the optimize feature, and eliminate errors with the fix feature. Using explain, we can rapidly document any query with rich natural language explanations of its function. All in all, we take the chore away from developing SQL, so we can focus on the fun part – answering tricky questions and using data to drive better decisions.

What’s nextThe SQL AI Assistant is now available in tech preview on Cloudera Data Warehouse on Public Cloud. We encourage you to try it out and experience the benefits it can provide when it comes to working with SQL, please refer to the help document to find details. Additionally, check out the Cloudera Data Warehouse page to learn more about self-serve data analytics, or the enterprise AI page to find how Cloudera Data Platform can help you turn AI hype into business reality.

The post Introducing the SQL AI Assistant:Create, Edit, Explain, Optimize, and Fix Any Query appeared first on Cloudera Blog.

View Details

Cloudera is launching and expanding partnerships to create a new enterprise artificial intelligence “AI” ecosystem. Businesses increasingly recognize AI solutions as critical differentiators in competitive markets and are ready to invest heavily to streamline their operations, improve customer experiences, and boost top-line growth. That’s why we’re building an ecosystem of technology providers to make it easier, more economical, and safer for our customers to maximize the value they get from AI.

At our recent Evolve Conference in New York we were extremely excited to announce our founding AI ecosystem partners: Amazon Web Services (“AWS“), NVIDIA, and Pinecone.

In addition to these founding partners we’re also building tight integrations with our ecosystem accelerators: Hugging Face, the leading AI community and model hub, and Ray, the best-in-class compute framework for AI workloads.

In this post we’ll give you an overview of these new and expanded partnerships and how we see them fitting into the emerging AI technology stack that supports the AI application lifecycle.

We’ll start with the enterprise AI stack. We see AI applications like chatbots being built on top of closed-source or open source foundational models. Those models are trained or augmented with data from a data management platform. The data management platform, models, and end applications are powered by cloud infrastructure and/or specialized hardware. In a stack including Cloudera Data Platform the applications and underlying models can also be deployed from the data management platform via Cloudera Machine Learning.

Here’s the future enterprise AI stack with our founding ecosystem partners and accelerators highlighted:

This is how we view that same stack supporting the enterprise AI application lifecycle:

Let’s use a simple example to explain how this ecosystem enables the AI application lifecycle:

• A company wants to deploy a support chatbot to decrease operational costs and improve customer experiences.
• They can select the best foundational LLM for the job from Amazon Bedrock (accessed via API call) or Hugging Face (accessed via download) using Cloudera Machine Learning (“CML”).
• Then they can build the application on CML using frameworks like Flask.
• They can improve the accuracy of the chatbot’s responses by checking each question against embeddings stored in Pinecone’s vector database and then enhance the question with data from Cloudera Open Data Lakehouse (more on how this works below).
• Finally they can deploy the application using CML’s containerized compute sessions powered by NVIDIA GPUs or AWS Inferentia—specialized hardware that improves inference performance while reducing costs.

Read on to learn more about how each of our founding partners and accelerators are collaborating with Cloudera to make it easier, more economical, and safer for our customers to maximize the value they get from AI.

Founding AI ecosystem partners | NVIDIA, AWS, PineconeNVIDIA | Specialized Hardware

Highlights:Currently, NVIDIA GPUs are already available in Cloudera Data Platform (CDP), allowing Cloudera customers to get eight times the performance on data engineering workloads at less than 50 percent incremental cost relative to modern CPU-only alternatives. This new phase in technology collaboration builds off of that success by adding key capabilities across the AI-application lifecycle in these areas:

1. Accelerate AI and machine learning workloads in Cloudera on Public Cloud and on-premises using NVIDIA GPUs
2. Accelerate data pipelines with GPUs in Cloudera Private Cloud
3. Deploy AI models in CML using NVIDIA Triton Inference Server
4. Accelerate generative AI models in CML using NVIDIA NeMo

Amazon Bedrock | Closed-Source Foundational Models

Highlights:We are building generative AI capabilities in Cloudera, using the power of Amazon Bedrock, a fully managed serverless service. Customers can quickly and easily build generative AI applications using these new features available in Cloudera.

With the general availability of Amazon Bedrock, Cloudera is releasing its latest applied ML prototype (AMP) built in Cloudera Machine Learning: CML Text Summarization AMP built using Amazon Bedrock. Using this AMP, customers can use foundation models available in Amazon Bedrock for text summarization of data managed both in Cloudera Public Cloud on AWS and Cloudera Private Cloud on-premise. More information can be found in our blog post here.

AWS | Cloud Infrastructure

Cloudera is working on integrations of AWS Inferentia and AWS Trainium–powered Amazon EC2 instances into Cloudera Machine Learning service (“CML”). This will give CML customers the ability to spin-up isolated compute sessions using these powerful and efficient accelerators purpose-built for AI workloads. More information can be found in our blog post here.

Pinecone | Vector Database

Highlights:The partnership will see Cloudera integrate Pinecone’s best-in-class vector database into Cloudera Data Platform (CDP), enabling organizations to easily build and deploy highly scalable, real time, AI-powered applications on Cloudera.

This includes the release of a new Applied ML Prototype (AMP) that will allow developers to quickly create and augment new knowledge bases from data on their own website, as well as pre-built connectors that will enable customers to quickly set up ingest pipelines in AI applications.

In the AMP, Pinceone’s vector database uses these knowledge bases to imbue context into chatbot responses, ensuring useful outputs. More information on this AMP and how vector databases add context to AI applications can be found in our blog post here.

AI ecosystem accelerators | Hugging Face, Ray:

Hugging Face | Model Hub

Highlights:Cloudera is integrating Hugging Faces’ market-leading range of LLMs, generative AI, and traditional pre-trained machine learning models and datasets into Cloudera Data Platform so customers can significantly reduce time-to-value in deploying AI applications. Cloudera and Hugging Face plan to do this with three key integrations:

Hugging Face Models Integration: Import and deploy any of Hugging Face’s models from Cloudera Machine Learning (CML) with a single click.

Hugging Face Datasets Integration: Import any of Hugging Face’s datasets via pre-built Cloudera Data Flow ReadyFlows into Iceberg tables in Cloudera Data Warehouse (CDW) with a single click.

Hugging Face Spaces Integration: Import and deploy any of Hugging Face’s Spaces (pre-built web applications for small-scale ML demos) via Cloudera Machine Learning with a single click. These will complement CML’s already robust catalog of Applied Machine Learning Prototypes (AMPs) that allow developers to quickly launch pre-built AI applications including an LLM Chatbot developed using an LLM from Hugging Face.

Ray | Distributed Compute Framework

Lost in the talk about OpenAI is the tremendous amount of compute needed to train and fine-tune LLMs, like GPT, and generative AI, like ChatGPT. Each iteration requires more compute and the limitation imposed by Moore’s Law quickly moves that task from single compute instances to distributed compute. To accomplish this, OpenAI has employed Ray to power the distributed compute platform to train each release of the GPT models. Ray has emerged as a popular framework because of its superior performance over Apache Spark for distributed AI compute workloads.

Ray can be used in Cloudera Machine Learning’s open-by-design architecture to bring fast distributed AI compute to CDP. This is enabled through a Ray Module in cml extension’s Python package published by our team. More information about Ray and how to deploy it in Cloudera Machine Learning can be found in our blog post here.

The post Announcing Cloudera’s Enterprise Artificial Intelligence Partnership Ecosystem appeared first on Cloudera Blog.

View Details

Without a doubt, 2023 has shaped up to be generative AI’s breakout year. Less than 12 months after the introduction of generative AI large language models such as ChatGPT and PaLM, image generators like Dall-E, Midjourney, and Stable Diffusion, and code generation tools like OpenAI Codex and GitHub CoPilot, organizations across every industry, including government, are beginning to leverage generative AI regularly to increase creativity and productivity.

Earlier this month, I had the opportunity to lead a roundtable discussion at the PSN Government Innovation show (2023 Government Innovation Show – Federal – Public Sector Network) in Washington, DC. There, I met with IT leaders across multiple lines of business and agencies in the US Federal government focused on optimizing the value of AI in the public sector. I’ll highlight some key insights and takeaways from my conversations in the paragraphs that follow.

Predictably, the roundtable participants I spoke with were guardedly optimistic about the potential for generative AI to accelerate their agency’s mission. In fact, most of the public servants I spoke with were predominantly cautious about the current limitations of generative AI, and underscored the need to ensure that models are used responsibly and ethically. As also expected, most had experimented on their own with large language models (LLM) and image generators. However, none of the government leaders I spoke with had deployed gen AI solutions into production, nor did they have plans to do so in the coming months, despite numerous applicable use cases within the federal government.

The underlying reason? Because the perceived potential benefits—improved citizen service through chatbots and voice assistants, increased operational efficiency through automation of repetitive, high-volume tasks, and rapid policymaking through synthesis of large amounts of data—are still outweighed by considerations about bias perpetuation, misinformation, fairness, transparency, accountability, security, and potential job displacement. Also, while agencies view embracing AI as a strategic imperative that will enable them to accelerate the mission, they also face the challenge of finding readily available talent and resources to build AI solutions.

Top operational problems in the public sectorRealizing the full potential of AI in the public sector requires tackling several operational problems that hinder government innovation and efficiency. Some of the primary operational problems highlighted at the PCN Government Innovation event include:

Civil Government: A major challenge facing the civil government is the inefficient and cumbersome procurement process. The lack of clear guidelines and the need for strict compliance with regulations results in a complex and time-consuming procurement process. AI-based procurement that uses natural language processing to process RFIs, RFPs, and RFQs, as well as text classification to streamline and automate processes such as supplier evaluation, contract analysis, and spend management, can streamline the procurement process and improve transparency and efficiency.

Defense and Intelligence Communities: The defense and intelligence communities face significant cybersecurity threats, with malicious actors trying to penetrate their systems continually. AI-enabled threat intelligence can help prevent cyberattacks, identify threats, and provide early warning to take necessary precautions. Innovations in AI-enabled data management in defense and intelligence communities also enable secure data sharing across the organization and with partners, optimizing data analysis and intelligence collaboration. By analyzing huge volumes of data in real time, including network traffic data, log files, security event, and endpoint data, AI systems can detect patterns and anomalies, helping to identify known and emerging threats.

State, Local, and Education: One of the significant challenges faced by state and local governments and education is the growing demand for social services. AI can optimize citizen-centric service delivery by predicting demand and customizing service delivery, resulting in reduced costs and improved outcomes. Academic institutions can leverage AI tools to track student performance and deliver personalized interventions to improve student outcomes. AI/ML models can process large volumes of structured and unstructured data, such as student academic records, learning management systems, attendance and participation data, library usage and resource access, social and demographic information, and surveys and feedback to provide insights and recommendations that optimize outcomes and student retention rates.

My final question to the roundtable was, “What are government agencies to do to optimize the value of AI today while balancing the inherent risks and limitations facing them?” Our government leaders had several suggestions:

1. Start small. Limit access and capabilities initially. Start with narrow, low-risk use cases. Slowly expand capabilities as benefits are proven and risks addressed.
2. Improve dataset quality. Ensure you can trust your data by using only diverse, high-quality training data that represents different demographics and viewpoints. Make sure to audit data regularly.
3. Develop mitigation strategies. Have plans to address issues like harmful content generation, data abuse, and algorithmic bias. Disable models if serious problems occur.
4. Identify operational problems AI can solve. Identify and prioritize potential use cases by their potential value to the organization, potential impact, and feasibility.
5. Establish clear AI ethics principles and policies. Form an ethics review board to oversee AI projects and ensure they align with ethical values. Update policies as needed when new challenges emerge.
6. Implement rigorous testing. Thoroughly test generative AI models for errors, bias, and safety issues before deployment. Continuously monitor models post-launch.
7. Increase AI model explainability. Employ techniques like LIME to better understand model behavior. Make key decisions interpretable.
8. Collaborate across sectors. Partner with academia, industry, and civil society to develop best practices. Learn from each other’s experiences.
9. Enhance AI expertise within government. Hire technical talent. Provide training on AI ethics, governance, and risk mitigation.
10. Communicate transparently with the public. Share progress updates and involve citizens in AI policymaking. Build public trust through education on AI.

The Year AheadThe next 12 months hold tremendous potential for the public sector with generative AI. As the technology continues to advance rapidly, government agencies have an opportunity to harness it to transform how they operate and serve citizens.

Learn more about how Cloudera can help you on your AI journey. Trust your data. Trust your enterprise AI. Enterprise AI | Cloudera

The post Optimizing the Value of AI Solutions for the Public Sector appeared first on Cloudera Blog.

View Details

Legal & General Investment Management (LGIM) is one of the largest global asset managers, managing £1.2 trillion on behalf of savers, retirees, and institutions worldwide. LGIM prides itself on being a responsible investor and is at the forefront of global index fund management and pension investment. Its strategies cover a broad array of asset classes and styles, including equities, bonds, property and alternatives, as well as multi-asset funds.

Implementing a cohesive data visionLGIM’s Global Chief Operating Office is responsible for the company’s technology, data, client servicing and the management of its strategic operating agenda. This includes delivering a transformation project to modernize LGIM, enable growth, and introduce more scalable and seamless services. In his role as LGIM’s Chief Technology Officer, Derrick Hastie is key to supporting this transformational agenda. He is responsible for introducing common tools and processes and enhancing the technology, tools and data available to employees, empowering them to use their expertise and creativity in innovative ways.

LGIM’s data ecosystem had become fragmented over time as it had grown. Data was hidden in silos and line-of-business teams were using multiple data management and analytics tools, many of which were not used to their full capability. To enable LGIM to better utilize its wealth of data, LGIM required a centralized platform that made internal data discovery easy for all teams and could securely integrate external partners and third-party outsourced data pipelines. To realize this cohesive data vision, LGIM adopted Cloudera Data Platform (CDP) Public Cloud.

“Cloudera provides us with the functionality to manage our entire data lifecycle, it’s my role to ensure that IT is collaborating with business teams to make certain they are armed with tools to create data products that effectively serve our clients. Moreover, widespread adoption of the platform allows our employees to unlock value from our data assets,” comments Derrick Hastie, CTO at LGIM.

A bridge between IT and business teamsLGIM worked with Cloudera’s Professional Services Team to build a centralized Data Marketplace that fosters a two-way relationship between technology and the business. The marketplace delivers a data-centric operating environment by increasing data accessibility and enabling advanced analytics. This allows LGIM’s business teams to focus on value creation by unlocking insights from market, trading, and risk data to better serve clients.

This cross-organizational approach that CDP facilitates has improved collaboration between LGIM’s IT and business teams, as well as reducing the number of local applications and tooling that IT has to manage. Data is now also linked bi-directionally throughout LGIM’s operations. The seamless inflow of retained and external datasets via CDP allows teams to create datasets locally and make them available for others to consume without the need for assistance from data engineers. As LGIM adds more use cases to the platform, its technical teams continue to work with Cloudera Professional Services to accelerate delivery, onboarding and optimization of additional workloads.

Data access unlocks new opportunities and builds resilienceCDP’s reliable and scalable architecture ensures data is always available, so LGIM can continuously offer its industry expertise to achieve positive long-term outcomes for its clients, investors and stakeholders. The out-of-the-box functionality of CDP also meant the LGIM IT team did not need to spend significant time integrating different tools, so it could more easily fulfill data lineage requirements and be confident its data is traceable and compliant.

“Cloudera Data Platform gives us confidence against the backdrop of a continually changing regulatory landscape. We know our Data Marketplace can run in the cloud or on-prem, and that we have full traceability of where data comes from, and where it leaves,” concluded Hastie.

Through implementing Cloudera public cloud on AWS, Legal and General Investment Management (LGIM) has been able to unleash data value with a sophisticated architectural runway realizing up to 45% savings in infrastructure costs for long-running workloads. Please read the full story here.

The post Cloudera Customer Story appeared first on Cloudera Blog.

View Details

During Week of Giving Clouderans across the globe took time out of their busy schedules to give back and support causes meaningful to them. For many colleagues, however, giving and volunteering during Week of Giving is just one of the many ways they support the causes meaningful to them.

We had the privilege of sitting down with Remus Lim, Regional VP of Sales in APAC who not only volunteered alongside his Singapore-based colleagues during Week of Giving but is dedicating an upcoming trip to philanthropy as well.

Over the Christmas holiday, Remus will be embarking on a 3-day trek in Taiga, Mongolia in support of Children’s Cancer Foundation, a well known organization in Singapore supporting cancer research and families impacted by childhood illness.

We thoroughly enjoyed learning about Remus’ passion and connection to CCF, the details of his upcoming journey and most of all ways we as Clouderans can support his fundraising efforts for this important cause.

Tell us a bit about this journey – what will you be doing? Who will you be traveling with? This December I will be joining a team from Beyond Limits and embarking on a 3-day trek in Taiga, Mongolia, an obscure but mystical part of the world that straddles the Mongolian-Russian border. It contains diverse terrain and stunning landscapes – think majestic steppe-taiga valleys and rugged peaks to breathtakingly dense boreal forests. Its beauty is unparalleled in most of the world).

Despite the beautiful scenery, the trek will not be easy. We’re expecting extreme weather that dips below -30°C as we traverse through 30-50 km of rolling hills and rugged, mountainous terrain. I have done many treks but not in nearly such extreme cold conditions. This physical intensity is to parallel the daily challenges our young cancer patients face daily.

Who else will you be traveling with?My wife and I will be trekking alongside other people who have been touched by cancer one way or another – teachers, students, businesspeople, caregivers, executives and people out of the workforce in need of a meaningful break.

The organizing company, Beyond Limits, focuses on charity expeditions and does an incredible job assembling a diverse group of volunteers.

What will a “typical day” look like? Honestly, I’m not sure but this is what I was told:

I’m excited to see how it is in real life!

This is such a momentous experience, what motivated you to sign up for such an intense physical challenge?Truth is, my wife signed me up! And the last time we did something meaningful together was during the Nepal Earthquake in 2015. I attended the briefing and was astonished to learn that approximately 140-150 children were diagnosed with cancer each year. In a small country of close to 6m people that number astounded me. I felt like it was something I could help address.

I chose to support CCF specifically because they are a well respected organization here in Singapore and they do such meaningful work. My first involvement with them was exactly 10 years ago when I participated in their event, Hair for Hope which is Children’s Cancer Foundation’s signature outreach event. It is the only head-shaving event in Singapore that serves to raise childhood cancer awareness and funds. (Every shaven head represents an understanding by an individual of the ordeals that a child with cancer is subjected to.) That year was also the year I lost my mum to cancer so participating in this event like this was particularly meaningful to me and I stayed involved in the organization ever since.

What inspired you to do this? The passing of Pak Erwin (our GVP for Indonesia) reminded me of the impermanence of life. My objective of going for this trek is to raise awareness and funds for CCF to support Children with cancer and their caregivers – and embrace life. Our fundraising goal is SGD100k and I know we can achieve that!

It is also motivating to me personally to go back to basics, to nature – tracing the steps of Gengis Khan.

What are you hoping to learn or experience while on this journey?Every trek is different and I’ve learned to expect the unexpected. I like the challenge of overcoming obstacles and most of the time, the challenge is mental. Be it for work or in life, the most important thing is attitude. I have never experienced sub 20 or sub 30 degrees so I anticipate this will be a challenge but I am looking forward to it! I see it as mental training.

Have you ever done something similar to this? If yes, did anything surprise you or inspire you to do this again? I did the Everest Base Camp trek in 2015 and fortunately for me, I left Nepal 5 days before the major earthquake. I called up my guide to find out if he and his family were safe. They were safe but lost their home and I donated some money to help him out.

The following year, my wife and I reached out to raise funds for the school children impacted by the earthquake at his hometown, a place called Dhading. It is in between Kathmandu and Pokhara connected by one road. We did a small trek at Poonhill with the same guide and moved on to Dhading to deliver School supplies to the school children living some 2,500m up on the mountain. Long story short, we escaped a landslide and had to get the supplies up the mountain by foot. We engaged a few porters of course. It was a tough trek up the muddy road, more for my wife but we found it really rewarding when we got up there seeing the smiles and deep appreciation from the children and their family.

Are there ways for Clouderans to get involved in this or follow your trek?Absolutely – I am still fundraising for my trek supporting CCF. If you would like to donate – please do! Every donation helps these Children and their families. It will be the best and most meaningful holiday gift.

Also, Beyond Limits organizes charity expeditions regularly. The next one is for the Breast Cancer Foundation.

The post #Volunteer Spotlight: Remus Lim appeared first on Cloudera Blog.

View Details

Recent Government Initiatives on Public Sector AI SolutionsIn recent years, governments across the globe have recognized the transformative potential of artificial intelligence (AI) and have embarked on initiatives to harness this technology to drive innovation and serve their citizens more effectively. These government-led efforts have had a profound impact on the development and adoption of AI solutions in the public sector, paving the way for a future where data-driven decision-making and automation are the norm.

One notable example of a government initiative that has shaped the AI landscape is the United States’ federal AI strategy. Launched in 2019, this strategy aims to position the US as a leader in AI research, development, and deployment. It focuses on five key pillars: investing in research and development; unleashing government AI resources; setting standards and policy; building the AI workforce; and advancing trust and security.

This strategy has spurred a wave of AI innovation within the public sector. Government agencies are actively exploring the use of AI for various applications, such as personalized citizen services, fraud detection, and smart infrastructure management. The US Department of Defense, for instance, is leveraging AI for predictive maintenance of military equipment, improving operational readiness and cost-effectiveness.

Similarly, other countries have also taken substantial steps to promote the use of AI in their public sectors. The United Kingdom’s Office for AI has launched initiatives to foster collaboration between academia, government, and industry, driving AI innovation and adoption. Estonia has implemented e-governance initiatives that rely heavily on AI to deliver efficient public services, attracting entrepreneurs from around the world.

These government initiatives have created a favorable environment for AI technology providers and startups to develop and refine solutions tailored to the unique needs of the public sector. As a result, AI is rapidly becoming a critical tool for governments to streamline operations, enhance public service delivery, and tackle complex challenges.

The Importance of Security, Governance, and Trustable DataWhile the potential of AI in the public sector is immense, ensuring the trustworthiness and reliability of AI solutions is critical in order to drive government missions forward. This requires a holistic approach that addresses the key areas of security, governance, and trustworthy data.

First and foremost, security is paramount when it comes to AI in the public sector. A breach or compromise of AI systems can have severe consequences, potentially compromising sensitive citizen data or even disrupting critical services. This summer’s MOVEit data breach has impacted over 2,600 organizations and 70,500,000 people worldwide, including Maximus (11 million), the state of Maine (1.3 million), the Louisiana Department of Motor Vehicles (6 million), and Oregon’s Department of Transportation (3.5 million), among others. Governments need to continue to prioritize building robust security mechanisms around AI solutions, incorporating techniques such as encryption, access controls, and threat detection to safeguard against cyber threats.

Governance is another crucial aspect that cannot be overlooked. AI systems must operate within a framework that promotes ethical practices, transparency, and accountability. Governments should establish clear guidelines and regulations surrounding the use of AI, ensuring that algorithms are fair, unbiased, and respectful of privacy rights. Collaborative efforts between government agencies, industry experts, and civil society organizations can foster the development of ethical AI frameworks and promote responsible AI deployment.

Lastly, trustworthy data forms the foundation of any AI solution. Governments must ensure that the data used for training AI models is of high quality, accurately representing the diverse range of scenarios and demographics it seeks to address. It is vital to establish stringent data governance practices to maintain data integrity, privacy, and compliance with regulatory requirements. Open data initiatives and data lakehouses, such as Cloudera’s open data lakehouse, provide governments with the means to consolidate and securely manage their data assets, ensuring its availability for AI applications while maintaining privacy and compliance.

By addressing the critical aspects of security, governance, and trustable data, governments can develop AI solutions that are reliable, transparent, and aligned with their mission to better serve the public. Open-source solutions like Cloudera Data Flow and Open Data Lakehouse provide the necessary infrastructure and tools for governments to build and deploy trustworthy AI solutions at scale.

Find out more about Cloudera Data Flow and CDP, the only hybrid data platform for modern data architectures with data anywhere here (Public Sector, Government Big Data Business Intelligence and Analytics (cloudera.com).

The post Building Trust in Public Sector AI Starts with Trusting Your Data appeared first on Cloudera Blog.

View Details

The Evolution of AI and LLMsArtificial intelligence (AI) and large language models (LLMs) have come a long way since their inception in the 1950s. From the pioneering research of English mathematician and logician Alan Turing to the recent breakthroughs achieved by models like GPT-3/GPT-4, AI has undeniably transformed industries and revolutionized human-computer interactions. But as AI becomes increasingly intertwined with our daily lives, developing an effective strategy to regulate it while optimizing value is more critical than ever.

As we rapidly approach the one year anniversary of ChatGPT’s release to the public, the number of users of the generative AI tool has skyrocketed from one million in the first five days to an estimated 180 million+. Given the resulting hype, not to mention the exponentially increasing value proposition to help drive innovation and advance government agency missions, President Biden’s October 2023 Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence was very timely. The order, which was 111 pages in total, charts a broad path, with short-term and long-term guidance towards responsible AI practices that protect privacy, address biases, and mitigate risks for years to come.

Short-term Impacts: Fostering Trust and TransparencyThe Biden EO establishes clear expectations for both federal agencies and the private sector, emphasizing the importance of trust, security, and transparency in AI development and deployment. The order also promotes fairness and mitigating biases, ensuring that AI technologies do not inadvertently perpetuate discriminatory outcomes. By implementing explainable AI methodologies, organizations will be able to comprehend the decision-making process of AI systems, enhancing accountability and public trust.

Moreover, the order prioritizes privacy and data protection, recognizing the need to safeguard sensitive information in an era of growing data breaches and cyber threats. With stricter guidelines on data handling and encryption, the government underscores its commitment to protecting citizens’ personal information. These short-term impacts lay the foundation for a more responsible and ethical AI ecosystem, fostering trust among citizens and stakeholders alike.

Long-term Impacts: Advancing AI for the Public GoodLooking ahead, the order is poised to drive transformative changes in the AI landscape, with far-reaching benefits for society. By prioritizing research and development, the government aims to advance AI innovations that serve the public good while addressing societal challenges. This long-term vision aligns with initiatives like the AI for Good Global Summit, where experts from various sectors collaborate to leverage AI to tackle issues such as climate change, healthcare disparities, and educational equity.

Furthermore, the executive order recognizes the importance of open-source AI solutions. Open-source frameworks and models provide a collaborative platform for researchers and developers, enabling them to build upon each other’s work while fostering transparency and innovation. By encouraging the adoption of open-source practices, the government promotes the democratization of AI, allowing smaller organizations and researchers to contribute meaningfully to the field, leading to a more inclusive and diverse AI community.

Collaboration and Overcoming ChallengesIt’s important to note that President Biden’s executive order on AI does not stand alone. It builds upon the collective wisdom of the AI community and complements existing frameworks like the AI Bill of Rights and the AI Risk Management Framework from the National Institute of Standards and Technology. These collaborative efforts across academia, industry, and government foster a multidimensional approach to addressing the challenges of AI, underscoring the power of collaboration in driving progress, overcoming biases, enhancing transparency, and securing the future of AI development and deployment.

By prioritizing trust, fairness, and transparency, the order lays a foundation for responsible AI practices that benefit individuals, communities, and society as a whole. It recognizes the potential of AI to address societal challenges and encourages collaboration to drive innovations for the public good. As the AI landscape continues to evolve, the principles outlined in this executive order will guide the way, fostering trust, security, and unlocking the true potential of AI technology.

Trusting in AI Requires Trusting Your DataGood AI rides on the back of good data. As governments strive to advance and accelerate their missions through the use of AI solutions, they must ensure the underlying data is of high quality and trustworthy. This can only be achieved through robust data management capabilities and well-established data strategy, governance, and security measures. With Cloudera, a world-class leader in open data lakehouse for trusted AI, public sector agencies can harness the power of generative AI to improve mission planning, intelligence analysis, and cybersecurity, ultimately enhancing national security efforts through cutting-edge technology solutions.

Find out more about CDP and the only open data lakehouse for both private and public cloud here (Cloudera Data Platform (CDP) | Cloudera).

The post Impacts and Takeaways From the Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence appeared first on Cloudera Blog.

View Details

In the dynamic world of machine learning operations (MLOps), staying ahead of the curve is essential. That’s why we’re excited to announce the Cloudera Model Registry as generally available, a game-changer that’s set to transform the way you manage your machine learning models in production environments.

Unlocking the power of model managementMachine learning has rapidly transformed the way businesses operate, but it has also introduced the need for robust model management. That’s where the Model Registry steps in. Think of it as your digital vault for machine learning models, a central hub that stores, organizes, and tracks every facet of your models and their life cycle. By providing a unified platform, it simplifies the complex task of model management across the entire life cycle of your machine learning projects.

What does the Model Registry offer?The Model Registry is designed to streamline these processes, offering a variety of tools and features.

Easy to use SDK: You can use the familiar MLFlow library that offers an intuitive, easy-to-use solution for model tracking. It simplifies recording model parameters, metadata, and metrics ensuring transparent bookkeeping. You can use the SDK to register your models in the Model Registry, enabling efficient management and deployment within your MLOps workflows.

Version Control: The Model Registry empowers you to store and manage multiple versions of your machine learning models. You can track each iteration, compare changes, and ensure that you always have access to the version that suits your needs. Model Registry eliminates versioning chaos and allows for a more systematic approach to model iteration.

Artifacts Management: The system efficiently handles the import and export of model artifacts in standard formats, promoting compatibility with different systems. It focuses on storing model artifacts in the Model Registry, linking development and production environments. This approach aids in straightforward model management and smooth transition across various stages of the project life cycle.

Lineage Tracking: It’s essential to maintain traceability in MLOps. The Model Registry records who made changes to a model, when these changes were made, and what the changes entailed. This creates a transparent and accountable record of a model’s evolution, which is important for effective model management and meeting regulatory requirements.

Robust APIs: The Model Registry’s APIs facilitate integration with CI/CD pipelines and essential tools in MLOps. They are designed to complement existing workflows, helping to streamline the transition of models from development to production. This integration supports the efficient operation of machine learning projects in a rapidly evolving landscape.

The future of MLOpsThe evolving landscape of MLOps is increasingly embracing hybrid and multi-cloud systems, offering significant flexibility for machine learning operations. This approach allows organizations to train their machine learning models in a private cloud environment and then deploy them to a public cloud, or vice versa. The adaptability of this method caters to various needs and scenarios, providing optimal environments for both the development and deployment phases. A key component in facilitating this flexible, cross-environment approach is the Model Registry. Its development is geared towards easing the transition between different cloud systems. This functionality is a prominent part of our road map, aiming to streamline the process of managing and deploying models across diverse cloud platforms, thereby enhancing the efficiency and scalability of machine learning workflows.

Get started todayThe Model Registry is now officially accessible in CML Public Cloud, ready to support both experienced data scientists and newcomers in machine learning. To harness the full potential of General Availability (GA), upgrade your CML Workspaces and deploy your new Model Registry! We encourage you to explore its features and see how it can assist in your machine learning projects. You can find more information about the new Model Registry in our community articles: How to set-up Model Registry and How to use Model Registry.

The post Announcing General Availability of Model Registry appeared first on Cloudera Blog.

View Details

With its rise in popularity generative AI has emerged as a top CEO priority, and the importance of performant, seamless, and secure data management and analytics solutions to power those AI applications is essential. Cloudera Private Cloud Data Services is a comprehensive platform that empowers organizations to deliver trusted enterprise data at scale in order to deliver fast, actionable insights and trusted AI. It offers a fully featured open data lakehouse with services including data warehousing, machine learning, and data engineering, in hybrid and multi-cloud environments. With the flexibility to harness the power of all your data, wherever it may be stored, Cloudera’s data services allow businesses to gain valuable insights and make informed decisions.

Cloudera and Dell are excited to announce a new milestone in our journey: the Quality Assurance Test Suite (QATS) certification for Dell PowerScale to work seamlessly with Cloudera Data Services.

Architecture overviewThe significance of Cloudera Private Cloud Data ServicesCloudera provides an integrated data platform that has become synonymous with immense open data lakehouse scalability and interoperability. With the recent release of Apache Iceberg, an engine-agnostic table format, businesses can deploy a single copy of data that caters to multiple use cases, realizing the elimination of data silos, simplified ETL pipelines, and reduced storage cost. This means you can expect simpler data management and drastically improved productivity for your business users. Cloudera brings cloud-native agility and innovation in the data analytics domain and offers a wide array of data services, including data warehousing, data engineering, and machine learning/AI for hybrid and multi-cloud environments. These unmatched data services provide the foundation for organizations to not only manage their data efficiently but also derive valuable insights from it.

The power of Dell PowerScaleIn the ever-evolving landscape of data management and analytics, businesses are constantly seeking solutions that offer scalability, performance, and ease of integration. Dell has been delivering PowerScale, the world’s most flexible (1), efficient (2), and secure (3) scale-out NAS solution that empowers you to easily store, protect, and manage your file data anywhere. Our partner ecosystem of 250+ application integrations and in-house vertical expertise ensure that you can power your workloads no matter which industry you are in. And now with the help of Cloudera, we’ve added the latest most comprehensive data platform, Cloudera Private Cloud Data Services, to the list. Limitless scale and extreme performance, delivered,

The significance of QATS certificationCertifications in the tech world are like badges of honor. They signify that a product has undergone rigorous testing and meets specific quality and compatibility standards. QATS, in particular, is known for its stringent testing procedures, making their certification highly respected.

As articulated by Martin Glynn, Sr Director Product Management,

“The QATS certification for our PowerScale and ECS products is another example of how Dell and Cloudera have partnered to bring best-in-class data management solutions to our joint customers. With PowerScale and ECS, customers can build a modern, scalable hybrid data platform on-premises, bringing many of the benefits of the public cloud to their data center.”

Dell PowerScale and ECS products have been certified by Cloudera from the inception of Cloudera’s QATS certification program. When Dell PowerScale received the QATS certification for Cloudera Private Cloud Data Services, it was a significant milestone. This certification validates that PowerScale seamlessly integrates with the entire Cloudera platform stack, allowing businesses to streamline their data management, analytics, and ML/AI workflows without any compatibility hiccups.

In the words of Cloudera’s Chief Product Officer, Dipto Chakravarty,

“Cloudera Private Cloud Data Services is the industry’s only Kubernetes-based private cloud platform that enables the full analytics life cycle for our customers, and stands ready for hybrid cloud. Now it also includes Dell PowerScale as a native object store for Cloudera’s platform. With the completion of QATS certification of PowerScale, customers can experience all the benefits of separating compute and storage, independent workload scaling, workload isolation, and the ability to seamlessly run hybrid cloud workloads without code changes.”

The power of the Dell and Cloudera partnershipDell and Cloudera have been delivering solutions together for more than 10 years. The integration of Dell PowerScale with Cloudera Private Cloud Data Services brings several advantages to the table:

Simplified Data Management: PowerScale’s compatibility with Cloudera Private Cloud Data Services means that data can be seamlessly ingested, stored, and analyzed within the same ecosystem. This eliminates the need for complex data transfers and reduces the risk of data silos.

Scalability: Both PowerScale and Cloudera Private Cloud Data Services are designed to scale horizontally, which means you can expand your infrastructure as your data grows. This scalability ensures that your data management and analytics capabilities can keep pace with your evolving business needs.

Performance: Dell PowerScale’s high-performance capabilities are complemented by Cloudera Private Cloud Data Services’ robust analytics and ML/AI tools. Together, they enable organizations to process and analyze large datasets quickly, resulting in faster insights and improved decision-making.

Data Security: Security is a paramount concern when dealing with sensitive data. The combination of PowerScale and Cloudera Private Cloud Data Services provides robust data protection mechanisms, ensuring the confidentiality and integrity of your data.

Cost-Efficiency: By streamlining data management and analytics workflows, this integration can yield cost savings by optimizing your resources and allocating them more efficiently.

In conclusionThe QATS certification of Dell PowerScale for compatibility with Cloudera Private Cloud Data Services is a testament to the commitment of Dell Technologies and Cloudera to providing seamless solutions for data-driven organizations. This certification not only ensures compatibility but also signifies the potential for improved efficiency, scalability, and performance in data management and analytics.

If your organization is looking to harness the power of data and make data-driven decisions with confidence, the combination of Dell PowerScale and Cloudera Private Cloud Data Services is worth exploring. It’s a partnership that brings together the best of both worlds, and the QATS certification is the seal of approval that you can trust for a successful data journey.

1. Based on internal analysis of publicly available information sources, February 2023.
2. Based on Dell analysis comparing efficiency-related features: data reduction, storage capacity, data protection, hardware, space, lifecycle management efficiency, and ENERGY STAR certified configurations, June 2023.
3. Based on Dell analysis comparing cyber-security software capabilities offered for Dell PowerScale vs. competitive products, September 2022.

The post Cloudera’s QATS Certification for Dell PowerScale Unleashes a New Era of Data Management appeared first on Cloudera Blog.

View Details

Earlier this month (November 6 through 8, 2023) a few hundred Apache Flink enthusiasts descended upon a Hyatt Regency Lake near Seattle for the annual Flink Forward conference. Cloudera was happy to participate, both as a sponsor of the conference and supporter of the open source community. Flink is, relatively speaking, a newer technology. However, it continues to gain adoption and inspire new development in the core engine as well as supporting technologies. Flink Forward is a great opportunity to learn about the cutting edge of streaming and stream processing technologies. This blog is a summary of what we observed there for anyone who was unable to attend or just wants to stay on top of what’s happening in streaming.

Takeaway No. 1: The Flink community is amazingI’d like to offer a proper hats-off to Veverica for organizing a fantastic conference. The conference had a laser focus on the open source technology and the developers who bring it to their organizations. No vendors pretending OS tech was their own secret sauce. No glorified advertisements masquerading as case studies. Just Flink-oriented content and training. The tech itself now boasts 1.4 million downloads, 21,000 GitHub stars, and 1,600 code contributions. There are individual Flink clusters in production as big as 4 million cores and 2,000 cluster nodes, clocked at 4.1 billion events/s. However you want to measure it, it’s safe to say that Flink has taken the mantle of “industry standard.”

Cloudera perspective: Flink is here to stay. When choosing open source or open core, a key consideration is the support of the community and the sustained development of the tech. No enterprise wants to bet on technology that will be out of fashion next year. Flink is a distributed engine that can be deployed on commodity hardware where it is lightning fast at astronomical scale. Vendors making claims of being faster than Flink should be viewed with suspicion.

Takeaway No. 2: The majority of Flink shops are in earlier phases of maturityWe talked to numerous developer teams who had migrated workloads from legacy ETL tools, Kafka streams, Spark streaming, or other tools for the efficiency and speed of Flink. Many critical downstream applications consume data processed by Flink, especially telcos, financial services, and e-commerce, where real-time processing needs are pronounced. But the burden of development and maintenance of these solutions often fell on small teams of Java programmers. There’s still a good percentage of self-managed Flink deployments that offer a series of challenges to solve in order to scale Flink. Many architects and team leaders expressed to us a desire to democratize stream processing to larger user bases, especially SQL analysts and/or a desire to move from manual configuration and maintenance of Flink environments to more of a PaaS model to maintain performance while freeing up development resources.

Cloudera perspective: This is exactly why we built SQL Stream Builder, a SQL-based no-code UI for analysts and domain experts. By democratizing access to streaming data, and bringing domain expert users into the development cycle, we help accelerate iterations on stream processing applications. This is vital when onboarding new data, or changing logic to meet evolving needs as is the case in fraud monitoring. Join our webinar December 14 to see a demonstration and ask questions.

Takeaway No. 3: Efforts to simplify deployment architectures are expected to help further accelerate adoptionMany organizations are moving their Flink deployments to Kubernetes. This will help accelerate deployment across environments and to optimize performance and resource utilization on an ongoing basis. DataOps rejoice—this is good news for Flink as it removes barriers to adoption and lowers the overall cost of deployment, significantly impacting the ROI on Flink pipelines and applications, especially when consolidating disparate processing tools.

Cloudera Perspective: Deployment architecture matters. Hybrid matters! Cloud-only solutions will not meet the needs for many use cases and run the risk of creating additional barriers for organizations. Cloudera is embracing Kubernetes in our Data in Motion stack, making our Flink PaaS offering more portable, scalable and suitable for data ops.

Takeaway No. 4: There is growing realization that Kafka is not enoughNumerous developers and architects expressed a desire to de-load Kafka and are looking to Flink for that purpose. Consider a few factors: First, many have been using Kafka as long-term storage and have seen their clusters grow without the same elasticity and accessibility one would expect from a modern data lake. Kafka has included “friends” Kconnect and Kstreams, but neither of those actually reduce the amount of data streamed, with Kconnect offering an all-or-nothing approach to bringing data into the stream. It should come as no surprise that streams have grown considerably over the years and here we are now where a common Flink use case is to simply filter streams to reduce the load on Kafka.

Cloudera perspective: The market has evolved. Organizations are moving beyond a Kafka-is-everything mentality when it comes to streaming. Workloads that don’t expressly require the many-to-many data sharing that publish/subscribe model solves for might be better for a universal data distribution too like NiFi for real-time needs or an open table format like Iceberg where making data accessible in near real time is acceptable. Cloudera offers Kafka with Flink and NiFi and Iceberg to provide a complete set of capabilities for streaming data that help organizations capture, process, and distribute and store any and all data needed to deliver the real time insights their applications and business users need.

Takeaway No. 5: Stream Processing and Lakehouse capabilities need each other. Veverica unveiled support for Apache Paimon, a new Apache project that seems poised to support this Kafka-offloading trend as part of a broader integration with data at rest. While an integrated storage solution for Flink is highly valuable it’s still early and not clear how the market will react to Paimon or “streamhouse” terminology. The project does tout some bells and whistles but ultimately little in terms of fundamental differentiation against Apache Iceberg. The Paimon community is nascent and heavily centered in one geo. Adoption has yet to really catch on. It’s unclear that there is enough incentive to do so—is there significant room between ultra low-latency Flink use cases and low-latency availability of Iceberg? What use cases are there where Iceberg low latency is too slow but real-time stream processing is unnecessary? Flink 2.0 is coming soon and has loads of upgrades for Iceberg integrations that can take advantage of killer features like time travel while Iceberg continues to develop an ecosystem of integrations that include Flink. Sink v2 is part of the Iceberg roadmap and will be a game changer for Flink SQL, providing incremental file compaction that will improve performance and reduce costs. It’s a positive sign that Iceberg will continue to develop integrations with Flink—after all, Iceberg has wide adoption from big organizations like Netflix, Apple, Citi, and Bloomberg, who also happen to have large Flink footprints and will be motivated to improve integrations between the two.

Cloudera perspective: Data Lakehouses have established themselves as core architectures at organizations across industries and it is becoming more clear that there is a need for Stream Processing capabilities that can be easily combined with lakehouse platforms.

Paimon might be a technology solution in search of a problem. For now, Flink plus Iceberg is the compute plus storage solution for streaming data. It’s important to place your bets strategically when choosing critical pieces of data infrastructure. There is a tremendous opportunity to simplify data architectures by combining a single unified processing engine with a single open-table storage solution. Over time, the open source community tends to consolidate efforts on a standard. Cloudera is monitoring the evolution and demand from our customers for Paimon at this stage.

Conclusion:All in all, Flink Forward was a fantastic conference. Cloudera is proud to support and contribute to the open source community and will be looking forward to sponsoring Flink Forward again. It feels like Flink is hitting an inflection point in adoption so we expect this time next year the community will have grown and matured a great deal!

For more information on how Cloudera is bringing Flink to the enterprise with SQL stream builder join our webinar Dec 14.

Download Cloudera Stream Processing Community edition for FREE and get zero to Flink in less than an hour. Our SQL Stream Builder console is the most complete you’ll find anywhere.

Sign up for a free trial of Cloudera’s NiFi-based DataFlow and walk through use cases like stream filtering and cloud data warehouse ingest.

The post 5 Key Takeaways from Flink Forward 2023 appeared first on Cloudera Blog.

View Details

The greater tech community was front row for a high-stakes corporate saga this past weekend, complete with more plot twists than the Succession series finale. The unexpected dismissal of OpenAI CEO Sam Altman, followed by a threatened employee mutiny, followed by Microsoft’s fastest hire ever (I’m not sure that I believe that Sam cleared all the HR requirements in that time), followed by the reinstatement of Sam Altman as the CEO of OpenAI, has reignited a crucial conversation in the tech community: the importance of not solely relying on third parties to provide AI solutions for critical business functions, and instead leveraging the open source community to bring those workloads in-house.

Why building in-house LLM solutions is crucial1. Strategic Control and Independence: Developing LLM solutions in house affords businesses greater control over their AI capabilities, turning black boxes into glass boxes, which is especially important for AI solutions that contribute to critical business operations. This autonomy ensures that companies are not at the mercy of external entities’ strategic decisions or operational upheavals. 2. Customization to Business Needs: In-house development allows for the customization of AI models to align with specific business objectives and operational requirements. While this level of customization can be achieved with third-party solutions, the data required to enable meaningful context in a model is likely proprietary or regulated, thus eliminating the option to customize with a third-party solution. 3. Intellectual Property and Competitive Advantage: Developing proprietary AI technologies can be a significant competitive advantage, especially in an era of increased democratization thanks to the prevalence of cutting-edge open source foundation models. It also ensures that intellectual property remains within the company, safeguarding against potential legal and security issues.

Challenges and considerations for in-house developmentWhile the benefits of in-house LLM development are clear, it’s important to acknowledge the challenges. These include the need for substantial investment in talent, technology, and training. The good news is that open source foundation models and companies like HuggingFace that make them easily available have considerably reduced the gap between the proprietary models coming out of groups like OpenAI and Anthropic and what a less specialized enterprise team can deliver. Companies must weigh these costs against the potential long-term benefits and consider their specific circumstances when deciding on their AI strategy.

The OpenAI incident: a wake-up callThe situation at OpenAI serves as a wake-up call for businesses to reassess their AI strategies. For companies that are heavily reliant on AI, the risk of external dependencies has become glaringly evident. The need for a more controlled, stable, and predictable approach to AI integration is paramount and more feasible than ever.

Preparing for an AI-driven futureIn conclusion, the recent events at OpenAI highlight the inherent risks of depending solely on third-party AI services. As AI continues to transform industries, building and owning in-house LLM solutions offers a strategic path for businesses seeking stability, customization, and independence in their AI endeavors. The journey towards in-house AI capabilities may be challenging, but the potential rewards for those who navigate it successfully are substantial, and Cloudera is here to partner with you on your path. Check out our Enterprise AI page to learn more!

The post Riding the OpenAI Rollercoaster appeared first on Cloudera Blog.

View Details

Of the many things one might take for granted, access to banking and financial services may not immediately come to mind. But as a thought experiment, imagine trying to buy a home or a car without the ability to take out a loan. Try depending on cash payments from your employer, or relying on alternative banking solutions like short-term payday loans, check-cashing services, and prepaid debit cards.

This may not seem like a reality to you personally, but as I noted in a previous blog, for the 1.7 billion adults worldwide who do not have access to formal financial services, it is very real and presents massive challenges to driving economic development and poverty reduction.

Lack of access to financial services and lack of education (inclusive of financial education) are two of the leading contributors to global poverty. Poverty affects us all, both socially and economically, not just the poor. According to the Center for American Progress, “The costs to the US associated with childhood poverty total about $500 billion per year, or the equivalent of nearly 4 percent of GDP.”

They state the annual effects of childhood poverty:

• Reduces productivity and economic output by about 1.3 percent of GDP
• Raises the costs of crime by 1.3 percent of GDP
• Raises health expenditures and reduces the value of health by 1.2 percent of GDP (source: CAP)

Providing affordable and accessible financial services to underserved or low-income individuals and communities—and empowering them with the resources they need to improve their economic well-being—is a problem that institutions need to more aggressively solve and it can be solved with data, and the will to succeed.

Data plays a pivotal role in the path to achieving financial inclusion. In this blog post, we’ll explore why financial inclusion makes good business sense and how data and AI are vital to transforming access to financial services.

Business reasons for financial inclusionIn addition to a compassionate desire to elevate people from all social and economic strata, institutions should pursue financial inclusion for several compelling business reasons.

First, promoting financial inclusion can lead to regulatory and governmental incentives or grants, including tax benefits or favorable regulations. Collaborating with governments and international organizations on financial inclusion initiatives can enhance reputation and create new opportunities for growth and innovation. For instance, the UN’s 2030 Agenda for Sustainable Development has identified 17 goals for sustainability—and this can’t be highlighted enough—of which financial inclusion is “positioned prominently as an enabler in eight of the 17.” Financial institutions that implement

Second, by engaging previously untapped markets, financial institutions can expand their customer base, which can lead to increased revenue and profitability.

Finally, it enhances customer loyalty and trust, as clients who feel supported in their financial journey are more likely to remain loyal to the institution.

Making data and AI work for the underservedBy leveraging data and emerging technologies that analyze it more comprehensively and efficiently, financial institutions can expand their reach to underserved populations, improving financial inclusion and empowering individuals with greater access to economic opportunities and financial stability.

Here are some real-world ways data and AI can serve the underserved.

Predictive analytics for targeted outreach: Understanding how individuals and communities conduct business and interact with financial services is key to making the greatest impact on the underserved. Data and AI can help financial institutions identify potential communities that are currently underserved by analyzing demographic, geographic, and behavioral data. They can target outreach efforts to reach these underserved populations with micro-branches as well as appropriate financial services and educational resources.

Learn how JP Morgan Chase is “combining business, data, policy, and philanthropic resources, to improve housing affordability and stability as well as homeownership opportunities for Black, Latino, and Hispanic households in the greater Washington, DC area.”

Credit scoring and risk assessment: Traditional credit scoring models rely on narrow and limited financial data, making it difficult for individuals without a well-established credit history to access loans or other financial products. AI algorithms can analyze a broader range of data sources, including non-traditional data like mobile phone usage, utility payments, and social media activity, to assess creditworthiness beyond credit history. This enables financial institutions to extend credit to a wider range of people, including those previously considered high-risk.

Read how Bank Rakyat Indonesia leveraged Cloudera to build an agile and reliable predictive augmented intelligent solution to enhance its credit scoring system.

Simplification of the application process: AI can play a significant role in helping underserved communities complete applications for financial services by guiding applicants through the application process and breaking it down into easy-to-understand steps while explaining complex financial terminology. AI-driven tools and apps can provide educational content to improve financial literacy, helping users better understand the implications of their financial decisions. They can also provide real-time translation services when one’s native language is not an option on the application. AI can assist in verifying and processing necessary documents by scanning, recognizing, and validating IDs, income documents, and other paperwork. AI algorithms can also analyze an applicant’s financial situation and offer personalized advice on which financial products or services suit their needs.

Cloudera Data Platform (CDP) has been instrumental in helping financial institutions take steps to address financial inclusion in underserved communities. CDP is a full data life cycle, hybrid data platform that enables banks to store, process, and analyze large volumes of data, including both structured and unstructured data, in a single, integrated platform. With CDP banks quickly and easily access the data they need to better understand the financial needs and behaviors of underserved communities, and design financial products and services that meet their unique needs.

Learn more about how Cloudera is helping the financial services industry here.

The post Data and AI as the Key to Unlocking Financial Inclusion appeared first on Cloudera Blog.

View Details

Have you ever wondered how massive business and consumer apps handle that kind of scale with concurrent users? To deploy high-performance applications at scale, a rugged operational database is essential. Cloudera Operational Database (COD) is a high-performance and highly scalable operational database designed for powering the biggest data applications on the planet at any scale. Powered by Apache HBase and Apache Phoenix, COD ships out of the box with Cloudera Data Platform (CDP) in the public cloud. It’s also multi-cloud ready to meet your business where it is today, whether AWS, Microsoft Azure, or GCP.

Support for cloud storage is an important capability of COD that, in addition to the pre-existing support for HDFS on local storage, offers a choice of price performance characteristics to the customers.

To understand how COD delivers the best cost-efficient performance for your applications, let’s dive into benchmarking results comparing COD using cloud storage vs COD on premises.

Test Environment:

The performance comparison was done to measure the performance differences between COD using storage on Hadoop Distributed File System (HDFS) and COD using cloud storage. We tested for two cloud storages, AWS S3 and Azure ABFS. These performance measurements were done on COD 7.2.15 runtime version.

The performance benchmark was done to measure the following aspects:

• Read-write workloads
• Read only workloads

The following configuration was used to setup a sidecar cluster:

• Runtime version: 7.2.15
• Number of worker nodes: 10

The cluster running with HBase on cloud storage was configured with a combined bucket cache size across the cluster as 32TB, with L2 bucket cache configured to use file-based cache storage on ephemeral storage volumes of 1.6TB capacity each. We ensured that this bucket cache was warmed up almost completely, i.e. all the regions on all the region servers were read into the bucket cache. This is done automatically whenever the region servers are started.

All the tests were run using YCSB benchmarking tool on COD with the following configurations:

• Amazon AWS
  • COD Version: 1.22
  • CDH: 7.2.14.2
  • Apache HBase on HDFS
    • No. of master nodes: 2 (m5.8xlarge)
    • No. of leader nodes: 1 (m5.2xlarge)
    • No. of gateway nodes: 1 (m5.2xlarge)
    • No. of worker nodes: 20 (m5.2xlarge) (Storage as HDFS with HDD)
  • Apache HBase on S3
    • No. of master nodes: 2 (m5.2xlarge)
    • No. of leader nodes: 1 (m5.2xlarge)
    • No. of gateway nodes: 1 (m5.2xlarge)
    • No. of worker nodes: 20 (i3.2xlarge) (Storage as S3)
• Microsoft Azure
  • Apache HBase on HDFS
    • No. of master nodes: 2 (Standard_D32_V3)
    • No. of leader nodes: 1 (Standard_D8_V3)
    • No. of gateway nodes: 1 (Standard_D8_V3)
    • No. of worker nodes: 20 (Standard_D8_V3)
    • Apache Hbase on ABFS
    • No. of master nodes: 2 (Standard_D8a_V4)
    • No. of leader nodes: 1 (Standard_D8a_V4)
    • No. of gateway node: 1 (Standard_D8a_V4)
    • No. of worker nodes: 20 (Standard_L8s_V2)

Here is some important information regarding the test methodology:

• Data size
  • 20 billion rows =~20TB
• Table was loaded from 10 sidecar worker nodes (2 billion rows per sidecar node) onto 20 COD DB cluster worker nodes
• Performance benchmarking was done using the following YCSB workloads
  • YCSB Workload C
    • 100% read
  • YCSB Workload A
    • Update heavy workload
    • 50% read, 50% write
• YCSB Workload F
• Read-Modify-Update workload
• 50% read, 25% update, 25% read-modify-update

The following parameters were used to run the workloads using YCSB:

• Each workload was run for 15 min (900 secs)
• Sample set for running the workloads
  • 1 billion rows
  • 100 million batch
• Following factors were considered while carrying out the following performance runs:
  • Overall CPU activity was below 5% before starting the run to ensure no major activities were ongoing on the cluster
  • Region server cache was warmed up (in case of Amazon AWS, with S3 and in case of Microsoft Azure, with ABFS) to the full capacity ensuring maximum amount of data was in cache
  • No other activities like major compaction were happening at the time of starting the workloads

Important findingsThe test started by loading 20TB of data into a COD cluster running HBase on HDFS. This load was carried out using the 10 node sidecar on the 20 node COD cluster running HBase on HDFS. Subsequently, a snapshot of this loaded data was taken and restored to the other COD clusters running HBase on Amazon S3 and Microsoft Azure ABFS. The following observations were made during this activity:

• Loading times = 52 hrs
• Snapshot time
  • Cluster to cluster = ~70 min
  • Cluster to cloud storage = ~70 min
  • Cloud storage to cluster = ~3hrs

Key takeawaysThe following table shows the throughput observed on running the performance benchmarks:

Based on the data shown above, we made the following observations:

• Overall, the average performance was better for a S3 based cluster with ephemeral cache by a factor of 1.7x as compared to HBase running on HDFS on HDD.
• Read throughput for S3 based cluster is better by around 1.8x for both HBase and Phoenix as compared to the HDFS based cluster.
• Some factors that affect the performance of S3 are:
  • Cache warming on S3: The cache should be warmed up to its capacity to get the best performance.
  • AWS S3 throttling: With the growing number of region servers and hence, the number of network requests to S3, AWS may throttle some requests for a few seconds which may affect the overall performance. These limits are set on AWS resources for each account.
  • Non atomic operations: Some operations like move do a lot of data copy instead of a simple rename and HBase relies heavily on these operations.
  • Slow bulk delete operations: For each such operation, the driver has to perform several operations like listing, creating, deleting which results in a slower performance.

As mentioned above, the cache was warmed to its full capacity in case of S3 based cluster. This cache warming took around 130 minutes with an average throughput of 2.62 GB/s.

The following chart shows the cache warming throughput with S3:

The following charts show the throughput and latencies observed in different run configurations:

The following few charts show comparative representation of various parameters when HBase is running on HDFS as compared to HBase running on S3.

AWS-HBase-Throughput (Ops/sec)

The following chart shows the throughput observed while running workloads on HDFS and AWS S3. Overall, AWS shows a better throughput performance as compared to HDFS.

AWS-HBase-Read Latency

The chart below shows the read latency observed while running the read workloads. Overall, the read latency is improved with AWS with ephemeral storage when compared to the HDFS.

AWS-HBase-Write Latency

The chart below shows the write latency observed while running the workloads A and F. The S3 shows an overall improvement in the write latency during the write heavy workloads.

The tests were also run to compare the performance of Phoenix when run with HBase running on HDFS as compared to HBase running on S3. The following charts show the performance comparison of a few key indicators.

AWS-Phoenix-Throughput(ops/sec)

The chart below shows the average throughput when the workloads were run with Phoenix against HDFS and S3. The overall read throughput is found to be better than the write throughput during the tests.

AWS-Phoenix Read Latency

The overall read latency for the read heavy workloads shows improvement when using S3. The chart below shows that the read latency observed with S3 is better by multifold when compared with the latency observed while running the workloads on HDFS.

AWS-Phoenix-Write Latency

The write heavy workload shows tremendous improvement in the performance because of the reduced write latency in S3 when compared to HDFS.

Azure

The performance measurements were also conducted on HBase running on Azure ABFS storage and the results were compared with HBase running on HDFS. The following few charts show the comparison of key performance metrics when HBase is running on HDFS vs. HBase running on ABFS.

Azure-HBase-Throughput(ops/sec)

The workloads running on HBase ABFS show almost 2x improvement when compared to HBase running on HDFS as depicted in the chart below.

Azure-Hbase-Read Latency

The chart below shows the read latency observed while running the read heavy workloads on HBase running on HDFS vs. HBase running on ABFS. Overall, the read latency in HBase running on ABFS is found to be more than 2x better when compared to HBase running on HDFS.

Azure-Hbase-Write Latency

The write-heavy workload results shown in the below chart show an improvement of almost 1.5x in the write latency in HBase running on ABFS as compared to HBase running on HDFS.

Things to consider when choosing the right COD deployment environment for you* Cache warming while using cloud storage + After the initial creation of the cluster, a warming-up process is initiated for the cache. This process entails fetching data from cloud storage to gradually populate the cache. Consequently, the cluster’s responsiveness to queries might experience a temporary slowdown during this period. This slowdown is primarily due to queries needing to access cloud storage for uncached blocks directly, all while contending with the cache population for CPU resources.

The duration of this warming-up phase typically falls within the range of three to five hours for a cluster configured with 1.5TB of cache per worker. This initial phase ensures optimized performance once the cache is fully populated and the cluster is running at its peak efficiency.

• AWS S3 throttling (AWS)

The inherent latency linked with such storage solutions is expected to cause slowness in retrieving data from cloud storage. And also, each access results in incurring a cost. However, the cloud storage’s built-in throttling mechanism stands as another significant factor that affects performance and resilience. This mechanism confines the number of allowed calls per second per prefix. Exceeding this limit results in unattended requests, with the potential consequence of halting cluster operations.

In this scenario, cache warming takes on a pivotal role in avoiding such situations. By proactively populating the cache with the data, the cluster can bypass a reliance on frequent and potentially throttled storage requests.

• Non-atomic operations
  • Operations within cloud storage lack atomicity, as seen in cases like renames in S3. To address this limitation, HBase has implemented a store file tracking mechanism which minimizes the necessity for such operations in the critical path, effectively eliminating the dependency on these operations.

ConclusionThe table below shows the total cost of ownership (TCO) for a cluster running COD on S3 without ephemeral cache (DR scenario) and with ephemeral cache (production scenario) as compared with a cluster running COD on HDFS.

We observed that the overall throughput of HBase with cloud storages with bucket cache is better than HBase running on HDFS with HDD. Here’s some highlights:

• With cached warm-up, cloud storage with cache yields 2x better performance with low TCO as compared to HDFS. The performance with cloud storage is attributed to local cache based on SSD where HDFS using costlier EBS-HDD requires three times of storage to account for replication.
• Write performance is expected to be same as both form factors uses HDFS as the base for WAL but as we are flushing and caching the data at the same time there is some 30% impact was seen

DR Cluster: This cluster is dedicated to disaster recovery efforts and typically handles write operations from less critical applications. Leveraging cloud storage without local storage to support cache, users can expect to achieve approximately 25% cost savings compared to an HDFS-based cluster.

Prod Cluster: Serving as the primary cluster, this environment functions as the definitive source of truth for all read and write activities generated by applications. By utilizing cloud storage solution with local storage to support cache, users can realize a substantial 40% reduction in costs

Visit the product page to learn more about Cloudera Operational Database or reach out to your account team.

The post Cloudera Operational Database (COD) Performance Benchmarking: Comparing HDFS and Cloud Storage appeared first on Cloudera Blog.

View Details

Are you struggling to manage the ever-increasing volume and variety of data in today’s constantly evolving landscape of modern data architectures? The vast tapestry of data types spanning structured, semi-structured, and unstructured data means data professionals need to be proficient with various data formats such as ORC, Parquet, Avro, CSV, and Apache Iceberg tables, to cover the ever growing spectrum of datasets – be they images, videos, sensor data, or other type of media content. Navigating this intricate maze of data can be challenging, and that’s why Apache Ozone has become a popular, cloud-native storage solution that spans any data use case with the performance needed for today’s data architectures.

Apache Ozone, a highly scalable, high performance distributed object store, provides the ideal solution to this requirement with its bucket layout flexibility and multi-protocol support. Apache Ozone is compatible with Amazon S3 and Hadoop FileSystem protocols and provides bucket layouts that are optimized for both Object Store and File system semantics. With these features, Apache Ozone can be used as a pure object store, a Hadoop Compatible FileSystem (HCFS), or both, enabling users to store different types of data in a single store and access the same data using multiple protocols providing the scale of an object store and the flexibility of the Hadoop File system.

A previous blog post describes the different bucket layouts available in Ozone. This blog post is intended to provide guidance to Ozone administrators and application developers on the optimal usage of the bucket layouts for different applications.

To start with, Ozone’s namespace includes the following conceptual entities:

Fig.1 Apache Ozone Namespace layout

• Volumes are the top level namespace grouping in Ozone. Volume names must be unique and can be used for tenants or users.
• Buckets can be used as parent directories. Each volume can contain multiple buckets of data. Bucket names must be unique within a volume.
• Keys store data inside buckets. Keys can be files, directories, or objects.

Bucket Layouts in Apache OzoneFile System Optimized (FSO) and Object Store (OBS) are the two new bucket layouts in Ozone for unified and optimized storage as well as access to files, directories, and objects. Bucket layouts provide a single Ozone cluster with the capabilities of both a Hadoop Compatible File System (HCFS) and Object Store (like Amazon S3). One of these two layouts should be used for all new storage needs.

A description of the bucket layouts and their features are below.

Fig 2. Bucket Layouts in Apache Ozone

Interoperability between FS and S3 APIUsers can store their data in Apache Ozone and can access the data with multiple protocols.

Protocols provided by Ozone:

• ofs
  • ofs is a Hadoop Compatible File System (HCFS) protocol.
• ozone fs is a command line interface similar to “hdfs dfs” CLI that works with HCFS protocols like ofs.
  • Most traditional analytics applications like Hive, Spark, Impala, YARN etc. are built to use the HCFS protocol natively and hence they can use the ofs protocol to access Ozone out of the box with no changes.
  • Trash implementation is available with the ofs protocol to ensure safe deletion of objects.
• S3
  • Any cloud-native S3 workload built to access S3 storage using either the AWS CLI, Boto S3 client, or other S3 client library can access Ozone via the S3 protocol.
  • Since Ozone supports the S3 API, it can also be accessed using the s3a connector. S3a is a translator from the Hadoop Compatible Filesystem API to the Amazon S3 REST API.
  • Hive, Spark, Impala, YARN, BI tools with S3 connectors can interact with Ozone using the s3a protocol.
  • When accessing FSO buckets through the S3 interface, paths are normalized, but renames and deletes are not atomic.
  • s3a will translate directory renames to individual object renames at the client before sending them to Ozone. Ozone’s S3 gateway will forward the object renames to the FSO bucket.
  • Access to LEGACY buckets using S3 interface is the same as access to FSO bucket if, ozone.om.enable.filesystem.paths=true otherwise, it’s the same as access to OBS bucket.
• o3
  • Ozone Shell (ozone sh) is a command line interface used to interact with Ozone using the o3 protocol.
  • Ozone Shell is recommended to use for volume and bucket management, but it can also be used to read and write data.
  • Only expected to be used by cluster administrators.

Fig 3. Interoperability between FS and S3 APIOzone’s support for interoperability between File System and Object Store API can facilitate the implementation of hybrid cloud use cases such as:

1- Ingesting data using S3 interface into FSO buckets for low latency analytics using the ofs protocol.

Fig 4. Ingest using S3 API and consume using FS API

2- Storing data on-premises for security and compliance which can also be accessed using cloud-compatible API.

Fig 5. Ingest using FS API and consume using S3 API

When to use FSO vs OBS Bucket LayoutsFig 6. When to use FSO vs OBS

• Analytics services built for HDFS are particularly well suited for FSO buckets:
• Apache Hive and Impala drop table query, recursive directory deletion, and directory moving operations on data in FSO buckets are faster and consistent without any partial results in case of any failure because renames and deletes are atomic and fast.
• Job Committers of Hive, Impala, and Spark often rename their temporary output files to a final output location at the end of the job. Renames are faster for files and directories in FSO buckets.
• Cloud-native applications built for S3 are better suited for OBS buckets:
• OBS buckets provide strict S3 compatibility.
• OBS buckets provide rich storage for media files and other unstructured data enabling exploration of unstructured data.

SummaryBucket layouts are a powerful feature that allow Apache Ozone to be used as both an Object Store and Hadoop Compatible File System. In this article, we have covered the benefits of each bucket layout and how to choose the best bucket layout for each workload.

If you are interested in learning more about how to use Apache Ozone to power data science, this is a great article. If you want to know more about Cloudera on private cloud, see here.

Our Professional Services, Support and Engineering teams are available to share their knowledge and expertise with you to choose the right bucket layouts for your various data and workload needs and optimize your data architecture. Please reach out to your Cloudera account team or get in touch with us here.

References:

[1] https://blog.cloudera.com/apache-ozone-a-high-performance-object-store-for-cdp-private-cloud/

[2] https://blog.cloudera.com/a-flexible-and-efficient-storage-system-for-diverse-workloads/

The post Apache Ozone – A Multi-Protocol Aware Storage System appeared first on Cloudera Blog.

View Details

We are excited to announce the winners of the 2023 EMEA Partner of the Year awards, presented this month at our EMEA Partner Summit in Lisbon, Portugal.

These awards recognize Cloudera’s EMEA partners who are dedicated to transforming the future of data by leveraging the most complete hybrid, multi-cloud data platform.

• EMEA Partner of the Year: IBM

IBM continues to be a key strategic partner, on track for another breakout year! In FY23 together we delivered record Total Contract Value (TCV) to more than 75 joint customers, across 21 countries within EMEA. Cloudera was named as IBM’s preferred partner of choice for Real-Time Data Movement using Kafka. Additionally, the unyielding support and engagement received from their executive team have played a pivotal role in our shared success.

• EMEA GSI Partner of the Year: Accenture

Accenture is a dedicated partner that has leaned in with Cloudera to pursue and create opportunities for joint success. As a technology partner in their Cloud Innovation Center in Rome, Italy, we work with Accenture to bring clients tested, cutting-edge, and tailored IT solutions. They have worked in collaboration with Cloudera and our other top partners to drive Public Cloud migration and Sovereign AI initiatives, and they have been active participants in our premier Evolve events throughout EMEA.

• EMEA Cloud Partner of the Year: AWS

Our collaboration with AWS continues at an astounding rate, with 200% year-on-year growth in our Amazon Web Services (AWS) business, entry into a multi-year strategic collaboration agreement, and the commitment they’ve demonstrated at our Evolve events, with AWS keynote speakers across 8 events in EMEA alone.

• EMEA VAR/VAD Partner of the Year: TD Synnex

TD Synnex is our single global distributor, with strong collaboration and commitment in each geo. We work vigorously to develop strategies in-market, and in EMEA we continue to look for ways to provide more value-added products to our partner ecosystem.

• EMEA IHV Partner of the Year: Dell Technologies

Dell Technologies has been a crucial partner since Cloudera’s founding in 2008. In EMEA we have recently accelerated our joint Go to Market efforts, working with their Storage and Compute field sales teams, and we’ve also started targeting the High-Performance Computing Market in collaboration with Dell, AMD, and NVIDIA.

• EMEA ISV Partner of the Year: Talend a Qlik Company

Talend by Qlik is a breakout star in our partner ecosystem and drives our joint Hybrid strategy, providing customers with a coordinated solution to maximize their investments in our platforms. Their Data Health and Lineage capability adds business context to data, enhancing data governance and, in turn helping enterprises accurately assess data risks.

Regional Partner Award Winners:In addition to the overall EMEA awards, Cloudera also recognized the following partners on a regional level:

• North EMEA Partner of the Year: Matrix

MATRIX is the leading technology service company in Israel, with 11,200 employees worldwide. We have had a successful, long-standing partnership that is highlighted by our collaboration on Project Nimbus, a cross-government project intended to provide a comprehensive framework for providing cloud services to the Government of Israel.

• Central EMEA Partner of the Year: SVA

SVA is one of Germany’s leading IT service providers. Winning CEMEA Partner of the Year for the second time in a row not only recognizes our joint success but also illuminates SVA’s high level of expertise in setting up and operating big data projects on CDP.

• South EMEA Partner of the Year: Agile Lab

Agile Lab is a highly trusted partner based in Italy. They have been a Cloudera partner for more than 5 years. Agile Lab produces a technology-agnostic, modular platform, that empowers modern enterprises to discover, elevate, and productize their data both in traditional environments and on fully compliant data mesh architectures.

• Middle East, Turkey, and Africa Partner of the Year: Solutions by STC

Solutions by STC has been a Cloudera partner since 2018. They have operated in Saudi Arabia for over 25 years, providing innovative, integrated technology solutions to enterprises and the Kingdom of Saudi. Our partnership continues to flourish, working closely together to address strategies and solutions across multiple verticals within the kingdom and across the region.

The winners were determined based on revenue, year-over-year growth, resource and certification investment in Cloudera, and technology alignment.

Insuring a Successful Partner Ecosystem The Annual EMEA Partner Summit is an important component to the collaboration and growth of our EMEA Partner Ecosystem.

Congratulations to each of our EMEA Partner of the Year award recipients. Thank you for a tremendous year, and we look forward to even greater success together in the years ahead!

The post EMEA Summit Partner Awards for AWS, Accenture, IBM, Talend a Qlik Company, Tech Data, and Dell Technologies appeared first on Cloudera Blog.

View Details

Elevate your AI applications with our latest applied ML prototypeAt Cloudera, we continuously strive to empower organizations to unlock the full potential of their data, catalyzing innovation and driving actionable insights. And so we are thrilled to introduce our latest applied ML prototype (AMP)—a large language model (LLM) chatbot customized with website data using Meta’s Llama2 LLM and Pinecone’s vector database.

Innovation in architectureIn order to leverage their own unique data in the deployment of an LLM’s (or other generative model), organizations must coordinate pipelines to continuously feed the system fresh data to be used for model refinement and augmentation.

This AMP is built on the foundation of one of our previous AMPs, with the additional enhancement of enabling customers to create a knowledge base from data on their own website using Cloudera DataFlow (CDF) and then augment questions to the chatbot from that same knowledge base in Pinecone. DataFlow helps our customers quickly assemble pre-built components to build data pipelines that can capture, process, and distribute any data, anywhere in real time. The entire pipeline for this AMP is available in a configurable ReadyFlow template that features a new connector to the Pinecone vector database to further accelerate deployment of LLM applications with updatable context. The connector makes it easy to update the LLM context by loading, chunking, generating embeddings, and inserting them into the Pinecone database as soon as new data is available.

Fig 1. High-level overview of real-time data ingest with Cloudera DataFlow to Pinecone vector database.

Navigating the challenge of “hallucinations”Our newest AMP is engineered to address a prevalent challenge in the deployment of generative AI solutions: “hallucinations.” The AMP demonstrates how organizations can create a dynamic knowledge base from website data, enhancing the chatbot’s ability to deliver context-rich, accurate responses. Its architecture, known as retrieval-augmented generation (RAG), is key in reducing hallucinated responses, enhancing the reliability and utility of LLM applications, making user experience more meaningful and valuable.

Fig 2. An overview of the RAG architecture with a vector database used to minimize hallucinations in the chatbot application.

The Pinecone advantagePinecone’s vector database emerges as a pivotal asset, acting as the long-term memory for AI, essential for imbuing interactions with context and accuracy. The use of Pinecone’s technology with Cloudera creates an ecosystem that facilitates the creation and deployment of robust, scalable, real-time AI applications fueled by an organization’s unique high-value data. Managing the data that represents organizational knowledge is easy for any developer and does not require exhaustive cycles of data science work.

Utilizing Pinecone for vector data storage over an in-house open-source vector store can be a prudent choice for organizations. Pinecone alleviates the operational burden of managing and scaling a vector database, allowing teams to focus more on deriving insights from data. It offers a highly optimized environment for similarity search and personalization, with a dedicated team ensuring continual service enhancement. Conversely, self-managed solutions may demand significant time and resources to maintain and optimize, making Pinecone a more efficient and reliable choice.

Embrace the new capabilitiesOur new LLM chatbot AMP, enhanced by Pinecone’s vector database and real-time embedding ingestion, is a testament to our dedication to pushing the boundaries in applied machine learning. It embodies our commitment to providing refined, innovative, and practical solutions that meet the evolving demands and challenges in the field of AI and machine learning. We invite you to explore the improved functionalities of this latest AMP.

The post Harness the Power of Pinecone with Cloudera’s New Applied Machine Learning Prototype appeared first on Cloudera Blog.

View Details

Enterprises see embracing AI as a strategic imperative that will enable them to stay relevant in increasingly competitive markets. However, it remains difficult to quickly build these capabilities given the challenges with finding readily available talent and resources to get started rapidly on the AI journey.

Cloudera recently signed a strategic collaboration agreement with Amazon Web Services (AWS), reinforcing our relationship and commitment to accelerating and scaling cloud native data management and data analytics on AWS. Our vision is to make it easier, more economical, and safer for our customers to maximize the value they get from AI. In this post, we share our vision and the integrations that are available to our customers on Cloudera Data Platform with generative AI on AWS. Generative AI offerings on AWS include Amazon Bedrock, Amazon SageMaker JumpStart, AWS Trainium, AWS Inferentia, Amazon CodeWhisperer, AWS HealthScribe, and Generative BI in Amazon QuickSight.

Our vision: building AI with CDP on AWSCloudera’s AI vision in alignment with AWS is to enable customers to leverage the 25 exabytes of data managed in Cloudera to build differentiated AI in their specific industry. Our vision is built on two pillars:

1. Build AI with Cloudera, powered by generative AI on AWS: Enable customers to build AI applications rapidly and cost-effectively by building capabilities and integrations between Cloudera Machine Learning and generative AI on AWS.
2. Build AI in Cloudera, powered by generative AI on AWS: Enable AI-powered productivity for data practitioners using Cloudera Data Platform (CDP) by building generative AI features into CDP.

Let us dive into what is happening in each of these pillars between AWS and Cloudera.

Building AI with Cloudera, powered by Amazon BedrockWe are building generative AI capabilities in Cloudera, using the power of Amazon Bedrock, a fully managed serverless service. Customers can quickly and easily build generative AI applications using these new features available in Cloudera.

CML text summarization AMP built using Amazon BedrockWith the general availability of Amazon Bedrock, Cloudera is releasing its latest applied ML prototype (AMP) built in Cloudera Machine Learning: CML Text Summarization AMP built using Amazon Bedrock. Using this AMP, customers can use foundation models available in Amazon Bedrock for text summarization of data managed both in Cloudera Public Cloud on AWS and Cloudera Private Cloud on-premise.

LLM Text Summarization AMP showcases how our customers can quickly build and deploy AI applications leveraging foundation models available in Amazon Bedrock to perform automated text summarization. This allows enterprises to distill lengthy documents, articles, or communications into concise and coherent summaries, facilitating quick decision-making and enhancing productivity. By harnessing the capabilities of Amazon Bedrock and our AMP, organizations can streamline their data analysis processes, extract crucial information, and gain a competitive edge.

Below is a high-level architecture and process flow for Cloudera’s Text Summarization AMP built using Amazon Bedrock:

In building this AMP, Cloudera’s research and development team explored and chose Amazon Bedrock.* With Amazon Bedrock, customers can interact via a single API and select from a wide range of industry leading foundation models. * As a fully managed service, there is no need to set up or manage any infrastructure, allowing customers to get started on building their application immediately. * We can fine-tune the Amazon Bedrock model using our own labeled data to create an accurate customized model for our specific problem. * Amazon Bedrock is integrated with AWS security capabilities, which customers were familiar with and helped them avoid a new infosec review, another major time saver. * Customers use the AWS tools and capabilities they are familiar with to deploy reliable, secure, and scalable generative AI applications.

For this use case, we selected Amazon’s Titan Text model for its strong track record with text summarization use cases and the use of responsible AI best practices in its creation.

Here’s an example of Cloudera’s AMP in action with the Amazon Bedrock API request code that’s automatically generated by the application based on the input text exposed. This AMP can be used on any Cloudera system running on-premise or any public cloud directly integrated with Amazon Bedrock APIs.

CML AWS Inferentia and AWS Trainium planned integrationsThe LLM Text Summarization AMP is just the beginning of the benefits our customers will gain from Cloudera and AWS generative AI product integrations. Cloudera is working on integrations of AWS Inferentia and AWS Trainium–powered Amazon EC2 instances into Cloudera Machine Learning service (CML). This will give CML customers the ability to spin-up isolated compute sessions using these powerful and efficient accelerators purpose-built for AI workloads.

AWS Trainium–powered Amazon EC2 instance support will bring efficiency improvements to the training phase of machine learning models within CML. Amazon EC2 Trn1 instances deliver faster time to train while offering up to 50 percent cost-to-train savings over comparable Amazon EC2 instances.

With AWS Inferentia, CML customers can leverage custom-designed inference chips, enabling faster and more cost-effective inference for their self-hosted machine learning models. Amazon EC2 Inf2 instances deliver up to nine times higher throughput and up to 80 percent lower cost per inference than comparable Amazon EC2 instances.

Customers can also use AWS Neuron SDK to train and deploy models on Amazon EC2 Trn1 and Amazon EC2 Inf2 instances as on-demand instances, reserved instances, and spot instances, or as part of a savings plan: US East (Northern Virginia), US West (Oregon), and US East (Ohio).

Building AI in Cloudera, powered by Amazon BedrockWe offer in-built generative AI capabilities within Cloudera services and applications, so customers can easily interact and benefit by getting faster outcomes.

CDP’s SQL code AI assistantWe couldn’t be more excited about building generative AI capabilities into CDP to power data practitioner productivity.

CDP’s SQL code AI assistant powered by Amazon Bedrock is already under development. This generative AI tool lets analysts generate and edit SQL queries using natural language statements. It can also optimize SQL queries to make them run more efficiently, explain what a SQL query is doing in plain English, and automatically find and fix errors in queries that won’t run. We are using the Claude v2 Foundation model from Anthropic available in Amazon Bedrock for this text-to-sql generation feature.

This tool alone will revolutionize how analysts get work done—allowing them to spend more time on creating business value and less time on writing code.

Below is the high-level architecture for CDP’s SQL code AI assistant:

We want to analyze sales by store so we click the generate button in HUE (our standard SQL editor UI). Then we write what data points we want in natural language and click go.

The AI assistant finds the relevant tables needed and writes the SQL query with a detailed explanation of its logic in seconds. All we have to do is review, click insert, and run it.

What’s next?Even with these integrations in our development pipeline we are just scratching the surface of what we will build using CDP and AWS AI services. Stay tuned for updates as we bring our vision to life by following our What’s New product feed. We’re more committed than ever to making it easier, economical, and safer for our customers to maximize the value they get from AI.

Resources to build generative AI with CDP on AWSTo learn more, check out new generative AI features available in Cloudera Machine Learning page. Subscribe to the 60-day CDP Public Cloud trial and start learning to build solutions with CDP on AWS. Learn about generative AI on AWS using AWS Training Resources and Amazon Bedrock Workshop.

The post Build Modern Innovative Solutions on Cloudera Data Platform Using the Power of Generative AI with Amazon Bedrock appeared first on Cloudera Blog.

View Details

As Halloween night quickly approaches, there is only one question on every kid’s mind: how can I maximize my candy haul this year with the best possible candy? This kind of question lends itself perfectly to data science approaches that enable quick and intuitive analysis of data across multiple sources. Using Cloudera Machine Learning, the world’s first hybrid data cloud machine learning tooling, let’s take a deep dive into the world of candy analytics to answer the tough question on everyone’s mind: How do we win Halloween?

So many factors go into obtaining the best possible candy portfolio. First of all it’s all about maximizing the number of doors knocked. This requires a densely populated location. However, this isn’t an option for every trick or treater. For example, I grew up in rural Montana where trick or treating required a car and snowshoes to get to each home (okay, not snowshoes, but definitely snow boots). If you find yourself in this situation, I highly recommend tracking average candy output per home each year. For example, if the Roger’s have handed out king size candy bars every year, it might be worth the extra 10 minute drive.

So far we’ve talked about quantity, but just as important is quality. This variable is largely out of your control, and can be dependent on the region you live in. I recently found out that there are companies that actually track the candy sales by state each year. CandyStore.com is one of these companies (on a side note, check out their website if you have a hankering for rare sweets). They released a blog this year with the results from their annual data mining, it includes the top 3 candies purchased for each state and the quantity purchased in pounds.

Some of the top purchased candies are wild. For example, take my home state of Montana, they purchased over 24 thousand pounds of Dubble Bubble Gum. You read that right, Dubble Bubble Gum, the rock-hard, 4-chews-with-flavor gum that everyone yearns for. Other states are a bit more of what you expect, Florida knows that no one can resist a classic like the Reeses Peanut Butter Cup, and Nevada plays it safe with a Hershey’s Mini Bar, a Halloween staple.

This got me thinking though, based on this data, there is likely a difference in taste between those buying the candy and those actually consuming it. Is there an easy way that we could identify these candy market imbalances? Luckily, when CML isn’t solving the world’s most ambitious predictive challenges for enterprise businesses, it’s the perfect tool for this kind of agile and ad-hoc data science discovery. To analyze and satisfy our candy questions, I’ll spin up JupyterLab natively in CML and immediately have access to both scalable compute and secure granular data to tackle this challenge in just a few clicks — let’s get started.

How to avoid the bad candyIf we want to find the states that bought “bad candies”, we need some way to quantify consumer taste preferences for various sweets. Enter The Ultimate Halloween Candy Power Ranking from FiveThirtyEight which contains the survey results from over 269,000 randomly generated candy matchups (i.e. do you like candy A or B better). The end result was a win percentage for 86 different mainstream candies.

Now, if we merge these two data sets together by candy name, we’re able to build a visualization that highlights the top purchased candy in each state, and the preference for that candy. The more black a state is, the more disliked the top candy purchased in that state is. When you hover over a state (or tap if you’re on your phone), the first number is the win percentage for the top candy in that state, you’ll also see the name of the candy and the amount of that candy purchased in 2023, according to CandyStore.com.

There are a few things that stick out to me. Louisianans must have a hankering for candy that kind of tastes like soap, because their top candy purchased is the rarely traded for Lemonhead, coming in at only 39% on FiveThirtyEight’s win percentage. In past candy analyses, Montana had elected Dubble Bubble as their top candy, but they seem to have found the error of their ways and our now focused on more liked candies since the Twix is the new #1 in the Big Sky state. Any state that is buying Candy Corn more than any other candy clearly has something against the children knocking on their doors. Yes, I’m looking at you Utah. Candy Corn’s win percentage is only 38%. So, if you’re a fan of Candy Corn or Lemonheads (aka if you have numb taste buds) you now know where to travel this holiday to find a surplus of your favorite disliked candy.

Analysis like these aren’t earth shattering, but not every analysis needs to be. What every analysis should be though is easy to do. Cloudera provides a variety of tools in the Cloudera Data Platform (CDP) that allow you to easily work with your data. If you want to give a tool like CML a try and run your own candy analysis, head over to our Demo page to learn more about everything that Cloudera has to offer.

The post The Ultimate Map to finding Halloween Candy Surplus appeared first on Cloudera Blog.

View Details

Cloud transformation is ranked as the cornerstone of innovation and digitalization. The legacy IT infrastructure to run the business operations—mainly data centers—has a deadline to shift to cloud-based services. Agility, innovation, and time-to-value are the key differentiators cloud service providers (CSP) claim to help organizations speed up digital transformation projects and business objectives. The public cloud is increasingly becoming the preferred platform to host data analytics–related projects, such as business intelligence, machine learning (ML), and AI applications. However, the reality is that the “move to cloud” is a turbulent flight for many of them. The main challenges are pointed out as a lack of resources/expertise, security, and from a different perspective, cloud cost management.

The decentralized resource provisioning approach, foundational to cloud solutions and infrastructure, faces a big challenge when it comes to governing the costs of on-demand resource allocation/utilization. Besides the advantages and changes in the technical aspects cloud brings, it imposes a paradigm shift from the traditional way of managing costs and procurement. The traditional approach prioritizes capital expenditure (CapEx), whereas cloud economics favors operating expenses (OpEx).

Innovation and digital transformation can be accelerated in the cloud, taking advantage of the pay-as-you-go model, with no upfront payment and faster time to market. A wide range and virtually unlimited resources and services are available on demand, and IT spending fluctuates based on consumption. Now development teams and engineers can spin up IT resources by themselves, just with a click of a button. The FinOps framework is helping organizations to obtain the best ROI for their cloud transformation. New cool technologies like GPU are enabling more use cases, especially in the Genearative AI (GenAI) and machine learning areas, and the cloud can easily provide those resources on an on-demand basis.

Why FinOps?Cost management is pointed out as the main challenge companies are facing during their cloud journey. According to the Flexera 2023 survey1, 82 percent of enterprises and SMBs pointed to cost management as the main challenge they are trying to overcome. As cloud security is not a big concern anymore and it is more mature (pointed out by 79 percent of all enterprises as a challenge), the market has seen an increase in the adoption of cloud services in the last few years. For the “as a service” model, the cloud offering is more consolidated, mature, and easy to adopt, which opens up the door for more migration to the cloud and consequently, more cloud infrastructure and platform consumption. It is not strange to hear and know about companies adopting a cloud-first strategy in the short term. According to Gartner Hyper Cycle2, Cloud computing solutions are situated on the plateau of productivity, so more and more companies are starting that journey.

FinOps3—Cloud Financial Operations—is a framework that allows companies to take the most of their investments in the cloud; in other words, maximize ROI. By promoting cross-functional integration and communication, different areas and departments contribute to keeping costs aligned with the business objectives. Technology, finance, and operations collaborate to bring financial accountability to cloud spend. But FinOps is not only about cost management and control. It also improves the procurement process with cloud providers, negotiating better discount rates and plans, maximizing the cloud investment.

Cultural ShiftAs organizations seek agility and innovation, cloud computing is the most appropriate way to prototype and launch new products and services, reducing the time to market. That same agility generates inefficiencies and loss of governance, leading to under-estimated cloud requirements, most of the cases resulting in the waste of cloud resources. The traditional “lift and shift” approach is not appropriate when building cost-effective cloud architectures. According to the latest Oomnitza report4, more than half of the surveyed companies recognize they have wasted 10 percent or more of their cloud budget on underutilized, unmanaged, and unattended cloud resources, which reflects the need for a discipline to manage the cloud and integrate financial, operation, and engineering teams.

FinOps—a project part of the Linux Foundation—breaks down the walls that separate technology, business, and finance to improve the cloud unit economics. It’s not just a set of procedures or a piece of software to monitor cost and trigger alerts. It requires a profound cultural change in how to manage and coordinate IT resources, business needs, procurement, and finance to take advantage of the variable cloud model. Transparency is one of the main pillars allowing visibility on how, when, and who uses the cloud resources. The idea is not to spot overspending and find culprits, but instead to improve the usage and investments. Real-time consumption data and reports will allow the teams to make informed decisions on the current spending and also improve financial forecasting and planning. All that combined will improve financial awareness and accountability for cloud transformation.

Cloudera FinOps CapabilitiesCDP is a cloud-native platform helping companies accelerate cloud adoption to run their data and analytics workloads. Replication Manager, Observability, and Data Catalog are examples of tools that are part of the Control Plane suite, helping companies to leverage the cloud as their primary infrastructure or as an extension of their data centers for data analytics initiatives and projects.

Besides the technical aspects of moving to the cloud, Cloudera provides tools and features to make sure the workloads are running properly and making the most of the cloud infrastructure. FinOps, as a cloud management discipline, proposes a journey consisting of three phases to help organizations improve their cloud investments. This is how unique Cloudera capabilities are helping customers in the FinOps journey and avoiding cloud overruns.

InformVisibility and accountability for workload spending.

This is the first phase of the FinOps process. It is about training all stakeholders with the information and knowledge necessary to make informed decisions about using the cloud. It is about empowering business organizations with better visibility, allocation, benchmarking, and budgeting. Proper time-accurate visibility of the cloud is necessary to make intelligent decisions due to customized pricing, discounts, and the on-demand and elastic nature of the cloud. A team’s productivity can be optimized through accurate features of benchmarking.

Consumption ReportCDP Public Cloud gives visibility to all stakeholders of Cloudera services executing. The Consumption Report offers an aggregated view of how the related credits are being used, but also administrators to drill down by services, date/time, and environment.

Resource taggingCDP Public Cloud allows administrators to easily add tags to the Data Service and resources the platform deploys on the company’s cloud tenant. Afterward, those tags are also used to track resource usage, assign usage to cost centers/departments, and trigger automation policies. Tags can be defined at tenant, environment, and service level, which adapts to different needs and strategies, and at the same time is propagated to the CSP infrastructure.

OptimizeReduce cloud spending and increase cloud efficiency.

Optimize the next phase of the FinOps process and focus on discovering cost-saving opportunities. In what areas of your organization can you adjust resources and take advantage of the benefits of discounts based on current use? After the businesses are empowered, the natural stage is to optimize their cloud footprint. For that, cloud service providers offer several resources, including reserved capacity—also known as Saving Plans—which can improve the ROI of cloud investments.

Cloudera ObservabilityCloudera Observability is an observability tool that continuously discovers and collects performance telemetry across applications and infrastructure components running in CDP environments. It performs real-time correlations that indicate existing and potential problems and includes prescriptive guidance to address them. Platform administrators are using these insights and recommendations to optimize job performance and execution, which has a direct impact on cloud infrastructure usage.

ChargebackThe Cloudera Observability Financial Governance Chargeback feature collects CPU, memory, and resource usage data from your environment, allocates those charges to your custom cost centers, and visually displays the results. It provides in-depth visibility into the workload resource costs of your environment’s infrastructure that can be used for planning, budgeting, forecasting, and optimization.

OperateContinuously evaluate and automate cloud financial management practices.

Operating is the final phase of the FinOps process, as organizations continually evaluate their performance concerning business objectives and look for ways to improve their FinOps practices. After defining optimization efforts, automation allows organizations to implement policies that will continually adjust cloud resources to control costs without impacting performance. It is essential to analyze business goals and the steps taken to achieve them.

Compute resource allocationCloudera Data Services includes different configurations and parameters to make sure compute allocation dynamically scales up and down accordingly to achieve the right performance/cost balance. Platform administrators can set the right amount of compute resources to meet workload needs, alongside auto-scale threshold, making sure to allocate resources when they’re needed, helping to control and manage cloud spending.

Spot instancesRecently Cloudera added the support for spot instances to facilitate cloud cost savings for workloads that are not SLA-bound. Spot instances are infrastructure provided with high disocunts—80 to 90 percent—that can be reclaimed at any time by the cloud provider. For fault-tolerant workloads such as Apache Spark, utilizing spot instances can provide significant cost savings. Because spot instances can be requested at any time by the CSP (impacting job performance) Cloudera recommends only using them for workloads without strict SLA requirements.

Cloudera <3 FinOpsThe features listed above are a partial list of the FinOps capabilities CDP Public Cloud provides. Cloudera is constantly innovating on new functional and integration features and adding and expanding FinOps capabilities to ensure companies have the best ROI for their cloud investments. Besides the out-of-box features, our Cloud Solutions Architect and Customer Success teams proactively guide and assess customers through the journey to provide a safe and cost-effective flight to the cloud.

Cloudera FinOps capabilities span all deployment form factors of CDP cloud-native platform, helping companies to consistently address the technical and financial challenges of moving to cloud.

Now data teams can decide where and when to execute their data analytics workloads, optimizing cost and taking advantage of each infrastructure option. Predictable and persistent workloads can still run on-premise and improve the investments made on the traditional hardware—bare metal and virtualization. On the contrary, seasonal and transient workloads can be easily moved to the public cloud infrastructure. Data is also driving the decisions of where and when to deploy critical workloads seemingly in a hybrid cloud.

Next StepsLearn more about our brand-new Observability tool that is revolutionizing the way companies take advantage of Lakehouses and help to be FinOps friendly:

Cloudera Observability product page

Blog: Beyond Monitoring: Introducing Cloudera Observability

References:

1- https://info.flexera.com/CM-REPORT-State-of-the-Cloud-2023-Thanks?revisit#challenges

2 – https://www.gartner.com/en/newsroom/press-releases/2022-08-04-cloud-platform-hc-press-release

3 – https://www.finops.org/

4 – https://www.oomnitza.com/resources/2023-snapshot-survey-saas-cloud-spend-optimization-and-automation/

The post Cloud Analytics Powered by FinOps appeared first on Cloudera Blog.

View Details

In the bustling world of technology and corporate dynamics, finding a workplace that truly values employee well-being and fosters growth is like finding a needle in a haystack. But Prashant Bajpai, a staff software engineer in Bangalore, says he found that needle. In fact, what he found was so much more.

Prashant shares that working at Cloudera is like having a basket of fruit that includes everything you could want. “Cloudera to me is like a fruit basket. It not only takes care of my health by providing an incredible work-life balance, it also keeps me motivated to reach new career heights by cultivating a healthy work environment. What more could you ask for?”

An exciting work environment within open sourceIt’s no secret that we’re a stimulating open source company. What might not be well known externally is how our employees feel about their contributions. Prashant says, “Working at Cloudera, you’re constantly being motivated and encouraged to add value not only to the products you support but to the open source community at large. This to me is a remarkable opportunity that advances my personal and professional development.”

He feels that so much of this is credited to his leadership, specifically Vinod Kumar Vavilapalli, Jayush Luniya, Rajesh Kumar Mishra, who strive to build and maintain a workplace that allows for learning, growth, and celebration.

“Every day, there is open dialog that encourages new ideas and allows for healthy debates on best ways to move the needle forward. We have all-hands meetings that keep everyone in alignment with larger organizational goals and short-term goals of the team. We also are in constant communication with managers, which helps keep us all focused.”

To add to that, Prashant feels that a large part of what makes such a rewarding environment is the celebration of product releases to acknowledge employees. “Our culture recognizes employees’ contributions and celebrates the work that’s been done for every product release.”

Not all work and no play Burnout seems like the price to pay for working within such a fast-paced technical environment, but Prashant shared that burnout is easily fended off. “We’re provided with many regular holidays, flexible PTO, and “Unplugged Days,” where I can unplug from work and spend time with family, travel, go mountaineering, or simply relax at home. This has helped prevent burnout and keep me motivated in my daily work.”

Cloudera is not just a workplace for Prashant, it’s an endless list of opportunities that make up his metaphorical fruit basket. Our unwavering commitment to employee well-being, open source contributions, and a healthy work environment are what represent the fruit he needs to nurture both his personal and professional growth. In one year’s time at Cloudera, Prashant is confident sharing that his life has been more fruitful since joining.

Are you interested in a more fruitful life? Check out the open roles and apply today!

The post Connecting with Clouderans appeared first on Cloudera Blog.

View Details

We’ve all heard that data helps businesses make better decisions. The good news? This isn’t just speculation: research shows that companies who use data to drive decision making increase revenues by an average of more than 8%, are 23 times more likely to attract new customers, and are 19 times more likely to be profitable as a result.

Most modern companies have access to rich and varied data sources embedded into their daily operations. Many of those data sources may be accessible to employees throughout the organization—not just tech and data specialists. But unless staff at every level grasp the power of data and have the skills to wield it properly, it becomes a wasted resource. That’s why organizations should focus on creating a culture of data.

What can a data-driven culture help organizations accomplish?

Maximize the impact of every employeeEach employee has a unique perspective, and every individual within the company serves a valuable role in supporting corporate goals. While a social media coordinator might glean valuable insights into consumer trends on a daily basis, an office manager might uncover opportunities for cost savings while analyzing purchasing data.

By empowering employees at all levels to understand data, to integrate it into their workflows, and to make data-driven decisions, companies enable individual contributors to drive impact that senior leaders and middle management can’t see, and that’s exciting.

Trounce the competitionData and predictive analytical tools are widespread and easily accessible: do you know how your competitors are using them? Data-driven insights help businesses get the jump on their competition, enabling them to predict trends in customer sentiment, build superior forecasting models, and expose market opportunities for competitive advantage.

Why not spend a Friday afternoon researching the business use cases that companies in your industry are solving with data and analytics technology? Are you more or less data-driven than they are?

If you’ve got ground to make up, begin by democratizing data. Make unrestricted data available far and wide but govern it. That’s very important. The enterprise needs to be singing from the same hymnal. Often that requires a centralized data engineering unit who manages data for everyone. With architectures like data mesh, that may change in the future.

Future-proof the organization Agile companies are successful companies. When you limit data and data tools to certain departments, you limit organizational outcomes. By continuously investing in employees to upskill, increase data literacy, and embrace new technologies and perspectives, businesses not only strengthen the capability and power of their teams, they insulate themselves from the global shortage of technical resources.

With new technologies and tools hitting the market at a rapid pace, embracing the power and potential of data helps future-proof their workforce and keeps companies looking forward.

So how can organizations create a data-driven company culture and ensure they’re making the most of the tools at hand?

Unify the entire enterprise Data-driven companies unify the entire enterprise around data. Here’s how. The C-suite evangelizes corporate objectives: everyone knows what they’re fighting for. Middle managers reinforce and decompose them into function-specific objectives: individual contributors understand how they make a difference. Everyone is aligned with a common purpose. When employees see how their work contributes to top line/bottom line growth and customer experience, they ooze pride. Ooze! Then they double-down on loyalty.

Elevate data literacyAligning to corporate objectives is just the beginning.

“When people begin to believe in the data, it’s a game changer,” said Ted Colbert, executive vice president and CEO, defense, of Boeing. “They begin to change their behaviors, based on a new understanding of all the richness trapped beneath the surface of our systems and processes.”

How can an organization help bring its employees on board? As early as in a 2018 McKinsey report on the importance of data culture, industry leaders outlined the importance of identifying “data translators” within an organization.

A data translator is someone who can “bridge both worlds—data science and on-the-ground operations.” They already work with staff on a day-to-day basis, and are a trusted partner. A good data translator will convey the importance of a data-driven mindset with lower-level and non-technical staff, and help train them on how to use the tools at their disposal to achieve results.

Data is here to stayData is existential to organizations today. Tomorrow it will be more. Those who embrace a data-driven culture and empower their staff at all levels to make impactful data-driven decisions will be well-positioned. Those that don’t will get left behind.

Interested to dig deeper? Watch this on-demand webinar to learn more about the foundations of a data-driven organization.

The post Think Like a Data Scientist: The Importance of Building a Data-Driven Company Culture appeared first on Cloudera Blog.

View Details

As Hispanic Heritage Month draws to a close, we wanted to conclude the celebration with an employee spotlight featuring the new lead for the Cloudera Latin X, Employee Resource Group (ERG), Joel Martinez. We talked with Joel about his career in sales, growing up in High Point, North Carolina, and his continued rediscovery of his Hispanic heritage with his move to Austin, Texas where he currently resides.

Finding his career fit early in the process.

Joel joined Cloudera two years ago as an inside sales representative (ISR) out of the Austin office. His tasks included sales activities such as prospecting, business development, and demand generation for new and existing accounts.

During his tenure, Joel’s role has expanded. Now, Joel has account executive responsibilities in the new business development organization, comprised of new logo teams in the central and western US. He is responsible for covering 2,000+ accounts, following up with leads, and creating sales motions to get introductions with key decision-makers.

Joel started his career after his freshman year in college in his hometown of High Point, North Carolina. To earn some extra money Joel started working at Vector Marketing, best known as a direct sales company that sells kitchen supplies, most notably its line of cutlery (Cutco). “I learned how to pitch the product, upsell, and service previous customers,” he said. This ended up evolving into a full-time role as he advanced to manage his own team.

After six years with Vector, Joel wanted to explore the tech sector. He brought his sales and management experience to a small startup in Durham that specialized in a niche compliance software to facilitate public records associated with the federal government’s Freedom of Information Act (FOIA) requirements. After gaining some experience in the tech sector, a friend referred Joel to an ISR role at Cloudera and the rest is history.

“What I like about working at Cloudera is I’m involved in solving big challenges across industries,” he said. “The companies we enable with our technology literally help save lives across healthcare, biotech, and pharmaceuticals, as well as support industries that keep the world going, like utilities, transportation, and banking. It’s pretty cool to be part of that. As a company, we can have such a direct impact on a lot of people globally and for me, that is really motivating.”

Rediscovering his Hispanic heritage, sharing with others, and helping the community

While discovering his passion for sales during his college years he was also rediscovering his ethnic background through his continued study of Spanish literature. Joel is of Mexican and El Salvadorian descent, but growing up he didn’t speak Spanish much at home.

“Growing up, I sometimes was made to feel different. As a result, I didn’t want to speak Spanish in the house or embrace my heritage—I wanted to fit in. Eventually, I realized I was missing out on being able to communicate with my parents effectively because I was losing my Spanish. That inspired me to learn everything I could about the language and culture, not just Mexican and Salvadorian, but all the cultures that make up the Hispanic people and nations.”

Joel’s desire to embrace his heritage and incorporate it more substantially prompted him to step into the role of Latin X, Employee Resource Group (ERG) lead at Cloudera. “I was already involved in Big Brothers/Big Sisters here in Austin and familiar with “Latinitas,” an Austin-based organization designed to help Hispanic girls get into STEM that Cloudera has supported in the past. So I thought the ERG would be a good point of collaboration.”

Advice for early career sales professionals

In reflecting on his career so far, Joel identified one thing to keep in mind at any stage in one’s sales career. “I learned this mindset from one of my first managers: Understand that your success in sales is a direct result of how much you learn and therefore the more you know about your product, the sales process of an organization, etc. can directly affect your compensation. So while taking action is vital to a successful career in sales, devoting time to regular practice and coaching is an investment in your future.”

He adds, “For example, I always work to understand sales processes and sales motions inside and out, thoroughly studying the products and technologies, how the partner ecosystem works, and the competitive landscape. In addition, I am currently studying for an Amazon Web Services cloud practitioner certificate because I know it will help me better understand the space and, in turn, better serve our customers.”

Living in Austin and the road ahead

Joel was excited to join the Cloudera Austin office and to try out living in a bigger city. Joel says, “Prior to moving, I only visited Austin once but I loved the city, so I was like, let’s give it a try!”

He adds, “It’s a super active city and that is my favorite thing about it. There’s just so much to do—the river and trails are awesome. I’m always walking or running and I just got a mountain bike. There are a lot of social events as well, including some great places to dance Salsa and Bachata.”

“Looking ahead in my career I’d like to move up to a manager role so I can help scale the business development organization and start mentoring others as I did in my previous business,” he said. “I got a lot of satisfaction out of helping others grow in their careers. I see how data analytics is changing the world and I’m happy to be part of it and create an organization where others can be a part of it too.”

We thank you Joel for contributing to Cloudera and sharing your story!

The post #Cloudera Life Employee Spotlight: Joel Martinez appeared first on Cloudera Blog.

View Details

The world faces multiple environmental sustainability challenges — from the climate crisis and water scarcity to food production and urban resilience. Overcoming these hurdles offers opportunities for innovation through technology and artificial intelligence.

That’s why Cloudera and AMD have partnered to host the Climate and Sustainability Hackathon. The event invites individuals or teams of data scientists to develop an end-to-end machine learning project focused on solving one of the many environmental sustainability challenges facing the world today.

Participants will be given access to Cloudera Machine Learning running on AMD hardware to enable swift, powerful computations and breakthrough innovations — a pairing that will help data scientists craft climate and sustainability solutions. At the completion of this hackathon, every line of code from the winning prototypes will be made public so that the event can contribute to the collective effort to address the climate crisis and other pressing environmental sustainability challenges.

This isn’t your ordinary hackathon — it’s meant to yield real, actionable climate solutions powered by machine learning. Participants can choose from the following categories for their prototype:

• Climate Smart Agriculture: With the world’s population expected to hit nearly 10 billion by 2050, finding sustainable ways to feed all of these people is critical for addressing global hunger as well as mitigating the climate crisis. Climate-smart agriculture (CSA) is an integrated approach to managing landscapes — cropland, livestock, forests and fisheries — that address the interlinked challenges of food security and climate change. Machine learning (ML) has the potential to advance climate-smart agriculture by providing valuable insights, predictions, and decision support to farmers, researchers, and policymakers. This includes climate modeling and prediction, crop yield prediction, pest and disease detection, irrigation management, precision agriculture, soil health assessment, crop selection and rotation, carbon sequestration, supply chain optimization, decision support systems, climate adaptation strategies, and data-driven research.
• The Water Crisis: While water is something many take for granted, its scarcity is becoming one of the most pressing sustainability challenges for businesses, governments, communities, and individuals around the world. Besides being fundamental to sustaining life, water also is integral for agriculture, manufacturing, and industrial processes. The climate crisis is a water crisis, too. As the planet warms, this leads to increased evaporation, changing and unpredictable precipitation patterns, rising sea levels, and melting snow pack and glaciers, among other challenges. Addressing water scarcity is becoming a critical issue. Possible projects include forecasting water consumption based on historical data, weather data, and population growth; using satellite imagery to detect changes in the environment that might indicate underground leaks in large pipelines; or predicting the amount of rainwater that can be harvested in specific regions based on weather forecasts and historical data to aid in designing effective rainwater harvesting systems.
• Sustainable Cities: Cities are responsible for 70 percent of global greenhouse gas emissions. That means that the climate crisis will be won or lost in our urban environments. Many of these emissions are driven by industrial and transportation systems reliant on fossil fuels. But machine learning and big data offer promise for developing the smart cities of tomorrow. By improving efficiencies and enabling better decision-making, we can address the sustainability challenges afflicting cities around the world. Possible projects include air quality prediction and monitoring, Predicting energy demand in different parts of the city to optimize electricity distribution, or using imagery to classify waste types for more efficient recycling processes.

For this Hackathon, participants will be tasked with using publicly available datasets (suggestions for each theme are provided) to create their own unique Applied ML Prototype (AMP) focused on solving or gaining further insight into a climate or sustainability challenge. Cloudera’s Applied Machine Learning Prototypes are fully built end-to-end data science projects that can be deployed with a single click directly from Cloudera Machine Learning, or accessed and built yourself via public GitHub repositories..

The climate crisis won’t wait — we hope you’ll join us in using the power of data science and machine learning to help address it once and for all. Learn more about how you can participate in the hackathon here.

The post Cloudera and AMD Spur Data Scientists to Take Climate Action appeared first on Cloudera Blog.

View Details

The AI technologies of today—including not just large language models (LLMs) but also deep learning, reinforcement learning, and natural-language processing (NLP) tools—will equip telcos with powerful new automation and analytics capabilities.

AI-powered automation is already driving significant margin growth by reducing costs. But to truly drive transformation telcos must ensure AI models are driven by accurate, high-quality, trusted data, and determine how to manage and govern massive volume at scale. And not just in ad hoc instances in pockets of the organization, but as a part of the infrastructure of the business as a whole. This is the essence of “Trusted AI Everywhere.”

“Trusted AI Everywhere” explainedTrusted AI poses significant challenges. One notable example is the tendency of LLMs to produce hallucinations—i.e., outputs that read smoothly and seem plausible, but are unfounded or nonsensical. Embedded biases, whether explicit or hidden, can also perpetuate harmful outcomes. Other challenges include the lack of transparency and explainability in AI systems and the need for continuous monitoring and inherent observability to maintain their effectiveness. Addressing these and other challenges is a precondition for trusted AI.

To do so, open source AI is particularly primed to spearhead the AI revolution. Not just because it’s rapidly closing the feature-and-function gap with commercial/proprietary solutions, but because it’s inherently transparent and adaptable. Open source’s benefits, proven in the realm of enterprise software, resonate even more in the context of AI. The ability to customize and scrutinize source code helps ensure trust and security, and the benefits of open source’s collaborative governance model help mitigate commercial AI’s “black box” problem.

To that end, “Trusted AI Everywhere” marries the ethos of trusted AI with the insight that AI’s maximum impact comes when it’s seamlessly integrated across a telco’s entire enterprise. This isn’t about isolated pockets of trustworthy AI, like chatbots in the contact center; it’s about ensuring pervasive trustworthiness, reliability, and observability. These concepts of observability and explainability are crucial not only because they make it easier to diagnose and resolve issues, but also because they contribute to our understanding of the behavior of AI solutions—whether they’re applied to network optimization, customer service, data analytics, or other use cases.

“Trusted AI Everywhere” encompasses three primary aspectsFirst, it involves the use of Ggenerative AI and LLMs, along with other AI technologies, to empathetically interact with users—customers, employees, and partners—improving the quality of interactions. By leveraging AI-powered sentiment analysis and affective computing, telcos can transform the interaction experience, promoting increased engagement, improving the efficacy of marketing and operations, and enabling enhanced decision-making.

Moreover, this first aspect of “Trusted AI Everywhere” extends beyond customer service or marketing. By promoting a question-and-answer driven interactive experience—and by synthesizing, contextualizing, and surfacing insights derived from an enormous amount of information—AI solutions can transform human decision-making, leading to better, more responsive decisions and actions.

Second, trusted data forms the bedrock of trusted AI, as AI models are only as good as the quality of their underlying data platform. Eliminating inconsistencies, errors, and redundancies is pivotal, as is understanding the lineage of data. Finally, the data sets used to drive AI models must be diverse, complete, unbiased, and representative of the problem space for which the model was designed. In a sense, these are classic data management problems; given the scale and complexity of AI development, however, they’re considerably more difficult to address. In addition, AI development poses significant challenges to data governance, especially with respect to explainability, regulatory compliance, security, and privacy.

The third and final component is omnipresence—the “Everywhere” component of “Trusted AI Everywhere.” The potential of AI is best realized when it is embedded across a telco’s business processes, not only as a means to improve or optimize these processes but in order to ensure observability into them. Customer service is one obvious application for embedded AI, which can provide personalized, efficient, and round-the-clock customer engagement. AI can also play a foundational role in helping telcos optimize their networks and operations, with observability enabling telcos to more quickly and reliably detect and pinpoint network performance problems, developing automated AI solutions that support both proactive health monitoring and the autonomous rectification of issues. Autonomous networks continue to be a goal for the most advanced telco operations. The same is true of supply chain management and logistics, human resources, finance, product development, and other essential business processes. Business partner interactions—distributors, resellers, roaming partners, and content providers—can similarly be driven by automated systems.

Another dimension of “Everywhere” is that telcos must deploy AI from the network edge to their core businesses. In addition to embedding AI to support back office (billing and payment processing, network operations, etc.) and front office (customer service, sales and marketing, etc.) functions, this might take the form of using AI to automate predictive maintenance for edge devices, like RAN base stations and towers or WAN endpoints. It could involve optimizing the way the fleet is deployed or developing an ability to dynamically schedule which routes they take. It might entail leveraging AI to improve the availability, performance, and security of core network infrastructure, e.g., supporting dynamic traffic prediction and load balancing across network technologies, adaptive network configuration, fault prediction and avoidance, and power consumption.

Conclusion“Trusted AI Everywhere” inaugurates a paradigm shift in the telco space, focusing on integrating AI seamlessly across all telco operations. Key pillars of this change are:

1. A unified data infrastructure ensures access to quality data, irrespective of its location, be it on-premises or various cloud services. This equips telcos to capitalize on AI’s insights.
2. A preference for transparent, open-source AI over proprietary systems. Open-source solutions offer trust and explainability, essential for decision-making and reducing AI adoption risks.
3. Pervasive AI integration up, down, and across a telco’s business operations.

Learn more about how Cloudera helps Telcos deliver Trusted AI Everywhere.

The post Transforming Telco with Trusted AI Everywhere appeared first on Cloudera Blog.

View Details

AI is the next revolutionary technology that will accelerate the mission of the Department of Defense. Newly boundless in its applications, “AI” joins “cyber” and “cloud” as the most important information technologies that have arrived in the last 25 years. In July 2023, the Department of Defense (DoD) marked the one-year anniversary of the Chief Digital and Artificial Intelligence Office (CDAO), which brought together the DoD Chief Data Officer (CDO), Joint Artificial Intelligence Center (JAIC), Defense Digital Service (DDS), and Advancing Analytics (ADVANA) Office.

The CDAO was tasked to shape AI policies and data strategies for the DoD and its related agencies. Notable achievements for the year can be found here, including the identification of a Digital Hierarchy of Needs, which highlighted “four areas necessary to accelerate and scale data, analytics, and AI/ML adoption in support of DoD priorities,” a prescient exercise for what was to come.

When OpenAI released ChatGPT as a part of a free research preview in November of 2022, no one could have predicted it would become the fastest-growing web platform in history. This single event ushered in the generative AI revolution that has affected industries across the public sector, including the DoD.

Task Force LimaIn response, on August 10, 2023, under the direction of Deputy Secretary of Defense Dr. Kathleen Hicks, the CDAO launched Task Force Lima, a specialized team comprised of partnerships across the DoD, the intelligence community, research professionals, and other agencies dedicated to minimizing redundancy while exploring generative artificial intelligence technologies (and their impacts to mission) such as large language models (LLMs). With a team of AI experts, data scientists, and engineers, the unit is set to become a hub for research, risk assessment, development, and implementation of AI systems in defense operations.

Exploring opportunities while highlighting risksWhen trained on the vast stores of government data, generative AI has the potential to provide myriad benefits to the DoD in operational areas “such as warfighting, cybersecurity, business affairs, supply chain management, health, readiness, and policy” including:

• IT Workforce Augmentation: Generative AI has the potential to augment the IT workforce at the DoD in several ways, enhancing efficiency and productivity through automating mundane/routine IT tasks such as software patching, malware analysis, system monitoring, and data entry. This augmentation enables the workforce to focus on more complex tasks and accelerate productivity.
• Cybersecurity: Generative AI has the potential to significantly boost cybersecurity by enhancing threat detection and response capabilities. Through the automated analysis of vast data sets and historical attack patterns, generative models can screen for and predict and simulate potential cyber threats to enable cyber warriors to focus on more complex tasks and enhance defense capabilities.
• Planning and Training: AI algorithms can help in optimizing mission planning by considering/integrating factors such as terrain, weather, enemy capabilities, and logistical constraints in real time while suggesting optimal routes, tactics, and strategies for military operations, thus improving outcomes for mission planning.
• Medical and Health Applications: Generative AI models can assist in analyzing medical images such as X-rays, MRIs, and CT scans as well as enhancing the quality of medical images, making it easier for providers to use AI to optimize, identify, and diagnose health issues. It has already been demonstrated that the use of AI has improved the effectiveness of medical image scanning by over 20 percent.

It’s all about data…and its trustworthinessAs I mentioned above, generative AI can present significant opportunities but is not without challenges. Regarding the challenges the task force must tackle, the most crucial is determining the data associated with the selected challenge and its attendant security. The risk of accidental or intentional data leakage or misuse can be a significant challenge, which means the DoD must meet or exceed the various regulations and legal frameworks governing data use, privacy, and security through rigorous identity management.

Coupled with that is the new AI-influenced threat landscape. Generative AI models are vulnerable to adversarial attacks, where malicious actors can manipulate input data to generate misleading or harmful outputs. In defense applications, this will lead to the creation of deceptive content or misinformation.

Verification and authenticity are concerns as generative AI can produce incredibly realistic and convincing text, images, and videos. This poses challenges in verifying the authenticity of content, making it difficult to distinguish between AI-generated and human-generated information.

Managing data to support AI initiatives Solving these data challenges requires robust data management capabilities that enable the DoD to efficiently collect, store, and analyze vast amounts of data while ensuring the secure and compliant handling of sensitive information.

With Cloudera, a world-class leader in Open Data Lakehouse for Trusted AI, the DoD can harness the power of generative AI to improve mission planning, intelligence analysis, and cybersecurity, ultimately enhancing national security efforts through cutting-edge technology solutions.

The post DoD Launches Task Force Lima to Explore Generative AI appeared first on Cloudera Blog.

View Details

There are some who believe that growing in your professional career and a desire to travel the world don’t mix well. I am not one of those people – in fact, I’m proof that these two ambitions can blend together to create a beautiful life.

I’m Kinga Kamaras. My title at Cloudera is a Strategic Customer Success Manager. It’s a role I enjoy and growing in my career is a big ambition of mine. But I’m more than just an employee.

A company culture that values work-life balanceIf you were to ask me what the greatest asset in life is, I would say “Time”. Once it’s gone, it’s gone forever. You never get time back. I believe that time is the biggest gift you can give someone and get in return. What’s exciting is that it’s a gift Cloudera provides by creating a company culture that values real work-life balance.

The way I see it, you work hard, you play hard and repeat the process. When I work, I am 100% dedicated to what I do, but when I’m clocked out, I’m a passionate traveler with a long bucket list of worldwide destinations and adventures that await.

Since joining Cloudera, I’ve been fortunate enough to visit 5 continents and some exotic locations like Zanzibar, Dominican Republic and Malaysia!

Plugging into the #ClouderaLife

For those that don’t know, you might be asking, “How are you able to get the time to visit so many locations?” The answer, Unplugged Days.

Unplugged Days are company-wide extended weekends or even full weeks off that are meant to allow employees time to unplug from work and plug into other parts of life. For me, that means traveling!

These days have been so valuable to me. During my time off, I refresh my mind by prioritizing self care on my travels. Sometimes, I push myself to get out of my comfort zone and engage with new people and cultures I’m completely unfamiliar with. Also, Unplugged Days offer me the ability to connect with friends and spend quality time with my loved ones. Having the time to travel, decompress and connect with family and friends has brought me so much joy. I can confidentiality say that I am a Clouderan that truly loves living the #ClouderaLife.

Components of a happy employee One thing I can’t forget to mention is that Cloudera is filled with great teammates that have helped me grow, keep me motivated and on track to reach my larger career ambitions. But I believe that proper work-life balance is key to creating a fully satisfied and engaged employee. I’ve found that balance at Cloudera and I couldn’t be more thankful.

If pursuing your passions and work life balance are important to you, check out what opportunities Cloudera has for you! Join the #ClouderaLife.

The post Connecting with Clouderans appeared first on Cloudera Blog.

View Details

Telecommunications companies are currently executing on ambitious digital transformation, network transformation, and AI-driven automation efforts.

While navigating so many simultaneous data-dependent transformations, they must balance the need to level up their data management practices—accelerating the rate at which they ingest, manage, prepare, and analyze data—with that of governing this data.

To do this, telcos must reimagine their approach to data architecture: transitioning from legacy, siloed data architectures to a modern data architecture—anchored by a data platform able to integrate data across on-premises and cloud environments, and the network edge.

The Opportunity of 5GFor telcos, the shift to 5G poses a set of related challenges and opportunities.

Large 5G networks will host tens of millions of connected devices (somewhere in the 1,000x capacity compared to 4G), each instrumented to generate telemetry data, giving telcos the ability to model and simulate operations at a level of detail previously impossible.

This exponential growth in connected devices will force telcos to up their game, first by provisioning the capacity they need to scale and maintain next-gen 5G data networks, and later by improving the effectiveness of their data management and governance practices.

But predictive modeling and machine learning (ML) will enable rapid extraction of meaningful insights from this data, gleaning information about customer preferences, behavioral patterns, and needs, making it possible to transform business operations and services, radically personalize the customer experience, and develop new products and services that just weren’t feasible with 4G.

The consolidation waveAnother consideration is a likely wave of consolidation driven by the desire among larger telcos to distribute the burden of technological investment, leverage economies of scale, gain competitive advantages in existing markets, or expand into new markets.

Consolidation presents perhaps the biggest overall challenge, not only with respect to the complexity of integrating dissimilar IT systems and data platforms, but also that of merging and reconciling business processes and operations. Add to this, too, the difficulty of integrating potentially dissimilar compliance frameworks: for example, separate telcos might be operating under different regulatory guidelines, appropriate to specific jurisdictions or business practices, requiring the merged entity to formalize a single, unified framework for compliance.

These transformations require a major rethinking of data architectureThe onus is on telcos to revamp their data architectures so they can collect, process, and analyze data at or close to real time—i.e., at the network edge—to accommodate the lower-latencies and larger volumes of data in the 5G era and beyond, as well as to make it easier to integrate systems, data, and processes in the event of merger and acquisition (M&A) scenarios. This has a few implications for next-gen data platform architecture:

First, streaming data presents a novel set of data management and governance challenges, requiring a data architecture that’s suitable for low-latency, high-velocity data processing.

Second, telcos must be able to “push out” data processing so it takes place closer to the connected devices that generate telemetry data, reducing data latency and minimizing traffic. This means devising ways to process data at the network edge, as well as making decisions about which data to persist for historical analysis—and which to discard.

Third, telcos must adopt a hybrid data platform capable of spanning the cloud, on-premises, and edge environments. They will need the elastic capacity of the cloud to accommodate the continuous, high-volume data flows generated by 5G devices; the massive volumes of historical data used to feed operational analytics and support long-term planning; and the large, multivariate data sets used to train ML models.

Fourth, by unifying control and visibility across the on-premises, cloud, and edge environments, a hybrid data platform makes it easier for telcos to navigate disruptive changes, like M&A scenarios. By automating data management tasks and supporting a wide variety of access protocols, it accelerates the work of integrating dissimilar systems and processes. And by building in identity and access management (IAM), role-based access control (RBAC), and data governance capabilities, it helps simplify M&A consolidation projects.

Integrating these capabilities into a data platform gives telcos the flexibility to navigate changing conditions while enforcing data security, compliance with regulations, and delivering novel products.

Scaling data engineeringIn the telco world, the scale of data engineering has always been constrained by factors like the shortage of skilled data engineers and the limitations of legacy platforms and tools.

A hybrid data platform breaks down this barrier, integrating ML- and AI-based tools that make it easier to manage, integrate, and analyze data, as well as monitor governance and compliance.

It also makes data professionals more productive, providing a rich set of ease-of-use features and exposing a variety of interfaces—like RESTful APIs, query interfaces, and language-specific bindings—they can invoke using their preferred tools. It incorporates features that make it easier to build, test, and deploy data pipelines, as well as schedule and monitor them in production. In addition, it automatically manages dependencies between tasks, keeping track of a task’s progress and ensuring that it completes successfully before triggering any dependent tasks. For data engineers, data scientists, and other experts, a hybrid data platform simplifies access to distributed data, enabling them to design reliable, idempotent, low-latency data pipelines that integrate real-time data from the network edge to feed operational analytics, or ML-powered, AI-automated applications and services.

A hybrid data platform that’s as close to turnkey As possibleNo combination of point solutions or open-source software (OSS) adds up to a turnkey hybrid data platform, especially when taking into account the challenge of integrating new OSS technologies with legacy telco systems.

However, Cloudera Data Platform (CDP) is a best-in-class platform that is 100 percent compliant with upstream OSS projects. CDP is the foundation of Cloudera’s Universal Data Distribution (UDD) vision, which describes a data architecture capable of spanning the on-premises, cloud, and edge environments that breaks down legacy silos and enables transparency and interoperability across distributed environments.

Cloudera DataFlow, one of CDP’s integral components, handles both batch and streaming data, ensuring reliable, “right-time” access to information. CDP includes built-in support for advanced security features like IAM and RBAC, which facilitate secure access to data while safeguarding privacy. CDP automatically enforces compliance policies, continuously monitoring and reporting on data access, changes, and movements. And by automating compliance enforcement, telcos reduce the risk of human error and adhere to regulatory requirements while minimizing manual effort.

And by selecting a best-in-class platform like CDP, they effectively outsource the daunting task of building and maintaining a bespoke hybrid data platform from scratch.

Download the e-book A Hybrid Data Cloud for Accelerated Insight and learn more about the benefits of a hybrid data platform.

The post Four Ways Telcos Can Realize Data-Driven Transformation appeared first on Cloudera Blog.

View Details

In the ever-evolving landscape of the financial services Industry, change is a constant and transformation is a requirement—to stay at pace with new regulations, risk mitigation, and the technological developments that support transformation. And just as financial services experiences its cycles, this time of year I find myself returning to the topic of cost reduction.

Financial services institutions are constantly challenged to strike the delicate balance between innovation and cost-saving as they seek out opportunities to decrease expenses while funding new application development projects. Historically, firms have relied on high-cost, third-party solutions to help identify savings opportunities, however, the landscape is rapidly changing, and the emergence of AI and machine learning (ML) has ushered in a new era of possibilities. These cutting-edge technologies provide lower-cost alternatives for discovering efficiencies within financial operations, all while enhancing the quality of services offered. In this dynamic environment, harnessing the power of AI and ML is proving to be a game-changer for firms seeking to thrive in an increasingly competitive market.

The use cases for these technologies continue to expand and improve, shaping the industry in ways we could only dream of. In fact, some of the insights presented in this blog have been assisted by the power of large language models (LLMs), highlighting the synergy between human expertise and AI-driven insights.

In this year’s post, I’ve collected cost-reduction strategies—augmented by the capabilities of generative AI—and innovative approaches that promise to accelerate cost-reduction and reshape the financial services landscape in the year ahead.

New technology means new opportunitiesAI generally and ML, generative AI, and LLMs specifically, have the potential to significantly reduce costs for financial services at large by automating tasks, improving productivity, and reducing the need for manual labor. And by investing in these technologies, companies gain the added benefits of increasing competitive advantage and improving customer experience.

Although this list is not exhaustive, I’ve provided an at-a-glance aggregate of use cases for quick wins, quicker wins, and longer-term strategies.

• Enterprise Knowledge Base (EKB) and Chatbots
  • Chatbots, or virtual assistants powered by generative AI, can be used to create customer-facing and employee-facing tools that assess user requests and provide personalized responses, reducing the need for human representatives and related costs. They can provide instant and accurate responses to common customer queries such as account balance, transaction history, loan information, and general banking procedures.
  • An EKB can be used to assist employees in accessing accurate and up-to-date product, regulatory compliance, and internal IT support information.
• Automated Research and Reporting
  • Automate the process of gathering, analyzing, and reporting financial data and market trends, reducing the time and cost of manual research for faster decision-making.
• Portfolio Optimization
  • Analyze a portfolio of investments and identify opportunities to optimize returns while managing risk. This can help investors reduce the time and cost of manual portfolio management and potentially improve investment performance.
• Content Generation, Text Classification, and Clustering
  • Automate website content for FAQs and help sections, keeping customer-facing content up to date.
  • Automatically create personalized messages, account statements, and transaction summaries, and notify customers of upcoming events or offers.
  • Group customers based on their transaction history, demographics, behavior patterns, and other relevant data to improve marketing and personalization outcomes.
  • Detect patterns and indicators of potential fraudulent activities using transaction data, customer profiles, and other relevant information.

Quick wins! Legal and Compliance + Summarizing regulatory requirements: AI can summarize regulatory requirements by analyzing large volumes of regulatory texts and extracting key information, making it easier for financial services firms to understand and comply with complex regulations. + Automating compliance monitoring: Analyzing and interpreting regulatory texts is time consuming, but AI can automate these tasks to help financial services firms understand and comply with complex regulations, saving time and costs associated with manual compliance monitoring. + Enhancing regulatory change management: Financial services firms must stay up to date with regulatory changes by monitoring and analyzing regulatory updates. AI can enable them to quickly adapt their compliance processes and reduce the risk of non-compliance. + Improving regulatory reporting: AI can automate the process of generating regulatory reports, ensuring accuracy and consistency while reducing the time and effort required for manual reporting. + Expedited legal research:* Generative AI tools can quickly search and analyze relevant case law, legislation, and secondary sources, enabling legal professionals to access pertinent information with ease.

Quickest wins! Language Translation + Multilingual customer service: With generative AI, customer support representatives can communicate effectively with customers who may not speak the bank’s primary language. + Document translation: When collaborating with multinational groups, generative AI can translate contracts, agreements, policies, and other legal/ business documents ensuring accurate written communication. * Code Development and Testing Assistance + Code generation: One of the benefits of generative AI is that it can be used to generate code automatically, reducing the time and effort required by developers. By analyzing existing code and patterns, generative AI algorithms can generate new code that is optimized for specific use cases. + Testing: Generative AI can be used to generate test cases automatically, reducing the time and effort required for developers to test their code. By analyzing the code and identifying potential edge cases, generative AI algorithms can generate test cases that cover a wide range of scenarios. + Debugging: Identifying and fixing bugs in code is essential for application security. By analyzing the code and identifying potential issues, generative AI algorithms can suggest fixes that can be implemented by developers. + Optimization: By analyzing the code and identifying potential optimizations, generative AI can update code automatically, improving performance and reducing resource usage. + Synthetic data generation: Generative AI can generate synthetic data for testing purposes and for training machine learning models, helping developers improve the accuracy of their models and make more informed decisions. + Automated documentation generation: Generating documentation is time consuming and tedious. AI can automatically generate documentation as developers write code, reducing the need for manual documentation while improving efficiency. + Code comments: Generative AI can be used to generate code comments that describe the function and purpose of each element of the code, helping developers identify inconsistencies or potential sources of errors. + Understanding legacy code: Generative AI can be used to analyze and understand the structure and functionality of legacy code, making it easier for developers to work with and maintain, including: - Generating unit tests: Automatically generate unit tests for legacy code, helping developers identify and fix potential issues and improve code quality. - Code refactoring: Assist in refactoring legacy code by suggesting improvements and identifying potential sources of bugs or inefficiencies. - Translating legacy code to modern languages:* Translate legacy code written in outdated languages, such as COBOL, to more modern languages, making it easier to maintain and integrate with newer systems.

Getting StartedUtilizing AI and its related technologies for cost reduction may seem like a huge lift, but the opportunities in financial services are many, and overcoming the perceived challenges is getting easier every day.

Cloudera offers a variety of solutions designed to transform AI initiatives into tangible, cost-reducing outcomes. Facilitate experimentation and accelerate the development of AI applications within Cloudera Machine Learning using Applied ML Prototypes (AMPs). Deploy internal LLMs and vector databases with LLMs trained on private data sets in a secure environment to experiment without risk, all while safeguarding sensitive data on premises, ensuring compliance and reducing the costs associated with prolonged cloud-based workloads.

We’re here to help you execute on your cost-reduction strategies. If you are interested in learning more about Cloudera’s solutions for enterprise AI or discussing any of the above use cases, please reach out to me.

The post Accelerating Cost Reduction: AI Making an Impact on Financial Services appeared first on Cloudera Blog.

View Details

Recently, Confluent hosted Current 2023 (formerly Kafka summit) in San Jose on Sept 26th and 27th. With few conferences curating content specific to streaming developers, Current has historically been an important event for anyone trying to keep a pulse on what’s happening in the streaming space. Over 2,000 attendees and lots of new solutions were on display, and the event proved to be a clear look into the current (no pun intended) state of streaming and where it is headed. This blog is for anyone who was interested but unable to attend the conference, or anyone interested in a quick summary of what happened there. I will cover key takeaways from Current 2023 and offer Cloudera’s perspective.

Five Takeaways from Current 2023: 1- The people have spoken and Apache Flink is the de facto standard for stream processing

This may seem obvious to many who are already familiar with Flink, but it is worth pointing out. Architecture decisions have long-term effects and an important consideration when choosing a stream processing engine is whether the technology will stagnate or continue to evolve with contributions from the open source community. Will I be able to find developers for this three years from now? The answer from the community is a resounding yes. Flink is here to stay.

It makes perfect sense that Apache Flink has emerged as the standard. Flink was launched in 2015 as the world’s first open source streaming-first distributed stream processing engine and has since grown to rival Spark in terms of popularity. And the layered APIs from low-level operations to high-level abstractions gives Flink appeal to a broad range of users. The adoption of Flink mirrors growth in streaming data volumes and maturity of the streaming market. As organizations shift from the modernization of data-driven applications via Kafka towards delivering real-time insight and/or powering smart automated systems, Flink

At Current, adoption of Flink was a hot topic and many of the vendors (Cloudera included) use Flink as the engine to power their stream processing offerings as well. Use cases such as fraud monitoring, real-time supply chain insight, IoT-enabled fleet operations, real-time customer intent, and modernizing analytics pipelines are driving development activity. The value of consolidating different processing frameworks onto a single comprehensive framework to minimize technical overhead and maintain innovation speed is well understood.

The big announcement everyone was waiting for was the unveiling of Apache Flink in Confluent Cloud. The actual unveiling was a bit underwhelming as the SQL console left a lot to be desired, and outside of serverless auto-scaling functionality there was no “wow” factor. As of this writing, the product is still not GA and will not be made available on-prem, but the unveiling is still important due to the sheer size of the Confluent user base. Adoption will follow, and it’s safe to say that we have passed the tipping point— Flink is the future of streaming.

Cloudera’s perspective: Cloudera saw the increasing volumes of data our customers were moving via streams early on. They were suffering rising costs and were struggling to provide real-time insight to demanding stakeholders. So we bet big on Flink in 2020 and started developing tooling to bring it to the enterprise, and have a mature Flink product used by customers in banking, telco, manufacturing, and IT. kSQLdb, Spark Structured Streaming, and other proprietary approaches that fall short of the truly open and distributed stateful stream processing capabilities that Flink brings to the table will likely decelerate.

2- But there is an intriguing new category of competitor emerging, the “streaming database”

There are a handful of vendors positioning streaming databases as an alternative to Flink for stream processing. Their core value proposition is that streaming databases are inherently faster than Flink due to in-memory processing and state management. This makes sense in theory, but there are pretty wild claims out there as far as just how much faster they are, and with a lack of independent benchmarks in the industry a healthy dose of skepticism is warranted. But the tech is interesting and the allure of DB tooling that can “do-it-all” is strong.

Cloudera’s perspective: There is much value to be captured by bringing real-time processing capabilities to streaming architectures. Kafka-centric approaches leave a lot to be desired, most notably operational complexity and difficulty integrating batch data, so there is certainly a gap to be filled. Real-time databases have their place in the streaming ecosystem, but that place is in publishing and making the result sets widely available after a highly scalable engine like Flink has processed the data. Cloudera does this via materialized views that are accessible via API. Also, why solve for connectivity and data distribution again if it’s already solved for? How long does streaming data live inside the database and what happens when it expires? Is this yet another database? What about data lock-in? With highly interdependent capabilities, how difficult will it be to make modifications as business and data requirements evolve?

This class of technologies is very interesting, but still new—“wait and see” is perhaps sage advice.

3- Change data capture is red hot and Debezium is the de facto standard in this space

Judging by the sheer number of questions from the audience about CDC in general and Debezium specifically, it’s safe to say that Debezium has become for CDC what Flink is for stream processing. It makes perfect sense—similar to Flink, Debezium is an open source distributed service frequently used with Kafka to extend the value of streaming and capture new use cases. Debezium works by continuously reading the change logs of popular databases and publishing to Kafka topics, effectively transforming legacy batch systems into rich streams of data.

Debezium does have certain complexities of course, namely resource management and schema evolution. But there is much value to be captured here.

Cloudera perspective: Data freshness matters. It’s difficult to imagine a use case where fresher data isn’t inherently better data. Change Data Capture is an important part of the streaming ecosystem. Cloudera supports Debezium connectors for Kconnect and Flink and will soon release a NiFi processor as well, giving users fine grain control over data distribution.

4- Tooling for the Kafka ecosystem is improving

It’s no secret that Kafka deployments can be quite complex. Setting up clusters, monitoring and managing brokers, partitions, and topics, handling message ordering, exactly once guarantees, schema evolution and security: these all add up to operational overhead. Data lineage and debugging can be a nightmare to unravel. As the streaming space grows in maturity one thing that stood out is the improved tooling in the space. Confluent’s future vision for the data portal is a great example of the effort to provide better tooling and smoother user experience around discoverability and governance. Many vendors are providing enhanced tooling to provide observability and improve performance or to extend the ecosystem by integrating other frameworks such as MQTT and Pulsar.

Cloudera perspective: Cloudera began providing support and building tooling for the Kafka ecosystem in 2015 and has developed stable enterprise solutions. The Streams Messaging Manager tool is included in our free community edition of Cloudera Streams Processing. Furthermore, Cloudera SDX provides an integrated set of security and governance tools across the entire data lifecycle, including streaming. The Kafka platform shifting from Zookeeper to Kraft as is a huge relief for anyone managing Kafka operations. KRaft is already in tech preview for our next release.

For these reasons and more, IBM recently chose Cloudera as strategic Kafka partner of choice to bring cost efficient, scalable solutions to our enterprise customers.

https://blog.cloudera.com/ibm-technology-chooses-cloudera-as-its-preferred-partner-for-addressing-real-time-data-movement-using-kafka/

5- There is still room for growth and maturation in the streaming space

While adoption of streaming technologies has steadily increased, the average streaming maturity level is still in the early stages. Streaming maturity is not about simply streaming more data; it’s about weaving streaming data more deeply into operations to drive real-time utilization across the enterprise. The number of use cases supported by a single Kafka topic is a better indicator than a raw measure of volume like events per second. Surprisingly few users had multiple use cases for most of their Kafka topics. Another hallmark of streaming maturity is the efficiency of the entire system in terms of resource utilization and ease of developing or modifying new use cases. Real-time processing can significantly reduce the volume of data in the stream and that’s a good thing. The majority of data streamers are just beginning to experiment here.

More forward-looking talks focused on expanding the impact of streaming data. Real-time anomaly detection and other time series operations on event streams. Operationalizing python for real-time ML pipelines was a hot topic. Others focused on the big picture efficiency, looking for ways to reduce load on Kafka by integrating with Apache Pinot for example (link below to an NYC-based Meetup on this topic). There was conspicuously little content specific to generative AI, which was a bit surprising given the attention the industry at large has given the topic in 2023. Streaming data absolutely has a tremendous role to play in generative AI, in fine tuning foundational models, optimizing prompts, contextualizing and augmenting outputs, etc. Stay tuned for plenty more on that topic!

Cloudera perspective: Data streams are part of a much broader data lifecycle. Kafka can’t do it all. Kafka shines when utilized as the real-time bus for application integration and as the message buffer for analytics workflows. When stretched beyond those core capabilities however, it becomes overly complex and carries significant technical overhead. That’s why a complete approach to streaming is needed. An efficient and scalable streaming architecture should be simple yet complete with tooling to address continuous iterative development cycles. That includes first class support for data distribution (aka universal data distribution), edge data capture, stream filtering, independently modifiable stream processing that is accessible to analysts, and integration with data at rest for low cost accessible storage. Lastly, real-time processing and movement of multi structured data including prompts and embeddings is critical for harnessing the transformative power of AI.

Download Cloudera Stream Processing Community edition for FREE and get zero to Flink in less than an hour. Our SQL Stream Builder console is the most complete you’ll find anywhere.

Sign up for a free trial of Cloudera’s NiFi-based DataFlow and walk through use cases like stream filtering and cloud data warehouse ingest.

Join myself and Developer Advocate Tim Spann in New York City for the latest on real-time, including generative AI and more, cohosted by Cloudera and Apache Pinot based Startree.

The post 5 Key Takeaways from #Current2023 appeared first on Cloudera Blog.

View Details

Cloudera recently released a fully featured Open Data Lakehouse, powered by Apache Iceberg in the private cloud, in addition to what’s already been available for the Open Data Lakehouse in the public cloud since last year. This release signified Cloudera’s vision of Iceberg everywhere. Customers can deploy Open Data Lakehouse wherever the data resides—any public cloud, private cloud, or hybrid cloud, and port workloads seamlessly across deployments.

With Cloudera Open Data Lakehouse in the private cloud, you can benefit from following key features:

• Multi-engine interoperability and compatibility with Apache Iceberg, including NiFi, Flink and SQL Stream Builder (SSB), Spark, and Impala.
• Time Travel: Reproduce a query as of a given time or snapshot ID, which can be used for historical audits, validating ML models, and rollback of erroneous operations, as an example.
• Table Rollback: Allow users to quickly correct problems by resetting tables to a good state.
• Rich set of SQL (query, DDL, DML) commands: Create or manipulate database objects, run queries, load and modify data, perform time travel operations, and convert Hive external tables to Iceberg tables using SQL commands.
• In-place table (schema, partition) evolution: Effortlessly evolve Iceberg table schema and partition layouts without rewriting table data or migrating to a new table, for example.
• SDX Integration: Provides common security and governance policies, as well as data lineage and auditing.
• Iceberg Replication: Provides disaster recovery and table backups.
• Easy portability of workloads to public cloud and back without any code refactoring.

In this multi-part blog post, we’re going to show you how to use the latest Cloudera Iceberg innovation to build an Open Data Lakehouse on a private cloud.

For this first part of the blog series we will focus on ingesting streaming data into the open data lakehouse and Iceberg tables making it available for further processing that we will demonstrate in the following blogs.

Solution OverviewPre-requisitesThe following components in Cloudera Open Data Lakehouse on Private Cloud should be installed and configured and airline data sets:

• Cloudera Data Platform Private Cloud Base 7.1.9
• Cloudera Flow Management 2.1.6
• https://github.com/jingalls1217/airlines-source-data.git (make sure to unzip the flights.csv.gzip file in the flights directory)

In this example, we are going to use NiFi as part of CFM 2.1.6 to stream ingest data sets to Iceberg. Please note, you can also leverage Flink and SQL Stream Builder in CSA 1.11 as well for streaming ingestion. We use NiFi to ingest an airport route data set (JSON) and send that data to Kafka and Iceberg. We then use Hue/Impala to take a look at the tables we created.

Please reference user documentation for installation and configuration of Cloudera Data Platform Private Cloud Base 7.1.9 and Cloudera Flow Management 2.1.6.

Follow the steps below for using NiFi to stream ingest data into Iceberg tables:1- Create the routes Iceberg table for NiFi ingestion in Hue/Impala execute the following DDL:2- Download a pre-built flow definition file found here: https://github.com/jingalls1217/airlines/blob/main/Data%20Flow/NiFiDemo.json

3-Create a new process group in NiFi and upload the flow definition file downloaded in step 2. First click the Browse button, select the NiFiDemo.json file and click the Add button.4- Update parameters as shown in table below:5- Click into the NiFiDemo process group: 1. 1. Right click on the NiFi canvas, go to Configuration and enable the Controller Services. 2. Open each Process Group and right click on the canvas, go to Configuration and Enable any additional Controller Services not yet enabled.

6- Start the Routes ingest to Kafka flow and monitor success/failure queues:7- Start the Routes Kafka to Iceberg flow and monitor success/failure queues: 8- Inspect the Routes Iceberg table in Hue/Impala to see the data that has been loaded:SELECT * FROM airlines.routes_nifi_iceberg;

ConclusionIn this first blog, we showed how to use Cloudera Flow Management (NiFi) to stream ingest data directly to the Iceberg table without any coding. Stay tuned for part two, Data Processing with Apache Spark.

To build an Open Data Lakehouse on your private cloud, download Cloudera Data Platform Private Cloud Base 7.1.9 and follow our Getting Started blog series.

And since we offer the exact same experience in the public and private cloud you can also join one of our Two hour hands-on-lab workshops to experience the open data lakehouse in the public cloud or sign up for a free trial. If you are interested in chatting about Cloudera Open Data Lakehouse, contact your account team. As always, we welcome your feedback in the comments section below.

The post Getting Started With Cloudera Open Data Lakehouse on Private Cloud appeared first on Cloudera Blog.

View Details

In a previous blog post, we explored the power of Cloudera Observability in providing high-level actionable insights and summaries for Hive service users. In this blog, we will delve deeper into the insight Cloudera Observability brings to queries executed on Hive.

As a quick recap, Cloudera Observability is an applied observability solution that provides visibility into Cloudera deployments and its various services. The tool allows automatic actions to prevent negative consequences like excessive resource consumption and budget overruns. Among other capabilities, Cloudera Observability delivers comprehensive features to troubleshoot and optimize Hive queries. Furthermore, it provides insights from deep analytics for a variety of supported engines using query plans, system metrics, configuration, and much more.

An essential goal for a Hive SQL developer is ensuring that queries run efficiently. If there are issues in the query execution, it should be possible to debug and diagnose those quickly. When it comes to individual queries, the following questions typically crop up:

1. What if my query performance deviates from the expected path?
   • When my query goes astray, how do I detect deviations from the expected performance? Are there any baselines for various metrics about my query? Is there a way to compare different executions of the same query?
2. Am I overeating, or do I need more resources?
   • How many CPU/memory resources are consumed by my query? And how much was available for consumption when the query ran? Are there any automated health checks to validate the resources consumed by my query?
3. How do I detect problems due to skew?
   • Are there any automated health checks to detect issues that might result from skew in data distribution?
4. How do I make sense of the stats?
   • How do I use system/service/platform metrics to debug Hive queries and improve their performance?
5. I want to perform a detailed comparison of two different runs; where should I start?
   • What information should I use? How do I compare the configurations, query plans, metrics, data volumes, and so on?

Let’s check how Cloudera Observability answers the above questions and helps you detect problems with individual queries.

What if my query performance deviates from the expected path?Imagine a periodic ETL or analytics job you run on Hive service for months suddenly becomes slow. It’s a scenario that’s not uncommon, considering the multitude of factors that affect your queries. Starting from the simplest, a job could slow down because your input or output data volume increased, data distribution is now different because of the underlying data changes, concurrent queries are affecting the use of shared resources, or system hardware issues such as a slow disk. It could be a tedious task to find out where exactly your queries slowed down. This requires an understanding of how a query is executed internally and different metrics that users should consider.

Enter Cloudera Observability’s baselining feature, your troubleshooting partner. From execution times to intricate details concerning the Hive query and its execution plan, every vital aspect is considered for baselining. This baseline is meticulously formed using historical data from prior query executions. So when you detect performance deviations for your Hive queries, this feature becomes your guide, pointing you to metrics of interest.

Am I overeating, or do I need more resources? As an SQL developer, striking a balance between query execution and optimal use of resources is vital. Naturally, you would want a straightforward way to find out how many resources were consumed by your query and how many were available. Furthermore, you also want to be a good neighbor when using shared system resources and not monopolize their use.

The “Cluster Metrics” feature in Cloudera Observability helps you achieve this.

Challenges may also arise if you have fewer resources than your query needs. Cloudera Observability steps in with several automated query health checks that help you identify the problems due to resource scarcity.

How do I detect problems due to skew?In the realm of distributed databases (and Hive is no exception), there is an essential rule that data should be distributed evenly. The non-uniform distribution of the data set is called data “skew.” Data skew can cause performance issues and lead to non-optimized usage of available resources. As such, the ability to detect issues due to skew and provide recommendations to resolve those helps Hive users considerably. Cloudera Observability comes armed with several built-in health checks to detect problems due to skew to help users optimize queries.

How do I make sense of the stats?In today’s tech world, metrics have become the soul of observability, flowing from operating systems to complex setups like distributed systems. However, with thousands of metrics being generated every minute, it becomes challenging to find out the metrics that affect your query jobs.

The Cloudera platform provides many such metrics to make it observable and aid in debugging. Cloudera Observability goes a step further and provides built-in analyzers that perform health checks on these metrics and spot any issues. With the help of these analyzers, it’s easy to spot system and load issues. Furthermore, Cloudera Observability provides you the ability to search metric values for important Hive metrics that may have affected your query execution. It also provides interesting events that may have happened on your clusters while the query ran.

I want to perform a detailed comparison of two different runs; where should I start?It’s not unusual to observe a degradation in query performance for various reasons. As a developer, you are on a mission to compare two different runs and spot the differences. But where would you start? There is so much to find out and compare. For example, starting from the most straightforward metrics like execution duration or input/output data sizes, to complex ones like differences between query plans, Hive configuration when the query was executed, the DAG structure, query execution metrics, and more. A built-in feature that achieves this is of great use, and Cloudera Observability does this precisely for you.

With the query comparison feature in Cloudera Observability, you can compare all of the above factors between two executions of the query. Now it is effortless to spot changes between the two executions and take appropriate actions.

As illustrated, gaining insight into your Cloudera Hive queries is a breeze with Cloudera Observability. Analyzing and troubleshooting Hive queries has never been this straightforward, enabling you to boost performance and catch any issues with a keen eye.

To find out more about Cloudera Observability, visit our website. To get started, get in touch with your Cloudera account manager or contact us directly.

The post Unlock the Full Potential of Hive appeared first on Cloudera Blog.

View Details

Your company collects data from different sources and then you analyze the data to help make the right decisions. But you aren’t quite getting the results that you expect. Maybe the insights aren’t accurate. Perhaps the process is time consuming and cumbersome. Or you are only currently using data for a few use cases and struggle to implement organization wide.

Using data effectively is the key to competing in today’s world as customer expectations continue to rise. Moving from a best guess mindset to a data driven organization requires a significant shift throughout the organization in terms of technology, processes, and culture. Your organization is not alone — many organizations struggle to move towards data as the cornerstone of their organization.

Here are five challenges that you need to overcome to become a data leader:

Bad data governanceYour insights are only as good as your data. If your data is full of errors, missing information and duplications, then your results will not be accurate. Your organization could make an inaccurate decision with significant consequences based on faulty or missing data. Your company’s data success depends on your data governance, which is how you ensure your data is clean, accurate, and easy to use. However, many organizations either do not have data governance programs or the business participation an effective program requires.

By using data mesh, you move true data ownership to the business units, which improves the quality. Data now becomes a product. The data mesh platform manages all of the security and governance, meaning that your insights are based on high quality data, thanks to the data observability and data cataloging capabilities within the platform. That facilitates data governance process automation which improves productivity as well as accuracy.

Data silosFor accurate cross-functional insights, a high percentage of the data from the organization might be required. However, many organizations have data silos, for instance when each department’s data is historically stored in disparate locations. Additionally, structured and unstructured data is often separate. By using a data platform that combines all data and creates a single view, your data provides the complete picture of your organization. By eliminating data silos, your data insights enable smarter and more accurate business decisions.

Lack of skillsMany organizations do not have on-staff data scientists due to expense or availability. If your organization lacks such skills, here’s what to do: employ technology that enables your data analysts to attack data science problems without necessitating a degree in statistics. That means an easy to use machine learning interface and open source, license-free machine learning algorithms. There are literally hundreds available including large language models which are all the rage. Such a strategy builds data science skill with the advantage that your data analysts already know their data domains, shortening the time to value.

Lack of real-time dataCreating a data-driven organization often requires near real-time data. If you set up an automation for sales contracts, you need the latest information about customer status and pricing — not yesterday’s data. Real-time data also allows you to create personalized customer experiences, such as customizing the offers and information on your home page. Additionally, using IoT sensors for automations is only possible with the ability to collect and manage real-time data.

Using real-time data starts with using cloud-based technology, such as hybrid cloud. By replacing legacy infrastructure, employees and sensors can access data from any location and see the latest data. Next, your organization should turn to a data platform that provides real-time data analytics and insights. Your organization can then make quick decisions based on the most up to date information possible. By operating on real-time data, your organization can have a competitive advantage over other organizations through insights, personalization, and automations.

Legacy infrastructureBecoming a data leader is challenging, if not impossible to achieve, if your organization is using outdated legacy technology. Outdated technology causes many of the other challenges that keep you from becoming a data leader — need for specialized skills, data silos, and lack of real-time data. While on-prem legacy technology certainly has its place, by moving toward a modern private cloud or hybrid alternative where possible, your organization has access to the data that you need to make the right business decisions and create the personalized experiences that customers expect.

Some organizations get frustrated and abandon their efforts for using data. By instead using a data platform, such as Cloudera, your organization can overcome these common challenges with using data. With the right tools and processes, your organization can begin moving towards becoming a data leader.

Ready to become a true data leader? See how to take the next steps and get there fast.

The post Five Common Pitfalls on the Path to Becoming a Data-Driven Enterprise appeared first on Cloudera Blog.

View Details

Fast moving data and real time analysis present us with some amazing opportunities. Don’t blink—or you’ll miss it! Every organization has some data that happens in real time, whether it is understanding what our users are doing on our websites or watching our systems and equipment as they perform mission critical tasks for us. This real-time data, when captured and analyzed in a timely manner, may deliver tremendous business value. For example:

• In manufacturing, fast-moving data provides the only way to detect—or even predict and prevent—defects in real time before they propagate across an entire production cycle. This will reduce defect rates, increasing product yield. We can also increase effectiveness of preventative maintenance—or move to predictive maintenance—of equipment, reducing the cost of downtime without wasting any value from healthy equipment.
• In telecommunications, fast-moving data is essential when we’re looking to optimize the network, improving quality, user satisfaction, and overall efficiency. With this, we can reduce customer churn and overall network operational costs.
• In financial services, fast-moving data is critical for real-time risk and threat assessments. We can move to predictive fraud and breach prevention, greatly increasing the protection of customer data and financial assets. Without real-time analytics we won’t catch the threats until after they’ve caused significant damage. We can also benefit from real-time stock ticker analytics, and other highly monetizable data assets.

By capitalizing on the business value of fast-moving and real-time analytics, we can do some game changing things. We can reduce costs, eliminate unnecessary work, improve customer satisfaction and experience, and reduce churn. We can get to faster root-cause analysis and become proactive instead of reactive to changes in markets, business operations, and customer behavior. We can get the jump on competition, reduce surprises that cause disruption, have better organizational operational health, and reduce unnecessary waste and cost everywhere.

The need for real-time decision support and automation is clear.However, there are some key capabilities that will make real-time analytics a practical and applied reality. What we need is:

• An openness to support a wide range in streaming ingest sources, including NiFi, Spark Streaming, Flink, as well as APIs for languages like C++, Java, and Python.
• The ability to support not just “insert” type data changes, but Insert+update patterns as well, to accommodate both new data, and changing data.
• Flexibility for different use cases. Different data streams will have different characteristics, and having a platform flexible enough to adapt, with things like flexible partitioning for example, will be essential in adapting to different source volume characteristics.

On top of these core critical capabilities, we also need the following:

• Petabyte and larger scalability—particularly valuable in predictive analytics use cases where high granularity and deep histories are essential to training AI models to greater precision.
• Flexible use of compute resources on analytics—which is even more important as we start performing multiple different types of analytics, some critical to daily operations and some more exploratory and experimental in nature, and we don’t want to have resource demands collide.
• Ability to handle complex analytic queries—especially when we’re using real-time analytics to augment existing business dashboards and reports with large, complex, long-running business intelligence queries typical for those use cases, and not having the real-time dimension slow these down in any way.

And all of this should ideally be delivered in an easy to deploy and administer data platform available to work in any cloud.

A unique architecture to optimize for real-time data warehousing and business analytics:Cloudera Data Platform (CDP) offers Apache Kudu as part of our Data Hub cloud service, providing a consistent, dependable way to support the ingestion of data streams into our analytics environment, in real time, and at any scale. CDP also offers the Cloudera Data Warehouse (CDW) as a containerized service with the flexibility to scale up and down as needed, and multiple CDW instances can be configured against the same data to provide different configurations and scaling options to optimize for workload performance and cost. This also achieves workload isolation, so we can run mission critical workloads independent from experimental and exploratory ones and nobody steps on anyone’s toes by accident.

Fig. 1: Kudu & Impala for Real-Time Data Warehousing

Key features of Apache Kudu include:Support for Apache NiFi, Spark Streaming, and Flink pre-integrated and out of the box. Kudu also has native support for C++, Java, and Python APIs for capturing data streams from applications and components based on those languages. With such a wide range of ingest types, Kudu can get anything you need from any real-time data source.

• Full support for insert and Insert+update syntax for very flexible data stream handling. Being able to capture not just new data, but also changed data, greatly facilitates Change Data Capture (CDC) use cases as well as any other use case involving data that may change over time, and not always be additive.
• Ability to use multiple different flexible partitioning schemes to accommodate any real-time data, regardless of each stream’s particular characteristics. Making sure data is able to land in real time and be accessed just as fast requires a “best fit” partitioning scheme. Kudu has this covered.

Key features of Cloudera Data Warehouse include:* Powerful Apache Impala query engine capable of handling massive scale data sets and complex, long running enterprise data warehouse (EDW) queries, to support traditional dashboards and reports, augmented by real-time data. * Containerized service to run both multiple compute clusters against the same data, and to configure each cluster with its own unique characteristics (instance types, initial and growth sizing parameters, and workload aware auto scaling capabilities). * Full lifecycle support including Cloudera Data Engineering (CDE) for data preparation, Cloudera Data Flow (CDF) for streaming data management, and Cloudera Machine Learning (CML) for easy inclusion of data science and machine learning in the analytics. This is especially necessary when combining real-time data with prepared data, and adding predictive concepts into our augmented dashboards and reports.

CDW integrates Kudu in Data Hub services with containerized Impala to offer easy to deploy and administer, flexible real-time analytics. With this unique architecture, we support stable and consistent ingestion of huge volumes of fast moving data, tougher with flexible, workload-isolated data warehousing services. We get optimized price/performance on complex workloads over massive scale data.

Ready to stop blinking and never miss a beat?Let’s take a close look at how to get started with CDP, Kudu, CDW, and Impala and develop a game changing real-time analytics platform.

Check out our recent blog on integrating Apache Kudu on Cloudera Data Hub and Apache Impala on Cloudera Data Warehouse to learn how to implement this in your Cloudera Data Platform environment.

The post Don’t Blink: You’ll Miss Something Amazing! appeared first on Cloudera Blog.

View Details

As data ecosystems evolve security becomes a paramount concern, especially within the realm of private cloud environments. Cloudera on Private Cloud with the Private Cloud Base (CDP PvC Base) stands as a beacon of innovation in the realm of data security, offering a holistic suite of features that work in concert to safeguard sensitive information. With the latest 7.1.9 release, the journey towards a more secure data ecosystem continues—one where businesses can unlock the full potential of their data with peace of mind.

How does this release elevate security?Cloudera on private cloud integrates a unified security platform that orchestrates the full spectrum of security measures. From access controls and identity management to encryption and auditing, this comprehensive approach ensures that every facet of your data ecosystem is protected against potential threats and vulnerabilities. This latest version delivers security enhancements for data at rest, data in transit, and Federal Information Processing Standards (FIPS) compliance, as well as compliance with various regulatory requirements.

Platform security for data in transitThe platform uses transport layer security (TLS) and secure socket layer (SSL) protocols to establish a secure communication channel between different components of the platform for better privacy and data integrity. Using a cryptographic protocol safeguards data from being intercepted or modified during transit, thwarting potential cyber threats. Therefore, it is critical to update security protocols to adapt to the changing threat landscape and defend against the latest attack methods.

This version securely leaps forward by offering TLS version 1.2, which delivers a refined set of cipher suites, strengthened cryptographic algorithms, and a robust handshake process for improved security and resilience. Additionally, the release introduces Oracle TCP/IP using SSL (TCPS) support to facilitate secure communication between PvC Base components and Oracle backend DB. This ensures data administration and monitoring happens securely through the TCPS connection protocol.

Enhancing encryption for data at restSeveral data at rest encryption mechanisms such as key management systems (KMS) ensure that sensitive information is shielded from potential threats and unauthorized access. The latest release uses Ranger KMS to provide unified key management services for encryption in lieu of key trustee server (KTS). This enhances customer experience, as this centralized approach streamlines policy management and ensures consistency in access control rules. Furthermore, for existing users of KTS, import of keys from KTS and NavEncrypt as well as automation of NavEncrypt nodes from old KTS servers to Ranger KMS servers has been streamlined so data encryption and security remains uncompromised.

In this latest version, perimeter security is enhanced as well. The Knox HttpFS feature provides a secure way to access HDFS resources through a web interface using HTTP methods. Additionally, Knox token authentication can now be used to establish secure connections and manage user access. Token-based authentication provides efficient and scalable user authentication using tokens, which are easily rolled, renewed, and revoked and therefore, reduce the risk of exposure of user credentials.

Custom Kerberos principals and service usersIsolation is an important concept when securing infrastructure to minimize the potential impact of vulnerabilities. Today, CDP services use default names for Kerberos principals with matching service user names on host machines. However, using the default configuration can extend the accessibility of a single service beyond the cluster it’s installed on in a multi-cluster deployment. To address this, organizations seeking more advanced techniques for segregating multiple clusters can create custom Kerberos principals (CKP) along with corresponding custom service users. The CKP empowers organizations to restrict services from accessing data on clusters belonging to distinct lines of business or projects. It is worth noting that users gain access to this functionality during the process of creating a new cluster or adding a service to an existing cluster.

FIPS 140-2 updatesFor organizations entrusted with confidential financial data, healthcare records, or government information, adherence to rigorous security standards like FIPS is not only a strategic choice but also a legal obligation.

Configuring the release to use FIPS 140-2 compliant cryptography within an operating system (OS) configured for FIPS-mode is currently supported for customers deploying on RHEL 7.8 and 7.9. With the addition of Red Hat Enterprise Linux (RHEL) 8.8 FIPS support, customers using RHEL 8.8 can now deploy the platform configured to use FIPS 140-2 compliant cryptography, on an FIPS-mode enabled RHEL 8.8 operating system. There are a number of CDP components that support configurability to use FIPS 140-2 compliant cryptography today. However, as part of our efforts to stay steadfastly aligned with the highest security benchmarks, we have extended the configurability to use FIPS 140-2 compliant cryptography support with the Phoenix, NavEncrypt, Ranger Key Management Service, and Key Trustee Server components. Full list of supported components for FIPS 140-2 are available at the required prerequisites for FIPS for CDP page.

Security vulnerabilities remediationWe take fixing security vulnerabilities seriously! We have an effective vulnerability remediation process that proactively scans, prioritizes, fixes, and monitors common vulnerabilities and exposures (CVE) to demonstrate our commitment to our customers’ security. The 7.1.9 release improved platform security and InfoSec compliance with 100+ resolved critical CVEs. This reduces the attack surface in customer deployments for a more secure ecosystem that prevents vulnerabilities from being exploited at any point.

ConclusionIn this world of technological evolution and the ever-changing landscape of cybersecurity, the proactive effort to keep security protocols up to date is crucial. By enforcing strong encryption mechanisms for data at rest, data in transit for enterprises, and supporting CGI standards for regulated industries, Cloudera on private cloud demonstrates a commitment to maintaining the integrity and availability of their systems and information.

Let us empower your organization to navigate the cybersecurity landscape confidently and effectively with this latest release! If you’re not already on the latest release, get started today! It’s now easier than ever to upgrade or migrate from previous versions to this one. To find your perfect path to 7.1.9, click on our migration and upgrade guide.

The post Strengthening Your Data Ecosystem with Unrivaled Security appeared first on Cloudera Blog.

View Details

Organizations increasingly rely on streaming data sources not only to bring data into the enterprise but also to perform streaming analytics that accelerate the process of being able to get value from the data early in its lifecycle. As lakehouse architectures (including offerings from Cloudera and IBM) become the norm for data processing and building AI applications, a robust streaming service becomes a critical building block for modern data architectures.

Apache Kafka has evolved into the most widely-used streaming platform, capable of ingesting and processing trillions of records per day without any perceptible performance lag as volumes scale. It is frequently used with several other Apache technologies as part of a larger streams processing platform, event driven architecture or a modern data lakehouse.

Cloudera Data Platform not only provides a trusted, enterprise-grade Apache Kafka distribution including unique tools for monitoring and operations, but also a complete streaming data platform including Apache Flink for stream processing and Apache NiFi for data movement, accelerating time to value for customers and enabling stream processing at scale.

Based on a partnership designed to bring IBM’s advanced data and AI solutions to more organizations across the expansive Apache Open Source Database ecosystem, IBM Technology is partnering with Cloudera as our preferred partner for addressing real time data movement built on Cloudera’s Data Flow leveraging Kafka. This will help new and existing customers to leverage the power of both solutions and provide significant benefits across three areas:

1. Increase developer agility to build data pipelines faster
2. Accelerate public cloud data migration with the best TCO
3. Deliver trusted data across hybrid environments with auto generated rules

“Trusted data is the backbone of all AI applications and Cloudera offers streaming solutions with comprehensive security and governance capabilities that already drive mission critical applications for our joint customers today. We expect this partnership to help accelerate workload migration to public cloud, drive new customer adoption, and deliver cutting edge AI solutions to our joint customers”

— Abhas Ricky, Chief Strategy Officer, Cloudera

A history of collaborationThe IBM and Cloudera strategic partnership builds on a longstanding relationship creating solutions for enterprise data cloud businesses.

IBM and Cloudera’s common goal is to accelerate data-driven decision making for enterprise customers, working on defining and executing the best solution for each customer. These solutions include on-premises and public cloud offerings with the data management, security and governance needed to build an enterprise data and analytic big data solution.

You can now elevate your data potential and activate AI’s capabilities through the synergic integration between IBM watsonx and Cloudera. Together, IBM and Cloudera help you unlock data potentials ensuring that your data-driven transformation is not just comprehensive but orchestrated for tomorrow’s business needs.Learn more about how you can benefit from a well-supported data management platform and ecosystem of products, services and support by visiting the IBM and Cloudera partnership page.

The post IBM Technology Chooses Cloudera as its Preferred Partner for Addressing Real Time Data Movement Using Kafka appeared first on Cloudera Blog.

View Details

Perhaps your C-suite is already a bit crowded. The typical hierarchy will include a CEO, COO, CFO, CTO, CMO, CIO, and a few more. Adding another position may not be terribly appealing, but there is one C-suite role every company should consider—chief data and analytics officer (CDO or CDAO).

The CDO is the point person for your data strategy: the leader who oversees how data is collected, managed, and put to use to improve the organization; the person who ensures that wherever there are opportunities to monetize data, those opportunities aren’t being squandered.

The CDO is an essential role in a data-driven organization. Data is the lifeblood of modern business, the fuel that powers digital transformation, and every company should have a data strategy. That being the case, you need a CDO to spearhead and defend that strategy.

Without a data champion, the C-suite can overlook and even ignore data. Gartner’s CDO Agenda Survey for 2023 found that 78% of surveyed data leaders rank organizational data strategy and vision as one of their top three priorities, and 68% are prioritizing initiatives based on alignment to strategic goals. With multiple stakeholders across an organization and the overwhelming amounts of data, CDO is the point person who can help advocate your data initiatives and priorities directly to the CEO.

Here are seven compelling reasons for why your company needs a CDO:

1. CDO establishes a data-driven culture The C-suite heavily influences the culture at the company, and if data is not part of the C-suite, it isn’t likely to be part of the culture. Anyone can talk about having a data-driven culture, but an actual strategy requires action, not just talk. The CDO drives the implementation and use of platforms, tools, and processes to support a data strategy. The CDO also helps shape, communicate, and implement the data strategy and vision to all stakeholders.

2. CDO inspires the data team To succeed, leaders need to inspire their teams to be passionate, productive, and willing to work with other stakeholders toward common goals. Leaderless and uninspired data teams are likely to feel misunderstood and such organization-wide efforts as data governance can be hard to implement. The CDO inspires and motivates their team through clear communication, well-defined goals and—when needed—course correction.

3. CDO sets data goals Collecting and storing data from various sources is not sufficient for a sound data strategy. Data strategy begins and ends with business value. Data teams need to understand the types of data they are handling and where it comes from to properly set KPIs and goals for the data. Without well-defined objectives, spearheaded by your CDO, the insights you expect from your data may never come to light.

4. CDO prevents treating data as an afterthought Without a CDO, an organization is likely to assign data-related tasks to the CIO or Chief Digital Officer—someone already weighed down with other responsibilities. As a result, the data strategy may not get the focus it deserves, and may even be treated as an afterthought. If this happens, a company cannot truly become data-driven.

5. CDO ties data strategy to ROI Data-related decisions should be made with ROI in mind. The implementation of modern data architecture technologies—data lakehouses, data mesh, and data fabric—should produce concrete returns on investment to ensure continued support from leadership. Those returns are often measured in business value. The CDO plays a vital role in ensuring data-related investments are delivering results and communicating those results to all relevant stakeholders.

6. CDO takes accountability As with any other part of the business, accountability is needed in the data team. As the team leader, the CDO is in charge of guiding the team and course-correcting whenever it becomes necessary. As such, the CDO ultimately is the person held accountable for data successes and failures, ensuring their visibility for the C-suite.

7. CDO improves competitiveness According to Harvard Business Review, 41% of surveyed CDOs define their success by achieving business objectives. This translates to improved agility and competitiveness, something that all organizations with an eye on the future should be striving for.

Even if your C-suite is a bit crowded, your organization needs a CDO to drive the organization’s data strategy and promote a data culture across all the departments. Without a dedicated, accountable data leader, you may still derive insights from the data you collect, store, and analyze. But you will need a CDO to ensure the organization reaches the full potential of those insights. Watch this video to find out how one CDO helped to transform their business.

The post Think Your Company Doesn’t Need a Chief Data Officer? Here Are 7 Reasons Why It Does appeared first on Cloudera Blog.

View Details

The world is awash with data, no more so than in the telecommunications (telco) industry. With some Cloudera customers ingesting multiple petabytes of data every single day— that’s multiple thousands of terabytes!—there is the potential to understand, in great detail, how people, businesses, cities and ecosystems function. This information is essential for the management of the telco business, from fault resolution to making sure families have the right content package for their needs, to supply chain dashboards for businesses based on IoT data.

The world has changed—business and people are connected! Access and the exchange of data is critical for managing the operations in many industries. For example in retail, connections between customers and stores/distribution centers allow the businesses within the ecosystem to understand who’s shopping, where they are shopping and what they are buying. In government services keeping citizens safe and protected— awareness and real-time data access is key. In the automotive aftermarket industry, doing over the air updates for connected vehicles ensures performance of a vehicle is top of mind. The telco industry is at the heart of managing the communication networks that make this happen. Set against this, of course, are the demands of privacy and consumer protection, as well as data security. The fact remains, however, that carriers have a latent asset that is potentially of great use to third-party businesses. The question is how to monetize the data assets enabling connected businesses and people?

Real-world data monetization opportunities in telcoPart of the assessment begins with identifying uniqueness. For telco businesses—especially wireless telecommunications—a key element of uniqueness is location within a geography.Understanding in great detail how traffic flows, how pedestrians move in a city, over time, is extremely important for city planners, retailers, real-estate executives, and infrastructure projects. Recognizing anomalies in established patterns can be valuable for policing and security services, as well as providing interesting insights such as advance warning of traffic incidents. Digital out-of-home advertising is an increasingly programmatic domain, and location information can help to drive better targeting to public signage.

Online, of course, the data set is even more rich. Many telcos have extensive in-country content assets, beginning with their online retail store, but often extending into rich, paid-for content such as sports, movies, and gaming. Modeling consumption patterns in these environments can provide insights for advertisers, while combining those insights with in-home set-top-box activity can further enrich the model.

The exponential growth of IoT and more devices being connected to the internet further creates an opportunity to model behavior indoors—whether footfall in commercial buildings, dwell patterns in retail stores, or in home occupancy. This data can provide rich opportunities for advertisers, store managers, and content partners interested in developing new offerings for the home, and countless other consumer businesses, like household appliance manufacturers offering warranty support, or utilities companies like water, gas, and electricity.

Laying the groundwork for management and monetizationIn order to generate a return from data, the design of the business starts with the market—as described above—identifying segments and targets where the data yield will be monetized. Generally, this should fit into a B2B portfolio of “conventional” telecommunications offerings such as connectivity, customer premise equipment (CPE), and other devices. Next—working backwards—collaboration tooling such as dashboards and developer environments like those on Amazon Web Services (AWS), are also key components to consider. Underneath all of this are the data assets themselves, with appropriate governance (e.g. masking and anonymization to protect privacy concerns), security (user access control), and comprehensive observability (for billing purposes), all of which come as standard features in Cloudera SDX (Shared Data Experience). Ultimately, the data management must be well governed, robustly persisted, across a hybrid data cloud, allowing for maximum utilization of infrastructure resources in the public cloud, such as AWS, or onprem where regulations or client demand requires it.

Internal and external data productsTelcos enjoy a level of intimacy in their customer relationships that few other organizations can rival. Combined with the fine-grained levels of detail about their customer’s location, mobility, preferences, and behaviors derived in near-real time from the network, telcos are uniquely positioned to capitalize on the power and potential of their data.

In addition to the direct internal and external data monetization opportunities outlined above, telcos can leverage their cloud-based IT architectures to act as “data brokers.” This means ingesting and combining first- and third-party data sources into their analytical infrastructure to create enriched, monetizable data products that can be leveraged by internal stakeholders and marketed externally to drive a wide variety of use cases and capabilities cross vertically, across any number of value and supply chains.

Building your data monetization strategy on Cloudera and AWSAWS provides several products and solutions to support this approach. AWS clean rooms overcome many of the challenges associated with data sharing and privacy protection by providing an analytics service that helps customers and their partners more easily and securely analyze and collaborate on their collective datasets—without sharing or revealing the underlying data.

Organizations can now create secure data clean rooms in minutes, and collaborate with the Cloudera Data Platform (CDP), synergistically combining strengths to provide a robust and comprehensive solution for telcos seeking to monetize their data through advanced data management and analytics capabilities. CDP seamlessly integrates with AWS’s scalable cloud infrastructure and services, allowing telcos to efficiently collect, store, process, and analyze vast amounts of data. Leveraging Cloudera’s powerful data management tools alongside AWS’s scalable computing resources, storage options, and specialized services can optimize data-driven insights, enabling informed decision-making, innovative applications, and streamlined data monetization strategies.

Data can be shared internally and with third parties via AWS Data Exchange (ADX), allowing telcos to seamlessly discover, subscribe to, and integrate valuable data sources directly from the data lake—for example cell tower coverage or geospatial data sets. Deeper insights and additional context can be introduced through integration with Amazon Marketing Cloud (AMC). This potentially enables Telcos to combine their knowledge of their customer base with anonymized and segmented retail-based insights, such as product browsing and buying behaviors derived from amazon.com to drive more effectively targeted campaigns.

Once these layers are in place, the data offering can be made available as a persistent, subscription-based offering, or as a consumption-based API, exposing well-governed and compliant data resources to a market hungry for insights.

Cloudera and AWS collaborate to help telcos reap cost savingsCloudera and AWS are collaborating to address telecommunications industry challenges all over the world. For example, we can provide a customer-specific analysis of deploying hybrid CDP to demonstrate cost savings with public cloud vs data center overhead. We can show organizations how to use intelligent storage tiering solutions from AWS combined with Cloudera CDP to minimize the cost of operations.

Also, with a hybrid architecture, Cloudera and AWS can show organizations how to move less sensitive data into the public cloud while maintaining an on-prem presence under the umbrella of a single data platform—that’s a powerful capability for a transforming industry. Based on all of this, organizations can build future-proof LLM and generative AI solutions that can transform the employee and customer experience.

Learn more about how Cloudera and AWS are working together .

The post Telecommunications Data Monetization Strategies in 5G and beyond with Cloudera and AWS appeared first on Cloudera Blog.

View Details

I started my current career path with Hortonworks in 2016, back when we still had to tell people what Hadoop was. Once I got to work with all the amazing open-source Apache tools I was hooked. I found Apache NiFi especially interesting. Soon after, I became a huge fan of Apache Kafka. Coupled with amazing […]

The post A Talented Team, Innovative Technology, and The Opportunity to Grow. There Is No Place Like Cloudera appeared first on Cloudera Blog.

View Details

Technology Spotlight

The post Revolutionize your data experience with Cloudera on private cloud appeared first on Cloudera Blog.

View Details

With the emergence of new creative AI algorithms like large language models (LLM) fromOpenAI’s ChatGPT, Google’s Bard, Meta’s LLaMa, and Bloomberg’s BloombergGPT—awareness, interest and adoption of AI use cases across industries is at an all time high. But in highly regulated industries where these technologies may be prohibited, the focus is less on off the shelf generative AI, and more on the relationship between their data and how AI can transform their business.

With AI, financial institutions and insurance companies now have the ability to automate or augment complex decision-making processes, deliver highly personalized client experiences, create individualized customer education materials, and match the appropriate financial and investment products to each customer’s needs. It’s the most revolutionary technological development in at least a generation.

But it’s also fraught with risk. Institutions must design AI systems that are not only transparent, reliable, fair, and accountable, but also comply with privacy and security requirements, as well as align with human values and norms. This June, for example, the European Union (EU) passed the world’s first regulatory framework for AI, the AI Act, which categorizes AI applications into “banned practices,” “high-risk systems,” and “other AI systems,” with stringent assessment requirements for “high-risk” AI systems. Under the terms of the AI Act, “high-risk” AI systems require a compulsory self-assessment by providers, with certain critical applications (like AI used in medical devices) also subject to review under existing EU regulations.
Given the complexity of the datasets used to train AI systems, and factoring in the known tendency of generative AI systems to invent non-factual information, this is no small task.

There’s also the risk of various forms of data leakage, including intellectual property (IP) as well as personally identifiable information (PII) especially with commercial AI solutions. This puts the onus on institutions to implement robust data encryption standards, process sensitive data locally, automate auditing, and negotiate clear ownership clauses in their service agreements. But these measures alone may not be sufficient to protect proprietary information.

The AI Moment in ContextAll extant AI solutions are “narrow” in the sense that they cannot approximate or surpass the cognitive capabilities of human beings: they’re unable to reason, reflect, or imagine, and they aren’t capable of genuine emotional understanding. That said, Generative AI and LLMs appear to do all of these things, producing original, “creative” outputs by learning from input data. ChatGPT, Bard, LLaMa, and BloombergGPT rely on a new type of neural network architecture, called a transformer model, which uses a special form of weighting to capture relationships and context across different parts of a sentence or sequence.

The reality of LLMs and other “narrow” AI technologies is that none of them is turn-key. Financial institutions implementing AI must grapple with the challenge of reshaping their core business process and application workflows, along with the difficulty of transforming their corporate cultures.

Perhaps the biggest challenge of all is that AI solutions—with their complex, opaque models, and their appetite for large, diverse, high-quality datasets—tend to complicate the oversight, management, and assurance processes integral to data management and governance. The way to manage this is by embedding data integration, data quality-monitoring, and other capabilities into the data platform itself, allowing financial firms to streamline these processes, and freeing them to focus on operationalizing AI solutions while promoting access to data, maintaining data quality, and ensuring compliance.

The Danger of Black-Box AI SolutionsWe believe the best, most pragmatic solution for AI in financial services and insurance is what we call–“Trusted AI.” But before more is said about what this is, let’s walk through some of the issues that a financial institution needs to take into account when it considers a commercial AI service.

First, there’s the challenge of protecting one’s business-critical IP—e.g., proprietary data, business strategies, methodologies, etc. Storing or processing this information in an external AI service could inadvertently leak or expose these critical assets.

Second, there’s the problem of safeguarding PII, transaction records and other types of sensitive or confidential data. Even when backed by robust security measures, an external AI service is a tempting, outsized target for potential security breaches: each integration point, data transfer, or externally exposed API becomes a target for malicious actors.

Third, there’s the “black-box” element: viz., the design and behavior of a commercial AI service’s algorithms is usually proprietary, not to mention intentionally obscured. This lack of transparency makes it difficult for financial institutions to thoroughly vet and validate the AI service’s outputs against regulatory standards.

Fourth, AI-powered automation is most transformative when it’s embedded throughout an institution’s business processes and workflows. Because AI is so tightly interpenetrated with core processes, standardizing on a commercial AI service could lead to vendor lock-in, stifling innovation, placing significant power in the hands of a single vendor, constraining the institution’s ability to negotiate terms and prices—and ceding control over future decision-making.

Introducing “-Trusted AI”Enter “Trusted AI.” Trusted AI is the ethos behind Enterprise AI across the organization, including Generative AI and LLM capabilities. Models are trained on a financial institution’s secure data, deployed and run internally, on their own infrastructure—or externally, in virtual private cloud (VPC) infrastructure, in the case of non-sensitive workloads. This not only ensures greater control and flexibility, but also helps safeguard the integrity of proprietary assets, like IP, while also providing enhanced protection for sensitive data while enforcing the rigorous security and compliance standards unique to the financial sector. And because an open-source AI model’s code is public, its inputs and outputs are understandable and explainable, ensuring transparency.

While it’s true that commercial providers currently dominate the AI space, the history of open-source software suggests this dominance will diminish—in this case, quite rapidly. Open-source AI isn’t just quickly catching up to OpenAI, Google, Meta, and Microsoft: mere months after ChatGPT’s debut, open-source AI models are almost as fast, in addition to being more customizable, affordable, and transparent. Just like the open-source system, database, and machine learning (ML) technologies of the past, AI models are narrowing the gap with proprietary alternatives at an incredibly rapid pace.

There’s one more thing. The foundation of Trusted AI is a hybrid data platform that is able to present a unified view of the data that’s distributed across a financial institution’s on-premises and multi-cloud environments. This platform uses AI and automation to abstract the complexity of data access, movement, integration, and analysis. By embedding intelligence at the data platform-level, it becomes possible to accelerate the pace at which financial institutions can operationalize AI solutions.

The combination of built-in data management and governance capabilities provides a solid foundation for firms to embed Trusted AI across their operations. In this blog series, well dive into the advantages of Trusted AI and the broader ramifications of AI adoption, exploring how financial institutions can bootstrap and evolve their AI strategies, from initial steps to what mature AI adoption looks like.

Let’s kick things off with a proposed Maturity Model for AI in Financial Services:

An AI Maturity Model for Financial Services1- Foundational AI Integration

At this foundational stage, financial institutions begin by prioritizing open-source AI tools, understanding that commercial and cloud solutions can expose them to risks. The foundation of this stage is a hybrid data platform that’s capable of seamlessly integrating data across the institution’s landscape, while automating or accelerating common tasks.

• Deploy a hybrid data platform. Leverage open-source technologies on a hybrid data platform that automates or accelerates tasks like data ingestion, transformation, and schema design, ensuring that sensitive data and IP remain secure wherever the data is located.
• Basic Process Automation. Start with the low-hanging fruit, using open-source ML/AI to automate basic tasks, like transaction classification, basic fraud detection, daily reconciliation processes, and “first-level customer support responsiveness.
• Leverage open-source LLMs to design chatbots & digital Assistants, deploying 24/7 customer support bots built on open LLM frameworks.
• Train and upskill employees. Initiate basic AI training programs for staff. Develop workshops, e-learning modules, and hands-on sessions designed to familiarize employees with the fundamentals of AI and its applications within the finance sector.

2- Intermediate AI Integration

At this level, financial institutions and insurance companies build on top of a foundational hybrid data platform to tap deeper into AI’s potential, focusing on enhancing the user experience, promoting data-driven decision-making, and implementing robust cybersecurity layered defenses.

• Automate loan and credit decisioning. Go beyond traditional credit scoring, using AI to examine customer behaviors to predict creditworthiness and identify default behaviors. Models must be fair, responsible and remove bias to ensure that AI systems don’t inadvertently discriminate.
• Enhance the Customer Experience. Accelerate and/or automate routine processes like KYC verification, speeding up loan or underwriting approvals, and ensuring error-free account setups.
• Use AI to automate financial crime prevention. Create basic AI systems to detect potential fraudulent activities, monitor online financial activities, and discover system loopholes.
• Systematize governance. Leverage the hybrid data platform’s built-in capabilities to automatically monitor data quality levels and align with regulatory standards. Formalize rules, standards, and best practices that guide how data is to be managed and used.
• Create core feedback mechanisms. Establish initial channels for user and employee feedback to refine AI applications. For example, implement embedded feedback options in AI-driven apps, analyze responses using open-source Natural Language Processing (NLP) tools for continuous refinement.
• Facilitate communication between stakeholders. Enable reporting to internal teams about the statuses of AI projects. Create dashboards that highlight project milestones, challenges, and advancements, ensuring stakeholders stay informed and provide input.

3- Advanced AI Integration

At this stage of adoption, financial institutions and insurance companies engage more intensively with AI and its capabilities, extracting more valuable insights from data. The hybrid platform’s automation capabilities are crucial in this stage, allowing for more rapid adaptation and richer analytics.

• Push predictive analytics to optimize operations and enhance profitability. Leverage AI to analyze previously untapped data sources, such as social media sentiment, geo-location data, and customer feedback. Glean insights into customer behavior and market trends that also correspond to overlooked sales opportunities. Identify activities or factors that directly impact revenue and/or earnings, e.g., loan default rates or customer retention.
• Simplify regulatory compliance. Use NLP to analyze and break down regulatory documents, translating complex legal jargon into actionable tasks.
• AI-ify risk management. Leverage ML/AI to refine risk models, incorporating data from diverse sources, and predicting outcomes based on market sentiment, climate data, etc.
• Even more training and upskilling. Introduce advanced AI training and programs, including hands-on projects that simulate real-world financial scenarios, or mentorship programs hosted by AI experts. Offer opportunities for employees to specialize in specific AI domains, such as fraud detection or predictive analytics, tailored to the institution’s needs.
• Plan to scale for the future. Prepare for higher AI demands, assessing the state of the institution’s infrastructure capacity while taking Into account future data processing needs.
• Formalize ethics and bias testing. Develop and implement automated tests to identify biases in AI models, ensuring that models align with ethical standards and fairness criteria. Third-party audits or reviews add credibility to claims of fairness and transparency.

4- Transformative AI Integration

With a strong open-source foundation and a hybrid data platform fully operational, AI becomes deeply ingrained in an institution’s core processes. Robust security mechanisms, such as IAM and RBAC, ensure that only authorized individuals can access sensitive AI models and data.

• Track market trends. Advanced analytics processing vast data volumes to forecast market trends, currencies, stocks, and investment timings.
• Step up to comprehensive cybersecurity. Invest in AI-powered intrusion detection systems (IDS) or security information and event management systems (SIEM). Use these tools to continuously scan transactional data, user activities, system logs, etc., ensuring a rapid response to data breaches, building trust with stakeholders and customers.
• Transform the Customer Experience, Create highly personalized user experiences, using AI to analyze customer behavior—transaction histories, browsing patterns, and service inquiries—to offer personalized financial advice, product recommendations, and tailored alerts, enhancing the user experience and deepening client engagement.
• Process Automation 2.0. Go beyond basic tasks, automating complex processes and workflows. By now, institutions should achieve significant gains in operational efficiency.
• Create integrated feedback mechanisms. Establish iterative loops with stakeholders for AI model refinement. By collecting and analyzing feedback, institutions can incrementally improve their AI systems, ensuring they remain accurate, relevant, and user-friendly.
• Supercharge communication. Regularly communicate AI strategies, milestones, and future goals not just to stakeholders, but to the organization as a whole.

5- Fully Mature AI Integration

At full maturity, financial institutions and insurance companies realize the power of Trusted AI built on top of a hybrid data platform, accelerating AI operationalization, with Trusted AI embedded across all operations.

• Step up to advanced AI oversight. Benchmark against global best practices and ensure that AI ethics are deeply integrated into all AI initiatives, with robust mechanisms for ongoing review, stakeholder feedback, and rapid adaptation to new ethical challenges. Collaborate with external ethical boards to reinforce the commitment to ethical AI.
• Develop next-gen personalized financial products. Leverage AI to design dynamic financial solutions, like AI-optimized savings plans, predictive investment portfolios, and personalized insurance offerings that adjust in real-time to each customer’s financial situation.
• Practice real-time risk management. Use AI to assess risk in real-time, adjusting portfolios and investment strategies automatically based on global events, market fluctuations, etc.
• Automate wealth management. Offer advanced robo-advisory services, using AI solutions to optimize asset allocation, tax strategies, retirement planning, and other practices.
• Anticipate regulatory changes. Tap the power of AI to model the potential impact of regulatory changes, ensuring that you’re one step ahead in compliance.
• Explore cross-industry integration. Use AI to identify opportunities to partner with retail, real estate, health and other industries to develop and market integrated financial solutions.
• Identify opportunities for environmental, social, governance (ESG) initiatives. AI can assist in assessing and investing in sustainable projects, a growing trend in the finance sector.

ConclusionCloudera is the ideal hybrid data platform for financial institutions and insurance companies seeking to adopt or advance AI initiatives due to our unique combination of robust data management capabilities and advanced analytics tools. With Cloudera’s proven track record in handling large-scale data infrastructures, Cloudera offers the reliability and security necessary for the sensitive and complex data environments in which financial institutions operate. Cloudera’s ability to seamlessly integrate and process diverse data sources, combined with its comprehensive suite of machine learning and AI tools, empowers institutions to harness the power of generative AI for predictive modeling, risk assessment, fraud detection, and personalized customer experiences. With Cloudera, financial institutions can unlock valuable insights from their data while adhering to strict regulatory standards, ultimately gaining a competitive edge in the rapidly evolving landscape of AI-driven finance.

Find out more about CDP, modern data architectures and AI here.

The post How Financial Services and Insurance Streamline AI Initiatives with a Hybrid Data Platform appeared first on Cloudera Blog.

View Details

At Cloudera, we’re known for making innovative technological solutions that drive change and impact the world. Our mission is to make data and analytics easy and accessible to everyone. And that doesn’t end with our customer base.

We also aim to provide equitable access to career opportunities within data and analytics to the workforce of tomorrow.

The Teen Accelerator Program is an inspiring step in shaping a brighter future.In July, Cloudera closed its second year of the Teen Accelerator program – a program hosted by the Diversity Equality + Inclusion team, run in partnership with The Boys and Girls Clubs of Middle Tennessee and of the Peninsula. Its objective is to level the playing field for high school students to gain access to careers in technology by offering a unique blend of personal and professional development opportunities that can transform their lives. To date, we are proud to have served 97 students.

To learn more about the Teen Accelerator Program, check out How Cloudera Is Opening Doors for Underserved Youth.

A catalyst for growth, confidence and a lot of fun along the way.The impact has been immeasurable. As we celebrate closing out the program with a 6 week internship, the students have grown in ways we could not have anticipated.

What we learned from them is that the biggest transformations can come from what may seem to us as the simplest of experiences. Things like working across different time zones, shadowing customer calls, hearing about the career paths of various employees, helping plan events, facilitating a meeting, and learning to build LinkedIn profiles, left lasting impressions.

From one intern: “Being able to work at Cloudera was one of the best times of my life. Everyone was so warm and was always willing to help you. I never expected to have so much fun but this internship exceeded my expectations. Having the chance to take on responsibility with all tasks I handled was refreshing and allows me to have skills that I can use for the future. Additionally, Cloudera gave me the chance to step out of my comfort zone and grow as a person. This summer allowed me to prepare for my future but it also gave me the opportunity to create fond memories.”

A resounding sentiment from all involved is they’ve seen a leap in confidence within the students. One Boys & Girls Club staff member shared their perspective.

“2 years ago, this student came to the clubhouse as a super shy individual, hair covered their face and they never looked up or said much. Fast forward to now, they’re demonstrating SO much confidence and walking with a different beat in their step. This student is consistently reaching out to the staff to share updates and highlights from this internship, and is loving their experience. The Teen Accelerator program has helped provide a major opportunity to increase their confidence and overall engagement!”

Carolyn Bai, who facilitates the program, shares her thoughts on its impact.

“I am incredibly proud of the students’ ambition to push themselves outside of their comfort zone to pursue things they never would have thought to before.”

Growth is a two way street. The students aren’t the only ones who find value and growth from this program.

Behind the scenes, Clouderans play a pivotal role in this initiative. Employees volunteer their time and expertise serving as mentors and intern managers, often for the first time. As the students gain experience working in the tech industry, Clouderans gain experience being people managers and helping lead the charge to reach shared goals.

“I gained the experience of being responsible for someone’s professional development. It was amazing to see how much they could achieve during their internship, and how strong a mentor/mentee relationship should be to assist each other in achieving our goals.” One intern manager shared.

By participating in the program, Clouderans cultivate valuable leadership skills all while making a positive impact. It’s a relationship that is mutually beneficial to all involved!

A shining example of how corporations can change the trajectory of today’s youth.The results are impressive, the experience is fun and the impact is lasting.

Cloudera’s Teen Accelerator Program is creating a ripple effect in the futures of the students that participate. It’s more than just another internship program. It’s a tool to help bridge the gap for underserved youth as it aims to inspire students to reach beyond their imagination.

We recognize that in order to create a more equal and diverse workforce in the future, we need to invest today in students, especially those that may not have access to resources and guidance. Stephanie Han, a co facilitator, shared that “with continued leadership support, I am excited to continue on this journey and see all that year 3 brings!” and it seems safe to say that’s a sentiment everyone can agree on.

The post Expanding Possibilities: Cloudera’s Teen Accelerator Program Completes Its Second Year appeared first on Cloudera Blog.

View Details

Overview

The post LLM AMP vol.1 appeared first on Cloudera Blog.

View Details

Machine Learning (ML) is at the heart of the boom in AI Applications, revolutionizing various domains. From powering intelligent Large Language Model (LLM) based chatbots like ChatGPT and Bard, to enabling text-to-AI image generators like Stable Diffusion, ML continues to drive innovation. Its transformative impact advances multiple fields from genetics to medicine to finance. Without […]

The post How to Ensure Supply Chain Security for AI Applications appeared first on Cloudera Blog.

View Details

Cloudera Partner Network Introduces New Sales, Technical and Industry Accreditations

The post New Accreditations for Cloudera Partners appeared first on Cloudera Blog.

View Details

Introduction The snapshots feature of the Apache Hadoop Distributed Filesystem (HDFS) enables you to capture point-in-time copies of the file system and protect your important data against corruption, user-, or application errors.  This feature is available in all versions of Cloudera Data Platform (CDP), Cloudera Distribution for Hadoop (CDH) and Hortonworks Data Platform (HDP). Regardless […]

The post HDFS Snapshot Best Practices appeared first on Cloudera Blog.

View Details

Embracing change with passion: The road back to event marketing and a blueprint for joy

The post #ClouderaLife Employee Spotlight: Shallan Miller appeared first on Cloudera Blog.

View Details

June 21st was a busy day for Cloudera Cares in the UK. Not only did we host the Evolve London Flagship event with a first of its kind, volunteer component, we also flew the Cloudera flag at a Cloudera Cares event with Mission Motorsport. Hear from this month’s volunteer spotlight, Paul Wooding about his day volunteering at two of Cloudera’s impactful UK-based events.

The post Volunteer Spotlight: Big Day in the UK! appeared first on Cloudera Blog.

View Details

In their effort to reduce their technology spend, some organizations that leverage open source projects for advanced analytics often consider either building and maintaining their own runtime with the required data processing engines or retaining older, now obsolete, versions of legacy Cloudera runtimes (CDH or HDP). However, both of these options are associated with substantial […]

The post Why Reinvent the Wheel? The Challenges of DIY Open Source Analytics Platforms appeared first on Cloudera Blog.

View Details

Sahana Nag Story

The post Career & Motherhood: How Cloudera Helped Me Transition Into Motherhood With Twins appeared first on Cloudera Blog.

View Details

Introduction Ozone is an Apache Software Foundation project to build a distributed storage platform that caters to the demanding performance needs of analytical workloads, content distribution, and object storage use cases.  The Ozone Manager is a critical component of Ozone. It is a replicated, highly-available service that is responsible for managing the metadata for all […]

The post Boosting Object Storage Performance with Ozone Manager appeared first on Cloudera Blog.

View Details

Discover Hidden Insights With Cloudera Observability

The post Unlock The Full Potential Of Hive appeared first on Cloudera Blog.

View Details

CDO Spotlight

The post One Big Cluster Stuck: Environment Health Scorecard appeared first on Cloudera Blog.

View Details

Introduction For more than a decade now, the Hive table format has been a ubiquitous presence in the big data ecosystem, managing petabytes of data with remarkable efficiency and scale. But as the data volumes, data variety, and data usage grows, users face many challenges when using Hive tables because of its antiquated directory-based table […]

The post From Hive Tables to Iceberg Tables: Hassle-Free appeared first on Cloudera Blog.

View Details

Iceberg is an emerging open-table format designed for large analytic workloads. The Apache Iceberg project continues developing an implementation of Iceberg specification in the form of Java Library. Several compute engines such as Impala, Hive, Spark, and Trino have supported querying data in Iceberg table format by adopting this Java Library provided by the Apache […]

The post 12 Times Faster Query Planning With Iceberg Manifest Caching in Impala appeared first on Cloudera Blog.

View Details

Apache Impala and Apache Kudu make a great combination for real-time analytics on streaming data for time series and real-time data warehousing use cases. More than 200 Cloudera customers have implemented Apache Kudu with Apache Spark for ingestion and Apache Impala for real-time BI use cases successfully over the last decade, with thousands of nodes […]

The post Integrating Cloudera Data Warehouse with Kudu Clusters appeared first on Cloudera Blog.

View Details

The recent failures of regional banks in the US, such as Silicon Valley Bank (SVB), Silvergate, Signature, and First Republic, were caused by multiple factors. To ensure the stability of the US financial system, the implementation of advanced liquidity risk models and stress testing using (MI/AI) could potentially serve as a protective measure. Technology alone […]

The post How to Manage Risk with Modern Data Architectures appeared first on Cloudera Blog.

View Details

Making connections to help customers and community.

The post #ClouderaLife Employee Spotlight: Peyton Kettering, Account Executive appeared first on Cloudera Blog.

View Details

During the COVID-19 pandemic, telcos made unprecedented use of data and data-driven automation to optimize their network operations, improve customer support, and identify opportunities to expand into new markets. This is no less crucial today, as telcos balance the needs to cut costs and improve efficiencies while delivering innovative products and services. The way to […]

The post Five Ways A Modern Data Architecture Can Reduce Costs in Telco appeared first on Cloudera Blog.

View Details

CDO Spotlight

The post One Big Cluster Stuck: The Right Tool for the Right Job appeared first on Cloudera Blog.

View Details

Six ways a hybrid data platform modernizes financial services

The post Do You Know Where All Your Data Is? appeared first on Cloudera Blog.

View Details

To celebrate Global Accessibility Awareness Day, Cloudera’s Capable ERG volunteered with Be My Eyes, a free app that connects Blind and low-vision people with 1:1 support at a moment’s notice to help solve daily tasks.

The post Volunteer Spotlight: Clouderans Volunteer with Be My Eyes appeared first on Cloudera Blog.

View Details

Australia and New Zealand Banking Group (ANZ), one of Australia’s Big Four banks and one of New Zealand’s top banks, offers commercial and retail banking and financial services from more than 1,100 branches and offices. The bank offerings include standard deposit and lending services, credit cards, wealth management, agribusiness finance and insurance, and foreign exchange […]

The post ANZ Bank’s Advances Data-Driven Analytics With Cloudera appeared first on Cloudera Blog.

View Details

CDO Spotlight

The post One Big Cluster Stuck: Platform Health appeared first on Cloudera Blog.

View Details

By now, almost everyone across the tech landscape has heard of the Zero Trust (ZT) security model, which assumes that every device, application, or user attempting to access a network is not to be trusted (see NIST definitions below). But as models go, the idea is easier than the execution. To provide guidance to federal […]

The post How Cloudera Supports Zero Trust for Data appeared first on Cloudera Blog.

View Details

OpeningIncreased costs and wasted resources are on the rise as software systems have moved from monolithic applications to distributed, service-oriented architectures. As a result, over the past few years, interest in observability has seen a marked rise. Observability, borrowed from its control theory context, has found a real sweet spot for organizations looking to answer the question “why,” that monitoring alone is unable to answer.

Gartner highlights applied observability as one of the top strategic technology trends for 2023, pointing to its impact on business: “By 2026, 70% of organizations that successfully applied observability will achieve shorter latency for decision making, enabling competitive advantage for target business or IT processes.”

Enter Cloudera Data Platform (CDP): a platform distributed across not just one but multiple infrastructures as it is deployed across private and public clouds. Keeping tabs on everything going on in such a complex ecosystem (containers, services, and more) through monitoring alone is no mean feat and impacts areas like resource utilization and performance, as well as SLAs and SLOs. With data and analytics literally all over hybrid cloud, financial governance is hard, plain and simple. Allocating spend to individual teams or use cases becomes a forensic accounting exercise and predicting future spend one that involves a crystal ball. Or rather: keeping tabs was no mean feat; financial governance was hard. Those challenges are now in the past for CDP customers.

Introducing Cloudera ObservabilityWe are both pleased and proud to announce that Cloudera Observability is now available to Cloudera customers. Cloudera Observability is CDP’s single pane of glass observability solution, continually discovering and collecting performance telemetry across data, applications, and infrastructure components running in CDP deployments on private and public clouds. With advanced, intelligent analytics and correlations, it provides insights and recommendations to address tricky issues, optimize costs, and improve performance.

An extensible solution, Cloudera Observability provides a range of capabilities to achieve its objectives, all accessible through a single interface:

• Active system monitoring: Provides monitoring and insights through both current and historical analysis of infrastructure usage, services, workloads, and users.
• Service health monitoring: Delivers continuous service monitoring to identify bottlenecks thanks to automatic correlation of events and logs.
• Workload optimization: With intelligent performance tuning recommendations and continuous refinement of tuning rules based on usage information.
• Financial governance: With cost management across CDP to avoid budget overruns and allow capacity projections for planning purposes.
• Self-service analytics: For complete impact analysis and visibility across deployments as well as automatic actions for the capabilities mentioned in the previous points.

With this insight into CDP deployments, organizations can exclude themselves from the 82% of organizations that know they incur unnecessary cloud costs and make the most effective use of those resources (or any resources for that matter). What’s more, knowing what team uses how much for which workloads and what the cost impact of this is, gives the opportunity to escape the 86% of companies who are unable to get a global view of spending on infrastructure. All this insight, and more, is now available in Cloudera Observability Premium.

Free expedited support for allCloudera Observability Essential, available at no additional cost, provides key troubleshooting support to keep your workloads at their healthiest. As such, it provides tremendous value for all CDP customers, since it bundles the power of Cloudera Support to resolve issues faster. With that baked in experience, Cloudera Observability helps you perform root-cause analysis and provide prescriptions to cure the problem. What’s more, when further help is needed, Cloudera Observability Essential will streamline the exchange of CDP system configuration and operation data between deployments and Cloudera Support. That saves time by reducing the multiple to-and-fro conversations, expediting support, and letting customers get back to creating value and insight from their data faster.

Benefits all aroundCloudera Observability outshines traditional observability tools by bottling the collective wisdom and decades worth of experience of Cloudera Support, product engineering, and professional services in a single solution, empowering users to monitor, understand, and optimize their CDP deployments with ease and confidence.

Because no one knows Cloudera like Cloudera.

As a result, customers benefit from:

• Improved platform performance: Thanks to proactive identification and optimization of inefficient processes and workloads.
• Better adherence to SLAs and SLOs: Through predictability for both business-critical workloads and dynamic, self-service analytics.
• Financial transparency: With full insight into resource consumption across hybrid cloud deployments.
• Cost control and optimization: Sith the ability to allocate data and workload costs to teams as well as projects, and more efficient use of resource.
• Enhanced Cloudera support experience: Through secure, real-time observability sharing from deployed CDP instances to Cloudera support experts when cases arise.

All this, ultimately, makes for more productive end users that can focus on driving insight and value from their data, rather than trying to tweak and tune their analytics. That’s Cloudera Observability’s role.

SummaryCloudera Observability is available as a free “essential” tier that provides rich insights and root cause analysis as well as expedited support, and also has a feature-rich “premium” tier that delivers deep self-service insight, financial governance, and optimization capabilities.

To learn more about Cloudera Observability, we’ve created this short video. We also invite you to register now for our upcoming Cloudera Now event on June 14 and see Cloudera Observability in action. If you’d like to get started right away, please contact a member of your Cloudera account team or get in touch.

The post Beyond Monitoring: Introducing Cloudera Observability appeared first on Cloudera Blog.

View Details

Results from the “Best in Flow” data pipeline development contest with Cloudera DataFlow in the public cloud

The post Aaand the New NiFi Champion is… appeared first on Cloudera Blog.

View Details

Riding the wave of the generative AI revolution, third party large language model (LLM) services like ChatGPT and Bard have swiftly emerged as the talk of the town, converting AI skeptics to evangelists and transforming the way we interact with technology. For proof of this megatrend look no further than the instant success of ChatGPT, […]

The post Generative AI for the Enterprise appeared first on Cloudera Blog.

View Details

Data asset standardization is the purposeful and carefully planned consolidation of redundant, contradictory reports, processes, and databases into enterprise standards. The proliferation of data assets can have the greatest adverse impact on environmental health; standardization has many health benefits:

• Reduces the likelihood that ill-constructed assets take down processes, nodes, and clusters
• Reduces contention and competition for compute and storage
• Reduces process and service failures and associated troubleshooting effort
• Reduces effort spent maintaining and supporting redundant assets

Although the impacts of data asset standardization on environmental health can be higher than any other category in this series, the business value benefits exponentially outweigh them: standard data definitions, improved data governance, consistent data interpretation, greater data trustworthiness, and improved data-driven decision making. Ideally you are realizing these benefits using Cloudera Data Catalog.

Total data standardization is a multiyear journey and likely unnecessary, but the low hanging fruit is ripe for the picking. We strongly recommend embarking on this journey until returns diminish.

Report StandardizationTake these steps:

1. Inventory reports, including ownership, usage statistics, and report frequency.
2. Target for retirement any reports unused in the last year, then in the last 6 months. Pay particular attention to report frequency as low usage of an annual report may be appropriate.
3. Select a report archival method commensurate with your customer partnership dynamic (we hope you’re not in data purgatory )
   1. Two weeks before, a week before, and the day of the archival, notify report owners as to which reports you intend to archive, allowing them a grace period to object and provide justification for the reports continued existence.
   2. Conversely, archive them without notification and restore a report when anyone shouts about it.
4. Archive targeted reports. In Tableau, we prefer to simply assign report ownership to a system user which prohibits further use while enabling us to easily restore it if requested and justified.
5. Repeat the exercise quarterly. In our experience, 80-90% of reporting inventory can be archived in as little as 2 quarters.
   1. If your visualization tool employs extract jobs, stop them, and note any database archival targets.
6. Occasionally investigate the appropriateness of report refresh rates and negotiate.
7. Over time, consolidate additional assets by grafting heavily used report features and functions into enterprise standard dashboards then retire redundant legacy reports. Admittedly, this is difficult and time consuming work usually undertaken as a means to trusted data, not environmental health.

DB Standardization1. As before, inventory database assets, ownership, refresh frequency, and associated usage statistics. 2. Target temporary/testing databases and user databases owned by former FTE. 3. Communicate far and wide. We’re not as brave as to archive dbs without notification and permission in most cases. We enjoy our jobs and want to keep them. 4. Archive the databases. We usually archive into a common archival database. In our experience, this can reduce 35-55% of production tables. 5. Occasionally negotiate refresh rates and data retention policies with database owners. 6. We strongly recommend taking the multiyear journey to standardize centralized data assets into enterprise standards as much as possible as it can significantly improve data trustworthiness and accurate data-driven decision making.

Pipelines and Jobs StandardizationDatabase asset standardization will identify archival opportunities for (1) the pipeline inventory, here referring to processes which move data from one repository or source to another repository or curated dataset, as well as (2) the jobs inventory, here referring to queries which provide views or persist data within the environment. Standardizing processes is high effort with diminishing returns on environmental health; therefore, begin with processes that:

• Frequently fail
• Are most critical
• Are most frequently updated
• Are the most resource intensive

As always, if you need assistance identifying or executing data asset standardization, engage our Professional Services experts. We did!

The post One Big Cluster Stuck: Data Asset Standardization appeared first on Cloudera Blog.

View Details

During Women’s History Month, Cloudera hosted a fantastic fireside chat featuring Irma Laxamana, Chief Legal Officer for Cloudera, and Cloudera’s CHRO, Amy Nelson. The discussion was wide-ranging from reflecting on career lessons learned, to advice on navigating the workplace. Below are the highlights of the chat.

About Irma Laxamana

Irma is the Chief Legal Officer at Cloudera leading a global team of lawyers and legal professionals supporting all areas of the business. She has been a practicing attorney for over 22 years, mostly at in-house legal departments at technology companies like Wind River and VMWare as well as smaller startups. In 2013, she joined Hortonworks as one of its first 3 attorneys, and there she helped to grow and scale the legal function through its IPO and hyper-growth stages. When Hortonworks merged with Cloudera, she joined the combined company, and in late 2021, she was promoted to the role of CLO.

About Amy Nelson

Amy joined Cloudera eleven months ago as the company’s Chief Human Resources Officer (CHRO), leading all Human Resources (HR) functions and the Diversity Equity and Inclusion (DE&I) organization. Amy has been in HR for almost 30 years. She spent 20 years at Dell Inc. in several Human Resources Business Partners (HRBP) and Talent Acquisition leadership roles before joining Quest Software which had been divested by Dell as part of a private equity spin-off company. There she led the company’s HR integration efforts and oversaw several HR functions, including the HRBP and Talent Acquisition organizations.

What was the best piece of advice you have received in your career that was unexpected or you didn’t think a lot about at the time, but was of impact later?Irma

While a good work ethic and investment in the details are definitely important and will help you build a good reputation, as you move up into leadership roles in fast-moving businesses, striving for perfection in all instances can be a hindrance. “The best piece of advice I got from a trusted manager was that although you may want to strive for perfection, don’t wait for it in every situation before you move forward and act. As I moved up in levels, this became more and more important as with more responsibilities, decisions for resolutions are tougher. So, I give it the best I can, being as thoughtful as I can, but I do not wait for everything to fall into place before I make a decision.”

Amy

Be open to new opportunities even if it’s a stretch role. “When I was at Dell working in talent acquisition, I didn’t have any plans to move into other areas of HR, but I remember having a skip-level career conversation with the Senior Vice President of HR. He advised me to be more open to and proactively consider growth opportunities. He also told me not to get too comfortable in my current role. This planted the seed for me to look for other opportunities to broaden my skill set. Several months later, I accepted another role outside of talent acquisition which helped moved my career forward.”

When people come to you looking for advice, what is the skill you emphasize first? Amy

Be resilient. Resilience is a mindset that can be very powerful in an ever-changing work environment. “Market conditions are changing; companies are changing. We are shifting all the time, having adaptability as a skill set is only going to make you more successful in your role.”

Irma

Be self-aware and have an eye for the big picture. “Being self-aware of your strengths allows you to be your best with your work but being self-aware of how you are perceived by others also helps you – in how best to partner with others and interact to collaborate for outcomes. Also, it’s easy to focus on what’s in front of you, but as you grow your level of responsibilities, building business and strategic acumen is invaluable. Look at the bigger picture and ask yourself ‘What am I doing to help the business move forward?’ or ‘What can I do new or different to help the business move forward?’”

With gender disparity in the technology and legal fields, as women, how have you navigated this space?Amy

Build a network that includes mentors and sponsors. “What’s been important for me has been building a network. It’s one of the most crucial things you can do to really get ahead in your career. It’s more than just meeting people, it’s about building respectful relationships and creating those connections. Establishing mentors and senior-level sponsors can drive your career advancement. Make sure you surround yourself with people who would mention your name in a room full of people. I wouldn’t be here today without mentors and sponsors.”

Irma

Build a network, including by participating in professional organizations to seek help and to help others. “I can’t emphasize enough the importance of finding the right network for yourself and participating in it. Coming into the VP level and leadership level in legal, I more proactively got involved in groups like The Women’s General Counsel Network, and The Silicon Valley General Counsel Network where you find you’re not alone in the work that you’re doing and the problems you’re trying to solve, and the blind spots you have, others have. There are people that can help you, and you can also find ways to help others.”

How do you navigate work-life boundaries especially as you progress through your career and hold more responsibility and accountability? Irma

Knowing your priorities and your purpose and knowing they can change over time. “It can be very difficult, and you have to be very intentional about it. Having the right support system around you and being aligned as a family or household is important. Always be clear about your purpose and know that it can change over time. For me, at first, it was building a strong foundation for my family. Later it was about finding self-fulfillment in my career, learning and growing, feeling like I was challenging myself. And while learning is a continuous journey, now I explore what I can do to be a sponsor or mentor.”

Amy

It’s a skill and a discipline. Saying yes is easier than saying no. “Work-life balance is the most important skill any professional can have, but also the most difficult to learn. Even though I believe in a strong work ethic, and will always do what it takes to get the job done, it is also important to resist the urge to be continuously working. Having a healthy work-life balance means leaning on my team and peers at times. Even though I have a tendency to want to take it all on myself, I’ve learned that is not the way to maintain a healthy work-life balance.”

Special thanks to Irma and Amy, for participating in this important discussion.

The post #ClouderaLife Women’s History Month Fireside Chat, Highlights appeared first on Cloudera Blog.

View Details

Originally from Brazil, Alex previously lived in Chile and now lives in Spain.

During his time living in Latin America in early 2016, Alex saw what he describes as a “knowledge gap”—seeing the way skills, content and expertise are shared in an open, friendly way at conferences in the US, Alex wanted to replicate that in Latin America.

To address this gap, Alex started planning meetups. “Our first meetup started in a Starbucks coffee shop, with just a few people around a table. Quickly, they transformed into gatherings with up to 70 people in a designated room. My goal was to create a space for data enthusiasts in Latin America to share knowledge with others.” Alex estimates he ran over 50 meetups with over 1,000 attendees in total, allowing people to connect over technology and meet a variety of vendors and people working in the space.

“It was a lovely experience to share my knowledge, get in touch with people, and train others! I saw the way the open source community works and how people engaged to share professional experiences, and that motivated me to do the same thing in Latin America.”

Using his contacts at a few universities and technology schools in Chile, he began to promote a space where data enthusiasts could connect, while also offering training opportunities for professionals and students to gain employment and therefore diminish the technical gap, starting with hands-on workshops showcasing the technology. But Alex likes a challenge, and wanted to do something bigger—a conference. His friend Max said, “Are you sure about that?” And Alex definitively answered “Yes!”

Exclusively using their contacts and networks, Alex and Max began to organize their first event. “That first event, we had 400 attendees at a one-day conference, combining hands-on labs with delivered content. The idea was to bring local people from local businesses and universities together to learn from that content. It was amazing! It was really hard work because we had no budget, but we achieved our goal—to bring knowledge and people together.”

With the revenue generated from that first conference, Alex and team donated to a local nonprofit supporting technology education for women and girls, and got to work planning their second event.

In partnership with Telefónica, the 2019 conference brought in 600 attendees, in addition to vendors such as Google and AWS. Alex and Max teamed up with Federico to launch a big data conference chapter in Buenos Aires, Argentina, with over 400 attendees. Even as the reach and input grew, the core mission of the events remained the same: to share knowledge and bring data enthusiasts together. “The people that attended these conferences started out with zero knowledge about data, and now they have not only data literacy but a network of peers. These attendees would never otherwise have the chance to hear from people in the industry about achievements, failures, and lessons learned,” he says.

“Their feedback was the fuel to motivate us to continue giving back more and more. These events are the biggest professional achievements in my career.”

Unfortunately, the pandemic put future conferences on pause (although they have since started again!), and during this time Alex moved to Spain. He started delivering more content on data to local universities. This culminated in an invitation from the School of Management in Madrid asking Alex to deliver data and open-source content and share professional experiences with an international audience of students.

“For some people, this helped them to realize that data is their passion, something they want to dedicate more time to as a professional specialty.”

Reflecting back, Alex notes that friendship was a key component to this work. “We found really good people to work with along the way. Because we had no financial resources, we had to be open, collaborative, and resourceful throughout the planning process.” At the events, friendship and networking was also a primary component. “These gatherings were an opportunity for attendees to meet reachable professionals, who are open to communication and willing to help.” Ultimately, improving access to employment and education in the data space, promoting friendship and connection and supporting diverse entry into the data industry is what Alex is passionate about.

Alex is now volunteering with Médicos del Mundo in Spain, thanks to an initiative to connect university alumni with NGOs to share their professional and specialist knowledge. “In our business, especially during the sales cycle, we are focussed on closing deals. But in this case, the impact is different—there’s no monetary return, but we connect with people with different skills, and I am learning more about the impact that these NGOs have. I’m currently part of a group of volunteers working with Médicos del Mundo, to prioritize their digital transformation over the next three years, making sure they can implement solutions that are impactful for them and the people they help, with the resources they have.”

The post #ClouderaLife Volunteer Spotlight: Alex Campos, Principal Technical Leader, Spain appeared first on Cloudera Blog.

View Details

SoftBank Corp. provides Japan-based mobile communications services, mobile device sales, fixed-line communications, and ISP services, with more than 80 million users nationwide. The company also provides a variety of solutions for enterprises, including data centers, cloud, security, global, artificial intelligence (AI), IoT, and digital marketing services.

Supporting Data Access to Achieve Data-Driven InnovationDue to the spread of COVID-19, demand for digital services has increased at SoftBank. With new revenue streams and more varied operational structures of business units, it was imperative for SoftBank to move to a scalable data platform, where they could have a reliable, single source of information. Cloudera Data Platform (CDP) will enable SoftBank to increase resources flexibly as needed and adjust resources to meet business needs.

New Data Lakehouse Enables Stronger Data GovernanceSoftBank needed to reduce the number of workloads on its existing platform and decided to adopt Cloudera to build a data lake capable of managing data more effectively. Team members with various Cloudera capabilities provided 24-hour support for upgrade. Cloudera’s Professional Services team and Solution Engineers worked together with SoftBank’s team, which ultimately resulted in a successful migration process.

The data lake will function as a reliable, single information source from which different business units can extract actionable insights, and as a centralized data management platform tohelp eliminate data silos, optimize costs and maximize operational efficiency.

Previously, SoftBank was only able to handle batch processing, but since the migration, they can now run different workloads with different use cases and analyses in real-time. In addition, by adding analytical capabilities for real-time processing, SoftBank has succeeded in reducing the time to value, acccelerating their impact on the business. .

With more users accessing data, the ability to secure and manage data, such as sensitive customer data in on-premise environments, becomes a higher priority. CDP enables SoftBank to set user access guidelines and adjust variables such as granularity and location. In addition, it has functions to review and update user access controls regularly as part of data governance.

SoftBank Aims to Expand Data Provision Further for Users and Corporate CustomersThe introduction of CDP has strengthened SoftBank’s business for both users and corporate customers. With more data from all points of contact with customers, telecommunications carriers can now execute a continuous loop of planning and verification and personalize targeted offers to improve the customer experience.

At the same time, telecommunications carriers’ user location data that has been aggregated, anonymized, and processed is converted into data products that are then provided to business customers. SoftBank aims to expand its data applicationss across a variety of industries by leveraging the huge volume of data collected from its broad user base.

Yoichi Aki of SoftBank’s Data Strategy Department, IT & Architect Division, Technology Unit, said:

“As we look for ways to expand possible services and provide new value to our customers while protecting users’ confidential data, the amount of data we manage with our abundant capabilities continues to grow. For this reason, we have come to recognize the need for a modern data architecture that enables us to align our data strategy with our business goals. We believe these new data analysis capabilities will boost what we can offer to our customers.”

The post SoftBank Selects Cloudera Data Platform to Leverage Customer Intelligence While Ensuring Data Security appeared first on Cloudera Blog.

View Details

Out of the box Cloudera Data platform (CDP) performs superbly but over time, if data architecture, data engineering, and DevOps best practices are not maintained, you can get stuck maintaining the wild, wild west. In this six-part series, we’re focused on improving the health of your environment.

Visibility and Transparency

Improving environmental health is impossible if you’re flying blind. You’ve got to see the challenges to overcome them. You’ll need to be able to suppress noise in order to focus your efforts on analyzing the root cause of the most significant challenges, and if you can proactively notify impacted users of the incidents, progress, and resolutions, you’ll earn credibility.

We would like to share monitoring, alert, and notification best practices to help you:

• Gain visibility into environmental health issues
• Manage automated alerts to focus attention on the most important issues
• Automate communication to keep users informed
• Repair root cause issues instead of merely overcome incidents
• Improve the health of the platform, services, and data processes
• Move from a reactive to proactive troubleshooting posture

The Instructions, Tools, and Code We Are Providing You

This blog is merely a gateway to many valuable tools, lines of code, and best practices to gain visibility into all that ails your environment. We provide automation, or instructions on how to automate monitoring and alerting processes so that you can dedicate effort not only to recover after incidents, not only to identify and resolve root cause problems, but to become proactive in those discovery and repair processes.

Alternatively, you could adopt Cloudera Observability and get tons of visibility immediately!

In addition to cluster monitoring on Cloudera Data Platform (CDP), we guide you through ways to monitor and troubleshoot issues using commonly used microservices: Cloudera Manager (CM), Hive, Impala, CDSW, Impala, Yarn, Nifi, Airflow, and even Jurassic cron jobs.

Some manuals guide you through proper implementation of out-of-the-box CM monitoring tools while others provide JSON files with steps to launch some of our homegrown monitoring capabilities. Others go into great detail to teach you how to connect your logs to visualizations, step-by-step.

Our great hope is that this will enable you to significantly improve the health of your environment. If we can clarify any steps for you, click the “Ask a Question” button on our Community page. Enjoy!

Links to the Goodies

• CM Monitoring/Troubleshooting/Dashboards
  • Deploy your Cloudera Manager Dashboards
• Cluster Monitoring (Tableau)
  • Leveraging Cloudera Manager Cluster Generated Alerts To Gain Control Of Your CDP Environment – Transparency and Visibility
• Impala Query Monitoring (Tableau)
  • Handling Impala Critical Query Exceptions – Transparency and Visibility
• Impala DataBase Monitoring (Tableau)
  • Monitoring Impala Usage
• Operational Data Engineering Process Monitoring – Transparency and Visibility
  • CDSW Monitoring (Tableau)
  • Yarn/Spark jobs Monitoring (Tableau)
  • Airflow jobs Monitoring (Tableau)
  • NiFi Monitoring (Tableau)
  • Cron Job Monitoring

If you need assistance gaining visibility, engage our Professional Services experts. We did!

The post Visibility and Transparency appeared first on Cloudera Blog.

View Details

Businesses everywhere have engaged in modernization projects with the goal of making their data and application infrastructure more nimble and dynamic. By breaking down monolithic apps into microservices architectures, for example, or making modularized data products, organizations do their best to enable more rapid iterative cycles of design, build, test, and deployment of innovative solutions. The advantage gained from increasing the speed at which an organization can move through these cycles is compounded when it comes to data apps – data apps both execute business processes more efficiently and facilitate organizational learning/improvement.

SQL Stream Builder streamlines this process by managing your data sources, virtual tables, connectors, and other resources your jobs might need, and allowing non technical domain experts to to quickly run versions of their queries.

In the 1.9 release of Cloudera’s SQL Stream Builder (available on CDP Public Cloud 7.2.16 and in the Community Edition), we have redesigned the workflow from the ground up, organizing all resources into Projects. The release includes a new synchronization feature, allowing you to track your project’s versions by importing and exporting them to a Git repository. The newly introduced Environments feature allows you to export only the generic, reusable parts of code and resources, while managing environment-specific configuration separately. Cloudera is therefore uniquely able to decouple the development of business/event logic from other aspects of application development, to further empower domain experts and accelerate development of real time data apps.

In this blog post, we will take a look at how these new concepts and features can help you develop complex Flink SQL projects, manage jobs’ lifecycles, and promote them between different environments in a more robust, traceable and automated manner.

What is a Project in SSB?Projects provide a way to group resources required for the task that you are trying to solve, and collaborate with others.

In case of SSB projects, you might want to define Data Sources (such as Kafka providers or Catalogs), Virtual tables, User Defined Functions (UDFs), and write various Flink SQL jobs that use these resources. The jobs might have Materialized Views defined with some query endpoints and API keys. All of these resources together make up the project.

An example of a project might be a fraud detection system implemented in Flink/SSB. The project’s resources can be viewed and managed in a tree-based Explorer on the left side when the project is open.

You can invite other SSB users to collaborate on a project, in which case they will also be able to open it to manage its resources and jobs.

Some other users might be working on a different, unrelated project. Their resources will not collide with the ones in your project, as they are either only visible when the project is active, or are namespaced with the project name. Users might be members of multiple projects at the same time, have access to their resources, and switch between them to select

the active one they want to be working on.

Resources that the user has access to can be found under “External Resources”. These are tables from other projects, or tables that are accessed through a Catalog. These resources are not considered part of the project, they may be affected by actions outside of the project. For production jobs, it is recommended to stick to resources that are within the scope of the project.

Tracking changes in a projectAs any software project, SSB projects are constantly evolving as users create or modify resources, run queries and create jobs. Projects can be synchronized to a Git repository.

You can either import a project from a repository (“cloning it” into the SSB instance), or configure a sync source for an existing project. In both cases, you need to configure the clone URL and the branch where project files are stored. The repository contains the project contents (as json files) in directories named after the project.

The repository may be hosted anywhere in your organization, as long as SSB can connect to it. SSB supports secure synchronization via HTTPS or SSH authentication.

If you have configured a sync source for a project, you can import it. Depending on the “Allow deletions on import” setting, this will either only import newly created resources and update existing ones; or perform a “hard reset”, making the local state match the contents of the repository entirely.

After making some changes to a project in SSB, the current state (the resources in the project) are considered the “working tree”, a local version that lives in the database of the SSB instance. Once you have reached a state that you would like to persist for the future to see, you can create a commit in the “Push” tab. After specifying a commit message, the current state will be pushed to the configured sync source as a commit.

Environments and templatingProjects contain your business logic, but it might need some customization depending on where or on which conditions you want to run it. Many applications make use of properties files to provide configuration at runtime. Environments were inspired by this concept.

Environments (environment files) are project-specific sets of configuration: key-value pairs that can be used for substitutions into templates. They are project-specific in that they belong to a project, and you define variables that are used within the project; but independent because they are not included in the synchronization with Git, they are not part of the repository. This is because a project (the business logic) might require different environment configurations depending on which cluster it is imported to.

You can manage multiple environments for projects on a cluster, and they can be imported and exported as json files. There is always zero or one active environment for a project, and it is common among the users working on the project. That means that the variables defined in the environment will be available, no matter which user executes a job.

For example, one of the tables in your project might be backed by a Kafka topic. In the dev and prod environments, the Kafka brokers or the topic name might be different. So you can use a placeholder in the table definition, referring to a variable in the environment (prefixed with ssb.env.):

This way, you can use the same project on both clusters, but upload (or define) different environments for the two, providing different values for the placeholders.

Placeholders can be used in the values fields of:

• Properties of table DDLs
• Properties of Kafka tables created with the wizard
• Kafka Data Source properties (e.g. brokers, trust store)
• Catalog properties (e.g. schema registry url, kudu masters, custom properties)

SDLC and headless deploymentsSQL Stream Builder exposes APIs to synchronize projects and manage environment configurations. These can be used to create automated workflows of promoting projects to a production environment.

In a typical setup, new features or upgrades to existing jobs are developed and tested on a dev cluster. Your team would use the SSB UI to iterate on a project until they are satisfied with the changes. They can then commit and push the changes into the configured Git repository.

Some automated workflows might be triggered, which use the Project Sync API to deploy these changes to a staging cluster, where further tests can be performed. The Jobs API or the SSB UI can be used to take savepoints and restart existing running jobs.

Once it has been verified that the jobs upgrade without issues, and work as intended, the same APIs can be used to perform the same deployment and upgrade to the production cluster. A simplified setup containing a dev and prod cluster can be seen in the following diagram:

If there are configurations (e.g. kafka broker urls, passwords) that differ between the clusters, you can use placeholders in the project and upload environment files to the different clusters. With the Environment API this step can also be part of the automated workflow.

ConclusionThe new Project-related features take developing Flink SQL projects to the next level, providing a better organization and a cleaner view of your resources. The new git synchronization capabilities allow you to store and version projects in a robust and standard way. Supported by Environments and new APIs, they allow you to build automated workflows to promote projects between your environments.

Try it out yourself!Anybody can try out SSB using the Stream Processing Community Edition (CSP-CE). CE makes developing stream processors easy, as it can be done right from your desktop or any other development node. Analysts, data scientists, and developers can now evaluate new features, develop SQL-based stream processors locally using SQL Stream Builder powered by Flink, and develop Kafka Consumers/Producers and Kafka Connect Connectors, all locally before moving to production in CDP.

The post Projects in SQL Stream Builder appeared first on Cloudera Blog.

View Details

We are excited to announce the winners of the 2023 Cloudera Partner of the Year Awards. These awards recognize Cloudera partners dedicated to transforming the future of data by leveraging the most complete hybrid cloud data platform. The announcement was made at Impact24 – Cloudera’s Annual Partner Kickoff, which took place on April 18 and 20.

Here are this year’s winners: Global Partner of the Year: IBM*

IBM is a key strategic partnership for Cloudera, with an extensive history of joint success working together. Our partnership has led to significant sales activity across the globe and consistent year over year double digit revenue growth.

• Cloud Partner of the Year: Amazon Web Services (AWS)

Cloud partnerships are critical to our joint success. The remarkable growth of our collaboration with AWS is a testimony to our joint investments in our partnership and the success of our joint customers.

• Emerging Partner of the Year: Talend

Talend is a breakout star in our partner ecosystem and has embraced our strategy, providing customers with a coordinated solution to maximize their investments in our platforms.

• Public Sector Partner of the Year: Carahsoft

Carahsoft continues to serve and drive momentum for the public sector by assisting in quoting, reference & proposal responses, lead generation, marketing and event planning. We are pleased to be presenting Carahsoft with Cloudera’s Public Sector Partner of the Year award for the second year in a row.

Accenture, Nuvento, MARTINEXSA recognized on a regional levelIn addition to the global awards, Cloudera also recognized the following partners on a regional level:

• EMEA Partner of the Year: Accenture

Accenture embodies the collaboration necessary for a successful partnership. Together, this team accelerates the journey to the cloud for our joint customers, while collaborating on the development of fresh, new data and AI use cases that bring added value to our customers.

• APAC Partner of the Year: Nuvento

NUVENTO brings deep domain knowledge in Data Management, Analytics and Machine Learning to our partner ecosystem. NUVENTO recognizes the importance of enabling and onboarding quality, new talent and ensuring they are Cloudera accredited and certified.

• Americas Partner of the Year: Martinexsa

MARTINEXSA exemplifies the coordination, accountability, ingenuity and activity it takes to grow together. Securing new logos, driving new pipeline, and delivering new use cases, while ensuring customer satisfaction is how they are consistently delivering exceptional results.

The winners were determined based on revenue, year-over-year growth, resource and certification investment in Cloudera, and technology alignment.

Components of a Thriving Partner EcosystemThe Annual Partner Kickoff and awards program are important components of the Cloudera Partner Network (CPN). CPN, launched November 2022, is designed to strengthen collaboration and solution building with our partners, and to help them guide their customers to adopt modern data strategies based on the Cloudera hybrid data platform.

We are investing more than ever in our partner ecosystem, and saw tremendous growth last year, with close to 400 partners joining our new Cloudera Partner Network. This network includes a competency based, points driven approach so that partners can clearly differentiate themselves and invest. Focusing on joint use cases is how we will win together!

Congratulations to each of this year’s Partner of the Year award recipients. Thank you for a tremendous year, and we look forward to even more success together this year!

The post Global Partner Awards for AWS, IBM, Talend and Carahsoft appeared first on Cloudera Blog.

View Details

Data is core to decision making today and organizations often turn to the cloud to build modern data apps for faster access to valuable insights. With cloud operating models, decision making can be accelerated, leading to competitive advantages and increased revenue. Can you achieve similar outcomes with your on-premises data platform? You absolutely can. Application modernization initiatives have led to cloud native architectures gaining popularity on premises, making it a sensible choice to extend to your data platform.

In the blogs that follow, as part of this multi-part series, we will shed light on the latest and greatest features released via Cloudera Data Platform (CDP) Private Cloud Data Services. These include data recovery service, quota management, node harvesting, optimizing TCO, and more.

CDP Private Cloud Data Services 1.5.0, Cloudera’s latest iteration of its best-in-class data platform, delivers cloud native superpowers on premises. At its core, CDP Private Cloud Data Services (“the platform”) is an end-to-end cloud native platform that provides a private open data lakehouse. It offers features such as data ingestion, storage, ETL, BI and analytics, observability, and AI model development and deployment. The platform offers advanced capabilities for data warehousing (DW), data engineering (DE), and machine learning (ML), with built-in data protection, security, and governance. Let’s look at what a cloud native powered data platform can achieve for your compute-hungry data workloads.

What is cloud native exactly?According to Cloud Native Computing Foundation (CNCF), cloud native applications use an open source software stack to deploy applications as microservices, packaging each part into its own containers, and dynamically orchestrating those containers to optimize resource utilization. This is exactly how the platform was designed from the ground up.

Traditional monolithic data platforms are complex, constraining end users who need quick access to insights and requiring technical IT support. Today’s end users demand speed, agility, and cutting-edge features. This is exactly where cloud native architectures excel, and why they are so popular.

Adopting a cloud native data platform architecture empowers organizations to build and run scalable data applications in dynamic environments, such as public, private, or hybrid clouds. Not only does this deliver faster and richer data services that end users expect, but also enables IT teams to operate a well-oiled platform with benefits such as simpler management and improved security.

What business benefits do cloud native architectures deliver?Cloud native architectures empower your business users to quickly and easily access data to deliver timely insights and make critical business decisions for their LOBs. By embracing a cloud native architecture for your data platform here are three key benefits you can expect for your data practitioners:

1. Greater agility allows for faster deployment of self-service data applications, enabling business users to quickly adapt to changing market conditions and enjoy a better user experience with your data platform.
2. Better innovation, first by enabling end users to adopt new features faster for better insights, and second, by allowing developers to run experimental workloads without risking production stability, fostering a culture of innovation.
3. Reduced cost by optimizing compute utilization to run more analytics with the same hardware allocation. Quick adoption of software updates further lowers maintenance costs.

Key technology benefits of cloud native architecturesSimpler platform management makes it easier for your IT platform team to service your data practitioners’ needs, and meet downstream business SLAs. As with the business benefits above, there are three key ways that cloud native architectures help simplify platform management:

1. Easily scale data platform resources to optimize hardware utilization and minimize costs for on-demand workloads.
2. Enhance platform resilience with automatic recovery from application failures and faster adoption of security measures, greatly improving overall platform reliability to meet SLAs.
3. Achieve true hybrid portability with “write once, run anywhere” capabilities, facilitating movement of applications and data between on-premises and public cloud environments without code changes help to optimize for cost, scalability, resilience, innovation, and/or ESG initiatives.

Now let’s unpack the cloud native “superpowers” that enable these business and technology benefits, namely workload isolation, independent scaling of storage and compute, and the ability to shift capacity to where it’s needed.

Workload IsolationThe platform makes workload isolation simpler. It’s powered by Kubernetes, providing container-level resource isolation by using namespaces. Secondly, it uses Apache Yunikorn, a modern, enterprise-grade resource scheduler for Kubernetes, enhancing resource utilization and providing strong user, group, and application isolation.

Through workload isolation the platform can deliver many of the business and technology benefits previously covered, including scalable compute, self-service analytics, workload resilience, independent upgrades, and application portability.

Scaling Storage and ComputeWith evolving business needs and expanding data use cases, it becomes critical to scale compute and storage resources independently to avoid waste, sub-optimal workload performance, and incurring unnecessary costs. The platform separates compute and storage by default, allowing flexible scaling to meet varied workload demands more efficiently.

Platform Managers can easily determine when and where to scale compute or storage resources, reacting quickly to changing demands—if a use case needs more data storage, buy more storage without attached compute, and vice-versa.

Shifting CapacityThe platform leverages Kubernetes’ auto-scaling, self-healing, and load balancing features for maximized resource utilization, creating spare capacity that is available for other tasks.

• Horizontal Pod Autoscaling (HPA) adjusts pod replicas based on CPU or memory utilization, scaling your application automatically.
• Kubernetes has a self-healing mechanism that monitors the health of your application and automatically restarts any failed containers or pods to minimize downtime.
• Kubernetes’ load balancing distributes traffic across multiple pods running an application to prevent overload and handle high traffic.

Shifting available capacity based on expected demand optimizes overall server utilization, yielding benefits for business and technology teams we previously covered, such as lower costs and better scalability.

Unified Data PlatformCDP Private Cloud Data Services is a unified data platform that provides on-premises flexibility with cloud-like capabilities. Enterprise solution architects benefit from the platform’s security, portability, and economies of scale, while data practitioners and citizen data scientists enjoy the simplicity of the cloud native user experience. Customers can standardize on a single data platform that consistently meets their processing, security, and governance needs, whether on-premises or in the public cloud, offering unmatched portability. This flexibility allows customers to choose where their workloads run, for the right platform at the right time and economics.

Adopting a cloud native architecture is essential for thriving in today’s fast-paced economy. Cloudera Data Platform empowers businesses to manage data assets and applications efficiently and securely, running millions of jobs daily across exabytes of data. CDP provides the speed, flexibility, and scalability required to drive insights for successful decision-making. To learn more about CDP Private Cloud Data Services, please visit our website and contact your sales representatives to learn about free trials.

The post Building Cloud Native Data Apps on Premises appeared first on Cloudera Blog.

View Details

Unwelcome…… are platform instability, downtime, hardware failure, poor performance, cluster resource contention, repeated process failures, runaway live queries, critical services alarms, invisibility into alarm cacophony… the list goes on. If those are ailments you would like to remedy …

Welcome!To this six-part series, where we’ll look at how to get control of the health of your Cloudera Data platform (CDP) environment. Out of the box, CDP performs superbly, but over time, if data architecture, data engineering, and DevOps best practices are not maintained, the Data City you’ve erected atop a solid CDP bedrock can become the wild, wild, west. Perhaps it’s time for some law and order to prevent further crimes against the tech.

More than a case study, we’ve interwoven best practices gleaned from multiple configurations and client sites into a comprehensive, easy to understand set of instructions to diagnose and resolve many of the issues that adversely impact CDP environmental health.

With each blog we’ll outline the symptoms and root causes of common environmental health challenges and prescribe solutions. Where we can, we’ll include valuable links to step-by-step instructions to guide you through successful implementation. When we conclude the series, we’ll share a homegrown tool, an environmental health scorecard, to monitor and manage the health of your environment.

There are many, many reasons that an environment may perform poorly, and certainly some resolutions take time and effort, but there is quite succulent low hanging fruit. Our great hope is that you find impactful quick wins that inspire you to pursue multiple avenues of health improvement. You may also decide to partner with our Cloudera Professional Services team who more than doubled a customer’s health score in two short quarters.

Categories of CDP Environmental HealthWe’ve categorized aspects of environmental health for this series.

Visibility and TransparencyInto the cluster, platform, services, and processes. We won’t be able to make much progress if we do not have proper visibility into the problems. That’s observability. In this blog we provide instructions and tools on how to gain visibility, suppress alarm noise, find and analyze the root causes of the most significant opportunities, and proactively notify your users when incidents occur

Data Asset StandardizationOf common datasets, pipelines, processes, and reports. Admittedly, data asset standardization is a multiyear journey; notwithstanding, addressing only your most problematic and resource-intensive processes and assets may yield more environmental health improvement than any other category. We’ll share best practices on how to locate and capitalize on those opportunities.

Platform HealthIncludes hardware and services settings and configurations. Cloudera Data Platform (CDP) must be configured properly to function well with high performance. Furthermore, as business needs continually change, so will your use of the platform, and that will necessitate re-tuning. To help you on that journey, we’ll list some common symptoms, link them to root cause analysis steps, provide proper configuration guidelines, and outline the steps to properly tune your environment.

The Right Tool for the JobIncludes the proper use of Impala, CDSW, Airflow, Nifi, and CM. You might be surprised at the adverse environmental impact of using CDSW as an ETL pipeline tool or using Impala to write unwieldy queries with an embarrassing number of joins. We’ve done it too. We confess. We’ll highlight the advantages of using Airflow to manage complex data pipelines with its facility to divide workflow into small independent tasks. We’ll list other do’s and don’ts.

Environmental Health ScoringBrings it all together by demonstrating how to measure, score, monitor, and control environmental health through dashboards that we provide for you along with instructions to hook them up to your logs.

If you’ve got the symptoms, the doctors are in. Let the healing begin!

The post Gaining Control of Your CDP Environment appeared first on Cloudera Blog.

View Details

What is a dead letter queue (DLQ)?Cloudera SQL Stream builder gives non-technical users the power of a unified stream processing engine so they can integrate, aggregate, query, and analyze both streaming and batch data sources in a single SQL interface. This allows business users to define events of interest for which they need to continuously monitor and respond quickly. A dead letter queue (DLQ) can be used if there are deserialization errors when events are consumed from a Kafka topic. DLQ is useful to see if there are any failures due to invalid input in the source Kafka topic and makes it possible to record and debug problems related to invalid inputs.

Creating a DLQWe will use the example schema definition provided by SSB to demonstrate this feature. The schema has two properties: “name” and “temp” (for temperature) to capture sensor data in JSON format. The first step is to create two Kafka topics: “sensor_data” and “sensor_data_dlq” which can be done the following way:

kafka-topics.sh --bootstrap-server <bootstrap-server> --create --topic sensor\_data --replication-factor 1 --partitions 1kafka-topics --bootstrap-server <bootstrap-server> --create --topic sensor\_data\_dlq --replication-factor 1 --partitions 1 Once the Kafka topics are created, we can set up a Kafka source in SSB. SSB provides a convenient way to work with Kafka as we can do the whole setup using the UI. In Project Explorer, open the Data Sources folder. Right clicking on “Kafka” brings up the context menu where we can open the creation modal window.

We need to provide a unique name for this new data source, the list of brokers, and the protocol in use:

After the new Kafka source is successfully registered, the next step is to create a new virtual table. We can do that from the Project Explorer by right clicking “Virtual Tables” and choosing “New Kafka Table” from the context menu. Let’s fill out the form with the following values:

• Table Name: Any unique name; we will user “sensors” in this example
• Kafka Cluster: Choose the Kafka source registered in the previous step
• Data Format: JSON
• Topic Name: “sensor_data” which we created earlier

We can see under the “Schema Definition” tab that the example provided has the two fields, “name” and “temp,” as discussed earlier. The last step is to set up the DLQ functionality, which we can do by going to the “Deserialization” tab. The “Deserialization Failure Handler Policy” drop-down has the following options:

• “Fail”: Let the job crash and then auto-restart setting dictates what happens next
• “Ignore”: Ignores the message that could not be deserialized, moves to the next
• “Ignore and Log”: Same as ignore but logs each time it encounters a deserialization failure
• “Save to DLQ”: Sends the invalid message to the specified Kafka topic

Let’s select “Save to DLQ” and choose the previously created “sensor_data_dlq” topic from the “DLQ Topic Name” drop-down. We can click “Create and Review” to create the new virtual table.

Testing the DLQFirst, create a new SSB job from the Project Explorer. We can run the following SQL query to consume the data from the Kafka topic:

SELECT * from sensors;

In the next step we will use the console producer and consumer command line tools to interact with Kafka. Let’s send a valid input to the “sensor_data” topic and check if it is consumed by our running job.

kafka-console-producer.sh --broker-list <broker> --topic sensor\_data>{"name":"sensor-1", "temp": 32} Checking back on the SSB UI, we can see that the new message has been processed:

Now, send an invalid input to the source Kafka topic:

kafka-console-producer.sh --broker-list <broker> --topic sensor\_data >invalid data We won’t see any new messages in SSB as the invalid input cannot be deserialized. Let’s check on the DLQ topic we set up earlier to see if the invalid message was captured:

kafka-console-consumer.sh --bootstrap-server <server> --topic sensor\_data\_dlq --from-beginning invalid data The invalid input is there which verifies that the DLQ functionality is working correctly, allowing us to further investigate any deserialization error.

ConclusionIn this blog, we covered the capabilities of the DLQ feature in Flink and SSB. This feature is very useful to gracefully handle a failure in a data pipeline due to invalid data. Using this capability, it is very easy and quick to find out if there are any bad records in the pipeline and where the root cause of those bad records are.

Try it out yourself!Anybody can try out SSB using the Stream Processing Community Edition (CSP-CE). CE makes developing stream processors easy, as it can be done right from your desktop or any other development node. Analysts, data scientists, and developers can now evaluate new features, develop SQL-based stream processors locally using SQL Stream Builder powered by Flink, and develop Kafka Consumers/Producers and Kafka Connect Connectors, all locally before moving to production in CDP.

The post Using Dead Letter Queues with SQL Stream Builder appeared first on Cloudera Blog.

View Details

Data has become an essential driver for new monetization initiatives in the financial services industry. With the vast amount of data collected from customers, transactions, and market movements, among other sources, this abundance offers tremendous potential for financial institutions to extract valuable insights that can inform business decisions, improve customer service, and create new revenue streams. With the emergence of new technologies, including privacy enhancing techniques (PETs) that further protect the customer, monetizing data has become more accessible than ever before.

Considering today’s mobile-first-, mobile-almost-everything world, there has been a surge in the use of mobile banking applications, only accelerated by the COVID-19 pandemic. With more people relying on mobile digital transactions to check balances, make deposits and payments, and execute trades, financial services firms are able to collect a massive amount of data on a wide range of customer behavior. These data and other sources represent key opportunities for data monetization. By leveraging these data and creating a data-driven culture, firms can generate new revenue streams and improve their operations.

Third party opportunitiesOne way for financial services firms to monetize their data is by selling it to third parties.

Here are some relevant examples from across the industry:

• Retail banks can sell their customer transaction data to marketing firms that are interested in understanding consumer behavior.
• Commercial banks can also sell data on business transactions and credit history to credit rating agencies and other financial institutions.
• Wealth managers can monetize their data by selling analytics and insights to their clients, such as customized investment recommendations based on an individual’s financial goals and risk profile.
• Asset and fund managers can sell data and analytics to their clients, such as performance data on different investment portfolios.
• Stock exchanges can monetize their data by selling real-time market data to financial institutions and news organizations interested in tracking market trends and movements.
• Central banks can sell data on economic indicators and monetary policy to financial institutions and researchers.

Developing new products and servicesIn addition to selling data, financial services firms can also monetize their data by using it to create new products and services. Customer 360 initiatives enable hyper-personalization to target the right customer with the next best action, which leads to more engaged customers and better outcomes from data monetization.

For example, a retail bank can use customer transaction data to develop personalized financial products, such as credit cards and investment portfolios, tailored to individual needs. Investment banks can use data on market trends and investor behavior to create new financial products and services, such as derivatives and structured finance products.

Factoring in complianceHowever, protecting customer data and adhering to data privacy laws is critical for financial services firms. The phrase “with great power comes great responsibility” comes to mind. In order to monetize their data while still respecting the privacy of their customers, these firms must implement robust data protection measures and adhere to relevant regulations. This includes implementing strong security measures, such as encryption and multi-factor authentication, and only collecting and using data with customer consent.

Financial services firms must also carefully consider how they share data with third parties, entering into data sharing agreements that outline the terms and conditions for data use. In addition, they must comply with relevant data privacy laws, such as the General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA), and be transparent about their data protection practices.

Shifting to a data-driven cultureTo fully realize the value of their data, financial services firms must create a data-driven culture that prioritizes the use of data in decision-making and innovation. This includes investing in modern data architecture, such as using a platform like Cloudera, which enables companies like Santander UK to store, process, and analyze large amounts of data in real time. By using data to inform their strategies and operations, financial services firms can better understand their customers, improve their risk management, and identify new opportunities for growth.

Overall, there are many ways for financial services firms to monetize their data, including selling it to third parties and using it to create new products and services. By adopting a data-driven culture and investing in modern data architecture, these firms can better leverage their data to drive innovation and generate new revenue streams while also protecting the privacy of their customers.

Learn how financial services firms are leveraging Cloudera for data-driven use cases from customer retention and profitability to reducing fraud and risk exposure.

The post Discovering Data Monetization Opportunities in Financial Services appeared first on Cloudera Blog.

View Details

As we celebrate International Women’s Day and Women’s History Month in the US, for this #ClouderaLife Employee Spotlight we sat down with Clouderan Sherry Zhou to talk about her career transition from biology to technology, her geographic transition from the US to the UK, and what she learned along the way.

Sherry is an Engineering Manager for the CDV (Cloudera Data Visualization) team. Her team’s objectives are to, first, make it easier for analysts to explore data, enabling them to uncover interesting trends in product features and performance. A second objective is to make it easier for less technical users to interpret data through the creation of dashboards.

Sherry started at Cloudera two years ago as an individual contributor and a technology lead for the team. She then took on a people manager role and is now managing a team of ten plus rotating interns.

“As an individual contributor I understood the big picture of the product, but becoming a people manager really expanded my perspective,” she said. “I learned a lot more about what goes into team planning and management of resources, and how the product relates to the team.”

Starting at and returning to Cloudera at the same timeSherry got her start in web development at a startup company, Arcadia Data, in the San Francisco Bay Area. She then brought her and her family to London and moved on to another company. “Living in London has been an exciting experience,” she said. “There has been a lot to learn and explore as well as a lot of adjustment in how everything works in the rhythm of the day-to-day.”

Meanwhile, Arcadia Data was acquired by Cloudera and a couple of years ago her former Arcadia colleagues, now Cloudera employees, contacted her. They were working on what is now called CDV (formerly an Arcadia product) and expanding, looking for those with previous experience in the product.

“Cloudera has a great culture. I really like how hard the company tries to be inclusive,” she said. “I also really like the company’s Cloudera Cares program, which matches donations to charities and causes.” She adds, “They are always coming up with new and interesting programs.”

From an early interest in technology to a biology career in the NIH, all led back to high-tech.Sherry always had an interest in technology while growing up. Her first foray into the world of web development started at the age of thirteen when she learned HTML to build a webpage to showcase photos from her math and science camp.

Despite her burgeoning interests in web design, she decided to study biology at Yale University and, after graduation, worked as an analyst at the National Institute of Health (NIH) in Washington DC. “Looking back at my time with the NIH, the part of my job I found the most fun was when I could automate my job by writing scripts,” she said. “I was drawn to this kind of work and ultimately decided I needed to explore it as a career.”

So Sherry took a deep dive into a new career in high-tech, by moving out to the San Francisco Bay Area. There she interned and then enrolled in a coding boot camp, which included training and aid in job placement. It even had a place to sleep as students had intense project schedules.

“Software development is a creative outlet for me,” she said. “You start with a problem and create something to solve that problem that didn’t exist before; it’s very fulfilling.”

“For anyone looking to advance their career in software development, I would say the more code you write in more varied projects the better it will be for you in the long run,” she said. “For example, participate in Hackathons to meet new people or explore different technologies and different ways of doing things. What you learn may not apply to your job today but in my case a lot of what I picked up wound up helping me later on.

Recognizing International Women’s Day and Women’s History Month (US)“It’s great to celebrate the progress women have made through International Women’s Day and Women’s History Month,” Sherry said. “When I was just starting out in my career in technology there were an overwhelming number of men in the field,” she said. “I had a great opportunity to volunteer at ‘Women Who Code’ events and similar organizations to help change that. The coding boot camp I was enrolled in was also an inclusive environment that brought together talent from all different backgrounds. As I got started I noticed that not all companies felt as welcoming, and I sought out companies to join that had a supportive team environment. On our team, I’m pleased to see we have a pretty even ratio of men to women.”

What’s Next?Sherry continues to develop along the management track while also growing her technical abilities. She is currently working on a master’s degree in software engineering at Oxford. In their spare time Sherry, her husband, and their one-year-old continue to take in the London culture and experience of living and working in the city.

As we close the conversation, we want to thank Sherry for her continued contribution to Cloudera and for telling her story.

The post #ClouderaLife Employee Spotlight: Sherry Zhou, Engineering Manager appeared first on Cloudera Blog.

View Details

One of the core principles that guides Cloudera and everything we do is a commitment to the open source community. As the entire Cloudera Data Platform is built on open source projects, we find it crucial to participate in and contribute back to the community. Applied ML prototypes are one of the ways that we accomplish this.

Applied ML Prototypes (AMPs) are fully built end-to-end data science solutions that allow data scientists to go from an idea to a fully working machine learning model in a fraction of the time. AMPs provide an end-to-end framework for building, deploying, and monitoring business-ready ML applications instantly. AMPs are available to deploy with a single click in Cloudera Machine Learning (CML), but every AMP is also available to the public as a public GitHub repository.

For the Cloudera and AMD Applied Machine Learning Prototype Hackathon, competitors were tasked with creating their own unique AMP for one of five categories (Sports and Entertainment, Environment, Business and Economy, Society, and Open Innovation). As you can tell, we left the guidance pretty open ended. This was a deliberate choice because we wanted to encourage competitors to work on whatever project their data hearts desired.

We had over 150 teams register to participate, and from those we selected nine teams as finalists. The final nine teams were given access to their own CML instance running on Amazon EC2 M6a instances powered by 3rd Gen AMD EPYC, and three weeks to develop their prototypes. These general-purpose M6a instances are designed specifically for balanced compute, memory and networking needs and deliver up to 10% lower cost versus comparable instances. What the competing participants delivered in the end astounded our team of judges, and they certainly didn’t make it easy to select a winner. However, after the dust settled, we are happy to share the following three winning Applied ML Prototypes.

First Place: Forecasting Evapotranspiration With Kats and ProphetDanika Gupta’s AMP checked all the boxes for the judges (see GitHub repository). It was a perfect example of everything that an AMP should be: a novel application of ML to a real-world problem, with well-written code, and a clean web application to communicate the results.

The project was aimed at helping make better water management decisions based on long-range forecasts of evapotranspiration (ET), which is an assessment of the release of water by evaporation from soil and transpiration from plants.

Using OpenET, a publicly accessible database of ET data assessed from satellite imagery, this project leverages forecasting models from the Kats library to create ET predictions for 10 cities in the California Bay Area. The accompanying web application was built with Streamlit, it allows users to select one of the 10 cities on a map and then view the historical ET data and predictions from each model for that city.

Second Place: Art Sale Price Prediction ModelOf the winning submissions, this AMP was the lone project worked on by a team (GitHub repository). Ishaan Poojari, Ge Jin, Idan Lau, and Jeffrey Lin are all students from NYU. For their AMP, they wanted to see if they could get into the New York art appraisal scene with their own ML backed art sale price predictor.

To accomplish the task, the team leveraged an ensemble method of combining predictions from a numerical and a computer vision model to accurately predict the price that a piece of art would sell at. For the numerical model they used a premade data set on Kaggle with art prices and other features from over the years to train a random forest model, and for the computer vision model they used a CNN from the TensorFlow Keras API on imagery downloaded from Sotheby’s.

Finally, to make their model accessible to the masses, they created a web application that allows users to upload an image and add some information about the piece of art and the artist that created it. The application will then provide a prediction of the price at which that piece of art would be sold for.

Third Place: Automatic Code CommentingThis AMP really speaks to my heart. What is the one thing that every developer hates? Going through and commenting their code! Okay, maybe some of us enjoy it, but the rest of us slackers are going to love this AMP.

Narendra Gangwani developed their AMP (see GitHub repository) to make the lives of developers everywhere easier, with a web application that allows you to enter the text of a Python function, and have accurate and descriptive comments with proper spacing added directly into the text.

The magic behind the scenes of the app is accomplished through an attention-based pre-trained transformer model (like BERT) that has been tuned with a sequence-to-sequence data set, with code-comment pairs for Python programming language.

What’s NextIn the coming months we will be incorporating these new projects into our official AMP Catalog, making them deployable with a single click for Cloudera customers, and their source code readily available via public GitHub repositories.

If you missed participating in this hackathon, but would like to take a crack at creating your own winning submission, follow Cloudera on LinkedIn and be on a lookout for the next AMP Hackathon later this year.

To learn more about how Applied ML Prototypes can reduce your data science team’s time-to-value, visit our AMP practitioner page.

If you’d like to learn more about AMD solutions on the cloud, visit the AMD page here: https://www.amd.com/en/solutions/cloud-computing

The post Applied ML Prototype Hackathon with AMD Winners appeared first on Cloudera Blog.

View Details

Cloudera Contributors: Ayush Saxena, Tamas Mate, Simhadri Govindappa

Since we announced the general availability of Apache Iceberg in Cloudera Data Platform (CDP), we are excited to see customers testing their analytic workloads on Iceberg. We are also receiving several requests to share more details on how key data services in CDP, such as Cloudera Data Warehousing (CDW), Cloudera Data Engineering (CDE), Cloudera Machine Learning (CML), Cloudera Data Flow (CDF) and Cloudera Stream Processing (CSP) integrate with the Apache Iceberg table format and the easiest way to get started. In this blog, we will share with you in detail how Cloudera integrates core compute engines including Apache Hive and Apache Impala in Cloudera Data Warehouse with Iceberg. We will publish follow up blogs for other data services.

Iceberg basicsIceberg is an open table format designed for large analytic workloads. As described in Iceberg Introduction it supports schema evolution, hidden partitioning, partition layout evolution and time travel. Every table change creates an Iceberg snapshot, this helps to resolve concurrency issues and allows readers to scan a stable table state every time.

The Apache Iceberg project also develops an implementation of the specification in the form of a Java library. This library is integrated by execution engines such as Impala, Hive and Spark. The new feature this blog post is aiming to discuss about Iceberg V2 format (version 2), as the Iceberg table specification explains, the V1 format aimed to support large analytic data tables, while V2 aimed to add row level deletes and updates.

In a bit more detail, Iceberg V1 added support for creating, updating, deleting and inserting data into tables. The table metadata is stored next to the data files under a metadata directory, which allows multiple engines to use the same table simultaneously.

Iceberg V2With Iceberg V2 it is possible to do row-level modifications without rewriting the data files. The idea is to store information about the deleted records in so-called delete files. We chose to use position delete files which provide the best performance for queries. These files store the file paths and positions of the deleted records. During queries the query engines scan both the data files and delete files belonging to the same snapshot and merge them together (i.e. eliminating the deleted rows from the output).

Updating row values is achievable by doing a DELETE plus an INSERT operation in a single transaction.

Compacting the tables merges the changes/deletes with the actual data files to improve performance of reads. To compact the tables use CDE Spark.

By default, Hive and Impala still create Iceberg V1 tables. To create a V2 table, users need to set table property ‘format-version’ to ‘2’. Existing Iceberg V1 tables can be upgraded to V2 tables by simply setting table property ‘format-version’ to ‘2’. Hive and Impala are compatible with both Iceberg format versions, i.e. users can still use their old V1 tables; V2 tables simply have more features.

Use casesComplying with specific aspects of regulations such as GDPR (General Data Protection Regulation) and CCPA (California Consumer Privacy Act) means that databases need to be able to delete personal data upon customer requests. With delete files we can easily mark the records belonging to specific people. Then regular compaction jobs can physically erase the deleted records.

Another trivial use case is when existing records need to be modified to correct wrong data or update outdated values.

How to Update and Delete Currently only Hive can do row level modifications. Impala can read the updated tables and it can also INSERT data into Iceberg V2 tables.

To remove all data belonging to a single customer:

DELETE FROM ice\_tbl WHERE user\_id = 1234; To update a column value in a specific record:

UPDATE ice\_tbl SET col\_v = col\_v + 1 WHERE id = 4321; Use the MERGE INTO statement to update an Iceberg table based on a staging table:

MERGE INTO customer USING (SELECT * FROM new\_customer\_stage) sub ON sub.id = customer.id WHEN MATCHED THEN UPDATE SET name = sub.name, state = sub.new\_state WHEN NOT MATCHED THEN INSERT VALUES (sub.id, sub.name, sub.state); When not to use IcebergIceberg tables feature atomic DELETE and UPDATE operations, making them similar to traditional RDBMS systems. However, it’s important to note that they are not suitable for OLTP workloads as they are not designed to handle high frequency transactions. Instead, Iceberg is intended for managing large, infrequently changing datasets.

If one is looking for a solution that can handle very large datasets and frequent updates, we recommend using Apache Kudu.

CDW basicsCloudera Data Warehouse (CDW) Data Service is a Kubernetes-based application for creating highly performant, independent, self-service data warehouses in the cloud that can be scaled dynamically and upgraded independently. CDW supports streamlined application development with open standards, open file and table formats, and standard APIs. CDW leverages Apache Iceberg, Apache Impala, and Apache Hive to provide broad coverage, enabling the best-optimized set of capabilities for each workload.

CDW separates the compute (Virtual Warehouses) and metadata (DB catalogs) by running them in independent Kubernetes pods. Compute in the form of Hive LLAP or Impala Virtual Warehouses can be provisioned on-demand, auto-scaled based on query load, and de-provisioned when idle thus reducing cloud costs and providing consistent quick results with high concurrency, HA, and query isolation. Thus simplifying data exploration, ETL and deriving analytical insights on any enterprise data across the Data Lake.

CDW also simplifies administration by making multi-tenancy secure and manageable. It allows us to independently upgrade the Virtual Warehouses and Database Catalogs. Through tenant isolation, CDW can process workloads that do not interfere with each other, so everyone meets report timelines while controlling cloud costs.

How to useIn the following sections we are going to provide a few examples of how to create Iceberg V2 tables and how to interact with them. We’ll see how one can insert data, change the schema or the partition layout, how to remove/update rows, do time-travel and snapshot management.

Hive:Creating a Iceberg V2 TableA Hive Iceberg V2 table can be created by specifying the format-version as 2 in the table properties.

Ex.

• CREATE Iceberg V2 table

| CREATE EXTERNAL TABLE TBL\_ICEBERG\_PART(ID INT, NAME STRING) PARTITIONED BY (DEPT STRING) STORED BY ICEBERG STORED AS PARQUET TBLPROPERTIES ('FORMAT-VERSION'='2'); |

• CREATE TABLE AS SELECT (CTAS)

| CREATE EXTERNAL TABLE CTAS\_ICEBERG\_SOURCE STORED BY ICEBERG AS SELECT * FROM TBL\_ICEBERG\_PART; |

• CREATE TABLE LIKE(creates an empty table based on another table)

| CREATE EXTERNAL TABLE ICEBERG\_CTLT\_TARGET LIKE ICEBERG\_CTLT\_SOURCE STORED BY ICEBERG; |

Ingesting DataData into an Iceberg V2 table can be inserted similarly like normal Hive tables

Ex:

• INSERT INTO

| INSERT INTO TABLE TBL\_ICEBERG\_PART VALUES (1,'ONE','MATH'), (2, 'ONE','PHYSICS'), (3,'ONE','CHEMISTRY'), (4,'TWO','MATH'), (5, 'TWO','PHYSICS'), (6,'TWO','CHEMISTRY'); |

• INSERT OVERWRITE

| INSERT OVERWRITE TABLE CTLT\_ICEBERG\_SOURCE SELECT * FROM TBL\_ICEBERG\_PART; |

• MERGE

| MERGE INTO TBL\_ICEBERG\_PART USING TBL\_ICEBERG\_PART\_2 ON TBL\_ICEBERG\_PART.ID = TBL\_ICEBERG\_PART\_2.IDWHEN NOT MATCHED THEN INSERT VALUES (TBL\_ICEBERG\_PART\_2.ID, TBL\_ICEBERG\_PART\_2.NAME, TBL\_ICEBERG\_PART\_2.DEPT); |

Delete & Updates:V2 tables allow row level deletes and updates similarly like the Hive-ACID tables.

Ex:

• Deleting a Row

| DELETE FROM TBL\_ICEBERG\_PART WHERE DEPT = 'MATH'; |

• Updating a Row value

| UPDATE TBL\_ICEBERG\_PART SET DEPT='BIOLOGY' WHERE DEPT = 'PHYSICS' OR ID = 6; |

Querying Iceberg tables:* SELECT statements

Hive supports both vectorized and non vectorized reads for Iceberg V2 tables, Vectorization can be enabled normally using the following configs:

1. set hive.llap.io.memory.mode=cache;
2. set hive.llap.io.enabled=true;
3. set hive.vectorized.execution.enabled=true

| SELECT COUNT(*) FROM TBL\_ICEBERG\_PART; |

• Time-Travel statements

Hive allows us to query table data for specific snapshot versions.

| SELECT * FROM TBL\_ICEBERG\_PART FOR SYSTEM\_VERSION AS OF 7521248990126549311; |

Snapshot ManagementHive allows several operations regarding snapshot management, like:

• Expire Snapshot:

| ALTER TABLE TBL\_ICEBERG\_PART EXECUTE EXPIRE\_SNAPSHOTS('2021-12-09 05:39:18.689000000'); |

• Set Current Snapshot:

| ALTER TABLE TBL\_ICEBERG\_PART EXECUTE SET\_CURRENT\_SNAPSHOT (7521248990126549311); |

• RollBack Snapshot:

| ALTER TABLE TBL\_ICEBERG\_PART EXECUTE ROLLBACK(3088747670581784990); |

Alter Iceberg tables* Schema evolution:

| ALTER TABLE … ADD COLUMNS (...); (Add a column)ALTER TABLE … REPLACE COLUMNS (...);(Drop column by using REPLACE COLUMN to remove the old column)ALTER TABLE … CHANGE COLUMN … AFTER …; (Reorder columns) |

• Partition evolution:

| ALTER TABLE TBL\_ICEBERG\_PART SET PARTITION SPEC (NAME); |

Materialized Views* Creating Materialized Views:

| CREATE MATERIALIZED VIEW MAT\_ICEBERG AS SELECT ID, NAME FROM TBL\_ICEBERG\_PART ; |

• Rebuild Materialized Views:

| ALTER MATERIALIZED VIEW MAT\_ICEBERG REBUILD; |

• Querying Materialized Views:

| SELECT * FROM MAT\_ICEBERG; |

ImpalaApache Impala is an open source, distributed, massively parallel SQL query engine with its backend executors written in C++, and its frontend (analyzer, planner) written in java. Impala uses the Iceberg Java library to get information about Iceberg tables during query analysis and planning. On the other hand, for query execution the high performing C++ executors are in charge. This means queries on Iceberg tables are lightning fast.

Impala supports the following statements on Iceberg tables.

Creating Iceberg tables* CREATE Iceberg V2 table:

| CREATE TABLE ice\_t(id INT, name STRING, dept STRING)PARTITIONED BY SPEC (bucket(19, id), dept)STORED BY ICEBERGTBLPROPERTIES ('format-version'='2'); |

• CREATE TABLE AS SELECT (CTAS):

| CREATE TABLE ice\_ctasPARTITIONED BY SPEC (truncate(1000, id))STORED BY ICEBERGTBLPROPERTIES ('format-version'='2')AS SELECT id, int\_col, string\_col FROM source\_table; |

• CREATE TABLE LIKE:
  (creates an empty table based on another table)

| CREATE TABLE new\_ice\_tbl LIKE orig\_ice\_tbl; |

Querying Iceberg tablesImpala supports reading V2 tables with position deletes.

• SELECT statements:

Impala supports all kinds of queries on Iceberg tables that it supports for any other tables. E.g. joins, aggregations, analytical queries etc. are all supported.

| SELECT * FROM ice\_t;SELECT count(*) FROM ice\_t i LEFT OUTER JOIN other\_t bON (i.id = other\_t.fid)WHERE i.col = 42; |

• Time-travel statements:

It’s possible to query earlier snapshots of a table (until they are expired).

| SELECT * FROM ice\_t FOR SYSTEM\_TIME AS OF '2022-01-04 10:00:00';SELECT * FROM ice\_t FOR SYSTEM\_TIME AS OF now() - interval 5 days;SELECT * FROM ice\_t FOR SYSTEM\_VERSION AS OF 123456; |

We can use DESCRIBE HISTORY statement to see what are the earlier snapshots of a table:

| DESCRIBE HISTORY ice\_t FROM '2022-01-04 10:00:00';DESCRIBE HISTORY ice\_t FROM now() - interval 5 days;DESCRIBE HISTORY ice\_t BETWEEN '2022-01-04 10:00:00' AND '2022-01-05 10:00:00'; |

Insert data into Iceberg tablesINSERT statements work for both V1 and V2 tables.

• INSERT INTO:

| INSERT INTO ice\_t VALUES (1, 2);INSERT INTO ice\_t SELECT col\_a, col\_b FROM other\_t; |

• INSERT OVERWRITE:

| INSERT OVERWRITE ice\_t VALUES (1, 2);INSERT OVERWRITE ice\_t SELECT col\_a, col\_b FROM other\_t; |

Load data into Iceberg tables* LOAD DATA INPATH:

| LOAD DATA INPATH '/tmp/some\_db/parquet\_files/'INTO TABLE iceberg\_tbl; |

Alter Iceberg tables* Schema evolution:

| ALTER TABLE ... RENAME TO ... (renames the table)ALTER TABLE ... CHANGE COLUMN ... (change name and type of a column)ALTER TABLE ... ADD COLUMNS ... (adds columns to the end of the table)ALTER TABLE ... DROP COLUMN ... |

• Partition evolution:

| ALTER TABLE ice\_pSET PARTITION SPEC (VOID(i), VOID(d), TRUNCATE(3, s), HOUR(t), i); |

Snapshot management* Expire old snapshots:

| ALTER TABLE ice\_tbl EXECUTE expire\_snapshots('2022-01-04 10:00:00');ALTER TABLE ice\_tbl EXECUTE expire\_snapshots(now() - interval 5 days); |

DELETE and UPDATE statements for Impala are coming in later releases. As mentioned above, Impala is using its own C++ implementation to deal with Iceberg tables. This gives significant performance advantages compared to other engines.

Future WorkOur support for Iceberg v2 is advanced and reliable, and we continue our push for innovation. We are rapidly developing improvements, so you can expect to find new features related to Iceberg in each CDW release. Please let us know your feedback in the comments section below.

SummaryIceberg is an emerging, extremely interesting table format. It is under rapid development with new features coming every month. Cloudera Data Warehouse added support for the most recent format version of Iceberg in its latest release. Users can run Hive and Impala virtual warehouses and interact with their Iceberg tables via SQL statements. These engines are also evolving quickly and we deliver new features and optimizations in every release. Stay tuned, you can expect more blog posts from us about upcoming features and technical deep dives.

To learn more:* Replay our webinar Unifying Your Data: AI and Analytics on One Lakehouse, where we discuss the benefits of Iceberg and open data lakehouse. * Read why the future of data lakehouses is open. * Replay our meetup Apache Iceberg: Looking Below the Waterline.

Try Cloudera Data Warehouse (CDW) by signing up for a 60 day trial, or test drive CDP. If you are interested in chatting about Apache Iceberg in CDP, let your account team know or contact us directly. As always, please provide your feedback in the comments section below.

The post Open Data Lakehouse powered by Iceberg for all your Data Warehouse needs appeared first on Cloudera Blog.

View Details

In this special edition of Cloudera Cares #VolunteerSpotlight for International Pet Day, Julia Ostrowski shares her experience of fostering cats and dogs for the past two decades.

About youWhere are you based?I live in beautiful downtown San Jose, CA and live in a house built in 1904.

What’s your job role? I have been with the Support org since starting with Hortonworks almost 8 years ago (cannot believe it’s been that long) as Director of Support Enablement, until this January when I moved to the IT department.

Introduce your foster animals!What are your foster dogs and cats names?I’m fostering two senior cats, Emma and Gianna, whose owner unfortunately died of cancer last year. When the owner’s husband died several years ago, she had to go into assisted living and I took in a cat for them at the time. Matt (now Panther) turned out to be the best, most amazing companion I could ask for, and so I adopted him a few months later.

So when I heard the wife had terminal cancer, I immediately volunteered to take in her cats. They are super easy going, very loving and it’s been a joy fostering them. I will likely have them for a long time as (unfortunately) senior animals are less desirable than younger animals, which is just fine with me!

Last year, my own dog Hoshi (a 16 year old poodle/shih tzu mix) died of kidney issues, and just a week later Rob Roy, my 15 year old foster hospice dog died.

Recently my other two fosters Maxie (a 7 year old Chi/terrier mix), and Bailey (a 7 month old Chi/pom puppy) were adopted. So now I’m fostering Johnny Boy (a 13 year old, 20 pound “your guess is as good as mine” dog), and Wrigley (also known as Mr. Wrigglesworth, Wriggly Piggly, and just plain “Piggy”–a 3 year old Chi mix).

Johnny is the smartest, clumsiest dog I’ve ever fostered who loves going for walks more than I love chocolate, and Piggy is by far the happiest, wriggliest dog I’ve ever met. He never stops wagging his tail and just loves people. And Johnny and Piggy get along great together, even sleeping in the same crate during the day.

How did you first get involved with fostering dogs and cats? I have been fostering cats for about two decades now for 13th St. Cats, a small, all-volunteer run rescue located primarily in downtown San Jose. At the time, we adopted out about 50 cats a year…and now we regularly place 300+ cats into loving forever homes annually.

I knew nothing about cats when I first became a foster, as my brother was allergic to cats, and we were always a dog family growing up. But I soon saw that cats are absolutely amazing in their own right and have fostered hundreds of cats and kittens over the years. Some have been ‘foster failures’ and ended up staying with me as my own cats until they crossed over the Rainbow Bridge.

The group also focuses on TNR (Trap Neuter Return) , which traps and neuters all the feral cats in a colony, adopting out the kittens and friendlier kitties, then returning the wild adults back to their territory and providing them with food and shelter.

I also started fostering dogs more than a decade ago, and now foster through a group called Every Pet’s Dream. This is a small but wonderful rescue that focuses on taking dogs with medical and behavioral issues that would be otherwise destroyed by the Shelter. I have fostered dozens of dogs for them, each one fantastic in their own way.

I prefer fostering (and adopting) the older dogs and cats as they have a special place in my heart, and am happy to foster animals with medical issues, such as kidney, thyroid, and heart problems, as well as diabetes. Neither rescue I volunteer for will end the life of a dog or cat due to treatable medical issues, or behavioral issues.

How do you volunteer for them – what do you do? For 13th St. Cats, I generally facilitate the monthly planning meetings and volunteer for their special fundraising events (like a recent Game Night co-hosted by the San Jose Woman’s Club). I also monitor and respond to calls made to our hotline, process adoption paperwork, and keep our Adoption Center (located in Petsmart) stocked with supplies.

What’s the most rewarding thing about volunteering with local animal rescues? Every bit of the time and energy spent in volunteering with animal rescues is worth it when you see the animals placed into wonderful loving homes! Hearing a once-terrified kitten purr for the first time…seeing a once-insecure dog gradually decompress and grow confident…there is nothing else like it.

It’s been an incredible experience over the last two decades, and I hope to continue for years to come. Dogs and cats are some of my favorite people

What would you say to someone considering volunteering with an animal rescue?Explore it! As a foster, all supplies (including food, bowls, beds, litter boxes, etc.) are provided by the rescue. You can let them know what type of foster situation you are looking for. Some folks can only foster dog-friendly dogs, if there are resident pooches already in the home, or they prefer to foster kittens and puppies (or, like me, prefer the more mature animal). Some prefer larger dogs, or smaller dogs…all those preferences are taken into consideration when placing animals into foster homes.

And even if fostering is not a possibility for you, there is a place for you as a rescue volunteer. There is always a need for data entry, fundraising (which is a constant, year-round necessity), staffing special events (like in-person adoption showcases), even driving animals to their medical appointments. And these days there is a special need for volunteers to help promote rescues and their adoptable animals on social media.

Rescue saves lives!The post #ClouderaLife Volunteer Spotlight Pet Day Special: Julia Ostrowski appeared first on Cloudera Blog.

View Details

The evolution of healthcare has come a long way since local physicians made house calls and homespun remedies were formulated using items from the kitchen spice rack. Today’s healthcare is driven as much by the promise of emerging technologies centered on data processing and advanced analytics as by developing new and specialized drugs. This has ushered in a new era of precision healthcare focused on the uniqueness of each patient and the multitude of variables that factor into more precise and effective detection and treatment regimens.

What is Precision Healthcare?Precision healthcare is a rapidly evolving data-driven approach that tailors treatments and prevention strategies to the individual characteristics of each patient, rather than a one-size-fits-all or groups of cohorts approach that matches the treatment to the disease itself. Precision healthcare leverages data and AI-powered advanced analytics to predict and prevent disease and to identify the most effective treatments for each patient.

The foundation of precision healthcare is the use of data from a variety of sources. And by combining and analyzing these different data sources, healthcare providers can gain a comprehensive understanding of each patient’s unique health profile, including their genome sequence, microbiome composition, health history, lifestyle, diet and environmental factors. The ability to analyze the more “granular level” of data of the individual and to compare and look for a ‘similar’ individual is key to establish the cohort of 1 with a level of precision.

How Data Drives Precision HealthcareOne of the key advantages of precision healthcare is the ability to use large datasets to identify patients at a higher risk of developing a particular disease. For example, if a patient has a family history of colon cancer, genetic testing can be used to determine if they have an increased risk of developing the disease. This information can then be used to develop a personalized prevention and screening plan, which may include more frequent colonoscopy screeningsor other interventions.

Data is also used to identify the most effective treatments for each patient. By analyzing a patient’s genomic makeup using machine learning (ML) algorithms, healthcare providers can identify specific mutations or genetic markers that may indicate a particular treatment will be more effective than others. For example, a patient with a certain genetic mutation may respond better to a specific chemotherapy drug than other patients with the same type of cancer.

In addition to genetic data, precision healthcare also uses data from other sources, such as electronic health records (EHRs) and wearable devices, to monitor patient health and identify potential health issues before they become serious.

For example, wearable devices can track a patient’s heart rate, activity level, and sleep patterns, providing insights into their overall health and well-being. This data can be used to identify early warning signs of potential health problems or adverse reactions to their care plan, allowing healthcare providers to intervene early and prevent more serious complications.

Data is also critical for research and development in precision healthcare. By using ML to analyze large amounts of data from diverse populations, researchers can identify new patterns and relationships that can inform the development of new treatments and prevention strategies.

But the use of data in precision healthcare also presents a number of challenges. Precision healthcare relies on the ability to collect and analyze massive amounts of data from a wide range of sources – unstructured, semi-structured, structured and streaming, including those mentioned above.. This data must be accurate, complete, and standardized in order to be useful for analysis and decision-making with the ability to monitor, predict and take action in real-time.

Focus on Breast CancerConsider mammogram mass detection for breast cancer. Radiologists spend a lot of time manually evaluating digital images looking to identify asymmetries, irregular density, clusters of calcifications, as well as areas of thickening skin. Detecting these abnormalities as well as changes in tumor size and appearance is crucial in determining the subsequent course of a patient’s treatment plan. This can be a tedious process open to interpretive or perception errors.

With Cloudera Machine Learning radiologists have a force multiplier at the edge, as ML-powered computer vision can analyze mammograms as well as CT scans, X-rays and MRIs with greater speed and accuracy and detect abnormal masses that are not visible to the human eye. This gives radiologists a supportive advanced analytics solution to make more informed clinical decisions, provide more timely diagnoses and ultimately increase caseload.

Solving The Data Challenge with ClouderaCloudera solves the data challenge by giving healthcare organizations a hybrid data platform that can manage and analyze data across the full data lifecycle – data distribution, data engineering, data warehousing, data science, and machine learning.

Healthcare organizations can ingest, process, store and analyze any type of data, in the data center, in public and hybrid clouds or at edge locations. By analyzing data at rest, in motion and streaming, healthcare organizations can use advanced analytics, machine learning and AI to identify patterns, detect anomalies and predict potential outcomes.

To learn more about how Cloudera supports Precision Health, join me in person at HIMSS23. We’ll be presenting a Lunch and Learn at the event and food will be provided.

Register Now

When: Wednesday, April 19 at 11:00 AM – 12:15 PM CT

Where: McCormick Place West Building

2301 S. Indiana Ave.

Level 4 | W470 A

Chicago, IL 60616

The post No Average Patient – Leveraging Data for Precision Healthcare appeared first on Cloudera Blog.

View Details

The best description of untrusted data I’ve ever heard is, “We all attend the QBR – Sales, Marketing, Finance – and present quarterly results, except the Sales reports and numbers don’t match Marketing numbers and neither match Finance reports. We argue about where the numbers came from, then after 45 minutes of digging for common ground, we chuck our shovels and abandon the call in disgust.”

How would you go about fixing that situation? How would you get the trust into trusted data?

Consult the Book of SpellsOur spells are cast from our Enterprise Business Glossary. Our wizard is Data Governance Director Suvayu Bose (no relation) who employs a very practical approach to data governance: establish C-suite commitment to the program, set strategic goals, identify data owners and data stewards, then get right to negotiating data definitions cross-functionally.

For data to be trusted, everyone must first agree to what it means, where it’s sourced, and how it’s derived.

Start with critical data elements, those data objects comprising the most important metrics and KPI to run the company. In this respect, Suvayu is quite the Svengali (no relation). If your numbers don’t conform to his data definitions, you’re up the QBR without a shovel.

1. Standardize Datasets

Here’s the first of three things Suvayu recommends to get the trust in trusted data: as data definitions are codified in the business glossary, establish those data objects in your enterprise datasets and evangelize them as the source of truth from which new data assets should be sourced.

Our company built the world’s best hybrid cloud data platform, bundled with integrated security, governance, and lineage, and yet we face the same challenges governing internal data that you might. We doubled-down on data governance in 2021, and in 18 short months we’re flying high, in part because we are standardizing our enterprise datasets. By sourcing new analytics from standard datasets, archiving legacy datasets, and repiping established analytics (only when feasible and purposeful!), we increase trust in data.

1. Standardize Reporting & Analytics

We’ve been great at data democratization for years but we’ve experienced the common adverse side effects that perhaps you face as well: the ungoverned proliferation of contrary reporting and analytics. Inventory shrinkage increases trust in the data by removing access to duplicative, contradictory reports.

First we retired reports and extract jobs with no/low usage: 85% of the inventory! That exposed additional db archival targets. We built enterprise standard dashboards for the company’s most important KPI and metrics, beginning with executive views then drilling down into middle management and individual contributor views. Then we consolidated an additional 5% of inventory by grafting important features of well-used reports into the enterprise standards.

1. Standardize Everything In-Between

With enterprise standard data objects and dashboards on the rise and legacy data assets in decline, we shutoff duplicative pipelines and queries and we watched the health of our environment skyrocket.

If you need help (we did), engage our Professional Services team to identify where your opportunities are and how to realize them.

The post Trusted Data: Alchemy For Misinformation appeared first on Cloudera Blog.

View Details

What is a materialized view?Cloudera SQL Stream Builder (SSB) gives the power of a unified stream processing engine to non-technical users so they can integrate, aggregate, query, and analyze both streaming and batch data sources in a single SQL interface. This allows business users to define events of interest for which they need to continuously monitor and respond quickly.

There are many ways to distribute the results of SSB’s continuous queries to embed actionable insights into business processes. In this blog we will cover materialized views—a special type of sink that makes the output available via REST API.

In SSB we can use SQL to query stream or batch data, perform some sort of aggregation or data manipulation, then output the result into a sink. A sink could be another data stream or we could use a special type of data sink we call a materialized view (MV). An MV is a special type of sink that allows us to output data from our query into a tabular format persisted in a PostgreSQL database. We can also query this data later, optionally with filters using SSBs REST API.

Why use a materialized view?If we want to easily use the results of our SQL job from an external application, MVs are the best and easiest way to do so. All we need to do is define the MV on the UI interface and applications will be able to retrieve data via REST API.

Imagine, for instance, that we have a real-time Kafka stream containing plane data and we are working on an application that needs to download all planes in a certain area, above some altitude at any given time via REST. This is not a simple task to do, since planes are constantly moving and changing their altitudes, and we need to read this data from an unbounded stream. If we add a materialized view to our SSB job, that will create a REST endpoint from which we will be able to retrieve the latest result from our job. We can also add filters to this request, so for example, our application can use the MV to show all the planes that are flying higher than some user-specified altitude.

Creating a materialized viewCreating a new jobAn MV always belongs to a single job, so to create an MV we must first create a job in SSB. To create a job we will also need to create a project first which will provide us a Software Development Lifecycle (SDLC) for our applications and allows us to collect all our job and table definitions or data sources in a central place.

Getting the dataAs an example we will use the same Automatic Dependent Surveillance Broadcast (ADS-B) data we used in other posts and examples. For reference, ADS-B data is generated and broadcast by planes while flying. The data consists of a plane ID, altitude, latitude and longitude, speed, etc.

To better illustrate how MVs work, let’s execute a simple SQL query to retrieve all of the data from our stream.

SELECT * FROM airplanes; The creation of the “airplanes” table has been omitted, but suffice it to say airplanes is a virtual table we have created, which is fed by a stream of ADS-B data flowing through a Kafka topic. Please check our documentation to see how that’s done. The query above will generate output like the following:

As you can see from the output, there are all kinds of interesting data points. In our example let’s focus on altitude.

Flying highFrom the SSB Console, click on the “Materialized View” button on the top right:

An MV configuration panel will open that will look similar to the following:

ConfigurationSSB allows us to configure the new MV extensively, so we will go through them here.

Enable MVFor the MV to be available once we have finished configuring it, “Enable MV” must be enabled. This configuration also allows us to easily disable this feature in the future without removing all the other settings.

Primary keyEvery MV requires a primary key, as this will be our primary key in the underlying relational database as well. The key is one of the fields returned by the SSB SQL query, and it is available from the dropdown. In our case we will choose icao, because we know that icao is the identification number for each plane, so it is a perfect fit for the primary key.

Retention and min row retention countThis value tells SSB how long it should keep the data around before removing it from the MV database. It is set to five minutes by default. Each row in the MV is tagged with an insertion time, so if the row has been around longer than the “Retention (Seconds)” time then the row is removed. Note, there is also an alternative method for managing retention, and that is the field below the retention time, called “Min Row Retention Count,” which is used to indicate the minimum number of rows we would like to keep in the MV, regardless of how old the data might be. For example we could say, “We want to keep the last 1,000 rows no matter how old that data is.” In that case we would set “Retention (Seconds)” to 0, and set “Min Row Retention Count” to 1,000.

For this example we will not change the default values.

API keyAs mentioned earlier, every MV is associated with a REST API. The REST API endpoint must be protected by an API Key. If none has been added yet, one can be created here as well.

QueriesFinally we get to the most interesting part, selecting how to query our data in the MV database.

API endpointClicking on the “Add New Query” button opens a pop-up that allows us to configure the REST API endpoint, as well as selecting the data we would like to query.

As we said earlier, we are interested in the plane’s altitude, but let’s also add the ability to filter the field altitude when calling the REST API. Our MV will be able to only show planes that are flying higher than some user specified altitude (i.e., show planes flying higher than 10,000 feet). In that case in the “URL Pattern” box we could enter:

planes/higherThan/{param} Note the {param} value. The URL pattern can take parameters that are specified inside curly brackets. When we retrieve data for the MV, the REST API will map these parameters in our filters, so the user calling the endpoint can set the value. See below.

Choose the dataNow it is time to select what data to collect as part of our MV. The data fields we can choose come from the initial SSB SQL query we wrote, so if we said SELECT * FROM airplanes; the “Select Columns” dropdown will have things like flight, icao, lat, counter, altitude, etc. For our example let’s choose icao, lat, lon and altitude.

OopsWe have a problem. The data fields in the stream, including the altitude, are all of VARCHAR type, making it infeasible to filter for numeric data. We need to make a simple change to our SQL and convert the altitude into an INT, and call it height, to differentiate it from the original altitude field. Let’s change the SQL to the following:

SELECT *, CAST(altitude AS INT) AS height FROM airplanes; Now we can replace altitude with height, and use that to filter.

FilteringNow to filter by height we need to map the parameter we previously created ({param}) to the height field. By clicking on the “Filters” tab, and then the “+ Rule” button, we can add our filter.

For the “Field” we choose height, for the “Operator” we want “greater_or_equal,” and for the “Value” we use the {param} we used in the REST API endpoint. Now the MV query will filter the rows by the value of height being greater than the value that the user would give to {param} when issuing the REST request, for example:

https://<host>/…/planes/higherThan/10000 That would output something similar to the following:

[{"icao":"A28947","lat":"","lon":"","height":"30075"}] ConclusionMaterialized views are a very useful out-of-the-box data sink, which provide for the collection of data in a tabular format, as well as a configurable REST API query layer on top of that that can be used by third party applications.

Try it out yourself!Anybody can try out SSB using the Stream Processing Community Edition (CSP-CE). CE makes developing stream processors easy, as it can be done right from your desktop or any other development node. Analysts, data scientists, and developers can now evaluate new features, develop SQL-based stream processors locally using SQL Stream Builder powered by Flink, and develop Kafka Consumers/Producers and Kafka Connect Connectors, all locally before moving to production in CDP.

The post Materialized Views in SQL Stream Builder appeared first on Cloudera Blog.

View Details

Over the past handful of years, systems architecture has evolved from monolithic approaches to applications and platforms that leverage containers, schedulers, lambda functions, and more across heterogeneous infrastructures. Cloudera Data Platform (CDP) is no different: it’s a hybrid data platform that meets organizations’ needs to get to grips with complex data anywhere, turning it into actionable insight quickly and easily.

While in the old world where questions around data quality or system performance were answered by monitoring a few logs and metrics, in a distributed landscape (like a hybrid data platform) it’s not that straightforward. There are many logs and metrics, and they are all over the place.

Monitoring alone will tell you when something’s not as it should be, but that’s not answering the question of “why?” That’s where observability comes in.

Pointing to “something” that could be an issue in the previous paragraph was intentional. There are various user roles that all have different questions “why?” as they use CDP. While a business analyst may wonder why the values in their customer satisfaction dashboard have not changed since yesterday, a DBA may want to know why one of today’s queries took so long, and a system administrator needs to find out why data storage is skewed to a few nodes in the cluster. Different types of observability for different aspects of CDP provide them with the answers: data, workload, and software observability as part and parcel of the platform.

Data observabilityFor a platform so concerned with data and the insight it brings, knowing whether the star player—data—is up to scratch is crucial. As Barr Moses outlined in her original article, data downtime is directly related to data systems complexity and immediately impacts insight and decision making. Luke Roquet recently drilled into the topic of data observability with Mark Ramsey of Ramsey International (RI) to also cover the five pillars (freshness, distribution, volume, schema, and lineage) that describe the quality and reliability of data.

These pillars and the metrics they provide are closely linked to the data governance capability CDP’s Shared Data Experience (SDX) delivers, and are surfaced in the data catalog. SDX continually captures and manages both the active and passive metadata for data assets and the processes that work on them. And, crucial for a hybrid data platform, it does so across hybrid cloud. With CDP, and SDX in particular, Barr’s concern that data governance is hard to achieve is directly addressed. Especially when implemented as a unified data fabric, CDP ensures proactive data governance and, with that, the basis for good data observability, reduced data downtime, and trusted data for better decision making.

Workload observability CDP’s key role for organizations is to turn data into insight and value at scale. To do so, the platform provides a range of analytics across the complete data life cycle. Data services and workloads cover ingesting data, enriching it, making it available for analysis in (operational) dashboards, or using it to build AI and machine learning models. Each of these analytics can be deployed to different infrastructures and may, on occasion, behave differently than expected. Although data downtime may be one of the causes of missed SLA and SLOs, implementation itself should be equally observed.

Observability always works from the same basis: metrics, traces, and logs; so too workload observability. Just as in the case of data observability, workload metrics and health tests help identify and troubleshoot issues as well as potential issues, while prescriptive guidance and recommendations address and optimize uncovered problems. Specifically for the main workload criteria of performance, baselines and historical analysis not only identify and address performance problems, but also create the basis for cost prediction and reduction (an area of increasing importance as financial governance increases). Within CDP, Workload Manager provides workload observability to ensure optimal performance, reduced downtime, and improved resource utilization.

Software observabilityAnd all this—this data, these workloads—are all deployed somewhere. On infrastructures ranging from bare metal data centers to public and private clouds, across hybrid cloud. Each has their own stacked layers of enabling technologies, from operating systems to containers to resources. Historically, this is where observability made its initial entry in the IT world.

For Cloudera as an organization too, software observability has been applied extensively in the area of support. Building on over 14 years of experience, Cloudera’s support organization draws on software observable insight from over 1.3 million nodes under subscription and has created sophisticated diagnostics tools that include predictive alerting based on diagnostic data. This allows Cloudera’s customers to receive advanced warning on hundreds of different known issues and security vulnerabilities to help avoid downtime, improve reliability, and reduce risk.

Observability futuresObservability will continue to evolve and has proven to deliver tremendous benefits. Baked right into the platform, CDP already provides the observability tools and insights for the full stack, all the way from the infrastructure to the end user. SDX’s data catalog provides data observability that highlights trusted data for better decision making across the business and helps reduce data downtime. Workload Manager adds workload observability for optimized processes and resource utilization.

As observability evolves, so will CDP. Cloudera is already hard at work bottling the software observability the support organization uses to bring the benefits and insight it brings closer to our customers. And being the open platform it is, we’re also looking at sharing CDP’s observability with other tools and vice versa.

Observability is an exciting area that provides the answers to the questions that crop up with increasingly complex hybrid cloud environments deployed at organizations. Get in touch now to learn more about CDP’s current and future observability capabilities.

The post Observe Everything appeared first on Cloudera Blog.

View Details

Align everything to corporate strategyI lead data and analytics at Cloudera. We’re called Cloudera Data Analytics (CDA). How very clever. Prior to forming the group, it was imperative to understand Cloudera’s corporate strategy: corporate objectives, product strategy, go-to-market strategy, key metrics and KPI. Our CDA charter must be aligned with corporate strategy, but shouldn’t everything we do be aligned?

Cloudera helps innovative organizations tackle transformational use cases and exact real-time insights. The analytics CDA delivers must enable the fulfillment of that mission, so we ask: will the requested data product drive customer success, achieve key efficiencies, or improve competitive advantage? If so, we absorb the request, knowing we are aligned with corporate objectives and contributing to enterprise success. If not, we respectfully reject the request.

For anyone operating from Data Org Purgatory – town motto Obliged To Say Yes! – aligning everything you do to corporate strategy should lift the obligation, liposuction your backlog, liberate your capacity to drive your strategic initiatives, and enable you to…

Sustain engagement of key stakeholders and senior executivesCDA envisions a maturity level where we set targets for the revenue that we influence or generate for Cloudera. How very ambitious, right? We are already held accountable for quantifiable efficiency gains that our data products achieve in the business. But despite our ambitions, we are still a humble shared services organization (SSO).

I’m seldom axiomatic, but I believe immutably that in high-functioning companies, shared services organizations are doomed to fail without sustained engagement from senior executives and key stakeholders.

Engagement is elemental to stay connected to corporate strategy, to ingest use cases that drive the highest business value, and thereby to remain enterprise-essential as a function.

Without engagement, stakeholders can become disinterested and doubt the SSO’s value proposition, and those are the front and rear turn signals into irrelevance and disbandment.

To engage key stakeholders:1- Actively, continually uncover perceived and actual needs.

Effective analytics organizations pursue high value use cases instead of waiting or wishing for them.

• Ask stakeholders: What questions can’t you answer? What is it that your teams cannot do today? What keeps you up at night?
• Interview managers and individual contributors.
• Ruminate on the corporate objectives yourself: what data products would help business units achieve them?Deliver, deliver, deliver

2- Deliver on commitments consistently; become indispensable to the business.

3- Meet with them frequently.

Don’t take no for an answer. You need to meet. Meeting will provide you a forum to…

Materialize awe and rave about it

To materialize awe, obtain those high-value use cases, deliver on them, then shout your success. How very 1-2-3. Admittedly anecdotal and generalized, I’ve noticed that CDOs are most often comfortable with delivery. Drumming up business requires some salesy soft skills. Touting success requires storytelling talent and some CDOs might find that unnatural. They shouldn’t.

Storytelling is so very important, and you’re already writing them. Your strategic plan is an outline for a compelling story you want to tell through your data functions. That informs your operating plan, a compelling work of fiction which unfurls according to the timeline in your roadmap. Your accomplishments are the materialization of your story, delivered on stage by your data functions. You’ve ideated the story. You’ve written the story. You’ve acted out the story. Have you raved about it?

Your executive check-ins are the reviews of your play, and here’s the beauty: you get to be the critic! Tell the story as it unfolds. Illustrate how well-receiving your audience has been (your users). Set a hook for the riveting plot points ahead. I get so excited to demonstrate our value prop that some check-ins feel like those queasy moments in job interviews where you’re not sure you’re coming across as prideful (the good kind) or braggadocious. Brag away.

Because excitement is contagious. It inspires engagement. I’ve even observed this paradox: a less engaged stakeholder stares at me like Napoleon Dynamite through a check-in, counts the seconds to the end of the session like Good Will Hunting, and then, months later, defends our value prop to peers: “I hear they’re doing great work.” Hypnotic suggestion. I now carry a pocket watch to every check-in.

And now I know.

Attending the Gartner Data & Analytics Summit in Orlando, live or virtually? #GartnerDADon’t miss Shayde Christian as part of the panel discussion, “Data Mesh or Data Mess? Best Practices for Data Products” #HybridCloud with Sanjeev Mohan (Analyst, SanjoMo) and Luke Roquet (Cloudera) on Tuesday, March 21, 11a in Swan Ballroom 4, WDW Swan Hotel. Hear a lively discussion about modern data architectures, with a CDO’s perspective. And meet us over booth #1327.

The post Align, Engage and Rave: 3 Things I Wish I Knew as Chief Data Officer appeared first on Cloudera Blog.

View Details

As the use of ChatGPT becomes more prevalent, I frequently encounter customers and data users citing ChatGPT’s responses in their discussions. I love the enthusiasm surrounding ChatGPT and the eagerness to learn about modern data architectures such as data lakehouses, data meshes, and data fabrics. ChatGPT is an excellent resource for gaining high-level insights and building awareness of any technology. However, caution is necessary when delving deeper into a particular technology. ChatGPT is trained on historical data and depending on how one phrases their question, it may offer inaccurate or misleading information.

I took the free version of ChatGPT on a test drive (in March 2023) and asked some simple questions on data lakehouse and its components. Here are some responses that weren’t exactly right, and our explanation on where and why it went wrong. Hopefully this blog will give ChatGPT an opportunity to learn and correct itself while counting towards my 2023 contribution to social good.

I thought this was a fairly comprehensive list. The one key component that is missing is a common, shared table format, that can be used by all analytic services accessing the lakehouse data. When implementing a data lakehouse, the table format is a critical piece because it acts as an abstraction layer, making it easy to access all the structured, unstructured data in the lakehouse by any engine or tool, concurrently. The table format provides the necessary structure for the unstructured data that is missing in a data lake, using a schema or metadata definition, to bring it closer to a data warehouse. Some of the popular table formats are Apache Iceberg, Delta Lake, Hudi, and Hive ACID.

Also, the data lake layer is not limited to cloud object stores. Many companies still have massive amounts of data on premises and data lakehouses are not limited to public clouds. They can be built on premises or as hybrid deployments leveraging private clouds, HDFS stores, or Apache Ozone.

At Cloudera, we also provide machine learning as part of our lakehouse, so data scientists get easy access to reliable data in the data lakehouse to quickly launch new machine learning projects and build and deploy new models for advanced analytics.

I like how ChatGPT started this answer, but it quickly jumps into features and even gives an incorrect response on the feature comparison. Features are not the only way of deciding which is a better table format. It depends on compatibility, openness, versatility, and other factors that can guarantee broader usage for varied data users, guarantee security and governance, and future-proof your architecture.

Here is a high-level feature comparison chart if you want to go into the details of what’s available on Delta Lake versus Apache Iceberg.

This response is a little dangerous because of its incorrectness and demonstrates why I feel these tools are not ready for deeper analysis. At first glance it may look like a reasonable response, but its premise is wrong, which makes you doubt the entire response and other responses as well. Saying “Delta Lake is built on top of Apache Iceberg” is incorrect as the two are completely different, unrelated table formats and one has nothing to do with the conception of the other. They were created by different organizations to solve common data problems.

I am impressed that ChatGPT got this one right, although it made a few mistakes with our product names, and missed a few that are critical for a lakehouse implementation.

CDP’s components that support a data lakehouse architecture include:

1. Apache Iceberg table format that is integrated into CDP to provide structure to the massive amounts of structured, unstructured data in your data lake.
2. Data services, including cloud native data warehouse called CDW, data engineering service called CDE, data streaming service called data in motion, and machine learning service called CML.
3. Cloudera Shared Data Experience (SDX), which provides a unified data catalog with automatic data profilers, unified security, and unified governance over all your data both in the public and private cloud.

ChatGPT is a great tool to get a high-level view of new technologies, but I’d say use it carefully, validate its responses, and use it only for the awareness stage of the buying cycle. As you go into the consideration or comparison stage, it’s not reliable yet.

Also, answers on ChatGPT keep updating so hopefully it corrects itself before you read this blog.

To learn more about Cloudera’s lakehouse visit the webpage and if you are ready to get started watch the Cloudera Now demo.

The post Educating ChatGPT on Data Lakehouse appeared first on Cloudera Blog.

View Details

In this post, I will demonstrate how to use the Cloudera Data Platform (CDP) and its streaming solutions to set up reliable data exchange in modern applications between high-scale microservices, and ensure that the internal state will stay consistent even under the highest load.

IntroductionMany modern application designs are event-driven. An event-driven architecture enables minimal coupling, which makes it an optimal choice for modern, large-scale distributed systems. Microservices, as part of their business logic, sometimes do not only need to persist data into their own local storage, but they also need to fire an event and notify other services about the change of the internal state. Writing to a database and sending messages to a message bus is not atomic, which means that if one of these operations fails, the state of the application can become inconsistent. The Transactional Outbox pattern provides a solution for services to execute these operations in a safe and atomic manner, keeping the application in a consistent state.

In this post I am going to set up a demo environment with a Spring Boot microservice and a streaming cluster using Cloudera Public Cloud.

The Outbox PatternThe general idea behind this pattern is to have an “outbox” table in the service’s data store. When the service receives a request, it not only persists the new entity, but also a record representing the message that will be published to the event bus. This way the two statements can be part of the same transaction, and since most modern databases guarantee atomicity, the transaction either succeeds or fails completely.

The record in the “outbox” table contains information about the event that happened inside the application, as well as some metadata that is required for further processing or routing. Now there is no strict schema for this record, but we’ll see that it is worth defining a common interface for the events to be able to process and route them in a proper way. After the transaction commits, the record will be available for external consumers.

This external consumer can be an asynchronous process that scans the “outbox” table or the database logs for new entries, and sends the message to an event bus, such as Apache Kafka. As Kafka comes with Kafka Connect, we can leverage the capabilities of the pre-defined connectors, for example the Debezium connector for PostgreSQL, to implement the change data capture (CDC) functionality.

ScenarioLet’s imagine a simple application where users can order certain products. An OrderService receives requests with order details that a user just sent. This service is required to do the following operations with the data:

1. Persist the order data into its own local storage.
2. Send an event to notify other services about the new order. These services might be responsible for checking the inventory (eg. InventoryService) or processing a payment (eg. PaymentService).

Since the two required steps are not atomic, it is possible that one of them is successful while the other fails. These failures can result in unexpected scenarios, and eventually corrupt the state of the applications.

In the first failure scenario, if the OrderService persists the data successfully but fails before publishing the message to Kafka, the application state becomes inconsistent:

Similarly, if the database transaction fails, but the event is published to Kafka, the application state becomes inconsistent.

Solving these consistency problems in a different way would add unnecessary complexity to the business logic of the services, and might require implementing a synchronous approach. An important downside in this approach is that it introduces more coupling between the two services; another is that it does not let new consumers join the event stream and read the events from the beginning.

The same flow with an outbox implementation would look something like this:

In this scenario, the “order” and “outbox” tables are updated in the same atomic transaction. After a successful commit, the asynchronous event handler that continuously monitors the database will notice the row-level changes, and send the event to Apache Kafka through Kafka Connect.

Implementation based on Cloudera Public Cloud and DebeziumThe source code of the demo application is available on github. In the example, an order service receives new order requests from the user, saves the new order into its local database, then publishes an event, which will eventually end up in Apache Kafka. It is implemented in Java using the Spring framework. It uses a Postgres database as a local storage, and Spring Data to handle persistence. The service and the database run in docker containers.

For the streaming part, I am going to use the Cloudera Data Platform with Public Cloud to set up a Streams Messaging DataHub, and connect it to our application. This platform makes it very easy to provision and set up new workload clusters efficiently.

NOTE: Cloudera Data Platform (CDP) is a hybrid data platform designed for unmatched freedom to choose—any cloud, any analytics, any data. CDP delivers faster and easier data management and data analytics for data anywhere, with optimal performance, scalability, security, and governance.

The architecture of this solution looks like this on a high level:

The outbox tableThe outbox table is part of the same database where the OrderService saves its local data. When defining a schema for our database table, it is important to think about what fields are needed to process and route the messages to Kafka. The following schema is used for the outbox table:

| Column | Type | | uuid | uuid | | aggregate_type | character varying(255) | | created_on | timestamp without time zone | | event_type | character varying(255) | | payload | character varying(255) |

The fields represent these:

• uuid: The identifier of the record.
• aggregate_type: The aggregate type of the event. Related messages will have the same aggregate type, and it can be used to route the messages to the correct Kafka topic. For example, all records related to orders can have an aggregate type “Order,” which makes it easy for the event router to route these messages to the “Order” topic.
• created_on: The timestamp of the order.
• event_type: The type of the event. It is required so that consumers can decide whether to process and how to process a given event.
• payload: The actual content of the event. The size of this field should be adjusted based on the requirements and the maximum expected size of the payload.

The OrderServiceThe OrderService is a simple Spring Boot microservice, which exposes two endpoints. There is a simple GET endpoint for fetching the list of orders, and a POST endpoint for sending new orders to the service. The POST endpoint’s handler not only saves the new data into its local database, but also fires an event inside the application.

The method uses the transactional annotation. This annotation enables the framework to inject transactional logic around our method. With this, we can make sure that the two steps are handled in an atomic way, and in case of unexpected failures, any change will be rolled back. Since the event listeners are executed in the caller thread, they use the same transaction as the caller.

Handling the events inside the application is quite simple: the event listener function is called for each fired event, and a new OutboxMessage entity is created and saved into the local database, then immediately deleted. The reason for the quick deletion is that the Debezium CDC workflow does not examine the actual content of the database table, but instead it reads the append-only transaction log. The save() method call creates an INSERT entry in the database log, while the delete() call creates a DELETE entry. For every INSERT event, the message will be forwarded to Kafka. Other events such as DELETE can be ignored now, as it does not contain useful information for our use case. Another reason why deleting the record is practical is that no additional disk space is needed for the “Outbox” table, which is especially important in high-scale streaming scenarios.

After the transaction commits, the record will be available for Debezium.

Setting up a streaming environmentTo set up a streaming environment, I am going to use CDP Public Cloud to create a workload cluster using the 7.2.16 – Streams Messaging Light Duty template. With this template, we get a working streaming cluster, and only need to set up the Debezium related configurations. Cloudera provides Debezium connectors from 7.2.15 (Cloudera Data Platform (CDP) public cloud release, supported with Kafka 2.8.1+):

The streaming environment runs the following services:

• Apache Kafka with Kafka Connect
• Zookeeper
• Streams Replication Manager
• Streams Messaging Manager
• Schema Registry
• Cruise Control

Now setting up Debezium is worth another tutorial, so I will not go into much detail about how to do it. For more information refer to the Cloudera documentation.

Creating a connectorAfter the streaming environment and all Debezium related configurations are ready, it is time to create a connector. For this, we can use the Streams Messaging Manager (SMM) UI, but optionally there is also a Rest API for registering and handling connectors.

The first time our connector connects to the service’s database, it takes a consistent snapshot of all schemas. After that snapshot is complete, the connector continuously captures row-level changes that were committed to the database. The connector generates data change event records and streams them to Kafka topics.

A sample predefined json configuration in a Cloudera environment looks like this:

| { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "database.history.kafka.bootstrap.servers": "${cm-agent:ENV:KAFKA\_BOOTSTRAP\_SERVERS}", "database.hostname": "[***DATABASE HOSTNAME***]", "database.password": "[***DATABASE PASSWORD***]", "database.dbname": "[***DATABASE NAME***]", "database.user": "[***DATABASE USERNAME***]", "database.port": "5432", "tasks.max": "1",, "producer.override.sasl.mechanism": "PLAIN", "producer.override.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"[***USERNAME***]\" password=\"[***PASSWORD***]\";", "producer.override.security.protocol": "SASL\_SSL", "plugin.name": "pgoutput", "table.whitelist": "public.outbox", "transforms": "outbox", "transforms.outbox.type": "com.cloudera.kafka.connect.debezium.transformer.CustomDebeziumTopicTransformer", "slot.name": "slot1"} |

Description of the most important configurations above:

• database.hostname: IP address or hostname of the PostgreSQL database server.
• database.user: Name of the PostgreSQL database user for connecting to the database.
• database.password: Password of the PostgreSQL database user for connecting to the database.
• database.dbname: The name of the PostgreSQL database from which to stream the changes.
• plugin.name: The name of the PostgreSQL logical decoding plug-in installed on the PostgreSQL server.
• table.whitelist: The white list of tables that Debezium monitors for changes.
• transforms: The name of the transformation.
• transforms..type: The SMT plugin class that is responsible for the transformation. Here we use it for routing.

To create a connector using the SMM UI:

• Go to the SMM UI home page, select “Connect” from the menu, then click “New Connector”, and select PostgresConnector from the source templates.

• Click on “Import Connector Configuration…” and paste the predefined JSON representation of the connector, then click “Import.”

• To make sure the configuration is valid, and our connector can log in to the database, click on “Validate.”
• If the configuration is valid, click “Next,” and after reviewing the properties again, click “Deploy.”
• The connector should start working without errors.

Once everything is ready, the OrderService can start receiving requests from the user. These requests will be processed by the service, and the messages will eventually end up in Kafka. If no routing logic is defined for the messages, a default topic will be created:

SMT plugin for topic routingWithout defining a logic for topic routing, Debezium will create a default topic in Kafka named “serverName.schemaName.tableName,” where:

• serverName: The logical name of the connector, as specified by the “database.server.name” configuration property.
• schemaName: The name of the database schema in which the change event occurred. If the tables are not part of a specific schema, this property will be “public.”
• tableName: The name of the database table in which the change event occurred.

This auto generated name might be suitable for some use cases, but in a real-world scenario we want our topics to have a more meaningful name. Another problem with this is that it does not let us logically separate the events into different topics.

We can solve this by rerouting messages to topics based on a logic we specify, before the message reaches the Kafka Connect converter. To do this, Debezium needs a single message transform (SMT) plugin.

Single message transformations are applied to messages as they flow through Connect. They transform incoming messages before they are written to Kafka or outbound messages before they are written to the sink. In our case, we need to transform messages that have been produced by the source connector, but not yet written to Kafka. SMTs have a lot of different use cases, but we only need them for topic routing.

The outbox table schema contains a field called “aggregate_type.” A simple aggregate type for an order related message can be “Order.” Based on this property, the plugin knows that the messages with the same aggregate type need to be written to the same topic. As the aggregate type can be different for each message, it is easy to decide where to route the incoming message.

A simple SMT implementation for topic routing looks like this:

The operation type can be extracted from the Debezium change message. If it is delete, read or update, we simply ignore the message, as we only care about create (op=c) operations. The destination topic can be calculated based on the “aggregate_type.” If the value of “aggregate_type” is “Order,” the message will be sent to the “orderEvents” topic. It is easy to see that there are a lot of possibilities of what we can do with the data, but for now the schema and the value of the message is sent to Kafka along with the destination topic name.

Once the SMT plugin is ready it has to be compiled and packaged as a jar file. The jar file needs to be present on the plugin path of Kafka Connect, so it will be available for the connectors. Kafka Connect will find the plugins using the plugin.path worker configuration property, defined as a comma-separated list of directory paths.

To tell the connectors which transformation plugin to use, the following properties must be part of the connector configuration:

| transforms | outbox | | transforms.outbox.type | com.cloudera.kafka.connect.debezium.transformer.CustomDebeziumTopicTransformer |

After creating a new connector with the SMT plugin, instead of the default topic the Debezium producer will create a new topic called orderEvents, and route each message with the same aggregate type there:

For existing SMT plugins, check the Debezium documentation on transformations.

Aggregate types and partitionsEarlier when creating the schema for the outbox table, the aggregate_type field was used to show which aggregate root the event is related to. It uses the same idea as a domain-driven design: related messages can be grouped together. This value can also be used to route these messages to the correct topic.

While sending messages that are part of the same domain to the same topic helps with separating them, sometimes other, stronger guarantees are needed, for example having related messages in the same partition so they can be consumed in order. For this purpose the outbox schema can be extended with an aggregate_id. This ID will be used as a key for the Kafka message, and it only requires a small change in the SMT plugin. All messages with the same key will go to the same partition. This means that if a process is reading only a subset of the partitions in a topic, all the records for a single key will be read by the same process.

At least once deliveryWhen the application is running normally, or in case of a graceful shutdown, the consumers can expect to see the messages exactly once. However, when something unexpected happens, duplicate events can occur.

In case of an unexpected failure in Debezium, the system might not be able to record the last processed offset. When they are restarted, the last known offset will be used to determine the starting position. Similar event duplication can be caused by network failures.

This means that while duplicate messages might be rare, consuming services need to expect them when processing the events.

ResultAt this point, the outbox pattern is fully implemented: the OrderService can start receiving requests, persisting the new entities into its local storage and sending events to Apache Kafka in a single atomic transaction. Since the CREATE events need to be detected by Debezium before they are written to Kafka, this approach results in eventual consistency. This means that the consumer services may lag a bit behind the producing service, which is fine in this use case. This is a tradeoff that needs to be evaluated when using this pattern.

Having Apache Kafka in the core of this solution also enables asynchronous event-driven processing for other microservices. Given the right topic retention time, new consumers are also capable of reading from the beginning of the topic, and building a local state based on the event history. It also makes the architecture resistant to single component failures: if something fails or a service is not available for a given amount of time, the messages will be simply processed later—no need to implement retries, circuit breaking, or similar reliability patterns.

Try it out yourself!

Application developers can use the Cloudera Data Platform’s Data in Motion solutions to set up reliable data exchange between distributed services, and make sure that the application state stays consistent even under high load scenarios. To start, check out how our Cloudera Streams Messaging components work in the public cloud, and how easy it is to set up a production ready workload cluster using our predefined cluster templates.

Further readingMySQL CDC with Kafka Connect/Debezium in CDP Public Cloud

The usage of secure Debezium connectors in Cloudera environments

Using Kafka Connect Securely in the Cloudera Data Platform

The post Reliable Data Exchange with the Outbox Pattern and Cloudera DiM appeared first on Cloudera Blog.

View Details

We just announced the general availability of Cloudera DataFlow Designer, bringing self-service data flow development to all CDP Public Cloud customers. In our previous DataFlow Designer blog post, we introduced you to the new user interface and highlighted its key capabilities. In this blog post we will put these capabilities in context and dive deeper into how the built-in, end-to-end data flow life cycle enables self-service data pipeline development.

Key requirements for building data pipelinesEvery data pipeline starts with a business requirement. For example, a developer may be asked to tap into the data of a newly acquired application, parsing and transforming it before delivering it to the business’s favorite analytical system where it can be joined with existing data sets. Usually this is not just a one-off data delivery pipeline, but needs to run continuously and reliably deliver any new data from the source application. Developers who are tasked with building these data pipelines are looking for tooling that:

1. Gives them a development environment on demand without having to maintain it.
2. Allows them to iteratively develop processing logic and test with as little overhead as possible.
3. Plays nice with existing CI/CD processes to promote a data pipeline to production.
4. Provides monitoring, alerting, and troubleshooting for production data pipelines.

With the general availability of DataFlow Designer, developers can now implement their data pipelines by building, testing, deploying, and monitoring data flows in one unified user interface that meets all their requirements.

The data flow life cycle with Cloudera DataFlow for the Public Cloud (CDF-PC)Data flows in CDF-PC follow a bespoke life cycle that starts with either creating a new draft from scratch or by opening an existing flow definition from the Catalog. New users can get started quickly by opening ReadyFlows, which are our out-of-the-box templates for common use cases.

Once a draft has been created or opened, developers use the visual Designer to build their data flow logic and validate it using interactive test sessions. When a draft is ready to be deployed in production, it is published to the Catalog, and can be productionalized with serverless DataFlow Functions for event-driven, micro-bursty use cases or auto-scaling DataFlow Deployments for low latency, high throughput use cases.

Figure 1: DataFlow Designer, Catalog, Deployments, and Functions provide a complete, bespoke flow life cycle in CDF-PC

Let’s take a closer look at each of these steps.

Creating data flows from scratchDevelopers access the Flow Designer through the new Flow Design menu item in Cloudera DataFlow (Figure 2), which will show an overview of all existing drafts across workspaces that you have access to. From here it’s easy to continue working on an existing draft simply by clicking on the draft name, or creating a new draft and building your flow from scratch.

You can think of drafts as data flows that are in development and may end up getting published into the Catalog for production deployments but may also get discarded and never make it to the Catalog. Managing drafts outside the Catalog keeps a clean distinction between phases of the development cycle, leaving only those flows that are ready for deployment published in the Catalog. Anything that isn’t ready to be deployed to production should be treated as a draft.

Figure 2: The Flow Design page provides an overview of all drafts across workspaces that you have permissions to

Creating a draft from ReadyFlowsCDF-PC provides a growing library of ReadyFlows for common data movement use cases in the public cloud. Until now, ReadyFlows served as an easy way to create a deployment through providing connection parameters without having to build any actual data flow logic. With the Designer being available, you can now create a draft from any ReadyFlow and use it as a baseline for your use case.

ReadyFlows jumpstart flow development and allow developers to onboard new data sources or destinations faster while getting the flexibility they need to adjust the templates to their use case.

You want to see how to get data from Kafka and write it to Iceberg? Just create a new draft from the Kafka to Iceberg ReadyFlow and explore it in the Designer.

Figure 3: You can create a new draft based on any ReadyFlow in the gallery

After creating a new draft from a ReadyFlow, it immediately opens in the Designer. Labels explaining the purpose of each component in the flow help you understand their functionality. The Designer gives you full flexibility to modify this ReadyFlow, allowing you to add new data processing logic, more data sources or destinations, as well as parameters and controller services. ReadyFlows are carefully tested by Cloudera experts so you can learn from their best practices and make them your own!

Figure 4: After creating a draft from a ReadyFlow, you can customize it to fit your use case

Agile, iterative, and interactive development with Test SessionsWhen opening a draft in the Designer, you are instantly able to add more processors, modify processor configuration, or create controller services and parameters. A critical feature for every developer however is to get instantaneous feedback like configuration validations or performance metrics, as well as previewing data transformations for each step of their data flow.

In the DataFlow Designer, you can create Test Sessions to turn the canvas into an interactive interface that gives you all the feedback you need to quickly iterate your flow design.

Once a test session is active, you can start and stop individual components on the canvas, retrieve configuration warnings and error messages, as well as view recent processing metrics for each component.

Test Sessions provide this functionality by provisioning compute resources on the fly within minutes. Compute resources are only allocated until you stop the Test Session, which helps reduce development costs compared to a world where a development cluster would have to be running 24/7 regardless of whether it’s being used or not.

Figure 5: Test sessions now also support Inbound Connections, allowing you to test data flows that are receiving data from applications

Test sessions now also support Inbound Connections, making it easy to develop and validate a flow that listens and receives data from external applications using TCP, UDP, or HTTP. As part of the test session creation, CDF-PC creates a load balancer and generates the required certificates for clients to establish secure connections to your flow.

Inspect data with the built-in Data ViewerTo validate your flow, it’s crucial to have quick access to the data before and after applying transformation logic. In the Designer, you have the ability to start and stop each step of the data pipeline, resulting in events being queued up in the connections that link the processing steps together.

Connections allow you to list their content and explore all the queued up events and their attributes. Attributes contain key metadata like the source directory of a file or the source topic of a Kafka message. To make navigating through hundreds of events in a queue easier, the Flow Designer introduces a new attribute pinning feature allowing users to keep key attributes in focus so they can easily be compared between events.

Figure 6: While listing the content of a queue, you can pin attributes for easy access

The ability to view metadata and pin attributes is very useful to find the right events that you want to explore further. Once you have identified the events you want to explore, you can open the new Data Viewer with one click to take a look at the actual data it contains. The Data Viewer automatically parses the data according to its MIME type and is able to format CSV, JSON, AVRO, and YAML data, as well as displaying data in its original format or HEX representation for binary data.

Figure 7: The built-in Data Viewer allows you to explore data and validate your transformation logic

By running data through processors step by step and using the data viewer as needed, you’re able to validate your processing logic during development in an iterative way without having to treat your entire data flow as one deployable unit. This results in a rapid and agile flow development process.

Publish your draft to the CatalogAfter using the Flow Designer to build and validate your flow logic, the next step is to either run larger scale performance tests or deploy your flow in production. CDF-PC’s central Catalog makes the transition from a development environment to production seamless.

When you are developing a data flow in the Flow Designer, you can publish your work to the Catalog at any time to create a versioned flow definition. You can either publish your flow as a new flow definition, or as a new version of an existing flow definition.

Figure 8: Publish your data flow as a new flow definition or new version to the Catalog

DataFlow Designer provides first class versioning support that developers need to stay on top of ever-changing business requirements or source/destination configuration changes.

In addition to publishing new versions to the Catalog, you can open any versioned flow definition in the Catalog as a draft in the Flow Designer and use it as the foundation for your next iteration. The new draft is then associated with the corresponding flow definition in the Catalog and publishing your changes will automatically create a new version in the Catalog.

Figure 9: You can create new drafts from any version of published flow definitions in the Catalog

Run your data flow as an auto-scaling deployment or serverless functionCDF-PC offers two cloud-native runtimes for your data flows: DataFlow Deployments and DataFlow Functions. Any flow definition in the Catalog can be executed as a deployment or a function.

DataFlow Deployments provide a stateful, auto-scaling runtime, which is ideal for high throughput use cases with low latency processing requirements. DataFlow Deployments are typically long running, handle streaming or batch data, and automatically scale up and down between a defined minimum and maximum number of nodes. You can create DataFlow Deployments using the Deployment Wizard, or automate them using the CDP CLI.

DataFlow Functions provides an efficient, cost optimized, scalable way to run data flows in a completely serverless fashion. DataFlow Functions are typically short lived and executed following a trigger, like a file arriving in an object store location or an event being published to a messaging system. To run a data flow as a function, you can use your favorite cloud provider’s tooling to create and configure a function and link it to any data flow that has been published to the DataFlow Catalog. DataFlow Functions are supported on AWS Lambda, Azure Functions, and Google Cloud Functions.

Looking ahead and next stepsThe general availability of the DataFlow Designer represents an important step to deliver on our vision of a cloud-native service that organizations can use to enable Universal Data Distribution, and is accessible to any developer regardless of their technical background. Cloudera DataFlow for the Public Cloud (CDF-PC) now covers the entire data flow life cycle from developing new flows with the Designer through testing and running them in production using DataFlow Deployments or DataFlow Functions.

Figure 10: Cloudera DataFlow for the Public Cloud (CDF-PC) enables Universal Data Distribution

The DataFlow Designer is available to all CDP Public Cloud customers starting today. We are excited to hear your feedback and we hope you will enjoy building your data flows with the new Designer.

To learn more, take the product tour or check out the DataFlow Designer documentation.

The post Cloudera DataFlow Designer: The Key to Agile Data Pipeline Development appeared first on Cloudera Blog.

View Details



We are thrilled to announce that the new DataFlow Designer is now generally available to all CDP Public Cloud customers. Data leaders will be able to simplify and accelerate the development and deployment of data pipelines, saving time and money by enabling true self service.

It is no secret that data leaders are under immense pressure. They are being asked to deliver not just theoretical data strategies, but to roll up their sleeves and solve for the very real problems of disparate, heterogenous, and rapidly expanding data sources that make it a challenge to meet increasing business demand for data—and do it all while managing costs and ensuring security and data governance. It’s not just the standard “do more with less”—it’s doing a lot more with less while growing complexity, which makes delivery a painful set of trade-offs.

With relentless focus on transforming business processes to be more responsive to timely, relevant data, we see that most organizations are now distributing data from more sources to more destinations than ever before. In this environment complexity can quickly get out of hand, leaving IT teams with a backlog of requests while impatient LOB users create sub-optimal workarounds and rogue pipelines that add risk. Sometimes referred to as “spaghetti pipelines” or the “Spaghetti Ball of Pain,” our customers describe scenarios where data-hungry LOBs go outside of IT and hack together their own pipelines, accessing the same source data and distributing to different places, often in different ways, paying little to no mind about enforcing data governance standards or security protocols. While the first or second non-sanctioned pipeline might seem like no big deal at first, risk compounds quickly and oftentimes isn’t truly felt until something goes wrong.

Security breach? Good luck getting visibility into the extent of your exposure where rogue pipelines abound. Data quality issue? Good luck auditing data lineage and definitions where policies were never enforced. Massive cloud consumption bill you can’t account for? Good luck controlling all the clusters deployed in haphazard ways. One customer told us bluntly, “If you think you’re not doing data ops, you’re doing data ops that you just don’t know about.”

The holy grail for data leaders is the elusive self-service paradigm, a balance between end user flexibility and centralized control. When it comes to data pipelines, self-service looks like centralized platform admins with visibility and enough control to manage performance and risk, while enabling developers to onboard new data pipelines when needed. A self-service data pipeline platform therefore needs to provide the following:

• Ability to build data flows when needed without having to involve an admin team
• Ability for new users to learn the tool quickly so they are productive
• Ability for developers to deploy their work to production or hand it over to the operations team in a standardized way
• Ability to monitor and troubleshoot production deployments

Self-service in data pipelines has the benefits of reducing costs, helping small administration teams scale to meet demand, accelerated development, and reduced incentive for costly workarounds. Business users benefit from self-service data pipelines as well—being simultaneously better able to develop their own innovative new data-driven solutions and better able to trust the data they are utilizing.

So how are data leaders to strike this balance and enable the self-service holy grail? Enter Cloudera DataFlow Designer.

Back in December we released a tech preview of Cloudera DataFlow Designer. The new DataFlow Designer is more than just a new UI—it is a paradigm shift in the process of data flow development. By bringing the capability to build new data flows, publish to a central catalog, and productionalize as either a DataFlow Deployment or a DataFlow Function, flow developers can now manage the entire life cycle of flow development without relying on platform admins.

Developers use the drag-and-drop DataFlow Designer UI to self-serve across the full life cycle, dramatically accelerating the process of onboarding new data. Resources are made maximally efficient with automated provisioning of infrastructure precisely at that specific point in the cycle and not left running continuously. Each phase is now more efficient:

• Development: Users can quickly build new flows or start with ReadyFlow templates without dependency on admins.
• Testing: With test sessions in a single integrated user experience users can get immediate feedback during development, reducing cycle times that can be extended frustratingly when flow definitions are not properly configured for deployment.
• Publishing: Users have access to a central catalog where they can more easily manage versioning of flows.
• Deployment: Users can work from deployment templates and quickly configure parameters, KPIs to monitor, etc.

Cloudera is delivering the most efficient, most trusted, and most complete set of capabilities on the planet today to capture, process, and distribute high velocity data to drive utilization across the enterprise. Business is demanding more data-driven processes. Developers are demanding more agility. The GA of DataFlow Designer helps our customers deliver on both. Furthermore, customers can realize infrastructure cost savings from a much lighter footprint across the data pipeline life cycle, while giving admin teams visibility and control. Self-service delivers the rapid development and deployment of data flows while combating the hidden costs and risks of rogue pipelines.

For more information or to see a demo, go to the DataFlow Product page.

DemoThe post Self Service is Simply Efficient – Cloudera DataFlow Designer GA announcement appeared first on Cloudera Blog.

View Details

As we celebrate Black History Month, for this #ClouderLife Spotlight we sat down with Clouderan Kimberly Lewis to talk about her career journey in human resources, growing up in New Orleans, and how her experience in basketball translated to leadership and career development in the workplace.

Kim is the director of human resources (HR) programs at Cloudera. She manages everything that happens on a programmatic basis in HR that has a strategy and a plan. “I work closely with the chief human resources officer on strategic planning as well as HR leadership,” she said. She also looks at each of the six HR leadership tracks to identify initiatives and help leadership be successful with pending and ongoing priorities.

Kim joined Cloudera two years ago. “At the time they were looking to fill a strategically focused role, which was a good fit with my past background as an HR business partner (HRBP),” she said. In her previous role, Kim led HRBP teams implementing strategies among all the HR areas. She functioned as the liaison between the field and corporate, where she needed to be an expert across all functional areas, developing a mindset for the larger company strategy.

As soon as she started at Cloudera there was an immediate need for her to implement an initiative around the workforce changes due to COVID-19 restrictions. “My very first responsibility was to oversee the COVID-19 committee at Cloudera,” she said.

“The best thing about Cloudera is the people that I get to connect with daily, there is nothing like it,” she said, describing her experience working at Cloudera. “The people I meet really want to do great work and they are excited about it. They are willing to sacrifice their time completely.”

Growing up in the Big Easy was about family, culture, and new opportunities through the love of basketballKim grew up in New Orleans in a close-knit, multi-generational family, but felt like the greater community of New Orleans was also family. “The passion of the city with a rich history wrapped in a cultural melting pot and anchored in the annual festivities of Mardi Gras. This is what makes New Orleans so special,” she stated.

It seemed like from the time Kim could walk she was involved with the game of basketball. Playing throughout her school days, she later earned a college basketball scholarship. Initially, Kim wanted to study architecture but chose a school that best fit her athletic aspirations that didn’t have architecture as a major. Then, after taking an employment law class she knew HR would be her major, and she has had a passion for it ever since.

With basketball, Kim went on to be a division-two point guard in her college years and then became a referee for semi-pro, and also coached at various levels since.

Soon after college, she settled in Atlanta to start her career where she has been ever since. Kim said, “They say if you’re in Atlanta longer than 10 years then you’re technically from Atlanta. I guess I now qualify.”

How lessons learned on the basketball court translate to the workplace

“Basketball and team sports have been a centerpiece throughout my life,” Kim said. “Being an athlete is very much a part of who I am and started my leadership foundation. I’m very much a servant leader and all about the team.”

Kim draws on lessons learned on the court to the workplace. Regarding organizational success, “working together accomplishes a lot more for yourself and the organization than being a star player or working as an individual,” she said. “Being a great player is one thing, but you don’t win unless you have a great team.”

In terms of looking at strategy, Kim has also drawn similar parallels. “Looking at the bigger picture versus the now is something I learned years ago in sports,” she said. “You might lose the game today knowing that there is a broader or bigger goal to win the championship. Similarly, in HR, for example, there are initiatives that we want to do today but must take a step back and look at the greater strategy to make sure the details align with our goals.”

These lessons also carry over to career development as she looked back at her own career path. “Not being afraid to fail,” she said. “I learned some of the biggest life lessons when things didn’t go right. Being able to reflect on what went well but learning the lessons of what didn’t work and not being afraid to adjust course.” She added, “Like in sports if you run a play and it doesn’t work, do you give up on the game? Or rather, take a step back and look at an alternative way to attack the situation and score?”

Another lesson that Kim attributes to her career progression is not being afraid to think differently or unorthodoxly. “It’s important in my role to be flexible and open to other ways of doing things,” she said. “Sometimes you have the best plans but things happen that change your trajectory. Our whole industry is always changing so we have to be flexible to change with it otherwise we will be left behind.”

Reflecting on Black History MonthIn reflecting on Black History Month Kim said, “It’s such an emotional time for minorities because, on the one hand, we reflect on the struggle, and on the other, the feeling of pride for the accomplishments of minorities. All the great things that we have contributed to society and seeing the progress is exciting.”

“Reflecting as a Black American woman, it wasn’t that long ago when women were first allowed to vote,” she said. “My mother was in school during segregation. When I sent my daughters to school, they had a choice that my mother, who is only two generations apart, did not.” She adds, “Black History, although a month-long event, is up to us to keep top of mind all the time.”

The final question: did you regret not becoming an architect?“I’m still passionate about architecture,” Kim replied. “When I travel, I make it a point to see the architecture of the area, it’s so fascinating to me. I try to get to any city that has fantastic architecture. I’m still a student of architecture. I’m also an ultimate DIYer. I have renovated my home many times over. But at the end of the day, my passion is for people, and HR is my way to contribute.”

As we close the conversation, we want to thank Kim for contributing so much to Cloudera and telling her story.

The post #ClouderaLife Employee Spotlight: Kimberly Lewis, Director of Human Resources Programs appeared first on Cloudera Blog.

View Details

“Cloudera CDO Prefers CDP!” What a shocker! Would I earn an ounce of credibility if you knew that I was previously a Cloudera customer before I became a Clouderan? When I joined L.A.-based nVizhen, a data consulting firm, I became a Cloudera partner. We implemented CDP (Cloudera Data Platform) for innovative G2k organizations so they could tackle transformational use cases and exact real-time insights. I witnessed first-hand the power CDP wielded across multiple industries.

Now I’m here. When CEO Rob Bearden asked me why I wanted to join Cloudera, I told him I had nowhere left to go but in.

But I have good reasons to prefer CDP!First, fullness.

Second, reach.

Thing#1 and Thing#2.As a CDO, I need full data life cycle capability. I must store data efficiently and resiliently, pipe and aggregate data into data lakehouses, and apply machine learning algorithms and AI to uncover actionable insights for our business units. CDP gets me all of it. Snowflake and Databricks can’t get me close. Sure, I could buffet a bevy of other tools and struggle to cobble them together while I boost Bayer stock by abusing their signature product, but even so, I still wouldn’t have shared security, lineage and governance. Cloudera CDP gets me that out of the box. It’s called SDX. Look at the fullness I get with CDP.

Not just fullness, through the entire data lifecycle, but reach over digital geography as well. With AWS, Azure, and GCP, I’m stuck. That’s a problem for me

I need the whole matrix. I need the reach across both dimensions: capability and infrastructure. I want burst flexibility to crunch my year end and quarter end numbers most cost effectively. I must be able to choose where my data resides because like everyone, I’m subject to data privacy regulation. And as business needs evolve, I must be able to shift my data and analytics where they are best placed without having to redo security and compliance, and without having to reimplement, rearchitect or redevelop. CDP provides that portability out of the box.

That brings me to the only two choices I have. 1. Without Cloudera CDP, I forgo security while spending considerable effort rewriting and integrating numerous applications to work on each cloud, and I wouldn’t be able to generate much business value while I do it. 2. With Cloudera CDP I can operate in any cloud without the need to rewrite my applications, and I get integrated lineage, governance, and security.

So for me CDP isn’t just a preference, it’s my only choice.

The post Why I Prefer Cloudera CDP appeared first on Cloudera Blog.

View Details

Recently, we announced enhanced multi-function analytics support in Cloudera Data Platform (CDP) with Apache Iceberg. Iceberg is a high-performance open table format for huge analytic data sets. It allows multiple data processing engines, such as Flink, NiFi, Spark, Hive, and Impala to access and analyze data in simple, familiar SQL tables.

In this blog post, we are going to share with you how Cloudera Stream Processing (CSP) is integrated with Apache Iceberg and how you can use the SQL Stream Builder (SSB) interface in CSP to create stateful stream processing jobs using SQL. This enables you to maximize utilization of streaming data at scale. We will explore how to create catalogs and tables and show examples of how to write and read data from these Iceberg tables. Currently, Iceberg support in CSP is in technical preview mode.

The CSP engine is powered by Apache Flink, which is the best-in-class processing engine for stateful streaming pipelines. Let’s take a look at what features are supported from the Iceberg specification:

As shown in the table above, Flink supports a wide range of features with the following limitations:

• No DDL support for hidden partitioning
• Altering a table is only possible for table properties (no schema/partition evolution)
• Flink SQL does not support inspecting metadata tables
• No watermark support

CSP currently supports the v1 format features but v2 format support is coming soon.

SQL Stream Builder integrationHive Metastore

To use the Hive Metastore with Iceberg in SSB, the first step is to register a Hive catalog, which we can do using the UI:

In the Project Explorer open the Data Sources folder and right-click on Catalog, which will bring up the context menu.

Clicking “New Catalog” will open up the catalog creation modal window.

To register a Hive catalog we can enter any unique name for the catalog in SSB. The Catalog Type should be set to Hive. The Default Database is an optional field so we can leave it empty for now.

The CM Host field is only available in the CDP Public Cloud version of SSB because the streaming analytics cluster templates do not include Hive, so in order to work with Hive we will need another cluster in the same environment, which uses a template that has the Hive component. To provide the CM host we can copy the FQDN of the node where Cloudera Manager is running. This information can be obtained from the Cloudera Management Console by first selecting the Data Hub cluster that has Hive installed and belongs to the same environment. Next, go to the Nodes tab:

Look for the node marked “CM Server” on the right side of the table. After the form is filled out, click Validate and then the Create button to register the new catalog.

In the next example, we will explore how to create a table using the Iceberg connector and Hive Metastore.

Let’s create our new table:

CREATE TABLE `ssb`.`ssb\_default`.`iceberg\_hive\_example` ( `column\_int` INT, `column\_str` VARCHAR(2147483647)) WITH ( 'connector' = 'iceberg', 'catalog-database' = 'default', 'catalog-type' = 'hive', 'catalog-name' = 'hive-catalog', 'ssb-hive-catalog' = 'your-hive-data-source', 'engine.hive.enabled' = 'true') As we can see in the code snippet, SSB provides a custom convenience property ssb-hive-catalog to simplify configuring Hive. Without this property, we would need to know the hive-conf location on the server or the thrift URI and warehouse path. The value of this property should be the name of the previously registered Hive catalog. By providing this option, SSB will automatically configure all the required Hive-specific properties, and if it’s an external cluster in case of CDP Public Cloud it will also download the Hive configuration files from the other cluster. The catalog-database property defines the Iceberg database name in the backend catalog, which by default uses the default Flink database (“default_database”). The catalog-name is a user-specified string that is used internally by the connector when creating the underlying iceberg catalog. This option is required as the connector doesn’t provide a default value.

After the table is created we can insert and query data using familiar SQL syntax:

INSERT INTO `iceberg\_hive\_example` VALUES (1, 'a');SELECT * FROM `iceberg\_hive\_example`; Querying data using Time Travel:

SELECT * FROM `iceberg\_hive\_example` /*+OPTIONS('as-of-timestamp'='1674475871165')*/; Or:

SELECT * FROM `iceberg\_hive\_example` /*+OPTIONS('snapshot-id'='901544054824878350')*/ In streaming mode, we have the following capabilities available:

We can read all the records from the current snapshot, and then read incremental data starting from that snapshot:

SELECT * FROM `iceberg\_hive\_example` /*+ OPTIONS('streaming'='true', 'monitor-interval'='1s')*/ Furthermore, we can read all incremental data starting from the provided snapshot-id (records from this snapshot will be excluded):

SELECT * FROM `iceberg\_hive\_example` /*+ OPTIONS('streaming'='true', 'monitor-interval'='1s', 'start-snapshot-id'='3821550127947089987')*/ ; ConclusionWe have covered how to access the power of Apache Iceberg in SQL Stream Builder and its possibilities and limitations in Flink. We also explored how to create and access Iceberg tables using a Hive catalog and the convenience options in SSB to facilitate the integration, so you can spend less time on configuration and focus more on the data.

Try it out yourself!Anybody can try out SSB using the Stream Processing Community Edition (CSP-CE). CE makes developing stream processors easy, from your desktop or any other development node. Analysts, data scientists, and developers can now evaluate new features, develop SQL-based stream processors locally using SQL Stream Builder powered by Flink, and develop Kafka Consumers/Producers and Kafka Connect Connectors, all locally before moving to production in CDP.

The post Streaming Ingestion for Apache Iceberg With Cloudera Stream Processing appeared first on Cloudera Blog.

View Details

To get the most out of any application, a graphical user interface improves your efficiency

and data streaming without exception. A UI should help you through the steps of an often-complex flow as the visible layer between your problem and solution. Even the most hardcore back end enthusiasts will admit that its significance is undeniable for a complete product. It has to be well organized and easy to understand, yet be able to provide the right tools in the right place. It offers an opportunity to simplify your users’ workflow, provide guidance, and clarify what is possible. For Cloudera’s SQL Stream Builder (SSB), we did not miss this chance.

“SSB was built to give analysts the power of Flink in a no-code interface,” wrote a colleague as a great summary. But it can be read differently depending on where the emphasis is placed. “Power of Flink” sounds exciting enough by itself, so in the next couple of minutes let’s dive into the “no-code interface” part and see where we began one year ago, what we have accomplished, and what challenges we encountered along the way.

A well-built UI from a past eraYou don’t have to be a SSB expert for this post. In a nutshell, our UI offers a feature-rich console with an SQL editor, display log messages, and execution results in real time, as well as providing an intuitive way to manage the following resources:

• SQL Jobs
• Virtual Tables
• User Defined Functions
• Data Sources
• Materialized Views
• API Keys
• Job Notificationshttps://docs.cloudera.com/csa/1.9.0/how-to-ssb/topics/csa-ssb-running-simple-job.html
• Connectors
• Data Formats

Although you could accomplish this with our original UI, it was lacking some important features that users might expect in the era of modern single-page applications. Things like maintaining user state for non-rendered elements, customizing layouts, and performant view transitions have all been attributes of a cutting-edge web application for some time now.

But when rebuilding an existing repository from scratch with a different tech stack, you should also focus on developer experience and long-term maintainability. It is obvious that making changes to a large JavaScript file takes more effort compared to a component-based approach where we only deal with smaller blocks. This is how we went from working with a mixture of vanilla JavaScript, Vue.js, and some helper libraries to unleashing the power of Angular and TypeScript.

The Angular wayWas this the perfect choice? If we remove all biases, no one knows for sure, but it’s more than enough to reach our goals. To mention a few benefits that we got out of the box:

• Well-structured repository
• Readable and testable code
• Fast development process
• Excellent component library.

In short, Angular is a full-fledged framework that provides solutions for server communication, routing within your application, and much more. TypeScript is the primary language for Angular application development, and also comes with many benefits in terms of language features, reference validation, project scalability, and code maintainability.

So we have our framework and language, but there are other things that have to be taken into consideration to prepare ourselves for the wilderness, to be able to deliver solutions quickly and with confidence.

Internal component libraryAngular CLI offers many features including a built-in code generator to generate components and other building blocks based on a schematic, but the end result is just a blueprint. At the time of writing, Angular does not support generating magically finished components that just fit well into your application.

Since we have to code them, we want to reuse as much of them as possible. Even though SSB UI has a lot of custom ones, the idea is to create generic, reusable components, services, and style definitions, and put them in a separate library for the benefit of other applications.

This library is created by talented Clouderans and based on the popular Angular Material component library and CDK. It proved itself a helpful companion on our journey.

State managementWhen a web page is loaded, the browser creates a Document Object Model (DOM) of the document that represents it as nodes and objects.

To keep our application performant we want to avoid an excessive DOM size and get rid of unused elements, but at the same time we want to preserve the state of specific views. If the user has interacted with it, or if it’s already populated with items rendered from a back end response we don’t want to lose them.

Our previous UI had multiple pages that you could visit using the sidebar, but user states were not stored, so switching pages meant losing states.

The Redux inspired NgRx library is an obvious choice for Angular. Along with organizing the application state and making API interactions consistent, with NgRx we can also simplify the communication between multiple components by managing a common state that can be accessed from anywhere. This enables us to implement the following features in a couple of lines:

• Navigate away from the active job without losing its state.
• Send log entries to the log viewer from any component via the injected NgRx store.
• Load resources once and keep the results in state.

The last point was not that straightforward. While NgRx state is useful for storing changes made by the user it doesn’t support the easy management of large data sets. This is not a new problem and there are great solutions to it already. One of them can be found in our internal library. It’s a utility based on ngrx/data (but simpler to use) that loads a given collection and makes it available in the store. This made us ready to easily manage large data tables.

API generationTo load resources and manage our Flink jobs, the SSB front end calls back end endpoints. No surprise so far. But whenever our back end API changes, the front end needs to follow. Introducing API generation based on our back end’s Swagger JSON files made it possible to catch errors at build time. Angular’s http client doesn’t have to be imported anywhere as it’s only used in the generated services. We also don’t have to worry about DTO type safety as interfaces are also auto generated based on the back end code. Pretty cool compared to what we had before.

SQL editorAnother improvement was the replacement of one of the central components of our UI. Switching from CodeMirror to Monaco Editor (the code editor that powers VS Code) came with many features out of the box. It is worth checking its playground if you have a similar use case.

New possibilities bring new challengesWhile the transition to our shiny new framework and libraries was taking place, SSB’s back end feature set kept evolving and its new capabilities required quick responses from the UI team.

After covering the architectural side of our new front end, let’s see how it supported the new UX that we were aiming for.

If you made it to this point, you deserve an image of the new, redesigned UI of SQL Stream Builder:

Restructured layoutAs part of the recently introduced SDLC concept, our resources are now organized as projects that can be synchronized with a remote repository.

This required a complete rework of what we had before. We no longer have separate routes for resources. Once you select a project you will step into an integrated, IDE-like layout that has two resizable sections.

On the left side you can find an explorer tree along with other project related views. From the explorer you can easily access resources within a project’s scope. On the right there is a workspace section with tabs for each opened resource.

This structure allows you to set up custom layouts. You can either see everything at once or only focus on what matters.

Adding a new context to the existing NgRx state enabled us to maintain all project-related information including layout state and tab management.

SSB’s project explorer is a tree component that contains resources of the active project as well as external ones. Resources can be opened and managed directly from the tree.

WorkspaceA tab group is the ultimate component for easy context switching. Tabs in SSB are loaded via URL, so reloading the page will load the same tabs.

ConclusionWe have covered the decisions that have been made to build a new UI that our product deserves. First we discussed the technical considerations, then some of the UX related improvements. Our API provides a lot of features already and the UI does not settle for less either. This makes Cloudera’s SQL Stream Builder one of the best options out there when it comes to data streaming with Flink.

Try it out yourself!Anybody can try out SSB using the Stream Processing Community Edition (CSP-CE). The Community Edition makes developing stream processors easy, as it can be done right from your desktop or any other development node. Analysts, data scientists, and developers can now evaluate new features, develop SQL-based stream processors locally using SQL Stream Builder powered by Flink, and develop Kafka Consumers/Producers and Kafka Connect Connectors, all locally before moving to production in CDP.

The post A UI That Makes You Want to Stream appeared first on Cloudera Blog.

View Details

In 2022, Cloudera had some great results – over 3,000 hours volunteered, $680,000 donated and stories of groups of Clouderans getting together to give back, worldwide.

What’s most important to us is the individual lives impacted.

In 2022, Cloudera supported:

• Mentees to navigate early stages of their careers.
• Veterans to re-build confidence through sport and re-engage with careers.
• Under-represented people to re-train into tech.
• Elderly people with notes of friendship in their native language.
• Clouderans to feel included at work.

Every part of our Cloudera Cares work affects an individual that has a story to tell, and that’s what we are celebrating in this report.

Read the full report here, or watch the video:

The post Cloudera’s Impact Report 2022 is Live! appeared first on Cloudera Blog.

View Details

Every day in the US thousands of legitimate prescriptions for the opioid class of pharmaceuticals are written to mitigate acute pain during post-operation recovery, chronic back and neck pain, and a host of other cases where patients experience moderate-to-severe discomfort.

While many of these drugs provide pain relief, the potential for misuse and outright abuse due to their addictive nature is extremely high. Since the 1990s, opioid abuse in the US skyrocketed to the point that in 2017 the Department of Health and Human Services declared the opioid crisis a public health emergency. This epidemic affects more than just individuals. Families and communities, as well as the US healthcare system at the federal, state, and local level, carry the burden emotionally and financially.

Solving this crisis can and should be supported by insights gained from the vast amounts of data available to healthcare providers and government agencies. The healthcare industry, in partnership with government agencies, must engage with a robust modern data management solution that can extract electronic health data in the form of controlled substance data, prescriber data, and patient data and transform it into meaningful, measurable, and actionable information.

The power of data to save livesKeck Medicine of USC (KMC), a university-based medical system in Southern California, has taken on this challenge with the help of Cloudera’s data management platform, CDP (Cloudera Data Platform). With CDP, KMC was able to develop the aptly named Controlled Substance Analytics platform to leverage data analytics in the fight against opioid abuse and misuse.

CDP gives KMC’s platform the ability to connect to any data source of any type (structured, unstructured, streaming) and the open data lakehouse enables multi-function analytics on both streaming and stored data. Data teams are empowered to collaborate with the tools of their choice all with the security and governance needed to meet the highly regulated healthcare industry compliance requirements.

Prior to implementation, basic tasks such as analyzing pharmacy orders for conspicuous opioid prescribing practices were resource-constrained and burdened by time-consuming manual processes, yielding little actionable insight.

With the Controlled Substance Analytics platform online, KMC has eliminated manual data collection and streamlined data processing. Each day, multiple data sets, including prescriptions and patient health records, are loaded from the electronic medical records (EMR) system directly into a Cloudera enterprise data lakehouse. Quantitative values are assigned to complex risk factors to illustrate trends, severity, and key indicators allowing KMC to perform the most complex data analytics to identify those at high risk for abuse or misuse of opioids.

In the healthcare industry and across the public sector leveraging data is critical to solving some of society’s biggest challenges, including the fight against opioid abuse.

Join us for an exclusive industry webinar on March 2, 2023, to hear KMC data leaders discuss specific examples of how data analytics have helped them identify suspicious prescription activity, limit the abuse of controlled substances, and much more.

The post Leveraging Data Analytics in the Fight Against Prescription Opioid Abuse appeared first on Cloudera Blog.

View Details

Cloudera’s SQL Stream Builder (SSB) is a versatile platform for data analytics using SQL. As apart of Cloudera Streaming Analytics it enables users to easily write, run, and manage real-time SQL queries on streams with a smooth user experience, while it attempts to expose the full power of Apache Flink. SQL has been around for a long time, and it is a very well understood language for querying data. The SQL standard has had time to mature, and thus it provides a complete set of tools for querying and analyzing data. Nevertheless, as good as it is sometimes it is necessary, or at least desirable, to be able to extend the SQL language for our own needs. UDFs provide that extensibility.

What is a UDF and why do we need it?SQL is a very useful language for querying data, but it has its limitations. With UDFs you can really enhance the capabilities of your queries. In SSB, today we are supporting JavaScript (JS) and Java UDFs, which can be used as a function with your data. Below we will show an example on how to create and use a JS UDF.

In the following example we use ADSB airplane data. ADSB is data about aircraft. The data is generated and broadcast by planes while flying. Anyone with a simple ADSB radio receiver can acquire the data. The data is very useful, and luckily easy to understand. The data consists of a plane ID, altitude, latitude and longitude, speed, etc.

For our UDF we would like to use the longitude value in order to find out what time zone the plane is in, and output a time zone value as an offset from the GMT time zone (i.e. GMT -3).

The ADSB raw data queried using SSB looks similar to the following:

For the purposes of this example we will omit the explanation of how to set up a data provider and how to create a table we can query. But let’s assume we have already set up such a table, based off of a Kafka topic that has the ADSB data streaming through it, and we have named it airplanes. Please check our documentation to see how that’s done.

The raw data above could be acquired by simply issuing the following SQL statement:

SELECT * FROM airplanes; As we stated earlier we would like to deal with the longitude values and use them to be able to generate a time zone in the usual GMT +- format. We are also not interested in rows that don’t contain a longitude so we can exclude those. We can also exclude most columns other than the icao, lon and the value we will generate. To achieve our goal, the SQL we require might look something like this:

SELECT icao, lon, TOTZ(lon) as `timezone` FROM airplanes WHERElon <> ‘’; The UDF (TOTZ)TOTZ does not yet exist. TOTZ is the custom UDF that we would need to craft in order to convert a longitude to a time zone, and output the appropriate string.

Planning the UDFA decimal longitude value can be converted to a time in seconds from the GMT by dividing the longitude by 0.004167:

Longitude / 0.004167 = seconds from GMT

Once we have the number of seconds from GMT we can calculate the hours from GMT by dividing the seconds from GMT by 3600 (3600 is the number of seconds in one hour):

Seconds from GMT / 3600 = hours from GMT

Finally we are only interested in the total number of hours from GMT, not in its remainder (minutes and seconds), so we can eliminate the decimal portion from the hours from GMT value. For example for Kahului, Maui, Hawaii, the longitude is -156.474, then:

-156.474 / 0.004167 = -37550.756s

To hours:

-37550.756 / 3600 = -10.43h

Thus our function should output “GMT -10”. Currently UDFs can be crafted using the JavaScript programming language in SSB (and Java UDFs can be uploaded, but in our post we are using JS). By right clicking on “Functions” and then the “New Function” button, a user can create a new UDF. A popup opens up and the UDF can be created. The UDF requires a “Name” one or more “Input Type”, an “Output Type” and the function body itself. The JS code has just one requirement, and that is that the last line must return the output value. The code receives the input value as the variable named “$p0”. In our case $p0 is the longitude value.

In case we want to pass multiple parameters to our function that can be done as well, we only need to make sure to adapt the last line accordingly and add the proper input types. For example if we have function myFunction(a, b, c) { … }, the last line should be myFunction($p0, $p1, $p2), and we should match the number and kind of the “Input Types” as well.

UDF code function totz(lon){ var numLon = Number(lon); if (isNaN(numLon) || lon == "") { return ""; } var seconds = numLon / 0.004167; var hours = seconds / 3600; // Return only the hours portion, and discard the minutes hours = Math.floor(hours); return "GMT " + (hours > 0 ? "+" : "-") + hours;}totz($p0); // this line must exist Testing the UDFAfter creating our UDF we can try our SQL and see what it produces.

Our TOTZ UDF did the job! We were able to quickly and easily extend the SQL language, and use the new UDF as if it was a native SQL function, and based off of the longitude value it was able to produce a string representing the time zone that the plane is flying through at the time.

ConclusionIn summary, Cloudera Stream Processing gives us the ability to build UDF’s and deploy continuous jobs directly from the SQL Stream Builder interface so that you can build streaming analytics pipelines that execute advanced/custom business logic. The creation and use of UDFs is simple, and the logic can be written using the usually familiar JavaScript programming language.

Try it out yourself!Anybody can try out SSB using the Stream Processing Community Edition (CSP-CE). CE makes developing stream processors easy, as it can be done right from your desktop or any other development node. Analysts, data scientists, and developers can now evaluate new features, develop SQL-based stream processors locally using SQL Stream Builder powered by Flink, and develop Kafka Consumers/Producers and Kafka Connect Connectors, all locally before moving to production in CDP.

The post Implementing and Using UDFs in Cloudera SQL Stream Builder appeared first on Cloudera Blog.

View Details

SQL Stream Builder (SSB) is a versatile platform for data analytics using SQL as a part of Cloudera Streaming Analytics, built on top of Apache Flink. It enables users to easily write, run, and manage real-time continuous SQL queries on stream data and a smooth user experience.

Though SQL is a mature and well understood language for querying data, it is inherently a typed language. There is a certain level of consistency expected so that SQL can be leveraged effectively. As an essential part of ETL, as data is being consolidated, we will notice that data from different sources are structured in different formats. It might be required to enhance, sanitize, and prepare data so that data is fit for consumption by the SQL engine. Data transformations in SSB gives us the ability to do exactly that.

What is a data transformation?Data transformation in SSB makes it possible to mutate stream data “on the wire” as it is being consumed into a query engine. This transformation can be performed on incoming records of a Kafka topic before SSB sees the data.

A few use cases when transformations can be a powerful tool:

• If the data being collected has sensitive fields that we choose not to expose to SSB.
• If the Kafka topic has CSV data that we want to add keys and types to it.
• If the data is in valid JSON format, but has non Avro compatible field names, has no uniform keys, etc.
• If the messages are inconsistent.
• If the schema you want does not match the incoming Kafka topic.

Similar to UDFs, data transformations are by default written in JavaScript. The one requirement that we do have is that after the data transformation is completed, it needs to emit JSON. data transformations can be defined using the Kafka Table Wizard.

The use caseThe data we are using here is security log data, collected from honeypots: invalid authentication attempts to honeypot machines that are logged and published to a Kafa data source.

Here is an excerpt of the log entries in JSON that is streamed to Kafka:

{"host":"honeypot-fra-1","@version":"1","message":"Sep 11 19:01:27 honeypot-fra-1 sshd[863]: Disconnected from invalid user user 45.61.184.204 port 34762 [preauth]","@timestamp":"2022-09-11T19:01:28.158Z","path":"/var/log/auth.log"}{"@timestamp":"2022-09-11T19:03:38.438Z","@version":"1","message":"Sep 11 19:03:38 honeypot-sgp-1 sshd[6605]: Invalid user taza from 103.226.250.228 port 41844","path":"/var/log/auth.log","host":"honeypot-sgp-1"}{"@timestamp":"2022-09-11T19:08:30.561Z","@version":"1","message":"Sep 11 19:08:29 honeypot-sgp-1 kernel: [83799422.549396] IPTables-Dropped: IN=eth0 OUT= MAC=fa:33:c0:85:d8:df:fe:00:00:00:01:01:08:00 SRC=94.26.228.80 DST=159.89.202.188 LEN=40 TOS=0x00 PREC=0x00 TTL=240 ID=59466 PROTO=TCP SPT=48895 DPT=3389 WINDOW=1024 RES=0x00 SYN URGP=0 ","path":"/var/log/iptables.log","host":"honeypot-sgp-1"} You probably notice a couple of non Avro compatible field names in the data, one of them being @timestamp, which contains an ISO formatted timestamp of when the security incident occurred. If you ingest this log data into SSB, for example, by automatically detecting the data’s schema by sampling messages on the Kafka stream, this field will be ignored before it gets into SSB, though they are in the raw data.

Further, if we’ve elected to use “Kafka event timestamps” as SSB row times, the timestamp that SSB records will be the time it was injected into Kafka. This might be OK for some cases. However, we will probably want to base our query on when a security incident actually happened.

We will solve this problem in three steps:

1. Write a data transformation that creates a new field with an Avro compatible name in each JSON entry. We populate the field with the value in the non Avro compatible @timestamp field.
2. We will change the schema of the data to include the new field that we emitted in step 1.
3. We will tell SSB to use this new field, that is now part of the schema as the event timestamp.

The data transformationThis data transformation should happen before the events are written into the SSB table. You can find “Data Transformation” as one of the tabs under the table.

At the core of the data transformation there is a “record” object that contains the payload of the log data. The data transformation is set up as a construct under the table.

We will want to create a new field called data_timestamp that is processed from the @timestamp field. We will create a local scoped variable to access the record’s payload dictionary. The timestamp field is parsed using the JavaScript Date module and added to a new key on the payload. We can, at that point, sanitize the fields that are not Avro compatible, and return it as a stringified JSON object.

var payload = JSON.parse(record.value);var output = payload;output['data\_timestamp'] = Date.parse(payload['@timestamp']);delete output['@timestamp'];delete output['@version'];JSON.stringify(output); We can now add the new field data_timestamp into the schema so that it will be exposed to SQL queries. We could just add the following fragment describing the new field and its time into the schema under the “Schema Definition” tab:

{"name" : "data\_timestamp","type": "long", "doc": "Injected from a custom data transformation" } The last step is to change the Kafka row time to use the new row that we just created. That function can be found under the “Event Time” tab’s “Input Timestamp Column.”

We can review the DDL changes that are going to be applied to the schema itself on “Update and Review.”

To summarize:

• A new big integer data_timestamp field is added.
• The eventTimestamp is used as the row time, formatted from the data_timestamp.

ConclusionIn this module, we have taken a deeper look at SSB’s data transformations. We looked at how to write a data transformation in JavaScript to extract a field from the payload and format it into a timestamp that can be configured as the SSB row time.

Try it out yourself!Anybody can try out SSB using the Stream Processing Community Edition (CSP-CE). The Community Edition makes developing stream processors easy, as it can be done right from your desktop or any other development node. Analysts, data scientists, and developers can now evaluate new features, develop SQL-based stream processors locally using SQL Stream Builder powered by Flink, and develop Kafka Consumers/Producers and Kafka Connect Connectors, all locally before moving to production in CDP.

Check out the full recording of the Deploying Stateful Streaming Pipelines in Less Than 5 Minutes With CSP Community Edition.

The post SQL Streambuilder Data Transformations appeared first on Cloudera Blog.

View Details

This month’s #ClouderaLife Spotlight features software engineer Amogh Desai. Here we discuss his background, how he got started at Cloudera, and his recent win at the Cloudera 2022 Global Hackathon.

Snatching victory from the jaws of defeatAmogh and his fellow hackathon team members felt the rush of victory after winning Cloudera’s 2022 global hackathon in the product development category. The team was on the road to redemption after coming just short of winning the last regional hackathon.

The global hackathon was composed of teams across the business to compete on the best new product ideas. Each team had 48 hours straight to develop their idea for demonstration to the contest judges.

“It was a great feeling to go home with the winning idea,” Amogh said.

Amogh and his team’s winning idea was a “Smart Instance Selector” which automates the identification of the most appropriate machine instances when deploying a product. This would solve the problem of relying on a customer who would have to select the instance. Selecting the instance by the customer is an imprecise method since there are many variables in getting it right, like machine type, size, and number of machines. It also happens that the cloud providers update their instance types and deprecate them all the time leading to installation failures, making the customers feel that the software is faulty when truly it is the hardware.

Meet Amogh DesaiAmogh lives in Bangalore and joined Cloudera, first as an intern and then full-time in July of 2021 as a software engineer. His day-to-day consists of development activities like writing and reviewing code, working on features around release timelines, and participating in design meetings for the team supporting the CDP Data Engineering product.

CDP Data Engineering helps offload orchestration and automation processes from in-house data engineers and data practitioners to free them up to get to work on their day-to-day responsibilities.

Amogh has the unique experience of working on CDP Data Engineering during his internship. At the time the product was still in its infancy. Then, when he joined Cloudera full time he continued to work on the product and saw it brought to market. “I got a lot of satisfaction out of working on it from the beginning and now to see customers using it,” said Amogh.

Prior to Cloudera, Amogh attended the Peoples Education Society (PES) University in Bangalore. PES University is the country’s leading teaching and research university. There Amogh majored in computer science and engineering with a specialization in data science. From there he was recruited into the Cloudera internship program.

Amogh is a second-generation engineer, his father, an automotive engineer, moved to various cities with his family as new opportunities arose. Amogh’s younger brother is also a budding engineer who just started his studies.

Working at Cloudera and the road aheadAmogh has really enjoyed working at Cloudera. “My work at Cloudera keeps me challenged. It’s like I have a new puzzle to solve every day. I like that as I feel I’m continually growing,” he said. “I also have a brilliant team that I can depend on, and we have a lot of fun. Also, I have been impressed with how encouraging management has been. They always give recognition to those who have worked hard.”

In reflecting on his career up to this point, Amogh’s advice to someone new to software development is to not be afraid to fail. “Often, we want to do everything perfectly or not show the team or boss that we made mistakes. But this holds one back because we really grow from making mistakes. Not to mention software is always evolving so you can’t possibly know everything. Ask questions and explore and people will help you.”

In visualizing where he’d like to take his career in the next five years, Amogh said, “I’d like to continue in the development track and grow my skills. I want to learn more and start moving into product architecture in the next two years while gaining experience in open-source software and then lead a team of junior developers.”

Speaking of learning more, Amogh is always trying to learn more about technology whether that be reading about it or discussing it with others. One way he does this is through blog writing. Check out his Blog on Medium where he discusses productivity and creative thinking around software development.

Thanks, Amogh, for your contributions to Cloudera and for sharing your story.

The post #ClouderaLife Spotlight: Amogh Desai, Software Engineer II appeared first on Cloudera Blog.

View Details

Special co-author credits: Adam Andras Toth, Software Engineer Intern

With enterprises’ needs for data analytics and processing getting more complex by the day, Cloudera aims to keep up with these needs, offering constantly evolving, cutting-edge solutions to all your data related problems. Cloudera Stream Processing aims to take real-time data analytics to the next level. We’re excited to highlight job monitoring with notifications, a new feature for SQL Stream Builder (SSB).

What problem are we solving with job notifications?The sudden failing of a complex data pipeline can lead to devastating consequences—especially if it goes unnoticed. A real-time financial fraud detector, or a complex architecture collecting and aggregating data to create insights and allow their customers to make data-driven decisions—these are systems that have little to no room for error or extended downtime. This is why we build job notifications functionality into SSB, to deliver maximum reliability in your complex real-time data pipelines.

How job notifications will make your life easierJob notifications can help make sure that you can detect failed jobs without checking on the UI, which can save a lot of time for the user. This feature is very useful, especially when the user has numerous jobs running and keeping track of their state would be hard without notifications.

ArchitectureFirst, we would like to introduce the architecture of job notifications. Let us use a figure to demonstrate how job notifications fit into SSB, then we will discuss each type separately.

OverviewIn SSB you can manage multiple projects. Projects represent the software development life cycle (SDLC) in SQL Stream Builder (SSB): you can create an environment for developing SQL jobs, verifying the results and then pushing them to production. The resources for job creation are shared within the jobs of a project, and there are resources available that can be used between the projects. The basic concept of a project can also be expanded for collaboration by sharing the projects with team members in streaming SQL console, or using source control to synchronize the project with a Git repository.

Job notifications also belong to projects. That means in one project you can define multiple notifications, and those notifications can only belong to the jobs of that project. In the figure below, you can see the architecture of a project from the perspective of job notifications. As of now there are two types of notifications: email and webhook. The notifications can also be organized into groups. The benefit of this is that if you want to assign the same set of notifications to multiple jobs you don’t have to do this one by one in every job, you can just create a notification group and assign that to the jobs. One notification can be included in multiple groups and a group can even contain another group.

In the figure below, the same job notifications are marked with the same color. As you can see in the project we have three jobs. In the first one we only have notifications, so if that job fails these four notifications will fire. In the second one we have a webhook notification and a notification group that has another webhook and an email notification, so if this job fails these three notifications will go off. The third job has a webhook notification, a group that contains an email notification, and another notification group that has two notifications, so if this job fails these four notifications will fire.

NotificationsAs I mentioned before, there are two types of notifications and you can assign them to groups. I will first introduce placeholders, which you can use to create notifications.

PlaceholdersThe email message or webhook request that is sent upon the trigger for a notification can be completely customized. More than that, SSB also allows the usage of placeholders, which can be used to provide all necessary information in the notification. With the ability to customize messages and to use placeholders, users will also potentially be able to automatically parse the incoming notifications and create automatic responses for them, thus guaranteeing that critical pipelines can be restarted without requiring human intervention.

The placeholders currently available for usage are:

• jobName
• jobStatus
• jobStatusDescription
• ssbJobId
• flinkJobId
• clusterId
• lastException

You can use a placeholder in the following format: “Houston we have a problem, your job with name ${jobName} has failed.”

Email notificationsEmail notifications are (as you could guess from its name) sending emails to the given email address upon job failure. To make this work some CM properties need to be configured:

• Mail server host for job notifications: The host of the SMTP server for job failure notifications
• Mail server username for job notifications: The username to access the SMTP server for job failure notifications
• Mail server password for job notifications: The password to access the SMTP server for job failure notifications
• SMTP authentication for job notifications: Enable SMTP authentication for job notifications (default value: True)
• StartTLS for job notifications: Use the StartTLS command to establish a secure connection to the SMTP server for job notifications (default value: True)
• Job notifications sender mail address: Sender mail address for job notifications
• Mail server port for job notifications: The port of the SMTP server for job failure notifications (default value: 587)

If you have these things set up properly and you add a notification to your job, you should get an email if the job fails.

Webhook notificationsWith webhook notifications you can make webhook requests upon a job failure. If you use the placeholders correctly, then you can use the defined webhook endpoints of external applications to handle the failures in a more efficient way. (For example, you can set up a webhook notification with Slack to send you a message directly if a job fails.)

In the case of webhook notifications you can set one property in CM:

• Job notifications webhook sender parallelism: Number of threads used by the job notification task to call user-specified webhooks when notifying about a failed or missing job (default value: 10)

DISCLAIMER: The payload template of a webhook notification must be a valid JSON! Also make sure to put placeholders within quotes!

E.g.:

• “name”: ${jobName} is invalid
• “name”:”${jobName}” is valid
• “name”:”whatever i want here ${jobName}” is also valid

Notification groupsAs I mentioned above you can assign your notifications into groups. This way you don’t need to add all the notifications to the jobs one by one. A cool thing about the groups is that they can also contain other notification groups.

How to use job notificationsSSB’s job notifications feature is a cool way to keep track of your failing jobs and thus minimize the downtime of them. You just need to make sure the “enable job notifications” functionality in CM is checked. The job-monitoring task periodically queries the state of your jobs, and triggers the assigned notifications if a failed job is found. The check interval in CM can be configured with the job notifications monitoring interval property (default value: 60s).

In this section I will show you some video examples for the usages of the job notifications.

Create and use an Email notification:Create and use a Webhook notification:Create and use a Notification GroupTry it out yourself!Anybody can try out SSB using the Stream Processing Community Edition (CSP-CE). CE makes developing stream processors easy, as it can be done right from your desktop or any other development node. Analysts, data scientists, and developers can now evaluate new features, develop SQL-based stream processors locally using SQL Stream Builder powered by Flink, and develop Kafka Consumers/Producers and Kafka Connect Connectors, all locally before moving to production in CDP.

The post Job Notifications in SQL Stream Builder appeared first on Cloudera Blog.

View Details

How Bad is Bad Code: The ROI of Fixing Broken Spark CodeOnce in a while I stumble upon Spark code that looks like it has been written by a Java developer and it never fails to make me wince because it is a missed opportunity to write elegant and efficient code: it is verbose, difficult to read, and full of distributed processing anti-patterns.

One such occurrence happened a few weeks ago when one of my colleagues was trying to make some churn analysis code downloaded from GitHub work.

I was looking for some broken code to add a workshop to our Spark Performance Tuning class and write a blog post about, and this fitted the bill perfectly.

For convenience purposes I chose to limit the scope of this exercise to a specific function that prepares the data prior to the churn analysis.

Here it is in all its glorious juiciness:from pyspark.sql.functions import udf,colfrom pyspark.sql.types import IntegerTypedef prepare\_data\_baseline(df): ''' Function to prepare the given dataframe and divid into groups of churn and non churn users while returnng the original datafrme with a new label column into a spark dataframe. Args: df- the original dataframe Returns: df - dataframe of the dataset with new column of churn added stayed - dataframe of the non -churn user's activities only. all\_cancelled - dataframe of the churn user's activities only. ''' *#Define a udf for cancelled* canceled = udf(lambda x: 1 if x == 'Cancellation Confirmation' else 0) *#define a new column 'churn' where 1 indicates cancellation of subscription, 0 otherwise* df = df.withColumn('Churn', canceled(df.page)) *#Dataframe of all that cancelled* cancelled\_df = df.select('page', 'userId','Churn').where(col('churn')==1) *#List of cancelled* list\_cancelled = cancelled\_df.select('userId').distinct().collect()*#list of cancelled users* *#Put in a list format* gb = []*#temporary variable to store lists* for row in list\_cancelled: gb.append(row[0]) canc\_list = [x for x in gb if x != '']*#remove the invalid users* *#Total number of users who canceled* print(f"The number of churned users is: {len(canc\_list)}") *#List of staying users* all\_users = df.select('userId').distinct().collect() gh = []*#a temporary variable to store all users* for row in all\_users: gh.append(row[0]) stayed\_list = set(gh)-set(gb)*#list of users staying* stayed\_list = [x for x in stayed\_list if x != '']*#remove the invalid users* *#Total number of users who did not cancel* print(f"The number of staying users is: {len(stayed\_list)}") *#Store both canceled and staying users in new dataframes containng all actions they undertook* all\_cancelled = df.select("*").where(col('userId').isin(canc\_list)) stayed = df.select('*').where(col('userId').isin(stayed\_list)) *#Redefine a udf for churn* churned = udf(lambda x: 0 if x in stayed\_list else 1, IntegerType()) *#Creat new column which will be our label column to track all users that eventually cancelled their subscription* df = df.withColumn('label', churned(col('userId'))) return df, stayed, all\_cancelled In this blog post, I will outline the steps I took to fix this code, and then measure the resulting difference in execution performance. In the process, I will explicitly state the best practices I will implement.

Let’s jump in this rabbit hole!

Define a non-regression test harnessStop!

Resist the temptation to start tweaking the code right away!

You want to be able to:

• Make sure that you do not introduce a regression by fixing the code
• Measure the improvements in terms of performance

This is where limiting the scope of the analysis to a function came in handy: it allowed me to use ad hoc and simple tooling:

• I isolated the original function in a prepare_data_baseline function in a separate prepareData_baseline.py file
• I created a new file called prepare_data.py with the new version of the prepare_data function
• I measured the time to perform the processing using the time library
• And I compared the resulting DataFrames with subtract

Because lazy evaluation defers the time when the code is actually executed, I added code that saves the DataFrames to files, thus forcing the materialization of the DataFrames via the execution of the code. I also added these lines in the scope of the time measurement.

And this is what it looks like:

from pyspark.sql import SparkSessionimport time, datetimefrom prepareData import prepare\_datafrom prepareData\_baseline import prepare\_data\_baselinespark = SparkSession \ .builder \ .appName("Churn Analysis Data Preparation Test Harness") \ .getOrCreate()spark.sparkContext.setLogLevel("ERROR")spark.conf.set('spark.sql.adaptive.enabled','false')print(f"AQE enabled: {spark.conf.get('spark.sql.adaptive.enabled')}")df = spark.read.json('data/mini\_sparkify\_event\_data.json')*#Baseline version*process\_time\_start = time.perf\_counter() *# Start timer: begin processing*df\_baseline, stayed\_baseline, all\_cancelled\_baseline = prepare\_data\_baseline(df)df\_baseline.write.mode("overwrite").json('data/df\_baseline')stayed\_baseline.write.mode("overwrite").json('data/stayed\_baseline')all\_cancelled\_baseline.write.mode("overwrite").json('data/all\_cancelled\_baseline')process\_time\_end = time.perf\_counter() *# Stop timer: end processing*process\_time = process\_time\_end - process\_time\_start *# Elapsed time for processing*totalTime = datetime.timedelta(seconds = process\_time)print(f"Preparing data took with the baseline version took {totalTime}")*#New version*process\_time\_start = time.perf\_counter() *# Start timer: begin processing*df, stayed, all\_cancelled = prepare\_data(df)df.write.mode("overwrite").json('data/df')stayed.write.mode("overwrite").json('data/stayed')all\_cancelled.write.mode("overwrite").json('data/all\_cancelled')process\_time\_end = time.perf\_counter() *# Stop timer: end processing*process\_time = process\_time\_end - process\_time\_start *# Elapsed time for processing*totalTime = datetime.timedelta(seconds = process\_time)print(f"Preparing data took with the new version took {totalTime}")*# Regression Testing*def diffDataFrame(df1,df2): return df1.subtract(df2).count()print(f"New processing introduced {diffDataFrame(df,df\_baseline)} differences in df.")print(f"New processing introduced {diffDataFrame(all\_cancelled,all\_cancelled\_baseline)} differences in all\_cancelled.")print(f"New processing introduced {diffDataFrame(stayed,stayed\_baseline)} differences in stayed.")spark.stop() Retro document the requirementsThis step was quite easy because of the comments that were present in the initial code.

This function:

• Takes a DataFrame containing activities from users,
• splits it into two groups of activities:
  • activities from users who eventually churned and
  • activities from users who did not, and
• adds a “label” column to the input DataFrame to tag activities that belong to users that eventually churned (1 if user churned 0 otherwise).

If that sounds suspiciously redundant to you I agree. But let’s table that issue for now; we will revisit it once we are satisfied with our new version of the code.

Refactor the codeThe main problem of the code is the use of Python lists to achieve the required results. Those lists are created by collecting the DataFrames onto the Spark driver where the for loops will be processed, making this code not scalable: above a certain number of users the driver memory might become overwhelmed and the program will crash.

Also this choice prevents the code from leveraging all the optimizations that come with DataFrames operations.

Then the code uses plain Pyspark UDFs for which you incur a performance penalty because of the need to:

• Deserialize the Spark DataFrame to its Java representation
• Transfer the resulting Java object to the Python process where the UDF will be executed
• Serialize back the output of the function to Spark format

Beware of the cost of Pyspark UDFsThere are ways to mitigate those issues by using PyArrow and vector UDFs when you really need to use them, but this is not one of those times.

First, the function creates a “Churn” column, which I guess is for convenience purposes. A user is identified as “churned” if they have been to the “Cancellation Confirmation” page.

This is achieved with a withColumn call and a UDF.

*#Define a udf for cancelled* canceled = udf(lambda x: 1 if x == 'Cancellation Confirmation' else 0) *#define a new column 'churn' where 1 indicates cancellation of subscription, 0 otherwise* df = df.withColumn('Churn', canceled(df.page))There is no need for a UDF in that case, those lines of code can be replaced by a simple column expression like so: *#define a new column 'churn' where 1 indicates cancellation of subscription, 0 otherwise* df = df.withColumn('Churn', (df.page == 'Cancellation Confirmation').cast('integer').cast('string')) I believe the correct type for that new column would be boolean, but for non-regression purposes I had to cast it to a string of 0 or 1.

Then the author proceeds to create two lists: one for the users that churned and one for the users that stayed. Since my goal is to avoid those lists, I am going to create the corresponding DataFrames instead:

all\_users = df.select(df.userId).distinct().where(df.userId != '') churned\_users = df.where(df.Churn == '1').select(df.userId).distinct().where(df.userId != '') stayed\_users = all\_users.subtract(churned\_users) First I create a DataFrame of all the non-empty users, then the DataFrame of users that churned, and define the users that stayed as the difference between the two.

The author uses the awkwardly created lists together with UDFs to create the all_cancelled and stayed DataFrames. Here is the code for the first one:

*#List of cancelled* list\_cancelled = cancelled\_df.select('userId').distinct().collect()*#list of cancelled users* *#Put in a list format* gb = []*#temporary variable to store lists* for row in list\_cancelled: gb.append(row[0]) canc\_list = [x for x in gb if x != '']*#remove the invalid users**…* all\_cancelled = df.select("*").where(col('userId').isin(canc\_list)) I realize now that the “Put in list format” loop is probably unnecessary.

To create the same DataFrame I just do the following:

all\_cancelled = df.join(churned\_users,'userId') The same technique is applied to create the stayed DataFrame:

stayed = df.join(stayed\_users,'userId') Last the author adds the “label” column to the main DataFrame by using a UDF:

*#Redefine a udf for churn* churned = udf(lambda x: 0 if x in stayed\_list else 1, IntegerType()) *#Creat new column which will be our label column to track all users that eventually cancelled their subscription* df = df.withColumn('label', churned(col('userId'))) Instead I just use a union:

df\_label = all\_cancelled.withColumn('label',lit(1)).union(stayed.withColumn('label',lit(0))) That triggered a regression because I did not include the null users. I wonder what use could be made of records with null users for training a model to predict churn from users’ behavior, but for non-regression purposes I added those too:

empty\_users = df.where(df.userId.isNull())… *#Add empty users for non regression purposes* df\_label = df\_label.union(empty\_users.withColumn('label',lit(1))) Last, I also had to reorder the columns of my DataFrames for my simple non-regression tests to be successful:

# Sort the columns

columns = ['artist','auth','firstName','gender','itemInSession','lastName','length','level','location','method','page','registration','sessionId','song','status','ts','userAgent','userId','Churn','label'] df\_label\_sorted = df\_label.select(columns) columns = ['artist','auth','firstName','gender','itemInSession','lastName','length','level','location','method','page','registration','sessionId','song','status','ts','userAgent','userId','Churn'] all\_cancelled\_sorted = all\_cancelled.select(columns) stayed\_sorted = stayed.select(columns)

This is my full version of the function:

**from** pyspark.sql.functions **import** lit**def** **prepare\_data**(df): ''' Function to prepare the given dataframe and divide into groups of churn and non churn users while returning the original DataFrame with a new label column into a spark dataframe. Args: df- the original dataframe Returns: df - dataframe of the dataset with new column of churn added stayed - dataframe of the non -churn user's activities only. all\_cancelled - dataframe of the churn user's activities only. ''' *#define a new column 'churn' where 1 indicates cancellation of subscription, 0 otherwise* df = df.withColumn('Churn', (df.page == 'Cancellation Confirmation').cast('integer').cast('string')) all\_users = df.select(df.userId).distinct().where(df.userId != '') churned\_users = df.where(df.Churn == '1').select(df.userId).distinct().where(df.userId != '') stayed\_users = all\_users.subtract(churned\_users) empty\_users = df.where(df.userId.isNull()) *#Store both canceled and staying users in new DataFrames containing all actions they undertook* all\_cancelled = df.join(churned\_users,'userId') stayed = df.join(stayed\_users,'userId') df\_label = all\_cancelled.withColumn('label',lit(1)).union(stayed.withColumn('label',lit(0))) *#Add empty users for non regression purposes* df\_label = df\_label.union(empty\_users.withColumn('label',lit(1))) *# Sort the columns* columns = ['artist','auth','firstName','gender','itemInSession','lastName','length','level','location','method','page','registration','sessionId','song','status','ts','userAgent','userId','Churn','label'] df\_label\_sorted = df\_label.select(columns) columns = ['artist','auth','firstName','gender','itemInSession','lastName','length','level','location','method','page','registration','sessionId','song','status','ts','userAgent','userId','Churn'] all\_cancelled\_sorted = all\_cancelled.select(columns) stayed\_sorted = stayed.select(columns) *#Total number of users who canceled* print(f"The number of churned users is: {churned\_users.count()}") *#Total number of users who did not cancel* print(f"The number of staying users is: {stayed\_users.count()}") **return** df\_label\_sorted, stayed\_sorted, all\_cancelled\_sorted Non regression and performanceI was able to verify that I had not introduced any regression in my version of the function on my desktop with Spark 3.3.

In order to get meaningful performance measurements I needed to use the full 12G JSON dataset. Otherwise, with small data, most of the time is spent on overhead and results vary wildly.

So I switched to our CML data service using Spark 3.2 and adapted the code accordingly.

CML uses Spark on Kubernetes and the default is dynamic allocation of executors. I had to disable that to get a stable environment and thus, meaningful measures:

import time, datetimefrom prepareData import prepare\_datafrom prepareData\_baseline import prepare\_data\_baselinefrom prepareData\_improved import prepare\_data\_improvedimport cml.data\_v1 as cmldatafrom env import S3\_ROOT, S3\_HOME, CONNECTION\_NAMEconn = cmldata.get\_connection(CONNECTION\_NAME)spark = ( SparkSession.builder.appName(conn.app\_name) .config("spark.sql.hive.hwc.execution.mode", "spark") .config("spark.dynamicAllocation.enabled","false") .config("spark.executor.instances", 3) .config("spark.executor.memory","32g") .config("spark.executor.cores",4) .config("spark.yarn.access.hadoopFileSystems", conn.hive\_external\_dir) .getOrCreate() )spark.sparkContext.setLogLevel("ERROR")spark.conf.set('spark.sql.adaptive.enabled','true')print(f"AQE enabled: {spark.conf.get('spark.sql.adaptive.enabled')}") That got me the desired result:

I then found out that the full 12G data set contained a corrupt record that I had to deal with, and while I was at it I converted the file to Parquet format to save me some time:

Convert early to compressed columnar formats (Parquet, ORC)I created a function that performs the tests to avoid repetitive code in which I also added calls to setJobGroup and setJobDescription to improve the readability of the Spark UI:

**def** **measureDataPreparation**(df,f,versionName): spark.sparkContext.setJobGroup(versionName,"") *# Start timer: begin processing* process\_time\_start = time.perf\_counter() df, stayed, all\_cancelled = f(df) spark.sparkContext.setJobDescription("Write /data/df") df.write.mode("overwrite").json(S3\_HOME + '/data/df') spark.sparkContext.setJobDescription("Write /data/stayed") stayed.write.mode("overwrite").json(S3\_HOME + '/data/stayed') spark.sparkContext.setJobDescription("Write /data/all\_cancelled") all\_cancelled.write.mode("overwrite").json(S3\_HOME + '/data/all\_cancelled') *# Stop timer: end processing* process\_time\_end = time.perf\_counter() *# Elapsed time for processing* process\_time = process\_time\_end - process\_time\_start totalTime = datetime.timedelta(seconds = process\_time) print(f"Preparing data with the {versionName} took {totalTime}") Use setJobGroup and setJobDescription to improve readability of the Spark UIAnd this is how the Spark UI looks as a result:

Since I had established that I had not introduced any regression, I also removed the regression tests.

Here is the the relevant part of the session’s output:

measureDataPreparation(df,prepare\_data\_baseline,"baseline version")The number of churned users is: 4982The number of staying users is: 17282Preparing data with the baseline version took 0:09:11.799036measureDataPreparation(df,prepare\_data,"no regression version") The number of churned users is: 4982The number of staying users is: 17282Preparing data with the no regression version took 0:01:48.224514 Great success! The new version is more than four times more efficient!

Further improvementsSince I no longer need to test for non regression I can remove the sorting of the columns.

I can also remove the code that prints the counts of the churned and stayed users. This code does not belong in a function that very likely will run unattended in a data pipeline.

It triggers distributed execution to compute results that nobody will see. It should be left to the code that calls the function to log that kind of information or not.

This is also an instance of breaking the following rule:

Remove code that helped debugging with count(), take() or show() in productionI checked the rest of the initial code, and after exhaustive data exploration and right before splitting the data set for training purposes, the author does remove the rows with null users. There is no point in carrying around this extra baggage all this time. In fact this breaks another rule of big data processing:

Filter earlyFinally, I removed the casting of the “Churn” column and left it as a boolean. I also checked that it was not used outside of this function and renamed it “churn” because I hated that uppercase “C” with all the passion of a thousand white hot blazing suns.

This is the final version of the code:

from pyspark.sql.functions import litdef prepare\_data\_improved(df): ''' Function to prepare the given DataFrame and divide into groups of churn and non churn users while returning the original DataFrame with a new label column into a Spark DataFrame. Args: df- the original DataFrame Returns: df - DataFrame of the dataset with new column of churn added stayed - DataFrame of the non -churn user's activities only. all\_cancelled - DataFrame of the churn user's activities only. ''' *#define a new column 'churn' where 1 indicates cancellation of subscription, 0 otherwise* df = df.where(df.userId != '').withColumn('churn', (df.page == 'Cancellation Confirmation')) all\_users = df.select(df.userId).distinct() churned\_users = df.where(df.churn).select(df.userId).distinct() stayed\_users = all\_users.subtract(churned\_users) *#Store both canceled and staying users in new DataFrames containing all actions they undertook* all\_cancelled = df.join(churned\_users,'userId') stayed = df.join(stayed\_users,'userId') df\_label = all\_cancelled.withColumn('label',lit(1)).union(stayed.withColumn('label',lit(0))) return df\_label, stayed, all\_cancelled ConclusionNow that I have achieved non regression using DataFrame exclusively, and that I also have an improved version, I should be able to measure the benefits of using the Spark cache and of the Adaptive Query Execution engine.

Here are the full results:

In this limited experiment, the number one factor that influences the performance of the execution is the refactoring of the Spark code to remove the distributed processing anti-patterns.

Caching the data, improving the code further, or using AQE all bring marginal improvements compared to the elimination of the technical debt.

The return on investment of training is always a thorny issue because of the difficulty to conveniently measure it in a spreadsheet but, with this experiment, I hope I have shown that the lack of skills should be a major concern for any organization running Spark workloads.

If you’d like to get hands-on experience with Spark 3.2, as well as other tools and techniques for making your Spark jobs run at peak performance, sign up for Cloudera’s Apache Spark Performance Tuning course.

If you need an introduction to AQE kindly refer to my previous blog post.

The post Spark Technical Debt Deep Dive appeared first on Cloudera Blog.

View Details

Financial inclusion, defined as the availability and accessibility of financial services to underserved communities, is a critical issue facing the banking industry today. According to the World Bank, 1.7 billion adults around the world do not have access to formal financial services, meaning that they cannot open a bank account or access credit, insurance, or other financial products. This lack of financial inclusion can lead to a range of negative outcomes, including difficulty saving for emergencies, reduced ability to invest in education or business opportunities, and increased vulnerability to financial shocks.

Access to financial services and credit can help lift individuals and entire underserved communities out of poverty. By providing access to financial services, banks can help underserved communities save for the future, invest in education and business opportunities, and build financial resilience. This, in turn, can help drive economic development and improve the overall quality of life in underserved communities.

Banks are also working to help underserved communities become more financially literate and responsible. This includes providing financial education and training programs via their mobile apps, as well as designing financial products and services that are easy to understand and use. By helping underserved communities to better understand their financial options and make informed decisions about how to manage their money, banks can help to increase financial inclusion and build stronger, more resilient communities.

To advance these efforts, banks are increasingly turning to data, data analytics, and machine learning (ML) and artificial intelligence (AI) to better understand and serve the financial needs of underserved communities. One way that banks are using these technologies is by leveraging non-traditional sources of data, such as payment history for things like utility bills, to better determine credit worthiness for individuals who may not have a traditional FICO or credit score. This can help banks to more accurately assess the creditworthiness of underserved individuals, enabling them to offer financial products and services that meet their needs, while at the same time managing their own risk.

Cloudera Data Platform (CDP) has also been instrumental in helping banks solve the issue of financial inclusion in underserved communities. CDP is a comprehensive hybrid data platform that enables banks to store, process, and analyze large volumes of data, including both structured and unstructured data, in a single, integrated platform. CDP allows banks to quickly and easily access the data they need to better understand the financial needs and behaviors of underserved communities, and design financial products and services that meet their unique needs. An example of this is PT Bank Rakyat Indonesia, who are leveraging CDP to help them focus on micro financial services initiatives.

Mobile banking apps are critical for banks looking to advance financial inclusion in underserved communities. With the widespread adoption of smartphones, mobile banking apps provide a convenient and accessible way for underserved individuals to access financial services, even in areas where traditional brick-and-mortar banks may not be present.

Finally, it’s worth noting that the cost of acquiring new clients has never been lower, making financial inclusion efforts more profitable for banks. With the increasing use of data analytics and ML/AI, banks are able to more effectively target underserved communities and design financial products and services that meet their unique needs. These technologies not only help to increase financial inclusion, but it can also drive revenue and profitability for banks as they expand their customer base.

Learn more about how Cloudera is helping the Financial Services industry here.

The post How Banks are Using Technologies to Help Underserved Communities appeared first on Cloudera Blog.

View Details

It’s no secret that women are vastly underrepresented in the tech sector. While women make up 57% of the professional workforce, they hold only 26% of professional computing jobs. That number is significantly lower for women of color: Black women make up 3% of the computing workforce, and Latina women only 1%.

Underrepresentation in tech is a complex, systemic problem, and no one company or organization can solve these issues alone. Enter: the Reboot Representation Tech Coalition.

Starting in April 2022, Cloudera joined 20 other organizations to help close the gender gap in the tech sector as a member of the Reboot Representation Tech Coalition. Together, Cloudera and organizations like Amazon, Intel, and Microsoft have pledged more than $26 million to support Black, Latina, and Native American (BLNA) women in computing.

The coalition was founded in 2018 following a first-of-its-kind report to collect data directly from tech companies to understand how they approach philanthropic and corporate social responsibility (CSR) initiatives focused on closing the gender gap in tech.

“The fourth industrial revolution will be powered by people, and technology companies are at the forefront of ensuring there are opportunities and access for all.” said Amy Nelson, Cloudera’s Chief Human Resources Officer. “Cloudera is proud to be a part of a group of committed, innovative companies who are working collectively to invest in change at a time when working together is the best avenue we have to impactful and lasting change.”

Reboot Representation was born from the notion that increasing the success of BLNA women in tech lies in targeted philanthropic investments in often overlooked programs and institutions that make education and careers in computing more equitable for women of color. Specifically, higher education and broader systemic change.

“In 2018, the Rebooting Representation report exposed the diminutive investments supporting Black, Latina, and Native American women and girls interested in tech. The severity of the issue made it clear that only a collective effort could create long-term change,” said Dwana Franklin-Davis, CEO of Reboot Representation. “Philanthropic dollars are powerful, and they can be even more powerful when pooled and spent intentionally, which is why the Reboot Representation Tech Coalition exists.”

The coalition’s immediate goal is to double the number of Black, Latina, and Native American women receiving computing degrees by 2025. Without this targeted investment, the number of BLNA women receiving these degrees wouldn’t double until 2052. In 2018, only 19% of computing degree recipients were women.

Degrees and education are important, but they’re just the beginning. The coalition’s long-term goal is to grow and facilitate cultures that lead to continued success for BLNA women and for the tech industry as a whole.

“Technology spaces have for decades been ones where women, especially Black, Latina, and Native American women, have felt—or been—unwelcome,” said Franklin-Davis. “Reboot Representation works across sectors to collaborate with leaders and changemakers in tech, higher education, and nonprofits to facilitate and grow a culture of inclusion, from classroom to internship to career.”

Not only do these efforts directly benefit women of color by setting them up for jobs and opportunities, but it’s also good business. According to the 2018 Rebooting Representation report, “Teams with greater gender diversity have certain dynamics that allow for more radical innovation, which is especially critical for tech companies.”

Additionally, companies with better diversity are more creative, innovative, and more profitable, according to Reboot Representation.

“Technology empowers, innovates, and adapts,” remarked Amy. “Moreover, technology is developed and deployed best when informed and reflective of our society. Industry leaders like Cloudera are demonstrating that companies can operate under these values by collaborating to invest in access and equality.”

The post Cloudera Collaborates with Reboot Representation to Boost Black, Latina and Native American Women in Tech appeared first on Cloudera Blog.

View Details

Cloudera Partners with Per Scholas to Educate a Diverse Technology Workforce Working in the technology sector comes with plenty of perks, from higher wages to opportunities for upward growth. But breaking into the tech industry isn’t always straightforward. Specialized jobs that come with higher pay and better benefits are typically only available for those who can afford for the training and skills required to make it in the industry.

As part of our commitment to making data and analytics easy and accessible for everyone, Cloudera has partnered with Per Scholas, a US–based nonprofit organization, to provide a no-cost software engineering training course for at least 20 individuals from diverse backgrounds in Charlotte, North Carolina.

Per Scholas, a national organization working to increase access and create opportunities for individuals who aspire to work in tech, equips motivated adults with the skills they need to successfully start careers in the industry. Per Scholas has been working to advance economic mobility for more than 27 years, as well as helping individuals traditionally underrepresented in technology gain high-growth careers in the industry.

“Per Scholas is in the business of creating opportunity,” said Mike Terrell, Per Scholas North Carolina managing director. “We are passionate about opening doors to high-growth careers in technology and supporting our learners in their journey from course work to securing that transformational career.”

“Partnering with Per Scholas in the United States market is in line with our commitment to accelerate reskilling solutions within the data Industry,” shared Abhas Ricky, chief strategy officer and executive sponsor of Cloudera’s corporate social responsibility program. “The organization not only provides a high-quality, industry-informed curriculum to highly motivated talent, but also ensures its learners receive continued education and wrap-around services that make their graduates top candidates in the workplace.”

Students enrolled in Per Scholas’ training receive rigorous technical instruction, professional development coaching, and connections to high-demand employment opportunities after graduation.

In addition to technical training, Cloudera’s funding also supports weekly professional development for each learner, helping them cultivate essential business skills like resume writing, time management, problem-solving, and effective communication. Learners also receive financial capability coaching and services provided by licensed social workers—ensuring that each learner is equipped with the skills necessary to enter into the tech sector and a holistic knowledge of ways to build economic and professional sustainability.

“Per Scholas gave me a lot of confidence. I know that I would not have qualified for the role I have now, from a technical perspective, without Per Scholas,” reflects Per Scholas North Carolina graduate Phillip Kinney.

In our 2022 cohort, learners were made up of 47% females, 80% people of color and achieved an 83% graduation rate in December 2022 (3% more than our target of 80% graduating). Alongside Cloudera, Per Scholas are now supporting December graduates into employment, with a target of 80% of graduates working in the technology sector within one year.

Per Scholas partners with a vast network of organizations to help their graduates gain employment in businesses of all sizes, from start-ups to Fortune 500 companies. Since their inception, they’ve helped more than 17,000 rebootpeople launch tech careers. With campuses in 19 cities, Per Scholas has now trained nearly 17,000 individuals from underrepresented backgrounds to launch vibrant new tech careers.

According to Per Scholas, historically 85% of learners graduate from the program, 80% of graduates obtain one or more industry-recognized IT certifications, and 80% of graduates gain employment with starting wages averaging more than four times their average pre-training income ($21/hour, or more than $43,000/year on average).

“We are so grateful to Cloudera for their support of our Software Engineering cohort,” said Mike Terrell, Per Scholas North Carolina managing director. “With their investment, we can continue our mission to train diverse learners in the tech skills they need to succeed.”

For more information, visit Cloudera’s corporate social responsibility website.

The post Per Scholas and Cloudera Innovate to Solve for the Skills Gap in Data appeared first on Cloudera Blog.

View Details

Have you ever considered how much data a single person generates in a day? Every web document, scanned document, email, social media post, and media download? One estimate states that “on average, people will produce 463 exabytes of data per day by 2025.”

Now consider that the federal government has approximately 2.8 million civilian employees and the department of defense has another 2 million active duty, Guardsmen, and Reservists. Add that to the nearly 19 million employees in state and local agencies and you have a combined agency population greater than the number of residents in Florida, the third-largest state. Each of these employees generates volumes of data on a daily basis. And this doesn’t even touch on the data generated by citizen services interfaces, machine or device-generated data such as video feeds, sensors, and communications data. The list could go on and on.

With these massive volumes of data, it’s common for agencies and enterprises to determine the data that is readily accessible and essential to mission success and prioritize for analytics. They essentially shine a light on the data that is most available and perceived as relevant to decision-making outcomes, while an unquantifiable amount of data stays in the “dark,” unused or unknown.

What is dark data?Gartner defines dark data as “The information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes (for example, analytics, business relationships and direct monetizing).” Some examples include employee records, internal and external communications, photo, video, and audio files, IoT sensor data, and streamed data.

By 2025, it’s estimated that the amount of data created, consumed, and stored will reach 180 zettabytes, with up to 90% of that unstructured and nearly all of it unused for decision making. This dark data resides everywhere in the enterprise, siloed in multiple data repositories, from laptops and mobile devices to data lakes and applications.

The purpose of this blog isn’t to emphasize the cyber risk of dark data but to spotlight its implications. Dark data is inherently considered unusable or not prioritized for analytics, which means it may be stored in less secure repositories, forgotten about or unmanaged, and left vulnerable to a data breach.

In the past, the accessibility and processing of this data was time and cost prohibitive to analytics at scale, but in the new age of hybrid data platforms finding, understanding, and utilizing dark data is possible and has huge implications for government applications, especially as AI and machine learning take hold within the enterprise.

• Agency Operations: Much of dark data is produced in day-to-day operations and has the potential to provide deep insight into how to improve operational efficiency for the public sector workforce while reducing costs for citizen services.
• IoT Insights: While data from IoT devices and sensors are regularly utilized for real-time alarms and control, understanding and analyzing IoT data opens the door to systems prediction use cases such as condition-based monitoring for aircraft maintenance, or optimization of emissions and water quality controls to avert future environmental catastrophes.
• Citizen 360: Having a complete picture of the citizen and their interactions with government agencies has wide-ranging positive outcomes from reducing fraud, waste, and abuse, to rooting out bias and providing more accurate and timely services.

Making dark data actionableOrganizations that succeed in optimizing the process of discovering, classifying, and leveraging their dark data to feed AI and machine learning models are well poised to reduce risk and unlock valuable insights that drive operational efficiencies. Enabling a solution to move previously untapped data to an analytics platform provides a broader and far more accurate view of data across the entire enterprise.

According to a Forbes analyst, “If IT can provide a unified data architecture that serves as an integrated layer connecting data endpoints and processes, it can make mission-critical data more discoverable, pervasive and reusable across all environments of an organization, including hybrid and multi-cloud environments.”

CDOs need to get a handle on their dark data now, as their organizations continue to gather increasing amounts of information every day. To help federal, state, and local government agencies minimize risk and leverage these “dark data” as a strategic asset, Cloudera Government Solutions provides universal data distribution and data in motion capabilities to discover, classify, and move all (including dark) data anywhere, to any application.

Learn how Cloudera Professional Services provides the building blocks to optimize your data value and deploy all modern data architectures.

The post How the Public Sector Can Maximize the Value of Dark Data appeared first on Cloudera Blog.

View Details

United Bank Limited (UBL), a Pakistani banking and financial services leader, serves over 11 million customers nationwide and operates 1,338 branches and 1,445 ATMs, along with its branchless banking proposition (combination ATM and online banking).

In 2022, UBL was awarded Best Bank for Digital Solutions by Asiamoney and Market Leader of Digital Banking in Pakistan by Euromoney, a testament to its track record as the best in digital banking.

UBL needed a superior data platform to handle the increasing volume and improve the businessWith UBL’s growing success, the bank needed to accommodate its growing volume of data. To this end, UBL embarked on a data analytics project that would achieve its goals for an improved data environment. First, it needed customized engagement models that would improve the customer experience. Next, it needed to enhance the company’s customer-centric approach for a needs-based alignment of products and services. And finally, UBL wanted to be able to strengthen its organizational decision-making capabilities. All of this needed to be achieved within the company’s operational efficiency and excellence standards.

UBL selected Cloudera for its data platform and Blutech Consulting—Pakistan’s leading data analytics company and the preferred partner of Cloudera—for the implementation. Both Cloudera and Blutech enabled UBL to achieve its vision around data analytics.

Overall, the implementation was seamless and used a single agile approach. Together with Cloudera and Blutech, UBL achieved this major milestone in under six weeks. This was made possible thanks to the right leadership and a sound technical team on the ground that understood the complexities around technology integration.

Mr. Faisal Anwar, CIO of UBL, was the champion of this end-to-end flagship program, which set the benchmark for the financial sector in Pakistan.

“The implementation has been fast and smooth, and our teams have already begun to witness the benefits of Cloudera’s data platform. We are positive that our continuing partnership with Cloudera and Blutech Consulting will be foundational to our customer-centric approach, considerably improving our customer responsiveness,” he said.

On a successful delivery, Mr. Kashif Jadoon, CEO of Blutech Consulting, said, “For UBL we aimed to achieve an innovative, unified platform that is built to meet UBL’s future needs and cater to the analytics demands of tomorrow. We work as a partner to our customer and Cloudera’s CDP is the only modern solution that enables us to deploy fast with minimal impact to operations while addressing the system, hosting, integration, and security.”

A complete data analytics solution returning key benefitsUBL has already realized some key benefits from using the Cloudera Data Platform, such as:

Machine learning and AI has been infused into key decision-making processes. This includes decisions around the optimal amount of cash in every bank’s ATM, or proactively classifying every digital transaction as fraud/non-fraud, which are now driven through data and AI.

Analytics for everyone. UBL has initiated Analytics for Everyone, a self-service business analytics capability for the bank’s various business units. Besides reducing its dependency on IT for providing insights, it also enabled the business units to make data-driven decisions.

Leadership visibility. The platform has created a consolidated view of the business for UBL’s management and top leadership. Executive dashboards have been built for senior leaders including the president and CEO and chief digital officer, to provide them with a holistic overview and the key data they require.

Single source of truth. A comprehensively built data foundation is used as a data hub built on CDP, which is treated as a single source of truth for downstream business applications. These applications connect to the data hub instead of connecting to individual source systems and having to navigate a complex series of data pipelines.

Mr. Kashif Riaz, head of data and AI at UBL, shared his thoughts on this project. “To unlock the true potential of our data, we need the best technology solution, which can enable us to provide greater experience and insights to our customers. With this major upgrade from HDP to CDP, UBL has not only unlocked new and enhanced functionality but will also be able to fuel its data and digital transformation journey moving forward.”

To find out more about how Cloudera empowers financial services leaders to realize the endless possibilities of data, read our latest FSI Guide, A world without limits.

The post United Bank Limited optimizes its data analytics with the Cloudera Data Platform (CDP) appeared first on Cloudera Blog.

View Details

In the past year, businesses who doubled down on digital transformation during the pandemic saw their efforts coming to fruition in the form of cost savings and more streamlined data management. Faced with even more pressure to remain resilient and agile amid looming global economic threats, Asia-Pacific (APAC) region businesses are looking to further mobilize emerging technologies such as artificial intelligence (AI) and machine learning that will optimize operational efficiencies and cost savings.

As more industries mature digitally and widely adopt AI and machine learning technologies, 2023 will be a pivotal year for organizations looking to deploy emerging tech solutions company-wide to fulfill business objectives. Here are three key trends that will likely dominate the priorities of APAC’s business leaders in the coming year.

1- Treating data as a strategic business asset Recent years have seen organizations generating unprecedented volumes of data as a by-product of their digitalization activities and increasing digital customer touch points. This is especially so in industries like telecom, retail, healthcare, manufacturing, insurance, and financial services. And with the anticipated deployment of 5G networks across the region, this volume of data will increase significantly.

In APAC, we have observed that organizations are doing (or aiming to do) more with their data, and reduce the time to value. Data contains valuable insights for critical business decision-making, and the most innovative and successful organizations recognize data as a strategic resource that demands its own strategy. How this strategy looks depends on the organization’s unique business needs as one affects the other. There is no one-size-fits-all approach; the strategy must continue evolving with the business’s priorities.

What is certain is that having an enterprise data strategy aligned to the organization’s cloud strategy and business priorities will help the organization drive greater business value by improving operational efficiencies and unlocking new revenue streams. According to findings from Cloudera’s Enterprise Data Maturity research report, organizations across the globe with data strategies in place for more than a year see an average profit growth of 5.97%.

With the right tools in place, distilling actionable insights from data to achieve business objectives or unlock new revenue streams is easily achievable for organizations of all sizes across industries, especially with the availability of self-serve functionalities that do not require specialized ops or cloud expertise.

2- Operationalizing adaptive AI systems for quicker business decision-makingWith the increase in demand for real-time data processing, streaming, and sharing, which power transformation into data-driven organizations, we anticipate more businesses investing in building adaptive AI systems that can ingest large amounts of data at frequent intervals and adapt to changes and variances quickly.

What will determine the winners from the laggards will hinge on the speed at which predictive analytics can be executed, and the cost-benefit ratio related to these algorithmic paradigms. An organization’s ability to create trust with usable and explainable AI for faster and more flexible decisions will separate the leaders from the pack.

We foresee organizations pivoting focus beyond the algorithm to things like business-ready predictive dashboards, visualizations, and applications that simplify the use of AI systems to reach conclusions. These will help business leaders quickly understand the impact to their business and act with confidence.

We have been working with APAC organizations to operationalize data analytics and AI solutions to unlock data-driven decision-making and operational efficiency, with them quickly seeing distinct business benefits. For example, Singapore’s United Overseas Bank (UOB) used machine learning to operationalize analytics and provide insights to users across the bank. Through the Cloudera Data Platform, UOB has launched a deposit analytics solution to ensure it can build stable deposits with optimal pricing, and provide consistent and accurate views of deposits. The results are higher revenues, lower risks, and increased productivity for the bank.

3- Continued move to the public cloud and hybrid cloud, optimizing deploymentsPublic cloud spend and workload volumes continue to accelerate for organizations of all sizes as cloud-first policies and cloud migration remain top of the agenda. However, a significant amount of this spend is wasted as organizations struggle to optimize costs effectively.

According to Flexera’s 2022 State of the Cloud Report, respondents self-estimated that their organizations wasted 32% of cloud spend in 2021, up from 30% the previous year. As cost optimization remains the top cloud initiative for organizations for the sixth year running, we will likely see organizations opt for more cost-effective strategies to deliver results quickly and efficiently, including:

• Migrating more workloads to the cloud to free up resources while driving agility
• Implementing data and analytics solutions that can manage the end-to-end data life cycle—from ingesting data from multiple sources to storing, processing, serving, analyzing, and modeling it to drive actionable insights
• Repatriating some machine learning workflows back on premise, where complex processes are more cost effective, to optimize cloud spend for compliance, governance, and security

This is where leveraging modern data architectures like data lakehouse, data fabric, and data mesh is essential to driving business efficiencies across diverse operations. In addition to managing data on premises and in public or private clouds, these modern data architectures are also intrinsically designed to handle complexities such as security and governance-related issues. They also address the concerns of IT teams in allowing access to organizational data.

Organizations can consider moving to hybrid data platforms to better manage the entire life cycle of data analytics and machine learning. The platforms must have features of openness and interoperability that allow ease of sharing and enable self-serve functionality, such as the Cloudera Data Platform (CDP), which has a built-in shared data experience (SDX) feature. These features provide businesses with a common metadata, security, and governance model across all their data.

Overall, organizations must take the time to evaluate their overarching business objectives before embracing cloud, edge, and data capabilities. It is crucial to determine the approach and strategies that best fit the unique needs of their business, and determine where these capabilities can benefit the entire organization and not just to solve specific problems.

Find out more about CDP for modern data architectures here.

The post 2023 Predictions: Data Trends That Will Dominate Business Agenda in APAC appeared first on Cloudera Blog.

View Details

Holiday season is a time to reflect on your year and support those less fortunate than yourself. Clouderans made a global impact by running a number of donation activities and local giving events to celebrate the season of giving.

November 29: Giving Tuesday—Global

Giving Tuesday, a day dedicated to donations and giving back, is the Tuesday after Thanksgiving in the US. As an organization, Cloudera supported the holiday with a 200% matching campaign, meaning for every $1 a Clouderan donated, Cloudera donated $2.

Overall Clouderans donated a total of $81,467 to 204 chosen nonprofits around the world.

November 29: Donating food to OVG on behalf of event attendees—Brasilia, Brazil

Instead of giving out swag at a client event, the Cloudera Brazil team partnered with a charity to donate one portion of food per attendee to the event. This resulted in over 200 portions of food being donated to in-need people in the local area, as well as a very happy Cloudera customer.

Thanks to the Brazil sales and marketing team for coming up with this idea, and special thanks to Thatiane Freire for making it happen!

December 5: Shoebox drive—Budapest, Hungary

The Cloudera Budapest team worked with two nonprofits to pack, decorate, and donate shoeboxes filled with gifts for in-need children. The team collectively donated about 65 shoeboxes, which will be distributed throughout the holidays.

December 5: Blanket drive for dogs—Budapest, Hungary

Working with a local animal shelter, the Cloudera Budapest team asked local employees to donate unused blankets, towels, and sheets. With colder weather and higher costs, many more animals need shelter this year, making the donations especially important. This event also gave Clouderans a chance to recycle unused items for a second life, therefore having an additional sustainable impact.

Thanks to Cloudera’s Budapest office for these donations, and special thanks to our organizers: Dániel Omaisz-Takács, Fanni Fulmer, Rita Szucs, and Glória Benkő.

December 7: Disney trivia quiz for Make-A-Wish Foundation—Global

This global virtual event provided Clouderans a chance to have fun while giving back. Clouderans were invited to put their Disney knowledge to the test in support of the Make-A-Wish Foundation. From a trip to Disney or meeting their favorite superhero, Make-A-Wish Foundation grants wishes to critically ill young people.

Thanks to everyone that took part, and special thanks to Stephanie Han for organizing!

December 12: Volunteering at Second Harvest Food Bank—Santa Clara, CA

Members of the Cloudera engineering team in the Bay Area took part in volunteering with a local food bank, Second Harvest. During the holiday season many food banks are looking for additional people to help volunteer and donate.

Thanks to those in the Santa Clara office for taking part, and special thanks to Stella Tarolli for organizing!

December: Giving Tree for KidsOut and The Society of St. Vincent De Paul—London, UK and Cork, Ireland

KidsOut supports in-need children across the UK to build confidence and happiness at their most vulnerable. Every year, KidsOut run a Giving Tree, where people are asked to buy certain gifts and donate. These donations are distributed to children in refuge over the holiday period.

Thanks to the UK and Ireland teams for taking part, and special thanks to Jade Kelly in the UK, and Suzanne Livesey and Aisling Holland in Ireland, for organizing!

The post Clouderans Celebrate the Holiday Season by Giving Back appeared first on Cloudera Blog.

View Details

Across the energy supply chain from generation to consumer, we can see that the trend toward investing in renewable energy has picked up pace as demand has grown for energy companies to actively pursue investments in energies with little or no environmental impact in the quest for decarbonisation. McKinsey estimates that by 2035, 50% of energy will be wind and solar.

The move toward renewable energy has a distinct and significant impact on energy generation and distribution that needs to be carefully managed. Efficient use of data will therefore be critical to improving the competitiveness and productivity of assets, both traditional and renewable generation.

Data efficiency in renewablesFrom the perspective of power generation from renewable sources, there are several challenges that companies need to tackle. The variable nature that fuels the renewable generation makes production less predictable. In that sense, access to the auction is more conservative and can lead to inefficiencies in the use of assets, as some of the energy produced may be discarded if it is not saleable in the intraday market.

Effective use of data can have a direct impact on the cash flow of wind and solar generation companies in areas such as real-time decision making. With the right insights, energy production from renewable assets can be optimized and better predict the future of supply and demand. In addition, using data well can allow better decisions to be made, such as the possibility of bypassing the day-ahead market and going directly to the intraday market and having a better return per watt generated.

Organizations working in traditional energy generation will have to adjust costs by improving the efficiency of these plants. In this respect, several studies project that a proper use of advanced analytics implies savings of between 5% and 7.5%. This is due to an improvement in plant uptime, supported by predictive maintenance applications, coupled with optimization of the fuel consumption that powers these plants, as well as fine-grained performance monitoring that eliminates over-production.

This need to improve production costs in the more traditional plants is becoming increasingly important if we add that more and more individuals are injecting energy into the system from solar panels. These actors are at both ends of the chain, as they produce and consume at the same time. This scenario suggests that in the not too distant future, there will be a large “long-tail” of producers that will have to be taken into account for any production forecasting model. For this reason, the need to have a data platform that allows for a fine-tuned analysis and prediction of supply and demand becomes a necessary tool to operate the business in a more efficient and profitable way.

Intelligent network managementAll of this introduces a new issue: grid management. The business of energy transmission and distribution has undergone little change in recent years, but can be stressed by a number of factors. Commonly it is the incorporation of a large number of small producers distributed throughout the territory, and the increase in the rise of electric vehicles that are causing this strain.

The latter will drive changes in consumption habits such as charging electric vehicles at home and the injection of energy from solar panels on roofs, which will cause midday energy peaks and create more volatile daily demand patterns. To cope with these changes in demand and avoid overloads distribution companies will have to invest in optimizing the grid, which may put pressure on profitability and cash flows.

Investment is unavoidable, so the question becomes about smart investment. How do companies future-proof their assets as well as prepare themselves for the future?

That’s where data comes in. McKinsey estimates that the use of data-driven technologies can drive operating and maintenance cost savings of more than 12%. For example, predictive maintenance, based on machine learning, will enable utility companies to take preventative action that avoids large-scale power outages and costs. Today, the costs of sensors, data capture, and information storage have significantly decreased and are one tenth of what they were 10 years ago, leading to the proliferation of data that enables advanced analytics-driving efficiencies.

These efficiencies are achieved through remote inspection solutions, avoiding staff travel, and cutting resolution times. Furthermore, thanks to predictive maintenance and the optimization of asset management efficiency, aspects such as energy loss and increased service time of transformer plants are improved.

This is possible thanks to the implementation of IoT solutions boosted by the introduction of communication improvements such as 5G or the future 6G technology, which will have a transmission speed of 1,000Gbp/s, compared to the 600Mbp/s of 5G. Through these technologies, energy companies will be able to collect data and analyze it in real time, allowing them to optimize operating costs, even defining predictive maintenance policies that guarantee the level of service and savings.

Towards a better customer experienceLastly, we examine retail companies, the energy marketers. The demand for energy in the retail market has been practically flat in recent years. Additionally, we are in a segment of the value chain where there is fierce competition and new competitors are more digital and more agile.

The authority and the struggle to retain or capture customers has become the precedent for these companies, where knowing the customer experience in detail and seeking to maximize customer satisfaction has to be the priority. In this endeavor—the use of data, profiling customers 360°—is imperative, as with the liberalization of the market, customer churn rates of up to 25% have been observed according to analysts.

In this context, recently created companies have the advantage of having been born digital, but nevertheless with less financial muscle and with the challenge of finding the scale that will make them viable in the medium and long term. On the other hand, traditional, larger companies have huge cost structures, are less agile, and need digitization to compete. Solutions such as automated voice analytics in call centers, the integration of these analytics with communications systems like mobile applications or the corporate website, as well as automatic consumption analysis and energy price will allow companies to have a better understanding of customers, increasing lifetime value and reducing churn.

Other aspects that can improve the efficiency of retail companies are better assessment of customer creditworthiness and consumption variation, minimizing defaults, and avoiding fraud. The impact of the use of different analytical techniques in this field increases the profitability of these companies by 5% to 10%, at the same time increasing the brand value by increasing customer satisfaction.

The use of advanced analytics in corporate centers is less talked about and has an estimated return of between 8% and 9% reduction in operating costs. Within this group, we can find aspects such as improvements in the management and investigation of accidents at work as well as their prevention, or other aspects such as what is called “people analytics” in reference to the use of data analysis tools for management and decision-making in people management in the company. These include recruitment, training, performance management, and employee retention.

Going into more technological aspects, the data platforms that must support this decision-making must be able to operate in a hybrid environment in which there is integration with the applications that reside in the company’s own data centers, as well as the possibility of working in public cloud environments.

Data governance is another key aspect in solving these problems, as the data life cycle is complex and diverse from the capture of the information when it is generated, until it is consumed in a system that exploits the final information. If we take into account these two points—hybridization and governance—unified data management can be a challenge if the appropriate technology is not in place.

In short, we have seen how the use of analytical aspects and data management has a direct impact on the entire value chain of the energy business. In a changing environment such as the one we are currently experiencing, with crises and peaks in demand, the agility and adaptability of energy companies will be a differentiating aspect that will be reflected in profitability, market share, and shareholder value.

We are facing a paradigm shift where data analytics in energy use can be a differential factor in obtaining greater profitability, with lower production costs, a great increase in efficiency in generation, distribution, and transmission and, of course, an improvement in the loyalty of the end customer who can also benefit from savings in their bills and efficiency in their consumption.

To learn more:* Read how utilities lever Cloudera to manage smart meters data . * Explore operational efficiencies with our Federated Learning Machine Learning Prototype. * Replay our webinar: Machine learning model deployment: Strategy to implementation * Try Cloudera DataFlow (CDF), Cloudera Data Warehouse (CDW), Cloudera Data Engineering (CDE), and Cloudera Machine Learning (CML) by signing up for a 60 day trial, or test drive CDP. If you are interested in chatting about how to manage the full data lifecycle with CDP, let your account team know or contact us directly. As always, please provide your feedback in the comments section below.

The post Optimizing the Energy Sector with Data Analytics appeared first on Cloudera Blog.

View Details

We are pleased to announce that Cloudera has been named a Leader in the 2022 Gartner® Magic Quadrant for Cloud Database Management Systems. Cloudera has been recognized in this cloud DBMS report since its inception in 2020. This year we’ve been named a Leader. This validates our significant momentum in global enterprises. And together, with our recent recognition in the Gartner Peer Insights Customer Choice Distinction for Cloud DBMS, cements our position as an industry leader.

We’re proud to be recognized for the data management and data analytics innovations we have delivered in the new Cloudera Data Platform (CDP). The latest version of CDP represents a major milestone and a bold step forward for our mission. Cloudera has always been in the forefront of disruptive technical innovation in data platforms. We make the data tools people need to confidently tackle the toughest data challenges. In other words, Cloudera makes hard data problems easier. We did it a decade ago when data was big and servers were expensive. We do it today when data is even bigger, and hybrid—and clouds—are expensive.

Only Cloudera delivers the hybrid and multi-cloud flexibility enterprises need today. We hear it from our customers and industry experts. For example, Gartner has provided very pointed guidance on the urgency and inevitability of hybrid and multi-cloud for data management. Only Cloudera has delivered a solution.

There are six key capabilities that cement our leadership in enterprise data platforms:

1-Hybrid and multi-cloud portability and scale to support the most demanding workloads. This is unique. Most other cloud DBMS vendors only work in the cloud. Anything else requires integration, sometimes between multiple vendors, which means complexity and risk.

Instead, Cloudera enables the same data services running on private and public clouds with replication capabilities, so companies can easily move workloads when needed. This is a strength, reflected in our 5.0 score on this in the associated Gartner Critical Capabilities for Analytical Use Cases. Cloudera’s platform enables teams to burst compute intensive machine learning workloads to the cloud. Notably, these same services simplify repatriating data workloads back to private clouds, to save on cloud infrastructure expenses. That’s game-changing for performance, budgets, and business continuity. Cloudera is closely partnered with the leading cloud service providers (CSPs), and has optimized our platform and services to run as efficiently on their infrastructure services as possible. This strong focus on multi-cloud and hybrid is essential for our enterprise customers.

2-A truly open data lakehouse. Cloudera has long had the capabilities of a data lakehouse, if not the label. Cloudera enables an open data lakehouse architecture that combines all the flexibility of the data lake with the performance of the data warehouse, so enterprises can use all data—both structured and unstructured. The integration of Apache Iceberg as a native table format in CDP enables customers to build one single data layer for all their workloads without ever moving their data or creating unnecessary silos. It eliminates ETL overhead, while increasing the usability of the data.

3-Streamlined deployment of complex, multi-function workloads. Enterprises run thousands of different workloads on Cloudera. Our open, interoperable platform is deployed easily in all data ecosystems, and includes unique security and governance capabilities. Many of our customers use multiple solutions—but want to consolidate data security, governance, lineage, and metadata management, so that they don’t have to work with multiple vendors. CDP provides integrated governance, in the form of our unique Shared Data Experience (SDX)—specifically highlighted in the recent Gartner report as a strength.

4-Ready for modern data fabric architectures. Cloudera’s investments in SDX are far ahead of the industry standards. We have been investing in development for years to deliver common security, governance, and metadata management across the entire data layer with capabilities to mask data, provide fine grained access, and deliver a single data catalog to view all data across the enterprise. This helps our customers quickly implement an unified data fabric architecture.

5-Integrated open data collection. This differentiator solves a major technical challenge for data projects. With Cloudera, enterprises can collect all data, and run all data workloads with a single data platform. Our new Universal Data Distribution (UDD) capability, launched earlier this year, can collect data from any source and deliver it to any destination for a scalable data pipeline. UDD works on any source and destination, even outside of Cloudera, making it very easy to integrate varied data sources.

6-Operational efficiency to optimize workload performance and cost. Only Cloudera includes integrated capabilities for the entire data lifecycle; data preparation to advanced analytics; and has automation built into all our data services. Low and no-code features and templates make it easy to get started with streaming and machine learning use cases. ReadyFlow galleries and Applied Machine Learning Prototypes (AMPs) significantly reduce time to value for data flow and ML projects. Our long history in leading MPP architectures means we make the best use of cloud native capabilities—delivering the highest performance at the lowest costs.

At Cloudera, our ultimate goal is to empower customers to transform their businesses by providing better, faster use of data with a hybrid, open, portable, and secure data platform for analytics, streaming, ML, and data management at scale. We are proud to be recognized for our history and leadership.

Get an introduction to the latest version of Cloudera’s Data Platform

https://www.cloudera.com/products/cloudera-data-platform.html

See how data and IT leaders use Cloudera to deliver outcomes faster

https://www.cloudera.com/why-cloudera/data-leaders.html

Get a complimentary copy of the Gartner 2022 Magic Quadrant for Cloud DBMS

https://www.cloudera.com/campaign/2022-gartner-magic-quadrant-for-cloud-database-management-systems.html

Note:

GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used here in with permission. All rights reserved.

Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner reearch publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

The post Cloudera Named a Leader in the 2022 Gartner® Magic Quadrant™ for Cloud Database Management Systems (DBMS) appeared first on Cloudera Blog.

View Details

Since we announced the general availability of Apache Iceberg in Cloudera Data Platform (CDP), Cloudera customers, such as Teranet, have built open lakehouses to future-proof their data platforms for all their analytical workloads. Cloudera partners are also benefiting from Apache Iceberg in CDP. For example, Modak Nabu is helping their enterprise customers accelerate data ingestion, curation, and consumption at petabyte scale. Today, we are thrilled to share some new advancements in Cloudera’s integration of Apache Iceberg in CDP to help accelerate your multi-cloud open data lakehouse implementation.

Multi-cloud deployment with CDP public cloudMulti-cloud capability is now available for Apache Iceberg in CDP. According to a recent Gartner survey of public cloud users, 81% of organizations are working with two or more public cloud providers. With CDP, customers can deploy storage, compute, and access, all with the freedom offered by the cloud, avoiding vendor lock-in and taking advantage of best-of-breed solutions. You can leverage Kubernetes (K8s) and containerization technologies to consistently deploy your applications across multiple clouds including AWS, Azure, and Google Cloud, with portability to write once, run anywhere, and move from cloud to cloud with ease. With a common interface in CDP that works across different cloud service providers, you can break down data silos while ensuring consistent security, governance, and traceability, all while seamlessly moving your Apache Iceberg–based workloads across deployment environments frictionlessly.

Advanced capabilitieThe new capabilities of Apache Iceberg in CDP enable you to accelerate multi-cloud open lakehouse implementations.

• Enhanced multi-function analytics

In addition to key data services in CDP, such as Cloudera Data Warehousing (CDW), Cloudera Data Engineering (CDE), and Cloudera Machine Learning (CML) already in use by our customers, we integrated Cloudera Data Flow (CDF) and Cloudera Stream Processing (CSP) with the Apache Iceberg table format, so that you can seamlessly handle streaming data at scale. Compute engines in these CDP data services can access and process data sets in the Iceberg tables concurrently, with shared security and governance provided by our unique Cloudera Shared Data Experience (SDX).

• Amazingly fast table migration

With in-place table migration, you can rapidly convert to Iceberg tables since there is no need to regenerate data files. Only metadata will be regenerated. Newly generated metadata will then point to source data files as illustrated in the diagram below.

• Data quality using table rollback

When data quality issues come to light, you can use table rollback to get back to a known high quality state. You can quickly restore data to a known good state, and take corrective actions faster and easier.

• Maintaining performance and manageability with improved table maintenance

Improve performance and overall manageability of Iceberg tables using the new table maintenance capabilities such as expiring old snapshots and removing their metadata, and compaction to combine small files for more efficient data processing.

• ORC open file format support

In addition to the Parquet open file format support, Iceberg in CDP now also supports ORC in the latest release. Support for these common industry standard open file formats further helps accelerate adoption of Iceberg and open lakehouse implementation.

• Accelerate analytics with materialized view support

In CDP, users can create materialized views on top of Iceberg tables. Materialized views are an industry standard practice for databases to accelerate analytics query execution by significant orders of magnitude.

• Performance and scalability

Cloudera developed unique features in CDP for Iceberg query performance and scalability for large data sets including I/O caching, dynamic partition pruning, vectorization, Z-ordering, parquet page indexes, and manifest caching.

General availability of ACID transactions with Iceberg tables Since we launched our support for Apache Iceberg in CDP, newer releases have been under development at Apache. Apache Iceberg version 0.14.1 (a.k.a. Apache Iceberg v2) provides support for data modification language (DML) operations such as row-level delete and update. With CDP’s Iceberg v2 general availability, users are able to maintain transactional consistency on Iceberg tables even when accessing the same data using multiple engines simultaneously. With Iceberg v2, you can access and process data, all while maintaining read consistency and multi-engine/user concurrent writes due to serializable isolation and optimistic concurrency control. In addition to DELETE and UPDATE SQL commands developed for DML, the MERGE SQL command is also offered to take advantage of row-level DML operations to simplify ETL data pipelines.

Integrated with Cloudera Data PlatformIceberg tables supported on CDP, automatically inherit the centralized and persistent Shared Data Experience (SDX) services—security, metadata, and auditing—from your CDP environment.

The following SDX security controls are inherited from your CDP environment:

• Authentication

CDP integrates with your corporate identity provider to maintain a single source of truth for all user identities.

• Fine grained authorization

Ensures that only users who have been granted adequate permissions are able to access the Iceberg tables and the data stored in those tables.

• Auditing

Apache Ranger provides a centralized framework for collecting access audit history and reporting data, including filtering on various parameters.

• Metadata management

Apache Atlas provides services to collect metadata when the service performs certain operations. You can use Atlas to find, organize, and manage different aspects of data about your Iceberg tables and how they relate to each other. This enables a range of data stewardship and regulatory compliance use cases.

SummaryCloudera’s integration of Apache Iceberg in CDP continues to benefit from new enhancements as we join the community in innovating on this modern table format. New capabilities such as multi-cloud deployment, ACID compliance, and enhanced multi-function analytics accelerate implementation for the multi-cloud open data lakehouse to meet ever-evolving requirements for modern data warehouse, data lake, AI/ML, data science, and more.

To learn more:* Replay our webinar Unifying Your Data: AI and Analytics on One Lakehouse, where we discuss the benefits of Iceberg and open data lakehouse. * Read why the future of data lakehouses is open. * Replay our meetup Apache Iceberg: Looking Below the Waterline.

Try Cloudera DataFlow (CDF), Cloudera Data Warehouse (CDW), Cloudera Data Engineering (CDE), and Cloudera Machine Learning (CML) by signing up for a 60 day trial, or test drive CDP. If you are interested in chatting about Apache Iceberg in CDP, let your account team know or contact us directly. As always, please provide your feedback in the comments section below.

Other Contributors to this article: Manish Maheshwari, Peter Ableda, Navita Sood , Imran Rashid, Priyank Patel, Michael Kohs, Ashish Shah, David Dichmann, Joseph Niemiec

The post Implement a Multi-Cloud Open Lakehouse with Apache Iceberg in Cloudera Data Platform appeared first on Cloudera Blog.

View Details

OCBC Bank optimizes customer experience & risk management with multi-phased data initiativeOCBC Bank is the second largest financial services group in Southeast Asia by assets and one of the most highly-rated banks in the world. Recognised for its financial strength and stability, OCBC Bank is consistently ranked among the World’s Top 50 Safest Banks by Global Finance. The bank and its subsidiaries offer a broad array of commercial banking, specialist financial and wealth management services, ranging from consumer, corporate, investment, private and transaction banking to treasury, insurance, asset management and stockbroking services.

In 2015, OCBC began a multi-phased initiative with Cloudera focused on giving customers access to its banking services through an easy, convenient user interface that delivered targeted and tailored products and services. The company recently migrated to Cloudera Data Platform (CDP) and CDP Machine Learning to power a number of solutions that have increased operational efficiency, enabled new revenue streams and improved risk management. OCBC also won a Cloudera Data Impact Award 2022 in the Transformation category for the project.

Real-time data analysis for better business and customer solutionsOCBC identified the need to upgrade its data lake technology as part of an enterprise data science initiative to introduce a more resilient infrastructure and platform capable of managing projects with increasing volume, variety and velocity of data, while also enabling real-time analytics.

Andrea Pisoni, Head of Data Science says “OCBC worked with Cloudera to design and secure big data platforms as part of its comprehensive data project. The partnership has enabled OCBC to better store, manage and harness the power of our data.”

One example is OCBC’s Next Best Conversation, a centralized platform that uses machine learning to analyze real-time contextual data from customer conversations related to sales, service and more. The bank was able to increase their revenue by more than $100M annually by using the data to identify the most relevant information for each customer and curate personal experiences across communication channels.

OCBC also developed a credit card fraud detection solution that reduced the volume of transactions reviewed by anti-money laundering compliance analysts and increased the accuracy rate of identifying suspicious transactions. They were also able to develop smarter processes on the platform by introducing chatbots to take over 10% of customer interactions on their website.

“We are committed to democratizing AI and transforming our key processes using data. It is critical that the migration process goes smoothly, and the seamless collaboration between OCBC and the Cloudera professional services team helped.” Donald Macdonald, CDO, OCBC

Ultimately, Cloudera’s support and platform will be helpful for us to accelerate our data strategy and allow us to continue innovating and grow with efficiency.”

To find out more about how Cloudera empowers financial services leaders to realize the endless possibilities of data, read our latest FSI Guide, A world without limits.

The post OCBC Bank Accelerates Its Data Strategy with Cloudera appeared first on Cloudera Blog.

View Details

How Agencies Can Gain the Cyber Edge with Smart Data SolutionsFor the vast majority of US citizens, the front lines of conflict are witnessed from thousands of miles away on the nightly news. But for government agencies, these physical conflicts are the tip of the iceberg as cyberattacks persist as an underlying constant, inflicting enduring damage regardless of geopolitical tension or location.

Consider the fact that the SolarWinds Orion supply chain breach (attributed to Russia) from 2020 continues to make news, with many customers still in the dark as to whether they were compromised or not. The US Security and Exchange Commission recently announced a class action lawsuit enforcement action taking place stating that SolarWinds didn’t do enough to secure its customers. The attack targeted a host of public and private sector organizations (18,000 customers) including NASA, the Justice Department, and Homeland Security, and it is believed the attackers persisted on SolarWinds systems for 14 months prior to discovery.

The question then, at the forefront of CISOs’ minds is, “How do you outpace the threats given the current IT estate?” Or better yet, “How do we empower people with enterprise data solutions that amplify positive outcomes in the security operations center?”

The answer: You must amplify the impact of data science talent on the mission and the cyber front lines, delivering advanced analytics products powered by machine learning (ML) into the hands of the threat hunters.

The benefits and challenges of ML operationsThe world is awash in data. Nearly everything around us is producing or consuming data at exponential rates, most of which is connected to the Web. The same holds true for the public sector. Data is everywhere as an opportunity and a target for malicious actors. But ML presents your cyber security operations with a force multiplier to leverage smart data solutions at scale in the cyber-mission. ML enables:

• Massive amounts of historic and real-time data to be synthesized and prepared for analysis
• Manual and repetitive tasks to be automated, freeing up resources for time-critical tasks
• Continuous learning from evolving data sets including labeled detections and alerts
• Automated predictive threat detection, evaluation, and response
• Augmentation of analyst insights with real-time ML detection
• Maximizing true positive detections while minimizing false positives

While these benefits are a potential boon for your cyber security operations, real-world implementation is often hindered by technical barriers, disparate systems, and interfaces that result in ML production swivel chair. Workflows become so cumbersome that projects never make it past pilot and most importantly, data scientists’ ML models rarely emerge from experimentation to operation.

Operationalize ML with the Cloudera Data Platform

In order to overcome the technical barriers, you need a unified data platform that consolidates the data management solutions that support ML operations into a single interface that delivers optimal performance, scalability, and security.

The Cloudera Data Platform (CDP) is a hybrid data platform for all your end-to-end machine learning (ML) production use cases. CDP enables you to collect, monitor, analyze, and store data from continuous streams, and once the data has been stored in the data lake house, CDP enables the entire ML life cycle by allowing your users to train, package, and serve ML models from a single secure platform. All with the integrated security and governance technologies required for compliance.

With CDP you can unlock the potential of your data scientists as ML models emerge from research projects to mission-critical components or services that can be productized and handed off to analysts without requiring details of the model, empowering the operator or cyber analyst while reducing the mean time for threat resolution.

To learn more, read our technical brief: Securely Train and Deploy Machine Learning Models in Production with Cloudera Data Platform.

The post How Agencies Can Gain the Cyber Edge with Smart Data Solutions appeared first on Cloudera Blog.

View Details

Over the past few months industry analysts have been making some pretty controversial recommendations for data management in the cloud. For a thoughtful and entertaining analysis, I strongly recommend you spend a few minutes watching the keynote session by Pat Moorhead, CEO Moor Insights & Strategy, at the Evolve 2022 Data event in New York. His takeaway: “The world is very much going to be hybrid and multi-cloud.” His delivery—priceless.

Pat isn’t the only analyst talking hybrid and multi-cloud for data management, although he may be the most entertaining. ZDNet had a thorough piece on why single clouds are passé. Workload portability with no cloud lock-in was cited as critical to digital initiative success. Gartner has also been providing very pointed guidance to clients on the urgency and inevitability of hybrid and multi-cloud for data management. If you haven’t been paying attention, here are the key points from my point of view.

Cloud is hybrid and multiIn March 2022, Gartner published A Strategic Roadmap for Migrating Data Management to the Cloud. The big takeaway for me was the urgency of true hybrid and multi-cloud capabilities. These two instructions really struck me:

“The future data ecosystem should leverage distributed data management components—which may run on multiple clouds and/or on premises—but are treated as a cohesive whole with a high degree of automation. Integration, metadata, and governance capabilities glue the individual components together.”

“Through 2026, 90% of data management tools and platforms that fail to support multi-cloud and hybrid capabilities will be set for decommissioning within three years.”

My translation: Don’t start any data management projects without a hybrid and multi-cloud data management platform.

That’s big. None of the hyperscalers do true hybrid and multi-cloud data management today. Most do uni-cloud, a single cloud that sometimes operates closer to the enterprise. It doesn’t manage or analyze data in the datacenter or on premises, but is designed to move data to their uni-cloud. Neither do the point solution (more on them later). Some do multi-cloud, but not hybrid and not really data management. They are more uni-service. Nothing wrong with uni, but it’s not hybrid.

Data management is hybrid and multiIn September Gartner published Innovation Insight: Data Ecosystems Will Reshape the Data Management Market. In my view they doubled down on the March guidance. The most important guidance from my point of view:

“The effective management of data is essential to every aspect of software, analytics, AI/ML, and every digital use case.”

“Hybrid will never disappear; as data ecosystems become the center of gravity for data management, they will enable the integration of data across a hybrid environment.”

“By 2024, 50% of new system deployments in the cloud will be based on a cohesive cloud data ecosystem rather than on manually integrated point solutions.”

My translation: If you don’t have effective data management, you will fail at AI and digital transformation. Data management must be hybrid, otherwise don’t start. And about point solutions: integration costs and lack of hybrid support make them economically and functionally undesirable.

Netting it all out, my read is that Gartner is “all in” on recommending hybrid and multi-cloud and multi-function as essential for data management platforms. The hyperscalers don’t have solutions today. Nor the point solutions for data management and analytics.

Businesses are going hybrid and multiWe are hearing and seeing the same from leading businesses around the world. Hybrid is the new normal. Hybrid data is the future and the future is now.

• A large European multinational insurer implements a central data science factory that leverages both data center and cloud resources to develop new use cases faster than ever.
• A large African telecommunications organization consistently manages and analyzes data from their businesses across the continent where different countries and regions have very different levels of infrastructure maturity.
• One of the world’s largest banks treats cloud infrastructure as a utility, running their data workloads in the cloud that is the most cost effective at the moment.

Cloudera makes hybrid and multi-cloud easierI encourage you to read both analyst papers thoroughly and checkout Pat’s Evolve 2022 keynote. They are content and controversy rich. And they confirm to me that the direction on data management is becoming very clear:

• Hybrid & multi-cloud is a given
• Hybrid is forever
• Hybrid and open data offer ultimate flexibility
• Point data solutions are a dead end
• Data management at scale is a pre-req for digital

Everyone needs to learn more about hybrid data. Cloudera makes hybrid data easier. Let’s talk.

The post Go Hybrid & Multi-Cloud or Don’t Go appeared first on Cloudera Blog.

View Details

Cloudera has been providing enterprise support for Apache NiFi since 2015, helping hundreds of organizations take control of their data movement pipelines on premises and in the public cloud. Working with these organizations has taught us a lot about the needs of developers and administrators when it comes to developing new dataflows and supporting them in mission-critical production environments.

In 2021 we launched Cloudera DataFlow for the Public Cloud (CDF-PC), addressing operational challenges that administrators face when running NiFi flows in production environments. Existing NiFi users can now bring their NiFi flows and run them in our cloud service by creating DataFlow Deployments that benefit from auto-scaling, one-button NiFi version upgrades, centralized monitoring through KPIs, multi-cloud support, and automation through a powerful command-line interface (CLI). Recently, we announced the general availability of DataFlow Functions, allowing NiFi flows to be executed in serverless compute environments, such as AWS Lambda, Azure Functions, or Google Cloud Functions.

With DataFlow Deployments and DataFlow Functions being available, flow administrators can now pick the best option for running their dataflows in production in the public cloud. Now, we shift focus on the needs of developers and addressing the challenges they face when building dataflows in the cloud.

Enabling self-service for developersDevelopers need to onboard new data sources, chain multiple data transformation steps together, and explore data as it travels through the flow. They value NiFi’s visual, no-code, drag-and-drop UI, the 450+ out-of-the-box processors and connectors, as well as the ability to interactively explore data by starting individual processors in the flow and immediately seeing the impact as data streams through the flow.

We’ve observed organizations using more and more data sources and destinations, as well as expecting a more diverse range of developers to build data movement flows. This observation further emphasizes the need for universal developer accessibility, which makes sure that developer tooling is easy to use for newcomers while giving power users the advanced options they need. A critical aspect of universal developer accessibility is to provide dataflow development as a self-service offering to developers. This is a challenge because developers are either required to manage their own local Apache NiFi installation, or a platform team is required to manage a centralized development environment that all developers can use.

What if there was a way to not require developers to manage their own Apache NiFi installation without putting that burden on platform administrators? What if we could provide an easy-to-manage, self-service development environment for developers that anyone can start using immediately?

These are the questions we asked ourselves, and I am excited to announce the technical preview of DataFlow Designer, making self-service dataflow development a reality for Cloudera customers.

A reimagined visual editor to boost developer productivity and enable self serviceAt the core of our new self-service developer experience is the new DataFlow Designer, which reinforces NiFi’s most popular features while making key improvements to the user experience—all presented in a fresh look and feel.

Figure 1: The Designer canvas with a brand new look and feel

A key improvement over the traditional Apache NiFi canvas is the new expandable configuration side panel, allowing developers to quickly edit processor configurations without losing focus of what’s happening on the canvas. The side panel is context-sensitive and instantly displays relevant configuration information as you navigate through your flow components.

Figure 2: Don’t lose sight of the canvas while applying configuration changes in the side panel

Another example of how the new flow designer makes a developer’s life easier is the ability to directly upload files through the designer UI. In traditional NiFi development environments, developers would either require SSH access to the NiFi instances to upload files or ask their administrators to do it for them. Having the ability to upload files like JDBC Drivers, Python scripts, etc. directly in the designer makes building new flows a lot more self-service.

Figure 3: Easily upload files directly through the designer without requiring SSH access to servers

Speaking of parameters—they are an important concept to make your dataflows portable. After all, it’s very likely that you are developing your flow against test systems but in production it needs to run against production systems, meaning that your source and destination connection configuration has to be adjusted. The best way to do this is by parameterizing these connection configuration values allowing you to plug in different values when creating a flow deployment in production. You can set default values for parameters as well as mark them as sensitive, which ensures that no one can see the value that was set.

Figure 4: Central management of flow parameters

The Designer supports on-the-fly parameter creation when configuring components as well as auto-complete by pressing CTRL+SPACE when providing a configuration value. As a result, parameter management is always at your fingertips right where you need it without requiring you to switch between views to look them up.

Figure 5: Parameter references in the configuration panel and auto-complete

Interactivity when needed while saving costsOne of NiFi’s unique features is the ability to interact with each component in a dataflow individually without having to stop the entire flow. This allows developers to make changes to their processing logic on the fly while running some test data through their flow and validating that their changes work as intended. For example, if your dataflow is reading events from a Kafka topic, which you want to filter and process but you’re not sure about the exact schema the events are in, you might want to peek at the events before writing your filter condition. With NiFi you can configure your source processor and run it independently of any other processors to retrieve data. Once you have retrieved the data, NiFi stores it in a queue, which allows you to explore the content and metadata attributes of the events. Once you know how your events look, you can move to the next step in your flow and define the filter condition and further processing logic. This makes it easy for developers to iterate and validate each processing step as well as onboard new data sources that they’re not familiar with.

We wanted to preserve the rapid, interactive development process while keeping the cost for required infrastructure low—especially during times when developers are not working on their flows. To meet this need we’ve introduced a new concept called test sessions with the DataFlow Designer.

When a developer creates a new dataflow, they are immediately directed to the Designer and can start building their flow without having to wait for any resources to be created. They can drag and drop processors to the canvas immediately, create parameters and services, and apply configuration changes.

Figure 6: Developers can start building dataflows immediately without requiring any NiFi resources to be allocated—note the grayed out processors indicating that no test session is active

As soon as they want to run a processor and test their flow logic, they can initiate a test session, which provisions NiFi resources on the fly within minutes.

Figure 7: Test sessions provide an interactive experience that NiFi developers love

Once a test session is active, developers can start or stop individual processors and services and explore data in the flow to validate their flow design. When the test session is no longer needed, developers can terminate it, freeing up the resources and saving costs. Test sessions act like on-demand NiFi sandboxes for developers.

Figure 8: Once a test session has been started, developers can interact with processors and monitor data as it is processed by their dataflow

A streamlined deployment process from development to productionDeveloping and testing dataflows is the first step in the dataflow life cycle, and needs to integrate well with deploying and monitoring dataflows in production environments. With the designer becoming available in CDF-PC, we can now support flow developers and flow administrators alike through a streamlined process.

Figure 9: Developers can create new draft flows as needed

Developers create draft flows, build them out, and test them with the designer before they are published to the central DataFlow catalog. Once they are in the DataFlow catalog, flow administrators can deploy them in their cloud provider of choice (AWS or Azure) and benefit from the aforementioned features like auto-scaling, one-button NiFi version upgrades, centralized monitoring through KPIs, and automation through a powerful CLI.

Figure 10a: Once a draft flow has been validated using a test session, developers can publish them to the DataFlow catalog for production deployments

Figure 10b: As part of the publication step, developers can leave comments and are redirected to the catalog from where they can initiate a deployment

Looking ahead and next stepsThe DataFlow Designer technical preview represents an important step to deliver on our vision of a cloud-native service that organizations can use for all their data distribution needs, and is accessible to any developer regardless of their technical background. Cloudera DataFlow for the Public Cloud (CDF-PC) now covers the entire dataflow lifecycle from developing new flows with the Designer through testing and running them in production using DataFlow Deployments or DataFlow Functions depending on the use case.

Figure 11: Cloudera DataFlow for the Public Cloud (CDF-PC) enables Universal Data Distribution

The DataFlow Designer is now available to CDP Public Cloud customers as a technical preview. Please reach out to your Cloudera account team or to Cloudera Support to request access.

Stay tuned for more information as we work towards making the DataFlow Designer generally available to CDP Public Cloud customers and sign up for our upcoming DataFlow webinar or check out the DataFlow Designer technical preview documentation.

The post Introducing Cloudera DataFlow Designer: Self-service, No-Code Dataflow Design appeared first on Cloudera Blog.

View Details

In case you were not aware, there’s a little event called the World Cup that’s happening right now. This World Cup has been notable for a couple reasons. The first being the timing—no summer watch party barbeques this time around, instead FIFA is breaking from tradition and running the tournament in the northern hemisphere winter months to spare the players the experience of playing soccer (Cloudera is headquartered in the US, so it is “soccer”) in temperatures exceeding 41.5°C (Cloudera is headquartered in the US, but we also recognize the superiority of the metric system).

The second notable fact about the 2022 World Cup is that this is only the second World Cup to be held entirely in Asia, the first being the 2002 tournament held in South Korea and Japan. However, it is the first World Cup to be held in the Middle East region of the world!

The third, and most interesting fact about the 2022 World Cup, is the new and innovative ways that technology and data are being used to improve the beautiful game, both on and off the pitch. For off the pitch innovations, Qatar has implemented solutions like a state-of-the-art cooling system, and even cameras and computer vision algorithms designed to prevent stampedes. For the fans, you don’t have to look far to find new and exciting ways that technology is enhancing their experience.

The data innovation that I was most excited to learn about though is the implementation of a human-in-the-loop (HITL) machine learning (ML) solution to assist referees in more accurately calling offsides. Officially, FIFA is referring to this ML solution as Semi-Automated Offside Technology (SAOT). Human-in-the-loop ML is not a new or novel technology, but the use of it at soccer’s largest stage is a major step for ML as a mechanism to improve the quality of officiating at professional sporting events.

What is human-in-the-loop machine learning?Machine learning is a subcategory of artificial intelligence where computer systems learn to do tasks based on data rather than being explicitly programmed to do so. HITL ML adds an additional step that requires a human (preferably a subject matter expert) to verify the tasks being performed by the computer system.

HITL ML essentially combines the strength of both ML and humans. ML has the unique advantage of being able to scale across multiple systems and process data exponentially faster than the human brain, allowing it to handle many more tasks than a human ever could. ML is not always perfect though, and so by including humans that are subject matter experts in the training of the system and the tasks being performed, you can minimize the likelihood of ML performing tasks incorrectly.

Humans can be involved in either the training of the system through providing it with the data that it learns from, or in the case of SATO, humans can be used to verify that the task performed was done so accurately.

A world-class machine learning solutionThe ML model implemented as part of SAOT is trained to classify a play as either offsides or not. It uses two main sources of data as inputs: The first is Adidas’ new IoT enabled ball, the Al Rihla Pro. This revolutionary ball contains a sensor inside of it that senses the ball’s inertia; that data is captured and reported a remarkable 500 times per second. The data from the ball provides a precise measurement of the direction that a ball was kicked, at the moment it was kicked (well, within 1/500th of a second that is).

The second piece of technology used as inputs to the ML model comes from 12 cameras mounted just underneath the roof of the stadium. These cameras capture 29 data points on each of the 22 players on the field, at a rate of 50 times per second.

That means that 17,400 positional data points from players on the field are used as inputs into SAOT’s model every second.

With these two sources of data, inertia data from the ball, and player positional data from the cameras, the SAOT ML model is able to provide a classification of whether each play is either offsides or onsides. Now here is where the HITL aspect of the solution comes into play. The offside prediction doesn’t go directly to the on-field referee, it is instead sent to the VMOs (video match officials) who then validate the offside prediction. The model provides the kick point of the pass that resulted in an offsides play, as well as a generated offside line with the 29 points of the offensive and defensive players at the moment of the pass. If the VMOs agree that the play was indeed offsides, they inform the on-field referee.

Other applications for human-in-the-loop machine learningA natural extension of this technology would be in other sports. Imagine if an automated system was used to inform NFL referees whether or not a player stepped out of bounds, or if the ball went across the goal line in mayhem of a QB sneak from the one yard line. In the NBA, HITL ML could be used to definitively classify a play as a block or a charge (the bane of any NBA fan’s experience).

What’s great about HITL ML is the speed at which it occurs. There would be no more five-minute review of the same camera angle in which we all disagree if there is court/grass between a player’s foot and a line. Instead, the prediction is instantly available. Additionally, because trained officials are still used to verify, inaccurate predictions are caught (and there’s no possibility of Sky Net going active and robots taking over the world).

In business, HITL methodologies can be used to minimize downtime due to an incorrect precision of a failure in a predictive maintenance application, and give confidence to business stakeholders that the output from ML models can be trusted.

It takes more than machine learning to solve these problemsNotice that the semi-automated offside technology solution wasn’t just an ML model sitting in the cloud. It required data to be streamed, transformed, loaded, analyzed, and reported, all within a matter of seconds. A solution for something like that requires data services for every step of this process, and these data services have to work together seamlessly, both on premise (the ball and cameras) and in the cloud (model training, predictions, and reporting web applications).

This is why Cloudera has built the hybrid data platform (the Cloudera Data Platform) with integrated data services for every step of the end-to-end data lifecycle, because anyone who has built ML solutions knows that it takes more than just an ML point solution in the cloud to deliver a business ready solution.

If you would like to learn more about how the Cloudera Data Platform is the hybrid solution you’ve been looking for, go here to learn more.

The post The Newest FIFA World Cup Referee: Human-in-the-Loop Machine Learning appeared first on Cloudera Blog.

View Details

In the spirit of showing thankfulness, I want to share my story of the opportunities given to me by my employer and the gratitude I have for this change of life.

My humble beginningsWhen I was 16 years old, I found myself in my mom’s backyard with a shovel creating a makeshift restroom because the water and electricity had been shut off for three days due to unpaid utility bills. I remember at that moment deciding this was not the future I wanted to create for myself and my future family.

I have been with Cloudera for three years as a field marketing specialist. This is my story of how Cloudera has helped me change the direction of my life, and put an end to generational poverty.

Like many women of Native American descent, I was raised somewhere between poverty and lower middle class. I married young, had children young, dropped out of college, and became a stay-at-home mom. Although I seemed to have a picture perfect family, I endured darkness and various forms of abuse for almost 20 years due to not having a way of escape. I had to make a change—I had to find a way for me and my children to survive, and make good on that promise to my 16-year-old self.

The start of something newIn 2019, I applied for an entry-level position with Cloudera and started as a part-time contractor with an opportunity for full-time employment, training, and career development. For the next two years, the field marketing team taught me how to perform my job responsibilities. I was introduced to tools that I had never been exposed to before, and was on the fast track to gaining valuable and transferable skills and experiences that would give me a competitive edge. In my personal life, I was able to become self-sufficient and started my life over with my children. In 2021, I was offered a full-time position with Cloudera with a further commitment to help me develop my career in the ever changing, fast-paced, B2B marketing world.

An opportunity to be thankful forBecause of Cloudera’s commitment to providing opportunities to all people, no matter their background or education, I drastically changed the trajectory of my life and the lives of my children. I became a part of an industry that before would have never been reachable for someone with my upbringing and demographics. Today, I watch my children enjoy a safe home environment while they pursue their education and explore opportunities that I never had growing up.

Cloudera has had a profound impact on my life. I am proud to be among the few who defied the odds. Cloudera helped make that possible. I no longer live in poverty and I live a safe and independent life full of opportunity for me and those close to me. For this, I am grateful.

Inspired by Ashley’s story? Check out our open roles and join #ClouderaLife.

The post A New Path in Life: How Cloudera Provided Me with the Opportunity for Change appeared first on Cloudera Blog.

View Details

Clouderans in 2022 have collectively donated hundreds of hours to causes they care about around the globe. This kind of support for nonprofits is essential to their running, and to driving impact in local communities. For International Volunteer Day 2022, we are excited to celebrate Cloudera’s volunteer spotlights from 2022!

“I believe it is important to work with people and organizations that share common values to achieve their goals.”

Feeding hundreds and reducing food waste: meet Carlos Zorzin

Carlos Zorzin, based in Australia, shared his experience volunteering for a local bakery that donates leftover bread at the end of each day to shelters. Carlos volunteers two to three times per month to reduce food waste and feed hundreds of people using local shelters to access food and temporary housing.

Find out more about Vinnies (St Vincent de Paul Society NSW) here.

“Thinking about the chain reaction—how many people (full families, seniors, children ‘down the line’) benefit from this work and this organization. It’s humbling.”

Reducing food insecurity through a community hub: meet Michael Billau

Michael Billau, based in North Carolina, shared his experience sorting and distributing donated food to families in need, via his local church. Michael shares the impressive scale of both his contribution, donations, and then distribution.

Find out more about Food Bank of Central & Eastern North Carolina here.

“I become a calmer person and find myself much happier when I can help address the needs of those around me.”

Combining practical needs (food, sanitation) with a support network: meet Thatiane Freir

Thatiane Freir, based in Brazil, is a tenured volunteer with her organization, donating every Saturday for the last two years there. The organization combines its big impact (1,000 meals and 3,000 lunch boxes per month), with a support network and much-needed love and support.

Find out more about Casa de Caridad Inacio Daniel here.

“Coming from a first-generation in the US family, I was also provided with programs like this one as a kid, so I want to give back to kids in similar situations!”

Building confidence and trust in talented young people: meet Sharon Li

Sharon Li, based in California, shared her experience mentoring with the Boys & Girls Clubs of America. Working with one young mentee consistently, Sharon found it so rewarding to regularly connect and see confidence, trust, and knowledge grow in this young person.

Find out more about Boys & Girls Clubs of America here.

“By the second time I participated, I saw a shift in the way the candidates engaged in the interview.”

Empowering under-represented people into employment: meet Lynne Montalbo

Lynne Montalbo, based in California, volunteers to support under-represented people take successful steps into their first professional roles. As Lynne reflects, transitioning into “the real world” is difficult for everyone, and so Lynne is there to give that essential bit of extra support during this important time.

Find out more about Braven here.

“Once you see how even a small amount of work can impact someone’s life, it’s hard to stop volunteering.”

Developing technology skills to increase diversity and equity in technology: meet Glaucia Esppenchutz

Glaucia Esppenchutz, based in Portugal, answers tech questions, mentors students, and translates materials into Portuguese to support the next generation of talent enter technology roles more equitably.

Find out more about freeCodeCamp here.

“The reward of helping those in need and the sense of team camaraderie outweigh the danger. It’s truly second to none.”

Combining longtime passion and knowledge with life-saving rescues: meet Barry Laide

Barry Laide, based in Ireland, shared his experience being on call 365 days a year, 24/7, in case a walker or mountaineer in his local area needs to be urgently rescued. It’s an adrenaline-driven way to volunteer!

Find out more about Kerry Mountain Rescue here.

“I learned how much trouble humans can cause for animals…our carelessness and neglect is the main reason why these animals (along with a lot of others) get in trouble, get injured, killed, maimed, or sick. This is why I am a regular volunteer and why I plan to continue in the future.”

Protecting the protected: meet Dániel Omaisz-Takács

Dániel Omaisz-Takács, based in Hungary, is passionate about the protected wild eastern hedgehog (and his favorite animal) in Hungary. He is trained to rehabilitate these little creatures, and also reflects that humans could do more to reduce our impact on their environments by avoiding cutting down and burning wooded areas.

Find out more about Hungary’s Hedgehog Foundation here

Thank you to all Clouderans for donating and volunteering in 2022. We look forward to hearing more of your stories in 2023 and beyond!

The post From Hunger to Hedgehogs: Clouderans Drive Impact in 2022 Through Global Volunteering Efforts appeared first on Cloudera Blog.

View Details

Bringing Better Data Observability Into the Enterprise Stack

The post You Can’t Hit What You Can’t See appeared first on Cloudera Blog.

View Details

What is Cloudera Operational Database (COD) Cloudera Operational Database enables developers to quickly build future-proof applications that are architected to handle data evolution. It helps developers automate and simplify database management with capabilities like auto-scale, and is fully integrated with Cloudera Data Platform (CDP). For more information and to get started with COD, refer to […]

The post How to Deploy Transaction Support on Cloudera Operational Database (COD) appeared first on Cloudera Blog.

View Details

What is CDP Operational Database (COD) CDP Operational Database enables developers to quickly build future-proof applications that are architected to handle data evolution. It helps developers automate and simplify database management with capabilities like auto-scale, and is fully integrated with Cloudera Data Platform (CDP). For more information and to get started with COD, refer to […]

The post Transaction Support in Cloudera Operational Database (COD) appeared first on Cloudera Blog.

View Details

Introduction Stream processing is about creating business value by applying logic to your data while it is in motion. Many times that involves combining data sources to enrich a data stream. Flink SQL does this and directs the results of whatever functions you apply to the data into a sink. Business use cases, such as […]

The post Enriching Streams with Hive tables via Flink SQL appeared first on Cloudera Blog.

View Details

Customer Go-Live

The post Habib Bank manages data at scale with Cloudera Data Platform appeared first on Cloudera Blog.

View Details

Cloudera’s November Volunteer Spotlight is Glaucia Esppenchutz, staff data engineer, based in Lisbon, Portugal.  Glaucia volunteers with Free Code Camp, an organization founded in 2014 that helps aspiring technicians learn to code for free.  Through the creation and publication of videos, articles, and interactive coding lessons—all freely available to the public—Free Code Camp is able […]

The post #Clouderalife Volunteer Spotlight: Glaucia Esppenchutz appeared first on Cloudera Blog.

View Details

It’s all about storytelling for the chief data and analytics officer

The post Once Upon a Time in the Land of Data appeared first on Cloudera Blog.

View Details

OpDB Store File Tracking

The post Unlocking HBase on S3 With the New Store File Tracking Feature appeared first on Cloudera Blog.

View Details

Cloudera Partners with Generation to Support Job Training and Placement Programs Around the World

The post Cloudera and Generation Partner to Reskill the Tech Talent of Tomorrow appeared first on Cloudera Blog.

View Details

From Washington, DC, to the cloud: a career guided by lessons on the battlefield and a lifetime of US Army service

The post #ClouderaLife Spotlight: Timur Nersesov, Senior Manager of Professional Services Strategy appeared first on Cloudera Blog.

View Details

Cloudera has been working on Apache Ozone, an open-source project to develop a highly scalable, highly available, strongly consistent distributed object store. Ozone is able to scale to billions of objects and hundreds petabytes of data. It enables cloud-native applications to store and process mass amounts of data in a hybrid multi-cloud environment and on […]

The post Ozone Write Pipeline V2 with Ratis Streaming appeared first on Cloudera Blog.

View Details

As the Channel becomes increasingly competitive it is critical that we establish a partner ecosystem that minimizes the friction of transacting business, and that helps partners build more strategic relationships and close larger, more profitable deals.

The post Cloudera Partner Network: Poised to Heat up Channel Growth appeared first on Cloudera Blog.

View Details

It’s no secret that IT modernization is a top priority for the US federal government. A quick trip in the congressional time machine to revisit 2017’s Modernizing Government Technology Act surfaces some of the most salient points regarding agencies’ challenges: The federal government spends nearly 75% of its annual information technology funding on operating and […]

The post When Private Cloud is the Right Fit for Public Sector Missions appeared first on Cloudera Blog.

View Details

How to mitigate your risk against threats in the cloud with CDP One

The post Protect Your Assets and Your Reputation in the Cloud appeared first on Cloudera Blog.

View Details

In this blog post we describe how to use the REST API of Apache Solr in CDP Public Cloud directly or through Apache Knox Gateway. We show indicative performance measurements, demonstrating the performance improvement one can achieve by reusing cookies set by the Knox Gateway.

The post Using Apache Solr REST API in CDP Public Cloud appeared first on Cloudera Blog.

View Details

Technology Spotlight

The post Accelerating Projects in Machine Learning with Applied ML Prototypes appeared first on Cloudera Blog.

View Details

Activate machine learning and analytic use cases without sacrificing security

The post 10 Keys to a Secure Cloud Data Lakehouse appeared first on Cloudera Blog.

View Details

Demand for both entry-level and highly skilled tech talent is at an all-time high, and companies across industries and geographies are struggling to find qualified employees. And, with 1.1 billion jobs liable to be radically transformed by technology in the next decade, a “reskilling revolution” is reaching a critical mass. Already underrepresented populations like workers […]

The post Reskilling Against the Risk of Automation appeared first on Cloudera Blog.

View Details

Cloudera’s Global Week of Service - a week each year the business dedicates to doing good. Learn about the impact our employees made throughout the week, including the first ever, Unplug with Kindness campaign.

The post Global Week of Service 2022: 430+ Clouderans Engage for Good! appeared first on Cloudera Blog.

View Details

Information technology has been at the heart of governments around the world, enabling them to deliver vital citizen services, such as healthcare, transportation, employment, and national security. All of these functions rest on technology and share a valuable commodity: data.  Data is produced and consumed in ever-increasing amounts and therefore must be protected. After all, […]

The post Cybersecurity: A Big Data Problem appeared first on Cloudera Blog.

View Details

In this post I will demonstrate how Kafka Connect is integrated in the Cloudera Data Platform allowing users to manage and monitor their connectors in Streams Messaging Manager while also touching on security features such as role based access control and sensitive information handling.

The post Using Kafka Connect Securely in the Cloudera Data Platform appeared first on Cloudera Blog.

View Details

3 Key Takeaways from the TM Forum Digital Transformation World 2022

The post Public or On-Prem? Telco giants are optimizing the network with the Hybrid Cloud appeared first on Cloudera Blog.

View Details

Learn how Cloudera used the power of its own software to reduce its public cloud spend by $12 million per annum

The post Cloudera Uses CDP to Reduce IT Cloud Spend by $12 Million appeared first on Cloudera Blog.

View Details

From deep agricultural community roots grew the strong branches of persistence, learning, and finding better ways to do things.

The post #ClouderaLife Spotlight: Elias Avila, Sr. Staff Proactive Support Engineer appeared first on Cloudera Blog.

View Details

Building the next generation of products and solutions for a hybrid data world

The post Cloudera Innovation Accelerator Debuts appeared first on Cloudera Blog.

View Details

A recent VentureBeat article , “4 AI trends: It’s all about scale in 2022 (so far),” highlighted the importance of scalability. I recommend you read the entire piece, but to me the key takeaway – AI at scale isn’t magic, it’s data – is reminiscent of the 1992 presidential election, when political consultant James Carville […]

The post AI at Scale isn’t Magic, it’s Data – Hybrid Data appeared first on Cloudera Blog.

View Details

Innovation Accelerator Spotlight: Data teams can collaborate to streamline data transformation and analytics pipelines in the open data lakehouse using any engine, and in any form factor to produce high quality data that your business can trust.

The post Cloudera’s Open Data Lakehouse Supercharged with dbt Core(tm) appeared first on Cloudera Blog.

View Details

Opportunities abound with Data, Analytics and AI/ML

The post Does Cost Reduction Play a Role in Digital Transformation? appeared first on Cloudera Blog.

View Details

Inevitably in any production deployment, the number of kafka nodes required to maintain cluster changes. Balancing performance and cloud costs requires that administrators scale up/scale down accordingly.

The post Scaling Kafka Brokers in Cloudera Data Hub appeared first on Cloudera Blog.

View Details

Learn how to distribute you ML workload in Cloudera Machine Learning when your data is too big or your workload is too complex to run on a single machine

The post How to Distribute Machine Learning Workloads with Dask appeared first on Cloudera Blog.

View Details

While the word “data” has been common since the 1940s, managing data’s growth, current use, and regulation is a relatively new frontier.  Governments and enterprises are working hard today to figure out the structures and regulations needed around data collection and use. According to Gartner, by 2023 65% of the world’s population will have their […]

The post Data Governance and Strategy for the Global Enterprise appeared first on Cloudera Blog.

View Details

Technology Spotlight

The post Announcing GA of DataFlow Functions appeared first on Cloudera Blog.

View Details

Cloudera DataFlow for the Public Cloud (CDF-PC) is a cloud-native service for Apache NiFi within the Cloudera Data Platform (CDP). CDF-PC enables organizations to take control of their data flows and eliminate ingestion silos by allowing developers to connect to any data source anywhere with any structure, process it, and deliver to any destination using […]

The post Serverless NiFi Flows with DataFlow Functions: The Next Step in the DataFlow Service Evolution appeared first on Cloudera Blog.

View Details

A primer on how to reconcile the seemingly similar but different trends that confuse data teams struggling with solving the intractable “everything everywhere all at once” problem.

The post The Top Three Entangled Trends in Data Architectures: Data Mesh, Data Fabric, and Hybrid Architectures appeared first on Cloudera Blog.

View Details

Learn about the differences between PaaS and SaaS Data Lakehouses

The post The Benefits of an All-in-One Data Lakehouse appeared first on Cloudera Blog.

View Details

Make Underwriting More Science than Art

The post Improve Underwriting Using Data and Analytics appeared first on Cloudera Blog.

View Details

Why volunteer? Well, for Cloudera’s Barry Laide volunteering is all about being a part of team! Barry volunteers with an organization called Kerry Mountain Rescue (KMR), to provide first aid and rescue to those in need in the uplands of the southwest of Ireland. Despite being dangerous on occasion, the group camaraderie keeps him motivated!

The post #Clouderalife Volunteer Spotlight: Barry Laide appeared first on Cloudera Blog.

View Details

Introduction This blog is intended to serve as an ethics sheet for the task of AI-assisted comic book art generation, inspired by “Ethics Sheets for AI Tasks.” AI-assisted comic book art generation is a task I proposed in a blog post I authored on behalf of my employer, Cloudera. I’m a research engineer by trade […]

The post Ethics Sheet for AI-assisted Comic Book Art Generation appeared first on Cloudera Blog.

View Details

An Introduction

The post SCIM (System for Cross-domain Identity Management) appeared first on Cloudera Blog.

View Details

Apache Ozone is a distributed, scalable, and high-performance object store, available with Cloudera Data Platform (CDP), that can scale to billions of objects of varying sizes. It was designed as a native object store to provide extreme scale, performance, and reliability to handle multiple analytics workloads using either S3 API or the traditional Hadoop API. […]

The post A Flexible and Efficient Storage System for Diverse Workloads appeared first on Cloudera Blog.

View Details

with Mark Ramsey, PhD - Chief Data Officer

The post Demystifying Modern Data Platforms appeared first on Cloudera Blog.

View Details

A key part of business is the drive for continual improvement, to always do better. “Better” can mean different things to different organizations. It could be about offering better products, better services, or the same product or service for a better price or any number of things.  Fundamentally, to be “better” requires ongoing analysis of […]

The post Choose Both: Data Fabric and Data Lakehouse appeared first on Cloudera Blog.

View Details

The promise of a modern data lakehouse architecture Imagine having self-service access to all business data, anywhere it may be, and being able to explore it all at once. Imagine quickly answering burning business questions nearly instantly, without waiting for data to be found, shared, and ingested. Imagine independently discovering rich new business insights from […]

The post The Modern Data Lakehouse: An Architectural Innovation appeared first on Cloudera Blog.

View Details

New report from TM Forum highlights governance requirements in an era of hybrid infrastructure and ‘dynamic data architecture’.

The post New Practices in Data Governance and Data Fabric for Telecommunications appeared first on Cloudera Blog.

View Details

Cloudera’s open source licensing policies have evolved with the changing dynamics in open source innovation. For more information on Cloudera’s current policy, please contact OSSQuestions@cloudera.com. We are now well into 2022 and the megatrends that drove the last decade in data—The Apache Software Foundation as a primary innovation vehicle for big data, the arrival of […]

The post Large Scale Industrialization Key to Open Source Innovation appeared first on Cloudera Blog.

View Details

In the wake of the disruption caused by the world’s turbulence over the past few years, the telecommunications industry has come out reasonably unscathed. There remain challenges in workforce management, particularly in call centers, and order backlogs for fiber broadband and other physical infrastructure are being worked through. But digital transformation programs are accelerating, services […]

The post Modern Data Architecture for Telecommunications appeared first on Cloudera Blog.

View Details

Apache HBase has long been the database of choice for business-critical applications across industries. This is primarily because HBase provides unmatched scale, performance, and fault-tolerance that few other databases can come close to. Think petabytes of data spread across trillions of rows, ready for consumption in real-time. While application developers and database admins are well […]

The post Five Reasons for Migrating HBase Applications to the Cloudera Operational Database in the Public Cloud appeared first on Cloudera Blog.

View Details

Firms are burdened with tech debt and endless regulatory compliance, often leaving innovation last to receive the necessary budgets. Data-fuelled innovation requires a pragmatic strategy. This blog lays out some steps to help you incrementally advance efforts to be a more data-driven, customer-centric organization. Embrace incremental progress The financial sector’s evolution is unleashing myriad demands […]

The post Incremental Strategies to Move Your Data Strategy Forward Remove Obstacles to Unlock Possibilities in Financial Services appeared first on Cloudera Blog.

View Details

Data is the fuel that drives government, enables transparency, and powers citizen services. But while state and local governments seek to improve policies, decision making, and the services constituents rely upon, data silos create accessibility and sharing challenges that hinder public sector agencies from transforming their data into a strategic asset and leveraging it for […]

The post Breaking State and Local Data Silos with Modern Data Architectures appeared first on Cloudera Blog.

View Details

Cloudera Machine Learning (CML) is a cloud-native and hybrid-friendly machine learning platform. It unifies self-service data science and data engineering in a single, portable service as part of an enterprise data cloud for multi-function analytics on data anywhere. CML empowers organizations to build and deploy machine learning and AI capabilities for business at scale, efficiently […]

The post Building Custom Runtimes with Editors in Cloudera Machine Learning appeared first on Cloudera Blog.

View Details

CDP One brings industry’s first all-in-one lakehouse with fast and easy analytics

Technology Spotlight

The post Accelerate Analytics for All appeared first on Cloudera Blog.

View Details

Cloudera’s August Volunteer Spotlight is Thatiane Freire, account executive for the public sector, located in Brasília, Brazil, and one of the company’s Cloudera Cares Ambassadors.  Thatiane volunteers with a local organization called Casa de Caridade Inacio Daniel, which began with the goal to meet the day-to-day, foundational needs of the homeless community in Brasília. The […]

The post #Clouderalife Volunteer Spotlight: Thatiane Freire, Account Executive, Public Sector appeared first on Cloudera Blog.

View Details

Following the family tradition: be ever resilient, always curious, and never turn down an opportunity.

The post #ClouderaLife Spotlight: Preety Vatvani appeared first on Cloudera Blog.

View Details

We’ve come a long way since 1778 when George Washington’s spies gathered and shared military intelligence on the British Army’s tactical operations in occupied New York. But information broadly, and the management of data specifically, is still “the” critical factor for situational awareness, streamlined operations, and a host of other use cases across today’s tech-driven […]

The post How Universal Data Distribution Accelerates Complex DoD Missions appeared first on Cloudera Blog.

View Details

Cloudera has a strong track record of providing a comprehensive solution for stream processing. Cloudera Stream Processing (CSP), powered by Apache Flink and Apache Kafka, provides a complete stream management and stateful processing solution. In CSP, Kafka serves as the storage streaming substrate, and Flink as the core in-stream processing engine that supports SQL and […]

The post Getting Started with Cloudera Stream Processing Community Edition appeared first on Cloudera Blog.

View Details

The previous decade has seen explosive growth in the integration of data and data-driven insight into a company’s ability to operate effectively, yielding an ever-growing competitive advantage to those that do it well. Our customers have become accustomed to the speed of decision making that comes from that insight. Data is integral for both long-term […]

The post An Introduction to Disaster Recovery with the Cloudera Data Platform appeared first on Cloudera Blog.

View Details

With all of the buzz around cloud computing, many companies have overlooked the importance of hybrid data. Many large enterprises went all-in on cloud without considering the costs and potential risks associated with a cloud-only approach. The truth is, the future of data architecture is all about hybrid. Hybrid data capabilities enable organizations to collect […]

The post The future of data architecture is hybrid: choosing your hybrid-first data strategy starts at Cloudera Now 2022 appeared first on Cloudera Blog.

View Details

In June 2022, Cloudera announced the general availability of Apache Iceberg in the Cloudera Data Platform (CDP). Iceberg is a 100% open-table format, developed through the Apache Software Foundation, which helps users avoid vendor lock-in and implement an open lakehouse.  The general availability covers Iceberg running within some of the key data services in CDP, […]

The post How to Use Apache Iceberg in CDP’s Open Lakehouse appeared first on Cloudera Blog.

View Details

Z-order is an ordering for multi-dimensional data, e.g. rows in a database table. Once data is in Z-order it is possible to efficiently search against more columns. This article reveals how Z-ordering works and how one can use it with Apache Impala. In a previous blog post, we demonstrated the power of Parquet page indexes, […]

The post Speeding up Queries With Z-Order appeared first on Cloudera Blog.

View Details

Fine grained access control (FGAC) with Spark Apache Spark with its rich data APIs has been the processing engine of choice in a wide range of  applications from data engineering to machine learning, but its security integration has been a pain point. Many enterprise customers need finer granularity of control, in particular at the column […]

The post Applying Fine Grained Security to Apache Spark appeared first on Cloudera Blog.

View Details

How to fine-tune CDP Capacity Scheduler configs in Weight Mode after fs2cs migration

The post Fine-Tune Fair to Capacity Scheduler in Weight Mode appeared first on Cloudera Blog.

View Details

The telco and financial services industries within APAC have been at the forefront of adopting data and analytics to deliver differentiated products and services. With mature enterprise data strategies helping organizations achieve 5.97% higher profit growth,here’s how a modern data architecture and hybrid approach to data can propel them even higher.

The post Driving Success With a Modern Data Architecture and a Hybrid Approach in the Financial Services and Telco Industries appeared first on Cloudera Blog.

View Details

Co-author: Mike Godwin, Head of Marketing, Rill Data Cloudera has partnered with Rill Data, an expert in metrics at any scale, as Cloudera’s preferred ISV partner to provide technical expertise and support services for Apache Druid customers. We want Cloudera customers that rely on Apache Druid to know that their clusters are secure and supported […]

The post Simplify Metrics on Apache Druid With Rill Data and Cloudera appeared first on Cloudera Blog.

View Details

This month, Cloudera Cares is excited to spotlight Burt Wagner, senior solutions engineer from Alexandria, Virginia. Burt—who joined Cloudera earlier this year— volunteers regularly with the Boy Scouts of America. He started Scouting as an eight year old; it has always been an integral part of his life and something he now enjoys sharing with […]

The post #Clouderalife Volunteer Spotlight: Burt Wagner, Senior Solutions Engineer appeared first on Cloudera Blog.

View Details

Better to be prepared

The post Does Financial Crime Increase During a Recession? appeared first on Cloudera Blog.

View Details

In part 1 of this blog we discussed how Cloudera DataFlow for the Public Cloud (CDF-PC), the universal data distribution service powered by Apache NiFi, can make it easy to acquire data from wherever it originates and move it efficiently to make it available to other applications in a streaming fashion. In this blog we […]

The post Fraud Detection With Cloudera Stream Processing Part 2: Real-Time Streaming Analytics appeared first on Cloudera Blog.

View Details

Various methods to replicate HBase data including Replication Manager

The post Why Replicating HBase Data Using Replication Manager is the Best Choice appeared first on Cloudera Blog.

View Details

Cloudera has appointed Remus Lim as vice president of Asia Pacific and Japan, to drive adoption of the hybrid data platform across the region and support customers in their journey to become more data-driven. We’ve asked him to share his vision for Cloudera in APAC and reflect on his past few months since taking up […]

The post Scaling One Peak After Another appeared first on Cloudera Blog.

View Details

The need for data fabric As Cloudera CMO David Moxey outlined in his blog, we live in a hybrid data world. Data is growing and continues to accelerate its growth. It is changing in makeup and appearing in ever more places. Driving insight and value from it all is as much of an opportunity as […]

The post Beyond Data Fabrics: Cloudera Modern Data Architectures appeared first on Cloudera Blog.

View Details

Celebrating Pride Month, reflecting on work that still needs to be done

The post #ClouderaLife Spotlight: Autymn Harris and Jonathan Sanford appeared first on Cloudera Blog.

View Details

Cloudera Technology Spotlight

The post Supercharge Your Data Lakehouse with Apache Iceberg in Cloudera Data Platform appeared first on Cloudera Blog.

View Details

Solving the First-Mile Problem

The post Fraud Detection with Cloudera Stream Processing Part 1 appeared first on Cloudera Blog.

View Details

Much of the hype around big data and analytics focuses on business value and bottom-line impacts. Those are enormously important in the private and public sectors alike. But for government agencies, there is a greater mission: improving people’s lives. Data makes the most ambitious and even idealistic goals—like making the world a better place—possible. This […]

The post Making the World a Better Place with Data appeared first on Cloudera Blog.

View Details

Much of the hype around big data and analytics focuses on business value and bottom-line impacts. Those are enormously important in the private and public sectors alike. But for government agencies, there is a greater mission: improving people’s lives. Data makes the most ambitious and even idealistic goals—like making the world a better place—possible. This […]

The post Making the World a Better Place with Data appeared first on Cloudera Blog.

View Details

Regulators globally seek to reduce systemic risk in financial entities

The post Are You Ready for Cloud Regulations? appeared first on Cloudera Blog.

View Details

Cloudera customers run some of the biggest data lakes on earth. These lakes power mission critical large scale data analytics, business intelligence (BI), and machine learning use cases, including enterprise data warehouses. In recent years, the term “data lakehouse” was coined to describe this architectural pattern of tabular analytics over data in the data lake. […]

The post The Future of the Data Lakehouse – Open appeared first on Cloudera Blog.

View Details

In the second blog of the Universal Data Distribution blog series, we explored how Cloudera DataFlow for the Public Cloud (CDF-PC) can help you implement use cases like data lakehouse and data warehouse ingest, cybersecurity, and log optimization, as well as IoT and streaming data collection. A key requirement for these use cases is the […]

The post Build Hybrid Data Pipelines and Enable Universal Connectivity With CDF-PC Inbound Connections appeared first on Cloudera Blog.

View Details

Every large enterprise organization is attempting to accelerate their digital transformation strategies to engage with their customers in a more personalized, relevant, and dynamic way. The ability to perform analytics on data as it is created and collected (a.k.a. real-time data streams) and generate immediate insights for faster decision making provides a competitive edge for […]

The post Turning Streams Into Data Products appeared first on Cloudera Blog.

View Details

Introducing Michael Billau, customer operations engineer from Raleigh, North Carolina as Cloudera’s June Volunteer Spotlight! Michael volunteers with the Food Bank of Central and Eastern North Carolina. Learn more about his experience and the important work this organization is doing to meet the needs of its community.

The post #Clouderalife Volunteer Spotlight: Michael Billau appeared first on Cloudera Blog.

View Details

Customers’ Choice for Cloud Database Management Systems 

The post Cloudera Recognized as 2022 Gartner® Peer Insights™ appeared first on Cloudera Blog.

View Details

Here at Cloudera, we’re committed to helping make the lives of data practitioners as painless as possible. For data scientists, we continue to provide new Applied Machine Learning Prototypes (AMPs), which are open source and available on GitHub. These pre-built reference examples are complete end-to-end data science projects. In Cloudera Machine Learning (CML), you can […]

The post Cloudera’s Applied ML Prototype Catalog Continues to Grow appeared first on Cloudera Blog.

View Details

In the first blog of the Universal Data Distribution blog series, we discussed the emerging need within enterprise organizations to take control of their data flows. From origin through all points of consumption both on-prem and in the cloud, all data flows need to be controlled in a simple, secure, universal, scalable, and cost-effective way. […]

The post Streaming Edge Data Collection and Global Data Distribution appeared first on Cloudera Blog.

View Details

Humble beginnings, cultivating mentorship, and fostering mental health

The post #ClouderaLife Spotlight: Hassan Mirza appeared first on Cloudera Blog.

View Details

We live in a hybrid data world. In the past decade, the amount of structured data created, captured, copied, and consumed globally has grown from less than 1 ZB in 2011 to nearly 14 ZB in 2020. Impressive, but dwarfed by the amount of unstructured data, cloud data, and machine data – another 50 ZB. […]

The post The Future Is Hybrid Data, Embrace It appeared first on Cloudera Blog.

View Details

How to use the new Data Discovery and Visualization feature in Cloudera Machine Learning

The post The Power of Exploratory Data Analysis and Visualization for ML appeared first on Cloudera Blog.

View Details

The Need for a Universal Data Distribution Service

The post Moving Enterprise Data From Anywhere to Any System Made Easy appeared first on Cloudera Blog.

View Details

The Education Data Portal

The post Urban Institute Enacts Real Social and Policy Change Using Data appeared first on Cloudera Blog.

View Details

Cloudera Technology Spotlight

The post Tailored Support Designed for You appeared first on Cloudera Blog.

View Details

Observations from an ESG Europe event

The post Who is Ready for Climate Disclosures? appeared first on Cloudera Blog.

View Details

A career in transition, a community in need, navigating uncharted territory to solve for both

The post #ClouderaLife Spotlight: Margot Tien, Software Engineer appeared first on Cloudera Blog.

View Details

Deciding to adopt an AI-first strategy is the easy part. Figuring out how to implement it takes a little more effort. It requires a clear-eyed vision built around well-defined goals and a realistic execution plan. Being AI-first means setting up your organization for the future. By leveraging data, analytics, and automation, a company can gain […]

The post Becoming AI-First: How to Get There appeared first on Cloudera Blog.

View Details

How to fine-tune CDP Capacity Scheduler configs in relative mode after fs2cs migration

The post Fine-Tune Fair to Capacity Scheduler in Relative Mode appeared first on Cloudera Blog.

View Details

A guide to tune and troubleshoot performance of the Hive on Tez after upgrading to CDP

The post Optimizing Hive on Tez Performance appeared first on Cloudera Blog.

View Details

Fraud, waste, and abuse (FWA) in government is a constant, multi-billion dollar issue that challenges agency leaders at all levels and across all sectors, from healthcare to education to taxation to Social Security. The scope and scale of public spending—federal outlays alone were approximately $6.6 trillion in fiscal year 2020 according to the Congressional Budget […]

The post Winning With Data in the Fight Against Fraud, Waste, and Abuse appeared first on Cloudera Blog.

View Details

Artificial intelligence (AI) has been a focus for research for decades, but has only recently become truly viable. The availability and maturity of automated data collection and analysis systems is making it possible for businesses to implement AI across their entire operations to boost efficiency and agility. AI has the potential to transform operations by […]

The post AI-First Benefits: 5 Real-World Outcomes appeared first on Cloudera Blog.

View Details

Meet Lynne Montalbo, senior business systems analyst from Santa Clara, California, who exemplifies Cloudera’s culture of volunteering and giving by supporting Braven, a nonprofit organization, as a professional development mentor.

The post #Clouderalife Volunteer Spotlight: Lynne Montalbo! appeared first on Cloudera Blog.

View Details

Deliver Ultimate Deployment Freedom With CDP Hybrid Cloud

The post Choose Compliance, Choose Hybrid Cloud appeared first on Cloudera Blog.

View Details

Building autism awareness and taking action for family and community

The post #ClouderaLife Spotlight: Susana López Huertas, Senior Account Manager appeared first on Cloudera Blog.

View Details

Modern businesses have vast amounts of data at their fingertips and are acutely aware of how enterprise data strategies positively impact business outcomes. Despite this, only a handful of organisations interact with all stages of the data life cycle process to truly distill information that distinguishes future-ready businesses from the rest. Much potential remains untapped […]

The post A Window Into the Future of Data in Motion and What It Means for Businesses appeared first on Cloudera Blog.

View Details

Collaboration, not centralization is the new playbook

The post Data Is Now a Team Sport appeared first on Cloudera Blog.

View Details

Enterprise data will need smart connectivity and networked cloud

The post Telco 5G Returns Will Come from Enterprise Data Solutions appeared first on Cloudera Blog.

View Details

Behind the DoD and VA's multibillion-dollar move to a unified EHR system

The post A Lifetime of Data: Departments of Defense and Veterans Affairs Journey to Genesis appeared first on Cloudera Blog.

View Details

Enabling Proactive and Preventative Healthcare

The post The Sprint towards Digital Healthcare appeared first on Cloudera Blog.

View Details

Data holds incredible untapped potential for Australian organisations across industries, regardless of individual business goals, and all organisations are at different points in their data transformation journey with some achieving success faster than others.  To be successful, the use of data insights must become a central lifeforce throughout an organisation and not just reside within […]

The post From the Ground Up: The Truth About Data Innovation appeared first on Cloudera Blog.

View Details

How data driven by cloud computing is helping to model Earth’s future: from climate change effects to a new supercontinent

The post Data In Motion: NASA and Aurica appeared first on Cloudera Blog.

View Details

The term “AI-first” has received its share of attention lately, especially in the boardroom where strategies to gain a competitive advantage are always welcome. But before a company embarks on an AI-first strategy, it pays to understand what it is and how it will transform the organization. If you’re AI-first, that means you have figured […]

The post Becoming an AI-first Organization appeared first on Cloudera Blog.

View Details

Learn about our #Clouderalife Volunteer Spotlight Dániel Omaisz-Takács  a Budapest-based Visualization Specialist, volunteers with Hedgehog Foundation, an organization dedicated to helping wild eastern hedgehogs in their native Hungary.

The post #Clouderalife Volunteer Spotlight: Dániel Omaisz-Takács appeared first on Cloudera Blog.

View Details

As a public sector leader, you don’t need the value of data explained to you. You already understand its importance to your vital missions.  The challenge, rather, lies in locating data, streaming it, enriching it, and serving it, and then running analytics to maximize the value of the data that agencies already have—and the massive […]

The post Accelerate Your Mission at Cloudera Government Forum ’22 appeared first on Cloudera Blog.

View Details

This blog post was written by Elizabeth Howell, Ph.D as a guest author for Cloudera.  At a distance of a million miles from Earth, the James Webb Space Telescope is pushing the edge of data transfer capabilities. The observatory launched Dec. 25 2021 on a mission to look at the early universe, at exoplanets, and […]

The post Space-Based AI Shows the Promise of Big Data appeared first on Cloudera Blog.

View Details

Embedding Data and Analytics into the DNA of Life Sciences Organizations

The post Why Can’t we Advance Healthcare and Life Sciences this Fast all the Time? appeared first on Cloudera Blog.

View Details

Sometimes it takes a billion-dollar mistake to bring the murkier side of data ethics into sharp focus. Equifax found this out to their own cost in 2017 when they failed to protect the data of almost 150 million users globally. The catastrophic breach was bad enough on its own — but Equifax waited three months […]

The post Do Data Companies Need Chief Ethics Officers? appeared first on Cloudera Blog.

View Details

State and local governments generate and store enormous amounts of data essential to their ability to deliver citizen services. But how can they capitalize on all of their data to become engines of growth and innovation, empowering and enhancing their ability to provide services and better serve their communities? Data doesn’t arrive on the doorsteps […]

The post Hybrid Data Cloud Success for State and Local Governments appeared first on Cloudera Blog.

View Details

Over the past decade, Cloudera has matured to become a leading-edge technology company, supporting a diverse range of customers, across the globe. At Cloudera, we are passionate about helping our customers identify opportunities for innovation and growth, enabling them to accelerate their digital transformation, and aiding them to solve some of societies’ largest challenges. None […]

The post Best Workplaces in Ireland Award 2022 appeared first on Cloudera Blog.

View Details

Earlier this month, the multi-national carrier MTN announced a rebranding, and along with its logo refresh, announced that it was moving to focus on being a technology provider. The new look, “aligns with our evolution from a telecommunications company to a technology company,” said Nompilo Morafo, Chief Corporate Affairs officer at the company. Across APAC […]

The post The Telecommunications Service Provider Journey – From Telco to Techco appeared first on Cloudera Blog.

View Details

Please join us on March 24 for Future of Data meetup where we do a deep dive into Iceberg with CDP  What is Apache Iceberg? Apache Iceberg is a high-performance, open table format, born-in-the cloud that scales to petabytes independent of the underlying storage layer and the access engine layer. By being a truly open […]

The post 5 Reasons to Use Apache Iceberg on Cloudera Data Platform (CDP) appeared first on Cloudera Blog.

View Details

Introduction The Covid-19 pandemic has resulted in an unprecedented global economic landscape that is dominated by loose monetary policies, low borrowing costs and influx of capital in the equity markets. Against that backdrop, Mergers and Acquisitions (M&A) activity has surged since 2021 as companies are trying to take advantage of the current environment and adapt […]

The post How to Accelerate Value from Merger and Acquisition Strategies with Cloudera Data Platform (CDP) appeared first on Cloudera Blog.

View Details

Data Science tools, algorithms, and practices are rapidly evolving to solve business problems on an unprecedented scale. This makes data science one of the most exciting fields to be in. As exciting as it is, practitioners face their fair share of challenges. There are well-known barriers that slow down predictive modeling or application development. Finding […]

The post One Line Away from your Data appeared first on Cloudera Blog.

View Details

Data is the true currency of the digital age, and it plays an indispensable role in defining and accelerating the mission of Government agencies.  Every level of government is awash in data (both structured and unstructured) that is perpetually in motion. It is constantly generated – and always growing in volume – by an ever-growing […]

The post Accelerate Agency Missions with Data in Motion appeared first on Cloudera Blog.

View Details

The global pandemic has ushered in a new wave of economics. Driven by the rapid convergence of changing circumstances, data, automation and Artificial Intelligence (AI), businesses today have to contend with a whirlwind of internal and external pressures. Companies are experiencing pressure from growing customer demands, and amidst a global talent shortage, a workforce no […]

The post Planet and Profit: Reshaping Business Priorities for Good appeared first on Cloudera Blog.

View Details

Each year, various departments and teams across the organization welcome early talent to embark on internships which allow them to kickstart their careers within the technology and big data industries. One of those early talent interns is Trang Luong, who worked within the APAC Inside Sales team earlier this year for a six-months, supporting the […]

The post My Internship Experience with Cloudera appeared first on Cloudera Blog.

View Details

The world cannot ignore the horrific invasion of Ukraine and the plight of the Ukrainian people, who are facing death and devastation in the defense of their country. Our Cloudera team members and their families in Ukraine have been impacted in ways we cannot imagine, and their safety is our top priority.  Cloudera – and […]

The post Cloudera Statement Regarding Ukraine appeared first on Cloudera Blog.

View Details

With the launch of CDP Public Cloud 7.2.14, Cloudera Streams Messaging for Data Hub deployments has gotten some powerful new features! In this release, the Streams Messaging templates in Data Hub will come with Apache Kafka 2.8 and Cruise Control 2.5 providing new core features and fixes. KConnect has been added and gains additional capabilities […]

The post New Features in Cloudera Streams Messaging for CDP Public Cloud 7.2.14 appeared first on Cloudera Blog.

View Details

In celebration of International Women’s Day, we surveyed some of the most inspiring female executives from across the data industry to share with us their thoughts on this year’s theme - #BreaktheBias.

The post Women Leaders in Data Discuss Breaking Bias on International Women’s Day appeared first on Cloudera Blog.

View Details

Note: this publication has been updated from original blog post published on June 3, 2021 Cloudera Support’s cluster validations proactively identify known problem signatures contained in customers’ diagnostic data with the goal of increasing cluster health, performance, and overall stability. Cluster validations are included in a customer’s enterprise subscription at no additional cost. All customers […]

The post Validations – Cloudera Support’s Predictive Alerting Program appeared first on Cloudera Blog.

View Details

Over the past decade, Cloudera has matured to become a leading-edge technology company, supporting a diverse range of customers, across the globe. At Cloudera, we are passionate about helping our customers identify opportunities for innovation and growth, enabling them to accelerate their digital transformation, and aiding them to solve some of societies’ largest challenges. None […]

The post Best Workplaces for Women in Ireland 2022 appeared first on Cloudera Blog.

View Details

Bias is everywhere.  We’re surrounded by it.  And it’s natural. We are alive today as a species because of biases.  But it has a tangible impact on our personal and professional lives. Biases shape us and our experience.  As primary caregivers, women have felt the impact of biases and expectations more keenly during the pandemic. […]

The post #BreakTheBias: It’s a Journey appeared first on Cloudera Blog.

View Details

From the conference room to the dojo, walking the collaborative path makes all the difference

The post #ClouderaLife Spotlight: Vicki Zingiris appeared first on Cloudera Blog.

View Details

This blog features how bad data governance practices lead to data breaches, lawsuits, and regulatory fines — and no enterprise is immune. 

The post Why Data Governance Is Crucial for All Enterprise-Level Businesses appeared first on Cloudera Blog.

View Details

Using Agent-based Modeling to create Stress Test Scenarios

The post Manage the Demand of Stress Testing in Financial Services appeared first on Cloudera Blog.

View Details

Apache Impala is used today by over 1,000 customers to power their analytics in on premise as well as cloud-based deployments. Large user communities of analysts and developers benefit from Impala’s fast query execution, helping them get their work done more effectively. For these users performance and concurrency are always top of mind.  An important […]

The post Memory Optimizations for Analytic Queries in Cloudera Data Warehouse appeared first on Cloudera Blog.

View Details

In the latest installment of the EMEA Influential Women in Data webinar series, we welcomed Shirley Collie, Chief Health Analytics Actuary at Discovery Health to discuss everything from how the pandemic has impacted working, to the opportunities within data, and the importance of intentionality. A data-driven organization Shirley knows better than most about the impact […]

The post The Power and Possibility of Intentionality appeared first on Cloudera Blog.

View Details

Supporting the transformation from “telco” to “techco” in telecommunications companies to benefit from the right data strategy

The post Cloudera: Enabling the Cloud-Native, Data-Driven Techco appeared first on Cloudera Blog.

View Details

Over the past decade, the successful deployment of large scale data platforms at our customers has acted as a big data flywheel driving demand to bring in even more data, apply more sophisticated analytics, and on-board many new data practitioners from business analysts to data scientists. This unprecedented level of big data workloads hasn’t come […]

The post Introducing Apache Iceberg in Cloudera Data Platform appeared first on Cloudera Blog.

View Details

Why leading with empathy is the key when working across cultures

The post Leadership in 2022: Focus on Empathy appeared first on Cloudera Blog.

View Details

A step by Step Guide

The post Upgrade Hortonworks Data Platform (HDP) to Cloudera Data Platform (CDP) Private Cloud Base appeared first on Cloudera Blog.

View Details

A look at the five areas of machine Learning Model Governance with Cloudera Machine Learning (CML)

The post Of Muffins and Machine Learning Models appeared first on Cloudera Blog.

View Details

Introducing Cloudera Data Engineering in CDP Private Cloud 1.3

The post Make the leap to Hybrid with Cloudera Data Engineering appeared first on Cloudera Blog.

View Details

An introduction to Machine Learning and the Data Science Lifecycle

The post Getting Started with Machine Learning appeared first on Cloudera Blog.

View Details

After the launch of Cloudera DataFlow for the Public Cloud (CDF-PC) on AWS a few months ago, we are thrilled to announce that CDF-PC is now generally available on Microsoft Azure, allowing NiFi users on Azure to run their data flows in a cloud-native runtime.  With CDF-PC, NiFi users can import their existing data flows […]

The post Announcing the GA of Cloudera DataFlow for the Public Cloud on Microsoft Azure appeared first on Cloudera Blog.

View Details

Using creativity to find solutions for the global workplace, employee diversity, and beyond

The post #ClouderaLife Spotlight: Marque Blackman, Director of Global Workplace appeared first on Cloudera Blog.

View Details

Cloudera has been recognized as a Visionary in 2021 Gartner® Magic Quadrant™  for Cloud Database Management Systems (DBMS) and for the first time, evaluated CDP Operational Database (COD) against the 12 critical capabilities for Operational Databases.  Overall, Gartner recognized 20 vendors for the Magic Quadrant of which 16 were evaluated in the 2021 Gartner Critical […]

The post Gartner® Recognizes Cloudera in Critical Capabilities for Cloud Database Management Systems for Operational Use Cases appeared first on Cloudera Blog.

View Details

An Overview of the HBase to CDP Operational Database (COD) Migration Process

The post HBase to CDP Operational Database Migration Overview appeared first on Cloudera Blog.

View Details

Every Snowflake is Unique, but Some Must Be Less Unique than Others

The post The Most Unique Snowflake appeared first on Cloudera Blog.

View Details

The role of Business Strategy, Regulation and Data Access in Defining your Enterprise Data Cloud

The post Why Choose a Hybrid Data Cloud in Financial Services? appeared first on Cloudera Blog.

View Details

Embrace Location Data to Enhance Insurance Offerings and Drive Loyalty

The post Customizing Personal Lines Insurance with Location Data appeared first on Cloudera Blog.

View Details

According to 451 Research, 96% of enterprises are actively pursuing a hybrid IT strategy. Modern, real-time businesses require accelerated cycles of innovation that are expensive and difficult to maintain with legacy data platforms. Cloud technologies and respective service providers have evolved solutions to address these challenges.  The hybrid cloud’s premise—two data architectures fused together—gives companies […]

The post 96 Percent of Businesses Can’t Be Wrong: How Hybrid Cloud Came to Dominate the Data Sector appeared first on Cloudera Blog.

View Details

It's Time to Look Forward

The post Make the Leap to AI Driven Data Applications appeared first on Cloudera Blog.

View Details

Recognizing three organizations who have addressed some of the most challenging issues affecting society and the planet

The post Data for Good appeared first on Cloudera Blog.

View Details

The Four Security Engineering Best Practice Pillars: Perimeter, Data, Access and Visibility.

The post Security Reference Architecture Summary for Cloudera Data Platform appeared first on Cloudera Blog.

View Details

Gartner® recognized Cloudera in three recent reports – Magic Quadrant™ for Cloud Database Management Systems (DBMS), Critical Capabilities for Cloud Database Management Systems for Analytical Use Cases and Critical Capabilities for Cloud Database Management Systems for Operational Use Cases. Our position as a Visionary in the Gartner Magic Quadrant for Cloud DBMS market speaks to […]

The post Gartner® Magic Quadrant™ for Cloud Database Report Recognizes Cloudera as a Visionary appeared first on Cloudera Blog.

View Details

We are excited to announce the release of Cloudera Streaming Analytics (CSA) 1.6 for CDP Private Cloud Base. With this release, we build on the foundation on 1.4 and 1.5 – with a number of fixes, enhancements, and features. Starting with this release, we now have an aligned release cycle for CSA Community Edition (CE). […]

The post Cloudera Streaming Analytics 1.6 Release Notes appeared first on Cloudera Blog.

View Details

Key uses cases showing how data can help drive societal change

The post How Data is Helping Organizations to Improve the Employee Lifecycle appeared first on Cloudera Blog.

View Details

Since the release of Cloudera Data Engineering (CDE)  more than a year ago, our number one goal was operationalizing Spark pipelines at scale with first class tooling designed to streamline automation and observability.   In working with thousands of customers deploying Spark applications, we saw significant challenges with managing Spark as well as automating, delivering, […]

The post Cloudera Data Engineering 2021 Year End Review appeared first on Cloudera Blog.

View Details

The importance of including centralized data management, security, and governance into data projects from the start

The post Recognizing Organizations Leading the Way in Data Security & Governance appeared first on Cloudera Blog.

View Details

Manoj Shanmugasundaram has been with Cloudera for 5 and a half years bringing his talents to our Solutions Engineering team.  As a Principal Solutions Engineer, he says his core responsibility is “to take Cloudera’s latest and greatest technology and meet a customer’s complex business requirements, across the data lifecycle, on any cloud or the datacenter.” […]

The post #ClouderaLife Spotlight: Manoj Shanmugasundaram – Principal Solutions Engineer appeared first on Cloudera Blog.

View Details

Hybrid cloud data services and analytics is becoming a new enterprise standard as much of the global workforce has switched to a hybrid way of working. To implement hybrid data strategies, IT leaders need tactics to overcome internal and external hurdles.

The post How To Overcome Hybrid Cloud Migration Roadblocks appeared first on Cloudera Blog.

View Details

Data is the missing puzzle piece for organizations to couple DEI initiatives with actionable insights, identify areas where impact will be highest, and build metrics for greater accountability.

The post Why Company Data Strategies Are Indelibly Linked with DEI appeared first on Cloudera Blog.

View Details

Some of the most successful people I know listen more than they talk. We need leaders who focus on building the organizations they run, not their own egos.

The post It’s Time to Listen More to Your Employees! appeared first on Cloudera Blog.

View Details

Global use cases for enterprise-scale machine learning and industrialized AI

The post AI and ML: No Longer the Stuff of Science Fiction appeared first on Cloudera Blog.

View Details

On December 10th 2021, the Apache Software Foundation released a security advisory for Apache Log4j 2.0-2.14. This vulnerability is critical and is rated 10 out of 10 on the CVSS 3.1 scoring scale.

The post Cloudera Response to CVE-2021-44228 appeared first on Cloudera Blog.

View Details

Use cases showing how your data strategy can create massive business impact

The post Driving Industry Transformation Through the Use of Data appeared first on Cloudera Blog.

View Details

CDP Operational Database allows developers to use Amazon Simple Storage Service (S3) as its main persistence layer for saving table data.

The post Delivering High Performance for Cloudera Data Platform Operational Database (HBase) When Using S3 appeared first on Cloudera Blog.

View Details

Delays in implementing a data strategy mean less time for the organization to harness critical business insights. Results from our global study of business and IT decision makers show that organizations which had data strategies in place earlier also reaped the benefits sooner.

The post The Best Time to Kickstart Your Data Strategy Was Yesterday, the Next Best Time Is Now appeared first on Cloudera Blog.

View Details

Top use cases of cloud data strategies

The post How Hybrid and Cloud-Based Architectures are Unlocking the Power of Data appeared first on Cloudera Blog.

View Details

Increase in AI adoption means higher risks for data bias and misinformation. What are the challenges in building ethical AI systems that organisations should be aware of?

The post In AI we trust? Why we Need to Talk About Ethics and Governance (part 2 of 2) appeared first on Cloudera Blog.

View Details

Cloudera secures your data by providing encryption at rest and in transit, multi-factor authentication, Single Sign On, robust authorization policies, and network security.

The post Create your Private Data Warehousing Environment Using Azure Kubernetes Service appeared first on Cloudera Blog.

View Details

Global survey of IT and business executives reveals how a mature data strategy relates to business performance and resilience. Organizations fall under one of four categories when it comes to enterprise data maturity, and they need to be aware of how to address multi-dimensional challenges of a hybrid future.

The post The Cloudera Enterprise Data Cloud Maturity Report: Uncovering progressive steps towards a hybrid future appeared first on Cloudera Blog.

View Details

Use cases from organizations that work with multiple parts of the data lifecycle to collect, enrich, report, serve, and predict

The post Connecting the Data Lifecycle appeared first on Cloudera Blog.

View Details

Increase in AI adoption means higher risks for data bias and misinformation. With the sheer volume of data that today’s digital economy is expected to produce, the consequences can be far-reaching. We examine some of these.

The post In AI we Trust? Why we Need to Talk about Ethics and Governance (part 1 of 2) appeared first on Cloudera Blog.

View Details

At Cloudera, supporting our customers through their complete data journey also means providing access to game-changing technologies with trusted partners like Amazon Web Services (AWS)

The post Empowering Digital Innovation Through Data and the Public Cloud Together with Amazon Web Services appeared first on Cloudera Blog.

View Details

The Equality Committee dreamed up a vision to work with youth in underserved areas to provide access to experiences and education that they otherwise would not have. This blog describes the journey in executing on that vision.

The post How Cloudera Is Opening Doors for Underserved Youth appeared first on Cloudera Blog.

View Details

How to create an operational database

The post Getting Started with Cloudera Data Platform Operational Database (COD) appeared first on Cloudera Blog.

View Details

How the Cloudera Data Platform helps organizations overcome scalability challenges

The post Addressing the Three Scalability Challenges in Modern Data Platforms appeared first on Cloudera Blog.

View Details

Machine Learning: Equal Parts Business Opportunity and Operations Conundrum

The post Make Your Models Matter: What It Takes to Maximize Business Value from Your Machine Learning Initiatives appeared first on Cloudera Blog.

View Details

Cloudera’s annual customer awards showcasing the most cutting-edge innovative data projects across industries around the world

The post Celebrating Data Superheroes: The 2021 Data Impact Awards Winners appeared first on Cloudera Blog.

View Details

It’s no secret that Data Scientists have a difficult job. It feels like a lifetime ago that everyone was talking about data science as the sexiest job of the 21st century. Heck, it was so long ago that people were still meeting in person! Today, the sexy is starting to lose its shine. There’s recognition […]

The post New Applied ML Prototypes Now Available in Cloudera Machine Learning appeared first on Cloudera Blog.

View Details

Fully Serverless Solution for Running NiFi Flows

The post NiFi as a Function in DataFlow Service appeared first on Cloudera Blog.

View Details

Challenges and Opportunities Data Presents to AI

The post The Rise of Unstructured Data appeared first on Cloudera Blog.

View Details

Simplifying Operational Database and Replication Manager for the Public Cloud

The post Defining Simplicity for Enterprise Software as “a 10 Year Old Can Demo it” appeared first on Cloudera Blog.

View Details

Why switch to Ranger?

The post Sentry to Ranger – A Concise Guide appeared first on Cloudera Blog.

View Details

CDP offers federal agencies a platform-based approach to strengthen their response to cyberthreats

The post Cloudera Addresses Executive Order on Improving U.S. Cybersecurity with Data Analytics appeared first on Cloudera Blog.

View Details

An unexpected journey of adventure, leadership, and giving back

The post #ClouderaLife Spotlight: Paul Wooding, Senior Regional Sales Director appeared first on Cloudera Blog.

View Details

How data literacy can encourage children worldwide to transform from passive spectators of technology disruption to active participants of positive change in their local communities and the world.

The post A Fresh Squeeze on Data appeared first on Cloudera Blog.

View Details

This blog demonstrate how easy it is to adapt a script built with popular CPU based Python libraries, like Pandas and Scikitlearn, to instead run with GPU based Python libraries, like cuDF and cuML.

The post Switching from CPUs to GPUs for NYC Taxi Fare Predictions with NVIDIA RAPIDS appeared first on Cloudera Blog.

View Details

Data governance: three steps to success

The post Accelerate Insight with Proactive Data Governance Practices appeared first on Cloudera Blog.

View Details

People who care about not only their own success but that of their colleagues and the business

The post Cloudera Ireland Center of Excellence Certified as a Great Place to Work appeared first on Cloudera Blog.

View Details

From navigating treacherous waters to educating customers, it has always been about executing on a plan

The post #ClouderaLife Spotlight: William Dailey, Senior Technical Instructor appeared first on Cloudera Blog.

View Details

How CDP Operational Database can deliver high availability for your applications when running on multiple availability zones in AWS

The post High Availability (Multi-AZ) for CDP Operational Database appeared first on Cloudera Blog.

View Details

Commercial Lines has more data available than ever before to understand and tune its underwriting practices

The post Commercial Lines Insurance- the End of the Line for All Data appeared first on Cloudera Blog.

View Details

Meet Krishna Birla

The post #ClouderaLife Spotlight: Krishna Birla, Software Engineer appeared first on Cloudera Blog.

View Details

How to use data visualization to avoid bad candy hot spots and maximize supply and demand imbalances

The post The Ultimate Map to finding Halloween candy surplus appeared first on Cloudera Blog.

View Details

World Mental Health Day took place earlier this month. Many came forward to share their personal struggles with mental health to raise awareness and reduce the stigma surrounding these issues. The pressures of the pandemic may have exacerbated some deep-seated problems among some of us, which has led us to place greater emphasis on mental health.

The 2021 Global WellBeing report by professional services firm, AON, revealed that mental health and working environment are ranked among the top three employee wellbeing issues globally. Employers are faced with high turnovers amidst “The Great Resignation”. Resignations in the tech industry are among the highest at a 4.5% increase from the previous year due to increased workloads and burnout induced by a demand surge from the pandemic.

As leaders, we cannot afford to ignore the impact of overall employee wellbeing as well as engagement on an organisation’s growth and stability. We must be invested in the welfare of our teams and how they perceive their future in the organisation.

Leaders need to create a culture that values wellbeing A company’s culture is a deciding factor on whether a concerted strategy for employee wellbeing will succeed, and this is more often than not influenced by its senior leadership. The crucial step is to lean in.

Are you someone who leads from an ivory tower or from the frontlines? This fundamental question helps set the tone for how you will steer team dynamics. The challenges of remote working with dispersed teams have been a test of leadership. Frequency of project updates may have been reduced and managers are cautious to avoid coming across as micromanaging employees, unintentionally pushing them to the brink of exhaustion. In these circumstances, empathy needs to underscore every interaction.

Lean in and not away One way for leaders to demonstrate empathy is to listen often. At times, it is necessary to listen more than you talk.

The intrinsic value of a team is that its members bring different perspectives to the table because each of us as individuals are prone to blind spots.

Exhibiting openness invites team members to contribute actively, and they would be more likely to do so when the head of the team is attentive to the various angles surrounding a challenge before arriving at a solution.

Maintaining regular interactions with your teams sends a signal that you value their work. At the same time, consistent support from the leadership team helps teams chart the path ahead or resolve obstacles smoothly. Gallup estimates that managers account for 70% or more of the variance in employee engagement scores across business units. This illustrates just how significant leaders are in keeping teams connected, starting by leaning in instead of away from problems. Knowing that their work is valued keeps teams engaged and encourages them to continue their efforts. This also provides clarity on how each role aligns with the whole company’s mission, for employees to strive towards that shared vision of success such as by being more innovative and taking the initiative more often.

That being said, leaders should take a measured approach and refrain from jumping right in every single time the team encounters an issue. Ultimately, the aim is to empower teams to take ownership of their work and to take pride in what they have accomplished. Learning presents opportunities for teams to grow and achieve excellence in the long run.

Emphasise commitment in times of change In times of change and uncertainty, getting in the trenches with the team to understand what they are going through reiterates your commitment to common goals. Being proactive by offering perspective and mentorship goes a long way in building trust, instead of being reactive after trouble arises. Employees who report high trust have an average capacity for change that is 2.6 times greater than those with low trust.

Building trust naturally takes time, and tying in with the earlier point about engaging employees, being consistent about offering support plays a major role in reaching desired levels. Showing teams that you want to get to the finishing line together by dedicating your own time to the project imparts a greater sense of meaning to the task at hand. Having established a strong rapport, team members would be assured of whether their concerns would be heard and consequentially readily share information about issues at an early stage to be promptly addressed.

Lead by example I always remember this piece of advice,

“Don’t be the only person in the meeting who walks away with no action items.”

Drive the momentum by walking the talk. A leader’s mindset and actions diffuses to the rest of the team. Giving a clear direction shapes expectations for employees.

In doing so, an old mentor of mine showed me that a leader should not ask the team to do something they would not do themselves. And that increased my respect for him even more.

Too many leaders today don’t believe in being hands on, perhaps because they might believe they are above it or don’t want to do the work. Roll up your sleeves and do the work to be more attuned to what the team is facing, which then puts you in a better position to offer advice when the situation calls for it. In the same vein, recognising employees’ achievements when they have done well is a morale booster.

The pandemic has permanently altered the way we work and the way we perceive workplaces, hopefully for the better. Where once financial growth was the main, if not the sole priority, leaders are coming to the realisation that other non-financial indicators like employee well-being and engagement can propel or impair an organisation’s growth trajectory, as well as its relationships with critical stakeholders. It is heartening to see that 48% of employees globally say that organisational culture has changed and improved during the course of the pandemic. The question to ask ourselves now is, “Can we make empathy and compassion commonplace in the “new normal”?

Learn more about working at Cloudera and our commitment to diversity, equality and inclusion here.

The post Are you Somebody Who Leads from the Ivory Tower or from the Front Lines? appeared first on Cloudera Blog.

View Details

With the launch of the Cloudera Public Cloud 7.2.12, the Streams Messaging for Data Hub deployments have gotten some interesting new features! From this release, Streams Messaging templates will support scaling with automatic rebalancing allowing you to grow or shrink your Apache Kafka cluster based on demand. Another notable item is that Streams Replication Manager (SRM) will now support multi-cluster monitoring patterns and aggregate replication metrics from multiple SRM deployments into a single viewable location in Streams Messaging Manager (SMM.) Last but not least is Apache Atlas and Schema Registry (SR) Integration, now you will be able to view Kafka topic schemas in Atlas letting you navigate data lineage from consumers and producers and see the schema of the topic they use without having to navigate back to the SR UI.

Kafka Scaling Newly deployed Light and Heavy Duty Streams Messaging templates scaling up or down their Kafka brokers will now be possible with the addition of Cruise Control. Cruise Control will automatically rebalance the partition replicas on the cluster making use of the newly added brokers in the event of an up scale, or down scaling will move replicas off the hosts that are targeted to be decommissioned. It should be noted that at the time of writing this, clusters that upgrade to 7.2.12 will not have the ability to downscale and upscaling will not automatically rebalance, requiring the old school manual json assignment files to be created and run by operators.

• Clusters newly provisioned with 7.2.12 or higher
  • Support up and downscale operations
  • Support automatic partition rebalancing with Cruise Control
• Clusters upgraded to 7.2.12 or higher
  • Support upscale operations only
  • Partitions must be moved manually to any newly provisioned brokers after upscaling is finished.