Secrets of Data Analytics Leaders: Recent Episodes
Eckerson Group
Listen to data and analytics leaders share the secrets of their success. Wayne Eckerson, long-time global thought leader interviews guests who run data and analytics programs at Fortune 2000 organizations around the world. Tune in to stay abreast of the latest technologies, techniques, and trends in our fast-paced industry.
Discover best practices to scale Power BI effectively—improving performance, governance, and self-service across enterprise environments.Published at:https://datalere.com/articles/power-bi-best-practices-from-the-field-lessons-from-enterprise-deployments
Chief Data Officers are expanding data sharing to drive productivity, innovation, and better decisions. Data marketplaces simplify access by connecting users to ready-to-use, high-value data through a centralized, self-service platform.Published at:https://www.eckerson.com/articles/how-data-product-marketplaces-enable-seamless-data-consumption-and-generate-value
All data management activities—whether internally or externally focused—should either reduce costs or grow earnings.Published at:https://www.eckerson.com/articles/book-review-how-to-make-money-with-data
This blog offers five guiding principles to help CIOs, CDOs, and team leaders optimize hybrid data environments.Published at:https://www.eckerson.com/articles/balancing-act-five-principles-to-optimize-hybrid-cloud-environments
This blog defines the governance requirements that streaming data pipelines must meet to make artificial intelligence/machine learning (AI/ML) initiatives successful.Published at:https://www.eckerson.com/articles/streaming-data-governance-three-must-have-requirements-to-support-ai-ml-innovation
Anthony Cosgrove, Co-founder at Harbr, tells us how successful data products focus on clear value propositions, tailored placement, intuitive packaging, and appropriate permissions.Published at:https://www.eckerson.com/articles/what-is-a-data-product
This blog, the second in a series, explores the mix of infrastructure types that support modern AI.Published at:https://www.eckerson.com/articles/cloud-on-prem-hybrid-oh-my-where-ai-adopters-host-their-projects-and-why
Despite $180 billion spent on big data tools and technologies, poor data quality remains a significant barrier for businesses, especially in achieving Generative AI goals.Published at:https://www.eckerson.com/articles/poor-data-quality-is-a-full-blown-crisis-a-2024-customer-insight-report
This blog discusses how GenAI and natural language processing are transforming SQL query generation, allowing non-technical users to access and query data easily.Published at:https://www.eckerson.com/articles/from-4gl-to-genai-how-sql-automation-has-evolved
This article by Piotr Czarnas, founder of DQOps, outlines a proven, team-based approach to tackling persistent issues like invalid data, delayed reporting, and inconsistent formats.Published at:https://www.eckerson.com/articles/overcoming-the-challenge-of-low-data-quality
This article explores how Product Information Management (PIM) helps manufacturers and retailers manage rich product data through phased attribution and collaborative workflows.Published at:https://www.eckerson.com/articles/keys-to-succeeding-in-your-product-information-management-pim-journey
As cloud adoption accelerates, not all analytics workloads are heading in the same direction. This blog explores three strategic options for data and IT leaders.Published at:https://www.eckerson.com/articles/are-you-cloud-bound-the-case-for-migration-repatriation-or-keeping-your-analytics-projects-on-premises
This article breaks down the evolving landscape of AI/ML platforms, from AutoML to full-stack AI workbenches, and provides a structured tool evaluation framework to cut through vendor ambiguity.Published at:https://www.eckerson.com/articles/the-ai-ml-tool-evaluation-template-a-guide-to-smarter-selection
This guide provides a step-by-step framework to assess vendors, align priorities, and make informed decisions about enterprise data and analytics tools.Published at:https://www.eckerson.com/articles/the-buyer-s-guide-to-selecting-the-right-enterprise-data-analytics-tool
This blog, the second in a series, recommends six criteria to help data leaders evaluate tools for managing streaming data pipelines.Published at:https://www.eckerson.com/articles/ultimate-guide-to-streaming-data-pipelines-six-criteria-to-evaluate-and-select-the-right-tool
This blog defines streaming data, explains why companies need it, and explores how streaming data pipelines feed multi-faceted GenAI applications.Published at:https://www.eckerson.com/articles/why-and-how-streaming-data-drives-the-success-of-generative-ai
Change is inevitable, but adoption is key. From AI/ML tools to leadership shifts, success depends on aligning people, not just technology.Published at:https://www.eckerson.com/articles/managing-change-in-the-age-of-ai-the-head-heart-and-herd-framework
This blog covers integrating SAP and third-party systems to build a unified data foundation for analytics and AI in conversational use cases.Published at:https://www.eckerson.com/articles/analytics-and-ai-for-sap-environments-build-a-unified-data-foundation-to-drive-advanced-use-cases
This blog recommends four questions to help data and AI leaders compare homegrown and commercial options for retrieval augmented generation.Published at:https://www.eckerson.com/articles/build-or-buy-rag-four-questions-to-guide-your-approach-to-retrieval-augmented-generation-for-genai
What will change next year? In the age of generative AI, the answer is simple: Everything!Published at:https://www.eckerson.com/articles/predictions-2025-everything-is-about-to-change
Exploring data observability’s limits in data migration, integrity audits, and the need for specialized tools for reliability.Published at:https://www.eckerson.com/articles/the-shiny-allure-of-data-observability-its-limits-in-data-migration-integrity-audits-and-certification
A strategic approach to data monetization covering key processes, risks, and organizational shifts needed for success.Published at: https://www.eckerson.com/articles/unlocking-data-monetization-success
A fresh perspective on data architecture, advocating for an 'anti-monolith' approach inspired by engineering best practices.Published at: https://www.eckerson.com/articles/good-data-architecture-anti-monolith
Many practitioners view data mesh and data fabric as mutually exclusive approaches to data strategy. However, these paradigms complement each other. Data mesh focuses on decentralization and autonomy; Data fabric ensures centralized integration and governance. Let’s dive into how blending elements of both can offer flexibility and control to create the right fit for your organization’s data strategy.Published at: https://www.eckerson.com/articles/blending-data-mesh-and-data-fabric-crafting-a-balanced-data-strategy-2118cd34-e463-4468-b150-bdaf9e1c541d
As organizations grapple with data spread across various storage locations, solutions like Coginiti Hybrid Query offer a much-needed alternative to fragmented tools.Published at:https://www.eckerson.com/articles/a-novel-approach-for-reducing-cloud-data-warehouse-expenses-from-coginiti
This blog post explores the evolving landscape of data catalogs, highlighting ten key market trends driving the adoption of next-generation solutions.Published:https://www.eckerson.com/articles/ten-key-market-trends-in-next-generation-data-catalogs
Data teams must filter, blend, and refine raw data inputs to create the high-octane fuel that drives innovation with artificial intelligence and machine learning (AI/ML).Published at:https://www.eckerson.com/articles/refining-the-right-fuel-how-data-integration-drives-the-ai-ml-model-lifecycle
With numerous data catalog options available, all claiming to be the best, how do you make an informed decision without exhaustive research?Published at:https://www.eckerson.com/articles/unveiling-the-future-of-data-catalogs
This blog describes the need for data teams to establish a flexible yet well-governed data architecture to support dynamic AI/ML projects.Published at:https://www.eckerson.com/articles/multi-style-data-integration-for-ai-ml-three-use-cases
Many data leaders want to implement self-service, but don’t realize that they first have to implement the right architecture, governance, operating model, project delivery approach, data, and change management plan.Published at:https://www.eckerson.com/articles/self-service-is-the-outcome-not-the-driver-of-a-data-driven-organization
Explore the essential characteristics to choose the right conversational query tool for your needs and environment.Published at:https://www.eckerson.com/articles/modernizing-analytics-with-conversational-query-tools-five-must-have-characteristics
Data analytics is a balance of flexibility for innovation and governance to control risks. This blog discusses its implications for artificial intelligence (AI), including machine learning (ML) and generative AI (GenAI).Published at:https://www.eckerson.com/articles/ai-ml-innovation-requires-a-flexible-yet-governed-data-architecture
Non-profit organizations are more mission-driven, consensus-driven, and resource-constrained than commercial organizations. As a result, it’s imperative that non-profits develop a data strategy before plunging into building data solutions. It will save them time, money, and burnout in the long run.Published at: https://www.eckerson.com/articles/why-non-profits-need-a-data-strategy
Explore the reasons for data engineers to collaborate with data scientists, machine learning (ML) engineers, and developers on DataOps initiatives that support GenAI.Published at:https://www.eckerson.com/articles/dataops-for-generative-ai-data-pipelines-part-iii-team-collaboration
This blog explores three criteria to evaluate tools that manage unstructured data pipelines for GenAI.Published at:https://www.eckerson.com/articles/data-engineering-for-genai-three-criteria-to-evaluate-pipeline-tools
If your data team wants to implement data products, it would be wise to avoid these 12 pitfalls that can torpedo an initiative.Published at:https://www.eckerson.com/articles/12-pitfalls-to-avoid-when-implementing-data-products
This article compares data catalogs and data marketplaces and argues that you need both and will soon have both as vendors add data marketplace extensions.Published at:https://www.eckerson.com/articles/why-do-i-need-a-data-marketplace-when-i-have-a-data-catalog
This blog defines conversational BI, why companies should consider it, and how their power and casual users can best get the desired results.Published at:https://www.eckerson.com/articles/driving-results-with-conversational-bi-best-practices-for-power-and-casual-users
Data engineering is now considered a crucial job in IT as Generative AI, the hottest technology of this decade, relies on data engineers to provide accurate inputs.Published at:https://www.eckerson.com/articles/data-engineering-for-genai-how-to-optimize-data-pipelines-and-governance
Data engineers and data scientists must manage pipelines for unstructured data to ensure healthy inputs for language models.Published at:https://www.eckerson.com/articles/why-and-how-data-engineers-will-enable-the-next-phase-of-generative-ai
Companies that adopt DataOps increase the odds of success by making GenAI data pipelines what they should be: modular, scalable, robust, flexible, and governed.Published: https://www.eckerson.com/articles/dataops-for-generative-ai-data-pipelines-part-ii-must-have-characteristics
Most data leaders want to deliver data products, but few are doing it. Let's face it: most data teams today function as internal service bureaus that fulfill customer requests that arrive via ticketing systems, email, handwritten notes, or calls from colleagues looking for a favor. Most work double time to keep their request backlogs from ballooning from weeks to months. In this environment, few data leaders have time or capacity to switch from a project management approach to a product management one.Even if data leaders had time, most wouldn't know how to make this transition. Most have no experience in product management, nor do they have a good idea of a data product. So asking data leaders to deliver data products is like asking them to build a rocket ship that can travel to the moon.In this episode, Wayne Eckerson interviews Henrik Strandberg, a strong proponent of running data teams using product management principles. Henrik Strandberg is a seasoned data transformation leader who, for the past 25 years, has helped numerous organizations bridge gaps between business and technology. In stints at publishing and gaming companies, Henrik has developed a unique understanding of building and delivering data products at scale that delight customers.
GenAI can help data engineers become more productive, and data engineering can help GenAI drive new levels of innovation.Published at:https://www.eckerson.com/articles/achieving-fusion-how-genai-and-data-engineering-help-one-another
Discover how master data management (MDM) provides language models with high-quality enterprise data to improve their response accuracy.Published at:https://www.eckerson.com/articles/improving-genai-accuracy-with-master-data-management
Explore our four primary criteria for evaluating conversational BI products.Published at:https://www.eckerson.com/articles/genai-driven-analytics-product-evaluation-criteria-for-conversational-bi
The success of Generative AI depends on fundamental disciplines like DataOps.Published at:https://www.eckerson.com/articles/dataops-for-generative-ai-data-pipelines-part-i-what-and-why
With the increasing adoption of Generative AI, learn how data governance will add value to and benefit from Generative AI.Published at:https://www.eckerson.com/articles/data-governance-in-the-era-of-generative-ai
"Meet the business where it is." If you're on the data team, that's what you're expected to do to empower stakeholders with data. But how far should you go to meet the business? And shouldn’t the business be expected to move a little toward meeting the data where it is?Published at:https://www.eckerson.com/articles/meeting-the-data-where-it-is-time-for-the-business-to-step-up
The European Union recently passed the first of its kind legal framework on the development, use, and governance of artificial intelligence. It lays out rules and standards with the aim of ensuring technologies are safe and transparent, and do not violate the fundamental rights of an individual.Published at:https://www.eckerson.com/articles/the-eu-ai-act-and-the-emergence-of-new-global-standards
Most organizations are committed to responsible and ethical use of AI. Yet anticipating unintended consequences before designing and implementing AI can be challenging. This framework and process helps evaluate short-term and long-term impacts across multiple dimensions so you can mitigate AI’s unintended consequences.Published at:https://www.eckerson.com/articles/mitigating-ai-s-unintended-consequences
It's not easy being the head of data & analytics at a large organization. You must align a large team across multiple disciplines; you must deal with oodles of legacy systems and tools that hamper innovation; and you must deliver business value fast to keep executives at bay and your job intact. You also need to recruit dynamic managers who can push the envelope while meeting operational objectives. And when you falter--which you inevitably will-you have to rebound fast. No one knows these lessons better than Tiffany Perkins-Munn. She currently runs a 275-person data & analytics team at JP Morgan Chase that consists of data engineers, data scientists, behavioral economists, and business intelligence experts. She thrives on versatility, having earned a Ph.D. in Social-Personality Psychology with an interdisciplinary focus on Advanced Quantitative Methods. Building on this foundation, she has accumulated vast experience in the art of managing data & analytics teams during her 23 years in technical and managerial roles in the financial services industry. In this interview, you’ll learn:1. Tiffany’s secret for aligning a large data & analytics team and keep them from splitting into silos of specialization2. Her favorite techniques for recruiting the right people to her team. 3. How to wade through the thicket of legacy systems and deliver innovative solutions quickly. 4. The impact of GenAI on her operations and the financial services industry. 5. How to advance your careers in data & analytics.
Adopting community of practice principles, along with coaching and mentoring, is a practical approach to fostering and cultivating data literacy.Published at:https://www.eckerson.com/articles/a-people-first-approach-to-developing-data-literacy
This blog examines the upcoming trend of domain-specific LLMs and evaluates three different methods of implementation.Published at:https://www.eckerson.com/articles/the-next-wave-of-generative-ai-domain-specific-llms
Many machine learning (ML) use cases center on real-time calculations. This article defines streaming ML and its architectural components.Published at:https://www.eckerson.com/articles/machine-learning-and-streaming-data-pipelines-part-i-definitions-and-architecture
Companies need to invest heavily in teams and people, both at corporate and in the field, if they want to become a data-driven organization.Published at:https://www.eckerson.com/articles/organizing-for-success-part-iii-how-to-organize-and-staff-data-analytics-teams
Data management practices have changed substantially since the early 1990s and the dawn of data warehousing.Published at:https://www.eckerson.com/articles/the-continuing-evolution-of-data-management
Conventional data governance conflicts with today’s world of self-service analytics and agile projects. Published at:https://www.eckerson.com/articles/modern-data-governance-problems
Let's reflect on the events of the past year and prognosticate on what may transpire in the months ahead.Published at:https://www.eckerson.com/articles/trends-for-2024-our-team-gazes-into-the-crystal-ball
Data leaders must prepare their teams to deliver the timely, accurate, and trustworthy data that GenAI initiatives need to ensure they deliver results. They can do so by modernizing their environments, extending data governance programs, and fostering collaboration with data science teams.Published at:https://www.eckerson.com/articles/the-data-leader-s-guide-to-generative-ai-part-i-models-applications-and-pipelines
Data modeling is a core skill of data engineering, but it is missing or inadequate in many data engineering teams. These teams focus on moving data with little attention to shaping the data. They engineer processes, not products. Full data engineering is both process and product engineering, and that calls for data modeling.Published at:https://www.eckerson.com/articles/a-fresh-look-at-data-modeling-part-2-rediscovering-the-lost-art-of-data-modeling
The hardest part about implementing data products is fostering a product mindset among the people responsible for defining, governing, building, and shipping data products. It’s also important that an organization establish processes to facilitate the work of the product team and review boards.Published at:https://www.eckerson.com/articles/data-products-part-ii-data-products-require-product-thinking
Many organizations abandoned data modeling as they embraced big data and NoSQL. Now they find that data modeling continues to be important, perhaps more important today than ever before. With a fresh look you’ll see that today’s data modeling is different from past practices – much more than physical design for relational data.Published at:https://www.eckerson.com/articles/a-fresh-look-at-data-modeling-part-1-the-what-and-why-of-data-modeling
Data democratization is the buzzword to describe empowering enterprise stakeholders with data. While there have been advances in data management, governance, and analytics, something keeps getting in the way of achieving data democratization.Published at:https://www.eckerson.com/articles/data-democratization-and-the-duties-of-data-citizenship
Our industry’s breathless hype about generative AI tends to overlook the stubborn challenge of data governance. Data catalogs address this challenge by evaluating and controlling the accuracy, explainability, privacy, IP friendliness, and fairness of GenAI inputs.Published at:https://www.eckerson.com/articles/generative-ai-needs-vigilant-data-cataloging-and-governance
The need for an independent semantic layer continues to rise as data science gains traction in the enterprise. Its five primary elements—metrics, caching, metadata management, APIs, and access controls—support AI/ML use cases as part of data science projects.Published at: https://www.eckerson.com/articles/why-and-how-to-enable-data-science-with-an-independent-semantic-layer
Business leaders can address AI bias and use it to have rational discussions about management and human bias.Published at: https://www.eckerson.com/articles/weighing-the-risk-and-reward-of-ai-a-non-technical-guide-for-business-leaders
Most organizations view data as an asset to be actively managed with standards, controls, and discipline. Yet, they are passive and casual about metadata. Data is managed. Metadata happens. As data management becomes more complex, metadata management is becoming an essential discipline. It is time to think about metadata management from an architectural perspective.Published at: https://www.eckerson.com/articles/an-architectural-view-of-metadata-management
Kevin Petrie, the Vice President of Research at Eckerson Group, and Dan O’Brien, research analyst, discussed large language models (LLMs), which are neural networks that analyze text to predict the next word or phrase. These models use training data, often from the internet, to understand word relationships and provide accurate answers to natural language questions.
Generative AI initiatives require new data pipelines that prepare text files for querying by language models. Data engineers, scientists, and other stakeholders collaborate to design and implement these pipelines, which span text sources, tokens, vectors, vector databases, and LMs. Published at: https://www.eckerson.com/articles/the-new-data-pipeline-for-generative-ai-where-and-how-it-works
Dan and Wayne discussed the concept of data and analytics operating models, which refers to how organizations organize their data and analytics resources for alignment and efficiency.
This final blog in our series on the ROI of master data management recommends ways for data teams to iterate their MDM initiatives based on the successes and failures of their first project. Published: https://www.eckerson.com/articles/driving-roi-with-master-data-management-part-iii-project-iteration
Responsible AI ethical principles provide a clear, unifying purpose for the technological, business, and social goals of AI initiatives. Published at: https://www.eckerson.com/articles/the-opportunity-and-risk-of-generative-ai-part-iii-responsible-ai-ethics
Most definitions of a data product conflate it with a data asset. The only way to turn a data asset into a data product is to publish it in a data store along with metadata about subscription and delivery options, and terms of service that specify a bidirectional contract between data consumer and producer. Published at: https://www.eckerson.com/articles/let-s-be-clear-a-data-asset-is-not-a-data-product
Responsible AI can help data leaders comply with the fast-evolving regulatory environment of data and artificial intelligence. Published at: https://www.eckerson.com/articles/the-opportunity-and-risk-of-generative-ai-part-ii-how-responsible-ai-assists-compliance
US frontier history had races, risks, and rewards. Generative AI's future will follow a similar path. Published at: https://www.eckerson.com/articles/enterprise-data-and-the-taming-of-the-generative-ai-frontier
Dan and Jay discussed the concept of Data Fabric, an automated and AI-driven approach to managing modern data environments.
Simba Khadder and Kevin Petrie discuss strategies to overcome technical debt in implementation, the pivotal role of data in the success of ML projects, navigating regulatory compliance in machine learning, and the future of AI governance.
Learn how to attain an optimal return on investment (ROI) with MDM by choosing the appropriate architectural strategy and evaluating progress during the initial project implementation. Published at: https://www.eckerson.com/articles/driving-roi-with-master-data-management-part-ii-your-first-project
As organizations strive to meet the ever-growing demand for data, they are adopting data products to streamline delivery and ensure solutions provide value to business stakeholders. Learn about four traps that can disrupt data product development and how to avoid falling into them. Published at: https://www.eckerson.com/articles/four-traps-to-avoid-when-developing-data-products
Generative AI brings a promise to improve lives in a blistering innovation race, but also a threat to people, corporations, and even nations. Data analytics leaders must understand the risks of generative AI, both societal and business-related, to use it positively and avoid the destructive consequences seen with nuclear energy development. Published at: https://www.eckerson.com/articles/the-opportunity-and-risk-of-generative-ai-part-i-a-nuclear-explosion
The unbundling of the data ecosystem is causing organizations to “duct tape” products and frameworks together to build their solutions and data delivery processes. Organizations fail to build and deploy end-to-end, automated, repeatable data-driven systems, ignoring data engineering & dataops principles as well as best practices. Published at: https://www.eckerson.com/articles/dataops-in-data-engineering
This blog recommends guiding principles for successful implementation of language models to assist data engineering. Published at: https://www.eckerson.com/articles/should-ai-bots-build-your-data-pipelines-part-iv-guiding-principles-for-success-with-language-models-and-data-engineering
An emerging approach to generative AI will help data engineering teams achieve much-needed productivity gains while controlling risk. Published at: https://www.eckerson.com/articles/should-ai-bots-build-your-data-pipelines-part-iii-the-emergence-of-small-language-models-for-data-engineering
Dan O'Brien and Kevin Petrie discuss FinOps, which is a cost governance discipline for cloud-based analytics and operational projects.
An annual assessment of the positioning strategies of the leading 21 BI vendors finds a lack of differentiation that makes it difficult for buyers to compare products. In the BI market’s sea of sameness, Qlik is the only vendor that stands out with this clever, memorable position. Published at: https://www.eckerson.com/articles/independent-study-bi-vendor-messaging-shows-lack-of-differentiation
MDM creates business value in three ways: it streamlines infrastructure, streamlines processes, and reduces risk. Published at: https://www.eckerson.com/articles/driving-roi-with-master-data-management-part-1-build-your-business-case
“Universal” semantic layer tools introduced in recent years promise to standardize business metrics across the data stack, and eliminate silos of metrics trapped in semantic layers that are limited to specific data sources or BI platforms. This post offers considerations for adopting a universal semantic layer. Published at: https://www.eckerson.com/articles/the-universal-semantic-layer-more-than-enough
An Analytics Center of Excellence empowers business teams to meet their own data needs by changing the role of IT from developer to facilitator. The reality, however, is that IT needs be both a facilitator and a developer. Published at: https://www.eckerson.com/articles/analytics-center-of-excellence-part-i-how-to-shape-the-organization
Traditional companies must balance new and old technologies as part of an ever-modernizing data stack. This blog explores how companies strike the right balance to navigate economic uncertainty, AI disruption, and the need for tool consolidation. Published at: https://www.eckerson.com/articles/the-modernizing-data-stack-three-ways-to-balance-new-and-old
LLMs are hugely popular with data engineers because they boost productivity. But companies must adapt their data governance programs to control risks related to data quality, privacy, intellectual property, fai-Datarness, and explainability. Published at: https://www.eckerson.com/articles/should-ai-bots-build-your-data-pipelines-part-ii-risks-and-governance-approaches-for-data-engineers-to-use-large-language-models
Despite innovations in data architecture, infrastructure, and analytics, most organizations today still struggle to realize the promised value of data. Learn how the data mesh principle of data as a product can help, as part of a data mesh initiative or as a stand-alone strategy. Published at: https://www.eckerson.com/articles/data-products-part-of-a-data-mesh-initiative-or-a-stand-alone-strategy
Data mesh is a new paradigm for fulfilling the promised value of data. It decentralizes both data ownership and the data itself, shifting them toward the functional domains that create and use data to operate. But data mesh is not for everyone. Learn how to assess if you’re ready for data mesh. Published at: https://www.eckerson.com/articles/data-mesh-evaluating-your-organization-s-readiness-for-a-decentralized-data-future
There’s so much hype surrounding data products that you have to wonder if it’s just another buzzword. But there’s more to data products than buzz. In this article, you’ll learn how the concept is a meaningful step forward in the art and science of data management.Published at: https://www.eckerson.com/articles/best-practices-for-developing-and-scaling-data-products
A zone-based data refinery creates an agile, adaptable data environment that supports new and unanticipated business requirements quickly. It turns a monolithic data warehouse into a flexible data environment that gracefully adapts to new and unanticipated business requirements while maximizing reuse and standards.Published at: https://www.eckerson.com/articles/how-zone-based-data-processing-turns-your-monolithic-data-warehouse-into-a-flexible-modern-data-architecture
Many data engineers already use large language models to assist data ingestion, transformation, DataOps, and orchestration. This blog commences a series that explores the emergence of ChatGPT, Bard, and LLM tools from data pipeline vendors, and their implications for the discipline of data engineering.Published at: https://www.eckerson.com/articles/should-ai-bots-build-your-data-pipelines-examining-the-role-of-chatgpt-and-large-language-models-in-data-engineering
At IAPP Summit, privacy and data governance leaders expressed the importance of a collaborative operating model.Published at: https://www.eckerson.com/articles/the-convergence-of-data-governance-and-privacy-takeaways-from-the-global-privacy-summit
Embeddings are a learned way of representing data in space. Vector databases make it easier to work with embeddings generated from deep learning models. They will become an essential tool in the AI stack because they reduce the time to structure data and train models.Published at: https://www.eckerson.com/articles/the-why-what-who-and-where-of-vector-databases
A robust data workflow testing strategy helps ensure the accuracy and reliability of data processed within a pipeline. Use this checklist to meet your organization’s data quality requirements according to the dimensions of accuracy, completeness, conformity, consistency, integrity, precision, timeliness, and uniqueness.Published at: https://www.eckerson.com/articles/developing-a-robust-data-quality-strategy-for-your-data-pipeline-workflows
An operational data hub (ODH) is a pattern in data architecture that provides a central location and a standard protocol for operational systems to communicate about and share data among themselves. Operational systems post messages about data events (add, change, delete) and subscribe to messages of interest posted by other applications. The hub works to share data among applications without the clutter and chaos of point-to-point data feeds.Published at:https://www.eckerson.com/articles/operational-data-hub-responding-to-data-friction-and-technical-debt
Data mesh is a hot topic in the data world, generating conversations about the benefits and drawbacks of its decentralized approach. Concerns about an explosion of data silos and inconsistent data quality are justified. But to those who feel a bit like Chicken Little, maybe the sky is not falling.Published at:https://www.eckerson.com/articles/data-mesh-the-sky-is-not-falling
The data mesh paradigm is in a nascent stage with data personas and organizations craving clarity and quick answers. Best practices are yet to be crystallized. Mesh done incorrectly runs the risk of degenerating into silos.Published at:https://www.eckerson.com/articles/caution-data-leaders-plan-carefully-before-rushing-to-data-mesh
I got energized walking the show floor at the Gartner Data & Analytics event last month and learned a few things about the future of our industry. Published at:https://www.eckerson.com/articles/quick-recap-of-gartner-conference-2023
The modern data stack is a loose collection of technologies, often cloud-based, that collaboratively process and store data to support modern analytics. It must be automated, low code/no code, AI-assisted, graph-enabled, multimodal, streaming, distributed, meshy, converged, polyglot, open, and governed.Published at:https://www.eckerson.com/articles/twelve-must-have-characteristics-of-a-modern-data-stack
AutoML and the emerging approach of declarative ML help simplify the process of creating and refining ML models.Published at:https://www.eckerson.com/articles/automl-and-declarative-machine-learning-comparing-use-cases
One version of the truth is the holy grail of data and analytics. However, the promise of one version of the truth still evades us because even with consistent data, the truth is, as the film My Cousin Vinny demonstrates, a matter of perspective and context.Published at:https://www.eckerson.com/articles/one-version-of-the-truth-according-to-my-cousin-vinny
Data leaders know the importance of change management, but few understand the dynamics involved in driving adoption. A new book by Damon Centola shows how social networks spread and inhibit change.Published at:https://www.eckerson.com/articles/how-change-happens-driving-technology-adoption
Over the past 20 years or more, data architecture practices have focused almost exclusively on managing data for analytics. Operational data is much more than source data for analytics. We must give attention to operational data architecture or pay the price in data disparity, data friction, and technical debt.Published at:https://www.eckerson.com/articles/operational-data-architecture
Designed and implemented well, automated workflows can make the modern business just a little less chaotic and complex. This blog explores the opportunity for automated workflows to help cross-functional teams collaborate and standardize organizational master data.Published at:https://www.eckerson.com/articles/master-data-management-and-operational-workflows-two-modern-use-cases
Data fabric is one of those buzzwords that’s used so much and in so many ways that it often elicits an eyeroll—undeservedly so. The phrase is shorthand for a complex and important set of issues that we’re all working to manage. In this article we’ll review what data fabric is and why it’s important.Published at:https://www.eckerson.com/articles/data-fabric-s-use-of-abstraction-and-metadata
The data pipeline market comprises four segments: data ingestion, data transformation, DataOps, and orchestration. This blog defines three principles for successful pipelines: (1) watch the innovative startups; (2) use suites where you can; and (3) use point tools where you must.Published at:https://www.eckerson.com/articles/modern-data-pipelines-three-principles-for-success
The data mesh framework doesn’t specify a key component that completes the last mile of the architecture: a data provisioning environment. New technology that underpins modern data marketplaces complement data mesh, providing a frictionless way for data providers and data consumers to exchange data.Published at: https://www.eckerson.com/articles/data-mesh-s-missing-ingredient-a-data-marketplace
Traditional techniques for managing data quality break at scale. Machine learning algorithms can automate aspects of the data quality workload, ensuring that the data the business users consume is reliable. This article profiles three tools and approaches that use ML to automate data quality.Published at:https://www.eckerson.com/articles/three-data-quality-automation-tools-you-should-consider
We must treat metadata like a fully-vested member of the enterprise data landscape. A unifying taxonomy is a good place to start making metadata a focus of data management rather than just a tool. This article explores how to start wrangling diverse and distributed metadata.Published at:https://www.eckerson.com/articles/wrangling-metadata-making-it-the-object-of-data-management
We enter 2023 in a haze of uncertainty. Enterprises must rationalize analytics projects, shift to lower-risk use cases, and control cloud costs. They also must measure the ROI of analytics projects and use data governance to reduce business risk.Published at:https://www.eckerson.com/articles/analyzing-a-downturn-five-principles-for-data-analytics-in-2023
This blog defines governed data integration and describes how it enabled two manufacturers to synchronize data flows from the factory floor to the customer.Published at:https://www.eckerson.com/articles/governed-data-integration-for-manufacturers
A rising number of financial services firms are adopting the discipline of governed data integration to build 360-degree customer views.Published at:https://www.eckerson.com/articles/governed-data-integration-for-financial-services
Synthetic data and artificial intelligence (AI) complement each other but are both subject to the risk of AI bias. Consequently, companies need to implement architectural and governance controls to reduce the bias that synthetic data can inject into AI models.Published at:https://www.eckerson.com/articles/mitigating-the-risk-of-bias-in-synthetic-data-for-ai
As enterprises grow more dependent on the cloud and as the economy convulses, FinOps will soon become mandatory.Published at:https://www.eckerson.com/articles/the-rise-of-finops-cost-governance-for-cloud-based-analytics
Business domains have a range of data & analytics capabilities that enterprise data teams must support. The key is to ensure domain activity aligns with enterprise standards and best practices to ensure data consistency and avoid silos.Published at: https://www.eckerson.com/articles/an-operating-model-for-data-analytics-part-iv-red-team-composition
Active metadata is not a type of metadata, it’s a way of using metadata to power systems. Active metadata is a critical feature of modern data architectures such as data fabric and data mesh. It makes things work such as data access management, data classification, and data quality management. Published at: https://www.eckerson.com/articles/active-metadata-the-critical-factor-for-mastering-modern-data-management
This blog explores four use cases for data orchestration and examples of the supporting architectural elements. Published at: https://www.eckerson.com/articles/architecting-data-orchestration-four-use-cases
There are many models for bridging business and technical teams. These models can be more centralized or decentralized in nature, depending on the culture of the organization and nature of the business domain. Each requires a strong enterprise data teams comprised of multiple departments and roles. Published at: https://www.eckerson.com/articles/an-operating-model-for-data-analytics-part-iii-team-composition-and-dynamics
Data observability provides intelligence about data quality and data pipeline performance, contributing to the disciplines of DataOps and FinOps. Vendors such as DataKitchen, DataOps.live, Informatica, and Unravel offer solutions to help enterprises address these overlapping disciplines. Published at: https://www.eckerson.com/articles/the-blending-disciplines-of-data-observability-dataops-and-finops
Hybrid development teams are critical to the success of a data & analytics program. Data leaders must invest time, energy, and thought to the creation of these teams and how best to support them. It’s critical that they allocate staff time to nurture knowledge flows between component groups. Published at: https://www.eckerson.com/articles/an-operating-model-for-data-analytics-part-ii-knowledge-flows
“Metadata is data about data” is a bad definition. It’s vague and recursive. It’s like saying climate is the weather of weather. But if metadata is not data about data, then what is it? Follow my thought process working toward a better understanding of metadata and its role in today’s data landscape. Published at: https://www.eckerson.com/articles/fixing-metadata-s-bad-definition
Data Mesh gets a lot of discussion. Some see it as revolutionary—the first new data architecture thinking in years. Others view it as a dangerous backward slide to the chaos of data silos. The reality lies somewhere between—a big shift in architecture thinking with some inherent risk. A few good architecture practices help to realize the benefits while managing the risks. Published at: https://www.eckerson.com/articles/a-thoughtful-approach-to-data-mesh
Many machine learning projects fail because data scientists don’t have the right data. Techniques such as synthetic data is a novel algorithmic approach to address algorithmic risks. Published at: https://www.eckerson.com/articles/synthetic-data-for-ai-definition-risks-and-strategies
Many machine learning projects fail because data scientists don’t have the right data. Techniques such as synthetic data is a novel algorithmic approach to address algorithmic risks. Published at: https://www.eckerson.com/articles/synthetic-data-for-ai-definition-risks-and-strategies
Data orchestration uses caching, APIs, and centralized metadata to help compute engines access data in hybrid or multi-cloud environments. Data platform engineers can use data orchestration to gain simple, flexible, and high-speed access to distributed data for modern analytics and AI projects. Published at: https://www.eckerson.com/articles/data-orchestration-simplifying-data-access-for-analytics
Data access management (DAM) is the process of defining and enforcing policies that control access to application data throughout the enterprise. New approaches to DAM provide greater access and better protection through centrally managed policies that are universally enforced. Published at: https://www.eckerson.com/articles/data-access-management-zero-sum-game-over
This article, the third in a series, dives into the technologies that underpin modern approaches to location intelligence. It explores databases for industrial-scale geospatial applications, advanced business intelligence (BI) tools for exploratory analysis, and simple use-case specific platforms. Published at: https://www.eckerson.com/articles/location-intelligence-part-iii-enabling-technologies
The data mesh makes business domain experts the owners of their data, which they deliver as a “data product” to analytics teams using a self-service data platform and a federated governance framework. Published at: https://www.eckerson.com/articles/why-enterprises-should-implement-the-data-mesh-with-dataops
An operating model for data & analytics is critical for aligning resources across the enterprise and balancing the needs for agility and governance. An effective operating model is critical to data & analytics success and its creation and upkeeping should be the primary focus of a chief data officer. Published at: https://www.eckerson.com/articles/an-operating-model-for-data-analytics
In this article, Lawson Abinati lays out core principles for market positioning that apply across all industries, as well as to Business Intelligence (BI) professionals. Although many business intelligence (BI) managers see themselves as technologists first, unless they understand the soft skills of sales, marketing, and communication, they won't succeed professionally or make good on their organization's investments in BI. Published at: https://www.eckerson.com/articles/what-is-positioning-and-why-is-it-important
This article summarizes the major characteristics of a modern data architecture and serves as a guide for organizations that are in the midst of developing a new data strategy for the modern age. Published at: https://www.eckerson.com/articles/ten-characteristics-of-a-modern-data-architecture
An analytics center of excellence is the cornerstone of every data strategy, yet few data leaders know how to design one that works effectively. The key is to embrace federated techniques that balance standards and speed, agility and governance. This article explains the core components of an analytics center of excellence. Published at: https://www.eckerson.com/articles/how-to-design-an-analytics-center-of-excellence
Design patterns have proven valuable in many endeavors. Can data pipeline design patterns help to break the data engineering logjam? Published at: https://www.eckerson.com/articles/data-pipeline-design-patterns
The need for adaptable data management architecture has never been more pressing. Yet getting there seems to be more confusing than ever. The field is rampant with buzzwords: data lake, data lakehouse, data fabric, data mesh, data hub, data as a network. Making sense of the confusion begins with sorting out the buzzwords. Published at: https://www.eckerson.com/articles/data-architecture-complex-vs-complicated
Data mesh is an evolutionary concept that’s gained a lot of traction in the software engineering world. It’s both exciting and unnerving—exciting because it represents a sea change in engineering attitudes toward data; unnerving because of its potential to create organizational confusion. Published at: https://www.eckerson.com/articles/the-exciting-unnerving-vision-of-data-mesh
This article, the second in a series, shares cutting-edge examples of location intelligence applications from the real world to help readers understand what’s possible with geospatial data. Published at: https://www.eckerson.com/articles/location-intelligence-part-ii-real-world-examples-from-businesses-that-use-geospatial-data
The data mesh is an attempt by software engineers to remake the data industry in their image. There is a lot of goodness in the data mesh but will it work? The track record for other reform efforts is not good, although new technologies are putting wind into the methodology. Published at: https://www.eckerson.com/articles/data-mesh-translated-software-engineers-try-to-reform-data
Data Stewardship Experience strategies (personal growth, community, societal contribution, and disruptive innovation) can meet several cognitive, social, and psychological needs, and motivate professionals to become productive data stewards. It also removes the stigma of data governance as a rigid and bureaucratic gatekeeping discipline. Published at: https://www.eckerson.com/articles/improving-the-data-stewardship-experience-dsx-productive-motivational-strategies-for-data-governance
Decision AI is an emerging discipline that enables faster, smarter operational decisions by combining decision intelligence and contextual intelligence. Published at: https://www.eckerson.com/articles/decision-ai-and-the-opportunity-for-smarter-faster-actions
This audio blog, the third and final in a series, recommends five guiding principles for success. Published at: https://www.eckerson.com/articles/the-customer-360-data-program-and-cloud-connectors-guiding-principles-for-success
Consider key trends and challenges as you design an effective organizational architecture for data governance while generating value with pervasive analytics. Published at: https://www.eckerson.com/articles/organizational-architecture-can-make-or-break-your-data-governance-program
At Summit 2022, Snowflake paints a picture of a utopian data future where all data is treated equally regardless of where it comes from or what language is used to process it, where processing resources are limitless, and all data is secured and governed. Published at: https://www.eckerson.com/articles/snowflake-summit-2022-growth-vision-and-strategies
Providing analysts with tools that increase their efficiency is critical to helping them do more work with less effort. Published at: https://www.eckerson.com/articles/a-data-analyst-s-guide-to-the-data-catalog
This blog recommends criteria for enterprises to evaluate cloud connectors and ensure the benefits outweigh the costs. Published at: https://www.eckerson.com/articles/evaluating-cloud-connectors-for-the-customer-360-data-program
Machine learning models help respond to the time-based value and risks of business events. To achieve this on an ongoing basis, enterprises should build a streaming ML program based on sound business objectives, a cross-functional team, open platforms, and phased execution. Published at: https://www.eckerson.com/articles/machine-learning-and-streaming-data-pipelines-part-iii-guiding-principles
Enterprise data teams embrace DataOps to achieve new levels of efficiency and effectiveness in delivering data-driven solutions. Roche shows what’s possible when you combine a state-of-the-art cloud data platform with a data mesh architecture and DataOps solution. Published at: https://www.eckerson.com/articles/what-can-dataops-do-for-you-ask-roche
As we share data, we create data webs. If we allow copies of our data to proliferate throughout these webs, we reduce the value of the data and create data governance challenges. The solution is new, ownership-centric approaches to data sharing that don’t rely on traditional copy-based integration. Published at: https://www.eckerson.com/articles/zero-copy-approaches-to-data-sharing
Treating data assets as products helps businesses increase internal data consumption and become more data driven. It also creates opportunities for monetization. But to succeed in either scenario requires product-market fit. This article presents a process for finding it. Published at: https://www.eckerson.com/articles/7-steps-for-building-a-valuable-data-product
It’s tempting to dismiss observability as another overused buzzword. But this emerging discipline offers substantive methods for enterprises to monitor and optimize business metrics, IT operations, data pipelines, machine learning models, and data quality. Published at: https://www.eckerson.com/articles/the-five-shades-of-observability-business-operations-pipelines-models-and-data-quality
An analytics center of excellence is the cornerstone of every data strategy, yet few data leaders know how to design one that works effectively. The key is to embrace federated techniques that balance standards and speed, agility, and governance. This article explains the core components of an analytics center of excellence. Published at: https://www.eckerson.com/articles/how-to-design-an-analytics-center-of-excellence
Companies are investing in new solutions—such as data fabric, data access governance, and data observability—to keep pace with expanding business appetite for data. Pervasive use of metadata to solve data management problems means that metadata is itself a valuable data asset that we must proactively manage. Published at: https://www.eckerson.com/articles/metadata-is-data-so-manage-it-like-data
Data engineers are often responsible for numerous types of data tests: unit tests, integration/component tests, performance tests, and end-to-end tests. Their best hope is to find the automated data testing tools that work for their technologies and try them. Published at: https://www.eckerson.com/articles/testing-capabilities-and-tools-for-data-engineers-part-2
Data integration, governance, and consumption play a pivotal role in the machine learning lifecycle. New offerings from Informatica illustrate the types of tools data science teams need to handle data integration, governance, and consumption. Published at: https://www.eckerson.com/articles/integrating-governing-and-consuming-data-for-the-machine-learning-lifecycle
Data architecture-as-a-service or DAaaS is a metadata-driven approach that injects SQL guardrails into no-code data development tools so business users can develop their own data pipelines without creating data silos. This article defines 10 criteria for evaluating DAaaS-based products. Published at: https://www.eckerson.com/articles/data-architecture-as-a-service-how-to-evaluate-products
Today’s data architecture discussions are heavily biased toward managing data for analytics, with attention to big data, scalability, cloud, and cross-platform data management. We need to acknowledge analytics bias and address management of operational data. Ignoring operational data architecture is a sure path to technical debt and future data management pain. Published at: https://www.eckerson.com/articles/the-yin-and-yang-of-data-architecture
This audio blog, the first in a series, presents the fundamentals of location intelligence. It will explore how location intelligence has evolved in recent years and the kinds of insights it can provide. Published at: https://www.eckerson.com/articles/location-intelligence-part-i-leveraging-geospatial-data-to-drive-your-organization
Data architecture-as-a-service (DaaS) is a new self-service paradigm that empowers local data owners to create architecturally compliant data repositories. By abstracting data architecture within self-service tools, DaaS solves the problem of data silos, which wreak havoc on enterprise data consistency and trustworthiness. Published at: https://www.eckerson.com/articles/data-architecture-as-a-service-liberation-for-data-users
We in the West have watched Russia's invasion of Ukraine with disbelief and horror. How could this happen to a European country in the 21st century? Is there any justifiable rationale for the wanton destruction of people and property there? As we ponder these questions, our data colleagues in Ukraine have experienced the war firsthand.
To help us get a handle on Ukraine's role in the data economy and how teams based there are coping with Russia's military onslaught, Wayne interviews two software executives today who share how the war has affected their companies and how they are adapting to the evolving situation.
Dragos Georgescu is vice president and chief technology officer of DataClarity, an innovative data analytics vendor with a development shop in Lviv, Ukraine.
Bogdan Steblyanko is CEO of CHI Software, a software development company based in Ukraine with more than 500 employees spread across four development centers, including hard-hit Kharkiv in the east, which is the company's headquarters.
COVID, inflation, broken supply chains, and not-so-distant war make this a turbulent time for the modern consumer. During times like these, families tend to their nests, which leads to lots of home-improvement projects…which means lots of painting.
Today we explore the case study of a Fortune 500 producer of the paints and stains that coat many households, consumer products, and even mechanical vehicles. While business expands, this company needs to carefully align the records that track hundreds of suppliers, thousands of storefronts, and millions of customers.
Business expansion and complex supply chains make it particularly important—and challenging—for enterprises such as this paint producer, which we’ll call Bright Colors, to accurately describe the entities that make up their business. They need to be governed, validated data to describe entities such as their products, locations, and customers. Master data management, also known as MDM, streamlines operations and assists data governance by reconciling disparate data records into golden records and ideally a single source of truth.
We’re excited to share our conversation with an industry expert that helps Bright Colors and other Fortune 2000 enterprises navigate turbulent times with effective strategies for MDM and data governance.
Dave Wilkinson is chief technology officer with D3Clarity, a global strategy and implementation services firm that seeks to ensure digital certainty, security, and trust. D3Clarity is a partner of Semarchy, whose Intelligent Data Hub software helps enterprises govern and manage master data, reference data, data quality, enrichment, and workflows. Semarchy sponsored this podcast.
The number of women entering data professions is growing, and men need to adapt. This podcast is designed to enlighten men about the role of women in the data field. Our guests are all executives at data and analytics software companies who have held positions in other sectors of our field: Prukalpa Sankar, Cindi Howson, Debika Sharma.
Nothing has galvanized the data community more in recent months than two new architectural paradigms for managing enterprise data. On one side there is the data fabric: a centralized architecture that runs a variety of analytic services and applications on top of a layer of universal connectivity. On the other side, is a data mesh: a decentralized architecture that empowers domain owners to manage their own data according to enterprise standards and make it available to peers as they desire.
Most data leaders are still trying to ferret out the implications of both approaches for their own data environments. One of those is Srinivasan Sankar, the enterprise data & analytics leader at Hanover Insurance Group. In this wide-ranging, back-and-forth discussion, Sankar and Eckerson explore the suitability of the data mesh for Hanover, how the Data Fabric might support a Data Mesh, whether a Data Mesh obviates the need for a data warehouse, and practical steps Hanover might to take implement a Data Mesh built on top of a Data Fabric.
Key Takeaways: - What is the essence of a data mesh? - How does it relate to the data fabric? - Does the data mesh require a cultural transformation? - Does the data mesh obviate the need for a data warehouse? - How does data architecture as a service fit with the data mesh? - What is the best way to roll out a data mesh? - What's the role of a data catalog? - What is a suitable roadmap for full implementation?
Nothing has galvanized the data community more in recent months than two new architectural paradigms for managing enterprise data. On one side there is the data fabric: a centralized architecture that runs a variety of analytic services and applications on top of a layer of universal connectivity. On the other side, is a data mesh: a decentralized architecture that empowers domain owners to manage their own data according to enterprise standards and make it available to peers as they desire.
Most data leaders are still trying to ferret out the implications of both approaches for their own data environments. One of those is Srinivasan Sankar, the enterprise data & analytics leader at Hanover Insurance Group. In this wide-ranging, back-and-forth discussion, Sankar and Eckerson explore the suitability of the data mesh for Hanover, how the Data Fabric might support a Data Mesh, whether a Data Mesh obviates the need for a data warehouse, and practical steps Hanover might to take implement a Data Mesh built on top of a Data Fabric.
Gordon Wong is on a mission. A long-time business intelligence leader who has led data & analytics teams at HubSpot and FitBit, Wong believes BI teams aren’t data-driven enough. He says BI leaders need to think of themselves as small businesses owners and aggressively court and manage customers. He says too many don’t have metrics to track customer engagement and usage. In short, BI teams need to eat their own dog food and build success metrics to guide their activities.
If you are a data or analytics leader, do you know the value your team contributes to the business? Do you have KPIs for business intelligence? Can you measure the impact of data and analytics endeavors in terms the business understands and respects? Too often BI and data leaders get caught up in technical details and fail to evaluate how their technical initiatives add value to the business. This wide-ranging interview with a BI veteran will shed light on how to run a successful BI shop.
Fast-casual restaurants offer a fascinating microcosm of the turbulent forces confronting enterprises today—and the pivotal role that data plays in helping them maintain competitive advantage. COVID prompted customers to order their Chipotle burritos, Shake Shack milkshakes, and Bruegger’s Bagels for home delivery, and this trend continues in 2022. Supply-chain disruptions, meanwhile, force fast-casual restaurants to make some fast pivots between suppliers in order to keep their shelves stocked. And the market continues to grow as these companies win customers, add locations, and expand delivery partnerships.
These three industry trends—home delivery, supply-chain disruptions, and market expansion—all depend on governed, accurate data to describe entities such as orders, ingredients, and locations. Data quality and master data management therefore play a more pivotal role than ever in the success of fast-casual restaurants. Master data management, also known as MDM, streamlines operations and assists data governance by reconciling disparate data records into a golden record and source of truth. If you’re looking for an ideal case study for how MDM drives enterprise reinvention, agility, and growth, this is it.
We’re excited to talk with an industry expert that helps fast-casual restaurants handle these turbulent forces with effective strategies for managing data and especially master data. Matt Zingariello is Vice President of Data Strategy Services with Keyrus, a global consultancy that helps enterprises use data assets to optimize their digital strategies and customer experience. Matt leads a team that provides industry-specific advisory and implementation services to help enterprises address challenges such as data governance and MDM.
Keyrus is a partner of Semarchy, whose Intelligent Data Hub software helps enterprises govern and manage master data, reference data, data quality, enrichment, and workflows. Semarchy sponsored this podcast.
In our podcast, we'll define data quality and MDM as part of data governance. We’ll explore why enterprises need data quality and MDM, and how they can craft effective data quality and MDM strategies, with a focus on fast-casual restaurants as a case study.
Knowledge graphs are a new, human-friendly way of organizing and navigating data that makes it easy to infer relationships that aren't explicitly defined. Knowledge graphs now power many applications in the cloud, including Google Search, data fabrics, and data catalogs. They make it easy to glean insights that aren't manually baked into the model. This is why people say knowledge graphs provide a rich, semantic user experience.
Joe Hilleary, a senior research analyst at Eckerson Group, has been exploring knowledge graphs for the past 12 months. He has written several excellent blogs that explain knowledge graphs in a way that makes sense even for a modeling simpleton like me! We've combined his blogs into an e-Book called "Getting Started with Knowledge Graphs" which will publish shortly on our site. Listen to this podcast and then read the eBook if you want to understand the ins and outs of knowledge graphs.
It’s hard to find a data discipline today that is under more pressure than data governance. One on side, the supply of data is exploding. As enterprises transform their business to compete in the 2020s, they digitize myriad events and interactions, which creates mountains of data that they need to control. On the other side, demand for data is exploding. Business owners at all levels of the enterprise need to inform their decisions and drive their operations with data.
Under these pressures, data governance teams must ensure business owners access and consume the right, high-quality data. This requires master data management—the reconciliation of disparate data records into a golden record and source of truth—which assists data governance at many modern enterprises.
In this episode, our host Kevin Petrie, VP of Research at Eckerson Group talks with our guests Felicia Perez, Managing Director, Information as a Product Program at National Student Clearinghouse, and Patrick O'Halloran, enterprise data scientist as they define what data quality and MDM are, why you need them, and how best to achieve effective data quality and MDM.
The advent of big data, self-service analytics, and cloud applications has created a need for new ways to manage data access. New data access governance tools promise to simplify and standardize data access and authorization across an enterprise. Data management expert, Sanjeev Mohan, provides an industry perspective on this emerging technology and what it means for data analytics teams.
In the physical world, you can see a bridge rusting or a building facade crumbling and know you have to intervene to prevent the infrastructure from collapsing. But when all you have is bits and bytes - digital stuff, like software and data ---how can you tell if your customer-facing digital interactions or data-driven analytics and models are about to go up in smoke?
Observability is a new term that describes what we used to call IT monitoring. The new moniker is fitting given all the technology changes that have happened in the past decade. The cloud, big data, microservices, containers, cloud applications, machine learning, and artificial intelligence have created a dramatically complex IT and data environment that is harder than ever to manage. And the stakes are higher as organizations move their operations online to compete with digital natives. Today, you can't run digital or data operations without observability tools.
Kevin Petrie is one of the industry's foremost experts on observability. He is vice president of research at Eckerson Group where he leads a team of distinguished analysts. He recently wrote an article titled "The Five Shades of Observability" that describes five types of observability tools. In this podcast, we discuss what observability is, why you need it, and the types of available tools. We also speculate on the future of this technology and recommend how to select an appropriate observability product.
In this episode, we explore an area of data analytics that everyone knows they need to improve but no one knows how to do it. That is data literacy. Data literacy ensures that business people have the skills to accurately interpret data represented in charts, tables, and dashboards, as well as the knowledge to use those tools to gather and analyze data on their own.
To guide us through the nuances of data literacy and explain how to implement it in an organization, we invited a data literacy expert to share the secrets of his trade. Kirill Makharinsky is the founder of Enki, a San Francisco-based company that provides data-as-a-second language training services. Kirill is a serial entrepreneur, having previously co-founded ETG, one the largest online B2B travel companies in Europe, and Quid, a leading research and analysis tool.
Every December, Eckerson Group fulfills its industry obligation to summon its collective knowledge and insights about data and analytics and speculate about what might happen in the coming year. The diversity of predictions from our research analysts and consultants exemplifies the breadth of their research and consulting experiences and the depth of their thinking. Predictions from Kevin Petrie, Joe Hilleary, Dave Wells, Andrew Sohn, and Sean Hewitt range from data and privacy governance to artificial intelligence with stops along the way for DataOps, data observability, data ethics, cloud platforms, and intelligent robotic automation.
The COVID shock forces enterprises in every market to accelerate and reshape their data analytics strategies. This trend is likely to continue. “Data Elite” enterprises survived this year through a mix of agility, efficiency, and intelligence. They met these requirements of survival as they accelerated their digital transformations, adopted cloud data platforms and embraced advanced analytics. As these data leaders continue their momentum in 2021, the data laggards will strive to catch up.
In this episode, Kevin Petrie, VP of Research at Eckerson Group, interviews Sumeet Agrawal, VP of Product Management at Informatica, to discuss the impact of COVID on enterprises. Sumeet talks about the trends of adoption during the onslaught of COVID and how enterprises are navigating in the post-pandemic era.
Continuous Intelligence (CI) integrates historical and real-time analytics to automatically monitor and update various types of systems, including supply chains, telecommunications networks and e-commerce sites. CI encompasses data ingestion, transformation and analytics, as well as operational “triggers” that recommend or initiate specific real-time actions.
CI casts a wider net than traditional analytics because it includes contextual data, for example related to market behavior, weather patterns or social media trends, that help enterprises operate the core systems more intelligently.
In this episode, our VP of Research Kevin Petrie interviews Simon Crosby, CTO at Swim.ai, a continuous intelligence software vendor that focuses on edge-based learning for fast-data. He co-founded security vendor Bromium in 2010, later sold to HP Inc in 2019.
This blog compares Predictive vs Prognostic analytics and gives a quick view into systems dynamics and causal modeling. If it sparks your interest, watch for an upcoming series of articles connecting the practices of systems thinking, causal analysis, and analytics.
Originally published at: https://www.eckerson.com/articles/looking-at-the-future-through-analytics-predictive-vs-prognostic
This blog is about Continuous Intelligence (CI) and how it integrates historical and real-time analytics to operate, monitor and tune systems of all types. Our next blogs will explore architectural approaches to CI, and how to navigate the trade offs it introduces to your organization.
Originally published at: https://www.eckerson.com/articles/continuous-intelligence-the-nexus-of-data-integration-analytics-and-operations
This blog is about Graphics Processing Units (GPU) and how much of the focus and industry growth in the use of GPUs has come from their suitability for machine learning, especially neural networks, more commonly known as deep learning.
Originally published at: https://www.eckerson.com/articles/gpu-databases-getting-more-value-from-your-machine-learning-infrastructure
This blog is about the challenge of data sprawl and how the combination of AI-based entity matching, schema matching, and enhanced knowledge graphing is moving us ever closer to the vision of self-driving data.
Originally published at: https://www.eckerson.com/articles/using-data-knowledge-to-conquer-data-sprawl
This EG blog is about data modernization steps and guiding principles that can help maintain balance within schools, stores, and other face to face businesses affected by COVID-19.
Originally published at: https://www.eckerson.com/articles/covid-19-and-higher-education-a-case-study-in-data-modernization
This is an audio blog on BI on the Cloud Data Lake and how to improve the productivity of data engineers. We'll dive deeper into the question; what’s the best measure of success for data pipeline efficiency? This is part 2 of a two part blog.
Originally published at: https://www.eckerson.com/articles/business-intelligence-on-the-cloud-data-lake-part-2-improving-the-productivity-of-data-engineers
This audio blog is about business intelligence on the cloud data lake and why it arose and how to architect for it. This is Part 1 of a two part blog series.
Originally published at: https://www.eckerson.com/articles/business-intelligence-on-the-cloud-data-lake-part-1-why-it-arose-and-how-to-architect-for-it
This audio blog is about the data lakehouse and how it is the latest incantation from a handful of data lake providers to usurp the rapidly changing cloud data warehousing market. It is one of three blogs featured in the data lakehouse series.
Originally published at: https://www.eckerson.com/articles/all-hail-the-data-lakehouse-if-built-on-a-modern-data-warehouse
This is an audio blog about the perplexities of the Data Lakehouse and if it is, indeed, the "paradigm of the decade". To hear more of Eckerson Group perspectives on the data lakehouse be sure to check out the blogs from colleagues, Wayne Eckerson and Kevin Petrie, and the recording of our recent Shop Talk discussion.
Originally published at: https://www.eckerson.com/articles/an-architect-s-view-of-the-data-lakehouse-perplexity-and-perspective
This audio blog discusses the Data Lakehouse, a marketing concept that evokes clean PowerPoint imagery, and why and how the New Cloud Data Lake will play a very real role in modern enterprise environments.
Originally published at: https://www.eckerson.com/articles/data-lakehouses-hold-water-thanks-to-the-cloud-data-lake
This audio blog discusses cloud adoption and how data teams will migrate an increasing portion of their on-premises operational and analytics workloads to the cloud. They can best meet budget and project requirements by using data streaming technologies such as change data capture (CDC), which replicates real-time updates between data source and target.
Originally published at: https://www.eckerson.com/articles/the-next-wave-of-cloud-migrations-needs-data-streaming
This audio blog is about how the CHOP’s data and analytics (DnA) team uses near real-time data and information to decide how to marshal its resources to contain the pandemic. The culmination of all of this work has been an enterprise COVID-19 dashboard that is distributed to enterprise leadership daily. Originally published at: https://www.eckerson.com/articles/chop-harnesses-the-power-of-data-analytics-to-address-the-covid-19-pandemic
This audio blog is about the emerging concept of the data mesh and how enterprises are working tirelessly to centralize diverse, ever-multiplying datasets by transforming mountains of data they don’t understand, into information that analysts do understand. Originally published at: https://www.eckerson.com/articles/the-data-mesh-re-thinking-data-integration
As of this writing, billions of consumers live in quarantine. They buy what they need online, comforting themselves with food, TV, and toilet paper. Nobody is splurging at the mall.
To say the least, it is an interesting time to analyze discretionary consumer behavior. As Director of the Voice of Consumer Analytics at Adidas, Tiankai helps measure and manage the perception of a consumer brand that is mentioned on social media an average of 260,000 times per day. An amateur musician, Tiankai went viral himself lately with his series of “Quarantunes,” songs such as “Self Quarantine” and “Parent in Quarantine,” that poke fun at our homebound predicament.
Tiankai recently spoke with Eckerson Group about the art and science of consumer analytics, the COVID-19 conundrum, and (of course) the role of creativity in modern data analysis.
This audio blog focuses on data storytelling and how it uses numbers, narrative, and visuals to communicate insights that would otherwise be hard to absorb. Originally published at: https://www.eckerson.com/articles/data-storytelling-part-i-telling-it-like-it-is-and-was-and-will-be
This audio blog focuses on increase usage of NLP to navigate different formats, languages, terminologies, and biases and how this technology will help analyze the fast-growing body of research on COVID-19. Originally published at: https://www.eckerson.com/articles/how-covid-19-will-drive-adoption-of-natural-language-processing
Chief data officers (CDOs) first appeared in enterprise organizations after the Sarbanes Oxley Act became law in the United States in 2002 to improve corporate governance controls. CDOs started with a trickle, but have since become a flood, now populating more than two-thirds of large enterprises, according to a recent survey by NewVantage Partners.
To explore this dynamic role in detail, we invited Joe Dossantos, newly minted CDO for the data and analytics software vendor Qlik. Joe is responsible for data governance, internal data delivery, and self-service enablement. He also evangelizes data and analytics best practices to Qlik customers.
Prior to joining Qlik, Joe led TD Bank’s data strategy, and built and ran the Big Data Consulting Practice for EMC Corporation's Professional Services Organization.
A chief data officer not only defines a data strategy to meet current needs but also evolves the strategy to ensure that the organization derives value far into the future.
Originally published at https://www.eckerson.com/articles/seven-core-responsibilities-of-a-chief-data-officer-cdo
Data leaders who launch self-service analytics programs without knowing their business users risk unleashing chaos. Data leaders need to canvas the organization and understand who produces what information for whom and where.
Originally published at https://www.eckerson.com/articles/succeeding-with-self-service-analytics-know-thy-customer
Master Data Management is no shiny object. But like many traditional IT practices, MDM is being severely tested – and rendered all the more strategic – by digitalization and rising data volumes.
Originally published at https://www.eckerson.com/articles/five-master-data-management-best-practices-for-enterprises
The rise of machine learning has placed a premium on finding new sources of data to fuel predictive models. But acquiring external data is often expensive and many data sets are rife with errors and difficult to combine with internal data. But that’s going to change in 2020.
To help us understand the scale, scope, and dimensions of emerging data marketplaces is Justin Langseth, one of the visionaries in our space. Justin is a VP at Snowflake responsible for the Snowflake Data Exchange. Prior to Snowflake, Justin was the technical founder and CEO/CTO of 5 data technology startups: Claraview (sold to Teradata), Zoomdata (sold to Logi Analytics), Clarabridge, Strategy.com, and Augaroo. He has 25 years of experience in business intelligence, natural language processing, big data, and AI.
Data quality and leadership trust levels may not seem connected, but they’re inextricably linked. Here’s why ...
Originally published at https://www.eckerson.com/articles/using-data-quality-to-build-trust-in-the-business-leaders
Data is critical for learning about the needs of the market, product bugs and issues, competitive solutions, and many other things. As such, analytics plays an important role in the innovation process.
Originally published at https://www.eckerson.com/articles/how-can-analytics-support-business-innovation
All organizations need to go down a similar path of data maturity. While you can skip steps in technology, you can’t skip steps in business data maturity.
Originally published at https://www.eckerson.com/articles/evolutionary-not-revolutionary-invest-less-in-new-tech-and-more-in-your-data-values
In this episode, Wayne Eckerson and Matthew Schwartz discuss non-traditional uses of business intelligence tools. Although BI tools have been around for almost three decades, most companies just scratch the surface of what’s possible to do with those tools. Using web layers and APIs, a company can use their imagination to customize and leverage their exiting BI tool-set to monetize data, integrate tribal knowledge and build industry-specific proprietary products.
Matthew Schwartz is the chief technology officer of Sage Hospitality, one of the world's largest hotel operators. Although Matt is responsible for all aspects of Sage’s IT operations, he has a deep fondness for data and analytics, having served as a BI director for several companies, including PetSmart and Staples. Matt firmly believes in the power of BI tools to transform organizations.
This audio blog probes the business value in analytics by examining the concept of Action Distance.
Originally published at https://www.eckerson.com/articles/finding-value-in-analytics-action-distance
One of the hardest parts of running a data analytics program inside a large organization is governing data and reports. It’s simply too easy for the definition of core data elements and metrics to get out of sync and reports to contain conflicting information.
Angie Davis has straddled both the business and IT worlds for more than 20 years. She served as a business analyst in several organizations before switching to the information technology side of the business where she ran analytics teams, first at JD Irving for six years and more recently at Brookfield Renewable where she is an IT director. Angie has a degree in mathematics and electrical engineering from Dalhousie University in Halifax, Nova Scotia.
This audio blog focuses on the importance of establishing a strong relationship between business and technical teams and describes various business engagement models that sit at the heart of all successful data analytics programs.
Learn how to achieve the DataOps objectives of improved efficiency and data quality by migrating to a streaming architecture based on Apache Kafka.
Companies that excel at advanced analytics and data science maximize the value of their data. They unearth hidden opportunities and become innovators in the industry. Although each organization has different goals, the underlying processes and tools to become successful at analytics remain somewhat the same. In this episode, Alan Jacobson explains them one by one and finishes off with his top three recommendations.
Alan Jacobson is the chief data and analytics officer (CDAO) of Alteryx, driving key data initiatives and accelerating digital business transformation for the Alteryx global customer base. As CDAO, Jacobson leads the company’s data science practice as a best-in-class example of how a company can get maximum leverage out of its data and the insights it contains, responsible for data management and governance, product and internal data, and use of the Alteryx Platform to drive continued growth.
Alan was recognized as a top leader in the global automotive industry as an Automotive Hall of Fame Leadership & Excellence award winner and an Outstanding Engineer of the Year by the Engineering Society of Detroit, and works with the National Academy of Engineering and other organizations as an advisor on data science topics.
With the growing popularity of machine learning and artificial intelligence, creating a data science program is a key initiative at most companies today. However, it’s not always clear to executives how they can deliver a return on investments in data science. To explain this, we invited an expert who has spent most of his career in the data science trenches and has a clear-minded perspective on how to deliver ROI with data science.
Alan Jacobson is the chief data and analytics officer (CDAO) of Alteryx, driving key data initiatives and accelerating digital business transformation for the Alteryx global customer base. As CDAO, Jacobson leads the company’s data science practice as a best-in-class example of how a company can get maximum leverage out of its data and the insights it contains, responsible for data management and governance, product and internal data, and use of the Alteryx Platform to drive continued growth.
Prior to joining Alteryx, Alan held a variety of leadership roles at Ford Motor Company across engineering, marketing, sales and new business development; most recently leading a team of data scientists to drive digital transformation across the enterprise. As an Alteryx evangelist at Ford, Alan spent many years leveraging the Alteryx Platform across the company and witnessed first-hand the impact a culture of analytics can have on the bottom line and what it takes to succeed as a data-driven enterprise.
How do you organize a data analytics program to maximize value for the organization? Although there is no right or wrong way to do this, several patterns emerge when you examine successful organizations.
Originally published at https://www.eckerson.com/articles/organizing-for-success-part-ii-how-to-organize-a-data-analytics-program
Despite our reliance on reports, the state of reporting in most companies is awful. There are too many reports in too many locations created by too many different people and tools.
The goal of self-service analytics is to empower business people to build their own reports, dashboards, and predictive models. If that happens, does your company still need a corporate business intelligence team?
Originally published at https://www.eckerson.com/articles/organizing-success-part-1-organize-bi-team
Although dashboards will never disappear, they will be radically transformed by artificial intelligence. They will become more intelligent, predictive, timely, and conversational.
Before a company hires data science talent, they should understand the role and types of data scientists. Failing to differentiate between research, applied, and citizen data scientist can result in appointing the wrong people on crucial projects. To continue our previous episode's discussion, we invited Alex Vayner for a second time to get an answer to the question: What is a data scientist?
Alex Vayner is a Partner and Americas Data & AI Practice Leader for PA Consulting Group, an innovation and transformation consultancy. Alex has spent his entire career in data & analytics, with his last five roles focused on building and running high-performance data science teams and capabilities in consulting and corporate environments. Before joining PA Consulting, Alex ran the NA Data Science & AI practice at Capgemini. He joined Capgemini from Equifax, where he served as VP, Global Data Innovation Leader, building a team responsible for pioneering disruptive data & analytics solutions for clients across all industries.
Explore how data analytics leaders can balance the use of specialists and generalists.
Originally published at https://www.eckerson.com/articles/specialists-or-generalists
New challenges to analytics platforms have prompted SAS to create new responses. The software giant responds with automation and decision support tools.
Data science has made immense progress, but companies are still stuck with the question: how do you use data science to deliver real value to the business? They hire dozens of data scientists and invest in state-of-the-art technology, but only a few have delivered ROI and business impact. In this episode, Wayne Eckerson and Alex Vayner discuss what organizations need to do for data science success.
Alex Vayner is a Partner and Americas Data & AI Practice Leader for PA Consulting Group, an innovation and transformation consultancy. Alex has spent his entire career in data & analytics, with his last five roles focused on building and running high-performance data science teams and capabilities in consulting and corporate environments. Before joining PA Consulting, Alex ran the NA Data Science & AI practice at Capgemini. He joined Capgemini from Equifax, where he served as VP, Global Data Innovation Leader, building a team responsible for pioneering disruptive data & analytics solutions for clients across all industries.
IoT has created a tidal wave that data savvy organizations can turn into profitable business solutions. Most IoT data comes from sensors, which are now attached to almost every device imaginable, from factory floor machines and agricultural fields to your cell phone and toothbrush. But IoT is forcing companies to rethink their data architectures to ingest, process, and analyze streaming data in real-time.
To help us understand the impact of IoT on data architectures, we invited Dan Graham to our show for a second time. Dan is a former product marketing manager at both IBM and Teradata, renowned for combining deep technical knowledge with industry marketing savvy. During his tenure at those companies, he was responsible for MPP data management systems, data warehouses, and data lakes, and most recently, the Internet of Things.
Just-in-time design is the practice of designing working software in small increments that support a business-defined need or story. Just-in-time design, as well as just-in-time testing, is an integral part of the agile software methodology. In fact, you can’t really do agile without just-in-time design.
To help us understand the nuances of just-in-time design, we invited Aaron Fuller, a long-time data architect and member of Eckerson Group’s consulting network. Across an 11-year career as the enterprise data architect for an insurance company, he modeled data, created technical designs for a broad range of systems, established governance and stewardship, and led the establishment of their enterprise data warehousing, business intelligence, and enterprise architecture programs. As principal consultant and owner of Superior Data Strategies since 2010, he leads a team of highly skilled data professionals who are uniquely capable of planning and executing agile data projects.
Last month, I attended Domo’s annual user conference for the first time. I came a skeptic, but left a believer. Domo has invested large sums of money to create a comprehensive data and analytics platform that scales to run small and medium-size businesses, and possibly large ones. Most importantly, it has a cadre of highly satisfied brand-name customers who want to extend the platform to support all business users and their analytic applications.
Originally published at: https://www.eckerson.com/articles/catching-the-domo-spirit
Processing continuous data streams is becoming increasingly important. However, traditional analytics architectures were often not built for real-time scenarios. This article will illustrate challenges and discuss how streaming-first approaches can change the way we think about analytics architectures.
Originally published at: https://www.eckerson.com/articles/streams-everywhere-towards-streaming-first-architectures
This second article in a series on modern data architectures. It focuses on what drives customers to want a modern data architecture (i.e., fear and opportunity) in the first place. It then lists ten requirements that customers desire for a modern data architecture, ranging from “cloud-first” and “streaming-first” to “best of breed” and “predictable pricing”.
Originally published at: https://www.eckerson.com/articles/ten-things-companies-want-from-a-modern-data-architecture
Data virtualization has been around for decades and has always been controversial. In the 1990s, it was called virtual data warehousing or VDW-- or as some skeptics liked to say, "voodoo and witchcraft”. It’s also been known as query federation and more recently, data services. The idea is that business users don't need to know the location of the data; they merely need to log into the data service and all data appears as if it’s local to their server, modeled in a fashion that makes sense to them.
Andrew Sohn is the Global Head of Data and Analytics at Crawford & Company, a $1B+ service provider to the insurance and risk management industry, where he designed and leads its data and digital transformation strategy and program. With more than 25 years in the industry, Andrew has managed a broad range of infrastructure and application technologies. He’s a strong advocate of data virtualization technology and believes it is an integral part of a modern, agile data ecosystem.
Why is Data Quality still an issue after all these years? To get an answer to the prevalent question, Wayne Eckerson and Jason Beard engage in a dynamic exchange of questions which lead us to the root cause of data quality and data governance problems. Using examples from his past projects, Jason shows the value of business process mapping and how it exposes the hidden problems which go undetected under the standard IT lens.
In his most recent role as Vice President of Process & Data Management at Wiley, a book publisher, he was responsible for master data setup and governance, process optimization, business continuity planning, and change management for new and emerging business models. Jason has led business intelligence, data governance, master data management, Process Improvement, Business Transformation, and ERP projects in a variety of industries, including Scientific and Trade publishing, Educational Technology, Consumer Goods, Banking, Investments, and Insurance.
Being a change agent is hard. It's tough to inspire people and get them motivated to work on a shared vision. To understand the mechanics of digitalization and tactics required to implement them, Wayne Eckerson invited Andrea Ballinger so that she could share her hard-won lessons from her illustrious career as a technology leader. Andrea is currently leading a transformation program at LSU, revamping the university’s information technology resources across multiple campuses. Prior to that, she served as Interim CEO and President for the University of Illinois Alumni Association and CTO of Illinois State University. She began her data career at the University of Illinois where she earned a reputation as the foremost data warehousing expert in higher education.
The road to AI adoption is far more complex than one can imagine. Building data science models and testing them is only one piece of the puzzle. To understand the roadblocks and best practices, Wayne Eckerson invited Nir Kaldero in our latest episode to learn why organizations need to start paying more attention to people, culture and processes to make data science projects a success and how democratization skills pays off in the long run.
Nir Kaldero is the Head of Data Science, Vice President at Galvanize Inc. and the creator of the GalvanizeU Master’s of Science in Data Science program. A tireless advocate for transforming education and reshaping the field of data science, his vision and mission is to make an impact on a wide variety of communities through education, science, and technology. In addition to his work at some of the world’s largest international corporations, Kaldero serves as a Google expert/mentor and has been named an IBM Analytics Champion 2017 & 2018, a prestigious honor given to leaders in the field of science, technology, engineering, and math (STEM).
In this episode, Daniel Graham dissects the capabilities of data lakes and compares it to data warehouses. He talks about the primary use cases of data lakes and how they are vital for big data ecosystems. He then goes on to explain the role of data warehouses which are still responsible for timely and accurate data but don't have a central role anymore. In the end, both Wayne Eckerson and Dan Graham settle on a common definition for modern data architectures.
Daniel Graham has more than 30 years in IT, consulting, research, and product marketing, with almost 30 years at leading database management companies. Dan was a Strategy Director in IBM’s Global BI Solutions division and General Manager of Teradata’s high-end server divisions. During his tenure as a product marketer, Dan has been responsible for MPP data management systems, data warehouses, and data lakes, and most recently, the Internet of Things and streaming systems.
Recent technology developments are driving urgency to modernize data management. What do you do about architecture, modeling, quality, and governance to keep up with big data, cloud, self-service, and other trends in data and technology? Examining some best practices can spark ideas of where to begin.
Originally published at https://www.eckerson.com/articles/stepping-up-to-modern-data-management
Data catalogs have become the centerpiece of modern data management. They are the means to connect self-service analysts with the right data, the “go to” technology for data curation, and the new gold standard for metadata management. With many data catalog tools available, choosing the right data catalog is an important decision.
In this episode, Wayne Eckerson asks Steve Dine about the approach needed to migrate to the Cloud and architecture required to run analytics in the Cloud. Steve Dine talks extensively about the pitfalls to avoid during Cloud migration and finishes off by saying that even though security is a big issue, most organizations will have part of their architecture in the Cloud during the next two-three years. Steve Dine is a BI and enterprise data consultant and industry thought leader who has extensive experience in designing, delivering and managing highly scalable and maintainable modern data architecture solutions.
BI teams often work extremely hard but have little to show for their effort, and they can never get ahead of a continuous backlog of requests. The business rewards their hard work by reducing their budget and staff. Even if they know what needs to be done, they have little authority to make it happen. To succeed, BI leaders need to partner with the business and deliver quick wins to turn the tide of adoption in their favor.
Originally published at https://www.eckerson.com/articles/why-bi-teams-struggle-the-tipping-point-of-success
In this Episode, Wayne Eckerson asks Charles Reeves about his organization’s Internet of Things and Big Data strategy. Reeves is senior manager of BI and analytics at Graphics Packaging International, a leader in the packaging industry with hundreds of worldwide customers. He has 25 years of professional experience in IT management including nine years in reporting, analytics, and data governance.
A data analyst workbench will inevitably integrate data catalog, data preparation, and data analysis functionality. Data analysts don’t want to jump from tool to tool when executing a workflow that is both linear and iterative. Data analysts with this kind of workbench will be more productive and foster higher levels of reuse and data literacy.
Originally published at https://www.eckerson.com/articles/self-service-triumvirate-the-new-data-analyst-workbench
In this episode, Wayne Eckerson and Shakeeb Ahkter dive into DataOps. They discuss what DataOps is, the goals and principles of DataOps, and reasons to adopt a DataOps strategy. Shakeeb also reveals the benefits gained from DataOps and what tools he uses. He is the Director of Enterprise Data Warehouse at Northwestern Medicine and is responsible for direction and oversight of data management, data engineering, and analytics.
A classic management practice dictates that a newly-appointed leader must accomplish certain things in their first 90 days. While some of this is general knowledge, there are specifics when it comes to Data Management and Analytics If you have been recently named to head any group that has to manage or facilitate the use of data, at any level in the organization, then this audio blog post is for you.
Originally published at https://www.eckerson.com/articles/what-do-you-do-first-after-being-hired-as-a-bi-analytics-data-engineering-director
In this episode, Wayne Eckerson and Rich Fox discuss what differentiates data science from analytics, why and how data science addresses business needs, why balance scorecards are relevant, and why Excel is a problem. Throughout the podcast, Fox shares many real-life examples and personal experiences.
Fox is vice president of Data Science and Analytics at Apex Parks Group, one of the largest entertainment center companies in the United States, which operates amusement parks, water parks, and family entertainment centers.
With all the hype and attention around big data and huge data platforms, there can sometimes be some data envy. There are still organizations and companies that don’t have big data: are they not poised for analytics too? Can they not get insights as well? The BI Pharaoh gives tips on how to work with your little data just like the big boys.
Originally published at https://www.eckerson.com/articles/little-data-needs-love-too
We’re at the dawn of a new era in decision making made possible by the intersection of business intelligence and artificial intelligence. Rather than replace BI, AI will make BI more pervasive. AI-infused BI tools will be easier to use, generate more useful insights, and make business users more productive. Rather than replace human decision makers, AI will free them to focus on value-added activities and make decisions with data rather than rely solely on gut instinct.
Originally published at https://www.eckerson.com/articles/the-impact-of-ai-on-analytics-machine-generated-intelligence
In this episode, Wayne Eckerson and Rich Galan discuss the obstacles to delivering timely analysis, the problems that large volumes of data create, solutions to those issues, and where BI is headed in the near future. Rich is a veteran data analytics leader with 20 years of experience in a variety of data-driven organizations.
Data analysts who sit in each business function (i.e., sales, marketing, finance) are critical to the success of a self-service analytics strategy. The problem is that most data analysts don’t receive the training and support they need to be proficient with self-service data and analytics tools. The easiest way to improve the skills and satisfaction of most data analysts is simple: bring them together into a power user network.
Originally published at https://www.eckerson.com/articles/power-user-networks-the-key-to-self-service-analytics
In this episode, Wayne Eckerson and Steve Dine discuss modernizing data architectures. They cover the definition of a modern data architecture, telltale signs you need to modernize, best practices to modernize, and much more.
Dine is the President of Datasource Consulting, an EXL company. He has extensive experience designing, delivering, and managing successful, highly scalable and maintainable modern data architecture solutions.
Data pipelines become chaotic with pressures of agile, democratization, self-service, and organizational “pockets” of analytics. From enterprise BI to self-service analysis, data pipeline management should ensure analysis results are traceable, reproducible, and of production strength. Robust data pipelines rely on eight critical components.
Originally published at https://www.eckerson.com/articles/the-complexities-of-modern-data-pipelines
In this episode, Wayne Eckerson and Jen Underwood explore a new era of analytics. Data volumes and complexity have exceeded the limits of current manual drag-and-drop analytics solutions. Data moves at the speed of light while speed-to-insight lags farther and farther behind. It is time to explore intelligent, next generation, machine-powered analytics to retain your competitive edge. It is time to combine the best of the human mind and machine.
Underwood is an analytics expert and founder of Impact Analytic. She is a former product manager at Microsoft who spearheaded the design and development of the reinvigorated version of Power BI, which has since become a market leading BI tool. Underwood is an IBM Analytics Insider, SAS contributor, former Tableau Zen Master, Top 10 Women Influencer and active analytics community member. She is keenly interested in the intersection of data visualization and data science and writes and speaks persuasively about these topics.
In this podcast, Carl Gerber and Wayne Eckerson discuss Gerber’s top five data governance best practices: Motivation, Assessment, Data Assets Catalog, CxO Alliance, and Data Quality.
Gerber is a long-time chief data officer and data leader at several large, diverse financial services and manufacturing firms, who is now an independent consultant and an Eckerson Group partner.
He helps large organizations develop data strategies, modernize analytics, and establish enterprise data governance programs that ensure data quality, operational efficiency, regulatory compliance, and business outcomes. He also mentors and coaches Chief Data Officers and fills that role on an interim basis.
With today’s unprecedented velocity of change in data and technologies, the single version of the truth (SVOT) is sometimes looked upon as a nice to have. But SVOT is not optional. Despite the difficulties in constructing a single version of the truth, it is now needed more than ever to keep CEOs happy and to stay compliant with new regulations like the GDPR.
Originally published at https://www.eckerson.com/articles/single-version-of-the-truth-not-optional
Ethics is challenging because right and good are not always clear. More data, more kinds of data, and advanced analysis of data often conflict with concerns of data privacy, security, anonymity, and ownership. Resolving these conflicts requires acknowledgment, discussion, and the hard work of defining ethics-based policies and creating a culture of ethical conduct.
Originally published at https://www.eckerson.com/articles/data-ethics-the-new-data-governance-challenge
In this episode, Wayne Eckerson and Jeff Magnusson discuss the data architecture Stitch Fix created to support its data science workloads, as well as the need to balance man and machine and art and science.
Magnusson is the vice president of data platform at Stitch Fix. He leads a team responsible for building the data platform that supports the company's team of 80+ data scientists, as well as other business users. That platform is designed to facilitate self-service among data scientists and promote velocity and innovation that differentiate Stitch Fix in the marketplace. Before Stitch Fix, Magnusson managed the data platform architecture team at Netflix where he helped design and open source many of the components of the Hadoop-based infrastructure and big data platform.
To survive and thrive in today’s information economy, organizations need to establish a compelling data strategy that advances the company’s key objectives and goals. This is not a job for the chief information officer or even chief data officer; it’s a task that the CEO must own and champion and assemble his best and brightest to execute. This audio blog examines the ten step methodology that Eckerson Group uses to help organizations create a data strategy.
Originally published at https://www.eckerson.com/articles/how-to-create-a-data-strategy-part-1-overview
Data engineering is one of the hottest and most difficult jobs to fill in the field of analytics. Breadth and depth of required skills limits the number of people qualified to work as data engineers. If you’re seeking to hire data engineers, consider the 24 skill areas identified here as guidance to shape job descriptions and to screen candidates. If you’re seeking to become a data engineer, take the skills assessment to highlight your strengths and identify your gaps.
Originally published at https://www.eckerson.com/articles/data-engineering-coming-of-age
In this podcast, Wayne Eckerson and Joe Caserta discuss data migration, compare cloud offerings from Amazon, Google, and Microsoft, and define and explain artificial intelligence.
You can contact Caserta by visiting caserta.com or by sending him an email to joe@caserta.com. Follow him on Twitter @joe_caserta.
Caserta is President of a New York City-based consulting firm he founded in 2001 and a longtime data guy. In 2004, Joe teamed up with data warehousing legend, Ralph Kimball to write to write the book The Data Warehouse ETL Toolkit. Today he’s now one of the leading authorities on big data implementations. This makes Joe one of the few individuals with in-the-trenches experience on both sides of the data divide, traditional data warehousing on relational databases and big data implementations on Hadoop and the cloud.
Every organization that manufactures data for decision making is rethinking its data architecture. Compared to five years ago, there is a treasure trove of new technologies and techniques that promise to transform the way organizations compete and serve customers. This audio blog summarizes the major characteristics of a modern data architecture and serves a high-level guide for organizations that are in the midst of developing a new data strategy for the modern age.
Originally published at https://www.eckerson.com/articles/ten-characteristics-of-a-modern-data-architecture
In this podcast, Wayne Eckerson and James Serra discuss myths of modern data management. Some of the myths discussed include 'all you need is a data lake', 'the data warehouse is dead', 'we don’t need OLAP cubes anymore', 'cloud is too expensive and latency is too slow', 'you should always use a NoSQL product over a RDBMS.'
Serra is big data and data warehousing solutions architect at Microsoft with over thirty years of IT experience. He is a popular blogger and speaker and has presented at dozens of Microsoft PASS and other events. Prior to Microsoft, Serra was an independent data warehousing and business intelligence architect and developer.
In this podcast, Henry Eckerson interviews Dave Wells on the current health and future of the data warehouse. Wells acknowledges that data warehouses are struggling, but argues they are still necessary and cannot be replaced by data lakes. He then explains what the role of the modern data warehouse should be, practical steps forward for evolving the data warehouse, and much more.
Wells is an advisory consultant, educator, and industry analyst dedicated to building meaningful connections throughout the path from data to business value. He works at the intersection of information management and business management, driving business impact through analytics, business intelligence, and active data management. More than forty years of information systems experience combined with over ten years of business management give him a unique perspective about the connections among business, information, data, and technology. Knowledge sharing and skill building are Dave’s passions, carried out through consulting, speaking, teaching, and writing.
He is now the practice director of data management at Eckerson Group, cofounder and director of education at eLearningCurve, and a faculty member at The Data Warehousing Institute.
In this episode, Wayne Eckerson and Jeff Magnusson discuss a self-service model for data science work and the role of a data platform in that environment. Magnusson also talks about Flotilla, a new open source API that makes it easy for data scientists to execute tasks on the data platform.
Magnusson is the vice president of data platform at Stitch Fix. He leads a team responsible for building the data platform that supports the company's team of 80+ data scientists, as well as other business users. That platform is designed to facilitate self-service among data scientists and promote velocity and innovation that differentiate Stitch Fix in the marketplace. Before Stitch Fix, Magnusson managed the data platform architecture team at Netflix where he helped design and open source many of the components of the Hadoop-based infrastructure and big data platform.
In this episode, Wayne Eckerson and Mike Masciandaro discuss keys to creating successful reports that users will love and use. Masciandaro provides in-depth explanations of each key (easy, drill, monitor, accurate, relevant, timely, responsive, and secure).
Masciandaro is a veteran business intelligence practitioner who recently retired from an illustrious career at Dow Chemical as director of BI. During that time, Mike saw and did just about everything there is to do in the world of BI, data, and analytics. He is now intent on sharing his hard-won knowledge with others.
In this episode, Wayne Eckerson and Lenin Gali discuss the past and future of the cloud and big data.
Gali is a data analytics practitioner who has always been on the leading edge of where business and technology intersect. He was one of the first to move data analytics to the cloud when he was BI director at ShareThis, a social media based services provider. He was instrumental in defining an enterprise analytics strategy, developing a data platform that brought games and business data together to enable thousands of data users to build better games and services by using Hadoop & Teradata while at Ubisoft. He is now spearheading the creation of a Hadoop-based data analytics platform at Quotient, a digital marketing technology firm in the retail industry.
In this podcast, Henry Eckerson and Stephen Smith discuss the movement to operationalize data science.
Smith is a well-respected expert in the fields of data science, predictive analytics and their application in the education, pharmaceutical, healthcare, telecom and finance industries. He co-founded and served as CEO of G7 Research LLC and the Optas Corporation which provided the leading CRM / Marketing Automation solution in the pharmaceutical and healthcare industries.
Smith has published journal articles in the fields of data mining, machine learning, parallel supercomputing, text understanding, and simulated evolution. He has published two books through McGraw-Hill on big data and analytics and holds several patents in the fields of educational technology, big data analytics, and machine learning. He holds a BS in Electrical Engineering from MIT and an MS in Applied Sciences from Harvard University. He is currently the research director of data science at Eckerson Group.
In this episode, Mike Masciandaro and Wayne Eckerson discuss how to partner with the business and ensure high levels of customer satisfaction and adoption.
Mike is a veteran business intelligence practitioner who recently retired from an illustrious career at Dow Chemical. Mike has seen and done just about everything there is to do in the world of BI, data, and analytics. He is now intent on sharing his hard-won knowledge with others.
In this podcast, Wayne Eckerson and Joe Caserta discuss what constitutes a modern data platform. Caserta is President of a New York City-based consulting firm he founded in 2001 and a longtime data guy. In 2004, Joe teamed up with data warehousing legend, Ralph Kimball to write to write the book The Data Warehouse ETL Toolkit. Today he’s now one of the leading authorities on big data implementations. This makes Joe one of the few individuals with in-the-trenches experience on both sides of the data divide, traditional data warehousing on relational databases and big data implementations on Hadoop and the cloud. His perspectives are always insightful.
Dewayne Washington is back this week for part II of his Secrets of Data Analytics Leaders podcast with Eckerson Group. In part I, Dewayne and I discussed the role of the CIO. In this episode we discuss the keys to IT success.
Washington is a senior consultant with 20+ years of experience in BI and Analytics in over two dozen verticals. He is the former BI manager at Dallas/Fortworth International Airport and the current CIO at The Business of Intelligence. He is also the author of the book Get In The Stream, the ultimate guide to customer adoption, and his Data Warehousing and Mobile Solutions implementations have been featured in CIO Magazine and the Wall Street Journal. Washington is also a sought-after speaker and mentor for organizations striving to leverage BI and Analytics to meet business goals, thus earning him the title, BI Pharaoh.
In this podcast, Dewayne Washington speaks the unadulterated truth about the role of the CIO and discusses keys to success and common pitfalls. Washington is a senior consultant with 20+ years of experience in BI and Analytics in over two dozen verticals. He is the former BI manager at Dallas/Fortworth International Airport and the current CIO at The Business of Intelligence. Washington is also a sought-after speaker and mentor for organizations striving to leverage BI and Analytics to meet business goals, thus earning him the title, BI Pharaoh.
Mike Masciandaro is a veteran business intelligence practitioner who recently retired from an illustrious career at Dow Chemical. Mike has seen and done just about everything there is to do in the world of BI, data, and analytics. He is now intent on sharing his hard-won knowledge with others. You will learn the definition and purpose of a BI program, the role of subject matter experts, how to hire and retain talent, keys to delivering value as a BI program, and more.
In this podcast, Kevin Sonsky reveals the secrets to his success as a business intelligence leader at Citrix Systems. During the past 11 years, he has implemented an enterprise-wide self-service reporting environment that has delivered deeper insights into customer purchasing behavior. At the same time, he has established a grassroots governance program that has successfully standardized on dozens of key enterprise metrics and reports. Kevin is interviewed by Wayne W. Eckerson, long-time thought leader in the business analytics field.