Name: Building a Data Science Platform on Apache Iceberg and Nessie
Start: 2023-10-25T18:55:00Z
End: 2023-10-25T18:55:32.000Z
Location: BrightTALK
Rating: 0

Dremio is the easy and open data lakehouse, providing self-service analytics with data warehouse functionality and data lake flexibility across all of your data. Dremio increases agility with a revolutionary data-as-code approach that enables Git-like data experimentation, version control, and governance.

Data integration is the foundation of modern business. When organizations consolidate the ingestion, cleansing, and transformation of disparate data sources into high-performance pipelines, they can drive analytics insights into every decision.

In order to keep pace with fast-changing business requirements, enterprises must keep pace with best practices in data integration. Chief among these is migrating data integration pipelines to scalable, resilient, and agile cloud platforms.

In this panel discussion video, TDWI senior research director James Kobielus will engage data industry experts in an in-depth discussion of data integration trends and best practices. 

The discussion will focus on several key issues:
- What are the hallmarks of modern data integration?
- What trends are spurring enterprises to modernize their data integration capabilities?
- Why should enterprises modernize their data integration platforms, processes, and organizations?
- What core capabilities are essential for deploying a full-featured enterprise data integration stack?
- What emerging data integration best practices are needed to support sophisticated new use cases in artificial intelligence, distributed analytics, and low-latency streaming?
- What new techniques should enterprises consider for reducing the cost and improving the efficiency of their data integration processes?
- How feasible is it for enterprises to entirely automate their data integration processes?
- To what extent can and should self-service tools be used to help business analysts and other nontraditional roles build and deploy sophisticated data integration pipeline logic?
- What the essential first step for enterprises on their journeys to modern data integration?

Watch this video and learn the data integration trends and best practices for your organization.

Expert Panel Discussion – Data Integration Trends and Best Practices Webinar

In this engaging talk with Alex Merced, Developer Advocate, we'll explore how Dremio revolutionizes data access, delivering speed, simplicity, and substantial cost savings. 

Discover the power of Dremio as we dive deep into:
- Data Access at Lightning Speed: Learn how Dremio accelerates data access, making insights available in real-time.
- Simplicity in Data Preparation: Streamline your data pipeline with Dremio's intuitive interface for data transformation.
- Cost Efficiency: Uncover how Dremio’s optimizations save you money while improving performance
- Use Cases: Explore real-world success stories and applications of Dremio's data access solutions.
- Future-Proofing Your Data Infrastructure: Understand how Dremio ensures scalability and adaptability.

Watch this video to uncover the secrets of fast, easy data access without breaking the bank!

How Dremio provides you fast and easy data access while saving you money

Organizations are struggling with the proliferation of toolings in their data infrastructure and the exponential growth of ETL pipelines are slowing down data engineers to deliver value to the business. They want to spend more time making impactful decisions and working on high value projects. Fivetran significantly reduces the amount of time spent in building ETL pipelines with their no-code approach. Dremio is the easy and open data lakehouse, providing self-service analytics with data warehouse functionality and data lake flexibility across all your data. Together, Dremio and Fivetran bring the best solution for enabling organizations to GTM faster.

In this video, you will learn:
- What Iceberg table format is and why it matters in data lakehouses
- How to load source files into Iceberg tables using Fivetran 
- How to create a unified access layer for your data with Dremio Cloud

How To Build an Iceberg Data Lakehouse with Fivetran and Dremio

Product analytics offers a transformative opportunity for companies to elevate the customer experience and offer a way to proactively understand customer behavior. This personalized understanding allows companies to tailor their product offerings, provide targeted recommendations, and streamline customer journeys, resulting in a more engaging, satisfying, and loyalty-inducing customer experience. Effective product analytics is a comprehensive strategy to proactively manage support and promote customer success.

NetApp, a leading global company specializing in hybrid cloud data services, helps enterprises build a simple and secure way to drive innovation wherever their data and applications live. The customer experience is a core driver within NetApp’s portfolio of solutions offering.

Watch Aaron Sims, Technical Director at NetApp as he shares his experience building out a unified access layer for product analytics with Dremio.

In this episode, you will learn:
- NetApp’s journey to unified analytics with Dremio’s phased approach for Hadoop modernization
- How a unified access layer makes data easier to discover and explore for your end users without data duplication
- Ways to maximize your existing infrastructure investments for improved ROI and lower TCO with Dremio.

How NetApp is Redefining the Customer Experience with Product Analytics

Watch this video to learn about the common challenges of using traditional file formats on premises and how leveraging Apache Iceberg on AWS helps you overcome these challenges. You will also learn about the comprehensive and advanced features of Apache Iceberg with elaborate demos that showcase the unique capabilities of Apache Iceberg on AWS.

Dive deep of Apache Iceberg on AWS

Organizations who want to leverage their data lake for insights often struggle to deliver a consistent, accurate, high-quality view of their data to all of their data consumers. That challenge is often exacerbated by the need to make changes to data that impacts multiple tables.

In this video, we’ll share how data teams can use Dremio Arctic, a data lakehouse management service, to simplify data management and operations. Using Git for Data capabilities like branching, tagging, and commits, we’ll show how Dremio Arctic makes it easier than ever to:

- Create zero-copy clones of your data lake so data consumers can work on production-quality data without impacting other users.
- Quickly make updates to all of your tables and merge those changes atomically, so every user has access to an accurate and consistent view of the data lake.
- Reduce the costs and complexities associated with data lakehouse management.

Simplify Lakehouse Operations with Zero-Copy Clones and Multi-Table Transactions

This talk will present how Nessie, Iceberg, and Great Expectations is used to build a DataOps pipeline that ensures data quality and avoids “datastrophes.”

DataOps in action with Nessie, Iceberg and Great Expectations

Watch this live fireside chat with David Stodder, Senior Director of Research Business Intelligence at TWDI, and Nik Acheson, Senior Product and Strategy Leader at Dremio, as they talk about using Data Mesh to Advance Distributed Data Access, Agility and Governance. During this informative session, you will learn:

- Best practices for success in the data mesh journey so you can make it easier to discover, understand, and trust data
- The importance of metadata catalogs, business glossaries, and data intelligence for integrating discovery, access, and governance
- How data mesh, data fabrics, and data virtualization differ and are related
- The role of an open data lakehouse in a distributed data architecture
- Balancing self-service data domains with requirements for enterprise data governance
- Sorting out data virtualization, data mesh and data fabrics
- Role of metadata catalogs, business glossaries and semantic layer
- Data mesh and the open data lakehouse: How they fit together
- The data mesh journey: Lessons learned and best practices
- Improving the user experience and increasing business value

Using Data Mesh to Advance Distributed Data Access, Agility and Governance

Wie bringt ein Data Lake House die Smart Factory auf ein neues Level?

WORUM ES GEHT:
Erhalten Sie konkrete und aussagekräftige Einblicke in erfolgreich umgesetzte Projekte von führenden Industrieunternehmen.

ERFOLGSBEISPIELE IM FOKUS:
Octotronic zeigt ausgewählte Kundenprojekte, in denen die Integration von Daten die Produktionsleistung, die Ausschussquoten und die Entscheidungsfindung verbessert und so Schritt für Schritt den Weg zur intelligenten Fabrik ebnet.

Dremio erklärt das Data Lakehouse und wie durch die hohe Benutzerfreundlichkeit und Kosteneffizienz das Datenmanagement vereinfacht und damit schnellere (BI) Geschäftslösungen ermöglicht werden.

Mehr Zeit für Erkenntnisse durch digitale Produkte

Wie Sie Self-Service ermöglichen - Benutzerfreundlichkeit für Datenwissenschaftler, Analysten und Prozessingenieure
Wie Sie kosteneffizienter werden - für Ihre Daten vor Ort, in der Cloud und als Hybridlösung
FÜR WEN:
Ideal für Führungskräfte/Profis, die sich mit Digitalisierung der Produktion, digitaler Transformation, Entwicklung und Innovation beschäftigen

Verpassen Sie nicht, wie andere zu Smart Factories werden und nutzen Sie die Erfahrungen!

Smart Data - Smart Factory with Octotronic and Dremio (German)

Network protocols for transferring data generally have one of two problems: they’re slow for large data transfers but have simple APIs (e.g. JDBC) or they’re fast for large data transfers but have complex APIs specific to the system. Apache Arrow Flight addresses the former by providing high performance data transfers and half of the latter by having a standard API independent of systems. However, while the Arrow Flight API is performant and an open standard, it can be more complex to use than simpler APIs like JDBC.

Arrow Flight SQL rounds out the solution, providing both great performance and a simple universal API.

In this video, we’ll show the performance benefits of Arrow Flight, the client difference between interacting with Arrow Flight and Arrow Flight SQL, and an overview of a JDBC driver built on Arrow Flight SQL, enabling clients to take advantage of this increased performance with zero application changes.

Arrow Flight SQL High Performance, Simplicity, Interoperability Data Transfers

Imagine fast, intuitive analytics on all of your data where it lives - with the power of a data warehouse and the scale of a data lake. Dremio's open data lakehouse makes it easy to access, understand, and analyze all your data with a lightning fast SQL query engine and low-code and no-code options for all users. Learn what’s new in Dremio - including next gen Reflections SQL acceleration - and how you can accelerate self-service analytics at scale.

We’ll cover:
- Next gen Reflections SQL acceleration  and new Reflection Recommender to automatically create Reflections for your most important queries
- New Generative AI capabilities for text-to-SQL and more to make it possible for all users to interact with data
- Expanded table format support, including time-travel for Delta Lake
- Enhancements to our lightning-fast query engine that deliver even faster, more intelligent interactive analytics
- Our native Apache Iceberg lakehouse catalog, Dremio Arctic, now in Preview

What’s New in Dremio, including Next Gen Reflections

In the world of data acceleration and optimization, both materialized views and Dremio's Data Reflections stand out as pivotal tools. Our upcoming webinar titled "Materialized Views and Dremio's Reflections" aims to demystify these technologies, comparing their benefits, limitations, and unique features. 

Dive deep into understanding the core differences between materialized views and Dremio's Reflections. Whether you're a seasoned data professional or just starting out, this webinar offers insights to optimize your data strategy. Discover the nuances, best practices, and real-world applications of these powerful tools, and make informed decisions for your data architecture.

This video will cover:
- What are the shortcomings of Materialized Views
- How Dremio’s Reflections Overcome those shortcomings
- How Reflections can allow you reduce the number of pipelines you have write and maintain.

Materialized Views and Dremio Reflections

Using Rust, Apache Arrow, and table formats, data can be efficiently processed closer to the hardware and without any pauses. In this video, it will explain the pros and cons of Apache Arrow for data processing and compare the performance with Apache Spark — the "standard" in terms of distributed processing of big data. 

We will discuss the advantages of the Rust language, including Rust Arrow and the tools available, the missing pieces, and performance comparisons.

Fast Data Processing with Apache Arrow

Apache Iceberg’s rich metadata is its secret sauce, powering core features like time travel, query optimizations, and optimistic concurrency handling.  But did you know that this metadata is accessible to all, via easy-to-use system tables? 

This talk will walk through real-life examples of using metadata tables to get even more out of Iceberg and address questions such as: 
- What is the last partition updated and when? 
- Why are there too many small files? 
- What Iceberg maintenance procedures can give us better query performance? 
- Can we start building more advanced systems like data audit and data quality? 
- How many null values are being added per hour? 
- What is the latency of data ingest over time? 
- We will also cover metadata table performance tips and tricks, and ongoing improvements in the community. 

Whether you are already using Iceberg metadata tables or interested in getting started, watch this talk to learn how this under-utilized feature can help manage data tables more effectively than ever before.

Apache Iceberg's Best Secret: A Guide to Metadata Tables

In the rapidly evolving landscape of big data, Data Lakehouse is heralding a new age of unified analytics, blending the best elements of data lakes and data warehouses. Central to this convergence is the need for advanced table formats that can meet the demands of scalability, performance, and data reliability. This webinar dives deep into the world of Data Lakehouse table formats, specifically focusing on Apache Iceberg, Delta Lake, and Apache Hudi.

Who should watch this video?
Data engineers, data architects, data analysts, and other professionals interested in modernizing their data platform or seeking deeper insights into the technicalities and advantages of these advanced table formats.

Key Takeaways:
- Introduction to Data Lakehouse: Explore the genesis of the Data Lakehouse paradigm, its significance, and how it’s reshaping the way organizations think about big data storage and analytics.

- Demystifying Apache Iceberg, Delta Lake, and Apache Hudi: Understand the intricacies of these popular table formats, their architectural nuances, and how they differ from traditional table structures.

- Features Spotlight: Delve into the unique feature sets that each format brings to the table - from ACID transactions, time-travel queries, to efficient upserts and scalability features.

- The Relevance Quotient: Understand why these table formats matter in today's data-driven world. Learn about their roles in ensuring data consistency, improving query performance, and facilitating near real-time analytics on large datasets.

- Best Practices and Use Cases: Explore real-world scenarios where organizations have leveraged these formats to transform their data analytics operations, and glean best practices for successful implementation and optimization.

Equip yourself with the knowledge to harness their power, ensuring a robust and efficient data infrastructure for your organization.

The Who, What and Why of Data Lakehouse Table Formats

The data lakehouse is an architectural strategy that combines the flexibility and scalability of data lake storage with the data management, data governance, and data analytics capabilities of the data warehouse. As more organizations adopt this architecture, data teams need a way to deliver a consistent, accurate, and performant view of their data for all of their data consumers. In this webinar, we will share how Dremio Arctic, a data lakehouse management service:

- Enables easy catalog versioning using data as code, so everyone has access to consistent, accurate, and high quality data.
- Automatically optimizes Apache Iceberg tables, reducing management overhead and storage costs while ensuring high performance on large tables.
- Eliminates the need to manage and maintain multiple copies of the data for development, testing, and production.

Introduction to Dremio Arctic: Catalog Versioning and Iceberg Table Optimization

Watch this an insightful webinar video titled "ELT, ETL, and the Dremio Data Lakehouse," where we explore the cutting-edge capabilities of Dremio in revolutionizing data engineering and analytics workflows. This webinar delves into the strategic use of Dremio's innovative technologies to optimize Extract, Load, Transform (ETL) and Extract, Load, Transform (ELT) patterns for enhanced efficiency and cost-effectiveness.

The session will commence with an in-depth exploration of traditional ETL and ELT methodologies, highlighting the challenges faced by organizations in managing large-scale data transformations. We will analyze the critical role of ELT patterns in the modern data landscape and the growing significance of data lakes for storage and processing.

Subsequently, we will introduce Dremio, a powerful and flexible data lakehouse platform, as a game-changer for executing ETL and ELT operations. Dremio's unique architecture empowers users to directly query data residing in the data lake, eliminating the need for unnecessary data copies and reducing data movement overhead significantly.

During the webinar video, attendees will gain valuable insights into how Dremio's no-copy architecture minimizes data redundancy, accelerates data processing, and drastically reduces the associated costs. By harnessing the full potential of data lake storage, organizations can simplify their data engineering workflows, enhance data availability, and achieve unparalleled performance for analytical workloads.

Key webinar takeaways:
- A comprehensive overview of ETL and ELT patterns and their relevance in modern data environments.
- The rise of data lakes and the pivotal role of Dremio's data lakehouse platform in transforming data management paradigms.
- Understanding the benefits of Dremio's no-copy architecture in optimizing data processing and analytics.
- Best practices and practical implementation tips for leveraging Dremio effectively in ETL and ELT workflows.

ELT, ETL and the Dremio Data Lakehouse

Watch this insightful webinar featuring Jacopo Tagliabue of Bauplan Labs as he dives into the world of data science and machine learning pipelines. In this video, you'll discover the rationale behind Bauplan Labs' choice of open-source technologies, such as Apache Iceberg table format and Project Nessie transactional data catalog, for their cutting-edge platform. Gain valuable insights into why modern data platforms are increasingly adopting these technologies and how Nessie's git-like features can revolutionize your data management. Don't miss out on this opportunity to stay ahead in the world of data science and technology!

About Project Nessie - https://www.dremio.com/blog/introducing-nessie-as-a-dremio-source/

What you will learn:
- Why Modern Data Platforms are being built on Apache Iceberg
- Why Modern Data Platforms are being built on Nessie

Building a Data Science Platform on Apache Iceberg and Nessie

Project Nessie

Apache Iceberg

Data Management

Open Source

Data Architecture

Machine Learning

pipelines

Open Lakehouse

Data Science

Bauplan Labs

Welcome to the virtualization community on BrightTALK! Whether it affects servers, storage, networks, desktops or other parts of the data center, virtualization provides real benefits by reducing the resources needed for your
infrastructure and creating software-defined data center components. However, it can also complicate your infrastructure. Join this active community to learn best
practices for avoiding virtual machine sprawl and other common virtualization pitfalls as well as how you can make the most of your virtualization environment.

Virtualization

Are you an IT service management professional interested in developing your knowledge and improving your job performance? Join the IT service management community to access the latest updates from industry experts. Learn and share insights related to IT service management (ITSM) including topics such as the service desk, service catalog, problem and incident management, ITIL v4 and more. Engage with industry experts on current best practices and participate in active discussions that address the needs and challenges of the ITSM community.

IT Service Management

Cloud computing has exploded over the past few years, delivering a previously unimagined level of workplace mobility and flexibility. The cloud computing community on BrightTALK is made up of thousands of engaged professionals learning from the latest cloud computing research and resources. Join the community to expand your cloud computing knowledge and have your questions answered in live sessions with industry experts and vendor representatives.

Cloud Computing

Increasing expectations for good governance, effective risk management and complex demands for corporate compliance are presenting a growing challenge for organizations of all sizes. Join industry thought leaders as they provide you with practical advice on how to implement successful risk and compliance management strategies across your organization. Browse risk management resources in the form of interactive webinars and videos and ask questions of expert GRC professionals.

IT Governance, Risk and Compliance

The data center management community focuses on the holistic management and optimization of the data center. From technologies such as virtualization and cloud computing to data center design, colocation, energy efficiency and monitoring, the BrightTALK data center management community provides the most up-to-date and engaging content from industry experts to better your infrastructure and operations. Engage with a community of your peers and industry experts by asking questions, rating presentations and participating in polls during webinars, all while you gain insight that will help you transform your infrastructure into a next generation data center.

Data Center Management

The application development community features top thought leadership focusing on optimal practices in software development, SDLC methodology, mobile app development and application development platforms and tools. Join top software engineers and coders as they cover emerging trends in everything from enterprise app development to developing for mobile platforms such as Android and iOS.

Application Development

Practicing business intelligence allows your company to transform raw data into sets of insights for targeted business growth. The business intelligence and analytics community on BrightTALK is made up of thousands of data scientists, database administrators, business analysts and other data professionals. Find relevant webinars and videos on business analytics, business intelligence, data analysis and more presented by recognized thought leaders. Join the conversation by participating in live webinars and round table discussions.

Business Intelligence and Analytics

The IT project management community on BrightTALK includes thousands of IT project and portfolio management professionals. Find relevant webinars and videos on agile methodologies, scrum strategy, project management processes and more. Attend live webinars or view on demand content presented by recognized thought leaders in the IT project management industry.

IT Project Management

Network infrastructure professionals understand that a reliable and secure infrastructure is crucial to enabling business execution. Join the network infrastructure community to interact with thousands of IT professionals. Browse hundreds of on-demand and live webinars and videos to learn about the latest trends in network computing, SDN, WAN optimization and more.

Network Infrastructure

Welcome to the big data and data management community on BrightTALK. Join thousands of data quality engineers, data scientists, database administrators and other professionals to find more information about the hottest topics affecting your data. Subscribe now to learn about efficiently storing, optimizing a complex infrastructure, developing governing policies, ensuring data quality and analyzing data to make better informed decisions. Join the conversation by watching live and on-demand webinars and take the opportunity to interact with top experts and thought leaders in the field.

Big Data and Data Management

As an IT professional, many of the problems you face are multifaceted, complex and don’t lend themselves to simple solutions. The information technology community features useful and free information technology resources. Join to browse thousands of videos and webinars on ITIL best practices, IT security strategy and more presented by leading CTOs, CIOs and other technology experts.

Building a Data Science Platform on Apache Iceberg and Nessie

Presented by

About this talk

More from this channel