A deep dive into the Arrow Columnar format with pyarrow and nanoarrow

🔊 Recorded at PyCon DE & PyData Berlin 2024, 24.04.2024
https://2024.pycon.de/program/LERYUY/

🎓 Watch a comprehensive tutorial on the Arrow Columnar format with pyarrow and nanoarrow, exploring its intricate details and interactive examples to enhance your understanding of this efficient in-memory data representation standard.

Speakers:
Joris Van den Bossche, Raúl Cumplido, Alenka Frim

Description:
In a compelling talk titled "A deep dive into the Arrow Columnar format with pyarrow and nanoarrow," speakers Joris Van den Bossche and Alenka Frim from Voltron Data along with Raúl Cumplido, a Senior Software Developer, will elucidate the intricacies of the Apache Arrow columnar format. The speakers, who are active contributors and maintainers in the Apache Arrow project, will shed light on the significance and utility of the Arrow format for efficient in-memory columnar data representation.

The tutorial aims to offer a comprehensive understanding of the Arrow columnar format, including its different types and buffer layouts, providing participants with interactive demonstrations using the pyarrow and nanoarrow libraries. The talk emphasizes Apache Arrow as a multi-language toolbox for accelerated data interchange and in-memory processing, illustrating the format's role in enabling efficient analytic operations on modern hardware such as CPUs and GPUs.

Attendees will have the opportunity to explore the physical memory layout and various data types associated with the Arrow columnar format. The talk will feature practical code examples using pyarrow and nanoarrow libraries, allowing participants to create and inspect Arrow data effectively. Furthermore, the discussion will highlight the broader applicability of the columnar format, as it underpins multiple libraries like pandas, polars, datafusion, duckdb, cudf, influxdb, among others.

By delving into the nuances of the Arrow columnar format and providing hands-on demonstrations with pyarrow and nanoarrow, the speakers will equip participants

⭐️ About PyCon DE & PyData Berlin:
The PyCon DE & PyData conference unite the Python, AI, and data science communities, offering a unique platform for collaboration and innovation. The PyCon DE & PyData Berlin 2024 conference, hosted in partnership with the local Berlin PyData chapter, provided an exceptional experience, fostering deeper connections within the Python community while showcasing advancements in AI and data science. Attendees enjoyed a diverse and engaging program, solidifying the event as a highlight for Python and AI enthusiasts nationwide.

Follow us:
• LinkedIn:   / 28908640  
• X: https://www.x.com/pyconde
• X: https://www.x.com/pydataberlin

Links:
• Conference website: http://pycon.de
• Related sessions: http://2024.pycon.de/program/categori...

The conference is organized by
• Python Softwareverband e.V.: http://pysv.org
• NumFOCUS Inc.: http://numfocus.org
• Pioneers Hub gemeinnützige GmbH: http://pioneershub.org


If you enjoyed this session, please like, comment, and subscribe to our channel for more insightful talks and discussions.
Share this video with your network to spread the knowledge!

Hashtags:
#Python #PyConDE #PyData #OpenSource #AI #DataScience #MachineLearning #SoftwareDevelopment #LLMs #Community

Acknowledgements:
Special thanks to all the volunteers and sponsors who made this event possible.

About:
Python Softwareverband e.V.:
PySV is a non-profit that promotes the use and development of Python in Germany through events, education, and advocacy, fostering an open Python community.

NumFOCUS Inc.
supports open-source scientific computing by providing financial and logistical support to key projects like NumPy and Jupyter, promoting sustainable development and collaboration.

Pioneers Hub gemeinnützige GmbH:
is a non-profit fostering innovation in AI and tech by connecting experts and promoting knowledge exchange through events and collaborative initiatives.
www.pydata.org

PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.