ProgrammingPro #77: Python 3.13 Finally Debuts, Rspack 1.0, and Importing Arrow Data with Python
Welcome to this week’s edition of ProgrammingPro!
In today’s Expert Insight, we bring you an excerpt from the recently published book, In-Memory Analytics with Apache Arrow - Second Edition, which shows you how to import Arrow data with Python using the C Data API via the pyarrow and cffi modules.
News Highlights: Python 3.13 finally debuts with GIL disabling and JIT compiler; Rust firm at 13th in Tiobe, Mojo enters top 50; Rspack 1.0 is 23x faster than Webpack; and Julia 1.11 introduces a faster, lower-overhead Memory type.
My top 5 picks from today’s learning resources:
The #1 Mistake Developers Make with Redux and How to Fix It🚧
Confusing or misunderstood topics in systems programming: Part 0🧩
But there’s more, so dive right in.
Stay Awesome!
Divya Anne Selvaraj
Editor-in-Chief
PS: The October survey is still live. Do take the opportunity to leave us your feedback, request a learning resource, and earn your one Packt credit for this month.
🗞️News and Analysis🔎
Breaking Down Python 3.13’s Latest Features: Released finally on the 7th of October, the version includes a revamped interactive interpreter with multi-line editing, experimental disabling of the GIL, and a JIT compiler.
Rust resumes rise in popularity: Rust has returned to 13th place in the October 2024 Tiobe index and maintained 10th in the Pypl index. Mojo has also entered Tiobe's top 50. Read to learn more about current rankings.
Rspack 1.0 Released, 23x Faster than Webpack, Compatible with Top 50 Webpack Plugins: Rspack 1.0, a JavaScript bundler written in Rust from ByteDance, is compatible with over 80% of the top 50 Webpack plugins.
Julia language adds lower-overhead Memory type: Julia 1.11 introduces a new Memory type as a lower-overhead alternative to Array, offering significant speed improvements and more maintainable code.
Apollo Connectors enables developers to turn REST APIs into GraphQL endpoints: Apollo Connectors simplifies API integration, reduces complexity, and enables a unified API strategy.
UNO Platform 5.4 Improves App Performance: The version introduces over 290 new features and enhancements focused on improving app performance, including a 15% reduction in memory allocations.
ChatGPT Canvas offers a new visual interface for working with ChatGPT in a more collaborative way: The new offering, currently rolling out to select users, enhances collaboration on writing and coding projects.
🎓Tutorials and Learning Resources💡
Python
10 Jupyter Notebook Features You Didn’t Know Exist: Discusses features including magic commands, interactive widgets, auto-reload for modules, in-notebook documentation, and collapsible headings.
For more Python resources, go to PythonPro
September Python Releases from Packt at 30% off
on both eBook and Print Books
Print discounts end tomorrow on the 11th of October!
C# and .NET
Microsoft previews unified .NET API for AI, and delivers .NET 9 Release Candidate with go-live license: The AI extension library aims to standardize interactions with AI services, allowing developers to work with a common API.
🎓Tutorial | How to Use StateContainers in .NET MAUI to Manage State in Applications: Explains how to implement StateContainers with code examples and a demo for handling user authentication.
🎓Tutorial | Dispose or Finalize in C#? Discover which to use: Provides real-world examples, such as file handling, database connections, and graphics processing, to demonstrate when to use each method.
C and C++
🎓Tutorial | Reflection in C++26: Metafunctions for Enums and Classes: Demonstrates how developers can use reflection to iterate through enum members and retrieve class members by index or name.
Iterating through matched characters in modern C++: Explores using approaches like
find_first_of,std::rangesfrom C++20, and coroutines from C++23.Privileges relinquishing order in C: Explains the correct order for relinquishing privileges in C programs, specifically for setUID and setGID operations, to prevent security vulnerabilities.
Java
Oracle touts ‘tip and tail’ release model for Java library development: The model, aimed at improving the developer experience allows faster innovation for new development while maintaining stability for long-term projects.
🎓Tutorial | Event-driven architecture (EDA) on the modern stack of Java technologies: Explores the implementation of an EDA focusing on Transactional outbox, Inbox, and Saga patterns in microservices.
Java 23 New Features With Examples: Covers improvements like primitive type support in patterns,
instanceof, andswitchstatements, the introduction of Markdown documentation comments, and enhancements to concurrency.
JavaScript and TypeScript
🎓Tutorial | The #1 Mistake Developers Make with Redux and How to Fix It: Discusses the common mistake of overusing Redux, explains the difference between client and server state, and shows you the benefits of React Query.
Behind the Scenes at the JavaScript Registry (JSR): Discusses JSR, a modern open-source package repository for JavaScript and TypeScript that offers features like type declaration files, secure publishing, and more.
Announcing TypeScript 5.7 Beta: New features include checks for never-initialized variables, path rewriting for relative paths, support for ES2024, faster project ownership checks, JSON import validation, and improved performance.
Go
🎓Tutorial | Securing gRPC Services with JWT Authentication in Go: Explains how to secure gRPC services using JSON Web Tokens (JWT) authentication in Go, focusing on interceptors, metadata, and JWT handling.
The Go libraries that never failed us: 22 libraries you need to know: Lists libraries for routing, middleware, database management, OpenAPI, gRPC, testing, and more, while offering anti-pattern warnings and practical tips.
Rust
💼Case Study | Rust is rolling off the Volvo assembly line: Discusses how Volvo successfully integrated Rust into the development of their low-power processor ECU, improving code quality and reliability in their vehicles.
🎓Tutorial | Understanding Rust's Trait Objects: Vtables, Dynamic Dispatch, and Memory Deallocation: Explains how vtables store method pointers for trait objects, how TCO optimizes performance, and more.
Swift
🎥Swift & Interoperability - Tony Parker & Ben Cohen: Introduces the new direction for Swift's Foundation, including a major refactor of Foundation for Swift 6, enabling open-source contributions, and transitioning from C to Swift.
🎓Tutorial | Writing Cleaner Swift Code: Early Returns & Aligning the Happy Path Left: Explains how handling edge cases upfront and reducing nesting, results in cleaner, more maintainable code with reduced cognitive load.
PHP
👨🏫Open Access Course | Laravel API Course: Covers topics like optimizing a Laravel project for API-first development, data modeling, API design, authentication, rate limiting, versioning, pagination, error handling, and more.
SQL
🎓Tutorial | Advanced SQL for 10x Data Analysts - Part 1: Explores advanced SQL techniques, focusing on joins, nested data, analytical functions, and efficient query writing.
Ruby
Ruby on Rails 8.0 first beta release: A big bet on SQLite in production: The release promotes using SQLite in production environments for various features, including caching, queuing, and as the primary database.
Ruby 3.4.0 preview2 Released: The version introduces features like switching to the Prism parser, language changes, core class updates, and C API improvements.
Kotlin
🎓Tutorial | How Functional Programming Can Help You Write Efficient, Elegant Web Applications: Includes code examples and practical guidance on topics like pure functions, function composition, and functional DI.
🌟Best Practices and Advice🚀
Proactive Approaches to Securing Linux Systems and Engineering Applications: Key approaches discussed include dynamic vulnerability confirmation, regular security audits, and comprehensive patch management.
Brutalist Programming Manifesto: Advocates for simplicity, self-reliance, solving real problems, avoiding unnecessary complexity, and encouraging independent, practical thinking in software development.
Confusing or misunderstood topics in systems programming: Part 0: Explains complex systems programming concepts like processes, context switching, and the kernel in a clear, beginner-friendly way.
🎥Tacit Talk Episode 5: Combinatory Programming with Zach Smith: Explores how functional programming concepts such as combinatory and tacit programming, can simplify code and improve readability.
🧠Expert Insight📚
Here’s an excerpt from “Chapter 4: Crossing the Language Barrier with the Arrow C Data API” in the book, In-Memory Analytics with Apache Arrow - Second Edition by Matthew Topol, published in September 2024.
Importing Arrow data with Python
The common terminology for runtimes providing an interface for calling an API of another runtime or language is an FFI. In Python, we’re going to use a library called cffi , which is used by the pyarrow module to implement the C
data API. Make sure that you’re running the script we’re about to write in the same directory as the libsample.so library file that we created in the previous exercise:
First, the imports – the following highlighted line represents the essential case that we’re importing the already compiled
ffilibrary that is part of thepyarrowmodule:
import pyarrow as pa
from pyarrow.cffi import ffi
There are a few different ways to integrate the FFI module, but for the purposes of this exercise, we’re going to use dynamic loading of the library with the defined interface through the
cdeffunction. Then, we’ll load our shared object library withdlopen:
ffi.cdef("void export_int32_data(
struct ArrowArray*);")
lib = ffi.dlopen("./libsample.so")
Note that this matches our
extern "C"declaration from before.Now, we can create an
ArrowArraystruct and call the function we exported to populate it:
c_arr = ffi.new("struct ArrowArray*")
c_ptr = int(ffi.cast("uintptr_t", c_arr))
lib.export_int32_data(c_arr)
Then, we use the
pyarrowmodule to import data. Remember that since we’re passing around pointers, there’s no copying of the data buffers. So, it doesn’t matter whether this array has 1,000 elements or 1 million elements; we’re not copying the data here. Importing the data is just hooking everything up to point to the right areas in memory:
arrnew = pa.Array._import_from_c(c_ptr, pa.int32())
# do stuff with the array
del arrnew # will call the release callback
# once it is garbage collected
You can add a print statement in there if you like, to confirm that it is actually working as intended. That’s it. The library we created could create that array of data in any way we want, but as long as it properly populates the C struct, we’re able to pass the data around in the Arrow format without having to copy it.
What if we wanted to work in the other direction? We can read in our data with Python and then hand it off to something faster for processing, similar to how Spark used JPype to communicate the data from Python to Java without copying. How would we go about doing that?
Memory Analytics with Apache Arrow - Second Edition was published in September 2024. Packt library subscribers can continue reading the entire book for free or you can buy the book here!
Get the eBook for $39.99 $27.98
Get the Print book for $49.99 $34.98
🛠️Useful Tools⚒️
Drasi: Microsoft's open-source system for real-time change detection and automated reactions in dynamic systems.
One: a React framework on Vite, streamlining web and native app development with universal routing and seamless data management.
agents: a real-time AI framework for processing text, audio, images, and video, with OpenAI Realtime API support and plugins for popular LLMs.
That’s all for today.
We have an entire range of newsletters with focused content for tech pros. Subscribe to the ones you find the most useful here.
If your company is interested in reaching an audience of developers, software engineers, and tech decision makers, you may want to advertise with us.