ProgrammingPro #38: Software Predictions for 2024, Python’s Dead Batteries, Life after Open Source, and RFCs
Bite-sized actionable content, practical tutorials, and resources for programmers
“When you assume a new tech leadership role in an established technology stack, it’s easy to spot flaws in the current setup and be tempted to rebuild or overhaul everything. Yet, the more challenging and practical approach is to adapt, understand, and value the existing system while making internal improvements.”
– Shalabh Aggarwal, In an interview published in today’s Expert Insight
A very Happy New Year 2024, and welcome to a brand new issue of ProgrammingPro!
News Highlights: In 2024, experts predict the integration of Large Graphical Models in AI and language-to-SQL technologies, while LinkedIn's DPH Framework, now open-sourced, offers advanced methods for optimizing code delivery. Ruby 3.3.0 released at Christmas and Python 3.13 is sending critical modules to the boneyard.
My top 5 picks from today’s learning resources:
Migrating from Java 8/11 to Java 21, and Spring Boot 2 to Spring Boot 3.2☕
Unlock Complex Time Series Analysis in SQL with Range Queries⏳
Project Hygiene: Combatting Goodhart’s Law and Other “Project Smells”🧼
The hardest part of building software is not coding, it's requirements🤔
In today's Expert Insight, we feature an interview with Shalabh Aggarwal, CTO at CredFlow and author of the Flask Framework Cookbook, now in its 3rd edition. A must-read for insights into tech careers, career advancement, and aligning tech with business objectives.
Stay Awesome!
Divya Anne Selvaraj
Editor-in-Chief
PS: We are ready to take requests again. If you would like us to include a specific programming learning resource next week, take the survey.
🗞️News and Analysis🔎
Software predictions for 2024: Experts predict this year we will see a focus on Large Graphical Models (LGMs) in AI, increased GenAI literacy and upskilling, comprehensive integration of AI/ML in software development cycles, widespread adoption of language-to-SQL technology, and more. Read to learn about trends that will directly impact your work and career opportunities.
LinkedIn open sources its Developer Productivity and Happiness Framework: Key to the framework is adapting to evolving software development methods, like agile and CI/CD, and integrating metrics from development tools to optimize code delivery and maintenance. Read to learn about effective strategies and frameworks to measure and enhance productivity and happiness in your work environment.
What you need to know about Python's 'dead batteries': Key deprecations include modules like cgi and cgitb, used for obsolete CGI web applications, replaced by urllib.parse and email.message. Read to understand all changes to adapt your code for Python 3.13, and leverage more secure and efficient replacements.
Ruby 3.3.0 Released: The new version introduces the Prism parser, the Lrama parser generator, and the RJIT pure-Ruby JIT compiler, along with performance enhancements, notably YJIT. Read to learn more about new features and optimizations.
What comes after open source? Bruce Perens is working on it: A founding figure of the Open Source movement, Perens is now focusing on the "Post-Open Source" movement to address the evolving challenges in open source licensing. Read to understand the need for a radical shift from traditional paradigms.
🎓Tutorials and Learning Resources💡
Python
Holding references on classes and functions: This article demystifies Python's treatment of functions and classes as objects. Read to master this key concept for advanced programming and understanding complex code structures.
Xmas decoration, part 1: This article introduces real-world examples from Python libraries to illustrate how decorators work in different contexts. Read to gain skills to modify and extend the behavior of functions dynamically.
Don’t use Alpine Linux for Python images: This article highlights the challenges and inefficiencies in using Alpine for Python, despite its benefits for other languages like Go. Read to learn more.
Java
Migrating from Java 8/11 to Java 21, and Spring Boot 2 to the latest Spring Boot 3.2: This article focuses on new features, performance enhancements, and strategies for a smooth transition. Read to learn how to address challenges and discover best practices for a smooth transition.
Pinning: A pitfall to avoid when using virtual threads in Java: An experiment comparing virtual and platform threads in synchronized tasks shows that virtual threads' performance degrades significantly when thread count exceeds core count, leading to longer execution times. Read to learn how to avoid this pitfall when using synchronized blocks or native methods.
Logging in Java: A Comparison of the Top 5 Libraries: In this article, Log4j 2, Logback, SLF4J, Tinylog, and the Java Logging Framework are evaluated based on features like performance, configurability, ease of use, and compatibility. Read to be able to to select the most appropriate one for your project.
PHP
Tutorial: Deploy a PHP and Azure Database for MySQL - Flexible Server app on Azure App Service: Key steps covered include creating a MySQL Flexible Server instance, building a PHP app, setting up Azure App Service, and deploying the app with Local Git. Read to gain skills in cloud-based application management and deployment.
SQL
Unlock Complex Time Series Analysis in SQL with Range Queries: GreptimeDB (an open-source, cloud-native time-series database) has introduced an enhanced SQL Range query syntax, blending SQL's flexibility with specialized time-series querying capabilities. Read to learn how the syntax addresses the limitations of standard SQL and PromQL.
SQL as API: This article acknowledges the potential risks but suggests that there are scenarios where SQL can be an effective choice for APIs, such as in complex web-shop systems requiring advanced filtering. Read to learn about the potential benefits and methods of using SQL as an API language.
How Distributed Transactions Work in TiDB: Using a marriage ceremony metaphor, this article describes the two-phase commit protocol, highlighting TiDB's (an open-source distributed SQL database) support for distributed transactions. Read for insights into its architecture, model, and the Percolator algorithm.
Go
Background Jobs in GoLang — Your Ultimate Guide to Empower Your Applications: This comprehensive guide discusses utilizing Gocraft with Redis in GoLang for efficient background job management. Read for a guide to leveraging background processing to optimize GoLang applications.
Game Dev.
The C++20 Naughty and Nice List for Game Devs: Read for a nuanced evaluation of C++20 features for game developers, categorizing them as "naughty" or "nice" based on their utility, efficiency, and compatibility with large codebases and diverse platforms.
Tonc - GBA Programming in rot13:
Tonc is a community-maintained guide to Game Boy Advance programming, now updated using mdbook. Read to learn about GBA development, contributing to open source projects, and more.
C/C++/C#
Modern C++ 101: Refactoring Legacy Code: This article demonstrates how to refactor legacy C++ code, using a simple Circle class as an example. Read to learn modern C++ techniques, such as using inline static for class members, const-correctness and more.
Top 10 errors in C and C++ projects in 2023: This article highlights common pitfalls like undefined behavior, null pointer dereferencing, and multi-threading issues. Read to identify and avoid typical coding mistakes.
std::print in C++23: This feature simplifies code and offers compile-time error checking, user-defined type formatting, Read to learn more about the feature’s impact on safety and performance.
Invariance, Covariance, and Contravariance in .NET C#: This article explains how these principles relate to generic type arguments in C#, affecting reference conversions and assignment compatibility. Read to learn how the concepts impact type safety and flexibility in .NET C# programming.
JavaScript
Why JavaScript promises aren't technically monads: This article delves into the technicalities of monads in programming, using TypeScript examples to illustrate their properties and functions. Read to learn how the "flattening" behavior of promises contradicts monad laws.
Fast Buffer-to-String conversion in JavaScript with a Lookup Table: This article discusses optimizing Buffer-to-String conversion in JavaScript, particularly for Node.js environments. Read to learn how different data structures and algorithms can impact performance and memory usage in JavaScript in string manipulation tasks.
Visual Basic/VB.NET
Tutorial: Create a WPF application with Visual Basic: This tutorial covers project creation, window configuration, adding text, buttons, and code, as well as debugging and building a release version. Read to gain skills in Visual Studio IDE, handling UI elements, coding in Visual Basic, and debugging techniques.
Ruby
Performance impact of the memoization idiom on modern Ruby: This article introduces object shapes, a concept aimed at optimizing the way instance variables are stored and accessed. Read to learn how object shapes and inline caching affect performance, especially in the context of polymorphism and memoization patterns.
Tutorial: 3 Chain of Responsibility (CoR) implementations that will make you fall in love with this pattern: This article presents three implementations of the CoR pattern used at Wecasa, emphasizing its transformation of method calls into independent object calls, thus enhancing maintainability and readability. Read to learn more about the pattern's benefits.
Rust
Rust Multi threading: This article covers Rust's async capabilities, thread ownership, memory safety mechanisms like Mutex and Arc, and message passing between threads using channels. Read for practical code examples.
Swift
Creating a macOS app with Haskell and Swift: This article includes steps for creating a SwiftUI app, linking Haskell libraries, and initializing the runtime system, with future installments planned to cover more complex interactions and iOS development. Read to learn about setup procedures, creating and linking libraries, and runtime system initialization.
Swift Concurrency Waits for No One: This article emphasizes the importance of forward progress in Swift's concurrency model. Read to understand the risks of deadlocks and the challenges in integrating asynchronous and synchronous code.
Assembly Language (ASM)
i386 Assembly Language trick for storing data in .text: This article details a method in i386 assembly for embedding data in the .text section, demonstrated through a compact "Hello World" example, resulting in significantly reduced executable sizes. Read if you are involved in creating compact and position-independent executables
🌟Best Practices🚀
Project Hygiene: Combatting Goodhart’s Law and Other “Project Smells”: This article proposes an external review process to identify and correct problematic development practices, emphasizing the need for a proactive approach to software project management. Read to gain insights into identifying and resolving subtle yet significant issues in software development.
6 Common Patch Management Mistakes to Avoid: This article emphasizes the importance of a coherent patching strategy, leveraging automation, balancing risk, and not overlooking patches, especially from smaller vendors. Read to learn best practices for effective patch management.
Secure transactions with local-first: This article describes how to execute secure transactions in local-first applications such as ElectricSQL, contrasting it with cloud-first approaches and offering a state machine solution to emulate request-response workflows. Read to learn how you can maintain transaction integrity without external data inconsistencies.
The hardest part of building software is not coding, it's requirements: This article emphasizes the complexity of software development compared to chess and self-driving cars. Read to understand the limitations of AI in handling complex software projects, and the importance of clear specifications and agile methodologies in software development.
Develop the right thing every time and become a 10x engineer 🏆: The art of writing RFCs 🥋: This article discusses the importance of writing Request For Comments (RFCs) in software development to plan and discuss codebase changes before implementation. Read to gain insights into the significance of RFCs in software development, the criteria for deciding when to write one, and the structure of an RFC document.
🧠 Expert Insight 📚
Here’s an exclusive interview with Shalabh Aggarwal, who authored the Flask Framework Cookbook - Third Edition. The book is tailored to help you learn how to develop, deploy, and scale Flask applications using industry-standard practices, along with integrating AI and managing microservices across various platforms.
In the following interview, the author, reflects on his career journey, from founding a startup to working at Google, and now leading at CredFlow, highlighting his learnings along the way and his advice to those looking to grow in their tech roles.
Q. Could you give us an overview of your career journey that led you to become a CTO?
A. This is an interesting question to begin with! I actually started my career as the CTO of a startup that I helped establish back in 2010. I was still in the final year of my engineering degree and have always been one to swim against the tide. We created a successful and profitable business which grew by 300+% year-on-year for nearly 4 years. However, due to unforeseen circumstances, we had to close the company's doors, and I moved on to work with Google.
This was a great transition for me, where I moved from a leadership role to being an individual contributor. This role and the ones that followed helped me learn the nuances of building technology, systems, and engineering teams from scratch. I worked on multiple different technologies across various domains while building engineering teams of all shapes and sizes. Whilst doing all this, I gradually climbed the maturity ladder as well as the corporate ladder.
12-13 years into my career, I found myself capable enough to lead the technology landscape of an already established startup, CredFlow, and hence accepted the current CTO role I am in now. CredFlow is a Fintech SaaS platform which helps Small and Medium Enterprises across India manage and improve their cash flows by integrating with their financial software and providing them with credit facilities.
Q. What first got you interested in programming, and how did you get started?
A. My first rendezvous with programming occurred long ago when I was in the 4th standard of my schooling, i.e., around 8-9 years old. It was the first time I was officially introduced to computers and to writing some basic code in LOGO. I immediately fell in love with the whole concept of building something out of thin air with a few lines of code. The entire experience was very powerful and yet satisfying.
Moving ahead, I experimented with C, C++, Visual Basic, and Java before finally adopting Python as my primary programming language. I have worked with other programming languages like R, GoLang, JavaScript, etc., but Python remains my go-to language.
Q. Can you share some pivotal moments that shaped your programming career?
A. This is a difficult question to answer. At all stages in my career, I have encountered defining moments which have influenced how I think about programming challenges and approach their solutions.
If I have to recall a couple of those instances, one would be when I was working with Google in 2014, and I got to work on some experimental JavaScript libraries that were in the very early stages of development. Being a Python enthusiast, it wasn’t a particularly pleasant experience at the start, but as I delved deeper, my understanding of JavaScript and programming in general increased significantly. I was able to comprehend how systems and programming languages operate at a very fundamental level while developing highly scalable software.
Another instance was when I built a closed-loop P2P mesh network that would allow extended distance communication between mobile devices without the need for the internet. This project required an in-depth understanding of networking, particularly in maintaining awareness among multiple participants in a network. We ended up modifying core Android libraries and creating many of our own to meet this challenge.
Q. How has your role as a CTO changed with the latest tech trends?
A. As a technology leader working primarily with early-stage, fast-growing startups, I have learnt that the overall technology landscape changes very rapidly. Every 2-4 years, there is a major shift in technology trends, either with the introduction of a completely new technology or a significant overhaul of an existing one. It is wise to stay updated with these changing trends and adopt those that resonate the most, as some have the potential to alter how the world operates.
For instance, over the last decade, we have witnessed several major changes like the Internet of Things (IoT), wearables, cloud computing, electric vehicles, blockchain, and the most recent, artificial intelligence.
It's not that these technologies matured overnight. They have taken decades to evolve and reach a broader audience, but in the past decade, all of these and more have come to the forefront, each changing the world in its own way.
My journey over the past 15 or so years, from an engineer to a CTO, has evolved with these trends as necessary. I was among the first to adopt AWS autoscaling, using it to enhance my company's profitability. Similarly, a precursor to AI is natural language processing, and I have extensively used this in developing highly customized solutions for one of the world's most challenging languages, Mandarin, including on-the-fly translation to traditional and simplified Chinese as well as English. Currently, with the advent of AI and the potential shift towards Artificial General Intelligence (AGI), my role has evolved to include adopting these technologies to help my business and company grow more efficiently with less effort.
Q. Any advice for programmers looking to climb the ladder in tech?
A. Good one! The career ladder in technology is not too complex to understand. It generally offers two growth paths: people/technical management and system design/architecture. These paths are not mutually exclusive, and individuals can choose to switch depending on their skill set and aspirations for future growth.
I always advise my juniors and fellow engineers to maintain a long-term vision for their career and set a goal that they aim to achieve within a specified time frame. This approach makes it easier to measure progress towards the established goal, and adjustments can be made as necessary.
Initially, programming might seem to be the most important aspect of engineering, but with experience, one realizes that writing code is just a small part of the job. Other elements such as planning, strategizing, solution development, designing, etc., are equally important, if not more so.
Therefore, another piece of advice I offer to fellow software engineers is to recognize the importance of all aspects of software engineering and excel in these areas as well, while continuing to hone your programming skills.
Q. What programming languages or tools do you think will be big in the future, and why?
A. Looking ahead, I believe the significance of any specific programming language will continue to diminish. People will still choose the language that best suits their needs, whether it's based on syntax, community size, available resources, frameworks, etc. However, the primary focus will eventually shift to which programming language best supports the ecosystem around artificial intelligence and machine learning. Python has been leading thus far, but other languages are catching up rapidly.
The performance of a programming language usually isn’t a major concern unless a system is operating at a very large scale, which applies to only about 1% of overall use cases. Therefore, developers and companies will continue to select what works best for them.
Regarding tools, I anticipate that in the coming years, tools related to Large Language Models (LLMs), AI, Generative AI, and Artificial General Intelligence (AGI) will become more prevalent. Companies will begin to integrate these tools at all levels, and engineers will need to adapt to remain relevant.
Q. How do you encourage innovation and learning within your tech team?
A. I begin by fostering a culture of curiosity where individuals are encouraged to learn and experiment. They are rewarded for questioning current implementations and choices, and for seeking answers to their own questions. People are provided with additional time and resources to explore new ideas, and they often have the opportunity to apply some of these in their official assignments.
Organizing hackathons is another effective method to uncover hidden talents among engineers, challenging them to push their limits and create something valuable within the constraints of time and resources.
I prefer to lead by example, being open to new ideas and experimenting with novel approaches. By sharing my past experiences, I encourage my team not to fear taking risks in exploring new options.
Developing internal leaders is crucial; it involves people taking responsibility for certain aspects, and I support their growth as they continue to learn and innovate.
Q. In your experience, what are common challenges for developers on large projects?
A. I would like to refer to one of my LinkedIn posts on this same topic that I published last year: "One of the biggest enemies of any software system is complexity."
Over the past few years, I have repeatedly found myself echoing this statement in many of my discussions, irrespective of whether I'm speaking with my engineering team, the product team, stakeholders, or even casually with friends over coffee. This topic seems to surface frequently in most of my conversations.
During one such discussion, someone described a particularly intricate project to me, involving multiple technologies and numerous moving parts. They concluded with the phrase - "This is definitely one of the most complex projects". For me, as intriguing as it may sound, it raises a red flag.
Reflecting on this, I've noticed that many people assume that if a system is complex, it must be good or advanced.
But, in reality, the opposite is true.
As complexity increases, quality decreases. The effort required to maintain, debug, and enhance the system escalates dramatically. Furthermore, onboarding new engineers onto such a system can become extremely challenging. I've witnessed competent engineers struggle simply because the system was too complex, and they couldn't meet the company's expectations within the set timelines.
People often overlook or fail to recognize the importance of regularly assessing the system for unnecessary complexities. It's always beneficial to occasionally pause new feature development and review the system, identifying and eliminating any superfluous complexities. Failing to do this regularly can lead to the dreaded necessity of a complete system rewrite.
I label this a 'necessary evil' because of its significant impact on the business. It incurs considerable development costs, and typically, no new features are developed during this process.
Key takeaways:
Utilize design patterns to keep complex components abstracted while preserving overall simplicity.
Avoid introducing multiple moving parts into the system, such as unnecessary microservices and/or libraries. Just because it is possible doesn’t mean it’s advisable.
Regularly evaluate your system for complexity. If something can't be easily explained, it likely needs revisiting.
Employ tools to assist in this process. Monitoring tools are indispensable to an engineer.
Q. How do you stay up-to-date with the ever-evolving tech landscape?
A. I won't use many words to answer this question; I believe it's quite straightforward.
I stay abreast of technology trends by reading high-quality blogs and articles.
I remain hands-on by coding occasionally and undertaking personal projects as required.
I read extensively whenever possible, selecting from a plethora of sources that align with my interests.
I engage in conversations with colleagues, preferably those more knowledgeable than myself, aiming to learn from their experience and wisdom.
Q. Any insights on balancing tech expertise with business goals in your role?
A. As the CTO, comprehending the business goals thoroughly and aligning closely with them is crucial in my role. It's my responsibility to ensure that my team also focuses on the same short and long-term objectives set by the founders and leadership committee.
Collaborating with other departments and strategically prioritising projects is another vital aspect of my role. It's important to keep other departments informed and educated about technological progress and constraints.
Maintaining a long-term vision is essential, ensuring that the technology developed is scalable and adaptable to the evolving and expanding needs of the company.
Managing risk and consistently communicating with relevant stakeholders about progress, challenges, and opportunities is also of paramount importance.
Q. How important is teamwork across functions for a CTO, and how do you promote it?
A. I believe that teamwork is a fundamental pillar in building any organization. Fostering healthy teamwork across various departments is particularly crucial in tech companies, as it's often challenging for non-technical departments to grasp how software technology operates. Additionally, due to the semi-intangible nature of software, not everyone comprehends its complexities and intricacies.
Therefore, as the leader of the technology department, it's imperative for me to ensure there's an open exchange of ideas between my department and others. Often, other departments provide valuable feedback that can be integrated to enhance the overall product.
To foster teamwork and collaboration between the technology department and others, I regularly coordinate meetings with heads of other departments. In these meetings, I explain the work we do and discuss relevant metrics. This approach makes it easier for others to understand the role of technology by relating it to concrete numbers and seeing how it connects with their own deliverables.
Q. Looking back, what's one piece of advice you wish you had at the start of your tech leadership journey?
A. I have been fortunate to have exceptional mentors throughout my career, and I've learnt immensely at every stage. Indeed, I continue to learn something new each day.
However, one piece of advice particularly resonates with me: “When you assume a new tech leadership role in an established technology stack, it’s easy to spot flaws in the current setup and be tempted to rebuild or overhaul everything. Yet, the more challenging and practical approach is to adapt, understand, and value the existing system while making internal improvements.” It’s crucial to recognize that the decisions made during the original system's development were based on valid reasons at that time. Although all systems eventually become outdated and may require a rewrite, doing so merely because you, as a leader, don’t favor it is not always the wisest course of action.
This guidance has significantly influenced how I approach problems and handle them. I have grown to appreciate the work of others more, and by enhancing things from within, the depth of understanding I gain about the system and the business is extraordinary.
Flask Framework Cookbook - Third Edition was published in July 2023. You can buy the book here.
🛠️ Useful Tools ⚒️
gpt-pilot: an AI developer companion that creates apps by coding alongside developers, emphasizing a 95%/5% human-AI collaboration.
ml-ferret: an advanced MLLM for precise referring and grounding tasks, with the Ferret Model, GRIT Dataset, and Ferret-Bench evaluation, available for research and development.
promptbench: a PyTorch package for LLM evaluation, offering APIs for model performance, prompt engineering, adversarial prompt assessment, and more.
StreamDiffusion: a real-time image generation pipeline with batch processing, guidance optimization, GPU efficiency, IO management, and model acceleration.
That’s all for today.
We have an entire range of newsletters with focused content for tech pros. Subscribe to the ones you find the most useful here. Complete ProgrammingPro archives can be found here. Complete PythonPro archives are here.
📢 If your company is interested in reaching an audience of developers, software engineers, and tech decision makers, you may want to advertise with us.
If you have any feedback, leave your comments below.