Hands-On Software Engineering with Python with Brian Allbee
On systems thinking, designing for change, testing, and team velocity
Brian Allbee has been writing Python almost exclusively since 2012, working across cloud-based application development, machine learning integration at Dice.com, and backend systems in AWS using Step Functions and Python Lambdas.
Allbee, Staff Software Engineer at Cleerly and author of Hands-On Software Engineering with Python, now in its second edition published by Packt, joined Deep Engineering Live to talk about what separates engineering from programming, how to scale and refactor Python systems responsibly, and what it actually takes to grow into senior and staff-level roles.
Watch the full conversation below.
This session was recorded live as part of the Deep Engineering Live Interview Series. The transcript below has been lightly edited for clarity and readability. Audience members joined the conversation and asked questions directly during the session.
Q. Tell us about your background and the kinds of systems you have worked on.
Brian Allbee: I have been programming almost exclusively in Python since early 2012. Prior to that I worked in C Sharp dot net, Flex markup language, and PHP for application development. I landed on Python at a job I started early in 2012 at an ad agency where they needed somebody to come in and build an internal application that was more performant than their off-the-shelf solution for asset management. I fell in love with the language a little before that position started, but I was very happy that a language audit I did reinforced that Python was still the way to go because it had everything they needed.
Since then I have done client-facing cloud management application work, a handful of customer-facing applications I cannot get into too much detail on because they are still covered under NDAs, and the last six years I spent doing machine learning implementation and integration for Dice.com on the team that eventually became their applied data science and AI team. Currently I am doing backend system development in an AWS cloud context with Step Functions and Python Lambdas to deal with health insurance processing.
Q. What distinguishes a true software engineering approach from just programming, particularly for Python developers working on real-world systems?
Brian Allbee: I think learning to think in terms of systems, not just implementations, is probably the main thing. I feel that holds true whether the backing language is Python or not, and it does not stop with just the systems that an engineer is writing. On the technical side it extends out to the entire toolchain, anything that shapes the code itself or determines how the code is managed or handled. But it also extends to what I would call nontechnical systems in the sense of a set of principles or procedures that define how something is done.
I basically feel that programming is really focused on making sure that the code is correct, the correctness of the code itself. Where software engineering starts expanding out into more of a focus on sustainability as change occurs.
Q. For Python developers aiming to move into senior or staff engineering roles, given how much AI is now part of development workflows, what skills or mindset shifts do they need beyond raw coding proficiency?
Brian Allbee: I think that same systems-oriented thinking is still the big dividing line, and I believe that will hold true even if LLM-based code generation turns out to be the next big thing that all of its proponents argue it will be. Even in those scenarios, manual interaction with code at the level of the syntax of the code itself might dwindle over time, but there are still going to be sensitive domains where some of that remains necessary. More importantly, engineers need to understand how that code fits together even if they did not write it, and why it fits together the way it does.
Hand in hand with that is a broader understanding of the problems being solved. Software engineering, like every other engineering discipline I am aware of, is concerned with solving problems usually while operating within some set of constraints. Software engineering focuses on solving those problems by creating systems, and that goes back to the whole systems-oriented thinking. But solving the problem requires understanding that problem first. Even if code generation becomes largely or completely automated, someone still has to own that system and understand its constraints and its potentials for failure and how it is expected to evolve over time.
Q. What are some of the best practices for updating, refactoring, and scaling an existing Python codebase as it evolves?
Brian Allbee: I think most of the paths to success in that context, at least the ones I can think of that I have seen, do not really start with the architecture but with discipline behind the process. The approaches that have worked for me or that I have seen work well for others include keeping things as simple as possible, wrapping processes that get used over and over again into functions or methods or whatever context works best whenever possible, and not being afraid to use structured data.
If you are working in a team, make sure the team has some agreement about how much in-code documentation and by extension comments are expected and what it should provide. The same kind of team-level agreement about what code standards you want to apply. Stick to those until something comes up that is not covered, and revise them as needed. It is a growing process.
Writing code with an eye towards making it testable is key even if there is not an immediate need for testing. Future you, if you are writing good tests, will come back and thank you if they could.
Q. Can you share an example from your experience of tackling technical debt or redesigning a Python system to improve its maintainability and performance?
Brian Allbee: Honestly, I really cannot. I do not have any dramatic war stories here because I have worked with generally exceptionally healthy teams that treated technical debt as a first-class concern, not as an emergency. Technical debt is one of those product-level priorities. Whoever is making the prioritization decisions is going to be in control of when those get tackled if they get tackled.
If there is significant technical debt, making sure that you can communicate effectively, here is what the impact of this technical debt is to your product-level people or whoever is making those decisions, is going to be a key thing. That means being able to sit down and say, I understand that you do not want us to deal with this bug or whatever it happens to be. If we do not deal with this, it is going to lead to this, then that, and the longer we put that off, the more likely it is to lead to a really significant problem and the longer it is going to take for us to get past that.
Q. Modern teams often grapple with how much upfront system design to do versus driving straight into coding. How do you find the right balance between careful architecture and rapid agile execution?
Brian Allbee: I think understanding the full final scope of a project, even if it is just at a very high level, is critical to one side of that balance. The other side is knowing, again even if just at a high level, what constraints and non-project expectations are in play. You mentioned agile. Even if some form of agile is not part of a team’s day-to-day processes, there are some takeaways from agile that I think can still be beneficial. The entire idea of delivering work and software frequently on some sort of cadence is one of those. Iterating against the smallest deliverable units that can be identified would be another major factor.
I do not think the real risk is too much design or too little. It is designing without understanding the constraints that are involved. Iterating on the smallest meaningful units of work is going to be the most practical way to find that right balance between design and execution.
Q. Python has introduced features like data classes, type hints, and static typing tools in recent versions. How have these modern language features shaped your approach to designing Python software, and how would you recommend engineers fully embrace type hints in large projects?
Brian Allbee: Not as much as it might sound like, actually. Before I turned to Python I was working with C Sharp dot net, which is a statically typed compiled language. I came to really like the idea of static typing from a programming perspective even in dynamically typed languages like Python and JavaScript. If you dig back far enough into some of my very old and now unsupported blogs, you will find I even wrote some blog posts about implementing that sort of a structure in Python back as far as version 2.7.
I definitely recommend using the typing system that is in the language right now. At worst, it is additional documentation that modern IDEs can pick up on to help an engineer working with the code. With the inclusion of just one third-party package, I like TypeGuard myself but there are others, it is possible to achieve runtime static-like type safety and static type-like behavior in Python code. And pre-deployment tools like MyPy are going to pick up on your type hints to give you some extra quality control going into that process. I think about whether you enforce types at runtime or not, the design clarity is worth it.
Q. In building robust Python applications, how do you approach data modeling and validation? When does Pydantic make sense and when are simpler options sufficient?
Brian Allbee: I think it depends on the scope and intentions of the project. Pydantic is great for projects where there are complex requirements that can be derived from something like a JSON or OAS schema. It is also good for projects that are responsible for generating those JSON or OAS schemas. The downside is it is a larger package, coming in at around two and a half megabytes or so for the module itself and its primary dependency. So if package size is a concern, that might not be the best choice.
There are other options. Fast JSON schema combined with regular Python dictionaries and lists is a solid alternative and it is much smaller. If there is no need for any kind of schema documentation but there is still a desire for type checking, type-annotated Python classes will probably get you 80 plus percent of the way there in my experience. If you need mutable data structures and that is all you need, data classes are a good option. If you need an immutable data structure and type checking is not a concern at the level of the code structure, I usually start with something like a named tuple.
I think the right data modeling tool depends less on popularity and more on the system’s constraints and the scale, scope, and longevity of the project.
Q. What are your go-to best practices for testing Python systems at scale? How do you balance unit, integration, and end-to-end tests, and how do you ensure the test suite stays reliable and useful over time?
Brian Allbee: For my own projects, I tend to like a testing approach that exercises valid and invalid inputs for all of the parameters of every callable in the project. You combine that with judicious monitoring of missing lines in a code coverage report, and that has served really well for me in making sure that the targets of those tests are being both thoroughly and realistically exercised.
In a working environment, I like for the team to come to some sort of consensus about how the tests are organized, what tools are involved, and so on. I have seen what happens when different engineers who are not communicating with each other each go their own way. The tests that result, even if they are rigorous and well thought out, are oftentimes difficult to follow across different test modules because different people write tests in different ways.
When integration or end-to-end testing is not feasible, I try to push unit tests closer to behavioral testing even if that increases mocking complexity. Those kinds of scenarios, unit testing can still go a long way. Ultimately, though, tests are most valuable when they reflect how the system is actually used, whether that is managed at a unit testing, integration testing, or system testing level, not just how that testing code or the code itself is structured.
Q. AI is now being used in areas like test generation, debugging, and even test maintenance. How should engineers think about AI in testing without compromising reliability and confidence in their systems?
Brian Allbee: I think the fundamental important thing there is going to be getting some sort of a consensus from anybody who is involved, all of the stakeholders, and anybody who is going to be held accountable for failures in the code, as to what the guardrails need to be. I like AI from the standpoint of code generation for things that are not sensitive. If it affects somebody’s lives or livelihoods, I do not want randomly generated code out there without really good guardrails.
The approach that I have seen and tried myself that has the most promise is to take a test-driven development approach and define a test suite, and only allow humans to modify that test suite. Make sure it is really good, really solid, really rigorous, and covers all of the business needs, everything that you can come up with. And then you can let an AI process go to town on the code as long as there is a clear boundary there. You can tell it, you can write all the code you want, it must pass this test suite, you do not get to modify that test suite. Let it go to town. At that point I think you are probably about as well covered as you can be.
Q. How should Python teams set up CI/CD pipelines to improve code quality and deployment reliability? What best practices help the most and what pitfalls should be avoided?
Brian Allbee: The goals are all the same for CI/CD regardless of the language involved. You have to fetch the code, you have to test it, you have to build it and package it, and you have to deploy it. The basic sequence is common across the board. There may be additional tasks like checking that the deployment process is well-formed, or aspects that are not tied directly to the code itself.
The main value that CI/CD adds is not necessarily the automation. It is the fast, automated, trustworthy feedback that you get from one of those processes. I would say look for places where you can generate that feedback, find the break points that are going to happen, and make sure that anything that fails gets surfaced in a meaningful, timely, and useful fashion.
Q. What makes a Python application cloud-ready, and what are the most important design principles to bear in mind?
Brian Allbee: Cloud-ready can mean different things to different organizations, different teams, or even individual people. A containerized application is cloud-ready provided that it can be deployed appropriately, but so too are function-as-a-service constructs like AWS Lambda functions and their equivalents in other provider spaces. It all depends ultimately on the final deployment expectations.
Some key things to bear in mind include leveraging environment variables to help control behavior in different cloud accounts or environments within those accounts. You will find that they can carry over from local development to deployment processes and build pipelines all the way out to your final deployed product. They can always be replicated and manipulated locally, and that makes things easier and faster to change in a deployed application without having to redeploy an entire stack.
Be aware of and actively seek out systems that cloud providers offer for things that you need to deal with. Secret storage is a great example. Pull a secret in one time when a container initializes or a Lambda starts up, and then do not touch it after that. Know the best practices and constraints for your final deployed code. A great example is AWS Lambda functions. You cannot run a Lambda function for more than fifteen minutes, so once you have a good idea of how long a process can take, set that timeout accordingly and test against it.
I think cloud-ready is less about where code runs and more about designing for volatility and external constraints.
Q. How do statelessness and containerization fit into building scalable cloud systems, and why do they matter?
Brian Allbee: If you think about it at a basic structural level, the key concept that ties almost every cloud resident system together, containerization, stateless design, any of those, is that they are inherently disposable. A container can be killed at any time. A Lambda invocation could be terminated before it reaches a successful completion. Kubernetes pods restarting are probably routine events. Even in a serverless context, a virtual machine can be stopped and restarted without warning. Recognizing that means designing your processes around the expectation that your hardware can disappear at any point in time.
Statelessness in that context is in a very real way about making failure of your hardware cheap. There is no state to manage, there is no need to write code to reacquire that state. A process ends and is restarted. Planning for failures and designing around the idea that recovery from a failure is just starting a new instance is probably near the top of the list of factors shaping design decisions.
In a container-based context, the container is at that point the smallest unit to replace. The key factor to keep in mind there is making sure that the startup behavior is consistent and predictable. Ensure that the environments are repeatable and allow a failed container to be replaced automatically and seamlessly rather than relying on any kind of manual troubleshooting process.
Statelessness and containerization matter more because they make failure cheap and recovery routine than for any other purpose or reason. That is what it comes right down to.
Q. A member of the audience asked: How critical is containerization to scaling systems?
Brian Allbee: Containerization is one of the more popular mechanisms but it is not the only mechanism out there. Most of my experience with containerization has been in a cloud-oriented context, and the alternatives in an AWS context at least include things like Lambda functions, which technically are their own containers but you do not have to worry about containerization as one of the factors in your code that you are concerned with. You are literally just writing code to fit inside the context of that Lambda container and letting it go. It is a good skill to have, most definitely, something that is going to be of use and interest in a lot of jobs these days, but I do not know that it is a critical skill for all cases.
Q. A member of the audience asked: Does Flask scale well?
Brian Allbee: The scaling question is so context-dependent that it is really hard to say definitively. In a containerized structure where your data store is completely separated from your Flask environment and application, and you can spin up and drop new instances of containers, I think it scales as well as anything else out there.
You will probably find that FastAPI is going to be more performant, but there is also a lot more work that has to happen in a FastAPI context. Flask is probably about in the middle. It is a good balance between a lot of stuff already supported versus speed of operation. And then at the other end you have something like Django where it does everything for everyone but it is not going to be as performant at an instance-by-instance level. After that, it is really going to end up depending on how well you can spread that load out through load balancing across containers running your application, regardless of whether it is Flask or FastAPI or Django or something completely homebrewed. That is probably where you are going to see the most scalability capability out of all of those options.
Q. What is one trade-off you see Python developers consistently get wrong when building systems?
Brian Allbee: The one I would say is most consistently seen in my experience is going back to the idea of overengineering. I want to write this as an object-oriented system because object-oriented is the way to go. And the same could be said for functional programming. Understand your problem space and design the solution around that problem space, because that is what you are trying to do, provide a solution for that specific problem space.
The best example in my personal experience was a system that was written in Python by somebody who came from a C Sharp background. The project was ridiculously huge. Every function had its own module. Every class had its own module. You put everything together, the functional layers of the system were seven or eight deep depending on the context.
Way more complicated than it needed to be, and it was hard to manage and hard to maintain. If I could have gone back and talked to, let us call him Steve, I probably would have said, Steve, there is this really good book out there called Code That Fits In Your Head. Read that. It is all about keeping things at a manageable level because psychologically humans can only keep five to seven bits of information in the front of their memory at a given point in time. You are talking about seven layers worth of depth in a project structure. That is already saturating things. Keep it simple. Collapse things down to the point where you do not have to have nineteen different classes and fifteen different instances of all these other classes to deal with something that really should be capable of being managed as a single function.
Q. How can you tell when an engineer is ready for senior-level work?
Brian Allbee: It goes back to what we started with. If they start demonstrating that they are concerned with more than just whether the code is doing what it is supposed to do, whether this function works, if there is a certain amount of curiosity about why are we doing it this way, what is the advantage of taking a functional programming approach versus a procedural versus an object-oriented one, what are the trade-offs, and do they recognise the trade-offs. Those are the things that start really indicating somebody is actually ready to go beyond just, I have written this function, it is done, it is tested, it works, I am finished.
The senior engineers that I have tried to emulate and that I have seen do their best work really are not defined by the code that they write but by the systems that they shape and the teams that they are enabling. That involves some gatekeeping, asking why we are going down this particular design path, or why are we not using this brand new library that has just shown up in the last three months. There is a lot of broader scope in asking those questions of less senior engineers and guiding them to learn how to ask those same questions on their own. Why do we go down this road? What is the benefit? What are the trade-offs? Because there are always trade-offs. Always.
Q. What motivated the second edition of Hands-On Software Engineering with Python, and what changed?
Brian Allbee: A good part of it was just the time lag. It was seven years between the first edition and the second. But Python itself has changed significantly, not so much in the language core but in the maturity of its tooling and the breadth of problems that it is now used to solve. The ecosystem around testing, packaging, automation, and deployment has grown in ways that significantly change how Python is used in real-world systems. Its adoption has also expanded dramatically, particularly in cloud and large-scale environments and also in AI, where it is very much a go-to language right now.
Today, Python engineers are frequently expected to think about architecture, performance, testing, and operational concerns in ways that just were not as common when the first edition was written. All of those growth areas and the dramatically increased surface area of use of the language kind of begged for further discussion.
The first edition tells the story of a fictitious company called Handmade Stuff that is just starting to develop an application structure to deal with what they are trying to accomplish as a company. The second edition takes that story forward and says, okay, we have this application and it is functional but less than optimal, and there is a significant impetus from the organization to move into the cloud. So what would that look like? A lot of the principles are still absolutely the same. You still need that system-level thinking, you still need to understand the problem space, you still have to work out how you are going to deploy this. What it looks like is going to be extremely different.
Q. What is the one piece of advice you would give to Python developers trying to grow into stronger engineers in an AI-accelerated world?
Brian Allbee: If you come away from thinking differently about why you write the code that you are writing, not just how, then you are moving in the right direction. Engineers are on the hook to develop and ship working code. But I will go back to my basic principles more than anything else. Think in systems. Design for change. If you are working in a team, optimise for your team.
Ask how easily the code could fit into a larger system, how it will change over time, and how your design choices will affect the product later for the people who have to work with it after you are done with it. That mindset shift is a key thing in enabling an engineer to grow into more senior roles and to build software that holds up under the real-world pressures that you are going to run into.
Brian Allbee is a Staff Software Engineer at Cleerly and the author of Hands-On Software Engineering with Python, published by Packt.
 | A guest post by
|