I recently started a new role, and I wanted to talk about what I did in my last role—it wasn’t top secret or anything, but it was somewhat different than my normal operations. I’d like to thank everyone at Designmind for giving me this opportunity, as it was a very interesting role. I didn’t leave the job because I disliked the project, I mainly didn’t see my skillset as being compatible with a firm like Cognizant, but I wish everyone there the best.
The Project
When I started at Designmind, we weren’t quite billing on the project—my role was to be data architect, I guess, but that really pretty quickly evolved into being the everything architect. While I did some other client work while I was there, nearly all of what I did during my last job was supporting this project. The project itself was a new application development project for client who aimed to build a supply chain resiliency system. We had a small team of a project manager, a data engineer, and two developers. Sadly, early in the project, we had to remove the data engineer as his skills didn’t really align to the project. That left all of the data tasks to myself and my project manager, who was a massive help. We’ll talk more about data later.
The client specified a fairly specific technology stack, AWS, Python/Flask/SQLAlchemy, PostgreSQL, Kubernetes which I was happy to adopt. They also had some specific requests around DevOps and security where we differed, and went in another direction, based on some other business requirements that the application had. While, I’ve been experienced with all of these technologies, I also had to help our developers get up to speed. We had a sample app one of the developers wrote, so at the beginning of the project, I containerized all of that code, and wrote a bunch of scripts to automatically deploy the app, on Windows, Macs, and Linux, as our target would ultimately be Linux.
Getting Started with AWS
I’ve always worked with AWS, just not as much as I’ve worked with Azure. I’ve had the good fortune to work with both vendors and customers in various AWS projects, so I had a good feel for sizing and performance. The first thing I did was to define the network configuration and building a VPN—I built everything on a private network from the beginning of the project. The basic architecture of the app was that we were going to have front-end containers running React and Nginx, connecting to a couple of middle-tier containers running Python/Flask, with a database running Postgres. Given the fairly narrow scope of this deployment, I used Elastic Kubernetes Services and Amazon RDS for PostgreSQL. For local testing and development, I instructed the team to have a local PostgreSQL instance, and Docker with Kubernetes enabled, so we could install the same stack locally, and use similar deployment scripts.
This kind of leads us into DevOps—which I’m not sure why, but fell into my lap. The client wanted us to use Jenkins, however, the ecosystem and community is dying and it was really complicated and proprietary. I have very good bash scripting skills, so using GitHub actions was a more natural fit to me. We faced a couple of challenges in building out our DevOps workflows—the first was that the builds operated differently on AWS as opposed to locally—this is easily taken care of by making a call to 169.254.169.254 which is a cloud metadata endpoint which works on all clouds, but can let you identity where you code is running. This mattered because we were using IAM authentication and other conditional deployment steps based on build location. Not just in our build process, but in our Python middle tier, I implemented conditional logic to decide how to authenticate to the database.
The Data
I’ve written here before about our data flow process and how we used some AI tooling to improve it. Our data flow and engineering process was really confusing to a lot of traditional data engineering pros I talked to about the project. The biggest issue was we didn’t have regular flows of inbound data—we had two data sets coming from the federal government that were published nightly. Those were easy—I built an Amazon glue job that downloaded, wrote them to S3, and did some degree of cleanup. That Glue job, when complete triggered a data ingestion process—Glue has its limitations, but for this data, it was fine.
The rest of our data sources were either what we could find or sporadic. From the earliest days of this project it was very obvious to me that we would have a data sourcing problem. While the federal government did supply some data, we were either going to have to gather, scrap or buy data. We ultimately bought data from a vendor, who was terrible (bad inconsistent data). I asked the vendor at one point if they could provide a delta file (a file of just changes) and they didn’t know what that was. If you know my email, and are in the data market—email me and I’ll tell you who not to buy data from. That vendor data did provide a basis for our webscraping efforts, which was pretty cool. Most of our reference data like geography, congressional districts, etc. was open sourced and downloaded in our environment. My project manager helped a lot here, identifying and vetting potential new data sources and getting us started with getting them integrating them into the app.
What I Learned
At this point in my career, I’ve been functioning as an architect since about 2013. A lot of people try to define what an architect does, and they try to do it in the context of what actual building architects do, or what some business book written by someone who’s never done the job, or written a line of code thinks an architect does. A good architect has to be ahead of the project, to understand where the priorities of both the project and client wants. The architect needs to be flexible and forward thinking. I always make culinary comparisons, but the role is a lot like being an executive chef. It doesn’t matter if your create amazing recipes (or designs), if your line cooks (or developers) don’t have the skill set to execute them. You either have to increase the skills of those workers (best), hire new workers (hardest), or dial back the recipe/design to match the skill of your team. The architect also needs to think about the problems the team is going to have next—mostly from a technical perspective, but also organizationally, or tooling. If you can get in front of those problems, you can help your developers do their jobs better.
It was a cool experience to be working on an app dev project. I was happy to get to push my skill set and help others grow their own. I would have loved to have completed the project, but unfortunately circumstances got in the way.








