How to Bootstrap Engineering in a Startup — Part 3: Tooling and Processes

Photo by Josh Redd on Unsplash

This blog post is part of a series. If you want to start at the beginning, take a look at Part 1: Hiring.

With a team hired, you’re now ready to start building some serious tech. You now have a blank slate in front of you and a million tools and frameworks to pick from, without being tied down to an existing code base. At this point, you have a unique opportunity to standardize on a narrow set of technologies, or alternatively, decide upfront that freedom of tooling is something you want to allow.

If you’re like me, coming from a big co’s, you won’t even realize how much of your core tooling you’re missing, until you start hitting these gaps, when you’re trying to build the simplest thing.

Code repos, CI/CD, container image repos, AWS IAM, VPCs, basic infrastructure, code review policy, logging infrastructure, RPC, basic monitoring, user auth, UI frameworks, code style and linting, HTTPS certificates etc etc etc.

It gets overwhelming very quickly and we aren’t even talking about what product you should be building (which is another can of worms).

If left uncontrolled, each engineer will tend to build everything using the technologies they are most familiar with, and not necessarily talk to each other about it. It’s just normal engineer nature to be very biased towards the technologies that they already know.

Best thing to do is to democratize much of the tech stack choice: language, key frameworks, data encoding format, RPC, containers, serverless, UI framework etc. Have an open discussion about this. You should aim to be largely the referee, yourself, and less of a participant. If you’ve built the right team, you need to trust that they reach the right technical decisions. Occasionally you will need to override the team, but you should be coming at it as an equal: explain why you think otherwise and seek out guidance from the team. This will create an environment in which the team feels comfortable contributing with opinion and will typically enjoy working in.

Once a decision has been reached, it will be your job to enforce it. When someone else joins the team, you should give them a tour of pre-existing decisions and why they were made, so they can also buy into them. Some engineers will still want to do things differently — you’ll then need to make a call on case-by-case, regarding standardization vs freedom. They each have their own benefits and trade-offs and it’s your job to make the right call (assuming you are the head of engineering). In any case, don’t override people sharply — make them part of the process of decision and argue for both sides. Expect your engineers to also have this ability to argue for and against an idea — it creates a culture where everyone trusts the team as a whole to reach a good decision, rather than to chase personal goals.

As you work your way through standardizing the technology used by the team, you will be constantly faced with decisions between building the product vs building the infrastructure and process that helps with building things faster. In the early days, I’ve always prioritized the things that allowed us to build faster. For all investments that improve agility of development, the sooner you make them, the longer you will benefit from them. Dev infrastructure creates a foundation for agility, which is very much needed for iterating on the product fast. In the long-run it is more efficient that way. Here are some examples of key foundational infrastructure to tackle early on (these are SaaS-flavored — if you’re not building for the cloud, YMMV):

• Solid CI/CD with weekly releases at the least. Pipeline automated as much as reasonably possible (without wasting time trying to build the perfect system from day one).
• Solid capabilities for debugging in dev and in production (good logging)
• Solid dev environment, with the best you can get dev-test cycle agility. This is an area that you can never really spend too much time on. Every investment in this area gives back tenfold or hundred-fold ROI. If you’re using containers, you’re already a step ahead — use docker compose to mimic the way the services would work in production. If you’re using lambda or some serverless framework, make sure it runs in dev as closely as possible as it does in production. Make sure your UI engineer can also run the backend and vice-versa. Remember, that the dev env does not have to be an exact mirror with production: it needs to be a carefully chosen tradeoff optimized for fast dev-test cycles, at the cost of parity with production.
• Safe-enough releases: staging environment, feature flagging, good way to verify production is healthy (manual process is ok in the beginning — just make sure people know how to do it and that they have to do it)
• Production infrastructure that allows the team to sleep well during the night. Basic rotations for releases and on-call.
• Good user monitoring infrastructure (allows you to understand what the user does). Don’t go overboard here — make sure your user monitoring is aligned with what the product actually needs. Don’t add monitoring you don’t need — it can become a source of confusion quickly.
• Standardized engineering tooling (to the extent that it applies to your product): build system, linting enforced, programming language-specific infrastructure, basic in-house libraries for common routines, auth system, RPC (if you’re microservice-heavy) etc.
• Standardized engineering processes: code reviews, code style agreed upon, a place for the wiki.
• Basic DB schema update tooling

When it comes to DBs, I wholeheartedly recommend a SQL-based DB (unless you have some specific requirements). When we started ShiftLeft, I pushed hard on a future-proof DB. So future-proof, that it was more like 10 years into the future of the company. We were nowhere near that kind of requirement and we weren’t going to be any time soon. That was a time when the team voiced (a very valid) concern against my opinion. It’s times like these that you know the team is working well: when you’re about to make a mistake, but the team is there to convince you not to do it. We went with postgres and looking back, it was one of the best tech-related decisions we made.

The reason why SQL-based was such a good decision for us was because we had no idea from day one of all the possible ways we would query the data later on. A NoSQL DB would require you to shape-shift the data in all kinds of ways to allow for new querying use-cases. Whereas with a SQL DB, it’s often just a matter of adding another index… or doing a join. Or if all fails, you can just write a quick migration script to rewrite the data in another form. These are things that allow you to be very agile and focus on your ever-changing product, rather than babysit a DB. It’s, again, an investment in agility.

All in all, all of these things are fundamental building blocks that you should have, no matter what SaaS product you’re building. Even if you pivot dramatically, these are things you would likely keep intact. So it’s well worth investing in a solid foundation.

Higher-level features that you’re experimenting with depending on user feedback should be developed with less rigor for perfection — in fact this is where you should make sure you’re not wasting time with the best implementation until they have become validated by users. After you have validation, you can go back and do things the right way; or, if they did not validate, replace them with another experiment.

This is where your contribution is key: you have the choice between building better foundation vs building product experiments and more often than not, throwing them away. You’ll need to find the right balance between making progress on the product side while also making progress on the foundational aspects.

If you’re too focused on the foundation, you end up not building a product for a long time and thus not learning anything new about your users (remember, at the end of the day, understanding your users is the most important thing early on). If you’re too focused on the product, you take too long between iterations, because nothing works and you have to reinvent the wheel every time. It’s a fine line — you’ll need to step back to look at the whole picture and use your best judgement.

Before ending this blog post, I’ll leave you with some of our own early tech decisions that we made at ShiftLeft, to fuel your inspiration:

• In general, follow the 12-factor app methodology
• Java/Scala for code analysis (our pre-existing research was written in Java). Google Java style guide.
• Initially Gradle for building Java/Scala, later on we switched to SBT
• Go for everything else in the backend. Standard go fmt, go lint and go vet enforced at build time.
• Makefiles for building Go
• React for UI (ES6 with AirBnB style guide)
• gRPC for inter-service communication
• Protobuf for all serialization. Only exception was the UI: we decided to use the JSON format of Protobuf because JS proto support in the browser was bad… and also, difficult to debug with popular tooling. Reason for picking protobuf overall was the forwards-backwards compatibility features which allowed us to move fast and not worry about rollout plans.
• Postgres for persistence
• All services are packaged as Docker containers
• Everything has to be cloud-agnostic (no AWS-specific technology). Reason for this was the possibility that one day we would have to deploy on-prem. I wouldn’t normally recommend this: achieving product-market fit faster is far more important than staying cloud agnostic (which is very very hard).
• AWS as our cloud of choice
• Docker Compose for a standardized dev stack
• Mesosphere for orchestration of containers in prod
• Jenkins for CI/CD
• DIY auth, then later on, Auth0 (and more recently we’re looking to go back to DIY)
• Hashicorp Vault for secrets
• At least one code review approval before merging

If you’ve enjoyed this post, follow me here or on Twitter (@VladAIonescu) for more. I’m planning to write a series of weekly blog posts about what I’ve learned from building ShiftLeft.