Full Cycle Development: A Giant Leap to Optimal Operations Excellence at Netflix
The “fog of war” turns as a familiar term to military commanders, which refers to the uncertainty in situational awareness typically arising in a military onslaught. Whereas application development is, by no means, a war, developers often find themselves experiencing similar uncertainty—or so-called “fog of development”— upon working on a dozen endpoints in a platform with hundreds of different endpoints. Indeed, with so many moving pieces, it’s easy for developers to get lost in the development and operation battle.
Fortunately, within such an ever-evolving landscape, there exist several disruptive technologies, tools, team structures and methodologies to address this concern while also speeding up the delivery of digital products. Among the most transformative ones is the industry-shaking approach of full cycle development, which was coined and best characterized by Netflix with its winning ethos of “Operate What You Build”.
So, what is the rationale behind the novel approach? How about its potentials for operations transformation? How can you, as a business owner, leverage this methodology to optimize your operational excellence, especially on a budget?
It’s high time to figure out your own answer!
Why Full Cycle Development Matters? – Insights from the Case Study of Netflix
To briefly introduce, full cycle development refers to a software development methodology where a development team takes full ownership of the whole software development lifecycle, typically consisting of requirements, development, testing, deployment, and operations. To understand the ins and outs of this novel approach as well as its related concepts, let’s dive deep into the development and operations (DevOps) transformation story of Netflix’s Edge Engineering team who minted this term.
From Siloed Ultra-Specialized Teams…
As shared by Greg Burrell, Senior Reliability Engineer, Netflix’s Edge Developer Productivity team, its engineering team previously adopted the model of specialized teams, where work was performed by individual (siloed) teams focused on development, test, and a network operations center (NOC). Actually, this has been a common approach within the software development sector, in which implementation follows a production line inspired model, and work is “handed off” incrementally as functionality is built and operationalized.
More often than not, teams employing such conventional organizational structures lack understanding of the general context – the “big picture”. For instance, developers and testers do not know much about the production systems whilst the NOC teams are vague on the applications’ details, which resulting in high communications overhead, particularly when an issue occurs.
Generally, this triggers a lengthy troubleshooting and fixing process since engineers move cautiously due to the lack of familiarity with applications, systems and current state. This can foster an over-reliance on back-and-forth on the phone and ultimately lead to a lossy feedback cycle. In the end, developers tended to stay away from production unless “something was on fire” while operations teams would “band-aid over problems”.
.. To a Hybrid Model with Centralized Operations Team …
To address those technical pain points, the Edge Engineering experimented with a hybrid model, where the NOC is replaced by a centralized operations (“CORE”) team, who also interact with and empower the development teams. Besides, developers could also push code themselves when needed as well as be held responsible for off-hours production issues and support requests.
Those practices proved to enhance the feedback and learning cycles for development and operations teams to some extent, having only partial responsibility left inherent gaps.
To put it simply, even though developers could do their own deployments and debug pipeline breakages, they would often defer to the operations release specialist. In a similar fashion, as for the operations-focused team, they were motivated to manage the day-to-day work, yet still found it tough to prioritize automation so that others did not need to rely on them.
In recognition of these problems and in search of a better way, Netflix Edge Engineering took a step back and decided to start from its first principles: “Operate What You Build.”
… and Ultimately a Giant Leap to Full Cycle Developers
The principle of “Operate What You Build” puts the development and operations (DevOps) tenets in action by formulating the team of developers who are also responsible for operating and supporting that system.
Since creating silos, with different people performing each role, adds communication overhead, introduces bottlenecks, and inhibits the effectiveness of feedback loops, breaking down specifying silos and encouraging shared ownership of the full software life cycle enabled Netflix to optimize its learning and feedback procedure. Particularly, distributing DevOps responsibility to each development team, instead of externalizing it, helps generate direct feedback loops and aligns incentives. Teams that once suffer operational pain are empowered to remediate the pain by adjusting their system design or code. Meanwhile, they are held accountable for both functions. Each development team owns deployment issues, performance bugs, alerting gaps, capacity planning, partner support, and so on.
In addition to maximizing learning and feedback efficiencies, ownership of the full development life cycle plays a critical role in scaling through developer tools.
Netflix engineering team took another massive step by forming centralized teams – Cloud Platform, Performance & Reliability Engineering, Engineering Tools, to name a few – with a view to designing common tooling and infrastructure that simplifies and automates common development needs. Serving as force multipliers, those centralized teams turned their specialized knowledge into reusable building blocks.
Empowered with these tools in hand, every single development team within Netflix can focus on addressing problems in their specific product domain. As soon as additional tooling needs arise, centralized teams would assess whether the needs are common across multiple dev teams and if they are, collaborations ensue.
The combination of these ideas led to the emergence of the “full cycle developers” who are responsible for certain operational aspects of service delivery, and are supported through training and a range of self-service tooling and infrastructure. Whereas centralized teams create and maintain common tooling and infrastructure, each team within the organization has the freedom to deviate from this “paved road”.
Full cycle developers are required to apply engineering discipline to all areas of the software life cycle, including design, development, test, deploy, operate, and support. They analyze issues from a developer’s perspective and reevaluate them by wearing other hats. By favoring systems-focused over humans-focused thinking and automation over manual approaches, Netflix obtains the capability to scale its business and attain phenomenal success.
Challenges of Full Cycle Development Approach
Whilst rethinking your business operations under full cycle development approach is critical to its sustainability, there exist a handful of challenges, if not say trade-offs, that you should put into perspective.
Obviously, moving to the full cycle development model requires a mindset shift: not all developers – or all teams – embrace such a working ideology. Indeed, some developers deem design, development, and sometimes testing, as the primary value-creating ways while having an anti-pattern of viewing operations as a distraction, favoring short-term fixes to operational and supporting issues so that they can get back to their “real job”.
To make sure that this model works, adaptations are necessary. Meanwhile, teams should be staffed appropriately with enough headroom to manage builds and deployments, handle production issues, and respond to partner support requests. Business owners have to formulate detailed plans devoted to training as well as invest and leverage modern working tools.
Furthermore, partnerships need to be fostered with centralized teams to develop reusable components and solutions. Such investment areas as automating alert responses and building self-service partner support tools need to be prioritized alongside business projects. In the absence of any factor mentioned, the business runs the risk of overload and burnout.
Besides, full cycle developers should be expected to be knowledgeable and effective in all areas of the software life cycle. And again, this poses a burden on appropriate staffing and extensive training programs to impart knowledge gaps and build up necessary skills.
So, as for small businesses and startups, what should business owners do if they are yet to have adequate staffing structure and financial capacity to invest in building and training centralized teams like at Netflix? Outsourcing services, with an affordable pricing scheme and a dedicated on-demand team of well-trained professionals, emerge as a seamless match for them to try as a first pass. Let’s thoroughly evaluate your business needs and be truly mindful of bringing in the least complexity necessary.
Is Full Cycle Development a Right Choice for You?
As staffing is expensive, an integration of the offshore team into your staff stands as the perfect fit for your business. At EnvZone, we offer a team of full-cycled approach developers who work directly with your company and handle the whole development process so that you don’t have to worry about other third-party providers.
For your bottom-line, we ensure our assigned team follow the best practices, bridge the gap between concepts, the successful realization and launching of the products. Your assigned team will always be ready to offer support at all stages of your product creation, including but never limited to:
- Feasibility Study
- Prototyping & Design
- Development
- Customization & Deployment
- Support & Maintenance
As soon as our project manager identifies your business pain points and growth objectives, we will select development teams to combine the power of Agile with Lean product development and system thinking. It synchronizes alignment, collaboration, and delivery for multiple Agile teams. As a result, it dramatically improves business agility by accelerating productivity, time-to-market, quality and engagement which impact the outcomes directly.
Not Sure How the Process Looks Like?
So, how do we work to make an impact on your business? Let’s take a look at our process framework.
#1. Discovery Phase
The purpose of the discovery process is to gather information and to determine if this solution is meeting your criteria. Within this phase, there are four key deliverables to consider:
- Initial Meeting: The focus of the initial meeting is YOU. We listen to know what you need and understand your motivation.
- Needs Assessment: We hold a consultation to understand the rudiments of your project. Once all your requirements are met, your team will start working on your project under your full supervision.
- Team Selection: To ensure the quality, we perform thorough vetting of our international team, so you have only the best of the best, highly skilled and experienced hands, working on your projects.
- Proposals: Based on your requirements and company structure, we will run through our best, vetted, and ready developing teams to give you our proposal that outline the needed support for your project.
#2. Initial Engagement
This works as the first real interaction between us and you. Similar to the discovery process, we typically conduct four engagement practices as follows:
- Detailed Requirements: The detailed requirement is an emphasis on the operation and process. It specifies all of the rules that must be enforced to ensure that we understand what to build.
- Wireframes & Clickable Prototype: We create wireframes and clickable prototypes leveraging tools and techniques to present the design concept and the basic function for your project.
- Research: The research process is involving the definition of the problem, determining what approach to take and ensuring that our developers on the project understand your requirements.
- Detailed Quote & Project Timeline: We use the project timeline to track major milestones for your project. The purpose of a detailed quote is to provide as much information as possible. We will show what work will be done, the hourly rate for people doing the work, etc.
#3. Development Process
Following discovery and initial engagement is the development process whose primary purpose is to analyze your requirements and building quality and correctness of good software.
Particularly,
- Predefined Sprints: We deliver each sprint individually and make sure that all of the issues are corrected before handing over to your team.
- Quality Assurance: Quality assurance is a process in which the technical aspects of your projects are verified to ensure the products match with your project requirements and quality standards.
- Deployment and Launch: We will support you through the necessary deployment and launch processes. Once the deployment meets your requirements, all workloads will be moved to another system.
Should you be still concerned about our handy yet super-powerful offerings? Then, let’s sign up for a free account and start assembling your team today.
The Bottom Lines
Over the recent years, the technology development landscape has witnessed disruptive changes, especially in terms of developers’ roles – from those being highly specialized, to dabbling in multiple parts of the stack, to today’s blurring of operational lines in a hybrid world. One emerging tech role is that of full cycle developer.
Whilst the working ideology of full cycle development comes as a forward-looking and profitable approach, challenges do still exist within the fast-pace changing industry and we always keep learning. Should you wish to lend a helping hand from a dedicated team of professionals, don’t hesitate to contact us and collaborate to drive an impact on your business.
