“Right-fitting”性能反馈循环到自动化的管道

Feedback loops become increasingly critical the faster you deliver software. Effectively implementing automated approaches to feedback loops is all about right-fitting the who\/what\/when\/where aspects. Typically, automated feedback loops live in pipelines and CI. But what about performance testing? Isn\u2019t it too complicated to fit into DevOps and automated contexts?<\/p>

The short answer is, of course not. But it takes a little engineering thought and setting the right goals first. Why do performance engineers need pipeline-driven performance feedback sooner? A few solid reasons:<\/p>

• It\u2019s often cheaper to address obvious faults early in product and feature lifecycles.<\/li>\t
• Long cycles introduce delays and context switching, which increases the likelihood of faults.<\/li>\t
• Automating feedback requires us to drive unnecessary variance out of our processes.<\/li>\t
• Having feedback helps us make better-informed decisions moment by moment.<\/li>\t
• Frequently generating performance data allows us to understand trends over time, not just \u201cbig bang\u201d point samples.<\/li><\/ul>

  Traditional load testing and performance testing has focused on large, complex end-to-end testing of fully baked pre-release systems. Most automated build and packaging cycles take minutes, not hours, so traditional performance test cycles don\u2019t fit well into delivery cycles from a time perspective. Is end-to-end testing still necessary? Yes. Is it necessary for this process to inherit flaws, faults, anti-patterns, and bad assumptions that we could have addressed earlier? Absolutely not.<\/p>

  An intelligent approach requires \u201cright-fitting\u201d load and performance testing into automated pipelines so that we get early signals that are actionable (i.e., feedback loops). From an engineer\u2019s perspective, in order to avoid a mess of end-to-end late-cycle validation, we need to break down the performance verification process into meaningful smaller feedback loops that fit into shorter work cycles. This way, developers and teams can receive performance feedback about their changes earlier and more often so that by the time we test end-to-end, it\u2019s not a dumpster fire.<\/p>

  3 big challenges to modernizing load and performance testing<\/strong><\/h3>

  • Not all systems and components are created equal,<\/strong>\u00a0so no one single approach applies across the board. Fitness and candidacy for early vs. late testing requires discussion; pick the right targets and apply a progressive testing strategy.<\/li>\t
  • The more frequently you do something, the \u201ctricky bits\u201d sum to a volume of toil<\/strong>\u00a0(unless automated). Specifically, for load testing, manual pre-configuration of infrastructure, data, and environments often takes a lot of time.<\/li>\t
  • If we only do something infrequently, it takes more effort than if we did it regularly<\/strong>. Flaky environments, brittle test scripts, out-of-sync data . . . because we don\u2019t exercise them continuously, our ability to address these kinds of problems doesn\u2019t improve.<\/li><\/ul>

    A proper performance testing tool belt includes a number of practices that inherit these challenges: load testing (i.e., simulating realistic conditions), monitoring, mocking\/virtualization, test data management\/sanitization, experience sampling, among others. Most enterprises I work with have either partially or fully automated many of these components with the help of tools like Delphix, Mockiato,\u00a0Tricentis NeoLoad<\/a>, Grafana, and Selenium. The good news is that there is nothing insurmountable to any of these approaches when automating them; having a solid vision and an approach aligns and organizes our efforts.<\/p>

    Automated continuous performance and load testing<\/strong><\/h3>

    It\u2019s easy to say that the goal of test automation is to \u201cgo faster\u201d or \u201caccelerate delivery.\u201d That applies when it works reliably and produces actionable outcomes, but getting to \u201creliable\u201d and \u201cactionable\u201d takes more than load testing tools. It\u2019s like the DevOps \u201cpeople, process, technology\u201d Venn diagram. If you\u2019re only solving for one of these aspects of the problem, you\u2019re not really solving the whole problem. So how do we solve for the \u201cpeople\u201d and \u201cprocess\u201d elements involved in continuous load and performance testing?<\/p>

    Enter automation. Automating our processes puts us up to a challenge to articulate our goals, requirements, and activities in a way that machines can execute on our behalf. It shines light on gaps in our processes, technologies, and skills . . . but that\u2019s a good thing. We need to know our gaps in order to properly address them.<\/p>

    To automate performance tests that produce meaningful, actionable outcomes, people need to communicate about goals and outcomes. Discussing SLAs, SLOs, and SLIs up front is an important part of \u201cleft shifting\u201d performance practices. There should be some kind of performance criteria intake process. It could be something as simple as a templatized form or questionnaire about the systems and timeframes that performance testing should target. The goal is to help produce baseline automation artifacts like SLA definitions and API test details.<\/p>

    As you combine these artifacts into a pipeline (i.e., our \u201cprocess\u201d), you will also have to automate the load infrastructure provisioning\/deprovisioning process so that you can run your load tests properly isolated from the Systems Under Test (SUTs). These days, containers and Kubernetes have come a long way to providing a standards-based approach to automating infrastructure, but there are always a million other ways to manage these resources too. Whatever provisioning strategy you employ, it shouldn\u2019t be complicated for teams to run a test. Autonomy to obtain feedback when necessary is a key component to accelerating delivery.<\/p>

    If your performance practices aren\u2019t automated, you\u2019ll scale-fail<\/strong><\/h3>

    Finally, automation shouldn\u2019t just be in the service of a single team. There\u2019s no force-multiplier to that kind of work in larger organizations when it\u2019s just a local efficiency (for a Performance CoE team). Conversely, automation that many teams can use, even if they aren\u2019t the experts that set it up, is what really helps organizations go faster.<\/p>

    In DevOps, the work of performance and reliability engineering isn\u2019t just running tests or analyzing\/consolidating results; it\u2019s providing product teams a way to do these things themselves, while also providing safety guardrails and \u201cgood practices\u201d around these processes so that teams can grow their own performance competencies over time.<\/p>

    I see many mature large-scale performance practices that have transitioned from the \u201cperformance team as a siloed service\u201d mindset to a \u201cperformance processes as self-service\u201d offering. One reason is that there\u2019s just no way to scale \u201cpeople\u201d performance expertise to hundreds of teams by adding more full-time bodies. But mainly it\u2019s because we can now formalize (i.e., automate) our performance testing practices into work where machines help with the heavy lifting.<\/p>

    In many ways, automating your load testing practices appropriately \u201cbrings the pain forward\u201d and puts the forward tension on organizational aspirations to accelerate delivery. Rather than the reflexive approach to leave behind \u201ctraditional practices,\u201d we need to re-think them, inherit their useful points of wisdom, and adjust to the constraints and challenges of today.<\/p>

    Where do we go from here?<\/strong><\/h3>

    Running performance tests early and often at smaller volumes is the key to establishing \u201cright-fit\u201d automated performance feedback loops into pipelines. Start small, targeting APIs, to build confidence and competence.<\/p>

    Practical how-to guidance on laying the foundation for a successful transition to an automated Continuous Testing approach \u2014 including strategies for prioritizing what to automate, best practices for developing dedicated performance pipelines, overcoming test infrastructure obstacles, and ensuring trustworthy go\/no-go decisions \u2014 is detailed in this\u00a0Practical Guide to Continuous Performance Testing<\/a>.<\/p>","quoteUsername":"paulsbruce","translations":{"Common":{"like":"Like","unlike":"Unlike"},"Forum":{"Quote":"Quote & Reply","Share":"Share"}}}">

    • 就像
    • 报价和回复
    • 分享