Skip to content

How AI is used for scryer media projects

TL;DR

AI is used responsibly, with guardrails to ensure quality goes up over time.

Summary

AI is a fantastic tool for quick and effective code generation; however, it is imperfect and must be used with care and caution. Scryer’s core maintainer, NZBMan, has a strong background in software engineering and architecture and has hand-rolled over 500k lines of enterprise-grade code over his career. Quality is the core principle of all Scryer Media products, regardless of contributor.

This document lays out the process of agentic coding used by core maintainers and the expectations on any contributions.

Quality is core

At the end of the day, software exists to provide a workflow for humans. This may be interactive like an app, or automated, or a combination. The quality of software should be judged against “does this software do what I need it to, with the safety features I require?”

To satisfy that requirement, all Scryer Media products use a comprehensive end-to-end (e2e) suite that tests the application against user behavior and automated flows. This suite is exercised regularly during development and must be green before any release or PR is merged. This suite is above and beyond the thousands of unit tests that run to ensure the code is behaving properly. For Scryer alone there are 170+ unique flows that run to exercise all the different expected workflows the tool provides.

NZBMan has found in his career that ensuring a working product is the goal of any software delivery pipeline, and the best way to ensure that is to test it as users would. In the world of agentic coding, this has never been more critical. A solid e2e suite catches regressions and side-quests before they ever hit users.

Security is non-negotiable

These tools are hosted in homelabs and must be secure. They do not rely on being “inaccessible”, they all implement industry best practice for security. Regular secuity audits are done of each product using frontier or even research grade (private access) AI security tooling. Remediations are done where they are sensible for the tools and deployment scenarios.

AI tooling use

Each maintainer is free to use their preferred agentic pipeline; however, any contributions will be judged against at least one (usually two) frontier models. As of writing this, those would be the premium models provided by OpenAI/Anthropic.

Feature development process

Every feature should go through a deliberate feature development process before any agentic coding starts. Agents work well when they have a strong implementation plan.

Large features must go through a full RFC process using multiple frontier models to create a solid (and documented) plan, with direct human oversight and review of the plan. This process can take anywhere from 3-20+ revisions of an RFC to get it to a spot where it’s ready for implementation. Full maintainers have access to the private plans repository, large features from contributors must also provide a link to their RFC before any code reviews will be conducted.

Medium to small features will often go through a much simpler /plan workflow with a single model and then adversarial reviews from another model after implementation. Contributions for smaller features must still explain clearly in their PR what the goal of the feature is and disclose AI usage to create it.

Validation and verification

Every feature must be validated before acceptance. This includes both adversarial and validation reviews as well as human review and UAT. Large features will get at least one e2e fixture, and most medium features will as well. Small features may not, depending on the criticality. Nearly every code change should require a new unit test or modification of an existing one.

Change management

Bug fixes and new features inevitably will have a ripple effect on other code. This is exacerbated by agentic coding; therefore, change management is critical. It revolves around three key things:

  • Unit test conformance
  • e2e conformance
  • UAT

Unit test conformance

This can be one of the most frustrating parts of agentic coding. Agents will write tests on day 1 that test day 1 behavior, then when behavior changes on day 2, a user will test it and validate it works, but an agent will then push the new code back to the old unit test assertions. This creates a very frustrating “whack-a-mole” pattern.

The key is to prompt agents well so that they are forced to validate why the test failed before they can change any code. This will teach them to verify if the code is bad or the test assertion is outdated.

e2e conformance

Many changes require a harness change in the e2e suite. This has the same foibles as unit test conformance, and any failure in the harness must be proven through thorough research into commit history. Oftentimes the harness does need to change.

UAT

Every new feature must be verified by a human. Period. Full stop.

Community contribution

As most Scryer Media projects are open source, contribution is encouraged and welcome! All contributions must follow the guidelines here and any CONTRIBUTORS.md of the project. Low quality submissions will be rejected. Repeat offenders of low quality submissions will be blocklisted.

Every PR will be given the same rigor as any internally developed feature. Expect your code to be analyzed deeply by a frontier model. Not all code is expected to come out completely clean as every model has a different viewpoint, but low quality submissions will be easy to spot. Your feature will be accepted into a release version of Scryer that will then be put through the full e2e suite before release.

Maintainers will decide if the PR requires e2e additions and add those internally. The e2e suite is not published publicly.