Security spotchecks are the driving force of The Red Guild. Thanks to them, we produce responsible disclosures, educational articles, Damn Vulnerable DeFi challenges, videos, talks and workshops.
Security spotchecks are our vehicle towards a more resilient application layer in Ethereum. But...
What is a security spotcheck?
That's our own definition. Quite broad, isn't it?
Let's dissect the key concepts of a security spotcheck at The Red Guild.
- A spotcheck is an exploration. It doesn't have a formal and structured roadmap nor process. Quite the opposite. We let ourselves loose in the jungle of the codebase, and take it from there. This works because all guild's members have a background in professional security reviews of web2/web3 systems. We trust our shared experience and intuition to serve as compass.
- A spotcheck is self-directed. We're not influenced by external actors when choosing its length, scope, and components. We explore smart contracts and off-chain components alike, based on our curiosity and expertise. Each guild member brings a unique skill set to the table, approaching the code in their preferred way. This combination of views and approaches makes the spotcheck even more robust.
- A spotcheck is time-bounded. We don't explore indefinitely. Since there's no external actor involved, then we're responsible for setting the deadlines. We commit ourselves right from the beginning to a soft delivery date. To the extent that it works in our favor, and doesn't create artificial pressure.
- A spotcheck aims to gain first-hand insights. Meaning that it's us who deep-dive into the code. We do read outside documentation, articles, etc. But always with a healthy dose of disrespect, double-checking everything against the code we're reviewing. By looking for "insights" instead of specific results, we open ourselves up to the learning process. We avoid affecting our creativity with early expectations of concrete actionable outputs.
- A spotcheck focuses on security. We don't explore a system out of mere curiosity. We want to assess how secure it is, uncovering and responsibly disclosing any security vulnerabilities. Also, security goes beyond finding or not a critical issue. Thus we pay attention to overpowered roles, unexpected edge-cases, misconfigured parameters, harmful MEV scenarios, users' footguns, fragile integrations, and more.
Our meta-process for a security spotcheck
There are artifacts and meta-processes that organize our spotcheck experience. They transform what could be a random wandering into an organized and coordinated exploration of guild members.
Evolving target pool
We have a shared list of projects that may be suitable for a spotcheck. We continuously update and prioritize it based on our interests and expertise, as well as on what's relevant for Ethereum at the time. To give you an idea, currently we have 24 potential targets in this list.
Scoping & resources page
For each spotcheck we create a shared table with all assets that we'll consider in-scope. It includes additional metadata of the asset. For example, deployed smart contracts include their deployment addresses. We also use this page as a central place to store links, diagrams, past security reports, and any resources that may come in handy to learn.
Specific pages for assets
From the scoping table we produce a page for each of the in-scope assets. These individual pages are the place where we share and document specific notes, comments, issues, small tests and questions.
Async updates
We work at different times and time zones, so for every spotcheck we create a dedicated text channel to keep in touch asynchronously on our progress. There aren't expectations on how much somebody should share. Although dropping a few messages once in a while to at least say you're alive is reassuring.
Sync meetings
We meet at least once a week in a synchronous meeting to discuss status of our work. We share and discuss anything we think relevant, and make sure everyone knows what the rest is up to: progress, code smells, findings, TO-DOs, open questions, motivation, etc.
These meetings are also the place where we decide when to wind down, and let a spotcheck go.
Monthly retrospectives
On top of the regular syncs, we write a monthly retrospective that we share with our funders. It helps us record all we've done in a month, stay accountable with our work and supporters, and reflect on ways to keep improving the guild.
We then transform these retros into our monthly public updates.
tincho
Outputs of a security spotcheck
From a spotcheck we gain insights into a system. We learn about clever implementations, code smells, footguns, edge cases, unexpected use-cases and powers, undocumented features, weaknesses, MEV opportunities, and of course, security vulnerabilities. All valuable information that we must repurpose to favor the Ethereum ecosystem.
Part of our job is to transform some rather abstract learning into a concrete shareable output. We spend quite some time discussing, designing and producing them. Some examples?
Security vulnerabilities turn into responsible disclosures. Which may turn into an article or a talk. Which may turn into a challenge of Damn Vulnerable DeFi. A weakness or non-exploitable bug may turn into a public GitHub issue (like this, this, this, this or this). A clever implementation that we found interesting may turn into a few articles, along with an in-depth code walkthrough.
Lots of possibilities. That's the beauty of exploring, learning and sharing.
Is a spotcheck an audit?
Nope. A security spotcheck is not an audit. Auditors are expected to provide high assurances that they have uncovered as many issues as reasonably possible in the given time for the agreed scope. Audits usually involve a thorough, in-depth, line-by-line review of a codebase. Auditors closely engage with clients, and (hopefully) build a lasting partnership, where auditors become security advisors at multiple stages of the software development cycle. That's so not a security spotcheck.
We work at our own pace, we define the scope, we don't produce a formal audit report, and are not constrained by client requests. Seldom we fully cover a codebase. Instead, we're likely to focus and deep-dive into specific features and components that pick our interest. For example, those that are newer, or look more fragile, or less documented, or are critical for the system.
Depending on each case, we use a combination of manual, semi-automated, and fully automated approaches for the reviews. Always trying to pick the right tool for the job.
Most importantly, instead of charging up-front, our spotchecks are pro bono. Because we're already funded by external, independent entities (right now, the Ethereum Foundation) who are genuinely interested in the well-being of the Ethereum ecosystem.
Are spotchecks like bug hunting?
If they're not audits, then you might be tempted to think that a security spotcheck is like bug hunting. Well, they do look alike. Our explorations can lead to uncovering security issues that yield bounties.
But our approach and mindset is different.
Unlike traditional bug hunting, security spotchecks at The Red Guild are not driven by potential payouts of vulnerability disclosures. We don't prioritize our targets based on the financial return of investment of our time.
During the spotcheck we may or not find a security vulnerability. If we don't, yet we think it's worthwhile, we might still reach out to developers privately, via bounty programs, or in public issues. There we'll share and discuss less severe problems, edge-cases or other weaknesses that somebody strictly seeking profit wouldn't.
Thanks to being funded by third-parties, we can take the time to do this complementary work that regular bug hunters cannot afford. And then use some alchemy to transmute what we learned into educational articles, videos, talks, workshops and challenges.
The future of security spotchecks
We're still looking for the best way of running security spotchecks. We know that spotchecks at The Red Guild, and their results, will evolve and mature as we continue experimenting.
These days we're finding new ways to turn these explorations into a more coordinated and collaborative effort. How?
Incorporating tooling for faster feedback on code snippets, building detection scripts with static analyzers, and trying out new frameworks to drive explorations.
As always, we'll be sharing our work in our monthly updates. Subscribe to receive them in your inbox!
What targets would you like The Red Guild to review in the next spotcheck? What kind of outputs would you like to see? Let us know!