by Grayson Hardaway, Engineering @ r2c
Bento is a new code check tool just released by the team here at r2c! We love code checking tools because they help us write better code, faster. However, sometimes we’re left wanting for a bit more…
Oh, ESLint rules aren’t enabled by default? Which is better, Airbnb’s or Google’s rule list? Bandit is giving me thousands of errors? Oh, those are just
asserts. Why does Bandit consider
assert a security violation? 🤔 Shouldn’t this be easier?…
Wouldn’t it be great to have a code check tool where you could just run
check and it showed you the results you cared about?
Developer time is valuable, and we frequently hear from developers that their linters demand a lot from them. Your linter shouldn't bother you with things you don't care about, so we turn off checks based on these three things:
Therefore, Bento’s default configuration explicitly disables all style and preference checks. Bento only reports violations that might** actually be a problem**.
For ESLint (with the Airbnb config), this amounts to disabling all checks labeled “style” by both ESLint and Airbnb, checks with “prefer” in the name, and a handful of checks that our manual examination determined did not indicate the presence of an actual bug.
For Flake 8, this amounts to most
We hypothesized we could get a good estimate for checks that developers like and dislike by mining the configuration files for code checking tools in open source code. Using our at-scale code analysis platform, app.r2c.dev, we mined the ESLint configurations of the top 1,000 npm packages (as of April 2019), and 128 Bandit configurations on GitHub.
Perhaps unsurprisingly, about 38% of repositories globally disabled
no-console. Apparently, people like
console.log, which makes sense! We disable this check in Bento.
Of the Bandit configs, nearly 80% disable
assert_used - B101! We disable this check in Bento, too. About 45% also disable
random - B311, so we disable it as well.
As you might imagine, this is the first step in tailoring checks to your code. In the future, Bento will show you the checks you want to see by identifying which checks get fixed… and which get ignored. This will be incredibly powerful—and necessary—as we add support for more code checks.
Granted, the definition of “bad” is subjective! 🤗 Generally speaking, our definition of a “bad check” is a check with a high false-positive rate and a low chance of being a bug.
As an example,
B105 checks for
hardcoded_password_strings. In theory, this check is great!
- it prevents checking in hard-coded secrets, and
- it prevents easily guessable defaults — for example,
However, in practice, this check is very noisy. The check itself looks for assignments or comparison operators that “look like a password.” This check commonly has a false positive on the variable name
token, frequently used in string parsing operations.
We use our static analysis platform to test checks on real-world code. We can test on thousands of open source repositories at once to see where a check yields a true or false positive. Usually, it is easy to spot false positives—such as
token in string parsers.
# https://github.com/cobbler/cobbler/blob/91e3ff1d2a05eeb24107ef72e09c1a3d800ae304/cobbler/web/templatetags/site.py#L245 while not self.at_end(): token = self.get_token() if token == 'not': # B105 fires here if self.at_end(): raise self.error_class('No variable provided after "not".') token = self.get_token() negate = True
To be more scientific, we can measure the accuracy of the check. After rigorously reviewing 20 repositories, including Django and Ansible,
B105 hits nothing of importance 93% of the time. While the theory is good, this check is rather noisy. Therefore, Bento disables this check. (FYI, neither Django nor Ansible use hardcoded passwords!...except in their test files. 😉)
Checks we ship in Bento are reviewed this way. If a check is “bad,” it’s disabled. If “badness” cannot be determined, we leave it enabled. We figure our users will tell us if it’s bad! 😅
We care a lot about code quality at r2c, and we love our linters! We’ve made it even easier to write better code faster by filtering out the checks that we don’t want linters to tell us about. You can run Bento on your code to take advantage of these checks by following this link: https://bento.dev/.