Module Idea: TruffleHog

[domwhewell-sage]

Description
I see bbot discovers and reports github repos, S3 buckets, AzureStorage etc.
In my testing I haven't seen it scan the github repos it discovers for secrets.. trufflehog has an organization flag which will enumerate all repos belonging to an organization and search them for secrets. It also has flags to include members and the repository's and to include forks which may be useful.

[TheTechromancer]

This is a great idea, however there are some challenges in implementing it. Right now, our github module searches github for files containing the target domain and raises them to httpx, which pulls down the raw file and raises it as an HTTP_RESPONSE. By default this event type is not displayed, but it is distributed internally to modules such as excavate and secretsdb which parse it for URLS, domain names, secrets, etc.

Looting the target's entire github is something I've wanted to do for a while, but the difficulty is that it's very hard to know for sure whether the github org is truly owned by the company. This is the same difficulty as we have with buckets. We sort of loot them as we see them rather than searching for them, because it's very easy to step out of scope and generate garbage data.

So basically, we are searching github for secrets etc. via the secretsdb module, but not quite in the way you've described. It's possible there's a better way to implement it that I haven't thought of. Curious to know your thoughts.

[domwhewell-sage]

Hi @TheTechromancer,
My thoughts initially went to trufflehog as it allows the organization flag and the ability to include members and forks. It also seems quite fast when running across large organizations.

For either module a good way to validate would be:

• Obtain the top level domain by stripping out the .com/.co.uk/.org etc. So blacklanternsecurity.com would become blacklanternsecurity which would be a potential organization name
• Make an unauthenticated API request to https://api.github.com/orgs/{potential_org_name} (Unauthenticated because if a token is included you may get 403 if the organizations security policy blocks PAT's)
• This response JSON will contain a "blog" key with the link to the organizations site match this with the original DNS_NAME or skip the module if not

A list of potential organization names could be created using all of the discovered DNS_NAME events that are in scope

[TheTechromancer]

Ohh interesting. I like that approach; try the API and if it returns any in-scope domains, we can assume it's in-scope.

If we're doing it that way we can also try:

• Azure tenant name (*.onmicrosoft.com)
• Github links (extracted by the social module)

[TheTechromancer]

@domwhewell-sage I have just noticed a bug in secretsdb which seems to be causing it not to work properly with the github module. Up to now secretsdb has been our "one stop shop" for extracting secrets, etc. We are using the regexes from this repo, which combines the signatures of trufflehog and gitleaks along with some custom validation logic to weed out common false positives.

Anyway, it's a small bug and an easy fix, but it highlights the need for some polishing in this area. Secrets-patterns-db was a cool idea but it's not being actively maintained. Rather than maintaining our own regexes, I would ideally like to have a weekly CI pipeline that aggregates the latest signatures from all the competing "secrets mining" tools, cleans/dedups them, and publishes them in a JSON file for the BBOT module to consume. The only reason I haven't done this already is simply because I haven't found time.

I am not inherently opposed to implementing trufflehog directly, but I think there's the potential for us to create something much more powerful if we can build it natively into BBOT's recursion.

[domwhewell-sage]

Ok that sounds good, by implementing truffle hog we may loose other goodies that could be in the HTTP_RESPONSE like user emails, and the ability to produce CODE_REPO events

Also an interesting side note from blackhat while at a GitHub copilot talk, they also mentioned that they would be using AI to detect generic secrets https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/enabling-ai-powered-generic-secret-detection

[TheTechromancer]

Migrating to discussion.