Get the top HN stories in your inbox every day.
satvikpendem
throwup238
You can also set the RUSTC_WRAPPER environment variable to make it system wide.
sodality2
It should be system wide already because it’s in .home/cargo
Already__Taken
no you'd have to make that in every users /home to be system wide. Else it's just user-wide
pie_flavor
The parent comment is also system wide.
sgammon
If you’re interested in a drop in remote cache for sccache, check out Buildless
We just released S3 and Redis support. https://less.build
Buildless also supports Gradle, Maven, Bazel, CCache and Turbo
xjia
What is the benefit of using a remote cache instead of a local ~/.cache directory? Is it only for sharing build results among team members? How do you make sure the build results are not spoofed?
aseipp
Not just team members; if you make your cache publicly readable, contributors to e.g. your GitHub/GitLab/Whatever project can also use them and get really fast builds, the first time they try to contribute. So a remote cache is nice to have, if it's seamless.
Nix works this way by default (and much of the community operates caches like this) and it can be a massive, massive time saver.
> How do you make sure the build results are not spoofed?
What do you mean "spoofed?" As in, someone put an evil artifact in the cache? Or overwrote an existing artifact with a new one? Or someone just stole your developers access and started shoving shit in there? There's a whole bunch of small details here that really matter to understand what security/integrity properties you want the cache to uphold.
FWIW, I've been looking into this in Buck2/Bazel land, and my understanding is that most large orgs just use some kind of terminating auth proxy that the underlying connection/flow/build artifacts can be correlated back to. So you know this cache artifact was first inserted by build B, done by user X, who authenticated with their key K, etc etc.
sgammon
Exactly — just like Git, everything is ultimately identified with a key which can tie back to a stable identity thru OIDC or similar mechanisms. At least that’s how we did it.
yjftsjthsd-h
Nix only caches at the package level, doesn't it?
sgammon
Sharing with team members, sharing with CI, and the ability to pull from more than just what's on your machine (i.e. a larger addressable cache than you are willing to keep on disk). Cache objects also compound across projects, so it's nice to ship them up somewhere and have them nearby when you need them.
Re/spoofing, obviously it's all protected with API keys and tokens, and we're working on mechanisms to perform end-to-end encryption. In general, build cache objects are usually addressed by a content-addressable-hash, so that also helps because your build typically knows the content it's looking for and can verify.
That isn't true for all tools, though, so we're working to understand where the gaps are and fix them.
xjia
IIUC the actual computation (e.g. compiling, linking, ...) happens on client (CI or developer) machines and the results are written to the server-side cache.
By spoofing I meant to say that an authenticated but malicious client (intentionally or not, e.g. a clueless intern) may be able to write malicious contents to the cache. For example, their build toolchain could be contaminated and the resulting build outputs are contaminated. The "action" per se and its hash is still legit, but the hash is only used as the lookup key -- their corresponding value is "spoofed."
The only safe way I can imagine to use such a remote cache is for CI to publish its build results so that they could be reused by developers. The direction from developers to developers or even to CI seems difficult to handle and has less value. But I might be missing some important insights here so my conclusion could be wrong.
But if that's the case, is the most valuable use case to just configure the CI to read from / write to the remote cache, and developers to only read from the remote cache? And given such an assumption, is it much easier to design/implememt a remote cache product?
Thorrez
>In general, build cache objects are usually addressed by a content-addressable-hash
How does that work? I would think the simplest case of a build object that needs to be cached is a .o file created from a .c file. The compiler sees the .c file and can determine its hash, but how can the compiler determine the hash of the .o file to know what to look up in the cache? I think the compiler would need to perform the lookup using the hash of the .c file, which isn't a hash of the data in the cache.
sgammon
(Fwiw, group conversation encryption tech like MLS is somewhat applicable, and that's the sort of pattern we're looking at, but it would be cool to know if that's moving to you on the problem of safety w.r.t. builds.)
throwawaaarrgh
It's for sharing and aggregating. Ccache is useful locally, but really shines when combined with Distcc, a distributed compiler. Every host contributes a cache object that other hosts can use, and every host can use the cache object contributed by other hosts. So you don't even have to built it once yourself to benefit from the cache of everyone else. It therefore speeds up multiple hosts/users builds, distributed builds and the dev experience of individuals.
mgaunard
I built my own build system that does something similar.
I've set it up at work with two S3 buckets: trusted and untrusted. CI/CD read/write from trusted only. Developers read/write from untrusted, and read-only from trusted.
sgammon
We decided to back our main cache with in-memory storage for spicier performance. I’m curious how well S3 has worked for you here? Is it fast enough?
Or, maybe the blobs you’re dealing with are on the bigger end? That would also make sense
wilted-iris
I remember reading that sccache requires builds to be done in the same absolute directory by each user to share cache. Do you know if that's correct?
sgammon
Still the case, yes, but stay tuned as we’re working to fix that ;)
__float
Is this only a remote cache for Bazel, but it does not support the remote execution API at all? It's a little worrisome to trust all user outputs when you do not also control the execution of them. (In the "best" case this could mean caching non-reproducible ("works on my machine") build results, in the worst case this could be actively dangerous if a malicious user poisons the build cache.)
sgammon
It’s only a remote cache and that’s deliberate. We see it as much safer to only offer a cache that the user can control and use however they want
We would see taking over execution of your build as much more dangerous.
No question though that build caching in shared form, in SaaS form, needs extra special attention paid to security. Our product doesn’t introspect cache blobs and in fact doesn’t really want to. Once we figure out how to make the crypto work, we shouldn’t be able to see any of that data at all.
Access can be made public for reads (OSS) but is always identified for writes.
sgammon
(Also, speaking as a Bazel user now, the Remote Execution APIs have always been a bit brittle and hard to setup, use, and maintain; certainly harder than just setting a cache endpoint.
I’ve found that remote execution ends up returning much less benefit than remote caching, but that’s just me and it’s entirely possible I Did It Wrong the whole time)
sgammon
Yes!! Glad people are thinking about this. We just added Cache Projects which will be launching soon, it should allow this style of public cache sharing.
The intent with Buildless is to release a free-first toolchain that helps with build caching in earnest and makes the whole problem much less error prone. Then the Cloud stuff on top is for groups who need more gas. Cloudflare is generously supporting our upcoming free tier.
sgammon
Our beta is open, just shoot me an email at sam@less.build if you want to try it out!
phamilton
I sent an email and it was blocked: "Recipient address rejected: Access denied."
sgammon
Oh no. We just switched to MS365, maybe something is amiss.
sam@elide.dev is fine too, thank you deeply for reporting this to us
sgammon
This is fixed now. It looks like our users and domains made it over but not some domain aliases.
Seriously, thank you!
wongarsu
sccache is also delightfully simple to set up if you just want local storage. It's my go-to solution for sharing build artifacts between rust projects
MuffinFlavored
> It's my go-to solution for sharing build artifacts between rust projects
Could you expand a bit please? As somebody who is too cheap to get more than the 256gb MacBook, those heavy ~1gb random target/ directories add up over time. Didn't know there was an easy/cheap way around this?
wongarsu
sccache mostly helps to cut down build times. For saving space, the simplest trick is to use the same target/ for all your projects, by setting
export CARGO_TARGET_DIR=/absolute/path/to/target
Though beware that projects that share a target dir can't build simultaneously. If you have multiple projects open that can get annoying, because it can also effect the language server. I'm that case you can use gentler measures like cargo-sweep.
IshKebab
In my experience the time this saves is generally outweighed by the effort of setting it up and the many hours lost when it goes wrong and you don't think to try a clean build. It's certainly better than ccache in that regard but you really need something like Bazel if you're going to be aggressively caching C++ builds like this (or impure Rust builds).
(And if you're using Bazel or one of its brethren then they generally have native remote caching and execution support.)
goku12
Setting it up took hardly 5 minutes for me (for Rust). And it hasn't caused any issues so far. Forcing a fresh build of artifacts is also very easy.
Meanwhile, the only legitimate problem you mentioned is if it causes a build error and we don't immediately consider it as the source of the issue. But I use the check command so often that it is easy to suspect the cache if check succeeds and build fails.
saghm
> Setting it up took hardly 5 minutes for me (for Rust)
You're underselling it, honestly. For those who haven't looked into it, if you want to enable it for all Rust projects on your system, literally all you need is install the binary and then add this to ~/.cargo/config, and then it will be enabled whenever you invoke `cargo` (or even rustc directly iirc) from the user who's config file you modified)
[build]
rustc-wrapper = "/path/to/sccache"
IshKebab
When these things go wrong it often doesn't cause a compilation error; it can just cause inexplicable runtime behaviour.
You may be lucky and be building a pure or mostly pure Rust program, in which case it works pretty well. Throw in some C/C++ and it starts to degrade (though it's still better than with an actual C/C++ program because you aren't actually editing the C/C++ code generally).
And are you actually using remote caching/compilation? Because there's no way you can set that up in 5 minutes.
sgammon
After using sccache and Gradle with Buildless for months, years, I have literally never seen these tools mixup or use the wrong binary objects.
Knowing the internals of some of them, I’ve found that build cache clients are way more likely to miss with a cache key misalignment than they are to mixup two objects. I’m sure it’s possible, I’ve just never seen it happen in the wild, after extensive usage.
Generally speaking these tools are very conservative about two inputs matching: an identical file at a different path will cause a cache key change in sccache.
goku12
> And are you actually using remote caching/compilation? Because there's no way you can set that up in 5 minutes.
Who said anything about remote caching? You don't need it to benefit from it. It's useful if you build a lot of Rust code. A lot of packages turn up repeatedly as dependencies among several projects.
> You may be lucky and be building a pure or mostly pure Rust program, in which case it works pretty well. Throw in some C/C++ and it starts to degrade
You're making assumptions again. What is the issue with pure Rust code? Rust isn't like Python needing C or C++ support for process-intensive parts. And much of the C/C++ dependencies are dynamically-linked, with Rust wrappers. I haven't seen many projects that require Rust and C/C++ code to be built together and statically linked.
Besides, I haven't heard anyone complain about sccache that much. How prevalent is the degradation anyway?
dataangel
ccache pretty aggressively scans for things like __DATE__ and has direct understanding of compiler flags so instances of bad caching are pretty rare, I think I've only had it happen twice in years of use and once was a GCC bug. Also since ccache supports s3 out of the box now remote caching is pretty fast to setup. Remote compilation is not handled by ccache at all. You need distcc or icecc for that.
creer
I keep reading this "When these things go wrong" - which is necessarily very intermittent. What is it that goes wrong with these?
heads
We used this for a while at Speechmatics but our LM researchers have a well established workflow based on git working copies on NFS /home and we had a lot of instability between sccache and NFS.
Is sccache susceptible to cache misses when using full paths as cache keys? It would be very helpful if compiling “/home/heads/project/foo.c” could use the cached result of compiling “/home/thunderbong/project/foo.c”.
slavik81
With ccache that would be a cache miss by default. However, that could be made a cache hit by configuring the CCACHE_BASEDIR option. There doesn't seem to be an exact equivalent in sccache. https://github.com/mozilla/sccache/issues/35
phlip9
sccache only caches if builds are run from the same absolute path, so indeed different home dirs won't work
tentacleuno
As a bystander, what would the reasoning be for doing this? I would have assumed that they'd hash each file and use that as a key in a lookup table.
sgammon
In some languages, symbols are provided which evaluate to a file’s path or directory parent, so program behavior can vary even for the same content hash. That’s just one way paths can bleed in to violate hermeticity/correctness.
MereInterest
It sounds like somebody assuming a docker build, where everybody’s build will use the same file path. It’s still a very silly restriction, because not everything occurs within docker.
jandeboevrie
Regular ccache supports remote storage as well: https://ccache.dev/manual/latest.html#_remote_storage_backen...
CaptainOfCoit
I guess the difference is that sccache supports cloud storage (S3, R2, Google Cloud Storage, el al) out of the box while cache doesn't, as far as I can tell.
slavik81
The sccache description was written before remote storage was added to ccache. Remote storage is a relatively new feature for ccache, and it didn't exist when sccache was created.
kinkaid
The ability to use the local and remote caches in tandem is the most important feature to me. sccache will use one or the other exclusively, so successive builds will have to pay the latency and bandwidth costs every time an artifact is built rather than just unpacking from the local cache. ccache stages remote artifacts locally by default, so it only pays the network costs once per artifact. In CI builds they are more or less the same, but the local build experience for ccache is much nicer imo.
sgammon
(ccache is only about caching, not distributing builds, also...)
sgammon
both are supported by buildless as well https://less.build
fifteen1506
IDK what happened at Mozilla but keep going!
CaptainOfCoit
Worth noting that the first commit in sccache git repository was in 2014 (https://github.com/mozilla/sccache/commit/115016e0a83b290dc2...). So I suppose that what "happened" happened waay back.
Then in 2016 it seems like sccache was re-implemented in Rust (https://github.com/mozilla/sccache/commit/3da89195ce91a576cc...), from the initial Python implementation.
altairprime
Also around then, taskcluster happened. https://github.com/taskcluster/taskcluster/commit/54ffef79db...
throwawaaarrgh
It would be nice if every application in the world didn't have to hack on support for X number of different almost-identical storage vendors who all decided it would be better to have completely incompatible interfaces.
NFS isn't horrible, though it's limited. OTOH object storage is limited but has some other advantages. So there needs to either be the latter as a standard that can be adopted by all vendors, apps and OSes, or a new standard that fills the gaps of what apps want.
sgammon
sccache uses OpenDAL… https://opendal.apache.org/
szundi
My first thought is how to prevent bad guys injecting rootkit binaries into these systems.
hulitu
> Mozilla sccache: ccache with cloud storage
RCE to a new level. Trust us, it is secure.
Sytten
Can you combine that with cross rs? I positively hate the official GHA cache action so anything to replace that would be nice. But we cross compile with cross rs.
sgammon
Another alternative is Buildless with the Actions setup step. This sets up a remote and local endpoint for sccache inside actions, with a connection up to remote caching by Dragonfly
Scarjit
It should work if you modify the cross docker image, so that it uses the sccache executable as wrapper
Get the top HN stories in your inbox every day.
I use this with Rust, works great. Simply add a ~/.cargo/config.toml with
And it will work everywhere with cargo. I also like to combine it with the mold linker.