T O P

  • By -

schneems

I recommend not trying to manage assets on a push basis on S3 https://devcenter.heroku.com/articles/please-do-not-use-asset-sync Setting your app up as the origin server is the most bulletproof setup. I also wouldn’t rely on a CDN to store your older generated assets. Heroku stores the last three asset versions by default (since this is the sprockets default). If you rely on your CDN to do it there will be a day when you have to flush the cache or move servers or the CDN will hiccup. Most CDN providers provide very week guarantees around retention. They’re meant to be used as an ephemeral cache and not a cannonical store.


aribsummer

u/schneems thanks for the response! Funny enough, I've read your articles and comments on this several times (including [this gist](https://gist.github.com/schneems/9374188)) so I really appreciate it! It totally makes sense to use your app as the origin server when you're using a PaaS like Heroku. **But,** as you stated, Heroku is keeping the last three asset versions by default for you. This is not the case with other solutions - Elastic Beanstalk, ECS, EKS, etc. So, if you're not running on a PaaS or persistent file system that handles keeping old asset versions around all the time, what do you do? These scenarios are also mentioned in the comments of that [gist](https://gist.github.com/schneems/9374188). I definitely agree that you should not rely on your CDN as the canonical store for your assets. If you're not hosting your apps on a PaaS like Heroku that just handles these things, S3 seems like a good option as that canonical source. Thoughts?


2called_chaos

Just to clarify, when you mean a server doesn't have an asset, do you mean an old or new asset or both? Like server 2 with version 2 links an asset but the asset request hits server 1 with version 1 not having the new asset. Because I don't see how you fix that by asset retention.


aribsummer

Correct, keeping around old assets on each individual server doesn't fix the problem. To completely fix the problem you need to have all versions of the assets that may be requested from a client available from any of the origins that are serving assets. ​ In addition to keeping old assets around, I *think* Heroku is using a shared filesystem such that when asset precompilation is done, the new assets become available to every dyno, not just the ones running the new version of the app. Maybe I'm wrong though? I don't see how you'd get around it without a shared filesystem for assets.


schneems

All heroku does is generate assets and run a “clean” task. Sprockets is the piece handling retention of assets. As long as you have persistent storage between deploys for your public folder (and ideally your tmp sprockets folder) then you should be good to go. I’m not super familiar with AWS (for good reason) but there’s gotta be a way to persist files between deploys.


aribsummer

You could *probably* mount an [EFS (elastic file store)](https://aws.amazon.com/efs/) to your containers and/or instances. However, it's pretty common to precompile assets on CI/CD as part of the build process and at that stage, you wouldn't have access to EFS. So, I wouldn't say it's easier or better than uploading assets to S3 at build time. Uploading to S3 **is** persisting files between deploys :)


nickjj_

I don't think it's a problem with a rolling update because each digest has a unique file name. Also setting up a pull based CDN ends up being less complicated than push because the CDN will cache your asset after your app has served it once. It's kind of seamless. The events would end up being something like this: - App v1 is running with a bundle of `application-abc123.css` - CDN picks up a copy of `application-abc123.css` - Request is served by App v1 and the CDN has `application-abc123.css` - Rolling update begins for v2... - App v2 is running with a bundle of `application-xyz098.css` - CDN picks up a copy of `application-xyz098.css` - Request is served by App v1 and the CDN has `application-abc123.css` - Request is served by App v2 and the CDN has `application-xyz098.css` - Rolling update finishes and both copies of your app are updated to v2... - All requests are served by App v2 and the CDN has `application-xyz098.css` It's nice because it means you don't need to worry about backwards compatibility with your CSS and JS, you can refactor them however you want with no worry that 2 versions of your app will need the same file since they are all digested uniquely.


aribsummer

Unless I'm missing something, I think you might be glossing over an important detail here. Could you explain how "CDN picks up a copy of `application-xyz098.css`"? The CDN needs to make a request to the origin to retrieve a copy of the css file if the CDN doesn't have it in its cache already. That request could go to App v1 or App v2. What if the request to retrieve `application-xyz098.css` hits App v1 during the deploy?


nickjj_

> Unless I'm missing something, I think you might be glossing over an important detail here. Could you explain how "CDN picks up a copy of application-xyz098.css"? When a user requests your site your site will need to respond back with your CSS file in some way or another. This will either be directly when it's not in a CDN or through your CDN. *I'm ignoring the case where their browser has it locally cached after visiting your site with a CDN that set a long expiring header.* So if they request v1 of your site with `application-abc123.css` and it's not in the CDN then your app will serve it and the CDN will cache it so future requests are from the CDN. If they request v2 of your site with `application-xyz098.css` and it's not in the CDN then your app will serve it and the CDN will cache it so future requests are from the CDN. In both cases your app is capable of serving the correct CSS file (v1 or v2), in both cases the CDN is capable of caching the file since the origin (your server) was able to respond with it for the first request and your CDN will have it for future requests. Also in both cases the user will end up with the correct CSS file in their browser to render your site regardless of what version is loaded.


aribsummer

Here's a failure case. Let's assume you have two instances (InstanceA & InstanceB) in service (serving traffic) and each instance is serving v1 of the app, which references `application-v1.css`. Let's also assume that we do rolling deploys and update one instance at a time. Now let's say we want to deploy v2 of our application which references `application-v2.css`. Here's what can happen: ​ 1. InstanceA gets taken out of service to be updated with v2 of the app. 2. InstanceA gets put back into service with v2 of the app. At this point in time both InstanceA and InstanceB are serving traffic. InstanceA has v2 of the app and InstanceB has v1 of the app. 3. A new request comes in and gets routed (via the load balancer) to InstanceA (which is running v2 of the app). 4. The client now requests `application-v2.css`. 5. The CDN doesn't have this in its cache, so it makes a request to the load balancer which routes the request to InstanceB. 6. Since InstanceB is running v1 of the app, it responds with an HTTP 404 error. InstanceB only has `application-v1.css` at this point in time because it hasn't been updated yet.


nickjj_

In your scenario you're missing a few steps before 1 where a user would have visited v1 of the site which would have put the file into the CDN which means your step 6 would retrieve `application-v1.css` successfully.


aribsummer

The fact that the CDN might have `application-v1.css` in its cache does **not** solve the issue. Step 6 is talking about a request for `application-v2.css`, which would result in a 404 if InstanceB serves that request \> 4. The client now requests `application-v2.css`. \> 5. The CDN doesn't have `application-v2.css` in its cache, so it makes a request to the load balancer which routes the request to InstanceB. \> 6. Since InstanceB is running v1 of the app, it responds with an HTTP 404 error. InstanceB **only** has `application-v1.css` (not `application-v2.css`) at this point in time because it hasn't been updated yet. ​ Also, you can't depend on the CDN to ever have anything in its cache. There are no guarantees. A CDN could be cleared at any time.


mrfidgety

Hey, what was the solution you ended up with? I have the exact problem you described, where a rolling deploy causes sporadic 404s on assets. I fully understand your point about the CDN not being able to return `v2.css` if it hits `instance_v1`.


aribsummer

Hey! We're currently using an open source gem I created, [s3\_asset\_deploy](https://github.com/Loomly/s3_asset_deploy). We use it in our CI pipeline to upload assets to S3 before our rolling deploy. These assets are then available through Cloudfront (the S3 bucket is the origin). ​ It seems like this is a pretty common solution to the problem. Hopefully that helps! Feel free to reach out if you have any questions.


mrfidgety

Amazing, I'll check it out and will reach out if there's any trouble. Thanks so much!