Well, the problem was that the app was already in production, so the change required us to move all the existing assets to the S3 in order to prevent losing data (obviously!).
During our research we found an easy fix to migrate all the assets to S3. The solution all over the Internet is quite self-explanatory; just send the folder where your assets are stored to the S3 (the well-known public/system folder)!
This would work in a situation where the url of Paperclip configuration was previously defined. The default schema that Paperclip uses ends up in a file with the location like:
To prevent losing time figuring out where /000/000/ came from, the explanation for all the leading 0’s, and in order to create a flexible solution, we decided to make our own strategy.
We need to move all the model attachments;
We need to preserve all the styles;
This needs to be done without locking the server;
S3 bucket should be different according to Rails environments (test, production, staging);
We need to have minimum downtime.
First we need to be able to switch the configuration on Paperclip to use S3 after migration, and to use our server during migration (we need to get the attachments files).
Lets see a typical model with attachments:
To prepare the model to be migrated to the S3 service we need to add a switch that can be easily activated.
Lets create an s3_migrate.yml file with some configuration:
From line 1 until 4 is the normal configuration for the Paperclip S3 storage, in line 5 we added the switch to activate the S3.
If it's not activated, the application should serve the assets from the local server. Next we change our model definition in order to reflect these changes.
In this snippet we store the Paperclip options in a hash (PAPERCLIP_STORAGE_OPTIONS) and load the file containing the migration options. We merge the S3 settings if activated is true.
For now we've prepared our model to switch to S3 after the migration process is complete.
In order to migrate our assets to S3 we will need a custom task to perform the action. Here is the snippet responsible for it.
Create a file named paperclip_migrate.rake and put it on lib/tasks folder and paste the code above.
Run the task:
**After finishing the migration we need to update our s3_migrate.yml and activate the S3 storage. **
Just change the line activated to true. Restart the webserver and you are done!
Note 1: The path setting in s3_migrate.yml should be equal to the one in the line 44 of the task. Change that according to your scenario.
Note 2: After completing the migration and assuring that our assets were successfully migrated to S3 you can delete the files that are still stored in public/system.
Lets iterate through our requirements for this problem and validate each one of them.
As we add the ability to specify the class and objects dynamically, one can execute the task with whatever models wanted, example:
Accomplished!
In our task, after we have checked that there is at least one object in our database, we can query it to retrieve all the defined styles:
It will return what we expect:
Accomplished!
We use a rake task to perform the migration so it can be executed without affecting the usage of the application on production environment.
Accomplished!
In our s3_migrate.yml we define the path of assets in order to use the Rails.env variable, so the assets will be saved according to the environment that the application is running.
Accomplished!
We only need a single restart to the webserver! So we can say that the requirement was...
Accomplished!
And that's about everything you need to know to migrate Paperclip assets from a local server to Amazon S3.
Experienced in backend systems for the web using state of the art technologies. Full stack developer & product maker. Ruby/Rails, NodeJS, ElectronJS.
People who read this post, also found these interesting:
.png)
