Implementing direct file uploads with Shrine and Uppy in Ruby on Rails

Heres how Ive implemented direct file uploads to S3-compatible object storage with Shrine and Uppy in Rails

Mar 08, 2024

File Uploads - The naive approach

The naive approach to upload files would be for you to

Build a web page that lets your users upload files
when the user clicks button for file to be uploaded
1. The file gets sent directly to your server
2. Your server processes the file and passes the file on to your storage (e.g. S3)

However! This approach sets your server right in the middle of file uploading and puts a bunch of unnecessary extra load on your server which is not great. Your server wants to be serving requests quickly, not chugging away uploading files for long periods of time.

Direct File Uploads - The scalable approach

The direct file upload pattern is super cool, I’m writing it up to help me solidify my understanding how it works behind the scenes a bit better

The direct file upload pattern:

Build a web page that lets your users upload files
when the user clicks button for file to be uploaded
1. The browser client talks to your server which responds with the S3 storage URL and a pre-signed token
2. The client then uses that pre-sign response to directly upload the file directly to your S3 compatible storage
3. After the file is uploaded your client notifies your server the work is done

A deeper dive of the network traffic in a direct file upload

User clicks ‘Upload Receipt’ and the ‘Drop Files Here’ modal appears (powered by Uppy)
User selects a file to upload and adds it to the Uppy modal. Uppy javascript manages the following:
1. we see in the network traffic tab an asynchronous HTTP GET to the /s3/params route to retrieve the S3 upload parameters. Rails uses Shrine presign to return the S3 URL where the file should be uploaded and a bunch of params that S3 is expecting
2. Uppy then makes a request to this URL and use these parameters to upload the file directly to S3 (not seen in the network traffcic tab)
  1. This is the direct upload part, the browser is directly uploading to your S3 storage. Potentially speeding up the upload process and reducing server load.
Once Uppy has uploaded the file, itgenerates JSON representation of the uploaded file on the client side, and write its to the hidden attachment field and immediately submits the form wiht the attachment
1. We see in the network track a HTTP POST to /receipts with a payload
```
receipt[image]:
{"id":"db53260bb3cbbf377d43165f19d9bfa6.png",
"storage":"cache",
"metadata":{"size":164,"filename":"Screenshot from 2024-03-08 21-26-43.png","mime_type":"image/png"}}
```

This json snippet contains the location of the file on S3. The Rails receipts_controller then creates an instance of a Receipt with an attachment that references the file on s3

The File direct upload finishes successfully here.

Libraries used Implementing direct file uploads

Uppy

Ive found Uppy to be a great file uploading frontend library. The drag and drop modal is super simple to integrate and works well on mobile too (lets you use your camera to submit photos). It also integrates with a bunch of services like Dropbox, Instagram etc

Shrine

Shrine is great for direct uploads, and has a range of plugins. I also use it for adding metadata, generating thumbnails in the background, validating files, pulling files from remote locations

Implementing direct file uploads - how to

Heres a great step by step how-to by shrine for direct uploads to s3 with Uppy

Heres a complete working example of direct S3 uploads using shrine, uppy and rails that I found helpful too

My Code Snippets

phew substack is super unfriendly to embed readable/usable code snippets. Heres my key snippets across rails/shrine and uppy (use in combination with the above howtos)

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters

Show hidden characters


	<div class="ml-4 mt-4 flex flex-shrink-0 items-start">
	<%= link_to "Add Transaction", new_transaction_path, class: "hidden lg:block relative inline-flex items-center rounded-md bg-white px-3 py-2 text-sm font-semibold text-gray-900 shadow-sm ring-1 ring-inset ring-gray-300 hover:bg-gray-50" %>
	<%= form_with(url: receipts_path, class: "test", html: { id: "theForm" }) do \|form\| %>
	<div class="form-group" data-uppy-single-media="receipt[image]" >
	<%= link_to "Upload Receipt", "#uppy", class: "relative ml-3 inline-flex items-center rounded-md bg-white px-3 py-2 text-sm font-semibold text-gray-900 shadow-sm ring-1 ring-inset ring-gray-300 hover:bg-gray-50", data: { behavior: "uppy-single-media-trigger" } %>
	<%#uppy.js will inject hidden_field media[] per image uploaded on uppy.js::setupUppy::completed %>
	<%= form.hidden_field "receipt[image]", class: "upload-data" %>
	</div>
	<% end %>
	</div>

view raw index.html.erb hosted with ❤ by GitHub

Show hidden characters

	class Receipt < ApplicationRecord
	belongs_to :related_transaction, class_name: 'Transaction', optional: true, foreign_key: 'transaction_id'
	include ImageUploader::Attachment(:image) # adds an `image` virtual attribute
	acts_as_tenant :account

	# Use store_accessor to declare attributes for serialization
	store_accessor :data_extract, :job_id, :job_completed, :job_created_at, :job_updated_at, :raw, :extracted_purchase_date, :extracted_amount_cents, :extracted_category, :extracted_description, :category_id

	# Broadcast changes in realtime with Hotwire
	# after_create_commit -> { broadcast_prepend_later_to :receipts, partial: "receipts/index", locals: {receipt: self} }
	# after_update_commit -> { broadcast_replace_later_to self }
	# after_destroy_commit -> { broadcast_remove_to :receipts, target: dom_id(self, :index) }

	end

view raw receipt.rb hosted with ❤ by GitHub

Show hidden characters

	#/app/config/initializers/shrine.rb
	require "shrine"
	require "shrine/storage/file_system"
	require "shrine/storage/s3"
	require "shrine/storage/url"


	s3_options = {
	access_key_id: Rails.application.credentials[:au_digitalocean_access_key_id],
	secret_access_key: Rails.application.credentials[:au_digitalocean_secret_access_key],
	bucket: Rails.application.credentials[:au_digitalocean_spaces_bucket],
	region: Rails.application.credentials[:au_digitalocean_region],
	endpoint: Rails.application.credentials[:au_digitalocean_endpoint]
	}



	if Rails.env.test?
	require "shrine/storage/memory"
	Shrine.storages = {
	cache: Shrine::Storage::Memory.new,
	store: Shrine::Storage::Memory.new,
	}
	else
	#if Rails.env.development? \|\| Rails.env.production?
	Shrine.storages = {
	cache: Shrine::Storage::S3.new(prefix: "cache", **s3_options), # temporary
	#https://www.rubydoc.info/gems/shrine/2.1.0/Shrine/Storage/S3
	store: Shrine::Storage::S3.new(prefix: 'store', upload_options: { acl: 'private' }, **s3_options), # permanent
	cache_url: Shrine::Storage::Url.new
	}
	end

	#https://www.ironin.it/blog/store-your-files-on-s3-using-the-ruby-shrine-gem-part-3.html
	# Shrine.storages[:cache_url] = Shrine::Storage::Url.new

	Shrine.plugin :instrumentation, notifications: ActiveSupport::Notifications
	Shrine.plugin :activerecord
	Shrine.plugin :cached_attachment_data # for retaining the cached file across form redisplays
	Shrine.plugin :restore_cached_data # re-extract metadata when attaching a cached file
	Shrine.plugin :validation_helpers
	Shrine.plugin :determine_mime_type, analyzer: :marcel

	Shrine.plugin :derivatives
	Shrine.plugin :backgrounding
	Shrine.plugin :signature
	Shrine.plugin :default_url
	Shrine.logger = Rails.logger

	# The remote_url plugin allows you to attach files from a remote location. i.e. pull image from replicate to my s3
	Shrine.plugin :remote_url, max_size: 2010241024

	# Shrine.plugin :url_options, store: { host: "https://cdn.commsie.com" }
	#https://github.com/shrinerb/shrine/wiki/Adding-Direct-S3-Uploads
	Shrine.plugin :presign_endpoint, presign_options: -> (request) {
	# Uppy will send the "filename" and "type" query parameters
	filename = request.params["filename"]
	type = request.params["type"]

	{
	content_disposition: ContentDisposition.inline(filename), # set download filename
	content_type: type, # set content type (required if using DigitalOcean Spaces)
	content_length_range: 0..(100 * 1024 * 1024), # limit upload size to 10 MB
	}
	}

view raw shrine.rb hosted with ❤ by GitHub

Show hidden characters

	import {AwsS3} from "uppy";
	import Compressor from '@uppy/compressor'

	import Uppy from '@uppy/core'
	import Dashboard from '@uppy/dashboard'


	document.addEventListener('turbo:load', () => {
	document.querySelectorAll('[data-uppy-single-media]').forEach(element => setupUppySingleMedia(element))
	})



	//only one image - assumes going to POST /media
	//set up uppy for images i.e. allowedFileTypes: ['image/*']
	function setupUppySingleMedia(element) {
	console.log("setupUppySingleMedia")
	let trigger = element.querySelector('[data-behavior="uppy-single-media-trigger"]')
	//let form = element.closest('form')
	let field_name = element.dataset.uppy
	const hiddenInput = document.querySelector('.upload-data')

	trigger.addEventListener("click", (event) => event.preventDefault())

	let uppy = new Uppy({
	debug: true,
	autoProceed: true,
	allowMultipleUploads: false,
	logger: Uppy.debugLogger,
	allowMultipleUploadBatches: false,
	restrictions: {
	maxNumberOfFiles: 1,
	minNumberOfFiles: 1,
	allowedFileTypes: ['image/*'],
	maxFileSize: 5 * 1024 * 1024
	}
	})

	uppy.use(Dashboard, {
	trigger: trigger,
	closeAfterFinish: true,
	note: 'Upload Images only'
	})

	uppy.use(AwsS3, {
	companionUrl: '/', // will call the presign endpoint on `/s3/params`
	})
	uppy.use(Compressor)
	uppy.on('complete', (result) => {
	// Rails.ajax
	// or show a preview:
	// element.querySelectorAll('[data-pending-upload]').forEach(element => element.parentNode.removeChild(element))

	result.successful.forEach(file => {
	appendUploadedFile(element, file, field_name)

	setPreview(element, file)

	const uploadedFileData = {
	id: file.meta['key'].match(/^cache\/(.+)/)[1], // object key without prefix
	storage: 'cache',
	metadata: {
	size: file.size,
	filename: file.name,
	mime_type: file.type,
	}
	}

	// set hidden field value to the uploaded file data so that it's submitted
	// with the form as the attachment
	// force submit the form to create our 'Media' object
	hiddenInput.value = JSON.stringify(uploadedFileData)
	document.getElementById('theForm').submit();

	})

	})

	console.log("setupUppySingleMedia:finish")
	}

	}

view raw uppy.js hosted with ❤ by GitHub

Lessons learnt

Ive used direct file uploads with Digital Ocean Spaces successfully

I explored Cloudflare R2 but they DONT support HTTP POST which performs uploads via native HTML forms (which I believe Shrine requires)

Felix’s Substack

Discussion about this post