FUSE is an interface for managing anything that resembles a file/folder as a native file/folder on Linux. In practice, this lets you manage almost all your cloud storage like Google Drive, Amazon S3, Google cloud storage, SSH servers, and local files like zip archives and ISO images, as normal folders on your PC. You can use them from any file manager, command line, or your code as if they were normal system folders.
The technical term for connecting a local folder to a folder like destination (cloud drives or archives or anything else) is “mounting”.
The goal of this blog is to get you acquainted with the FUSE interface and several open-source implementations of the FUSE interface, those lets you mount all previously discussed subjects. The second part of this blog is a step-by-step explanation of how to mount an Amazon S3 bucket on Amazon Linux (EC-2 or LightSail instances) and the obvious differences the mounted volume has in comparison with normal folders.
Part I: Linux FUSE (Filesystem in Userspace) interface
Technically FUSE (abbreviation for Filesystem in Userspace) is an interface for programs to mount virtual filesystems on Linux as folders. The mounted filesystems behave identically to normal files/folders except that they are fully managed by the managing application virtually. This lets us use almost anything that seems like a folder in Linux. For example, ISO files, Archives, Cloud drives, Amazon S3, Google cloud storage, etc. can all be used as a normal folder. The mounted files and folders can be accessed from any file browsers and programs that access the file system including the command line. Think about seeing your Google drive on your personal computer as a folder with full read-write support from your favorite file manager! It is what FUSE API can do for you.
Seems interesting? It is just the beginning of the story. We have got the implementation of FUSE API as nice little executable binaries for almost all requirements that we use to come across in our daily life as developers. To list a few,
Mount ISO files: https://help.ubuntu.com/community/FuseIso
Mounting Archives: https://www.Linux.com/news/mounting-archives-fuse-and-archivemount/
Google cloud storage bucket: https://cloud.google.com/storage/docs/gcs-fuse
Amazon S3 buckets: https://github.com/s3fs-fuse/s3fs-fuse
Mount SSH servers file system: https://github.com/libfuse/sshfs
I strongly suggest you check through them once you have concluded reading this blog.
The most interesting part of FUSE is that we seldom need to code ourselves to use them.
In case your requirement is too specific to have it pre-built, you can code it yourself. FUSE interface is super easy to implement, even though it is a kernel-level library. We just need to implement all required file/folder access functions and pass that to the fuse_main function. The library will do everything else for us. Here is an example code (official libfuse library’s example: https://github.com/libfuse/libfuse/blob/master/example/hello.c).
If C language seems difficult to comprehend, we have alternative higher-level implementations (unofficial) of the library in almost every popular programming language. Here are few examples:
Part II: Mounting Amazon S3 as a folder on Amazon Linux
AWS already calculated the scenario of all dealing with such a requirement one day 😉. That is why they have implemented FUSE as s3fs-fuse for us. The steps are simple, just follow me
1. Install & enable EPEL
sudo amazon-Linux-extras install epel
2. Install s3fs-fuse
sudo yum install s3fs-fuse -y
3. Create a file anywhere you want & write your access key ID and secret access key into it (in the format: ACCESS_KEY_ID:SECRET_ACCESS_KEY). (You can find the keys from https://console.aws.amazon.com/iam/home#/security_credentials page)
Optionally you can run this command to get the same effect (replace ACCESS_KEY_ID & SECRET_ACCESS_KEY with your own security credentials).
echo "ACCESS_KEY_ID:SECRET_ACCESS_KEY" > /any/path/you/need/credentials.txt
4. Create a folder to which the bucket should be mounted (anywhere you want).
5. You are done. Run the following command to mount your S3 bucket to the newly created folder. (replace BUCKET_NAME with the name of your bucket)
s3fs BUCKET_NAME /new/folder/path -o passwd_file=/path/to/credentials.txt
In case you encounter some errors, or if the mounting didn’t work, add a debug logging parameter to see what actually went wrong behind the scenes. Postfix the above command with -o dbglevel=info -f -o curldbg
i.e. the command with debug logging enabled will be
s3fs BUCKET_NAME /new/folder/path -o passwd_file=/path/to/credentials.txt -o dbglevel=info -f -o curldbg
Detailed installation steps on other Linux variants and platforms are explained in s3fs’s official GitHub repository README file https://github.com/s3fs-fuse/s3fs-fuse
Even though the mounted directory will work exactly like a folder, there are a few notable differences.
1. The latency of reading, writing & listing files is higher in comparison with normal files and folders on disk.
2. Multiple clients can mount the same bucket on different machines at a time. But modifying the same file/folder (renaming, re-writing, deleting, etc.) from multiple machines concurrently might induce undefined behaviors.