File Controller and Finishing a simple File Storage Service Using VueJS, Flask, and RethinkDB
Aug 9, 2018 • 16 Minute Read
Getting Started
For more information about how to set up your workspace, checkout the first guide in this series: Introduction and Setup to Building a Simple File Storage Service Using VueJS, Flask and RethinkDB.
File Controller
The File controllers will be used for working with both Files and Folders, hence, there will be slightly more logic here than in our previous controller. We start by creating a boilerplate for our controller in /api/controllers/files.py module.
import os
from flask import request, g
from flask_restful import reqparse, abort, Resource
from werkzeug import secure_filename
from api.models import File
BASE_DIR = os.path.abspath(
os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
)
class CreateList(Resource):
def get(self, user_id):
pass
def post(self, user_id):
pass
class ViewEditDelete(Resource):
def get(self, user_id, file_id):
pass
def put(self, user_id, file_id):
pass
def delete(self, user_id, file_id):
pass
The CreateList class, as the name implies, will be used for creating and listing files for a logged in user. The ViewEditDelete class, also as the name implies, will be used for viewing, editing, and deleting files. The methods we're using in the classes correspond with the appropriate HTTP actions.
Decorators
We will start the implementation by creating a bunch of decorators which we will be using on the methods in our Resource classes. You'll want to separate this out into the /api/utils/decorators.py module.
from jose import jwt
from jose.exceptions import JWTError
from functools import wraps
from flask import current_app, request, g
from flask_restful import abort
from api.models import User, File
def login_required(f):
'''
This decorator checks the header to ensure a valid token is set
'''
@wraps(f)
def func(*args, **kwargs):
try:
if 'authorization' not in request.headers:
abort(404, message="You need to be logged in to access this resource")
token = request.headers.get('authorization')
payload = jwt.decode(token, current_app.config['SECRET_KEY'], algorithms=['HS256'])
user_id = payload['id']
g.user = User.find(user_id)
if g.user is None:
abort(404, message="The user id is invalid")
return f(*args, **kwargs)
except JWTError as e:
abort(400, message="There was a problem while trying to parse your token -> {}".format(e.message))
return func
def validate_user(f):
'''
This decorate ensures that the user logged in is the actually the same user we're operating on
'''
@wraps(f)
def func(*args, **kwargs):
user_id = kwargs.get('user_id')
if user_id != g.user['id']:
abort(404, message="You do not have permission to the resource you are trying to access")
return f(*args, **kwargs)
return func
def belongs_to_user(f):
'''
This decorator ensures that the file we're trying to access actually belongs to us
'''
@wraps(f)
def func(*args, **kwargs):
file_id = kwargs.get('file_id')
user_id = kwargs.get('user_id')
file = File.find(file_id, True)
if not file or file['creator'] != user_id:
abort(404, message="The file you are trying to access was not found")
g.file = file
return f(*args, **kwargs)
return func
The login_required decorator is used to validate that users are actually logged in before accessing the method's functionality. We use this decorator to protect certain endpoints by decoding the token to ensure its validity. We get the id field stored in the token and try to retrieve the corresponding user object. Then, also store this object in g.user for access within the method definition.
Similarly, we create the validate_user decorator which ensures that no other logged in user can access URL patterns labelled with another user's ID. This validation is purely based on the information in the URL.
Finally, the belongs_to_user decorator ensures that only the user who created a file can access it. This decorator actually checks the creator field in the file document against the user_id supplied.
Here are the views for creation of new files and listing of files:
class CreateList(Resource):
@login_required
@validate_user
@marshal_with(file_array_serializer)
def get(self, user_id):
try:
return File.filter({'creator': user_id, 'parent_id': '0'})
except Exception as e:
abort(500, message="There was an error while trying to get your files --> {}".format(e.message))
@login_required
@validate_user
@marshal_with(file_serializer)
def post(self, user_id):
try:
parser = reqparse.RequestParser()
parser.add_argument('name', type=str, help="This should be the folder name if creating a folder")
parser.add_argument('parent_id', type=str, help='This should be the parent folder id')
parser.add_argument('is_folder', type=bool, help="This indicates whether you are trying to create a folder or not")
args = parser.parse_args()
name = args.get('name', None)
parent_id = args.get('parent_id', None)
is_folder = args.get('is_folder', False)
parent = None
# Are we adding this to a parent folder?
if parent_id is not None:
parent = File.find(parent_id)
if parent is None:
raise Exception("This folder does not exist")
if not parent['is_folder']:
raise Exception("Select a valid folder to upload to")
# Are we creating a folder?
if is_folder:
if name is None:
raise Exception("You need to specify a name for this folder")
return Folder.create(
name=name,
parent=parent,
is_folder=is_folder,
creator=user_id
)
else:
files = request.files['file']
if files and is_allowed(files.filename):
_dir = os.path.join(BASE_DIR, 'upload/{}/'.format(user_id))
if not os.path.isdir(_dir):
os.mkdir(_dir)
filename = secure_filename(files.filename)
to_path = os.path.join(_dir, filename)
files.save(to_path)
fileuri = os.path.join('upload/{}/'.format(user_id), filename)
filesize = os.path.getsize(to_path)
return File.create(
name=filename,
uri=fileuri,
size=filesize,
parent=parent,
creator=user_id
)
raise Exception("You did not supply a valid file in your request")
except Exception as e:
abort(500, message="There was an error while processing your request --> {}".format(e.message))
The listing method is pretty straightforward. We filter the table for all files that were created by a certain user and stored in the root directory. We return this data for this endpoint and throw an exception if there are any errors.
Create
For the create action, it's a bit more involved. For this guide, we're assuming that files and folders will be created with the same endpoint. For files, we will need to supply the file as well as a parent_id, if we're uploading it in a folder. For folders, we will need a name and a parent_id value, again if we are creating this within another folder. For folders, we also need to send an is_folder field with our request to specify that we are creating a folder.
If we are going to store this within a folder, we have to ensure that the folder exists and is a valid folder. We also ensure that we are supplying a name field if we are creating a folder.
For file creation, we upload the file into a folder specifically named for the different users as mentioned before. In our case, we are using the pattern /upload/<user_id> for the different user file directories. We have also used the file information to populate the document we're going to be storing in the table.
We conclude by calling the methods for file and folder creation - File.create() and Folder.create() respectively.
Serializers
Notice that we have used the marshal_with decorator available with Flask-RESTful. This decorator is used to format the response object and to indicate the different field names and types that we'll be returning. See the definition of the file_array_serializer and file_serializer below:
file_array_serializer = {
'id': fields.String,
'name': fields.String,
'size': fields.Integer,
'uri': fields.String,
'is_folder': fields.Boolean,
'parent_id': fields.String,
'creator': fields.String,
'date_created': fields.DateTime(dt_format= 'rfc822'),
'date_modified': fields.DateTime(dt_format='rfc822'),
}
file_serializer = {
'id': fields.String,
'name': fields.String,
'size': fields.Integer,
'uri': fields.String,
'is_folder': fields.Boolean,
'objects': fields.Nested(file_array_serializer, default=[]),
'parent_id': fields.String,
'creator': fields.String,
'date_created': fields.DateTime(dt_format='rfc822'),
'date_modified': fields.DateTime(dt_format='rfc822'),
}
This can be added at the top of the /api/controllers/files.py module or in a separate /api/utils/serializers.py module.
The difference between both serializers is that the file serializer includes the objects array in the response. We use the file_array_serializer for list responses while we use the file_serializer for object responses.
We have also made use of a function called is_allowed() to help ensure that we support all the files that we are uploading. We created a list called ALLOWED_EXTENSIONS to contain the list of all the allowed extensions.
ALLOWED_EXTENSIONS = set(['txt', 'pdf', 'png', 'jpg', 'jpeg', 'gif'])
def is_allowed(filename):
return '.' in filename and \
filename.rsplit('.', 1)[1] in ALLOWED_EXTENSIONS
Finally, we conclude by adding in the resource class for ViewEditDelete in the /api/controllers/files.py module.
class ViewEditDelete(Resource):
@login_required
@validate_user
@belongs_to_user
@marshal_with(file_serializer)
def get(self, user_id, file_id):
try:
should_download = request.args.get('download', False)
if should_download == 'true':
parts = os.path.split(g.file['uri'])
return send_from_directory(directory=parts[0], filename=parts[1])
return g.file
except Exception as e:
abort(500, message="There was an while processing your request --> {}".format(e.message))
@login_required
@validate_user
@belongs_to_user
@marshal_with(file_serializer)
def put(self, user_id, file_id):
try:
update_fields = {}
parser = reqparse.RequestParser()
parser.add_argument('name', type=str, help="New name for the file/folder")
parser.add_argument('parent_id', type=str, help="New parent folder for the file/folder")
args = parser.parse_args()
name = args.get('name', None)
parent_id = args.get('parent_id', None)
if name is not None:
update_fields['name'] = name
if parent_id is not None and g.file['parent_id'] != parent_id:
if parent_id != '0'
folder_access = Folder.filter({'id': parent_id, 'creator': user_id})
if not folder_access:
abort(404, message="You don't have access to the folder you're trying to move this object to")
if g.file['is_folder']:
update_fields['tag'] = g.file['id'] if parent_id == '0' else '{}#{}'.format(folder_access['tag'], folder['last_index'])
Folder.move(g.file, folder_access)
else:
File.move(g.file, folder_access)
update_fields['parent_id'] = parent_id
if g.file['is_folder']:
Folder.update(file_id, update_fields)
else:
File.update(file_id, update_fields)
return File.find(file_id)
except Exception as e:
abort(500, message="There was an while processing your request --> {}".format(e.message))
@login_required
@validate_user
@belongs_to_user
def delete(self, user_id, file_id):
try:
hard_delete = request.args.get('hard_delete', False)
if not g.file['is_folder']:
if hard_delete == 'true':
os.remove(g.file['uri'])
File.delete(file_id)
else:
File.update(file_id, {'status': False})
else:
if hard_delete == 'true':
folders = Folder.filter(lambda folder: folder['tag'].startswith(g.file['tag']))
for folder in folders:
files = File.filter({'parent_id': folder['id'], 'is_folder': False })
File.delete_where({'parent_id': folder['id'], 'is_folder': False })
for f in files:
os.remove(f['uri'])
else:
File.update(file_id, {'status': False})
File.update_where({'parent_id': file_id}, {'status': False})
return "File has been deleted successfully", 204
except:
abort(500, message="There was an error while processing your request --> {}".format(e.message))
We created a get() method which returns a single file or folder object based on the ID. For folders, it includes listing information. You can see how this is done if you look at the belongs_to_user decorator. For files, we have included a query parameter should_download to be set to true if we want to download a file.
The put() method takes care of updating file and folder information. This also includes moving files and folders. Moving files is triggered by updating the parent_id field for a file/folder. The logic for both have been covered in the move() methods for the file and folder models.
The delete() method also comes a query parameter which specifies whether or not we want to perform a hard delete. For hard delete, records are removed from the database and the files are deleted from the file system. For soft delete, we only update the file status field to false.
We have created new methods here called update_where() and delete_where() in the RethinkDBModel class for deleting and updating a filtered set from the table:
@classmethod
def update_where(cls, predicate, fields):
status = r.table(cls._table).filter(predicate).update(fields).run(conn)
if status['errors']:
raise DatabaseProcessError("Could not complete the update action")
return True
@classmethod
def delete_where(cls, predicate):
status = r.table(cls._table).filter(predicate).delete().run(conn)
if status['errors']:
raise DatabaseProcessError("Could not complete the delete action")
return True
Conclusion
And that's it! We're done with our file storage API. Run the API to see it in action.
You can check out the codebase for the project here. Show some love by giving this guide a thumbs up at the top right.