Openstack Swift Image Compression Policy
Openstack Swift Image Compression Policy
The new policy for Openstack Swift
In this project, OpenStack Swift is used to providing web developers with a custom policy to automatically compress resources such as images. Recently, web developers are using resources that are cheaper and unlimited traffic with cloud storage service like Amazon AWS S3, which enables developers such as image servers to automatically compress resources and fix the speed of web pages thus we can increase the performance of web page.
October 1, 2016
Software-defined Network, Open Stack, Python
San Jose State University
Share projectOpen Website
openStack-swift-image-compressionThe new policy for Openstack Swift In this project, OpenStack Swift is used to providing web developers with a custom policy to automatically compress resources such as images. Recently, web developers are using resources that are cheaper and unlimited traffic with cloud storage service like Amazon AWS S3, which enables developers such as image servers to automatically compress resources and fix the speed of web pages thus we can increase the performance of web page.
- Ikwhan Chang ([email protected])
- Prashanth Iyengar ([email protected])
- Priya Vishnu ([email protected])
Description of the Project
- Analyzing the source code of the complicated OpenStack Swift, how to modify the code, how to modify the individual storage policy, and what factors See if the developer can set the desired policy.
- Identify the difference from the existing policy by putting the policy on the specific storage of the actual OpenStack Swift.
- When the user uploads the file, if the file is an image file, develop the logic to save the compressed image. In our project, we can create and use a compressed file in the same format as [File Name]_compressed.[Extension] according to one upload request of user or developer.
Goals of the Project
- Analyze the Swift Code as our first step. The OpenStack Swift source code is made up of a huge amount of Python sources. The file diskfile.py is composed of more than 3000 lines of source code and more than 50 methods. So understanding Swift is essential.
- Understood the swift proxy code flow. Once you have completed the primary analysis, you will understand what each method does for Swift, which consists of the actual proxy server and the object server.
- Identify the files that we need to modify to implement an additional storage policy. Analyze the source to see where you can link the policy to the real world.
- Identified the most suitable image compression technique. PIL (Python Image Library), Open CV, etc. to see what image compression algorithm can be applied.
- Modified the Swift code to add the new image compression policy and integrated the compression procedure
- Developed a web page to demonstrate the usage of the new policy.
Project ArchitectureFor our implementation, we need to design and analyze the architecture as shown in Figure 2. First we need to understand how the OpenStack Swift works, and the protocol it provides. OpenStack Swift provides its own Web Server Gateway Interface (WSGI) via the REST API and its own Command Line Interface (CLI). In other words, you can connect via the web through your own WSGI. First, we decided to use the web interface for testing. Next, we analyzed the architecture of the entire OpenStack Swift and planned to make plan A and plan B, as shown in Figure 3, about which image compression should go. OpenStack Swift consists of proxy Server and object Server. In actual practice, we thought that the two servers would be separated, so we tried to perform image compression on the proxy server where files were originally delivered and collected. In other words, when an object is received from a proxy server, the file is compressed and replaced. However, plan A had two major problems. First, when a lot of objects are delivered, the proxy server itself is slowed down, which hinders overall performance. The proxy server aims to deliver the object correctly to the object server depends on what is preset in the ring file. By bypassing this sort of bypass function from proxy to object, and doing something to the proxy server itself, performance can also be affected on object servers in zones where real objects are not stored. Second, when the proxy server receives the first file, it will split the file into chunks in a predefined unit of 500 bytes and send it sequentially. In this process, there is a certain metadata that is created first, but the compressed file does not correctly recognize it. Even if replacement is correct, it is not downloaded if it is actually stored in the object server. So, we decided to start with object server in plan B shown in Figure 2. Therefore, we analyzed the other policies and confirmed that the implemented replication policies are implemented in the object server. That is, the most important file is diskfile.py file. In this file, there is a specific manager module as shown in Figure 3, and writer module and reader module that are re-inherited in base module and processed differently based on user’s request of REST API. So, we decided to create a manager in diskfile.py like the other policy, and create a writer to modify the file processing here. As with plan A, because the creation of the metadata failed, our solution is as follows:
- Intercept the file in the middle.
- Compress the intercepted file and save it in /var/tmp/.
- Upload the file saved in /var/tmp/ with _compressed after the file name using Swift CLI.
- As in (1), if the file name is “_compressed”, do not proceed with steps (2)-(4).