↓Skip to main content

Using Cloudflare for Conveyor App Download Site

·9 mins
Software App Packaging App Publishing Tutorial

Introduction #

I recently created a JVM based application packaged with the fantastic Conveyor software and then I got to the point of making it available to the world. In this post I want to describe how I published the download site for my application using Conveyor, CloudFlare CDN and CloudFlare R2 storage. Whilst I will describe the steps I took, due to interfaces changing over time I will give general instructions on what to do rather than giving specific instructions on what buttons to press and such like. You probably will find it useful to read CloudFlare documentation alongside this post.

GitHub Releases was not the answer #

As my application is open-source and on GitHub and Conveyor supports GitHub Releases, it seemed logical to publish my application through GitHub Releases. Conveyor has documentation on publishing to GitHub and after following the instructions and issuing the copied-site command I had a new release on GitHub. It all looked right, that was until I tried downloading the Windows installer executable. Unfortunately running the installer it would start downloading the rest of my application, but it would get stuck and then error. After asking a question, it seems like GitHub does not always work well with these installers and the recommendation was to find another solution.

Choosing CloudFlare #

Conveyor also supports publishing to S3 compatible object storage. I havve a number of my domains with CloudFlare and I noticed their R2 storage is S3 compatible. CloudFlare’s free offering should provide sufficient resources for my project’s download site, but it is critical to correctly set up the CDN caching to reduce the operations on the R2 storage so you do not run the risk of getting billed for going over the free allowance.

Have domain for download site with CloudFlare DNS #

As mentioned it is important to have a CDN correctly configured to avoid excessive operations on the R2 storage. CloudFlare have such a CDN and using this makes the set up much easier. So if your domain or subdomain is not added to your CloudFlare account do this from the CloudFlare dashboard. My experience is that CloudFlare do a good job in guiding you through the process, giving you instructions on setting DNS records with your domain’s registrar should you not be using CloudFlare registrar. Once successfully added to your account you should find your domain listed on your account dashboard.

Activate CloudFlare R2 and connect with your domain #

Time to create the R2 storage bucket for the download site and connect it with your domain. Slightly unintuitively in my view, you start the process by selecting your domain in your account dashboard, then under zone rules you select zone rules cloud connector. Once you get to this point CloudFlare leads you through the process. So go ahead and create a new bucket, here are the settings I used:

  • Bucket name: It can be anything but using the application’s name followed by downloads will help me remember what it is for in the future.
  • Location: Automatic if fine but if you have requirements to restrict where it is located you can select something else.
  • Storage type: Standard storage is probably the most appropriate option here.
  • R2.dev subdomain access: Do not set this to public, CloudFlare says it isn’t suitable for production use due to limitations and you will access it through your own domain anyway.

Once you have pressed the button to create the bucket, you will then need to link your domain to the bucket. Select the bucket and go to settings. In there you will find an option to connect a custom domain. I chose to host the website separate from the download site, so I used a download subdomain, so for me I gave something like download.myapp.com as the domain. You can use a top level domain if you prefer and plan to store the rest of your website in the same storage bucket. I left the option for minimum TLS at the default level and pressed the button to connect the domain.

Create API access token #

Whilst we are in the R2 management part of the CloudFlare interface, we should create an API token which will allow Conveyor to automatically upload our site. So from the R2 overview page, open the API menu and select the option to manage API tokens. Now press the button to create an API token. Give your token a name and assign the object read and write permission. There are two more options relating to how long before the token expires and if it should be restricted to certain IP addresses, either accept the defaults or if you prefer to be more restrictive in the name of security go ahead and change these. When you press the button to create the token, you then will be presented with a page of details, it is important to take note of the information shown here as you will need it later for the conveyor configuration and you will not be able to get it again from CloudFlare. Make a note of the access key ID and secret access key. Also you can get the S3 endpoint which you need for conveyor, if you selected auto for location use the default endpoint otherwise use the endpoint relating to the juristiction you specified for your storage bucket. Once you have noted these details you can press the finish button.

Configuring CloudFlare CDN #

This possibly was the most complicated step and it took me a few tries to get it correct. Read the Conveyor documentation on configuring a CDN. In particular pay attention to the discussion of stable and volatile files. There was one extra thing which caught me out, when creating the caching rules make sure you do not cache origin errors, specifically 404 errors as this may make your site not work correctly.

To start setting up the CDN caching, go to your CloudFlare account dashboard home, select your domain and then select zone caching and zone caching rules.

Handling Stable files #

The stable files are those which will not be changed over time, therefore we can cache these for a long time and reduce the number of requests which hit the R2 storage. Remember when configuring the rule for stable files to ensure 404 errors, may be other errors, should not be cached. Create a new caching rule, here is my example set up, you will need to change the download.myapp.com hostname to match your domain you configured for the download site.

  • Name: Give it something that will help you remember the purpose of the rule.
  • Set rule to match a custom filter expression.
  • Set the expression to something like, modifying to match your domain: 
    (http.host eq "download.myapp.com" and (ends_with(http.request.uri.path, ".tar.gz") or ends_with(http.request.uri.path, ".zip") or ends_with(http.request.uri.path, ".msix") or ends_with(http.request.uri.path, ".deb") or ends_with(http.request.uri.path, ".delta")))
  • Set it to be eligible for cache.
  • Set ignore cache-control header.
  • Set the TTL to a long time, I went for 1 year.
  • Under status code TTL, set 404 response to no-cache. You could set other error codes to no-cache as well. This is an important step, if you don’t then there is a chance you will cache a 404 error for a long time and you will need to manually purge the cache to get your installer to work.
  • You can leave the other options as default.

Now save the rule.

Handling volatile files #

Unfortunately the default caching policy of CloudFlare will catch some of the volatile files so you need to also configure these. Create a second caching rule with the following settings:

  • Name: Whatever will help you remember its purpose.
  • Set match to custom filter expression.
  • Set the expression to something like the following, modifying for your domain name: 
    (http.host eq "download.myapp.com" and (http.request.uri.path eq "/debian/InRelease" or http.request.uri.path eq "/debian/Release" or http.request.uri.path eq "/debian/Packages" or http.request.uri.path eq "/debian/Packages.xz" or http.request.uri.path wildcard "/debian/Contents-*64" or http.request.uri.path wildcard "/debian/Contents-*64.xz" or http.request.uri.path wildcard "/*.exe" or http.request.uri.path wildcard "/*.appinstaller" or http.request.uri.path wildcard "/appcast-*64.rss" or http.request.uri.path eq "/metadata.properties"))
  • Set to bypass cache.
  • Leave the rest of the settings as default.

Now save the rule.

Configure Conveyor to upload to your Site #

Now CloudFlare is all correctly set up, it is time to configure Conveyor. Mostly the documentation for object storage is clear. The only element which caught me out was you need to set the region to auto. Also be careful about the endpoint, it differs if you set automatic location or specified a juristiction. Your app.site section of your Conveyor configuration will look something like:

app {  
  site {
    base-url = "https://download.myapp.com"
    copy-to = "s3:my-bucket/path/to/site"

    s3 {
      // Your bucket region.
      region = "auto"

      // Optional: access credentials.      
      access-key-id = ...
      secret-access-key = ...

      // Optional: override endpoint if using S3 from a different provider:
      endpoint = "https://8b4b4b976e702665ce730a8efbaad994.r2.cloudflarestorage.com"

      // Optional, 3 is the default.
      retries = 3      
    }
  }
}

Now if everything is correctly configured you should be able to run conveyor make copied-site and your application should be built and uploaded to your download site. Once done you can find the automatically generated download page at https://download.myapp.com/download.html and you should go ahead and try downloading and installing your application to confirm you have everything set up correctly.

Confirm caching is working #

As I mentioned you really want that CDN caching the requests, if not it could lead to you going over your free allowance in R2. So once you have performed the install test and may be periodically after that, you probably will want to go back into Cloudflare and look at the zone analytics for your domain to see how many requests are being cached. It is hard to say exactly what numbers you should expect, but most of the requests should be cached.

Conclusion #

Now I have all of this set up and working, it seems to be working well, but it does feel like it has been quite a journey to get here. If you are using this post to help you set up your own download site for an application packaged with Conveyor, please read through each section carefully as there are some critical settings which if configured wrongly can break the whole set up. Should you have problems, double check you set everything correctly and if you still have problems feel free to contact me.

Author
Michael Whapples
Software developer, target shooter, puzzler