0bs.tech

Reader

Read the latest posts from 0bs.tech.

from tim

Last year I founded chaos.cloud UG (haftungsbeschränkt*) together with Timo. Timo is known for building Dropshare, a macOS and iOS native file sharing app. Dropshare allows users to upload files by dragging and dropping to the menubar as well as screenshots, screenrecordings and capturing Android and iOS screenshots via USB. Dropshare Cloud is one of the supported storage options, built and operated by us. We try to offer the best user experience as well as advanced features like statistics, auto expiry for files, custom domain as well as SSL/TLS for everyone.


Storage

AWS S3 was the obvious option for storing files. It's highly reliable, reasonable priced and equipped with awesome APIs. We started out under the assumption that we could create a bucket per user and have S3 deliver the files. This quickly turned out to be a wrong assumption as creating more than 100 buckets is not easily possible. Luckily there is a workable solution: Creating temporary credentials via AWS Security Token Service (STS). The service allows creating temporary, limited-privilege login credentials for AWS services. It's configurable via the normal AWS IAM policies, so it's possible to create credentials that are only allowed to access parts of bucket by restricting the access rights to a prefix of the bucket. We operate a webservice that allows the app to request access key and secret for a users path on S3 via HTTPS. The credentials have a limited time of validity and are restricted to only operate on the users prefix on S3.

Delivery and statistics

Storing files is done, next part is delivering them for people to see. We proxy all files for several reasons: – We need to control the URL for the looks – We want to offer SSL – We want to offer custom domains (with free SSL certificates) – We need quick and granular billing To proxy the files we operate caching proxies. They are based on a custom stack built around OpenResty and Varnish, as well as custom statistics collection software.

OpenResty

We use OpenResty for flexible proxying as well as dynamic serving of SSL certificates. Requests for our shared domain are more or less passed to varnish and S3 directly, for custom domains we resolve the domain to the users prefix on S3 and then hand off to varnish. SSL is included in all our plans, the “Power” level also offers free SSL certificates on a users custom domain. To offer this service we leverage LetsEncrypt and dynamically request certificates for custom domains. OpenResty offers an ingenious way of loading SSL certificates dynamically during the SNI handshake which we use to fetch them from a local Redis readslave.

Varnish

Once we started serving files our own it was a wise choice to implement some form of caching. Varnish also renders custom error pages on S3 and has a little VCL logic to determine acceptable caching times. To be fully encrypted we take advantage of Stunnel, which we use to build a transparent TLS tunnel to S3, which Varnish itself doesn't support.

Statistics

Our plans are limited on storage and transfer. We retrieve storage stats from S3, traffic stats are generated on our side. A worker is constantly following the Varnish NCSA log, parsing it for the bytes transferred in the client body. This information is put in a few different sorted sets in Redis for tracking the traffic of every file of a a user, also broken down per month. Those stats are interpreted by our main Ruby on Rails app in workers that check the usage stats regularly, disabling file delivery if limits are overstepped.

File list

We decided we need to show a list of all files that have been uploaded for a user. The first naive implementation of this feature was just doing the S3 list call for the users prefix as the file browser was opened. This proved unbearably slow during the first extended tests with real amounts of uploads. Adding caching helped the situation for a while but meant we ran into all sorts of issues (users deleting a file or changing something meant our web UI was way off most of the time).

To solve this once and for all we resorted to AWS Lambda processing S3 triggers. A simple lambda function is called on each add/delete to S3. This function immediately messages us with the details of the change. We then update our internal files database with all information that is needed for displaying the file list. Switching from our cached list to this allowed file uploads to show almost immediately while reducing the complexity of our logic immensely. We currently have a worker running every night to make absolutely sure each and every file is found in our database, it yet has to find a difference.

This feature proved well worth the effort while implementing automatic expiry, as expired files are instantly reflected in the UI. Automatic expiry

To deal with the storage usage we offer automatic expiry of files. A user can select files based on extension or them being screenshots for setting an expiry on predetermined levels between one day and year. The selection is transferred to the app which sets S3 tags on the files. Using S3's auto expiry feature we set up rules that expire those files the set amount of time after the upload. This relatively new options frees us from iterating over all files and implementing such a logic ourselves, reducing the expenses for all S3 calls and worker capacity on our servers.

Payment

Finding the right payment solution was a more complex task than imagined. We started under the assumption that just going with Stripe would do the trick. We then decided that the support of PayPal would be desirable (we are in Germany, sadly having a credit card is far from normal here). After a quick trip to BrainTree land we switched to Paddle. They offer handling all the VAT stuff, processing PayPal and credit cards - so rewrite number three of the payment system is what happened, which since worked great. Paddle is less well supported than Stripe or Braintree, but implementing a few webhooks and a few models didn't take too long at all.


Be sure to check us out on Stackshare, try Dropshare Cloud and ask away in the comment section below. Also make sure to subscribe to Timo and me for more details about building our own product. *yep, it's the law to put that there…thanks Merkel

 
Read more...

from tim

Together with a friend I recently built Dropshare Cloud. We offer online storage for the file and screenshot sharing app Dropshare for macOS/iOS. After trying out Django for getting started (we both had some experience using Django) I decided to rewrite the codebase in Rails. My past experience developing in Rails made the process quick — and boring.

What's boring about Rails?

It's hard to find something that has not been done with Rails before. Awesome user management? Devise. Asynchronous job processing? Sidekiq. Creating certificates via Let's Encrypt? acme-client. This makes implementing services almost feel like playing with lego. Take this brick, grab that one, put on another one — done. Stumbling upon strange behaviour almost never happens. The enormous user base, from startup to Fortune 500 company mean it's stable and well tested. Finding solutions for almost all exceptions that can happen is as simple as copying the error message to Google and reading the first 3 results, tops.

Why this matters

Building a product is not about using the latest and greatest technology. I really love Elixir and Phoenix for example, but we still chose to implement Dropshare Cloud in Ruby on Rails. The maturity of the framework and widespread use outweigh the great performance of Phoenix/Elixir for us. The challenge when creating a product is not achieving sub-ms response times, it's about getting stuff shipped. Software architecture, awesome test coverage and great performance are all nice. They are also worthless if your idea doesn't stick.

Reducing the cycle time is king. As a small company we can only win in the market if we move faster than the competition. Implementing ideas like free SSL certificates in just 2 evenings is just what sets a small company apart from bigger competitors. We don't have a sprint planning, requirements engineering or UX reviews (we really should have those, though :D) — we decide to build something and just do it. Having a reliable framework back all those ideas and make it easy is what is key.

A word on performance

Benchmarking one server serving our Rails front page leads to ~50 RPS. “That’s so slow” I hear you say. And yes, it is. But: No one cares. If we had 50 requests per second for any time other than me testing we would spend enough money on this issue to just make it go away (scaling out is possible for us in the current situation). 50 requests/second means 3,000 requests/minute, 180,000 requests/hour or ~4.3 million hits per day. Even with mediocre conversion rates a few of those people would hopefully subscribe to our service and start paying money that could immediately be thrown at more hardware or motivate us to implement a caching strategy, rewrite parts of the service or maybe just optimize the database queries.

 
Read more...

from tim

Together with a friend I recently built Dropshare Cloud. We offer online storage for the file and screenshot sharing app Dropshare for macOS/iOS. After trying out Django for getting started (we both had some experience using Django) I decided to rewrite the codebase in Rails. My past experience developing in Rails made the process quick — and boring.

What's boring about Rails?

It's hard to find something that has not been done with Rails before. Awesome user management? Devise. Asynchronous job processing? Sidekiq. Creating certificates via Let's Encrypt? acme-client. This makes implementing services almost feel like playing with lego. Take this brick, grab that one, put on another one — done. Stumbling upon strange behaviour almost never happens. The enormous user base, from startup to Fortune 500 company mean it's stable and well tested. Finding solutions for almost all exceptions that can happen is as simple as copying the error message to Google and reading the first 3 results, tops.

Why this matters

Building a product is not about using the latest and greatest technology. I really love Elixir and Phoenix for example, but we still chose to implement Dropshare Cloud in Ruby on Rails. The maturity of the framework and widespread use outweigh the great performance of Phoenix/Elixir for us. The challenge when creating a product is not achieving sub-ms response times, it's about getting stuff shipped. Software architecture, awesome test coverage and great performance are all nice. They are also worthless if your idea doesn't stick.

Reducing the cycle time is king. As a small company we can only win in the market if we move faster than the competition. Implementing ideas like free SSL certificates in just 2 evenings is just what sets a small company apart from bigger competitors. We don't have a sprint planning, requirements engineering or UX reviews (we really should have those, though :D) — we decide to build something and just do it. Having a reliable framework back all those ideas and make it easy is what is key.

A word on performance

Benchmarking one server serving our Rails front page leads to ~50 RPS. “That’s so slow” I hear you say. And yes, it is. But: No one cares. If we had 50 requests per second for any time other than me testing we would spend enough money on this issue to just make it go away (scaling out is possible for us in the current situation). 50 requests/second means 3,000 requests/minute, 180,000 requests/hour or ~4.3 million hits per day. Even with mediocre conversion rates a few of those people would hopefully subscribe to our service and start paying money that could immediately be thrown at more hardware or motivate us to implement a caching strategy, rewrite parts of the service or maybe just optimize the database queries.

 
Read more...

from tim

When leaving my last job my ex-colleagues pooled some money and suprised me with a Pine64 Clusterboard. It's a 7-socket mainboard for Pine64 SOPINE A64 SODIMM-DDR3 formfactor modules which 2GB RAM, 4 Core 64bit Cortex A3 cores.

This board is the optimal solution for a low-power multi-machine cluster. It packs a built-in 1 GbE switch, power distribution, USB & GPIO breakout for all boards and provides eMMC passthrough for one.

So far this sounds like a great solution, sadly it has at least two flaws: – reboot of the boards doesn't work – software support by Pine64 is almost non-existant

Rebooting

The boards show a reproducible behavior of not properly resetting after a reboot-command has been issued. The serial interface just turns blank when a reboot is issued and the CPU halts, but no reset ever occurs. There are just two ways to resurrect the module: Pull the RESET pin to GND or remove and reinsert power to the whole cluster.

After some investigation, reading forums and code of the early stage bootloader process, I actually started compiling my own bootloaders from U-Boot and the arm trusted firmware and hacking in an early reset. This showed that the board would not reset at all and it's almost certainly a hardware issue. This was confirmed by a friend who owns the sopine64 baseboard, which is basically the single-module holder for the sopine64.

I found a hacky solution to this issue that is sufficient for now by just connecting the whole clusterboard to a ZigBee controlled socket and therefore allowing remote reset from my smart home controller. In the long-term I still want to actually figure out what keeps the PMIC (power management IC) from resetting the CPU (or the CPU from completing a reset).

Software Support

Pine64 seems very dedicated to offering affordable hardware at awesome prices – something at which they clearly succeed. What lacks is great mainline software support for the boards though. This is very nicely covered by armbian – but they receive no funds or the like for supporting those commercial products. This leads to unnecessary situations like needing to patch the device tree to support the way Ethernet is implemented on the clusterboard.

My Setup

When writing this post I started writing down a list of all the things that I run on my clusterboard right now, but as this list grew quite a bit this will be a post for another time.

 
Read more...