Blog

Android – Wait for two networks requests to finish with Kotlin

In Android, UI operations happen in the main thread while longer running operations such as network requests and database transactions happen in the background thread.

When building mobile apps against public APIs, there are times where you have to make a network request to multiple endpoints in a single screen, but only display the entire screen once all of the network requests have completed. As mentioned at the beginning of this post, network requests are done in a separate thread, which is usually managed by whatever networking libraries you are using (it’s probably Retrofit), RxJava, or if you’re doing it manually in an AsyncTask.

Threads are hard. I don’t like threads. I have a difficult time understanding it, working with it, and managing it. Slight mismanagement of it can lead to all kinds of issues like race condition where it can become annoying enough to debug where you’ll want to flip a table. As mentioned above, when you have to build an a screen in a mobile app where you have to make a network request to more than one endpoint at a time, and only commence an operation (like displaying data on a screen) once all of the network requests have finished, you’ll have to figure out how to listen for when different requests, essentially threads, have finished loading.

There are multiple ways of going about this in Android. Some people will say look into CountDownLatch while some people will say to learn RxJava. While both suggestions will work, CountDownLatch looks too complicated for me to bother understanding and RxJava has too high of a learning curve for it to be a viable option if you need to learn how to do this quickly. If you use Kotlin, you can also look into coroutines but that’s apparently an experimental feature according to JetBrains.

While trying to figure out how to do this I found a Kotlin library called Kovenant that makes asynchronous operations super easy on Kotlin. If you’re unfamiliar with promises and asynchronous calls, the TLDR is that you run some code asynchronously and promises allow you to commence the next sequence of operation once that initial asynchronous operation has finished running. Kovenant also makes it easy to combine multiple promises, which is what we’ll be using to wait until two network operations are finished before commencing our UI operations. Let’s look at some code sample from the redesign I’m working on for MaterialReads.

In this redesign, I need to have data from two different endpoints: updates and reviews. Unfortunately, GoodReads do not provide an endpoint that reads both datasets in one API call, thus I have to make two. I want to be able to display a loading spinner while data from both endpoints are being loaded and then hide the loading spinner only if the two network requests have finished running its full course. This would be difficult to do manually but Kovenant makes it easy. As displayed in the code sample, we split the both network requests into two separate methods, and then create a deferred object. We only resolve the deferred object if the request has been successful in each network request.

In our refresh method, we can combine the two promises, which will allow us to wait until two network requests have finished loading and load the updates and reviews that have been retrieved from the API. In this example, we’re only combining two promises, but Kovenant allows you to combine up to 20 promises at a time.

If you Google how to wait and detect if multiple asynchronous operations have finished, you’ll get slews of answers that range from very manual (which also can mean error prone) to something like “just learn RxJava and reactive programming”. I found that the using something like Kovenant and its easy to use promises is the best way to solve this problem of asynchronous callbacks.

Common Table Expressions within Rails

We use ActiveAdmin heavily at work.

And I have a love and hate relationship with it. While it’s an easy way to quickly spin up an admin dashboard in Rails, it is so opinionated that any time you want to do anything slightly out of the ordinary, it quickly becomes a nightmare for me. I guess people may say the same thing about Rails, but I digress…

I recently had to modify a CSV export within ActiveAdmin so that it includes two rows in the CSV export for each one row that’s in the database. For the sake of example, let’s say that we have a payments table where we have three columns.

In ActiveAdmin, you define how you want the csv export to look like in a “csv” block like this.

The above block of code will produce a csv file that will have one row per database row in the payments table. However, the requirement that I was given called for two csv rows per one database row with the processing_fee being displayed as amount. Since we can’t force ActiveAdmin’s CSV processor to produce two rows, we need to feed the CSV processor the collection with two records per one record already preprocessed. At first, I decided to attempt this with Common Table Expressions.

What’s a Common Table Expression (CTE)?

I think of Common Table Expressions as a temporary table that you construct within a query that you can query from. I think of it as a more temporary lite version of database views. Let’s say that you need to produce a result set with data coming from multiple tables. You could perform complex joins and then query from the joined result set, or you could create a CTE and simplify your queries. In my specific case, I needed to create a temporary table that contained two rows per one row in the database with the second row displaying the processing_fee as amount. How do we achieve this with Common Table Expression?

The query that I came up with was

The above query will construct a temporary table called cte_payments with the result set from the query contained inside the parenthesis. Then you can query the cte_payments table as you normally would. With the above query, we can do the following within ActiveAdmin to create a custom scope to feed into ActiveAdmin’s CSV processor.

The above in the if request.csv? block work in creating a collection of result sets that you want. However, find_by_sql method returns an Array rather than the ActiveRecord::Relation collection and unfortunately ActiveAdmin requires that you feed ActiveRecord::Relation into its CSV processor. If you are working outside of ActiveAdmin or in a situation where an Array will work fine for you, you can stop reading here. If you need your custom query that you execute within Rails to return an ActiveRecord::Relation, then read on.

If you must need to have your ActiveRecord query return an ActiveRecord::Relation collection, and you would like to stick with using CTEs, you can try using the postgres_ext gem that was created by the folks at DockYard. Unfortunately, the gem is no longer maintained. You can also dig into Arel and play in that world. There’s an excellent blog post on how to work in Arel to utilize Common Table Expressions here. I personally think that working directly with SQL is easier than working with a gem that’s no longer maintained and trying to figure out Arel, so I opted for rewriting my CTE query with a regular subquery that I can use with the ActiveRecord’s from method. If I rewrite the CTE query that I wrote above as a subquery, and use ActiveRecord to return an ActiveRecord::Relation collection, it would look like the following.

The above method will return an ActiveRecord::Relation instead of an Array that you get with using Common Table Expressions.

Global IDs in Rails

I recently learned about Global IDs in Rails applications.

Just as a reference, the link to the Global ID repo is https://github.com/rails/globalid.

That GitHub repo does a good job of explaining what Global ID is but TLDR version is that it’s a global identifier that you can use to locate an instance of a model.

I found it to be pretty useful when passing in ids of various model instances in a job queue. For example, let’s say that you have a few models with one model having a belongs_to polymorphic association with the rest of the models. You create a record of this polymorphic model in a job class. Let’s have an example here.

In the PictureCreatorJob, thanks to us using the global_id of the user, we can locate the model without having to conditionally check which model we should be querying with ActiveRecord. I find that in these specific situations, Global ID can make your code more succinct and concise.

Rails and PostgreSQL Arrays

This is something that I recently learned while working on Building Localize Ninja.

I have a Project model, where I need to store two sets of data, the source_locale and the target_locales. Basically, a Project represents a localization project that the translators/collaborators will be working on translating the app. The source_locale will only contain one language which will probably be English for many people and target_locales will be one to many languages.

The default way I approach this would have been creating a locales table where I insert all language that will be available and then creating a join table between the projects and the locales table. However, I wanted to see if there was a way to associates projects with multiple target_locales without joining tables.

At first, I thought about storing comma separated values as a string directly into a column called target_locales in the projects table. And after querying, I could split the string by commas and determine which languages the project is targeting. This is a bit archaic but I’ve seen it being done this way in other projects I have worked on and it seems to work for them. However, it felt a bit too hackity hack hack for me so I looked into it further.

After a few minutes of research, I learned that PostgreSQL supports storing arrays directly in a column. With this, rather than storing a list of languages in a separate table and then joining tables every time I want to figure out which target locales the project has, I could define the list of languages in Ruby as a constant and store the list of target languages in the array column, avoiding having an extra table just for the sake of keeping a list of languages.

To add an array column that will contain strings, migration will Rails will look something like this.

Note that you have to define what data type the array will hold. In my case, it will be strings, thus I used the text data type. If I wanted to store numbers, I would have used the integer data type instead.

Validations

Let’s say that we want to allow having projects to have the same name as long as they have different values for the target_locales column. In Ruby, arrays are considered to be not equal if they have the same values in different order. For example,

Let’s say that we want our program to recognize that [1, 2, 3] == [1, 3, 2] are equal and return true since they technically contain the exact same values, albeit in a different order. Within the world of Rails, we could have a before_save callback so that we sort our array before saving it into the database like this.

Something like the above will sort the list of target_locales that we have so that all of the locales saved in the database will be nil and empty string free and be sorted so that we can easily compare the target_locales list of one project to another.

If you wanted to make the above more performant, you could do the above within PostgreSQL land, sorting the array by creating a custom function and then creating a custom rule that will execute within an insert statement. I actually prefer doing things like this within Ruby land because there have been times where I was debugging something and couldn’t figure out why something unexpected was happening and then realizing a few hours later that there was a custom database rule that was firing that I didn’t realize existed. I think when I’m coding Rails apps, I tend to be stuck in Ruby land and tend to concentrate on the code in front of me. Thus, I generally like to have as much logic written in Ruby as possible unless I have a good reason to move operations out into the database. Usually when I consider such options, it’s due to performance bottlenecks.

Since I’m currently at the beginning stages of the project, I am sticking with doing this in Ruby land. If the project ever gets some users and I see performance bottlenecks, I will consider moving the above logic out into the database.

Custom spec types with RSpec

We use RSpec as our testing framework at Modern Message. And I learned something new about RSpec today.

You know those types that you can pass into RSpec specs? For example:

Or

Starting from RSpec 3+, we have to specifically define these spec types. And it turns out that these spec types provide RSpec specific methods that you can use like post, expect, and etc that you use to write your tests. Also, it turns out that you can define your custom spec types as well.

Why am I writing about custom spec types?

At Modern Message, I’m currently working on a feature where I have to have common before and after hooks for multiple specs. To be specific, I’m currently using the stripe-ruby-mock library to stub out external requests made to Stripe in my tests.

For each test that utilizes the stripe-ruby-mock gem, we need a before and after hook that starts and stops the StripeMock class. You can read through the docs in the GitHub page for the library, but essentially it looks something like this.

In my specific circumstance, I am writing tests for webhook events that Stripe sends out. For example, Stripe has a webhook event called invoice.payment_succeeded. To create a corresponding model in my Rails app, I created a folder called stripe_service in my models folder and a corresponding ruby class file app/models/stripe_service/invoice_payment_succeeded.rb.

Do note that Stripe does have multiple webhook events and in my specific case (and probably in most Rails applications), you would have multiple classes and files in the stripe_service folder that represents different webhook events. And that means more spec files in spec/models/stripe_service/ folder with more repetitive before and after hooks starting and stopping the StripeMock class.

While I’m one of those people who believe that there’s nothing wrong with a little bit of duplication (especially in my case where we only have three webhook event classes), we can remove this before and after hook duplication with a custom spec type. Thus, if we create a custom spec type, our Stripe::InvoicePaymentSucceeded spec will look like this.

To create this custom stripe_service type, we need to go into our spec/spec_helper.rb file and define this new type. Here’s an example.

Here, the first thing we do is that we specify that all of the specs in the spec/models/stripe_service folder to have the type stripe_service. And in the following lines of the configuration, we use our new strip_service metadata type, we specify that before and after hooks both start and stop the StripeMock classes. With this configuration, each time we specify our specs to have the type stripe_service, it’ll automatically run the before and after hooks automatically, reducing duplication in our specs.

MaterialReads Android app released

I released my first solo Android app on the Google Play Store.

It’s an Android client for Goodreads website.

You can download it here. I built it mostly for the sake of finishing a project. The goal is to slowly add features, bugs, and performance enhancements to increase downloads counts and see where I can take it from here.

 

Goodreads OAuth in Android

For my solo project, I’m building a Goodraeds client. Mostly because I like to use Goodreads to track the books I’m reading and I don’t enjoy the default Android app provided by Goodreads.

Thus, I’m building my own alternative using their API. However, it has been a tricky process from the beginning since

  1. Goodreads API isn’t very well documented.
  2. Logging in via Goodreads means that I need to do OAuth authentication with their API, which as mentioned in point 1, is not very well documented, and also is OAuth 1.0.

It took me some time to figure out how to perform OAuth authentication in Android development and I’ll go over the steps on how to do this in the hopes that it’ll save other people a bunch of time.

Just as a note, since this is Android development, the code samples that I’ll be showing will work in any standard Java application or any language that can easily interpolate with Java and targets the JVM. Also, since I’m writing this app in Kotlin, do note that the code samples will be in Kotlin with the exception of one small portion that relies on a library. But if you need to use Java, you can always look at the code sample and figure out what’s going on pretty easily.

Things that we will need

We’ll need several components that come together to make OAuth authentication work. They are:

  • Goodreads developer key and secrets
  • Your own server that will act as a callback url for the Goodreads OAuth API
  • ScribeJava library for making OAuth easy

As far as setting up your own server that will act as a callback url for the Goodreads OAuth API, you’re on your own. What you’ll have to do is set up a simple API that will receive the callback url that includes the OAuth token and whether the user authenticated your app from accessing his/her data or not from Goodreads API with a route that you’ll set in the Goodreads API Developer dashboard.

1 – Register for the Goodreads developer keys

Just Google something like Goodreads API developer keys and you’ll find a link in Goodreads that will allow you to register for the developer keys so that you can access their APIs. Once you have registered yourself as a developer, you’ll be redirected to a page that looks like this that will provide you with your developer key, secret, and form with inputs where you can enter in your app’s name, domain, and a callback url. Fill these out appropriately with the exception of the callback url.

2 – Set up your own API to act as a callback url

I won’t go into the details of how to set up your own API to act as a callback url. It shouldn’t be too difficult if you’re somewhat familiar with backend development. All this server has to do is take in two parameters, oauth_token and authorize which will be coming from the Goodreads API to your callback server, and then render the results in a json format. I like to user Ruby on Rails for backend development purposes and for the sake of providing a simple example, this is what my controller looks like.

And my routes file looks like

Having my Rails controllers and routes like this will give me a url that goes something like https://myurl.com/goodreads_oauth_callback.

Once you have deployed your server, go back to the Goodreads page where your developer keys are listed, and fill out the Callback URL form. Just as an example, mine looks like this (sensitive data blurred out).

3 – Install ScribeJava

Depending on what you’re developing, you can do this several ways. In the context of Android development specifically, simply add this line in your app’s build.gradle file.

Notice here that I’m using an older version of ScribeJava. As of this writing, the latest version of ScribeJava is at 5.3.0. For whatever reason, I couldn’t get Goodreads OAuth to work with the latest version of ScribeJava as I kept running into an exception that I couldn’t find a solution to. Thus, rather than trying to figure out how to make the latest version of ScribeJava work, I decided to simply use a version that worked out of the box. If you want to try to get the latest version of ScribeJava to work, give it a try. If you want the easy way and get Goodreads OAuth authentication to work following my code samples, stick with version 4.2.0.

4 – Set up a Goodreads API adapter for ScribeJava

ScribeJava comes with a lot of different adapters to help make it easier for the user to work with various popular OAuth authentication services like Twitter, Facebook, and etc. However, it does not come with an adapter for Goodreads. Thus, we’ll have to write one. Don’t worry, it’s fairly simple. We’ll write this one in Java instead of Kotlin. Also, since Kotlin interpolates with Java, feel free to write this one in Kotlin if you desire.

5 – Authenticate the user via OAuth

This is the part where we’ll put everything together so that users can use Goodreads OAuth to authenticate with your app. One thing about the Goodreads OAuth is that user authentication happens via their login page. Thus, whatever you’re building, you’ll have to redirect your users to a webview of some sort to display the login form provided by Goodreads. The code sample below is for Android development, thus we’ll be using the standard Android WebView. If you’re not doing Android development, adjust your code accordingly.

While the above code should be relatively easy to follow and understand, I’ll explain the most important part where we retrieve the authenticated user’s access tokens and secrets that we’ll be using to make requests to the Goodreads API.

You’ll see that I override onPageFinished method in my GoodreadsAuthView webview. In this overridden method, I check to see if the url in the WebView contains the callback url that we defined in the Goodreads developer page. If it does, we will grab the query parameters in that url to retrieve the oauth token and whether the user has authorized your app to access his/her data. If the user has authorized your app to access his/her Goodreads data, the query parameter authorize will return a String value of 1.

From here, we check to see if the query parameter authorize has returned 1, and then retrieve the access token and the access token secret utilizing the ScribeJava library. We then use these tokens and the ScribeJava library to make special OAuth request to the https://www.goodreads.com/api/auth_user endpoint so that we can parse out the authenticated user’s user_id. The /api/auth_user endpoint is a regular endpoint defined in the Goodreads API documentation, but for whatever reason, I found it close to impossible to try to retrieve this with regular curl requests (or using Retrofit in Android). However, if I use ScribeJava’s built in functions, retrieving the user_id of the authenticated user is very simple. To be transparent, this part where we try to retrieve the user_id of the authenticated users is the part where most developers seemed to be having trouble based on the questions posted in the Goodreads developer forums. ScribeJava makes this process simple as one line of code.

Note that once we sign the request using the access token and ScribeJava’s built in functionality, we need to parse out the user_id from the XML that gets returned from the Goodreads API. Once we have retrieved the user_id, we can go ahead and save the user_id, access_token, access_token_secret, and then oauth_token in a SharedPreferences so that we can use them throughout the rest of the codebase.

Conclusion

This actually took me around 2 days to figure out how to do. Yes, it seems long to figure out how to do a simple OAuth authentication, but it’s a lot more difficult than it should be due to the lack of support and documentation from Goodreads. I’m hoping that this blog post helps anyone who’s trying to work with the Goodreads API. If you have any questions, feel free to leave a comment below or email me directly. Thanks for reading.

after_create vs after_save vs after_commit

Callbacks are useful tools to use in Rails applications.

The ones I tend to use most are ones that run before or after certain actions are taken upon an ActiveRecord model. Some of the common ones you may see in a typical Rails codebase are before_save, before_create, after_create, and after_save. In this blog post, I’ll be writing specifically about the difference between the after_create and after_save, and then throw after_commit to the mix since it can be preferable to use after_commit in specific situations.

First, let’s take a look at the difference between after_create, after_save, and after_commit.


after_create

This is called after ActiveRecord::Base.save is called on new objects that haven’t been saved yet (meaning that no record exists in the database)

after_save

This is the same thing as after_create but is called whether the object is a new or an existing record in the database.

after_commit

This is called after the database transaction has completed.


Now, the part about “after the database transaction has completed” for after_commit is important because this will determine whether you will prefer after_commit vs the after_create or after_save. Basically what this means is that after_create or after_save can still run before the changes are actually saved to the database, meaning the database can raise an error if the data you want to commit to the database violates your database constraints (like a uniqueness constraint violation). Thus, in certain situations where you want to work with data that you are sure that has been written to the database, after_commit is preferable to after_create or after_save.

This is one of those “gotchas” about various callback options a developer may have that’s not blatantly obvious. Choosing the right callback can save you a lot of headache in creating accidental bugs.

 

Ruby’s assign if not nil operator

I think most Ruby programmers know the ||= operator, where a variable is assigned a value if nil.

For those who don’t know, it works like this.

In the example above, variable a is assigned 1 because it is nil. If a wasn’t nil, 1 wouldn’t be assigned to a and a would just remain nil.

I recently came across the &&= operator while reading Confident Ruby by Avdi Grimm. It basically works like this. It works kind of like the ||= operator except that it assigned a value to a variable if it isn’t nil. An example would be.

I think this is a good way to make your code concise by getting rid of a lot of if statements. For example, I do this a lot in my code.

That if statement above can be replaced with &&=, making our code more concise.

This is a short post, just sharing something new I’ve learned in the past few days.

Adding columns with default values to large tables in Postgres

I recently had to add a new column with a default value to an existing table in a Rails application.

The Rails application was backed by PostgreSQL and the table that we were adding this new column to had over 2 million rows of data. Now, adding a new column with a default value in Rails is a simple task that involves writing a migration that looks like

 

In most cases, the above will work. However, if you have really big table with large amounts of data, which in this specific case there were over 2 million rows of data, the above migration will take an eternity to run. This is because adding a default value for a column in a table will get Postgres to go over every row and update the default values for the said column. If you have a large table, this operation can take some time. What’s worse is that because the above migration is locked in a transaction, the table in question becomes locked during the migration and your application will grind to a halt since your database table won’t be functional.

So… how do we solve this? Let’s first list our problems in the situation where we’re adding a new default column to a really large table.

Problems

  • Column creation (or simply adding default value to an existing column) causes all rows in that table to be updated at the same time, which can be time consuming in really large tables
  • Updates are slow in Postgres since it has to guarantee consistency
  • During the update, the table that’s in operation is locked, causing the application to grind to a halt until the update finishes

To solve this problem, we need to work around the above facts so that we can add the default column and allow all of the rows in the large table to be updated while keeping the application running. How do we do this? Well… we can do the following.

Solution

  • Adding a column to a table without default values in Postgres is fast. Thus, we should add the new column with a default value of null first to get the creation of the new column out of the way.
  • Rather than updating every row in the table all at once, we can split up the number of records and update them in batches. This makes the work more manageable for the database.
  • If we make the updates of the rows in the table with default values more manageable, the table won’t be locked during the operations and the application can keep running during the migration.

So, let’s do this. To do this in a Rails migration, we need to recognize a few more things about how Rails handles migrations. Rails migrations are done in a transaction. Meaning anything in that def change are being ran in a transaction to allow for rollbacks. If we do any data updates there, it means that all of those updates will be wrapped in a transaction, locking the entire table. This means that we’ll need a way to disable this auto-transaction mechanism of Rails migrations and handle our transactions manually, picking and choosing where we want to wrap our database operations in a transactions and where we want to disable wrapping our database operations in transactions. Thankfully, Rails comes with a method called disable_ddl_transaction! that we can use in our migration files to disable the default transaction locking behavior in our migrations. Also, we want to replace the def change method with the def up and def down  so that we can manually define our rollback strategy just in case we want to rollback our migration. Below is my attempt at utilizing this feature of Rails and handling the transactions on my own.

For the sake of demonstration, let’s say that we’re adding a integer column named number_of_retries with a default value of 0 to a table called users.

You can see that I wrap all database operations (adding a column, altering columns with default values, and etc.) in transactions, while I do not wrap the part where I update the default user values in transactions. The find_in_batches has a default batch_size of 1,000, but you can always increase that to a larger number for faster migration speed if you want. I just found that 1,000 to be a good number for stable migrations. I’ve had situations where the migration timed out when I set the default batch_size to a higher number like 10,000. Also, as you can see, having a separate def up and def down methods in the migration allows us a way to safely rollback the migration if needed.

The above migration will work. Granted, it won’t be an instant migration. You’ll still have to wait for the migration to run. However, it’ll prevent locking of the table which grinds the application to a halt, which ends up in bad experience for your users. Unfortunately as of now, there are no good simple ways to add new columns with default values quickly to an existing large table. The above is a workaround that utilizes certain principles to add new columns with default values on really large tables.

I hope that this post is helpful for those looking to solve similar problems on their production Rails applications.