sqlite3 isn't exactly new. It's been around for almost two decades now (much longer than I've known coding myself 😛). There's plethora of articles on the internet describing what it is, what makes it special, when you should ideally use it (and when not) and why "it's the only database you will ever need".

So, I'm not gonna repeat all those advices here. Instead I'll try to cover some of other exciting, specialised interfaces that sqlite3 offers and how we can profit from it!

#1 Custom functions in SQL

sqlite3 provides support for registering custom, application-defined functions in C that can be exposed to queries in SQL.

Custom functions can be used to integrate your queries with the rest of the system, or provide access to external functionality. They can be used to perform complex calculation or do statistical operations on data.

Custom functions can be used to marshal / un-marshal and / or manipulate data in alternative formats (such as json). They can be used to compute hashes and do much more!

sqlite3 comes with a bunch of built-in scalar and aggregate functions, and with routines to manipulate date / time information. There is also nalgeon/sqlean on Github that aims to pack a lot more set of functions into convenient extensions that can be loaded on-demand.

#2 Virtual Tables

Virtual Tables are one of the most exciting features of sqlite3 that I found! Essentially, virtual table allows us to register objects with sqlite3 that, from the perspective of an SQL statement, looks like any other table / view.

This allows the user to be able to query them just as they would do with any other table / view. When a user executes a query against a virtual table sqlite3 invokes callback methods on the virtual table object.

This has the potential to unlock a ton of different use-cases. For eg., there's an official csv virtual table implementation that allow users to query data from a csv file just as they would with a normal table. There are other implementations that make it possible to interface with other systems, such as augmentable-dev/askgit uses sqlite3 virtual tables to allow users to query data from git repositories, osquery.io that allow users to run queries against the operating systems!

#3 Authorizer to check / limit what a user can query

sqlite3 supports registering a custom authorizer function that can limit what data a query can access and what action can it perform. This, together with other defensive settings, can be used to turn sqlite3 into a sandbox where we can execute raw, user-supplied, untrusted statements with confidence that it won't cause any un-intended side-effects.

#4 Capture a change-set of differences in a session

sqlite3 session extension provides a mechanism for recording changes to some or all of the rowid tables in an SQLite database, and packaging those changes into a "changeset" file that can later be used to apply the same set of changes to another database. This can be used to capture diffs generated by multiple users, working asynchronously, capturing and merging it together into a unified database file.

Sessions can be useful when sqlite3 is used as an application data format __ and multiple users work on the same file at the same time. Each user can generate a patch for their side of work (much like how one would do it with a vcs like git) and the system could combine those into a unified database file.

#5 Online Backup API

sqlite3Online Backup API allows the contents of one database to be copied into another database, in an incremental fashion where the database is only locked for the brief periods of time when it is actually being read from. This allows other database users to continue uninterrupted while a backup of an online database is made.

It provides a basic building block onto which other, more complex replication solutions can be built.

#6 Dynamically loaded extensions

sqlite3 has support for building extensions that can either be loaded dynamically (as a shared object file) or linked statically into the final build.

The sqlite3ext.h defines the Extension API which the module can use to interact with the database connection (that loaded the extension). The extension can do pretty much anything that you could normally do with the sqlite3 core, like registering custom functions, defining virtual table modules and more!

One should consider building re-usable modules as loadable extensions to allow for easier distribution and better re-use.

In the follow-up articles, I would try to cover some of the features I've discussed here and how one can leverage those while coding with Golang using http://go.riyazali.net/sqlite, a library that I recently open-sourced 🤗

I'll cover the whole hole punching mechanism, perhaps, in a different article. But for context, the mentioned source used raw linux sockets and classic Berkeley Packet Filter to inject custom packets and do hole punching.

The source is written in C but I wanted my implementation in Golang. So, I started looking and soon came across golang.org/x/net/bpf which provides an implementation of raw bpf assembly codes and a virtual machine written in Golang. I wanted to use the assembly codes but not necessarily depend on the bpf.VM (that would've defeated the purpose of filters in the first place 😅).

Instead, I started looking for ways in which I could use setsockopt with SO_ATTACH_FILTER to attach the filter directly to the native file descriptor. I came across ipv4#PacketConn.SetBPF method from golang.org/x/net/ipv4 which takes in a []bpf.RawInstructions and applies it onto the underlying socket, although the socket in this case wasn't exactly the raw socket I was working with.

To circumvent this, I started looking further into the source, till I found sockOpt.setAttachFilter and, consequently, setsockopt's implementation on unix. The setsockopt used syscall.Syscall6 with syscall.SYS_SETSOCKOPT to implement the system call and passed through the given arguments as unsafe.Pointer to the syscall.

Putting it all together

After all this I came with something close the following code snippet to enable using berkeley filter with sockets in Golang 🚀

package filter

import (
    "golang.org/x/net/bpf"
    "golang.org/x/sys/unix"

    "syscall"
    "unsafe"
)

// Filter represents a classic BPF filter program that can be applied to a socket
type Filter []bpf.Instruction

// ApplyTo applies the current filter onto the provided file descriptor
func (filter Filter) ApplyTo(fd int) (err error) {
    var assembled []bpf.RawInstruction
    if assembled, err = bpf.Assemble(filter); err != nil {
        return err
    }

    var program = unix.SockFprog{
        Len: uint16(len(assembled)), 
        Filter: (*unix.SockFilter)(unsafe.Pointer(&assembled[0])),
    }
    var b = (*[unix.SizeofSockFprog]byte)(unsafe.Pointer(&program))[:unix.SizeofSockFprog]

    if _, _, errno := syscall.Syscall6(syscall.SYS_SETSOCKOPT,
        uintptr(fd), uintptr(syscall.SOL_SOCKET), uintptr(syscall.SO_ATTACH_FILTER),
        uintptr(unsafe.Pointer(&b[0])), uintptr(len(b)), 0); errno != 0 {
        return errno
    }

    return nil
}

Using this is simple too! You just have to define the filter program using bpf assembly codes, like:

// filter packet by checking if they are destined to local port 8080
var filter = Filter{
    bpf.LoadAbsolute{Off: 22, Size: 2},  // load the destination port
    bpf.JumpIf{Val: 8080, SkipFalse: 1}, // if Val != 8080 skip next instruction

    bpf.RetConstant{Val: 0xffff}, // return 0xffff bytes (or less) from packet
    bpf.RetConstant{Val: 0x0},    // return 0 bytes, effectively ignore this packet
}

and call ApplyTo on the socket's file descriptor.

// import "syscall"

// open a raw socket
fd, err := syscall.Socket(syscall.AF_INET, syscall.SOCK_RAW, syscall.IPPROTO_UDP)
if err != nil {
    // ... define error handling
}

// then apply the filter
err = filter.ApplyTo(fd)
if err != nil {
    // ... define error handling
}

And that's it! the kernel would filter out packets based on specified filter program and next time you do a syscall.Read on your fd you'd only see the packets you're interested in 🎉 🚀

I've started a new job @ CloudCover and have also started contributing to golang projects!

The inspiration for this post comes from a (slightly) confusing topic in golang which I initially struggled to understand, and that's interfaces!

This post doesn't cover what interfaces are! There are lots of really good articles on the internet for that.

What I really want to focus on is "how" you can use them and what common pitfalls to avoid (especially if you come from Java / C#)

#1 Define interfaces close to site of use

This allows the code (that makes use of the interface) to depend on abstractions rather than on concrete implementations.

And because interfaces in go are implemented implicitly, your actual implementation would just work auto-magically!

// UserService defines a service that's responsible to manage user profiles
type UserService interface {
    // FetchAll returns a cursor that lists all users on the platform
    FetchAll() UserIterator

    // ......
    // this interface could contains more (but related) methods
    // just make sure to not bloat it (see Rule #3)
}

// Now we could utilize UserService abstraction in our code.
// Here we pass it to a function that returns an http.HandlerFunc
// When that handler is executed, the code could utilize the supplied user service
// without having to worry about where the actual implementation came from

func ListAllUser(svc UserService) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        var cursor = svc.FetchAll()
        //...
    }
}

This allows you to decouple your code and it's dependencies enabling you to write modular code.

DO NOT place the interface in the package that provides the dependency and than make the provided service implement that interface. This seems like a common pattern used in other languages (namely, Java) where you'd declare an interface com.example.UserService and than provide a package-private implementation of it com.example.UserServiceImpl (along with a factory perhaps)

The real power of Go interfaces comes from the fact that they can be implemented implicitly! This reduces the overhead of arranging for an implementation before-hand and also allows a single type to implement multiple (but un-related) interfaces or implement interfaces that doesn't even exist yet (crazy right!)

A common example from the standard library to support this would be the fmt.Stringer interface which is defined close to it's site of use (the methods in fmt package) yet anyone can provide an implementation by adding a method with compatible signature String() string

#2 If it's gonna be used at multiple places, declare it only once

If you feel like your interface definition is going to be used in multiple places, it's better to declare it once (and maybe in a shared package)

Wait a minute.. doesn't that contradict #1 ?

Umm no, not really. You need to declare them in a single place and make the symbol (the interface) available to your site of use. This doesn't mean you need to couple it with your actual implementation.

A notable project which makes use of this pattern is drone.io

All the core interfaces used by drone are declared under the core package which is than used by various other packages and satisfied by another set of packages.

#3 Keep it small silly 😛

Try and stick with SOLID!

More specifically one should try to follow Interface Segregation principle.

Quoting from Wikipedia,

The Interface Segregation principle states that no client should be forced to depend on methods it does not use.

One way to achieve this would be to keep your interfaces lean and focused. Model your interfaces around behaviour, and not data, and prefer embedding over large interfaces.

A very popular example from the standard library would be the interfaces in io package

type Reader interface {
    Read(p []byte) (n int, err error)
}

type Closer interface {
    Close() error
}

type ReadCloser interface {
    Reader
    Closer
}

#4 Accept interfaces return structs

Lol no, I didn't come up with it 😅 in fact, no one's actually sure who did 🙄 but people often cite Postel's Law

The basic idea is that your method should accept (as arguments) values of interface types while should return values which are concrete types.

A few examples from the standard library would be

• io.Pipe method whose signature declares the concrete underlying type of the return values

   func Pipe() (*PipeReader, *PipeWriter)
  

• io.Copy method whose signature declares accepted argument types to be interfaces

    func Copy(dst Writer, src Reader) (written int64, err error)
  

The primary advantage here is that your function is now decoupled and depends on behaviour rather than concrete types of the arguments.

Yet consumers of your function have the flexibility to choose the way they want to use the return type (by casting it appropriately) based on their use case.

This doesn't mean that you should (or can) always declare methods like that. There are exceptions. For example, io.TeeReader method

func TeeReader(r Reader, w Writer) Reader

Here, it actually makes sense to wrap the underlying implementation and return an io.Reader

Another common use case for returning interface types from method is to ensure that the value being returned actually satisfies the interface. I think this is rather a hack around Go's philosophy that "interfaces are implemented implicitly".

If what you actually need is a way to ensure that your custom implementation satisfies an interface you can use interface guards.

Bonus

Interface guards

An interface guard (not an official term) is a simple no-cost hack to add checks in your code to ensure that your implementation satisfies some interface. Use this if you are building a custom implementation for an interface that is exported by a third-party library.

Following snippet is taken from Caddy's documentation on interface guards

var _ InterfaceName = (*YourType)(nil)

This is simple yet no-cost way of ensuring *YourType implemements InterfaceName at compile-time rather than running into errors at runtime.

Got doubts / queries / suggestions, feel free to open an issue on Github!

Good documentation gives the developers the necessary control to maintain a system - nochance @ stackexchange

But good code doesn't need documentation 🤔

Remember, documentation go way beyond the comments that are inserted in code. Often times, they're a combination of well describing code comments, design documents, business requirement documents and other relevant pieces.

Ain’t nobody got time for that ⏰

The main reason code goes undocumented is because of time!

Developers often argue that they could rather be more productive and be writing code than spending time documenting the software.

The case for documentation

No matter what you're building, chances are that some day you (or somebody else) would need to re-visit it, to fix some bug or extend it to add new feature or refactor it or any other reason.. and when that day comes, you will not remember so vividly what you wrote and why.

And if you do remember, there may be some edge cases or specific uses which may not be clearly apparent. The obvious solution is documentation.

And after all, when we, as developers, need to understand something about certain aspect of coding / library / framework, what do we do?

We go look at the documentation!

But aren't comments a code smell 🤷‍♂️

Umm... only if the comment describes what the code is doing ^[1]

static int add(int a, int b) {
  // add b to a and return the result
  return a + b;
}

👆 that's 🤷‍♂️

But have a look at following snippet from Spring Data Commons' CrudRepository.java

/**
 * Interface for generic CRUD operations on a repository for a specific type.
 *
 * @author Oliver Gierke
 * @author Eberhard Wolff
 */
@NoRepositoryBean
public interface CrudRepository<T, ID> extends Repository<T, ID> {

  /**
   * Saves a given entity. Use the returned instance for further operations as the save operation might have changed the
   * entity instance completely.
   *
   * @param entity must not be {@literal null}.
   * @return the saved entity; will never be {@literal null}.
   */
  <S extends T> S save(S entity);

  // other definitions omitted..
}

The comments here describe why the interface is there and what the function does (not how it does it!)

This makes it very clear for any future reader to know about the purpose of the interface and what the function is supposed to do.

This instance is just one example! There are several other places where comments can be very useful.

@antirez, creator of Redis, had put together an excellent blog about comments! Make sure you check it out.

👆only covers code comments / documentation.. what about other forms of documentation?

They are all important and are there to serve different use cases!

• Design docs are there to describe and postulate the approach a project took to solve a given problem
• System documentation represents documents that describe the system itself and its parts.
• User documentation covers manuals that are mainly prepared for end-users of the product and system administrators. User documentation includes tutorials, user guides, troubleshooting manuals, installation, and reference manuals.

MDN, Spring, Django, Stripe are all good examples of products / tools / services with an excellent documentation.

References

• Why should you document code?
• This Answer to "comments are code smell"
• Writing system software: code comments by @antirez
• Why documentation matters, and why you should include it in your code by Tomer Ben Rachel

To begin with, the Fork workflow (just workflow from now on) extend Github's Workflow and adds forking to the mix..

And why do that?

Several benefits.

• You don't need to give everyone write access to the central repository
• It keeps the central repository "clean" (arguable..) by not polluting it with developers' feature branches
• Developers can form subteams (by adding other dev's clone as a remote) (useful for collaborating, pair-programming etc.)
• Enforce review guidelines by limiting direct write access
• Very useful for remote teams (personal experience..)

Convinced? nice! read ahead to know more about how to implement..

Implementation

• Organsiation admins ^[1] (or users with write access) start by creating the central repository (the upstream)
• Members than fork the upstream under their own account (the origin for that dev)
• The developers than start by cloning their respective origin (each dev has complete access over their forks) and add the upstream remote

git clone git@github.com:user/repo
git remote add upstream git@github.com:organisation/repo

• One can then follow the git branching workflow that they are already familiar with.

git checkout -b feature/the-next-big-stuff

• Once they've started working on the feature, they can push it up to their fork.

git push -u origin HEAD

• Once they've pushed the code to origin, they can then open a Pull Request to the upstream where the project owner / maintainer can review the code and merge it (or suggest any changes)

Keeping the code in sync

When multiple developers are working on a project it's very likely that the upstream would move much faster than the devs' origin. In that case, the dev would need to periodically (ideally, they should do it daily) sync their origin with the upstream.

To do that, devs need to execute following (on their local machine) -

# fetch upstream and all branches
> git fetch upstream
# checkout local or own develop
> git checkout develop
# get the changes from the upstream and merge to your local.
> git merge upstream/develop
# push the code to the develop branch of your fork
> git push origin develop

The devs should also execute ☝️ once they've their PR merged.

And that's it! you've got a scalable git workflow model that would work well even for remote teams (we've used it quite extensively!)..

And just to make it clear.. we are not the first ones who've used this or anything like that 😛.. this flow is used extensively in open-source world where generally there is a central upstream canonical repository (where all issues are made and project is released) and each contributor has his/her own "fork" of that repository. The beauty of this flow is that it scales so well even for a highly distributed open source project!

References

• Understanding the GitHub Flow by GitHub
• A successful Git branching model by Vincent Driessen @ nvie
• A big shout to my friend @vinitkme for familiarising me with this!

It's just more overhead man..

🤷‍♂️ people often claim that it’s because they’re saving time by skipping the spec-writing phase.

But what they don't realise is that in the long run (even for a smaller project that's true), they might be spending a lot more time fixing, iterating and communicating all the same stuff (over and over again) what they could have done once if they had written a proper spec for it.

Joel Spolsky (@spolsky) has written [quite extensively] about this in his article.

So, why does it matter?

A well-structured design doc can save you days, or even weeks of wasted time!

The main goal of a design doc is to make you more effective by forcing you to think through the design and gather [early] feedback from others.

Point is that when you design your product in a human language, it only takes a few minutes to try thinking about several possibilities, revising, and improving your design whereas when you design the same product in a computer language, it takes weeks to do iterative designs / changes.

Another big reason of having well-written design docs is that they save us time communicating the stuff by making sure that everyone is on the same page, and that can help us build great products and innovate quickly. Everybody on the team can "just read the spec".

What happens when you don’t have a spec is that all this communication still happens, because it has to, but it happens ad-hoc..

How do I go about writing a design doc?

Umm... I won't go into too much detail. There are already several good articles on the internet that does that. I'll try and summarize the points (and maybe add a few from my side) into the following list -

• Try and stick with high-level overview without diving too much into implementation detail
• Define what service you're building and what are the objectives
• Provide a context to the readers, describing where the service fits in the overall architecture
• Outline goals and non-goals of the service and make them quite explicit
• Give a high level technical overview. Use user / case stories to depict scenarios
• Highlight your dependencies (what other services your service will depend on) and tell "why"
• Describe your choice of stack (but don't dive too much into technicalities) and the reason why you chose them
• Highlight any potential issues / open questions and get feedback from peers
• Add links to other sites, references or articles that might be helpful
• Get feedback early-on rather (and perhaps on each step)

I personally found that following ☝️ steps is generally very helpful and makes one more productive than doing it the other way

References

• How to write a good software design doc by Angela Zhang
• Painless Functional Specification - Part 1 by Joel Spolsky

Is it even a challenge? One could say that I could've simply used a library like Picasso or Glide to offload image loading to the tried and trusted tools. But my main challenge was not just to load the image but to load the perfect image.

And that means the image should fit-well into ImageView without much need to resize and/or scale it, and thus, giving a crisp and sharp look and improving UX.

So, what's the big deal? Why don't I just use the different bucket sizes (w185, w300, etc.) that TMDb offers?

The thing is that the aspect size of the images served by TMDb is a bit weird (IMO!). For example, for the w185 bucket, the image served has an aspect ratio of 185:278!

What can be done? Well, the only thing we can do is to download a bigger image (from original bucket) and resize it on device. But that's a total no-go because the original image is too large (2000x3000) and would consume signifacntly more resources. And also, performing resize on user's device is not a good idea!

What I initally thought was to setup a Thumbor server. But I quickly dropped the idea because then I would have to maintain the server (which I didn't want) for the (atleast) lifetime of the project or else the app would simply not work!

This led me to search for open, free and hosted alternatives. Although, I couldn't find a trustworthy solution, what I found was this Gist. In the Gist, the author reverse engineered (sort of) the URL for the Google Image proxy and the required parameters and leveraged it as a caching and resizing proxy to serve his images!

Cool! how to use that in the project app? Simple! create the primary TMDb image url, append it as a parameter to the Google Image Proxy url and load the resulting url into the ImageView using your favorite library!

So, here's a Utility class that I created to handle the stuff!

public final class ImageUtils {
  private ImageUtils(){
    /*  ¯\_(ツ)_/¯ */
  }

 // Google's image caching and resizing proxy
 private static final String GOOGLE_CACHE_BASE =
  "https://images1-focus-opensocial.googleusercontent.com/gadgets/proxy";


  /**
   * Create base tmdb url
   */
  @NonNull private static String tmdbUrl(@NonNull String poster){
    return String.format(
      "%s/original%s", 
      // or however you get the base url...
      BuildConfig.TMDB_IMAGE_BASE, 
      poster);
  }

  /**
   * Calculate aspect
   */
  public static int aspect(int width, int height) {
    return height / width;
  }

  /**
   * Calculate the other dimension (either height or width) 
   * given one dimension and an aspect
   *
   * Use as: calculateOtherDimension(aspect(4, 3), 400) -> 300
   *
   * @param aspect aspect ratio
   * @param dimension measurement of one dimension
   */
  public static int calculateOtherDimension(int aspect, int dimension){
    return aspect * dimension;
  }

  @NonNull public static String createTmdbImageUrl(
    @NonNull String poster, 
    int height, 
    int width){
      return Uri.parse(GOOGLE_CACHE_BASE).buildUpon()
        .appendQueryParameter("container", "focus")
        .appendQueryParameter("resize_h", "" + height)
        .appendQueryParameter("resize_w", "" + width)
        .appendQueryParameter("url", tmdbUrl(poster))
        .toString();
  }

}

Now to use it, just call it as

import static ImageUtils.*;

// ...
createTmdbImageUrl(
  movie.getPoster(), 
  300 /* height */, 
  calculateOtherDimension(aspect(2 /*w*/, 3 /*h*/), 300)
);
//...

{{< image src="https://images1-focus-opensocial.googleusercontent.com/gadgets/proxy?container=focus&resize_w=200&resize_h=300&url=http%3A%2F%2Fimage.tmdb.org%2Ft%2Fp%2Foriginal%2Fto0spRl1CMDvyUbOnbb4fTk3VAd.jpg" alt="Deadpool" position="center" >}}

And that's it! Using this you can load image of any aspect that you need and deliver a crispier and sharper experience to your users. But, I'd sincerely like to request to you to only use this for small, open source projects so that you don't need to maintain a server!
If you are developing a closed-source business application, than it would be much better to use something like Thumbor or ImageProxy as they are dedicated solutions and provide more features!

As you might've guessed, TMDb has a huge collection of movies, and so a need for paging the data was quite obvious from the very begining of the project.

So I started looking into ways in which I could've implemented paging with my RecyclerView. Turns out there are a few ways to achieve this!

1. The old way (as documented in this CodePath guide) is to use RecyclerView.OnScrollListener and listen for scroll events and load data accordingly! But with the release of the Android Paging Library, the guide is now considered deprecated!

2. The new way (yeah! you guessed it) is to use Android Paging Library and the rest of this post is all about how to integrate it!

Architecture

So you ask, how this Paging Library works?

Well, there are four major components in the Paging Library:

1. DataSource: The DataSource is where you write code to interact with your source (REST API, sqlite database, etc.) All the requests for data from the DataSource are made on a background thread and so you can safely write synchronous code to interact with your source, if you want to. DataSource is responsible to load a List<T> of your model object given a page/key and implement business logic for the same.

2. PagedListAdapter<T, VH>: PagedListAdapter is a RecyclerView.Adapter which uses a PagedList (see below) to load data into the UI. PagedListAdapter uses a DiffUtils.ItemCallback<T> implementation to calculate difference between your model objects in the list and animates add/remove operations on UI.

3. DiffUtils.ItemCallback<T>: The DiffUtils.ItemCallback<T> for your model class is utilized by the PagedListAdapter to calculate diff between two list and animate changes in the UI. You must provide an implementation of this callback. The callback is executed on a background thread to prevent stalling the UI.

4. PagedList<T>: A PagedList is a Java-collection which loads it's data in chunks (pages) from a DataSource. The PagedList connects your PagedListAdapter and your DataSource together and is responsible to lazy load data by calling appropriate methods on the DataSource.

{{< image src="https://cdn-images-1.medium.com/max/800/1*O1acN3yAOS70zbRMfThXrw.gif" alt="Paging Library Architecture" position="center" >}}

First, we create a model POJO class (referred in the post as Movie) and implement DiffUtil.ItemCallback<T>

public class Movie {
  // id of the movie
  private long id;
  // title of the movie
  private String title;
  // path to movie's poster
  private String poster;

  // DiffCallback to assist Adapter
  public static final DiffUtil.ItemCallback<Movie> DIFF_CALLBACK = 
    new DiffUtil.ItemCallback<Movie>() {
      @Override 
      public boolean areItemsTheSame(Movie oldItem, Movie newItem) {
        return oldItem.id == newItem.id;
      }

      @Override
      public boolean areContentsTheSame(Movie oldItem, Movie newItem) {
        return TextUtils.equals(oldItem.poster, newItem.poster)
            && TextUtils.equals(oldItem.title, newItem.title);
      }
    };
}

Then, we subclass PagedListAdapter and create our custom adapter which will be used to display Movie

// required imports
import android.arch.paging.PagedListAdapter;

public class MovieAdapter 
  extends PagedListAdapter<Movie, MovieAdapter.ViewHolder> {
  // ViewHolder removed for brevity... nothing special in the ViewHolder

  /**
   * Create new adapter and pass the diff callback to parent
   */
  public MovieAdapter(
      @NonNull DiffUtil.ItemCallback<Movie> diffCallback) {
    super(diffCallback);
  }

  // ... then the usual create(...) and bind(...) view holder methods
}

Next we need to create a DataSource that will connect with the API and fetch data! There are 3 types of DataSource provided by the Paging Library:

• ItemKeyedDataSource<Key, Value>: Use it if you need to use data from item N-1 to load item N

• PageKeyedDataSource<Key, Value>: Use it if pages you load embed keys for loading adjacent pages.

• PositionalDataSource<T>: Use it if you can load pages of a requested size at arbitrary positions, and provide a fixed item count.

You can read more about the types of DataSource and when to use which one on DataSource documentation

For our example, we will use PageKeyedDataSource<Key, Value>

// required imports
import android.arch.paging.PageKeyedDataSource;

public final class MoviesDataSource 
  extends PageKeyedDataSource<Integer, Movie> {
    // constant max retry count, see Bonus section for more!
    private static final int MAX_RETRY = 3;
    // retrofit service instance
    private final Tmdb.Api tmdbApi;

    public MoviesDataSource(@NonNull Tmdb.Api tmdbApi) {
      this.tmdbApi = tmdbApi;
    }

    @Override public void loadInitial(@NonNull LoadInitialParams<Integer> params,
      @NonNull LoadInitialCallback<Integer, Movie> callback) {

    // load the first page
    tmdbApi.getPopularMovies(1 /* page */).enqueue(new CallbackWithRetry<MovieResponse>(MAX_RETRY) {
      @Override public void onResponse(Call<MovieResponse> call, Response<MovieResponse> response) {
        if (response.isSuccessful()) {
          MovieResponse movieResponse = response.body();

          // send response back
          callback.onResult(movieResponse.getMovies(), null, // null because there is no page before this
              movieResponse.getPage() + 1); // TMDb follows incrementing integer as pages numbers
        } else {
          onFailure(call, new Exception("unknown error"));
        }
      }

      @Override public void onFinalFailure(Call<MovieResponse> call, Throwable t) {
        callback.onResult(Collections.emptyList(), null, null);
      }
    });
  }

  @Override public void loadBefore(@NonNull LoadParams<Integer> params,
      @NonNull LoadCallback<Integer, Movie> callback) {

    // params.key contains the page number of the page to load
    tmdbApi.getPopularMovies(params.key).enqueue(new CallbackWithRetry<MovieResponse>(MAX_RETRY) {
      @Override public void onResponse(Call<MovieResponse> call, Response<MovieResponse> response) {
        if (response.isSuccessful()) {
          MovieResponse movieResponse = response.body();
          // return response, since we are moving backward adjacent page is - 1 from current position
          callback.onResult(movieResponse.getMovies(), movieResponse.getPage() - 1);
        } else {
          onFailure(call, new Exception("unknown error"));
        }
      }

      @Override public void onFinalFailure(Call<MovieResponse> call, Throwable t) {
        callback.onResult(Collections.emptyList(), null);
      }
    });
  }

  @Override public void loadAfter(@NonNull LoadParams<Integer> params,
      @NonNull LoadCallback<Integer, Movie> callback) {

    // params.key contains the page number of the page to load
    tmdbApi.getPopularMovies(params.key).enqueue(new CallbackWithRetry<MovieResponse>(MAX_RETRY) {
      @Override public void onResponse(Call<MovieResponse> call, Response<MovieResponse> response) {
        if(response.isSuccessful()){
          MovieResponse movieResponse = response.body();
          // return response, since we are moving forward adjacent page is + 1 from current position
          callback.onResult(movieResponse.getMovies(), movieResponse.getPage() + 1);
        } else {
          onFailure(call, new Exception("unknown error"));
        }
      }

      @Override public void onFinalFailure(Call<MovieResponse> call, Throwable t) {
        callback.onResult(Collections.emptyList(), null);
      }
    });
  }
}

The only thing that remains for us is to connect our MovieAdapter with our MovieDataSource using a PagedList.

To use a PagedList, we first create a PagedList.Config. We also need to create two Executor which will be used to execute fetch and notification calls from the PagedList, i.e., data fetching via DataSource will execute on the fetch executor and load- and boundary- callbacks will be executed by the notify executor.

import java.util.concurrent.Executor;
import java.util.concurrent.ExecutorService;

// Ui thread executor to execute runnable on UI thread
class UiThreadExecutor implements Executor {
  private Handler handler = new Handler(Looper.getMainLooper());

  @Override public void execute(@NonNull Runnable command) {
    handler.post(command);
  }
}

// Background thread executor to execute runnable on background thread
class BackgroundThreadExecutor implements Executor {
  private ExecutorService executorService = 
        Executors.newFixedThreadPool(2);

  @Override public void execute(@NonNull Runnable command) {
    executorService.execute(command);
  }
}

Then in your Activity

@Override protected void onCreate(Bundle savedInstanceState) {
  super.onCreate(savedInstanceState);

  // usual onCreate code...

  // setup recycler view
  RecyclerView recycler = findViewById(R.id.movies);
  recycler.setLayoutManager(new GridLayoutManager(this, 2));
  // create new movie adapter
  MovieAdapter adapter = new MovieAdapter(Movie.DIFF_CALLBACK);
  // add adapter to recyclerview
  recycler.setAdapter(adapter);

  // create data source
  MoviesDataSource dataSource = new MoviesDataSource(getRetofitApi());
  // create configuration
  // see https://is.gd/l4hv42
  PagedList.Config config = new PagedList.Config.Builder()
      // this is not used by TMDb as we cannot pass page size to API
      // but is required by PagedList to use as hint
      .setPageSize(10)
      .setPrefetchDistance(5)
      .build();
  // create paged list
  PagedList<Movie> pagedList = 
    new PagedList.Builder<>(dataSource, config)
      // get list notifications on Ui thread
      .setNotifyExecutor(new UiThreadExecutor())
      // execute fetches on a background thread
      .setFetchExecutor(new BackgroundThreadExecutor())
      // build!
      .build();
  // update adapter's paged list (and data source)
  adapter.submitList(pagedList);
}

That's it! In a (sort of) few lines of code (if we remove the inherent verbosity in Java), we get a RecyclerView.Adapter that is capable of efficiently loading data and presenting it on the UI all while keeping it low on resources and preventing janky UI!

Bonus!

Here is the source of the retrofit2.Callback<T> with exponential backoff retry policy, i.e., it will retry the request each time on failure with exponentially increasing delay to prevent network and API abuse!

// retrofit callback implementation with retry policy
// adapted from https://stackoverflow.com/a/41884400/6611700
// adapted from https://is.gd/KaEwPt (exponential delay)
abstract class CallbackWithRetry<T> implements retrofit2.Callback<T> {
  // exponential backoff delay
  private static final int RETRY_DELAY = 300; /* ms */

  // max allowed retries
  private final int maxAttempts;
  // current attempts
  private int attempts = 0;

  CallbackWithRetry(int max){
    this.maxAttempts = max;
  }

  @Override public final void onFailure(Call<T> call, Throwable t) {
    attempts++;
    if(attempts < maxAttempts){
      int delay = 
          (int) (RETRY_DELAY * Math.pow(2, Math.max(0, attempts - 1)));
      new Handler().postDelayed(() -> retry(call), delay);
    } else {
      onFinalFailure(call, t);
    }
  }

  // failure hook called when retries are consumed
  public abstract void onFinalFailure(Call<T> call, Throwable t);

  private void retry(Call<T> call){
    call.clone().enqueue(this);
  }
}

Hope you enjoyed the post! If you've any doubts regarding any of the stuff here feel free to open an issue on GitHub{:target="_blank"} and I'll try my best to explain it! Till then goodbye!

Then, one day while checking into the forum an idea struck me! Why not make an app for the discussion forum itself! I knew the forum was powered by Discourse and quickly started looking up for ways in which in could create an app for it...

And voila! I found the official Discourse API. I quickly went through the API and found that for most of the calls the user needed to be authenticated (due to obivious reasons). But the only way to authenticate was to use an Admin generated API key. Obviously, I couldn't generate the API keys myself and didn't want to bother the forum moderators or admins. Besides that the Admin API key would grant complete read/write access to the forum to anyone who have it, not something a forum admin would want. So I thought to scrape the idea...

But then something struck me! I knew a user could log into the forum from a browser (very obvious) and the Discourse frontend (written in Ember) communicated to the backend (written in Ruby) using an API. I suspected if it was the same API? Turns out the API used by Discourse frontend is a superset of the public API and it uses the _t cookie as the Session Cookie.

And so I started looking for ways I could use this to create a login for the user of my app. Turns out it's not that easy.

First I thought to use Chrome Custom Tabs but it turns out that you cannot pass cookies between an app and a chrome custom tab for security reasons

Then I tried using the WebView and succeeded to some extent. But the solution was very fragile as there is no well-defined /login page on Discourse and the user could easily navigate away from the WebView. I know I could've prevented the navigation but Discourse allows 3^rd party login integration and then I would have to account for all possible domains a user could have navigated too! And besides these there are no autofill and password remembering features in WebView which means the user would have to login manually then.

{{< image src="https://media.giphy.com/media/YoGX4OZFjUwRq/giphy.gif" alt="Lo and Behold" position="center" >}}

And then I found what I was actually looking for! The Discourse User API key specification!! First created in Aug'16, this feature facilitates “application” access to Discourse instances without needing to involve moderators. The spec provides a flow similar to OAuth but with a few tweaks so that the Discourse instance doesn't have to know about the app prior to authentication. The token generation flow is documented here

The problem with User API key spec is that it's documentation is outdated and the canonical way is to follow the source of the Discourse Mobile App which is a companion app and provide features like Push notificaton, etc. (not browsing!) The app is written in React Native.

After a lot of trial-and-error I finally go it to work natively (in Java) on Android. And following is the Gist of what I did.

1 . Create a DiscourseAuthAgent class.

  public class DiscourseAuthAgent {
    // Randomly generated or static client id for your app
    private String clientId;

    // Random string to verify that the 
    // reply indeed comes from Discourse server
    private String nonce;

    // RSA Public key
    private String publicKey;

    // RSA Private key
    private String privateKey;

    // API key we get from the server
    private String apiKey;
  }

2 . To generate the RSA KeyPair

  String[] generateKeyPair(){
    String[] result = new String[2];

    // generate KeyPair
    KeyPairGenerator kpg = KeyPairGenerator.getInstance("RSA");
    kpg.initialize(2048);
    KeyPair keyPair = kpg.generateKeyPair();

    // get and base64 encode public key
    byte[] publicKey  = keyPair.getPublic().getEncoded();
    String b64PublicKey = Base64.encodeToString(publicKey, Base64.DEFAULT);
    result[0] = b64PublicKey;

    // get and base63 encode private key
    byte[] privateKey = keyPair.getPrivate().getEncoded();
    String b64PrivateKey = Base64.encodeToString(privateKey, Base64.DEFAULT);
    b64PublicKey[1] = b64PrivateKey;

    return result;
  }

3 . For the clientId and nonce you should generate a random string of length 32 and 16 respectively. I tried with other length options but failed so I guess it's good to go with these values.

  static String randomString(final int sizeOfRandomString) {
    // taken from https://stackoverflow.com/a/12116194/6611700
    String ALLOWED_CHARACTERS = 
      "0123456789qwertyuiopasdfghjklzxcvbnm";

    final Random random= new Random();
    final StringBuilder sb= new StringBuilder(sizeOfRandomString);

    for(int i=0;i<sizeOfRandomString;++i)
      sb.append(
        ALLOWED_CHARACTERS.charAt(
          random.nextInt(ALLOWED_CHARACTERS.length())
        )
      );
    return sb.toString();
  }

4 . Now to generate a Uri,

  /**
   * @param appName Your Application name that will be displayed to the user
   * @param scopes Permission scopes that you want (e.g. read, write)
   * @param redirectUri Uri to redirect to after authentication
   */
  Uri generateAuthUri(String appName, String[] scopes, String redirectUri) {
    clientId = randomString(32 /* size */);
    nonce = randomString(16 /* size */);
    String[] rsaKeyPair = generateKeyPair();
    publicKey = rsaKeyPair[0];
    privateKey = rsaKeyPair[1];

    return Uri.parse(BASE_URL).buildUpon()
      .appendEncodedPath("user-api-key/new")
      .appendQueryParameter("scopes", TextUtils.join(",", scopes))
      .appendQueryParameter("client_id", clientId)
      .appendQueryParameter("nonce", nonce)
      .appendQueryParameter("auth_redirect", redirectUri)
      .appendQueryParameter("push_url", "https://api.discourse.org/api/publish_android")  // placeholder
      .appendQueryParameter("application_name", appName)
      .appendQueryParameter("public_key", getFormattedPublicKey(Base64.decode(rsaPublicKey, Base64.DEFAULT)))
      .build();
  }

  String getFormattedPublicKey(byte[] unformatted){
    String pkcs1pem = "-----BEGIN PUBLIC KEY-----\n";
    pkcs1pem += Base64.encodeToString(unformatted, Base64.DEFAULT);
    pkcs1pem += "-----END PUBLIC KEY-----";
    return pkcs1pem;
  }

5 . Finally to validate the result of the authentication and to get the API key from the redirect, we need to decrypt the payload query parameter that was sent with the redirection. The payload is encrypted using our publicKey and can only be decrypted by our privateKey

  boolean handleAuthIntent(Intent data){
    if(null == data.getData())
      return false;

    try {
      String payload = java.net.URLDecoder.decode(
        redirection.getData().getQueryParameter("payload").replace("+", "%2B"),
        "UTF-8"
      ).replace("%2B", "+");

      // get back the private key
      byte[] privateKeyBytes = Base64.decode(privateKey, Base64.DEFAULT);
      PKCS8EncodedKeySpec spec = new PKCS8EncodedKeySpec(privateKeyBytes);
      PrivateKey privateKey = KeyFactory.getInstance("RSA").generatePrivate(spec);

      // initialize the Cipher
      Cipher cipher = Cipher.getInstance("RSA/NONE/PKCS1Padding");
      cipher.init(Cipher.DECRYPT_MODE, privateKey);

      // decrypt the payload into json
      byte[] decryptedBytes = cipher.doFinal(Base64.decode(payload, Base64.DEFAULT));
      String decrypted = new String(decryptedBytes);

      // create json object, verify nonce and let the fun begin!!!
      JSONObject payloadJson = new JSONObject(decrypted);

      // compare nonce
      if(!nonce.equals(payloadJson.getString("nonce")))
        throw new IllegalAccessException("invalid nonce");

      apiKey = payloadJson.getString("key");

      // and we are all done...
      return true;
    } catch (Exception e) {
      // failed to decrypt!
      // app must die :(
      throw new RuntimeException(e);
    }
  }

6 . And that's it!! Now you just need to persist the clientId and apiKey (preferably in SharedPreferences) so that you can use them later to authenticate your requests.

Now to use the DiscourseAuthAgent class in your (say) LoginActivity, add this to your manifest :

  <activity android:name=".LoginActivity" android:launchMode="singleTask">
    <!-- deep link redirection handler after authentication -->
    <intent-filter>
      <action android:name="android.intent.action.VIEW" />
      <category android:name="android.intent.category.DEFAULT" />
      <category android:name="android.intent.category.BROWSABLE" />
      <data android:scheme="discourse" android:host="auth_redirect" />
    </intent-filter>
  </activity>

This will make sure that the same instance of the Activity gets the deep linking intent so that you can use the same DiscourseAuthAgent instance.

In the example LoginActivity, start authentication by getting Uri and launching the default browser or a Chrome Cutom Tab (preferred).

  // on some button click
  // redirect uri must be exactly the same, I tried with other but failed
  Uri rediectTo = discourseAgent.generateAuthUri("My Amazing App", new String[]{ "read", "write" }, "discourse://auth_redirect");
  CustomTabsIntent customTabsIntent = new CustomTabsIntent.Builder()
    .setToolbarColor(getResources().getColor(R.color.colorPrimary))
    .build();
  customTabsIntent.launchUrl(this, redirectTo);

Then, when Chrome redirects the user back to your Activity:

@Override protected void onNewIntent(Intent intent) {
  super.onNewIntent(intent);
  if(null != intent.getData() && intent.getData().toString().startsWith(REDIRECT_URI)){
    // redirection intent
    Timber.d("reply: %s", intent.getData());
    if(discourseAgent.handleAuthIntent(intent)){
      // authentication completed!
      Timber.d("Session authenticated!");
    } else {
      Timber.d("Failed to authenticate session :(");
    }
  }
}

And well that's it! Now that you have an api key you can make autheticated calls to the Discourse Public API using the key. Just send the key in the User-Api-Key HTTP header and your client id in the User-Api-Client-Id HTTP header.