Friday, November 16, 2018

My Guidelines for designing a restful APIs so frontend devs won't ask where I live

As a backend developer, I have to play nice with the frontend people. Let's face it, I don't want to be prison shanked on the lunch line. Easier said than done because of the different ideas that people have floated around like these three.
Each one of these have pros and cons but those are academic papers that I don't care to write right now. But I've worked and talked to a lot of frontend devs and they don't really care if we use any of the three examples that I gave. 

For them, a good API:
  1. Is well documented - makes sense; I'm a fan of drfdocs. Also Swagger gets a lot of mentions.
  2. Is versioned - good point; because changes to existing ones break too many things
  3. Is idempotent - to be honest I had to look this one up but it's the same idea of pure functions. Given the same input, it should return the same value (output)
  4. Uses HTTP errors correctly - For example, don't return errors with 200 status. Oops.
  5. Uses HTTP verbs sensibly - they want none of that 'GET /v1.0/delete/100' because that shit is stupid.
  6. Is NOT SOAP - dear lord up above in heaven, not ever SOAP. Ever
I also don't like SOAP. 

Thursday, October 25, 2018

2 months into Flutter

So I've been learning Flutter trying to get in deep. I'm been using Flutter for make an Android app and so far, I'm largely impressed but I have met a few small annoyances.

The good. 

1. Hot reload is the killer feature. I won't be a happy mobile developer if have to wait for the damn AVD to restart to see my changes. 

2. Android Studio support. I'll have to disclose that I'm also a Pycharm user so you could say this is a biased assessment. I've tried Flutter with VSCode, it works but having to manage the Android stuff like AVDs manually is a pain.

3. Good looking apps fast. Once you figure out how the layout system works along with the theming system, you'll have a blast creating your app UI also tweaking it.

4. Aside from good looking app, you also have nice responsive apps. Although I have to admit, the app I built didn't have that many moving parts.

The bad.

1. Dart. The language is like Java and JavaScript had a bastard child. Being a Python dev for the last decade, the curly braces just irks me. Also, it has a lot of optional syntax rules that bug me. 

2. The learning curve. Flutter isn't particularly easy especially for those without CS or programming training or experience. For example, how it handles databases and API calls is via Futures; Learning how to use Futures isn't trivial especially if you want a maintainable code base.

3. Missing key IDE features. The feature I'm taking about is the "build" feature. If you are using Android Studio, you'll still be forced to use the command line to build your final release APK. You can't do this from the IDE. May be the future.

I got nothing if you're an IOS developer.

Conclusion:

Flutter works although it has some of rough edges. I'd like to think these rough edges are teething problems. I expect a couple of these rougher edges be polished away before the end of year. I'm liking Flutter so far and I think I'm in good company - Alibaba used Flutter for their Xianyu app.

Friday, August 17, 2018

Cleaning up your global NPM packages

Sooner or later when you're working with any web app, you'll have to deal with nodejs because frontend requirements will force you to do so. And sooner or later, you'll have a mess of npm packages in your global space. You'll want to clean it up but when you issue the command npm list you get a whole mess of packages with their dependencies.

However there's a useful trick you can do that will make npm only list the top level packages without their dependencies.


$ npm list -g --depth 0


Here's a sample result:

The trick is the --depth 0 parameter. It suppresses the listing of a package's dependencies.

We can now tame our unruly global npm packages.

Ha!

Wednesday, July 25, 2018

Pyenv folder (.venv) with Pipenv

Pipenv, by default will use your virtualenv folder location if you set that environment variable - which probably be a .virtualenv or .env folder in your home directory.

But sometimes you would like to have the virtualenv folder within the project folder - ie. project/.venv No sweat, pipenv supports this workflow if set the PIPENV_VENV_IN_PROJECT environment variable.

For Macs: $ export PIPENV_VENV_IN_PROJECT=1

For Windows: > set PIPENV_VENV_IN_PROJECT=1

Do note that in Windows, you can also use setx.

Then we can run either a: pipenv install or pipenv --three to get started.




Friday, May 25, 2018

Cleaning up your local repo from too many dead branches

You'll eventually have too branches in your local that's untracked - ex. Not in origin or Github. This gets annoying since too many of these branches are often dead. So time to prune them:

git remote prune origin

You would assume that would work but nope. This only works on the remote repo and not your local. But you can to do this:

git branch --merged master | grep -v '^[ *]*master$' | xargs git branch -d

This is uncomplicated since this just soft deletes all branches that are already merged - the -d flag in git branch.

But if you need to check against the remote repo then a more complex command is called for:

git branch -r | awk '{print $1}' | egrep -v -f /dev/fd/0 <(git branch -vv | grep origin) | awk '{print $1}' | xargs git branch -d

The only rub here that is only works on bash or zsh shells. Also you need to run git fetch --prune first because using this.

Powershell

git checkout master; git remote update origin --prune; git branch -vv | Select-String -Pattern ": gone]" | % { $_.toString().Trim().Split(" ")[0]} | % {git branch -d $_}
Reference:

  • https://stackoverflow.com/questions/13064613/how-to-prune-local-tracking-branches-that-do-not-exist-on-remote-anymore/16906759#16906759

Wednesday, May 9, 2018

Understanding the two kinds of smart contracts

Stellar is a platform built with Blockchain that enabled someone to build a financial system like say a payment system. One feature on Stellar is Stellar Smart Contracts (SSC) and I've been trying to wrap my head around it.

So what is a smart contract? An SC is just an agreement between 2 or more parties formalised over a computer network. Think paper contracts but with a computer network as a witness. The catch is that there are TWO kinds of smart contracts - Turing complete OR non-turing.

Turing complete SCs like Ethereum's Solidity include code that EXECUTES ON THE CONTRACT when given certain inputs. What can be executed is still subject to constraints on the contract like say time - ie. You can execute this contract after 48 hours.

Non-Turing complete SCs like Stellar's do not include code with the contract. So you have to execute the contract on your side - ie. your own server. But similar to the other type of SC, what can be executed is still subject to constraints.

Turing complete SCs are like self contain black boxes with buttons and sensors. It does stuff on it's own based on what button is pressed or what the sensors pick up.

Non-Turing SCs on the other hand are like those hand-crank jack-in-box toys, who (when, how, etc.) can crank it defined is identified on the toy. On it's own it doesn't do much. It just sits there.

So what use cases are talking about?

Non-Turing SCs seem to be pretty good for escrow type transactions. Stellar's SC examples are in fact, escrows.

Turing complete SCs seem to fit into long term, repeating type transactions like retirement payments.

Which one to use? I can't say. I've been learning Stellar via a Python SDK. It's been interesting so far.

Monday, March 26, 2018

Why use the middleware approach for Django analytics?

Because of either proxy servers, ad blockers (like ABP, uBlock, etc.), or browser settings that stop JS scripts. Depending on the level of users and how much value they place on their privacy, they can outright BLOCK all forms of analytics. This ends up with our analytics data being unreliable or at least viewed with less confidence.

But why the middleware?

Django middleware is a good place because it's called for all request and response cycle.

The upside to this is:

  1. It's easy to customise. Django middleware is just a plain Python class with some methods. 
  2. The Django middleware is well-documented. 
  3. Allows us to setup whatever rules like exclude errors and only track HTTP 200 responses.
  4. We can setup tracking to be asynchronous. This way, our pages are not at the mercy of an external resource.
Writer's note: If your analytics needs are simple, you can install django-google-analytics and follow Usage #2 - Middleware + Celery.

There are a couple of downsides to this though:
  1. We could be impacting performance in a big way if the analytics middleware is doing too many things or worst, is misconfigured. 
  2. Not as bad as #1 but if we went down the asynchronous path, we will end up with a Celery server with some message broker like RabbitMQ on the tech stack. Another thing we'll have to manage. 
TL:DR For analytics with Django, HTML tag slows down pages and could be blocked; better to use a middleware approach.

Friday, February 23, 2018

Using Django forms to validate POST data in Rest Framework

Django Rest Framework already has a couple of ways to validate data. I believe the "preferred" way is to validate data from serializers. For example:


from rest_framework import serializers

class SomeSerializer(serializers.ModelSerializer):

    def validate(self, data):
        errors = {}
        year_built = data.get('year_built')
        year_rennovated = data.get('year_rennovated')
        
        if year_rennovated < year_built:
            errors['error'] = "You can't renovate something that doesn't exist!"
            raise serializers.ValidationError(errors)
            
        return data

But you can use Django forms, especially if you have existing ones that have the same "shape" of your post data.

Here's an example form:

from django import forms

class BuildingForm(forms.Form):

    year_built = forms.IntegerField(required=True)
    year_rennovated = forms.IntegerField(required=True)
                            
    def clean(self):
        cleaned = super(BuildingForm, self).clean()
        year_built =  cleaned.get('year_built')
        year_rennovated =  cleaned.get('year_rennovated')

        if year_rennovated < year_built:
            raise forms.ValidationError(u'Oops!')


And then in our API View:

from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework import status

class BuildingView(APIView):
    def get(self, request):
        model_form = forms.BuildingForm(request.data)
        if model_form.is_valid():
            data = serializers.someSerializer(obj).data
            return Response(data, status=status.HTTP_200_OK)
        else:
            return Response({'errors': model_form.errors}, status=status.HTTP_400_BAD_REQUEST)

Thursday, January 11, 2018

Django lazily evals URL patterns so I thought the "include" function was broken

Little known things sometimes kick your ass and you'll need to call a friend.

Until today, I didn't realise that Django lazily evaluates it's URLs. I discovered this when the code me and my team were working on had a cascading URL include. So we had something like:

# main urls.py
urlpattern = [ 
   url(r'^path/', include('app1.urls'),
]

# 2nd url - app1's url.py
urlpattern = [
   url(r'^app1/', include('app1.subapp.urls'),
]

# 3rd url - subapp for app1 urls.py
urlpattern = [
   url(r'^subapp/$, actual_view.as_view()),
]

Problem started when I assumed that the full URL path will show in the debug page on the browser. I was looking for /path/app1/subapp/. I only saw the /path listing.

This is where I thought that Django was not registering the rest. It never dawned on me that by just doing "localhost:8000/path/" will allow me to see the rest of the URL pathing. Django just shows the same or next urls not the urls 2 levels in.

Thank you Eric (@riclags) for pointing out that fact. 

I'm a dumbass sometimes.

Friday, January 5, 2018

When JSON has URLs but you need data (or how to chain API calls and merging the result)

The Problem

Let's say you have an API (ie. http://api.example.org/stuffs) that returns an array of stuff. But the API dev who worked on it, the bastard that he is, decided that certain fields, like say price is a hyperlink. So it looks like:

[{
   "id": "exde01"
   "name": "Stuff1",
   "type": "just stuff",
   "weight": "3.22",
   "price": "http://api.example.org/price/exde01"
},
{
   "id": "exde02"
   "name": "Stuff2",
   "type": "just stuff",
   "weight": "3.25",
   "price": "http://api.example.org/price/exde02"
}
.....
]

Ok. Now when you do an API call to the price url you'd get something like:

{
   "inlu_price": 69.96
   "exlu_tax": 49.99,
   "price": 123.25
}

So now the problem is how do we loop over the first JSON result with stuff and calling each price URL and then appending back the result. Essentially, we want the end result to look like this:

[{
   "id": "exde01"
   "name": "Stuff1",
   "type": "just stuff",
   "weight": "3.22",
   "price": {
     "inlu_price": 69.96
     "exlu_tax": 49.99,
     "price": 123.25
   }
},
{
   "id": "exde02"
   "name": "Stuff2",
   "type": "just stuff",
   "weight": "3.25",
   "price": {
     "inlu_price": 69.96
     "exlu_tax": 49.99,
     "price": 123.25
   }
}
.....
]

The Solution

My solution is in TypeScript and if it wasn't for reactive extensions (or streams) I'd probably have hanged myself. 

The first step is to get the main JSON. So I wrote a service for it.

    getAllStuff(): Observable {
        const url = "http://api.example.org/stuffs";

        return this.http.get(url}).map((response) => response.json());
    }

Now for the tricky part.

   this.restService.getAllStuff().switchMap((stuffs) => {
      return Observable.forJoin(stuffs.map((s) => {
         return this.http.get(s.price).map((res) => {
            s.price = res;
            return s;
         }
      }
   })
   .subscribe((r) => {
      console.log(JSON.stringify(r));
   });

This looks hard but it isn't really once you understand how it "flows".

First the getAllStuff() call will return the first set of results (technically, an Observable array) which we pass in into the forJoin() function.

What forJoin() does it take an array of API calls (as Observables) and spits out a single "merged" result. That's where the stuffs.map() does. But take note this is a two step process.

If you just focus on just the stuffs.map() . You can access stuff JSON array and drill down to the s.price, ie. ['http://api.example.org/price/exde01', 'http://api.example.org/price/exde02'].

This is why then we do a return with the this.http.get() function to produce that array forJoin() needs. ForJoin() then resolves these calls. After resolving, the result is passed back up to the switchMap() function which we can then subscribe to get the transformed JSON.

There we get our desired JSON result albeit with a bit of processing.

Now, if there's a 2nd URL in your main json, then you'll just need another switchMap(). You can chain multiple switchMaps().

References:

  • http://blog.danieleghidoli.it/2016/10/22/http-rxjs-observables-angular/
  • https://www.learnrxjs.io/operators/transformation/switchmap.html
  • https://stackoverflow.com/questions/38517552/angular-2-chained-http-get-requests-with-iterable-array

Wednesday, January 3, 2018

Being asked "Is Angular 1 dead?" too often

Happy new year everyone.

With the pleasantries out of the way let's get down to the topic at hand. All too often when I'm doing a tech demo or talk somewhere I get this question, "Is Angular 1 dead? Is it worth still learning?" or some variation of.

My answer is that Angular1 is not dead. It's no longer the "next big thing". That's why we no longer see much news of it. It's just a thing. We need to understand that web apps written in angular1 and there are thousands of these web apps out there, will never be rewritten in anything else. The cost is just too great and no sane PM or owner will pay for it. Remember the first adage of maintaining anything, if ain't broken, don't fix it.

Besides, Angular1 is here. It's good solution. It's well maintained. It's has good docs and bugs are getting fixed. It's just here, it's used by millions of developers. There's still value in learning Angular1 but how much value depends on your circumstances.

Usually technology doesn’t die, just new projects stop using it. You can still find plenty of websites out there, in development still using old Angular1 or even backbone, even plain jQuery.
If you plan on making your tech stack based on the latest headlines, you will never deliver anything. Never forget, there is always something new.