Tuesday, July 11, 2017

Splitting 40 character tokens or keys for readability with dashes

A good bit programming is about mangling text.

Python 3 has a lot of fun language features you can use to process text. Here is short Python 3 script that will split a 40 character length token (or text) into 6 segments. Of course, how many character per segment depends on the text length.


token = 'ad6d9c4e3fe09cfd24afdd62cc3705be02545272'

# double // so we don't have to do an int cast: int(len(token)/6) - Python 3 feature
chunks, chunk_size = len(token), len(token) // 6

keyed_token = [ token[i:i+chunk_size] for i in range(0, chunks, chunk_size)]

print('-'.join(keyed_token))


It should output: ad6d9c-4e3fe0-9cfd24-afdd62-cc3705-be0254-5272

Done!

Friday, June 30, 2017

Fixing CERTIFICATE_VERIFY_FAILED error on Macs with Python3

There's this annoying thing where Requests, a very popular Python library for working the internet doesn't work because of SSL certificates not being trusted.

This whole thing apparently is caused by OSX and Python3 having no certificates and can't validate SSL connections! This information is sort of hidden away in this itty bitty readme file on the certifi module which then pointed me to the readme file in python3. What. The. Fuck.

There, you'll know that for Python 3.6 on OSX, it requires a post-install step, which installs the certifi package of certificates. This is documented in the ReadMe, which you should find at /Applications/Python\ 3.6/ReadMe.rtf

The ReadMe will have you run this post-install script, which just installs certifi: 

/Applications/Python\ 3.6/Install\ Certificates.command

I should read the release notes for Python3 more closely.  

Wednesday, June 7, 2017

Dealing with outdated or abandoned pypi modules with VCS install

So I was using a small, simple but working Paypal module which unfortunately was forgotten or abandon by the original author. It was working on my Django 1.5 site up until I needed to upgrade to Django 1.10 because of the other modules. Also for security and stability reasons but that's another story for another time.

Fortunately, this Paypal module wasn't "super" broken. The errors were just a couple of lines in the templates and a function named patterns() from django.url.conf - urls.py.

This leads into a couple of solutions:

1. I could copy-pasta the code and turn it into a internal django app with the fixes.

2. I could fork it, fix and then setup by own pypi server.

3. I could find out if someone fixed this already and take it from there.

Of course, I'd pick #3 - Programmer. So, lazy. Look it up.

I was lucky to found out that someone has fixed all the issues and got it running on Django 1.10. All that's left is to fork his repo and install it into my virtual environment.

It's simple because pip already supports installing modules from a VCS.

$ pip install git+https://github.com/killertilapia/fixed-forked-project

If you need install a certain branch do this instead

$ pip install git+https://github.com/killertilapia/fixed-forked-project@branch

This also works with other VCS like Mercurial (hg+https), Bazaar (bzr+https) or for-fuck-sakes why! - SVN (svn+https).

Wednesday, May 31, 2017

Windows Python 3.6, VS2017 and the case of the missing basetsd.h

So you decided to join the Python 3 bandwagon on Windows with your new shiny Visual Studio. And then you start work on a fresh virtual environment and then you do a pip install or something. Sometimes, you'll end up with a compile error just like this:
WTF is this missing 'basetsd.h' file
You'll google this and end up with one of many stackoverflow topics stating you must install Visual Studio 2015 or some SDK.

Fuck that and save your time. You can do better because a programmer named Dimitri Janczak found a solution that works and I can confirm it works. 

The solution is as follows:

1. Install or update setuptools (> 34.0). Do note that setuptools is sort of tied with your version of Pip. So you might have to do: python -m pip install -U pip setuptools

2. The next step is to pick the correct command-line environment. What?! I know right, Visual Studio comes with multiple command prompts. 


Select the appropriate Native Tools Command prompt. Navigate to the project folder. Activate the virtual environment and do what you did last before you encountered the compile error.

If you goes to plan then you should get a successful result. 

Thursday, May 11, 2017

Using git push to deploy code to DigitalOcean

There'll be times using a continue integration server or stack is a big pain in the butt. Sometimes we just want to type git push from the terminal and the live app is updated. This is that tutorial although specific to DigitalOcean. I'll bet it will roughly work the same for other cloud services.

Assumptions
  • You have a Digital Ocean server (or droplet) running and configured
  • You have the ssh to said droplet working
So ssh to the droplet and then navigate to /var/www (or anywhere else you want to put your code) and make two directories:

$ mkdir liveapp.com && mkdir appcode.git
$ cd appcode.git

Inside the appcode.git folder we will create a "bare" git repository: git init --bare
A bare git repository doesn't contain our code but rather the internals of the .git folder. The whole point of this is to access the hooks folder within.

The hooks folder have scripts that can be run at certain steps within the git repo's lifecycle. We want a hook script to run after we "receive" a push to this repo. Hence we create a post-receive file inside the hooks folder.

$ vi post-receive

# Type this in

#!/bin/sh
git --work-tree=/var/www/liveapp.com --git-dir=/var/www/appcode.git checkout -f

# save and exit
# make the post-receive hook executable

$ chmod +x post-receive

There! We are done for the server side. We move to the our local machines.

All we actually need is need is to add a remote target in your working git repo. Assuming you added the droplet in your local ssh config then the command should be look like:

$ git remote add droplet ssh://user@ip-of-droplet/var/www/appcode.git

# add a new commit - this will not work even if you existing commits
# you must have a new commit
$ git add .
$ git commit -m "new commit"

# push to repo - could be github or bitbucket
$ git push

# push to droplet
$ git push droplet master

If all goes to plan, then our code will show in our new "git work tree" which is the liveapp.com folder inside our droplet.

A couple more thoughts:
  1. It's fairly easy to setup a sort of staging or testing folder/area with this.
  2. The post-receive script can do more things like a "restart app" or send mail to admin for notification. It's just a bash script.

Friday, May 5, 2017

Ghetto Python development with VS Code

Not everyone can afford Pycharm or Sublime or you just don't like the new Pycharm terms/license then this could be a alternative setup with Visual Studio Code.
  1. Install Visual Studio Code. It's has all the "f"s: fast, free, flexible, fowerful.
    Don't forget to install Python also.

    Tip: Install Python 3 unless you're maintaining legacy Python code then install Python 2
  2. Pick our plugins for Python. You can install plugins or extensions from the within VS Code; Just click on the "Extensions" button on the side menu or you can download them from the "marketplace" and install them manually. Go here for details.

    Here's our list of extensions.
    1. donjayamanne.python - our primary plugin so we can work with Python. It's got almost all the things we'd expect to make for a good Python IDE; Intellisense, Code formatting, refactoring, debugging, and linting.

      A word on linting though, The plugin doesn't come with it. You'll have to install your linting module like say Pylint or Flake 8 separately via pip. 

    2. Indent-raindow - Trust me, you'll need this. It makes seeing the code indents easier. 

    3. Jinja - This is an optional plugin. You'll probably only need this if you're working with a lot of Jinja templates.

      Don't forget to explore the marketplace for other customisations like themes.
  3. Start coding. Don't forget to make a virtual environment because VSCode does support them on a per project bases via a settings JSON file. Same with debugging.
  4. Push to a repo. VSCode already has a git client built-in so push your code to popular repos like Github, Gitlab or Bitbucket.
Code away!

Saturday, March 25, 2017

Visual Studio 2017 Xamarin stuck if you didn't install the SDK and NDK

I ran into this bug when I was upgrading from Visual Studio 2015 to 2017. I included the Xamarin option But I didn't include the Android SDK and NDK. I removed them from the install list. I did this because I already have both installed.

The "hang" occurs then you try opening the Xamarin in the Options window. Visual Studio will stop working here so you can't set the paths for Xamarin.

So I went digging through the Xamarin logs in the AppData folder. I know where the log folder is from a previous issue with Xamarin. Xamarin is a good idea but it's buggy as fuck. So navigate to %USERPROFILE%\AppData\Local\Xamarin\Logs\15.0\ folder and see which is the latest log and open it with your favorite text editor.

Once we confirm, if it's really the missing SDK and NDK paths, we then have to add them to Window's Registry.
  1. Open regedit
  2. Navigate to Computer\HKEY_CURRENT_USER\SOFTWARE\Xamarin\VisualStudio\15.0\Android
  3. Right-click and add new String Values
    • AndroidNdkDirectory - set it to the full path to where you install the NDK 
    • AndroidSdkDirectory - set it to the full path to where you install the SDK
  4. Close and restart the computer
And we should have fixed the issue.




Monday, March 6, 2017

Dealing with SSL HandShake failure with Python Requests (_ssl.c:590)

I was working with Requests trying to talk to a REST api when I encountered this SSL HandShake error. It was fairly easy to replicate because it happens every time I tried talking to a HTTPS endpoint or resource. It didn't matter if it was a GET or POST action. I suspect it'd be the same for other actions like PATCH or DELETE.

So what's the fix? Uninstall-install then maybe an update or upgrade.
  1. I updated Python to 2.7.11. You might not need to do this if your up-to-date. 
  2. I uninstalled requests. - $ pip uninstall requests
  3. I uninstalled urllib3. - $ pip uninstall urllib3
  4. Installed crypto dependencies or libs. This is where it gets hairy.

    First I had to update my machine's openSSL library. This could range from easy to hair pulling hard. Thankfully, I had brew on my machine so it was relatively easy.

    Anyhow you'll want to see this from the terminal after the update.

    $ openssl version
    OpenSSL 0.9.8zh 14 Jan 2016
    

    After you'll need to run this command:

    $ pip install pyopenssl ndg-httpsclient pyasn1
    

     This installs the rest of the crypto dependencies.
  5. I installed urllib3 back. - $ pip install urllib3
    I also installed urllib3[secure]. - $ pip install urllib3[secure]
    You might not need the secure version but I covering all the bases.
  6.  Install requests back. - $ pip install requests
    Same deal with urllib3, you might also want to install request with security.

    $ pip install request[security]
Why this works? I suspect that there's an installation-time check for either urllib3 (or requests) for the crypto stuff that keeps things from working without the uninstall-reinstall.





Tuesday, January 24, 2017

Doing query param request in Django

Typically, I want my API endpoint url to be flat like this:

mydomain.com/shit_api/v1/person/123

That url is pretty straight forward, it's trying to get a person with an ID of 123 from shit_api. But that's not always clear cut.

I've been working on a particular API that needed a pair of optional filters that we're a pain to work out in a regular expression. And this is where I found about Django's QueryDict object. With it I can write out API urls with params:

mydomain.com/shit_api/v1/something/?filter1=value1&filter2=value2

And on the urls and views, I can handle it as such:

#urls.py
urlpatterns = [
   url(r'^parts/$', SomethingView.as_view()), name='Something'),
]
...

#views.py
class SomethingView(View):
   def get(self, request, *args, *kwargs):
      fiter1_value = request.GET.get('filter1', 'default')
      fiter2_value = request.GET.get('filter2', 'default')

      #do more here

      return Response('some response')

The thing here is that request.GET.get() method having a default value. This make it that having it on the URL it isn't a problem. So call like:

mydomain.com/shit_api/v1/something/

Will still work.

Bonus since I don't have to deal with a convoluted regular expression. Yehey!

Friday, January 13, 2017

Tricks with Django URLs and knowing you suck doing RE

RE or regular expressions and I don't really see eye to eye. But as a programmer, I have to work with them. So it's awkward like your ex works in the same office as you. But lucky for us, we just stick to a couple of tricks to make it a bit palatable.

Numbers in URL

You'll need to use a \d+ in the pattern.

ex: (r’^status/(\d+)$’, ‘myapp.backbone_view’)

This would work on /api/status/1

Dealing with spaces

I found two ways to do this but then found out that they are the same: [\w|\W]+ AND .+

Yes that's a dot.

ex: (r’^status/([\w|\W]+)$’, ‘myapp.backbone_view’) OR
(r’^status/(.+)$’, ‘myapp.backbone_view’)

This would work on /api/status/dead on arrival

The browser will just replace the spaces with %20 and it should work.