Thursday, February 14, 2019

Random quirk with Windows, Git and an executable Bash script

The context of the problem was that I was making a Docker container with Linux image. It's to be deployed to a cloud service and I'm working from Windows machine. Of course the container would need code to run. To facilitate this, I wrote a basic bash script - scrape.sh.

On Windows, the scrape.sh file already is executable
For disclosure, I'm working with Scrapy here.

From my Windows terminal, the scrape.sh script seems to be already executable. I thought I was fine and I did the whole git cycle of add, commit and push.

The problem shows up when the container tries to run. It will spit out an error saying that it can't execute the scrape.sh script because it's lacking rights. It seems that git commits my bash script as an ordinary file not as an executable despite what I saw on the Windows terminal. The fix though was quite easy.


$ git add --chmod=+x -- afile
$ git commit -m"afile is now executable"


Relevant link:



Friday, January 4, 2019

Installing Scrapy on Windows

Data is the new oil they say and you want to start scraping sites for data. Fine! And since you are a Python developer you'd want to use Scrapy. Unfortunately for you you are a Windows user and errors abound.

Anyhow, you run pip install scrapy and you run into an error:

Scrapy failing to pip install in Windows

The error here is one of dependencies of Scrapy which is Twisted. Fortunately this is fixable.

The first thing we're going to do is download the "binary" wheel file for Twisted. There's a trick here though, YOU MUST DOWNLOAD THE CORRECT ONE THAT MATCHES YOUR PYTHON VERSION. So if you are on Python 3.6, you're looking for something that reads like Twisted-18.9.0-cp36-cp36m-win32.whl; if Python 3.7 then you're looking for something with cp37-cp37m. You get the idea.

The 32-bit or 64-bit probably also matters but I didn't test it.

Once you have that file downloaded somewhere, you can then do a pip install .

Try installing Scrapy again and you should be good to go.

As an added bonus, this seems to fix other Python packages in Windows that require the Visual Studio Windows C++ SDK like the mysql-python.

Tip for Pipenv users:

1. pipenv shell
2. pip install
3. pipenv sync


Friday, November 16, 2018

My Guidelines for designing a restful APIs so frontend devs won't ask where I live

As a backend developer, I have to play nice with the frontend people. Let's face it, I don't want to be prison shanked on the lunch line. Easier said than done because of the different ideas that people have floated around like these three.
Each one of these have pros and cons but those are academic papers that I don't care to write right now. But I've worked and talked to a lot of frontend devs and they don't really care if we use any of the three examples that I gave. 

For them, a good API:
  1. Is well documented - makes sense; I'm a fan of drfdocs. Also Swagger gets a lot of mentions.
  2. Is versioned - good point; because changes to existing ones break too many things
  3. Is idempotent - to be honest I had to look this one up but it's the same idea of pure functions. Given the same input, it should return the same value (output)
  4. Uses HTTP errors correctly - For example, don't return errors with 200 status. Oops.
  5. Uses HTTP verbs sensibly - they want none of that 'GET /v1.0/delete/100' because that shit is stupid.
  6. Is NOT SOAP - dear lord up above in heaven, not ever SOAP. Ever
I also don't like SOAP.