Updating powershell with powershell

To update powershell with powershell you, run the command: 

iex "& { $(irm https://aka.ms/install-powershell.ps1) } -UseMSI"

That's it. 

I'm using Windows Terminal btw.

NOTE: This does not update the modules external to PS. You'll have to update those on their own.

Scrapy a JS heavy website using Selenium

Scrapy doesn't really like JavaScript heavy websites especially the ones that load the rest of the HTML via a secondary requests using JavaScript.

To overcome this you either use Splash or Selenium. Unfortunately, Splash is no longer supported. It still works but moving forward, it's going to be Selenium. 

The good news here is that Scrapy already supports Selenium via a middleware: scrapy-selenium.

Steps to use scrapy-selenium:

1. Download a Selenium Driver. For example for Firefox get gecko. I'm assuming you have the browser also installed. 

2. Add the following settings to the settings.py file:

SELENIUM_DRIVER_NAME = 'firefox'

SELENIUM_DRIVER_EXECUTABLE_PATH = 'path/to/gecko'

SELENIUM_BROWSER_EXECUTABLE_PATH = 'path/to/firefox binary'

SELENIUM_DRIVER_ARGUMENTS=['-headless']  # '--headless' if using chrome instead of firefox

For example (on Windows):

SELENIUM_DRIVER_NAME = 'firefox'

SELENIUM_DRIVER_EXECUTABLE_PATH = 'c:\\Tools\\geckodriver.exe'

SELENIUM_BROWSER_EXECUTABLE_PATH = 'c:\\Program Files\\Mozilla Firefox\\firefox.exe'

SELENIUM_DRIVER_ARGUMENTS=['-headless']  # '--headless' if using chrome instead of firefox

3. In the spiders, just replace the Request() calls to SeleniumRequest()

4. Add a wait_until test on the SeleniumRequest() to would look like:

SeleniumRequest(url=url, 
                callback=self.parse, 
                wait_time=5,
                wait_until=EC.visibility_of_element_located((By.CSS_SELECTOR, 'div.search-results div.search-cell')))

In this one, we wait for the max of 5 secs or until the element, selected by class to be found in the HTML source. After that, we can use scrapy selectors to find the things we want.

So that's it.

Random quirk with Windows, Git and an executable Bash script

The context of the problem was that I was making a Docker container with Linux image. It's to be deployed to a cloud service and I'm working from Windows machine. Of course the container would need code to run. To facilitate this, I wrote a basic bash script - scrape.sh.

On Windows, the scrape.sh file already is executable

For disclosure, I'm working with Scrapy here.

From my Windows terminal, the scrape.sh script seems to be already executable. I thought I was fine and I did the whole git cycle of add, commit and push.

The problem shows up when the container tries to run. It will spit out an error saying that it can't execute the scrape.sh script because it's lacking rights. It seems that git commits my bash script as an ordinary file not as an executable despite what I saw on the Windows terminal. The fix though was quite easy.

$ git add --chmod=+x -- afile
$ git commit -m"afile is now executable"

Relevant link:

• Stackoverflow
• Git-docs

Windows Python 3.6, VS2017 and the case of the missing basetsd.h

So you decided to join the Python 3 bandwagon on Windows with your new shiny Visual Studio. And then you start work on a fresh virtual environment and then you do a pip install or something. Sometimes, you'll end up with a compile error just like this:

WTF is this missing 'basetsd.h' file

You'll google this and end up with one of many stackoverflow topics stating you must install Visual Studio 2015 or some SDK.

Fuck that and save your time. You can do better because a programmer named Dimitri Janczak found a solution that works and I can confirm it works. 

The solution is as follows:

1. Install or update setuptools (> 34.0). Do note that setuptools is sort of tied with your version of Pip. So you might have to do: python -m pip install -U pip setuptools

2. The next step is to pick the correct command-line environment. What?! I know right, Visual Studio comes with multiple command prompts. 

Select the appropriate Native Tools Command prompt. Navigate to the project folder. Activate the virtual environment and do what you did last before you encountered the compile error.

If you goes to plan then you should get a successful result. 

Flask on Windows and the case of the Pip install failing

At work I have a Mac and back home I have a Linux (Fedora) and Windows 10 machines. I started a Flask project on the Mac and then put it in a Github repo which I then cloned on my Windows 10 machine. And this is where I encountered the problem of pip failing to install the modules. I tried two variations to the call:

1. > pip install -r requirements.txt
2. > python -m pip install -r requirements.txt

This is with Python 3 and both will spit out a "Permission Denied" error relating to your appdata/Temp folder. The contents of my requirements.txt is pretty tame.

Here's a partial.

Flask==0.11.1
Flask-RESTful==0.3.5
Flask-Webpack==0.1.0
get==0.0.0
itsdangerous==0.24

It fails right on the get module. It say something like it can access the temp folder on your Windows machine which put me on a wild goose chase because the error isn't fucking related to permissions. I figured out the fix after watching the temp folder while pip attempted to install the modules.

Apparently, the Windows pip version does something really weird with the setup.py and/or egginfo where it attempts a build on a empty setup folder for the get module causing the error. The fix is to just change the version. In this case for get==0.0.0 to get==0.0.20.

The pip install command should go off with no problems now.

WTF Win8! No adhoc networking? No wonder it sucks.

This is more rant than developer talk.

A few nights ago, my laptop's hard disk was failing and I had to replace it and while I was at it I decided to update to Windows 8. I thought why the hell not? I did have a valid key and who know I might like it. I WAS SOOO FUCKING WRONG!!

The install process was painless, then there's a lot of eye candy and then it started to go downhill from there. First is that ribbon, strip on the right side, what is that all about? And why doesn't have to pop out ONLY WHEN I put my mouse pointer to the TOP RIGHT corner of the screen? And if I can customize it, WHERE THE FUCK is the customization screen so I can change it's behavior.

Second, WHY THE FUCK NO wireless adhoc connections? Yes, I found the couple of blogs telling how it can be done but involves opening the command line and THAT IS BULLSHIT! And if you make the argument of using third-party software like Connectify, FUCK YOU!

There's a third one, fourth one, fifth one but I don't want to talk about BECAUSE REASONS!

The only bright spot for me is that my laptop boots up and shuts down faster and my Visual Studio runs better. Almost everything else was a disaster.

Make up your mind! - How an error made me "ang amp @#$@#!"

I admit that I don't really use Microsoft software but that doesn't say they don't put out some fantastic stuff. So I decided to check out Microsoft's Expression Web 4. EW4 is for web design, also lucky for me the university that I am working for had an academic alliance license for it. Woohoo!

It should have been a straight-forward procedure installing EW4. But it had me to jump a couple of hoops.

First, I had to download the .NET 4 framework. Fine. OK. No problem. So, I got the web installer and ran it. Being a web installer, I had to wait for it to download the whole thing.

I also got Silverlight. What fun would I have with I couldn't run all those cool demos from EW4 without Silverlight. BTW, getting Silverlight should be pie if you are running Windows 7.

With everything set, ran the app and I then ran into this error:

I don't know about you but the error confused me. Take a closer look: "is either not designed to run on Windows..." Not designed to run on Windows? Are you sure?

Anyway, I did fix this error...eventually.