Keeping a Forked Repo up to date

Keep your fork up to date by tracking the original "upstream" repo that you forked. To do this, you'll need to add a remote:

# Add 'upstream' repo to list of remotes
git remote add upstream https://github.com/UPSTREAM-USER/ORIGINAL-PROJECT.git

# Verify the new remote named 'upstream'
git remote -v

Whenever you want to update your fork with the latest upstream changes, you'll need to first fetch the upstream repo's branches and latest commits to bring them into your repository:

# Fetch from upstream remote
git fetch upstream

# View all branches, including those from upstream
git branch -va

Now, checkout your own master branch and merge the upstream repo's master branch:

# Checkout your master branch and merge upstream
git checkout master
git merge upstream/master

If there are no unique commits on the local master branch, git will simply perform a fast-forward. Now, your local master branch is up-to-date with everything modified upstream.

Handling non-well-formed HTML in Scrapy with BeautifulSoup

With Scrapy, we can deal with non-well-formed HTML is many ways. This is just one of them.

BeautifulSoup has a pretty nifty feature where it tries to fix bad HTML like replacing missing tags. So if we put BeautifulSoup in the middle then whatever we get from a site is fixed before we parse it with Scrapy.

Fortunately, all we have to do is pip install Alecxe's scrapy-beautifulsoup middleware.

pip install scrapy-beautifulsoup

Then we configure Scrapy to use it from settings.py:

DOWNLOADER_MIDDLEWARES = {
    'scrapy_beautifulsoup.middleware.BeautifulSoupMiddleware': 400
}

BeautifulSoup comes with a default parser named 'html.parse'. We can change it.

BEAUTIFULSOUP_PARSER = "html5lib"  # or BEAUTIFULSOUP_PARSER = "lxml"

HTML5 is the better parser IMO but it has to be installed separately.

pip install html5lib

Gracefully dealing with different SSH keys for different domains or accounts

This problem often happens when you mix your personal ssh keys along with your company's and you work off a single workstation or laptop.

Even if the user and host are the same, they can still be distinguished in the ~/.ssh/config.

Host gitlab.com
  HostName git.company.com
  User git
  IdentityFile /home/whoever/.ssh/id_rsa.alice
  IdentitiesOnly yes

Host kitlab.com
  HostName git.company.com
  User git
  IdentityFile /home/whoever/.ssh/id_dsa.bob
  IdentitiesOnly yes

Then you can use gitlab.com and kitlab.com instead of the hostname in your git remote.

git remote add g-origin git@gitlab.com:whatever.git
git remote add k-origin git@kitlab.com:whatever.git

You probably want to include the option IdentitiesOnly yes to prevent the use of default ids.

Ref: https://blog.developer.atlassian.com/different-ssh-keys-multiple-bitbucket-accounts/