ERROR: Failed building wheel for Twisted, How to install Scrapy in Ubuntu 18.04

Hey, you there. Are you starting up with Scrapy and web-crawling, unable to install Scrapy on your Ubuntu distribution. Let’s get you sorted quickly.

Error:

compilation terminated.
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for Twisted
  Running setup.py clean for Twisted
Failed to build Twisted
Installing collected packages: Twisted, Scrapy
  Running setup.py install for Twisted ... error

This happens due one of the missing dependencies of python-dev, so let’s take the installation from the top.

I am running this in my XUbuntu 18.04, which means it will run it almost any Ubuntu flavor. We will be referring these installation instructions line by line, but since you have trouble reading docs then I feel inclined to copy and paste for you. First up, install this

sudo apt-get install python-dev python-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev

These cover up the dependencies for LXML as well as cryptography, next up

sudo apt-get install python3 python3-dev

Make sure this completes without errors, move on towards making a virtual environment. Read more about them.

~$ sudo pip install virtualenv
~$ mkdir temp-project
~$ cd temp-project
~$ python3 -m venv .venv
~$ source .venv/bin/activate
~$ pip install scrapy
~$ scrapy version
Scrapy 1.6.0

If the output looks something like this, then congratulations you have learnt 2 things today. That you need to read documentation more carefully, and not just glane through it. Because all of these instruction were already mentioned in the Scrapy Docs. The second thing, being Web Scraping for which you have just taken your first step.

Allow me to take your leave now, and always remember to live in the mix.