Hey, you there. Are you starting up with Scrapy and web-crawling, unable to install Scrapy on your Ubuntu distribution. Let’s get you sorted quickly.
Error:
compilation terminated. error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 ---------------------------------------- ERROR: Failed building wheel for Twisted Running setup.py clean for Twisted Failed to build Twisted Installing collected packages: Twisted, Scrapy Running setup.py install for Twisted ... error
This happens due one of the missing dependencies of python-dev, so let’s take the installation from the top.
I am running this in my XUbuntu 18.04, which means it will run it almost any Ubuntu flavor. We will be referring these installation instructions line by line, but since you have trouble reading docs then I feel inclined to copy and paste for you. First up, install this
sudo apt-get install python-dev python-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev
These cover up the dependencies for LXML as well as cryptography, next up
sudo apt-get install python3 python3-dev
Make sure this completes without errors, move on towards making a virtual environment. Read more about them.
~$ sudo pip install virtualenv
~$ mkdir temp-project
~$ cd temp-project
~$ python3 -m venv .venv
~$ source .venv/bin/activate
~$ pip install scrapy
~$ scrapy version
Scrapy 1.6.0
If the output looks something like this, then congratulations you have learnt 2 things today. That you need to read documentation more carefully, and not just glane through it. Because all of these instruction were already mentioned in the Scrapy Docs. The second thing, being Web Scraping for which you have just taken your first step.
Allow me to take your leave now, and always remember to live in the mix.