Tag Archives: Python

Integrate Python and Eclipse IDE

There are two main ways you can work with Python: through the command line or through an IDE. I’ve chosen the Eclipse IDE.

  1. Eclipse requires Java Virtual Machine (JVM) – Download and install the Java Developer Kit.
  2. Download and install the 32-bit Kepler version of Eclipse.
  3. Install plug-ins to integrate Eclipse and Python:
    • Mylyn:
      1. Help -> Install New Software
      2. Beside “Work with” – Add “Mylyn” – “http://download.eclipse.org/mylyn/releases/latest”
      3. Select All and follow prompts to install – restart Eclipse
    • Pydev:
      1. Help -> Install New Software
      2. Beside “Work with” – Add “Pydev” – “http://pydev.org/updatest”
      3. Select All and follow prompts to install – restart Eclipse
  4. Configure Pydev – within Eclipse, select Window -> Preferences -> Pydev -> Interpreters -> Python Interpreter :: Enter c:\Python34\python.exe in top panel and click way through to set up.

How To Set Up Python 3.3.3 on CentOS 6.4


When you get started with CentOS, however, you will be surprised to see that Python still is at version 2.6 and it should not be used by deployed applications regardless as it is reserved for system’s use.

We are going to talk about downloading and setting up Python 3.3.3 without breaking the system’s default. It is rather important to not to get involved with that as critical system tools such as YUM depend on it. Furthermore, we will install two popular must-have Python companions pip and virtualenv.

After we are done, you will be able to simultaneously use either versions of Python on your CentOS 6.4 VPS, create and use virtual environments and finally, download and manage Python packages for each version.

Preparing The System and Installing Python

The installation of Python on CentOS consists of a few (simple) stages, starting with updating the system and followed by actually getting Python and proceeding with the set up process.

Remember: You can see all available releases of Python by checking out the Releases page.

Updating the Default CentOS Applications

Before we begin with the installation, let’s make sure to update the default system applications to have the latest versions available.

yum update

Preparing The System for Development Installations

CentOS distributions are lean – perhaps, a little too lean – meaning they do not come with many of the popular applications and tools that you are likely to need.

This is an intentional design choice. For our installations, however, we are going to need some libraries and tools (i.e. development [related] tools) not shipped by default. Therefore, we need to get them downloaded and installed before we continue.

The recommended of doing this is simply downloading a bunch of tools by a single command with yum software groups.

YUM Software Groups consist of bunch of commonly used tools (applications) bundled together, ready for download all at the same time via execution of a single command and stating a group name. Using YUM, you can even download multiple groups together.

How to Install Development Tools using YUM on CentOS

yum groupinstall -y 'development tools'
yum install -y zlib-dev openssl-devel sqlite-devel bzip2-devel

Downloading, Building and Installing Python

Let’s begin with retrieving the (compressed) archive containing Python source code.

wget http://www.python.org/ftp/python/3.3.3/Python-3.3.3.tar.xz

Extracting the Compressed Source Archive

# Let's decode (-d) the XZ encoded tar archive:
xz -d Python-3.3.3.tar.xz

# Now we can perform the extraction:
tar -xvf Python-3.3.3.tar

Configuring and Installation

Before building the source, we need to make sure that all the dependencies are there and prepare the environment. This is achieved automatically by using ./configure to handle the task for us.

# Enter the file directory:
cd Python-3.3.3

# Start the configuration (setting the installation directory)
# By default files are installed in /usr/local.
# You can modify the --prefix to modify it (e.g. for $HOME).
./configure --prefix=/usr/local    

This procedure should execute without any hiccups – as we have downloaded all the necessary tools and applications. When it is complete, we will be ready to move on to the next step: building and installing.

Building and Installing

After configuring everything for the system we are working on, we can continue with building (compiling) the source and installing the application. Normally, one would use “make install”; however, in order not to override system defaults – replacing the Python already used by the system – we will use make altinstall.

# Let's build (compile) the source
# This procedure can take awhile (~a few minutes)

# After building everything:
make altinstall

Example for version 3.3.3:

make && make altinstall   # <--- Two commands joint together

[Optional Step] Adding New Python Installation Location to PATH

Note: If you have followed the instructions using the default settings, you should not have the need to go through this section. However, if you have chosen a different path than /usr/local to install Python, you will need to perform the following to be able to run it without explicitly stating its full [installation] path each time.

Once the installation is complete, we can access the generated binaries (i.e. the Python interpreter for the version we have chosen) only by specifying its full location (path) (e.g. /usr/local/bin/python2.7) – unless of course the path exists already in the PATH variable (i.e. the variable which tells contains information on where to look for files stated).

If you would like to be able to access the newly installed Python interpreter without explicitly telling each and every time where to look for it, its path needs to be appended to PATH variable:

# Example: export PATH="[/path/to/installation]:$PATH"
export PATH="/usr/local/bin:$PATH"

Setting Up Common Python Tools pip and virtualenv

Having installed Python, we can now finalize completing the basics for application production and deployment. For this, we will set up two of the most commonly used tools: pip package manager and virtualenv environment manager.

If you are interested in learning more about these two tools or just quickly refreshing your knowledge, consider reading Common Python Tools: Using virtualenv, Installing with Pip, and Managing Packages.

Installing pip on CentOS Using a New Python Installation

Before installing pip, we need to get its only external dependency – setuptools.

From the article on virtualenv and pip:

It [setuptools] builds on the (standard) functionality of Python’s distribution utilities toolset called distutils. Given that distils is provided by default, all we need left is setuptools.

Execute the following commands to install setuptools:

This will install it for version 2.7.6

# Let's download the installation file using wget:
wget --no-check-certificate https://pypi.python.org/packages/source/s/setuptools/setuptools-1.4.2.tar.gz

# Extract the files from the archive:
tar -xvf setuptools-1.4.2.tar.gz

# Enter the extracted directory:
cd setuptools-1.4.2

# Install setuptools using the Python we've installed (2.7.6)
python2.7 setup.py install

Installing pip itself is a very simple process afterwards. We will make use of the instructions from the article mentioned above to have it downloaded and installed automatically and securely using cURL library.

Note: To learn more about cURL, you can refer to the section explaining it here.

Let’s download the setup files for pip and have Python (2.7) install it:

This will install it for version 2.7.6

curl https://raw.github.com/pypa/pip/master/contrib/get-pip.py | python2.7 -

To learn about how to use pip, please refer to the article on Common Python Tools: Using virtualenv, Installing with Pip, and Managing Packages.

Installing virtualenv on CentOS Using a New Python Installation

Now that we have pip the package manager ready, getting virtualenv on the system is a breeze.

Run the following command to download and install virtualenv:

pip install virtualenv

To learn about how to use virtualenv, please refer to the article on Common Python Tools: Using virtualenv, Installing with Pip, and Managing Packages.

Submitted by: O.S. Tezer

Download search queries data using Python – 20111222


For all the developers who have expressed interest in getting programmatic access to the search queries data for their sites in Webmaster Tools, we’ve got some good news. You can now get access to your search queries data in CSV format using a open source Python script from thewebmaster-tools-downloads project. Search queries data is not currently available via the Webmaster Tools API, which has been a common API user request that we’re considering for the next API update. For those of you who need access to search queries data right now, let’s look at an example of how the search queries downloader Python script can be used to download your search queries data and upload it to a Google Spreadsheet in Google Docs.

Example usage of the search queries downloader Python script
1) If Python is not already installed on your machine, download and install Python.
2) Download and install the Google Data APIs Python Client Library.
3) Create a folder and add the downloader.py script to the newly created folder.
4) Copy the example-create-spreadsheet.py script to the same folder as downloader.py and edit it to replace the example values for “website,” “email” and “password” with valid values for your Webmaster Tools verified site.
5) Open a Terminal window and run the example-create-spreadsheet.py script by entering “python example-create-spreadsheet.py” at the Terminal window command line:

python example-create-spreadsheet.py

6) Visit Google Docs to see a new spreadsheet containing your search queries data.

If you just want to download your search queries data in a .csv file without uploading the data to a Google spreadsheet use example-simple-download.py instead of example-create-spreadsheet.py in the example above.

You could easily configure these scripts to be run daily or monthly to archive and view your search queries data across larger date ranges than the current one month of data that is available in Webmaster Tools, for example, by setting up a cron job or using Windows Task Scheduler.

An important point to note is that this script example includes user name and password credentials within the script itself. If you plan to run this in a production environment you should follow security best practices like using encrypted user credentials retrieved from a secure data storage source. The script itself uses HTTPS to communicate with the API to protect these credentials.

Take a look at the search queries downloader script and start using search queries data in your own scripts or tools. Let us know if you have questions or feedback in the Webmaster Help Forum.

Written by , Webmaster Trends Analyst

Barry Wise said…

Excellent – been waiting for this. Just curious to know why Google always seems to prefer Python over PHP

DECEMBER 23, 2011 AT 4:25 AMJavier said…

Schedule in Windows 7  http://windows.microsoft.com/en-US/windows7/schedule-a-task

DECEMBER 23, 2011 AT 8:30 AMDan DeVeney said…

Really great addition guys. Thanks! As an FYI to everyone else, I’ve found you can change the selected_downloads in the script from “TOP_QUERIES” to “TOP_PAGES” to pull you top pages report instead.

DECEMBER 23, 2011 AT 11:14 AMFrank said…

For those who like to use this to download the data for all their sites at once, I’ve changed the scripts somewhat: http://www.webkruscht.com/2011/downloading-data-from-google-webmaster-tools

DECEMBER 24, 2011 AT 8:22 AMThomas Hey’l said…This is really a great Christmas present. With a little help of JohnMu I’ve been able to pimp those scripts so their output becomes configurable with dates, regions, query types, pages or queries, etc.
Please add the mapping for TOP_QUERY_CHART and TOP_PAGES_CHART to the JSON list of downloadable files – this would really improve the tool.


DECEMBER 25, 2011 AT 7:32 AMlaborant said…Hi I’m stuck at 4) where example-create-spreadsheet.py is executed. There I get thid output:
Traceback (most recent call last):
File “C:\Python27\Lib\gdata-2.0.15\test2\exa
downloader.DoDownload(website, selected_do
File “C:\Python27\Lib\gdata-2.0.15\test2\dow
sites_json = json.loads(available)
File “C:\Python27\lib\json\__init__.py”, lin
return _default_decoder.decode(s)
File “C:\Python27\lib\json\decoder.py”, line
obj, end = self.raw_decode(s, idx=_w(s, 0)
File “C:\Python27\lib\json\decoder.py”, line
raise ValueError(“No JSON object could be
ValueError: No JSON object could be decoded

The gdata python client works I’ve tested it with the example code from the installation HowTo. I’ve modified email password and website.
Does anyone has an idea?

DECEMBER 31, 2011 AT 10:42 PMSteve said…

If you’re looking for PHP support: I wrote something up here: http://code.google.com/p/php-webmaster-tools-downloads/source/browse/

JANUARY 26, 2012 AT 4:39 AMcarinth said…

I was considering using Google and you associates for web publishing and investing. However since you have now abandoned Democracy and taken the Communist view of Corporate Might is Right by going along will an Illegal bill which promotes regression, monopolization, subjugation and dictatorship. I hope you guys crash and burn for what you have done.

JANUARY 29, 2012 AT 5:18 AMUnknown said…

Seems the script is no more working for External Links and other features. Looks like the _GetDownloadList function returns just 2 or 3 downloadable URLs…

JULY 5, 2012 AT 11:57 AMMichael Stitt said…

Can you point me to the documentation for tweaking these files? For example, This gives the top search queries for the past 30 days, but what if I only want data for the past week? Or yesterday? Thanks for the article!

MARCH 5, 2013 AT 2:15 PMMichael Stitt said…

Nice article! I realize you wrote it over 2 years ago but the scripts still work. Can you point me to any documentation showing how to change the date range? For example, what if I only want top queries for the past week, or day?

JULY 3, 2013 AT 2:30 AMAaron said…

I was able to customize the date range by editing downloader.py and changing “url = self._GetFullUrl(path)” to “url = self._GetFullUrl(path + ‘&prop=WEB&region&db=20130801&de=20130807&more=true&format=csv’)” where “db” is the beginning date” and “de” is the end date.

SEPTEMBER 27, 2013 AT 3:49 PMMatt said…

got it to work and it works like a charm. can anyone point me in the right direction on how to setup it up so that it creates automatic monthly exports?

OCTOBER 1, 2013 AT 5:13 AMClayton Sheppard said…

Does this work with python33?

OCTOBER 17, 2013 AT 11:09 AMGoogle Webmaster Central said…

Hi everyone,

Since over a year has passed since we published this post, we’re closing the comments to help us focus on the work ahead. If you still have a question or comment you’d like to discuss, free to visit and/or post your topic in our Webmaster Central Help Forum.

Thanks and take care,
The Webmaster Central Team