Scripted Access (UWS)

This web application is an instance of Daiquiri, which comes with a built-in UWS interface according to the specifications from the International Virtual Observatory Alliance (IVOA). The basic usage for UWS is described at Daiquiri – UWS. This allows to write scripts from the command line for sending jobs and retrieving results, checking the job’s status etc.

The URL for Gaia@AIP’s UWS service is:
https://gaia.aip.de/uws/query

Requesting this url returns a list of jobs, with their jobids and the phase (status).

There are many ways to send requests to a URL. Below we show an example how to do it with httpie or with a UWS-client written in Python.

Example: UWS job with httpie

Here is a first example for sending a UWS job from the command line and retrieving the results using httpie. See next section for an example with our UWS-client written in Python.

Requirements:

  • httpie for sending commands to the server (download from Github: httpie)
  • user account (register at Registration, accounts from the old MultiDark web app should work as well)

Please replace <username> and <password> in the commands below with your own credentials.

    • Create a job for retrieving the hundred stars with the highest proper motion:
      http --auth <username>:<password> --form --follow POST https://gaia.aip.de/uws/query query="SELECT tycho2_id,SQRT(POW(pmra, 2) + POW(pmdec, 2)) / 1000 as pm FROM GDR1.tgas_source WHERE tycho2_id IS NOT NULL ORDER BY pm DESC LIMIT 100" table="mytest"
      

      Results will be written to table “mytest” in your private database. If you skip the table name, the current timestamp will be used.

      You can also specify the type of queue you will need by adding queue=”short” or queue=”long”:

      http --auth <username>:<password> --form --follow POST https:/gaia.aip.de/uws/query query="SELECT tycho2_id,SQRT(POW(pmra, 2) + POW(pmdec, 2)) / 1000 as pm FROM GDR1.tgas_source WHERE tycho2_id IS NOT NULL ORDER BY pm DESC LIMIT 100" table="mytest" queue="long"
      

      By default, the short queue is used.

      The output that is returned contains a jobid. Use this as a reference to your job from now on. In this example, we use jobid=353132934652699. Please remember to change this number to your own jobid in the following commands.

    • Start the job:
      http --auth <username>:<password> --form --follow POST https://gaia.aip.de/uws/query/353132934652699/phase phase=run
    • Check the status:
      http --auth <username>:<password> --print b GET https://gaia.aip.de/uws/query/353132934652699/phase
      
    • Get a list of jobs (only lists the latest ~ 1000 jobs to avoid timeout and memory problems):
      http --auth <username>:<password> --print b GET https://gaia.aip.de/uws/query
      
    • If the job status is “COMPLETED”, you can get information about the results
      http --auth <username>:<password> GET https://gaia.aip.de/uws/query/353132934652699/results
      

      Look in the output of this command for the tag <uws:result ...> with the url of the result id. In our example, this is: https://gaia.aip.de/query/download/stream/table/mytest/format/csv

    • Download your results directly to the terminal:
      http --auth <username>:<password> GET https://gaia.aip.de/query/download/stream/table/mytest/format/csv
    • Alternatively download results to a file:
      http --auth <username>:<password> --download GET  https://gaia.aip.de/query/download/stream/table/mytest/format/csv

If you want to specify a different output file name, add the option --output </path/file>
to the command above.

  • Delete a job, if you don’t need it anymore:
    http --auth <username>:<password> --follow DELETE https://gaia.aip.de/uws/query/353132934652699
    

    This will delete the results from the server. The job information itself may still be archived on the server.

These steps could be combined in a Bash or Python script and adjusted to your needs.

UWS client

There’s now also a Python UWS client available, check it out from GitHub – uws-client. You can use it as a UWS library or install it and use the uws command to create and start jobs, list and delete them etc.

Here is the same example as above, but using the UWS-client:

  • Get a nicely formatted list of the latest jobs (limited to ~ 1000 jobs to avoid timeout and memory problems):
    uws --host https://gaia.aip.de/uws/query --user <username> --password <password> list
    

    Note that the UWS-client also allows you to filter the jobs by phase by adding the corresponding optional argument (–pending, –queued, –executing, –completed, –error, –aborted). E.g. for listing only pending jobs:

    uws --host https://gaia.aip.de/uws/query --user <username> --password <password> list --pending
    
  • Create a job:
    uws --host https://gaia.aip.de/uws/query --user <username> --password <password> job new queue="long" query="SELECT tycho2_id,SQRT(POW(pmra, 2) + POW(pmdec, 2)) / 1000 as pm FROM GDR1.tgas_source WHERE tycho2_id IS NOT NULL ORDER BY pm DESC LIMIT 100" table="mytest"
    

    Results will be written to table “mytest” in your private database. If you skip the table name, the current timestamp will be used.
    The output that is returned contains a jobid. Use this as a reference to your job from now on. In the example, we use jobid=353132934652699. Please remember to change this number to your own jobid in the following commands.

  • Submit the job with id 353132934652699:
    uws --host https://gaia.aip.de/uws/query --user <username> --password <password> job run 353132934652699
  • Create and submit a job in one step:
    This can be achieved by adding the option -r or --run when creating a job.
uws --host https://gaia.aip.de/uws/query --user <username> --password <password> job new queue="long" query="SELECT tycho2_id,SQRT(POW(pmra, 2) + POW(pmdec, 2)) / 1000 as pm FROM GDR1.tgas_source WHERE tycho2_id IS NOT NULL ORDER BY pm DESC LIMIT 100" table="mytest" --run
  • Check the job’s parameters:
    uws --host https://gaia.aip.de/uws/query --user <username> --password <password> job show 353132934652699
  • If the job status is “COMPLETED”, you can download the results using:
    uws --host https://gaia.aip.deuws/query --user <username> --password <password> job results 353132934652699 csv

    The results will be stored in a csv-file in the directory from which uws was called.

  • Abort or delete a job, if you want to stop it or if you don’t need it anymore:
    uws --host https://gaia.aip.de/uws/query --user <username> --password <password> job abort 353132934652699
    uws --host https://gaia.aip.de/uws/query --user <username> --password <password> job delete 353132934652699

    This will delete the results from the server. The job information itself may still be archived on the server.

See the README.md of the uws-client to learn more about it.

Scripts for download – available for CosmoSim

If you are looking for ready-made example shell-scripts for posting, downloading or deleting many queries automatically, then read this blog post – shell scripts or look at the cosmosim shell scripts at GitHub for the most recent versions.

Proudly powered by Daiquiri
©2016 GAIA@AIPImprint and Data Protection Statement