Description of web services provided by the Edinburgh GIN

View the Edinbugh GIN software version and release notes.

Web sites that use the INTERMAGNET web service

These web sites provide a user-friendly, interactive interface to the web services described below.

View INTERMAGNET observatory data: /GIN_V1/GINForms2

View INTERMAGNET data statistics: /GIN_V1/GINStatistics

View test observatory data at the Edinburgh GIN: /GIN_V1/GINTest2

HAPI Web Service

INTERMAGNET provides a HAPI (Heliophysics Application Programmer's Interface) web service: /GIN_V1/hapi

INTERMAGNET Web service

The INTERMAGNET web service allows access to INTERMAGNET data in the form of data files in community formats, interoperable formats (such as JSON) and plot files. Metadata and statistics on data provision and data usage are also available. A full description is given here so that other organisations may easily incorporate these services into their own products. The type of information that the services provide is controlled by parameters that are passed to the service in the URL's query string.

The web service is available from this base URL: /GIN_V1/GINServices. The services support both POST and GET http methods. The 'Request' parameter controls what information will be provided.

Some institutes choose to embargo their data for a short time. Details of the embargo and the ways in which it may be possible to waive the embargo for an observatory's data are described in the section on embargoes.

To help understand how to format URLs to retrieve data from the web service, a URL generation application is available. This application allows you to enter values for the data that you want to retrieve and then to see (and copy) the URL that can be used to retrieve the data.

HTTP status codes

The Intermagnet web services use the following status codes:

The capabilities service is designed to allow remote systems to interrogate the GIN and discover its capabilities. The service returns:

• A list of the observatories known to the GIN.
• A list of valid data types.

The following parameters may be used to control the capabilities request:

Use the Web Service URL Generator to see how to create URLs for this service.

The data service provides geomagnetic data from INTERMAGNET observatories either as data files or as magnetograms. The following parameters may be used to control the data request:

Use the Web Service URL Generator to see how to create URLs for this service.

The data directory service gives precise information on the amount of geomagnetic data available to the user. The following parameters may be used to control the data request:

Use the Web Service URL Generator to see how to create URLs for this service.

The data modification time service allows users to retrieve the modification time of geomagnetic data. Observatories may submit data at any time and may overwrite previously submitted data. This service allows users to discover when data was modified, meaning that users are able to monitor for updates to data. Internally data supplied to the service is grouped by observatory, cadence, data type and day. Modification times can be retrieved for each individual day of data. The following parameters may be used to control the data request:

Use the Web Service URL Generator to see how to create URLs for this service.

The data lag time service shows the difference between current time and the time of the most recently recorded sample from an observatory. The lag time is sampled once an hour for each observatory that sends data to the Edinburgh GIN. The lag time service allows operators to plot individual lag time figures for an observatory, also a monthly summary of lag times for all observatories. The following parameters may be used to control the data request:

Use the Web Service URL Generator to see how to create URLs for this service.

Providing bulk data downloads is a challenge for web service providers. Allowing too much data to be downloaded in one go can impact the performance of a web server. This system solves the problem by creating download scripts, which are retrieved by the user then run (on either UNIX or Windows or in Python) to download the data required by the user. The script is designed in such a way as to ease the burden on the web server. Because of this, no restrictions are placed on the amount of data that the user can download. Parameters fully control the data that the user requires - the download script will be created to allow retrieval of the data as specified in the parameter set. Further details are available on the download page of the interactive form.

Use the Web Service URL Generator to see how to create URLs for this service.

Statistics describing the amount of data received from each INTERMAGNET observatory are created on the 5th day of each month for the preceeding month. These statistics are emailed to all observatories and also archived. This service gives access to the archive. The archive consists of plain text files. Data are not available in any other format.

Each item in the archive has two dates:

1. The date that the statistics refer to - a year and a month
2. The date that the statistics were created on - a year, month and day. Typically this will be the 5th day of the month following the date that the statistics refer to, but it is possible that the statistics could be created on a different day, or that there could be multiple creation dates for the same statistics date.

Use the Web Service URL Generator to see how to create URLs for this service.

Each time a data file is downloaded from the Intermagnet web or ftp service, a record is made of the download. At the end of each day the total number of downloads for each observatory are recorded. At the end of the month these daily statistics are amalgamated into a monthly summary. These daily and monthly statistics on data downloads from the web and ftp servers are available through the data usage service.

Use the Web Service URL Generator to see how to create URLs for this service.

Embargoing of data and plots

Data providers can specify an embargo period, during which data submitted to INTERMAGNET will not be distributed to users. The embargo period is specified in hours and applies to each observatory individually. Requests for data that is more recent than the embargo period will fail, showing the data as missing.

Organisations can apply to INTERMAGNET for a username/password to access this web service without the embargo being applied to their requests. Valid reasons for such a request include providing a service to the community that has been agreed with data providers, or an arrangement between a user and a data provider to supply data more rapidly than the embargo would allow.

To use a username provided for this purpose, there are two options:

1. Send your http request using Basic authentication
2. Append two parameters to each request sent to the web service:
   • "username" - the username
   • "password" - the password

The web sites at /GIN_V1/GINForms2 and /GIN_V1/GINTest2 provide a "Login" button where you can use the same username to get interactive access to the web service data using the permissions set up for any user credentials that you have access to.

To set up an embargo for an observatory or to apply for a username and password contact us using the "comments" link at the end of this page.

Test and production observatories

Associated with each username is a flag which specifies whether to allow access to production observatories, test observatories or both. This flag is set up by the administrator at the time the username is created. The flag overrides the function of the 'testObsys' query string parameter.

Best available data

Observatories record the geomagnetic field using a combination of automated computer measurements (high resolution variations) and manual measurements (low frequency "baseline" measurements). Manual measurements, typically taken with periods ranging from from days to weeks, are collected over time and a "best fit" baseline calculated in order to combine the manual time series with the automatically recorded computer measurements. New manual measurements may affect previous calculations of the baseline. For this reason a number of "publication states" (or "data types") have been defined to describe the state a data set is in, from initial recording to final publication. During the publication process data may also be edited to correct spikes or fill in gaps from an alternative recording system. The names of the publication states are:

1. "reported" (or "variometer") data - newly recorded data without a baseline. This is a historic data type that data providers have been asked to stop sending.
2. "adjusted" (or "provisional") data - data with a provisional baseline. The data may also have been edited. There is no guarantee how close the data values will be to the final values.
3. "quasi-definitive" data - data with a provisional baseline where the accuracy is within a defined limit with respect to the final value.
4. "definitive" data - data where all adjustments have been completed and the data provider has declared that no further changes will be made.

Calculation of "best available" data

The "best available" publication state is intended to provide you with the most "finished" data that the data provider has supplied. If definitive data is available for the observatory and time period requested, it will be provided. If not quasi-definitive data will be searched for, followed by adjusted data and finally reported data.

Data in the Edinburgh GIN is stored in files containing a day of data. When determining which type of data is available, the GIN searches to see which individual "day files" exist. For data requests spanning multiple days, the GIN will search the full span of days requested and if any data is present in this span for a particular publication state, then all the data returned will be for that publication state. Publication states will not be mixed in the data returned from a single request. This is true for data requested in all the formats that the service provides, both plots and data files.

Because the data service returns data sets that only contain a single publication state, the data directory and data modification time services also follow this behaviour. For a single request spanning multiple days, these services will return lists that refer to a single publication state across the entire span of time requested. If no data is found for any publication state within the requested time window, the returned information may show the "publication state" or "data type" as "none" (to indicate that no data was found). If multiple observatories are specified in these requests, each observatory's data may be returned with a different publication state.

If information is sought about the availability or modification time of the best available data for each day from an observatory, for a time span exceeding one day, there are two ways to approach this. The first is to split the request into multiple days, requesting the best available data for each day. The second is to make four requests, one for each data type (reported, adjusted, quasi-definite and definitive), each for the entire time span required, and to analyse the returned data yourself to determine the best available data for each day.

Terms of use: https://www.bgs.ac.uk/legal-and-policy/terms-of-use/

Cookie policy: https://www.bgs.ac.uk/legal-and-policy/cookies-policy/

Privacy policy: https://www.bgs.ac.uk/legal-and-policy/privacy/

Comments: Mail the GIN manager