English
Français
Español
Português
Русский
中文
العَرَبِيَّة‎
UNOPS
IATI Data Store Request for proposal

Reference: RFP/2018/5113
Beneficiary country(ies): Multiple destinations (see 'Countries' tab below)
Registration level: Basic
Published on: 05-Jul-2018
Deadline on: 07-Aug-2018 23:59 0.00

Description
Tender description: IATI Data Store

-----

IMPORTANT NOTE: Interested vendors must respond to this tender using the UNOPS eSourcing system, via the UNGM portal. In order to access the full UNOPS tender details, request clarifications on the tender, and submit a vendor response to a tender using the system, vendors need to be registered as a UNOPS vendor at the UNGM portal and be logged into UNGM. For guidance on how to register on UNGM and submit responses to UNOPS tenders in the UNOPS eSourcing system, please refer to the user guide and other resources available at: https://esourcing.unops.org/#/Help/Guides

Documents
Contacts
Links
Countries
UNSPSC
Revisions
This tender has been posted through the UNOPS eSourcing system. Cet avis a été publié au moyen du système eSourcing de l'UNOPS. Esta licitación ha sido publicada usando el sistema eSourcing de UNOPS.
Countries/territories
Afghanistan
Albania
Algeria
American Samoa
Andorra
Angola
Anguilla
Antigua and Barbuda
Argentina
Armenia
Aruba
Australia
Austria
Azerbaijan
Bahamas
Bahrain
Bangladesh
Barbados
Belarus
Belgium
Belize
Benin
Bermuda
Bhutan
Bolivia
Bosnia and Herzegovina
Botswana
Brazil
British Indian Ocean Territory
Brunei Darussalam
Bulgaria
Burkina Faso
Burundi
Cambodia
Cameroon
Canada
Cape Verde
Cayman Islands
Central African Republic
Chad
Chile
China
Christmas Island
Colombia
Comoros
Congo
Cook Islands
Costa Rica
Côte d'Ivoire
Croatia
Cuba
Cyprus
Czech Republic
Congo, The Democratic Republic of the
Denmark
Djibouti
Dominica
Dominican Republic
Timor-Leste
Ecuador
Egypt
El Salvador
Equatorial Guinea
Eritrea
Estonia
Ethiopia
Falkland Islands (Malvinas) - A dispute exists between the Governments of Argentina and the United Kingdom of Great Britain and Northern Ireland concerning sovereignty over the Falkland Islands (Malvinas)
Faroe Islands
Fiji
Finland
France
French Guiana
French Polynesia
Gabon
Gambia
Georgia
Germany
Ghana
Gibraltar
Greece
Greenland
Grenada
Guadeloupe
Guam
Guatemala
Guinea
Guinea-Bissau
Guyana
Haiti
Honduras
Hong Kong, Special Administrative Region of China
Hungary
Iceland
India
Indonesia
Iran, Islamic Republic of
Iraq
Ireland
Israel
Italy
Jamaica
Japan
Jordan
Kazakhstan
Kenya
Kiribati
Korea, Democratic People's Republic of
Korea, Republic of
Kuwait
Kyrgyzstan
Lao People's Democratic Republic
Latvia
Lebanon
Lesotho
Liberia
Libyan Arab Jamahiriya
Liechtenstein
Lithuania
Luxembourg
Macao, Special Administrative Region of China
Macedonia, the former Yugoslav Republic
Madagascar
Malawi
Malaysia
Maldives
Mali
Malta
Marshall Islands
Martinique
Mauritania
Mauritius
Mayotte
Mexico
Micronesia, Federated States of
Moldova, Republic of
Monaco
Mongolia
Montserrat
Morocco
Mozambique
Myanmar
Namibia
Nauru
Nepal
Netherlands
Netherlands Antilles
New Caledonia
New Zealand
Nicaragua
Niger
Nigeria
Niue
Norway
Oman
Pakistan
Palau
Panama
Papua New Guinea
Paraguay
Peru
Philippines
Pitcairn
Poland
Portugal
Puerto Rico
Qatar
Reunion
Romania
Russian Federation
Rwanda
Saint Helena
Saint Kitts and Nevis
Saint Lucia
Saint Pierre and Miquelon
Saint Vincent and the Grenadines
Samoa
San Marino
Sao Tome and Principe
Saudi Arabia
Senegal
Seychelles
Sierra Leone
Singapore
Slovakia
Slovenia
Solomon Islands
Somalia
South Africa
Spain
Sri Lanka
Sudan
Suriname
Svalbard and Jan Mayen
Eswatini
Sweden
Switzerland
Syrian Arab Republic
Taiwan, Province of China
Tajikistan
Thailand
Togo
Tokelau
Tonga
Trinidad and Tobago
Tunisia
Turkey
Turkmenistan
Turks and Caicos Islands
Tuvalu
Uganda
Ukraine
United Arab Emirates
United Kingdom
Tanzania, United Republic of
United States of America
Uruguay
Uzbekistan
Vanuatu
Venezuela
Viet Nam
Virgin Islands, British
Virgin Islands, U. S.
Wallis and Futuna Islands
Western Sahara
Yemen
Zambia
Zimbabwe
Antarctica
Palestine, State of
Serbia, Republic of
Montenegro
Kosovo
South Sudan
J  -  Services
81000000  -  Engineering and Research and Technology Based Services
81110000  -  Computer services
81112000  -  Data services
81112006  -  Data storage service
New amendment added #1: The tender is hereby amended as follow:- Dead Line for submission has been extended by 2 days, to 7th August 2018 at 23:59 UTC.

Changed/edited on: 03-Aug-2018 13:42
Changed/edited by: webservice@unops.org
New clarification added: Question: IATI Registry - is it safe to say that existing IATI Registry is not undergoing any changes for the new implementation of Data Store? Answer: No changes will take place with regard to the way in which data is accessed. However validation service will be added and the resulting metadata will be available for the datastore to consume Page 3 - Phase -1, under ETL Section:>> Metadata should be maintained that records all validation errors (via Registry) and transformation processes for each activity.ETL validation errors should be reported back to the IATI registry? or should the validation errors be maintained in the Data Store and exposed via API? Answer: The development of a new standardised validation service that is available to all applications and users is under review. The DataStore can either access validation metadata from the Registry or run this standard validation. Users of the Datastore should have access to the validation metadata. In addition the DataStore should maintain its own logs of the ETL process, including any changes made to the data during these processes. Page 3/4 - Phase 2>> Analyst Interface and Publisher Interface --a.       Are there any technology requirements for developing these user interfaces? Web or Desktop tools? Answer: Beyond the overriding preference for Python, no. A desktop tool was not part of this RFP but the benefits of such a service would be considered. b.       If web, does it remain separate or should be integrated to thee main IATI website? Answer: It should sit on its own platform on an iatistandard.org sub-domain, but be seamlessly branded with the main website We found an existing implementation of Data Store on your github, wondering if you can highlight things that you definitely want us to improve (peformance, scalability, relability) or in the new implementation? Answer: Robust and reliable extraction (including accurate handling of deleted activities and datasets that are temporarily unavailable); a database structure that allows proper indexing and filtering of all fields; access to all data in one version of the standard; a sustainable, extensible architecture that makes the handling of upgrades to the standard easy to implement; access to metadata

Changed/edited on: 27-Jul-2018 10:28
Changed/edited by: webservice@unops.org
New clarification added: Question: Dear Team,Please confirm the understanding and answer following. UNOPS already publishes data to IATI and requires the below:Section: Outputs, Page 2:1. A data store that should store all IATI metadata and Activity/Organization’s transaction data fetched from IATI registry using Web APIs?Answer: Yes2. Create a data service accessible to anybody that should query the Data Store and render the results based on parameters/filters passed to it?Answer: Yes3. Data Store should keeping polling IATI registry for new data or metadata and update itself. What frequency you are looking at for these updates/refresh because it certainly does not require real time data streaming?Answer: Data must be refreshed at least once per day, but preferably more frequently.4. Is there an intention to fetch IATI metadata too using data service for IATI publishing?Answer: Yes, the Registry holds metadata on the publisher and on the validation of datasets.5.  How does UNOPS want to store IATI data/metadata in Data Store? In file format or in normalized table schema?Answer: The proposed architecture is part of the proposal.6. Since UNOPS is already IATI compliant, why does it not uses its current data store/data mart/data warehouse for rendering data to public? How UNOPS current data store would be different than the one mentioned/required in this RFP?Answer: The IATI data store is a data service for all data from all publishers to be easily accessible from a single sourceThe IATI data store is a data service for all data from all publishers to be easily accessible from a single source​7. When the data is published in previous version (i.e. 2.01) of IATI standards it is not advisable to transform same data in latest versions (2.03) and render in XML/JSON formats as it may lose its correctness/completeness (due to standard's tag/element changes). Does it still remain a requirement?Answer: Yes. This is an essential requirement to ensure that all data can be compared, irrespective of the version published​ Section: General1. At what frequency UNOPS currently publishes implementation data to IATI?Answer: UNOPS publishes IATI data on a monthly basis​ System Requirements, Page 41. Is it mandatory to develop data service using Python and its code repository should be versioned using Github only? Are you open for other technology stack?Answer: If a long-term sustainable solution is proposed it may be considered. See the previous request for clarification for an explanation

Changed/edited on: 26-Jul-2018 12:45
Changed/edited by: webservice@unops.org
New clarification added: Question: Under the System Requirements > Coding Standards, there is a requirement to use "python 3" and a specific library for XML processing (lxml). Is the python 3 requirement for the overall platform or just for XML processing? If this applies to the overall platform, can organizations propose alternatives, including justification for these alternatives?Answer: The python 3 requirement is for the overall platform. The reason for this is that all core IATI products are written in (or are being converted to) python to standardise the technologies employed by the IATI Secretariat’s Technical Team. An alternative open source codebase could be considered if a very clear case can be made for a long-term, sustainable hosting solution.

Changed/edited on: 11-Jul-2018 10:50
Changed/edited by: webservice@unops.org
New clarification added: The python 3 requirement is for the overall platform. The reason for this is that all core IATI products are written in (or are being converted to) python to standardise the technologies employed by the IATI Secretariat’s Technical Team. An alternative open source codebase could be considered if a very clear case can be made for a long-term, sustainable hosting solution.

Changed/edited on: 11-Jul-2018 10:46
Changed/edited by: webservice@unops.org