Downloading And Uploading Data on Google App Engine Using Java Application

Table of contents

Reading Time: 3 minutes

We need to download data as a CSV from the app engine data store. This would help us get the data back from the app engine, just in case you want to move away or update from the data store.

We’ll go through a step by step example for using this tool with a Java application.

1. Download Google App Engine SDK for Python from here.
Unzip this file. Place it in your path. For Linux users.

PATH="/path/to/where/you/unzipped/appengine:${PATH}"
export PATH

2. Now you have to set remote servlet with our Java application with which would be able to communicate with the app engine remotely. The following needs to be set up in your web.xml

<servlet>
   <servlet-name>RemoteApi</servlet-name>
   <servlet-class>com.google.apphosting.utils.remoteapi.RemoteApiServlet</servlet-class>
</servlet>

<servlet-mapping>
   <servlet-name>RemoteApi</servlet-name>
   <url-pattern>/remote_api</url-pattern>
</servlet-mapping>

Deploy your application to the app engine.

3. Now we are going to use appcfg.py to generate a configuration file:

appcfg.py create_bulkloader_config --url=http://APPID.appspot.com/remote_api --application=APPID --filename=config.yml

Once you execute this command, after the necessary authentication the following file is generated

# Autogenerated bulkloader.yaml file.
# You must edit this file before using it. TODO: Remove this line when done.
# At a minimum address the items marked with TODO:
#  * Fill in connector and connector_options
#  * Review the property_map.
#    - Ensure the 'external_name' matches the name of your CSV column,
#      XML tag, etc.
#    - Check that __key__ property is what you want. Its value will become
#      the key name on import, and on export the value will be the Key
#      object.  If you would like automatic key generation on import and
#      omitting the key on export, you can remove the entire __key__
#      property from the property map.

# If you have module(s) with your model classes, add them here. Also
# change the kind properties to model_class.
python_preamble:
- import: base64
- import: re
- import: google.appengine.ext.bulkload.transform
- import: google.appengine.ext.bulkload.bulkloader_wizard
- import: google.appengine.ext.db
- import: google.appengine.api.datastore
- import: google.appengine.api.users

transformers:

- kind: Mobile
  connector: csv
  encoding: utf-8
  
property_map:
    - property: __key__
      external_name: key
      export_transform: transform.key_id_or_name_as_string

    - property: model
      external_name: model
      # Type: String Stats: 1 properties of this type in this kind.

    - property: name
      external_name: name
      # Type: String Stats: 1 properties of this type in this kind.

    - property: timestamp
      external_name: timestamp
      # Type: Date/Time Stats: 1 properties of this type in this kind.
      import_transform: transform.import_date_time('%Y-%m-%dT%H:%M:%S')
      export_transform: transform.export_date_time('%Y-%m-%dT%H:%M:%S')

    - property: year
      external_name: year
      # Type: String Stats: 1 properties of this type in this kind.

4. It would create configuration for all entities. Replace these lines:

connector: # TODO: Choose a connector here: csv, simplexml, etc...
connector_options:
  # TODO: Add connector options here--these are specific to each connector.

with these

connector: csv
connector_options: encoding: utf-8

5. Now the next step is for downloading data.

appcfg.py download_data --config_file=config.yml --filename=data.csv --kind=<kind> --url=http://APPID.appspot.com/remote_api --application=APPID

It will asked to provide email and password credentials.

To learn more, see http://code.google.com/appengine/kb/general.html#rpcssl .
Please enter login credentials for java.latest.bootcamp-demo.appspot.com
Email: YOUR EMAIL
Password for YOUR_EMAIL:
[INFO    ] Downloading kinds: ['kinds']
.[INFO    ] Greeting: No descending index on __key__, performing serial download
.
[INFO    ] Have 1 entities, 0 previously transferred
[INFO    ] 1 entities (11304 bytes) transferred in 10.5 seconds

There’s now a CSV file named data.csv in our directory, with autogenerated bulkloader files.
Open CSV file in text mode. You can make changes in CSV file by using text editor.

6. Now for uploading data.

appcfg.py upload_data --config_file=config.yml --filename=data.csv --url=http://APPID.appspot.com/remote_api --application=APPID --kind=<kind>

To learn more, see http://code.google.com/appengine/kb/general.html#rpcssl .
Please enter login credentials for APPID.appspot.com
Email: your.email@gmail.com
Password for your.email@gmail.com:
[INFO    ] Starting import; maximum 10 entities per post
.
[INFO    ] 2 entites total, 0 previously transferred
[INFO    ] 2 entities (5394 bytes) transferred in 8.6 seconds
[INFO    ] All entities successfully transferred

The bulkloader took care of generating a mostly-finished configuration file for us, in a format that ensures the data we download can be re-uploaded again without loss of fidelity.