Client Datastore API: writing "Text" values

[mitchhentges]

Hello again!

According to the App Engine Entity Property Reference, there's a "Text" data type which allows Strings up to 1MB.

However, with the Client Datastore, the only text-related Value type is StringValue. How can I store a Text value? I was considering Blob, but that's actually its own, separate type in Datastore (according to the Entity Property Reference)

[mziccard]

Hi @mitchhentges, welcome back :)

This is indeed a good question. In the com.google.datastore.v1.Value proto (on which we rely) I see no way of setting a Text value. Setting a String value (using Value.Builder.setStringValue) longer than 1500 bytes fails with:

The value of property "bigString" is longer than 1500 bytes.

@eddavisson any insights on this? Is Text only available in AppEngine?

[mitchhentges]

Here's something interesting: if I save a 3kb text string using the App Engine API as "Text", then I read the entity with that 3kb text string using the Client Datastore API, the Value type is StringValue.

Logically, I cannot turn around and save that entity, because a StringValue must be under 1500 bytes.

[sai-pullabhotla]

I don't see any issues with storing/retrieving Strings that are longer than 1500 bytes. I was able to insert 1MB (1,000,000 ascii characters) just fine. You just have to make sure the property is not indexed when saving. If indexing is turned on, you are limited to 1500 bytes. When I tried to do 1024 * 1024 bytes, I did hit the max limit with the below Exception:

The value of property "name" is longer than 1048487 bytes.

Check out the limits in the official documentation:

https://cloud.google.com/datastore/docs/concepts/limits

[mitchhentges]

@sai-pullabhotla, running the following code throws a DatastoreException "The value of property "bork" is longer than 1500 bytes."

longString is ~11K

val datastore = DatastoreOptions.defaultInstance().service()
val longString = IOUtils.getResource("signature")
val keyFactory = datastore.newKeyFactory().kind("Test").namespace("4998Test")
val entity = FullEntity.builder(keyFactory.newKey()).set("bork", longString).build()
// or val entity = FullEntity.builder(keyFactory.newKey()).set("bork", StringValue(longString)).build()
datastore.put(entity)

1. It looks like, by default, all properties are indexed. How can I "unindex" a property using the Datastore Client API? According to the docs (see BaseEntity.Builder), you can only set() properties, not whether or not they're indexed
2. Can you provide an example of a block of code that will save over 1500 bytes of text?

[sai-pullabhotla]

@mitchhentges,

That's correct, by default all properties are indexed, unless you indicate otherwise. To exclude properties from indexes, you have to use the ValueBuilder. For String type, you would use the StringValue.Builder. Link to JavaDoc below:

http://googlecloudplatform.github.io/google-cloud-java/0.4.0/apidocs/com/google/cloud/datastore/StringValue.Builder.html

Use the excludeFromIndexes(true).

Example -

entityBuilder.set("propertyName", StringValue.builder(hugeString).excludeFromIndexes(true).build());

[mitchhentges]

Thanks champ!

[afornie]

Thanks @sai-pullabhotla , but with your solution it works in the sense that it ends up storing the value (as a String), but there is no way to force it to be a Text type, right? What if there is an appengine app reading these entities and because of the code it uses it will be mandatory that the property is indeed a Text?

[sai-pullabhotla]

@afornie - it is just a terminology difference between the AppEngine and google-cloud APIs. From the Datastore server point of view, String/Text are same. Strings with more than 1500 byes cannot be indexed. So, you should be able to access the data fine from all APIs.

[wguerlich]

@sai-pullabhotla Unfortunately that's not true... they are NOT the same type when read through the AppEngine API.
When reading BLOBs or Text values written through the Cloud Datastore API it will depend on the size of the value written how they will appear in AppEngine. If they're no larger than 1500 bytes they will appear as ShortBlob (which isn't a sub-class of Blob) and String respectively and as Blob and Text if they're larger.
There seems to be no way to force Cloud Datastore API writing the expected type, hence you can't share the same datastore instance without modifying the AppEngine code base first to take care of the unexpected return types.

[leewisestamp]

Why was this closed? It appears to still be an issue in the latest version of the Python client library. The workaround mentioned above is not sufficient because subsequently exporting those entities (which previously had a TextProperty field and now have a StringProperty field in its place) breaks the BigQuery schema when importing said exports.