Create a job from code

[tim-lebedkov]

I try to start and control a job from code, but cannot find the appropriate functions and interfaces. Could you please provide an example?

Generally, the public interfaces are hard to find. Maybe you could add the library to go-search.org and to godoc.org. Another possibility would be to move the implementation to a sub-package.

[CaptainCodeman]

Yeah, this is definitely something that can be improved on. I'm always triggering it either from an cron task or a manual request right now which uses the handler here which should give you some pointers:

datastore-mapper/job_spec.go

Line 106 in 38bbd5e

func (m *mapper) startJobHandler(w http.ResponseWriter, r *http.Request) {

The idea is that the unique job name is auto-generated from the AppEngine request id. Things like the bucket (if used) and overriding the number of shards and queue to use are also extracted from the request so they can be set when kicking off jobs (e.g. using postman).

Most of that could be wrapped up into a much simpler call for triggering from code maybe something like:

InitiateJob(name string, queue string, shards int, bucket string) error

Would that make sense / be what you'd expect?

[tim-lebedkov]

re. the public interfaces: I found "godoc", which is enough for me.

re. my use case: I would like to start a long running job for a particular user (nbased.com) and show the progress and whether it ends successfully. I found "MP_job", but I would need a key to access an entry there (e.g. Key(MP_job, 'all_table.example1/55ce1022')). If the records in "MP_job" are not a part of the public API, then I would need a replacement.

re. "InitiateJob": seems OK, but it should return a handle to the running job so that I could get its status.

[CaptainCodeman]

My plan (when I get some free minutes) is to add a REST API that can provide status for jobs together with a nice Polymer UI to display and control things. These should probably be mirrored / back by some convenient functions to fetch the state directly in code.

In the meantime, and because it sounds like you want it programatically anyway, you should be able to use the Lifecycle methods to add hooks for any notifications as jobs / namespaces / shards / slices start and finish. You could update whatever status entities you want from those.

See: https://github.com/CaptainCodeman/datastore-mapper/blob/master/job_spec.go#L48