The C API is weakly typed

[markshannon]

The C type system is weak, compared to Java or Rust, but we should use it where possible.

The function PyTuple_Size() should be infallible; all tuples have a size.
Yet, it can fail if passed a non-tuple, which can only happen because it is weakly typed.
Like PyTuple_Size(PyObject *), many API functions take PyObject * when they should take more specific types.

Here are a few examples:
PyTuple_Size()
PyList_Append()
PyDict_GetItem()

The error handling of these functions is awful. PyTuple_Size() and PyList_Append() raise a SystemError, not a TypeError if passed the wrong type. PyDict_GetItem() just acts as if the item is not in the container, not raising at all.

[encukou]

Note that C type safety does nothing for interop with other languages, so if we push for this we should also invest in that area -- e.g. publish enough parseable info to make safe wrappers easy to generate (#7).

[markshannon]

Violating type rules is unsafe. Wrappers don't make things safer, just more convenient.
E.g.
Suppose we have PyTuple_Length(PyTupleObject *). It might crash if you pass a PyFloatObject * (or more likely produce a meaningless result).
If we add a "safe" wrapper PyTuple_Size(PyObject *), it still crashes if you call PyTuple_Size(7).

[encukou]

Yup, C wrappers are useless. Let me try to rephrase.

Languages without a C compiler only see exported DLL symbols (and whatever they manage to parse from C headers).
If we ditch runtime checks, we should make it easy as possible to generate whatever boilerplate they need to ensure type safety on their end.

[vstinner]

I would prefer to do the opposite: move to HPy-like opaque handles to fix issue #22.

[steve-s]

One way of looking at this is that we still do not want to expose the memory layout and have opaque pointers, but we can still use C types to provide bit better user experience and some level of type checking, but not a real type system that would allow us to, for example, be absolutely sure that only handles pointing to tuples ever make it to PyTuple_Size.

We did consider this in HPy and I think it would be useful, but we need to iron out the API and do more research into use-cases. For example, this would be also useful for user defined types since those are also opaque. When porting NumPy we noticed that by replacing the types like DType* with generic HPy we loose valuable semantic info and also the little bit of type checking that C gives us. We might want to introduce it eventually.

Example:

typedef struct _HPy_s { intptr_t _i; } HPy;
typedef struct _HPyType_s { HPy h; } HPyType;

HPyType (*ctx_Type)(HPyContext *ctx, HPy obj);
int (*ctx_TypeCheck)(HPyContext *ctx, HPy obj, HPyType type);

// Usage:
HPyType t = HPy_Type(ctx, some_handle);

// Downcasting, always safe
HPy generic_handle = t.h;

// Upcasting, could use some helper functions/macros
// It can be safe or unsafe. Anyone can do unsafe cast using just (void*)
// as intermediate step (putting aside the C standard...), so IMO there's
// no point in not providing helpers for that and trying to "hide" it from users
HPyType type = HPyCast(ctx, HPyType, generic_handle);
if (HPy_IsNull(type)) { ...it was not a type... }

// Still passing in ctx, so that maybe in debug mode we can do some checks(?)
HPyType i_know_what_i_am_doing = HPyUnsafeCast(ctx, HPyType, generic_handle);

[vstinner]

See also issue #37.

[vstinner]

For me, the question here is also how much the Python release build should help developers when they misuse the C API. Provide a cute well formatted exception? Kill the process with SIGABRT on an assertion error? Ignore silently the error and attempt to provide a behavior / result which prevent a crash?

A practical problem is that a debug build of Python is not widely available: see issue #36. I'm in favor of removing assertions in a release build and require developers to use a debug build, but there are practical issues with that.

[vstinner]

I would prefer to move the API towards type agnostic functions like PyObject_GetItem() instead of PyDict_GetItem(), PyObject functions have a good API: return a strong reference and raise the expected exception in case of error.

So far, I'm not convinced that outside CPython it's really worth it to use specialized functions like PyTuple_GetItem() or PyTuple_GET_ITEM(), instead of generic PySequence_GetItem(). Did anyone run a (micro or macro) benchmark to see the benefits on specialized functions? Is it really a bottleneck in terms of performance?

The other problems is that the C API has no clear tradeoff between performance and stability. "It depends" on the function, on the type, on the context. Some types have specialized functions. Some types have macros. Some others don't. How can users discover what's the best practice?

[gvanrossum]

If "nogil" is accepted, at least internally we may have PyDict_FetchItem which is like PyDict_GetItem but returns a strong reference. In that case we might as well adopt that API and gently start deprecating PyDict_GetItem. For example.

[markshannon]

It is worth noting that debug builds should still do the type checks, so we expect any type errors that slip pass the compiler to be caught in testing.

uintptr_t PyTuple_GetSize(PyTupleObject *t)
{
    assert(PyTuple_Check(t));
    return Py_SIZE(t);
}

[markshannon]

I would prefer to move the API towards type agnostic functions like PyObject_GetItem() instead of PyDict_GetItem().

It makes sense to have both, and to use the appropriate one.

PyObject *obj, PyDictObject *dict;
PyObject_GetItem(obj);   // 👍 
PyDict_GetItem(dict);   // 👍 
PyObject_GetItem((PyObject *)dict); // Pointlessly inefficient
PyDict_GetItem((PyDictObject *)obj); // Unsafe

[erlend-aasland]

IMO, we should try to come up with a guideline regarding this problem before adding new APIs. I don't have a strong preference for strongly or weakly typed APIs; there are pros and cons for each variant as I see it¹. I think @markshannon's proposal in #31 (comment) of having both weakly typed and strongly typed APIs makes sense, though.

Footnotes

1. I find Steve's comment interesting ↩