Skip to content

API improvement of use_softmax and zero_infinity#180

Open
NKNaN wants to merge 1 commit intobaidu-research:masterfrom
NKNaN:api-improve
Open

API improvement of use_softmax and zero_infinity#180
NKNaN wants to merge 1 commit intobaidu-research:masterfrom
NKNaN:api-improve

Conversation

@NKNaN
Copy link

@NKNaN NKNaN commented Mar 4, 2024

From a suggestion proposed by PaddlePaddle community, it is recommended to add use_softmax and zero_infinity options to warpctc, comparing to the Pytorch API of torch.nn.functional.ctc_loss(log_probs, targets, input_lengths, target_lengths, blank=0, reduction='mean', zero_infinity=False).

I made an attempt to add this two options in this PR, and the logic is shown the following:

  • use_softmax: The attribute of log_probs in Pytorch receives the output of logsoftmax operation. Therefore, it is suggested when the users want to use the output from logsoftmax operation as an input, we could omit the softmax operation in our code. The proposed improvement is when use_softmax=False, perform exponential operation on the input logits.

  • zero_infinity: It is hoped to zero the infinity cost and associated gradient when zero_infinity=True. So first to make infinity value of cost available, I omit the truncation process done to the logits. And then I do zero operation after the calculation of cost and grad for each batch of the input.

I also add the corresponding test for this two cases. (PS. The test of inf_test seems only to pass when the truncation processes are omited.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant